Instructions to use urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ

SGLang

How to use urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ with Docker Model Runner:
```
docker model run hf.co/urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ
```

Agnes-3.0-Flash Preview W4A16 (AutoRound GPTQ)

4-bit weight-only (W4A16) AutoRound GPTQ quantization of Agnes-AI/Agnes-3.0-Flash - the 33B open-weights Preview checkpoint: multimodal (text/image/video), hybrid delta + global attention, 262 144-token context. 22.9 GB on disk against ~66 GB for the BF16 original.

This is an unofficial, community-produced quantization. All credit for the base model goes to Agnes AI - see their model card for benchmarks and citation. Note their version clarification: these are the open-weight Preview weights, not the production/API Agnes 3.0 Flash checkpoint.

A build containing vLLM PR #57003 is required to serve this checkpoint with vLLM. Current released builds do not include the Agnes architecture. The tested vLLM path is text-only with --language-model-only and --trust-remote-code.

Model Details


Base model	Agnes-AI/Agnes-3.0-Flash (Preview, 33B)
Architecture	`AgnesForConditionalGeneration` (custom code, `trust_remote_code=True` required)
Text layers	72 - 54 `agnes_delta_attention` + 18 `agnes_global_attention` (every 4th layer)
Global attention	24 query heads / 4 KV heads, head_dim 256, gated output, partial rotary 0.25, interleaved mRoPE
Delta attention	16 key heads / 48 value heads, head_dim 128, conv kernel 4, swish output gate
FFN	intermediate 17408 + parallel FFN 2048, hidden size 5120
Vision tower	27 layers, hidden 1152, patch 16, spatial merge 2 → 5120
MTP	1 layer, shared embeddings
Context length	262 144 tokens, RoPE θ = 10,000,000
Vocab size	248 320

Quantization Details

Quantized with Intel AutoRound v0.16.0 on Intel Arc Pro B70 (XPU) hardware.

Setting	Value
Scheme	W4A16 (4-bit weights, 16-bit activations)
Format	GPTQ (`auto_gptq`-compatible), symmetric, `desc_act=false`
Group size	64
Iterations	600
Calibration dataset	`opencode-instruct`
`lm_head`	Not quantized

What is and isn't in INT4:

Component	Precision	Size
Text decoder: `global_attn` q/k/v/o, `delta_attn` in_proj_qkv / in_proj_z / out_proj, `mlp` and `mlp.parallel_ffn` gate/up/down (666 linears)	INT4 g64	16.0 GB
`embed_tokens` + `lm_head`	BF16	5.1 GB
Vision tower (333 tensors)	BF16	0.9 GB
MTP block (15 tensors, in `model_extra_tensors.safetensors`)	BF16	0.9 GB
`delta_attn.in_proj_a` / `in_proj_b` (108 × `[48, 5120]`), `conv1d`, norms	BF16	0.1 GB

AutoRound excluded the in_proj_a/in_proj_b pairs on its own: they are recorded as -: dynamic keys in quantization_config. The vision tower and the MTP block were never entered for quantization and carry no such keys, so a runtime that builds every nn.Linear as GPTQ straight from quantization_config has to skip them explicitly.

Reproduce with (transformers==5.13.1, since 5.17 renamed LAYER_TYPE_CACHE_MAPPING to DYNAMIC_LAYER_TYPE_MAPPING and the bundled remote code expects the old name):

auto-round \
  --model Agnes-AI/Agnes-3.0-Flash \
  --scheme W4A16 \
  --group_size 64 \
  --iters 600 \
  --dataset "opencode-instruct" \
  --format auto_gptq \
  --low_gpu_mem_usage \
  --output_dir <output_dir>

Usage

vLLM (text only)

Use a vLLM build that includes PR #57003. Until the PR is merged and released, stock vLLM builds will not recognize AgnesForConditionalGeneration.

vllm serve urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ \
  --trust-remote-code \
  --language-model-only \
  --dtype bfloat16 \
  --max-model-len 2048

The text-only AutoRound GPTQ checkpoint was smoke-tested on Intel Arc Pro B70 with XPU Graph. Image and video serving have not been evaluated. The text-only MTP-2 benchmark below passed its coherence test.

vLLM MTP-2 benchmark

Enable two MTP draft tokens on the vLLM server with --speculative-config '{"method":"mtp","num_speculative_tokens":2}'. The 4096-token prompt and 256-token generation require a --max-model-len above 4352; increase the 2048-token limit in the basic example above before running this benchmark. On the server host, run:

uvx llama-benchy --base-url http://0.0.0.0:8000/v1 \
  --model urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ \
  --pp 4096 --tg 256 --concurrency 1 --depth 1 \
  --no-cache --exact-tg --latency-mode generation

Measured on 2026-09-15 at 15:58:38 with llama-benchy 0.4.0 and three runs per test. Coherence passed; average generation latency was 195.95 ms.

test	t/s	peak t/s	ttfr (ms)	est ppt (ms)	e2e ttft (ms)
pp4096 @ d1	1615.44 ± 2.37	—	2765.12 ± 3.72	2569.17 ± 3.72	2765.12 ± 3.72
tg256 @ d1	41.16 ± 5.64	52.33 ± 2.87	—	—	—

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id, subfolder = "urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ", "Agnes-3.0-Flash-w4g64"
tok = AutoTokenizer.from_pretrained(model_id, subfolder=subfolder, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, subfolder=subfolder, trust_remote_code=True, device_map="auto"
)

msgs = [{"role": "user", "content": "Explain gated delta attention in three sentences."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, temperature=1.0, top_p=0.95, top_k=20)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Images and video go through the bundled processor (AutoProcessor, also remote code) exactly as in the base model card - the vision tower is untouched by quantization.

Reasoning effort

The shipped chat_template.jinja accepts xhigh (default), medium, low, plus enable_thinking=False:

ids = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort="medium",
                              return_tensors="pt")

reasoning_effort="high" raises - the base card's wording predates this template.

Sampling

temperature=1.0, top_p=0.95, top_k=20, max_tokens ≥ 2000, per the checkpoint's generation_config.json.

Limitations

Requires trust_remote_code=True (custom architecture, inherited from the base model).
vLLM serving requires a build containing PR #57003; text-only XPU Graph serving has been smoke-tested and MTP-2 benchmarked. Image and video serving have not been evaluated. SGLang serving has not been tested with this checkpoint.
Only the text decoder is in INT4 embeddings, lm_head, vision tower and MTP stay BF16, so memory savings are ~3× rather than ~4×.
No accuracy evaluation has been run on this quantization yet; treat quality as unverified.

License

Apache 2.0, inherited from the base model.

Downloads last month: 11

Safetensors

Model size

7B params

Tensor type

I32

BF16

Model tree for urakozz/Agnes-3.0-Flash-W4A16-AutoRound-GPTQ

Base model

Agnes-AI/Agnes-3.0-Flash

Quantized

(20)

this model