Avuja/Qwen3.8-27B-int4-AutoRound

Qwen3.8-27B INT4 AutoRound

A W4A16 (INT4 weight, FP16 activation) quantization of https://huggingface.co/Qwen/Qwen3.8-27B, produced with Intel's AutoRound.

Built as a drop-in model slot for the club-3090 dual-RTX-3090 vLLM stack — format-identical to https://huggingface.co/Lorbus/Qwen3.6-27B-int4-AutoRound, whose recipe this replicates for the new 3.8 release.

TL;DR

• Base: Qwen3.8-27B (27B dense VLM, hybrid Gated DeltaNet + full attention, Aug 2026)
• Quant: INT4 W4A16, group_size 128, symmetric
• Tool: auto-round 0.14.2 (default recipe, 200 iters, torch.compile)
• Size: 19.6 GB (down from ~54 GB BF16) — **2.7x reduction**
• MTP preserved: the entire Multi-Token Prediction head (mtp.fc + all mtp.layers.0 linears) is kept in BF16, enabling native speculative decoding in vLLM — 1.55x measured decode throughput (127 vs 82 tok/s, single stream, RTX 5090) at 0.69 greedy draft acceptance on vLLM 0.27.1
• Purpose: the stock club-3090 dual compose serves this repo with a two-line change

Quantization details

Field	Value
Base	Qwen/Qwen3.8-27B (BF16 — never quantize from the FP8 repo)
Method	AutoRound (intel/auto-round), default recipe
Scheme	W4A16 (4-bit weights, FP16 activations)
Bits	4
Group size	128
Symmetric	yes
Packing format	auto_round:auto_gptq
Unquantized layers	linear_attn.in_proj_a/b, the full MTP head (mtp.fc, mtp.layers.0.* linears), all LayerNorms/RMSNorms, router gates, vision tower, embeddings, lm_head
Calibration samples	128 (default)
Iterations	200
torch.compile	enabled
GPU used for quant	1× RTX 5090 (32 GB, SM120), low_gpu_mem_usage=True
auto-round version	0.14.2
Quant wall time	~1h 10m (peak VRAM 27.7 GB, peak RAM 23.7 GB)

Unquantized layers — why

• linear_attn.in_proj_a/b: low-rank projections in the Gated DeltaNet blocks; shapes not divisible by 32 (group_size), so AutoRound skips them. Tiny parameter fraction.
• mtp.fc: the Multi-Token Prediction fusion layer — see the fix below.
• Norms, routers, vision tower, embeddings: precision-sensitive and small; kept at base precision (BF16), matching the reference quant and the official FP8 repo's own exclusion list.

MTP fix — what's different from a vanilla AutoRound run

A plain auto-round run on this architecture packs mtp.fc as INT4. In that form, vLLM's Qwen3_5MTP loader skips the layer entirely (it expects fc.weight, finds fc.qweight), and MTP speculative decoding silently runs at 0% acceptance.

This build goes one step further than the 3.6 reference quant and keeps the entire MTP head in BF16:

• mtp.fc: excluded from quantization natively via --fp_layers mtp.fc (auto-round ≥ 0.14 supports this), so it was never packed. Recorded in quantization_config.extra_config as 16-bit fp.
• mtp.layers.0 linears (7): auto-round 0.14.2's exporter RTN-packs these (untuned) at export time; they were replaced with the original BF16 weights from the base model post-hoc (single-shard surgery in model_extra_tensors.safetensors) and marked as 16-bit fp in extra_config.

The whole head is only ~220 MB, so size impact is negligible. Result: MTP works out of the box. If you serve without --speculative-config, the BF16 MTP weights are simply unused and harmless.

Validation (RTX 5090, vLLM 0.27.1)

Gate	Result
Code generation	PASS — correct, coherent Python
MTP acceptance	PASS — 0.69 greedy / ~0.58 sampled (non-thinking) / ~0.47 (thinking, temp 1.0); fail mode ≈0 excluded. An A/B test (INT4 vs BF16 MTP internals) scored identically, so the gap vs older ~0.8–0.9 reports is a vLLM-version difference, not quant quality
Decode speedup	1.55x (900-token completion: 7.1 s with MTP vs 11.0 s without, single stream)
Vision	PASS — correct scene description of an unlabeled photo

Note: on a single desktop-shared 32 GB GPU, a 131072 ctx OOMs at KV-cache init (hybrid-GDN state pool scales with max_num_seqs); validated at 32768 ctx / 32 seqs. The dual-3090 TP=2 target rig has sufficient headroom for full context.

Known issue — vLLM 0.27.x GDN+MTP engine bug (not a checkpoint defect)

On vLLM 0.27.x, the hybrid-GDN + MTP speculative-decoding path is unstable when prefix caching and/or async scheduling are enabled: the engine either crashes (CUDA illegal memory access) or — reported at TP=2 — permanently collapses MTP draft acceptance to 0% after a long session once any sequence crosses position 32,768, surviving context resets until restart (vllm#52873, closed as not-planned; crash family #37035/#34948).

This was investigated against this checkpoint: the artifact is bit-identical to the upload (sha256-verified), and with --no-async-scheduling --no-enable-prefix-caching the BF16 MTP head drafted at 0.79–1.00 windowed acceptance through 37k+ generated tokens and past position 32,768 with no collapse. The failure modes are engine-side, not weight-side; checkpoints with quantized MTP heads hit the same engine bugs.

Serving guidance: with the MTP drafter enabled, add --no-async-scheduling --no-enable-prefix-caching (cost: shared-prefix re-prefill). Without the drafter (num_speculative_tokens=0), prefix caching and async scheduling are safe. Capping --max-model-len below 32,768 does not make the unstable path safe — crashes were observed at ~21k ctx.

Quantization environment

python 3.12 · auto-round 0.14.2 · transformers 5.15.0 · torch 2.13.0+cu130 · quantized from a local BF16 checkout of Qwen/Qwen3.8-27B

---

Update 2026-09-06 — drop-in status for the club-3090 stack, and the collapse reproduced (engine-side)

TL;DR for anyone serving this checkpoint on dual RTX 3090s: the weights are fine — the vLLM 0.27.x GDN+MTP bug above is what kills long sessions, and on 2026-09-06 it was finally reproduced on this exact artifact, as a hard crash. Until vllm#50021 lands in a release you can pin, treat this checkpoint as a drop-in model slot only at MTP n=3 (or with the drafter off). Do not run MTP n=4 on stock vLLM 0.27.1.

What the re-test showed

Serving config under test: W4A8 int8 activations (see recipe below), prefix caching ON, async scheduling ON, fp8 KV, FlashInfer, TP=2 — i.e. the club-3090 dual-fast feature set. Identical growing multi-turn conversation each run (the same driver as the August investigation, greedy, thinking off):

Run	MTP n	Outcome
A	3	Clean — 39,900 cumulative generated tokens (2.7× the reported ~14.5k trigger), ctx to 44k, windowed acceptance 0.87–0.98 throughout, no errors
B	4	Engine crash at ~14,950 cumulative generated tokens — CUDA error: an illegal memory access was encountered at gpu_model_runner.py:3875/3877 synchronize_input_prep, Xid 31 MMU fault on both GPUs simultaneously, acceptance still healthy (0.91–0.98) at death

Run B matches, point for point, the crash that club-3090 filed as #1052 against this checkpoint's deployment: same threshold (~14.5k cumulative gen with async scheduling), same signature (dual Xid 31), same fault site (synchronize_input_prep — the exact frame their vllm-gdn-mtp-async-spec-order patch gates, upstream vllm#50021, still open). The same artifact at n=3, and at n=4 on W4A16 in August, ran past every threshold with the drafter healthy — so the discriminator is the serving configuration, not the weights.

Why the "checkpoint-specific" attribution doesn't hold: the buggy code derives an array index from the per-step accepted-token count without bounds-checking it. A checkpoint only changes the rhythm of those counts (a BF16 drafter head and a quantized one draft with different acceptance patterns), which moves where the race tends to fire — it cannot cause or prevent it. club-3090's own tracker agrees on the substance: the crash "reproduc[es] across two unrelated quant formats" (their #838), their replacement checkpoint later showed the same decay signature (#1096), and the fix they ship for n=4 is an engine patch, not a checkpoint. Frozenlock differs from this quant mainly in quantizing the MTP head and in calibration data — neither is a plausible mechanism for a bounds-check bug in vLLM's spec-decode path, and neither survives the A/Bs above as an explanation.

The recipe that works (validated 2026-09-06, 2×3090 TP=2, vLLM 0.27.1)

export VLLM_MARLIN_INPUT_DTYPE=int8        # W4A8: int8 activations on the Marlin int4 GEMMs
export VLLM_CACHE_ROOT=~/.cache/vllm-w4a8  # torch.compile cache cannot key on W4A8 — isolate it
vllm serve /path/to/this-checkpoint \
  --quantization auto_round --dtype float16 \
  --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 16384 --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-prefix-caching --attention-backend flashinfer \
  --override-generation-config '{"temperature":0.7,"top_p":0.80,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

• W4A8 needs two vLLM source patches on 0.27.1 — stock silently ignores the toggle for auto-round checkpoints, and the int8 kernel mis-reads AutoRound's negative group scales (garbage output). Both are small, idempotent, anchor-checked Python patchers, available as club-3090's w4a8-int8-act (patches: thread input_dtype through the inc route; fold negative scales into the weight codes at load). Measured single-variable A/B on this checkpoint: prefill 1359 → 1860 tok/s @10K (+36.9%), TTFT −27%, decode neutral, KV pool −14% (503,785 → 434,954 tokens). Rollback = unset the env var; the patches are runtime no-ops without it.
• Crash avoidance: n=3 is the tested depth. If you need n=4, either apply club-3090's vllm-gdn-mtp-async-spec-order + vllm-flashinfer-decode-pin patches (their fix for exactly the fault site above), or run with --no-async-scheduling (their #1059 A/B: eliminates the crash at ~0% single-stream cost). SPEC_N=0 is the always-safe fallback.
• Transient expandable_segments allocator OOM-retries during weight load at util 0.95 are normal under W4A8 (the fold repacks per layer); if a boot ever dies there, lower --gpu-memory-utilization a notch.

Drop-in note for club-3090 compose users: this checkpoint works as the model slot with the usual two-line change, but set SPEC_N=3 on their dual-fast tier (its n=4 default assumes their vendored spec-decode patches). Artifacts for the 2026-09-06 re-test (per-turn logs, CSVs, crash journal with kernel Xid lines) are available on request.