garnermccloud/Qwen3.8-Flash-Next-NVFP4-SSD-Stream

Qwen3.8 Flash-Next NVFP4 - SSD Stream

Run Qwen3.8 Flash-Next with 48 GB less RAM. Keep the speed.

This is RadixArk/Qwen3.8-Flash-Next-NVFP4, prepared for sglang-ssd-stream. RadixArk produced the NVFP4 checkpoint from Qwen3.8 Flash-Next; this layout loads its 48 GB lookup table from a local SSD instead of occupying system RAM. The required data is fetched while the GPU is already working.

What this model is

Source checkpoint	RadixArk/Qwen3.8-Flash-Next-NVFP4
Exact source revision	7b719225242aacd3dbd3f9407468c2ee9a9d2594
Base architecture	Qwen/Qwen3.8-Flash-Next
Serving engine	SGLang with sglang-ssd-stream
Prepared download	About 140 GiB

The source checkpoint uses ModelOpt NVFP4 W4A4 for the routed experts. Its attention, shared experts, embeddings, LM head, vision components, and MTP weights remain BF16; the predictive lookup table is FP8.

This prepared checkpoint does not retrain, fine-tune, or requantize the model. It makes one serving-oriented layout change:

1. The 128 FP8 lookup-table shards are removed from the normal model load.
2. Their rows are stored in one read-only 51,200,245,760-byte SSD sidecar.
3. ssd-stream.json records its exact shape, source revision, and SHA-256.
4. The remaining model and native MTP weights are preserved from the source.

Sidecar SHA-256: b070f9644adf93794d8a1030584ab705809387e64396a9327a68fa3a3a6666b3

Performance

RTX PRO 6000 Blackwell

Matched tests on one RTX PRO 6000 Blackwell used native MTP, CUDA graphs, and a 1,024-token completion:

RTX PRO 6000 result	Value
GPU memory	96 GB
Normal lookup-table RAM	47.68 GiB
SSD Stream working RAM	About 64 MiB
RAM returned	About 47.6 GiB
Normal RAM-loaded speed	148.5-156.2 tok/s
SSD Stream speed	164.7 tok/s

The measured result is the point of SSD Stream: removing the table from RAM did not make it the decode bottleneck. Content-heavy requests dominated by unseen, random lookup rows measured 126-137 tok/s.

The RTX acceptance suite also passed structured tool calls, unrelated images, a 120,043-token retrieval request, native MTP, CUDA graph replay, restart reuse, and a sustained alternating workload without swap, OOM, restart, or host-memory growth.

DGX Spark (experimental)

DGX Spark result	Value
Unified memory	128 GB
Normal lookup-table memory	47.68 GiB
SSD Stream working memory	About 64 MiB
Unified memory returned	About 47.6 GiB
Normal RAM-loaded speed	Pending hardware run
SSD Stream speed	Pending hardware run

The Linux aarch64 wheel and automatic GB10 profile are included. It pins SGLang's SM121 QSA implementation and starts with 262K context, BF16 KV, FP32 model state, native MTP 3/1/4, decode CUDA graphs, and one request at a time. The wheel and Rust io_uring reader pass the complete native ARM64 test suite. Full model startup, long-context behavior, and performance still need to be measured on the rented DGX Spark, so this profile remains experimental.

Run it

Use Linux with a Blackwell GPU and a local SSD:

curl -LsSf https://raw.githubusercontent.com/garnermccloud/sglang-ssd-stream/main/install.sh | sh
~/.local/bin/sglang-ssd-stream serve

The first launch downloads and verifies this snapshot, then starts SGLang's OpenAI-compatible API at http://127.0.0.1:30000/v1.

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.8-Flash-Next-NVFP4-SSD-Stream",
    "messages": [{"role": "user", "content": "Reply with: SSD Stream works"}],
    "max_tokens": 32
  }'

The CLI selects the validated single RTX PRO 6000 profile or the experimental single DGX Spark profile automatically.

Model capabilities

• Text, image, and video input with text output
• Reasoning and structured tool calling
• Native MTP speculative decoding
• Up to 262K context in the source architecture
• OpenAI-compatible chat-completions API through SGLang

Why Hugging Face shows 68B parameters and 8-bit precision

Those automatic labels describe visible storage tensors, not this model's logical architecture or its primary weight precision.

• The 68B figure counts about 60.4B packed uint8 storage elements plus 8.0B BF16 elements in the Safetensors files. Each packed byte can hold two NVFP4 values.
• The 51.2B FP8 lookup-table elements live in the SSD sidecar, so Hugging Face's Safetensors scanner does not count them.
• The 8-bit label comes from those storage dtypes. The routed-expert weights are NVFP4 W4A4; selected model components remain BF16; the streamed table is FP8.

The source model describes the architecture as approximately 180B logical parameters in total. The 68B header is therefore not a comparable parameter count.

Technical details

The model performs 16 deterministic table lookups per generated token. Each FP8 row is 160 bytes. SSD Stream starts those reads before the model reaches the block that consumes them, allowing SSD I/O to overlap GPU computation.

The extension:

1. copies row IDs asynchronously into pinned host memory;
2. deduplicates the exact 4 KiB filesystem pages required by those rows;
3. submits concurrent reads through a Rust io_uring engine;
4. restores rows in their requested order;
5. converts staged FP8 data on a separate CUDA stream; and
6. synchronizes only when a read outlasts the overlapping GPU work.

It reserves a 32 MiB registered page pool and two 16 MiB pinned staging buffers. It does not reserve a multi-gigabyte private cache. Native MTP, multimodal input, structured tools, and CUDA graph replay remain enabled.

sglang-ssd-stream is an independent extension for SGLang. It is not an official SGLang component.

License and attribution

The model weights retain the source model's license and terms. See:

• Qwen/Qwen3.8-Flash-Next
• RadixArk/Qwen3.8-Flash-Next-NVFP4
• sglang-ssd-stream, Apache-2.0