Qwen3-4B-Autoround-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Qwen3-4B-Autoround-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

  • Model architecture: Qwen3 4B
  • Quantization: A16W4 (BF16 activations, INT4 language weights), with mixed-precision exceptions retained from the prequantized checkpoint.
  • Pre-quantization: The source checkpoint was pre-quantized with AutoRound before compilation.
  • Maximum context length: 4096 tokens
  • Source model: Qwen/Qwen3-4B

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, four samples per input length, and max_new_tokens=128. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Input tokens Mean TTFT (seconds) Mean generation rate (tokens/second)
128 0.14 23.82
256 0.25 23.64
512 0.51 22.93
1024 1.06 21.61
2048 2.46 16.87
3072 4.92 16.01

Prerequisites

To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.

Installation and Deployment

Download the precompiled model directly on Modalix:

hf download simaai/Qwen3-4B-Autoround-a16w4 --local-dir /media/nvme/llima/models/Qwen3-4B-Autoround-a16w4

The model is installed under:

/media/nvme/llima/models/Qwen3-4B-Autoround-a16w4

Alternatively, download it on a host and copy it to Modalix:

hf download simaai/Qwen3-4B-Autoround-a16w4 --local-dir Qwen3-4B-Autoround-a16w4
scp -r Qwen3-4B-Autoround-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/

Usage

Validate the model directly on Modalix:

llima run Qwen3-4B-Autoround-a16w4

For application integration, see GenAI with LLiMa.

Recompilation

To build for an 8,192-token context, recompile the matching source checkpoint with this repository's compile.sh in a compatible SiMa Model Compiler and LLiMa compilation environment. Change --max_num_tokens 4096 to --max_num_tokens 8192 in the script, then follow its source-checkpoint and output-directory instructions.

The context budget includes input and generated tokens. A larger context increases KV-cache memory requirements and must fit the target device. The downloadable artifacts use a 4,096-token context; any performance measurements in this card refer to that build.

Limitations

  • This repository contains compiled Modalix artifacts and cannot be loaded with transformers.AutoModel.
  • Quantization can cause minor differences from the full-precision source model.

Resources

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for simaai/Qwen3-4B-Autoround-a16w4

Finetuned
Qwen/Qwen3-4B
Finetuned
(1,275)
this model

Collection including simaai/Qwen3-4B-Autoround-a16w4