eaddario/Olmo-3-7B-Instruct-GGUF

Experimental global target bits‑per‑weight quantization of allenai/Olmo-3-7B-Instruct

Using non-standard (forked) LLaMA C++ release b7540 for quantization.

Original model: allenai/Olmo-3-7B-Instruct

From the original model creators:

Model Details

Model Card for Olmo 3 7B Instruct

We introduce Olmo 3, a new family of 7B and 32B models both Instruct and Think variants. Long chain-of-thought thinking improves reasoning tasks like math and coding.

Olmo is a series of Open language models designed to enable the science of language models. These models are pre-trained on the Dolma 3 dataset and post-trained on the Dolci datasets. We are releasing all code, checkpoints, logs (coming soon), and associated training details.

The core models released in this batch include the following:

Stage	Olmo 3 7B Think	Olmo 3 32B Think	Olmo 3 7B Instruct
Base Model	Olmo-3-7B	Olmo-3-32B	Olmo-3-7B
SFT	Olmo-3-7B-Think-SFT	Olmo-3-32B-Think-SFT	Olmo-3-7B-Instruct-SFT
DPO	Olmo-3-7B-Think-DPO	Olmo-3-32B-Think-DPO	Olmo-3-7B-Instruct-DPO
Final Models (RLVR)	Olmo-3-7B-Think	Olmo-3-32B-Think	Olmo-3-7B-Instruct

⚠️ PLEASE READ THIS BEFORE USING THESE EXPERIMENTAL VERSIONS! ⚠️

An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. There are many approaches to accomplish this, including architecture simplification and knowledge distillation, but my focus has been primarily on quantization and pruning.

The method to produce these experimental versions involves using a custom version of llama-imatrix to generate an imatrix including the mean activations, and a custom version of llama-quantize, which computes a per-tensor weighted mean squared quantization error and a bias/projection term (if the imatrix includes activations), to automatically select the lowest error quantization recipe that achieves a global target bits‑per‑weight (bpw). More details on the implementation and test results here

There are two pull requests (#14891 & #15550) to merge these changes back into the core llama.cpp project. This may or may not ever happen so, until then, the modified versions will be available on GitHub.

For testing and comparison, I use models produced by Bartowski (see credits below) and Unsloth (Daniel and Michael Han do some really interesting stuff!) but when they don't provide versions of the required model, tests and comparisons are against standard quantization obtained by simply running llama-quantize with no further optimizations.

All experimental versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration. In llama.cpp, an imatrix is a calibration file derived from running representative text through the model and collecting activation statistics. It is used to weight quantization error so that error in more “important” directions (as estimated from activations) is penalized more heavily.

The process to generate these models is roughly as follows:

1. Convert the original model's safetensors to GGUF F16*
2. Estimate the Perplexity score for the F16 model (baseline) using the wikitext-2-raw-v1 dataset, and save the logits
3. Generate an imatrix from the most appropriate calibration dataset
4. Quantize the baseline model targeting a bpw average, allocating more bits to tensors estimated to matter more (e.g. llama-quantize --target-bpw 4.5678 --keep-bpw-state --imatrix imatrix.gguf baseline-model-F16.gguf 12)
5. Quantize the baseline model targeting a bpw average, treating each tensor equally instead of prioritizing some (e.g. llama-quantize --target-bpw 4.5678 --no-importance --keep-bpw-state --imatrix imatrix.gguf baseline-model-F16.gguf 12)
6. Calculate Perplexity, KL Divergence, ARC (Easy+Challenge), HellaSwag, MMLU, Truthful QA and WinoGrande scores for each quantized model
7. Keep version with the best 𝜌PPL scores (i.e. highest Cor(ln(PPL(Q)), ln(PPL(base))))
8. Repeat until all desired quants are created

*BF16 would be preferred, but F16 performs better on Apple's GPUs

Advantages and disadvantages of the global target bits‑per‑weight quantization process

Advantages

1. Target arbitrary size models

   • When specifying --target-bpw 4.5678 for instance, the algorithm will produce a model (nearly) exactly of that size, which is very useful for maximizing VRAM usage. In a system with 24GB VRAM and a 70B model, standard quants might produce a 16.8GB file (too small, quality left on table) or a 24.1GB file (won't fit). This approach can generate a 23.85GB file to utilize the hardware fully.

2. Data-driven mixed precision often can improve quality at fixed size

   • Instead of using hardcoded heuristics (e.g. make attn_v Q5_K for a 70B model), that may be sub‑optimal for a given architecture or size, the quantization mix is determined by the actual error sensitivity of the specific model's weights. This, in practice, often yields a better quality/size trade-off, especially in aggressive quantization scenarios (1.5 to 3.5 bpw), or for unusual architectures.

   • Please note: llama.cpp’s heuristics have been tuned across many models and are highly optimized; although the target bpw method produces better quality often (>75% based on tests with 130 models from 11 different families), it can also lose in surprising cases.

3. Allows better like-for-like comparisons between models and families

   • Standard llama.cpp quantization uses hardcoded rules like: "use Q4_K_M, except bump some tensors up/down, except fall back if incompatible, except keep some tensors unquantized..." and for that reason, two different models quantized with the same Q4_K_M type can end up with very different bpw (e.g. 4.75 and 4.30).

   • All things being equal, the performance of a model is usually proportional to its overall bpw size; models with a higher bpw tend to perform better than lower bpw models. Since model A has simply been given more bits, it will typically perform better (lower perplexity, better eval scores, etc.) even if the underlying quantization method is identical. That makes comparing the performance not a controlled experiment, because the comparison is between models with different effective compression ratios.

   • --target-bpw tries to address that by making the experiment more controlled: each model gets quantized to land on (approximately) the same global byte budget, so that the models' performance differences are more attributable to architecture/training differences, quantization error behaviour at the same compression ratio, optimizer’s allocation decisions, etc.

Disadvantages

1. Quantization process is significantly slower than standard

   • This approach can take 5x-10x longer as it quantizes a sample of most tensors into 15 different formats, dequantizes them back to floats, computes error diffs, and selects the best size/error option that fits the global bpw budget.

   • However, the --keep-bpw-state option will save the above-mentioned computations to disk so that future quantizations, in the permissible bpw range for the same model, can be generated at normal speed. It also allows to interrupt the computation process and resume it at a later time.

2. The optimization target is only a proxy for the model's performance quality

   • The process minimizes a per-tensor estimated error computed from sampled rows, not actual perplexity or divergence of output distributions (a future version may address this). Since errors interact nonlinearly across layers, there are no guarantees it will select the best possible quantization recipe subject to the bpw size constraint.

   • Furthermore, the process can operate in two modes: giving priority to important tensors (default) or treating each tensor equally (setting the --no-importance option). To my knowledge, there is no computationally feasible way to determine ahead of time which modality will yield better results, and two runs per model may be needed to obtain the best quality, but the default mode usually wins.

3. An imatrix with activations data is required for best results

   • Activation data is required to compute the bias factor (i.e. the systematic error projected onto activation directions). If the imatrix file does not contain activation data, the quantization recipe will likely be sub-optimal.

---

Models

Bits per weight, size, perplexity and KL Divergence scores

Model	BPW	Size (GB)	μPPL	𝜌PPL	μKLD	Same Top-P
Olmo-3-7B-Instruct-F16	16.0012	14.6	13.937178 ±0.118969	100%	N/A	N/A
Olmo-3-7B-Instruct-IQ1_L	1.7497	1.6	39.075286 ±0.317208	69.44%	1.939807 ±0.004377	42.288 ± 0.130
Olmo-3-7B-Instruct-IQ2_S	2.2496	2.1	19.043598 ±0.155349	83.01%	1.004207 ±0.003056	58.152 ± 0.130
Olmo-3-7B-Instruct-IQ2_XS	2.1243	1.9	20.990696 ±0.170417	80.45%	1.159149 ±0.003321	55.194 ± 0.131
Olmo-3-7B-Instruct-IQ2_XXS	1.9996	1.8	22.626560 ±0.184954	78.85%	1.261045 ±0.003530	53.370 ± 0.131
Olmo-3-7B-Instruct-IQ3_XXS	3.0000	2.7	14.554863 ±0.119633	92.07%	0.469920 ±0.001691	70.247 ± 0.120
Olmo-3-7B-Instruct-Q2_K	2.4998	2.3	17.060276 ±0.140060	86.62%	0.804364 ±0.002579	61.791 ± 0.128
Olmo-3-7B-Instruct-Q3_K_L	3.7498	3.4	14.399330 ±0.123325	97.44%	0.154932 ±0.000660	82.143 ± 0.101
Olmo-3-7B-Instruct-Q3_K_S	3.2498	3.0	14.898797 ±0.127742	94.17%	0.347211 ±0.001377	74.367 ± 0.115
Olmo-3-7B-Instruct-Q3_K	3.4995	3.2	14.147290 ±0.119956	96.16%	0.229317 ±0.000936	78.750 ± 0.108
Olmo-3-7B-Instruct-Q4_K_S	4.2495	3.9	13.952606 ±0.118807	98.40%	0.092717 ±0.000434	86.113 ± 0.091
Olmo-3-7B-Instruct-Q4_K	4.4995	4.1	13.944686 ±0.119057	99.04%	0.055153 ±0.000250	88.999 ± 0.082
Olmo-3-7B-Instruct-Q4_K_M-bartowski	4.8978	4.5	13.893317 ±0.116608	98.24%	0.101048 ±0.000448	85.230 ± 0.093
Olmo-3-7B-Instruct-Q4_K_M-unsloth	4.8978	4.5	13.891710 ±0.116580	98.24%	0.101428 ±0.000453	85.156 ± 0.094
Olmo-3-7B-Instruct-Q4_K_M-bpw	4.8972	4.5	13.976423 ±0.119540	99.32%	0.039396 ±0.000185	90.677 ± 0.077
Olmo-3-7B-Instruct-Q5_K_S	5.2495	4.8	14.019974 ±0.120138	99.49%	0.028791 ±0.000143	91.969 ± 0.072
Olmo-3-7B-Instruct-Q5_K	5.4996	5.0	14.020931 ±0.120185	99.64%	0.019615 ±0.000103	93.332 ± 0.066
Olmo-3-7B-Instruct-Q6_K	6.4985	5.9	13.970693 ±0.119646	99.84%	0.007703 ±0.000045	95.806 ± 0.053
Olmo-3-7B-Instruct-Q8_0	8.4980	7.8	13.993324 ±0.119895	99.96%	0.000852 ±0.000005	98.560 ± 0.031

ARC, HellaSwag, MMLU, Truthful QA and WinoGrande scores

Scores generated using llama-perplexity with 750 tasks per test, and a context size of 768 tokens.

For the test data used in the generation of these scores, follow the appropriate links: HellaSwag, ARC, MMLU, Truthful QA and WinoGrande

Model	ARC	HellaSwag	MMLU	Truthful QA	WinoGrande	Avg Score
Olmo-3-7B-Instruct-IQ1_L	49.6000	52.1333	30.0000	32.2667	56.4000	44.0800
Olmo-3-7B-Instruct-IQ2_S	57.3333	64.4000	32.2667	36.2667	60.9333	50.2400
Olmo-3-7B-Instruct-IQ2_XS	56.4000	60.4000	32.1333	37.6000	60.2667	49.3600
Olmo-3-7B-Instruct-IQ2_XXS	53.4667	59.2000	30.5333	35.7333	60.1333	47.8133
Olmo-3-7B-Instruct-IQ3_XXS	64.6667	73.0666	33.6000	38.8000	63.7333	54.7733
Olmo-3-7B-Instruct-Q2_K	61.7333	68.6666	33.0667	36.6667	64.9333	53.0133
Olmo-3-7B-Instruct-Q3_K_L	66.0000	75.7333	33.3333	39.2000	68.9333	56.6400
Olmo-3-7B-Instruct-Q3_K_S	64.5333	74.6666	34.1333	39.3333	65.8667	55.7066
Olmo-3-7B-Instruct-Q3_K	66.2667	76.5333	34.5333	37.7333	66.8000	56.3733
Olmo-3-7B-Instruct-Q4_K_S	66.1333	76.2666	34.2667	38.4000	69.0667	56.8267
Olmo-3-7B-Instruct-Q4_K	66.6667	76.8000	34.1333	39.7333	66.4000	56.7467
Olmo-3-7B-Instruct-Q4_K_M-bartowski	66.1333	76.4000	34.1333	39.0667	68.2667	56.8000
Olmo-3-7B-Instruct-Q4_K_M-unsloth	65.8667	77.2000	33.4667	39.4667	67.2000	56.6400
Olmo-3-7B-Instruct-Q4_K_M-bpw	66.8000	76.4000	33.6000	38.9333	67.3333	56.6133
Olmo-3-7B-Instruct-Q5_K_S	66.8000	76.8000	34.0000	38.8000	67.7333	56.8267
Olmo-3-7B-Instruct-Q5_K	67.0667	76.1333	34.6667	38.9333	68.2667	57.0133
Olmo-3-7B-Instruct-Q6_K	67.6000	76.9333	34.6667	39.0667	68.4000	57.3333
Olmo-3-7B-Instruct-Q8_0	67.8667	77.7333	34.2667	38.4000	67.7333	57.2000

Tokens per second benchmarks

Scores generated using llama-bench. Standard (llama-quantize with no optimization) Q4_K_M quantization included for comparison.

model	size	params	backend	threads	test	t/s
Olmo-3-7B-Instruct-Q4_K_M-bpw	4.16 GiB	7.30 B	Metal,BLAS	12	pp512	917.52 ± 1.54
Olmo-3-7B-Instruct-Q4_K_M-bpw	4.16 GiB	7.30 B	Metal,BLAS	12	tg128	76.58 ± 0.83
Olmo-3-7B-Instruct-Q4_K_M-bpw	4.16 GiB	7.30 B	Metal,BLAS	12	pp1024+tg1024	114.95 ± 1.99
Olmo-3-7B-Instruct-Q4_K_M-bartowski	4.16 GiB	7.30 B	Metal,BLAS	12	pp512	872.81 ± 10.09
Olmo-3-7B-Instruct-Q4_K_M-bartowski	4.16 GiB	7.30 B	Metal,BLAS	12	tg128	85.67 ± 0.27
Olmo-3-7B-Instruct-Q4_K_M-bartowski	4.16 GiB	7.30 B	Metal,BLAS	12	pp1024+tg1024	127.90 ± 0.78
Olmo-3-7B-Instruct-Q4_K_M-unsloth	4.16 GiB	7.30 B	Metal,BLAS	12	pp512	932.84 ± 2.02
Olmo-3-7B-Instruct-Q4_K_M-unsloth	4.16 GiB	7.30 B	Metal,BLAS	12	tg128	83.70 ± 2.34
Olmo-3-7B-Instruct-Q4_K_M-unsloth	4.16 GiB	7.30 B	Metal,BLAS	12	pp1024+tg1024	126.84 ± 0.66

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the original model the better, thus the closest to 0 the better.

AI2 Reasoning Challenge (ARC): a benchmark to evaluate the ability of AI models to answer complex science questions that require logical reasoning beyond pattern matching.

HellaSwag: the Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations (bit of a mouthful!) is a benchmark designed to test commonsense natural language inference. It requires the model to predict the most likely ending of a sentence.

MMLU: the Massive Multitask Language Understanding evaluates LLMs’ general knowledge and problem-solving abilities across 57 subjects, including elementary mathematics, US history, computer science, and law.

Truthful QA: evaluates how well LLMs generate truthful responses to questions. It identifies whether AI models can avoid generating false or misleading information, particularly in areas where human knowledge is prone to misconceptions.

Winogrande: based on the Winograd Schema Challenge, is a natural language understanding task requiring models to resolve ambiguities in sentences involving pronoun references.

Credits

LLaMa C++ has a large and vibrant community of contributors (~1,200 last time I checked) that actively maintain and extend its functionality, adding new models and architectures almost as fast as they appear. Considering the breakneck speed at which the AI/ML field is advancing, this alone is a remarkable feat!

While I'm grateful to all contributors, I want to recognise three in particular:

• Colin Kealty, for the many contributions and for being one of the best sources of high quality quantized models available on Hugging Face
• Georgi Gerganov for his amazing work with llama.cpp and the ggml/gguf libraries
• Iwan Kawrakow for being one of the key authors behind the many quantization algorithms and the imatrix functionality.