s-conia commited on
Commit
19d89a8
Β·
verified Β·
1 Parent(s): 1ebe76a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +38 -31
README.md CHANGED
@@ -4,61 +4,67 @@ license: apache-2.0
4
  license_link: https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE
5
  ---
6
 
7
- # Qwen3.5 Text-Only Variants
8
-
 
 
9
  Trimmed checkpoints of the Qwen3.5 model family with vision encoder weights removed β€” smaller files, lower VRAM, drop-in text-only replacement.
10
 
11
- > **⚠️ Disclaimer:** These models were tested exclusively with HuggingFace Transformers (β‰₯5.2.0). vLLM, SGLang, llama.cpp, Ollama, and other inference engines are **not supported** β€” partly because Transformers 5 support is still cooking in those projects, and partly because we just threw these checkpoints on the Hub while messing around in the lab. We didn't set out to build a production-ready model zoo; we just left the oven door open. Use accordingly, and don't @ us when your vLLM setup catches fire.
 
 
 
12
 
13
- ## Overview
 
 
14
 
15
- [Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-9B) is a multimodal model family (VLM) that integrates a vision encoder alongside the language model in a single unified checkpoint. While this enables image and video understanding, the vision weights are entirely unused during text-only inference β€” yet they still occupy disk space and VRAM.
16
 
17
- This project re-serializes each Qwen3.5 checkpoint using HuggingFace Transformers' `Qwen3_5ForCausalLM` class, which exposes only the language model component. The result is a standard causal LM checkpoint that:
 
 
 
18
 
19
- - loads directly with `AutoModelForCausalLM` β€” no special handling required
20
- - preserves all text generation capabilities of the original model
21
- - ships with a smaller file size and lower VRAM footprint
22
 
23
  ## Available Models
24
 
25
  | Model | HuggingFace Hub |
26
  | --- | --- |
27
- | Qwen3.5-0.8B text-only | `principled-intelligence/Qwen3.5-0.8B-text-only` |
28
- | Qwen3.5-2B text-only | `principled-intelligence/Qwen3.5-2B-text-only` |
29
- | Qwen3.5-4B text-only | `principled-intelligence/Qwen3.5-4B-text-only` |
30
- | Qwen3.5-9B text-only | `principled-intelligence/Qwen3.5-9B-text-only` |
 
31
 
32
  ## Size Reduction
33
 
34
- **Qwen3.5-0.8B**
35
 
 
36
  | Metric | Qwen3.5 | Text-Only | Reduction |
37
- | --- | --- | --- | --- |
38
  | File size (GB) | 1.75 | 1.50 | ~14% |
39
  | VRAM (GB) | 1.59 | 1.40 | ~12% |
40
  | Parameters (B) | 0.85 | 0.75 | ~12% |
41
 
42
- **Qwen3.5-2B**
43
-
44
  | Metric | Qwen3.5 | Text-Only | Reduction |
45
- | --- | --- | --- | --- |
46
  | File size (GB) | 4.55 | 3.76 | ~17% |
47
  | VRAM (GB) | 4.12 | 3.51 | ~15% |
48
  | Parameters (B) | 2.21 | 1.88 | ~15% |
49
 
50
- **Qwen3.5-4B**
51
-
52
  | Metric | Qwen3.5 | Text-Only | Reduction |
53
- | --- | --- | --- | --- |
54
  | File size (GB) | 9.32 | 8.41 | ~10% |
55
  | VRAM (GB) | 8.45 | 7.83 | ~7% |
56
  | Parameters (B) | 4.54 | 4.21 | ~7% |
57
 
58
- **Qwen3.5-9B**
59
-
60
  | Metric | Qwen3.5 | Text-Only | Reduction |
61
- | --- | --- | --- | --- |
62
  | File size (GB) | 19.32 | 17.90 | ~7% |
63
  | VRAM (GB) | 17.52 | 16.68 | ~5% |
64
  | Parameters (B) | 9.41 | 8.95 | ~5% |
@@ -114,13 +120,14 @@ You can also use `pipeline` for a simpler interface:
114
 
115
  > Qwen3.5 thinks by default, generating `<think>...</think>` content before the final response. To disable thinking, pass `chat_template_kwargs={"enable_thinking": False}` in your generation call or API request.
116
 
117
- ## Why Text-Only?
118
 
119
- - **Lower VRAM** β€” vision encoder weights are freed, reducing peak memory usage by 5–17% depending on model size
120
- - **Smaller checkpoints** β€” faster downloads and storage savings
121
- - **Simpler loading** β€” standard `AutoModelForCausalLM`, no multimodal dependencies
122
- - **Drop-in replacement** β€” identical tokenizer, same chat template, same text generation behavior as the original Qwen3.5 models
123
 
124
- ## How It Works
125
 
126
- The Qwen3.5 architecture consists of a vision encoder and a language model sharing a single checkpoint. During text-only inference the vision encoder is never called, but its weights are still loaded into memory. By loading the checkpoint with `Qwen3_5ForCausalLM` instead of `Qwen3_5ForConditionalGeneration`, HuggingFace Transformers instantiates only the language model component. Re-saving that model produces a checkpoint with no vision weights, which can subsequently be loaded with the standard `AutoModelForCausalLM` interface.
 
 
 
 
 
4
  license_link: https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE
5
  ---
6
 
7
+ # Qwen3.5 Text-Only
8
+
9
+ If all you need is text, these are the Qwen3.5 models for you.
10
+
11
  Trimmed checkpoints of the Qwen3.5 model family with vision encoder weights removed β€” smaller files, lower VRAM, drop-in text-only replacement.
12
 
13
+ > **⚠️ Disclaimer:** These models were tested exclusively with HuggingFace Transformers (β‰₯5.2.0). vLLM, SGLang, llama.cpp, Ollama, and other inference engines are **not supported** yet β€” partly because Transformers 5 support is still cooking in those projects, and partly because we just threw these checkpoints on the Hub while messing around in the lab. If you get any of these running on other engines, we'd love to hear about it β€” open a discussion or drop a community post. We didn't set out to build a production-ready model zoo; we just left the oven door open. Use accordingly.
14
+
15
+ For official details on the Qwen3.5 model family β€” architecture, benchmarks, training data, and intended use β€” see the [original Qwen3.5 model card](https://huggingface.co/Qwen/Qwen3.5-9B).
16
+
17
 
18
+ ## How It Works
19
+
20
+ The Qwen3.5 architecture consists of a vision encoder and a language model sharing a single checkpoint. During text-only inference the vision encoder is never called, but its weights are still loaded into memory. By loading the checkpoint with `Qwen3_5ForCausalLM` instead of `Qwen3_5ForConditionalGeneration`, HuggingFace Transformers instantiates only the language model component. Re-saving that model produces a checkpoint with no vision weights, which can subsequently be loaded with the standard `AutoModelForCausalLM` interface.
21
 
22
+ **Why bother?**
23
 
24
+ - **Lower VRAM** β€” vision encoder weights are freed, reducing peak memory usage by 5–17% depending on model size
25
+ - **Smaller checkpoints** β€” faster downloads and storage savings
26
+ - **Simpler loading** β€” standard `AutoModelForCausalLM`, no multimodal dependencies
27
+ - **Drop-in replacement** β€” identical tokenizer, same chat template, same text generation behavior as the original Qwen3.5 models
28
 
 
 
 
29
 
30
  ## Available Models
31
 
32
  | Model | HuggingFace Hub |
33
  | --- | --- |
34
+ | Qwen3.5-0.8B-text-only | [`principled-intelligence/Qwen3.5-0.8B-text-only`](https://huggingface.co/principled-intelligence/Qwen3.5-0.8B-text-only) |
35
+ | Qwen3.5-2B-text-only | [`principled-intelligence/Qwen3.5-2B-text-only`](https://huggingface.co/principled-intelligence/Qwen3.5-2B-text-only) |
36
+ | Qwen3.5-4B-text-only | [`principled-intelligence/Qwen3.5-4B-text-only`](https://huggingface.co/principled-intelligence/Qwen3.5-4B-text-only) |
37
+ | Qwen3.5-9B-text-only | [`principled-intelligence/Qwen3.5-9B-text-only`](https://huggingface.co/principled-intelligence/Qwen3.5-9B-text-only) |
38
+
39
 
40
  ## Size Reduction
41
 
42
+ We compared each text-only checkpoint against its original Qwen3.5 counterpart across three metrics: file size on disk, peak VRAM usage when loaded in `float16` with `device_map="auto"`, and total parameter count. Savings scale with the relative size of the vision encoder β€” smaller models see the biggest percentage drop.
43
 
44
+ **Qwen3.5-0.8B vs. Qwen3.5-0.8B-text-only**
45
  | Metric | Qwen3.5 | Text-Only | Reduction |
46
+ | --- | ---: | ---: | ---: |
47
  | File size (GB) | 1.75 | 1.50 | ~14% |
48
  | VRAM (GB) | 1.59 | 1.40 | ~12% |
49
  | Parameters (B) | 0.85 | 0.75 | ~12% |
50
 
51
+ **Qwen3.5-2B vs. Qwen3.5-2B-text-only**
 
52
  | Metric | Qwen3.5 | Text-Only | Reduction |
53
+ | --- | ---: | ---: | ---: |
54
  | File size (GB) | 4.55 | 3.76 | ~17% |
55
  | VRAM (GB) | 4.12 | 3.51 | ~15% |
56
  | Parameters (B) | 2.21 | 1.88 | ~15% |
57
 
58
+ **Qwen3.5-4B vs. Qwen3.5-4B-text-only**
 
59
  | Metric | Qwen3.5 | Text-Only | Reduction |
60
+ | --- | ---: | ---: | ---: |
61
  | File size (GB) | 9.32 | 8.41 | ~10% |
62
  | VRAM (GB) | 8.45 | 7.83 | ~7% |
63
  | Parameters (B) | 4.54 | 4.21 | ~7% |
64
 
65
+ **Qwen3.5-9B vs. Qwen3.5-9B-text-only**
 
66
  | Metric | Qwen3.5 | Text-Only | Reduction |
67
+ | --- | ---: | ---: | ---: |
68
  | File size (GB) | 19.32 | 17.90 | ~7% |
69
  | VRAM (GB) | 17.52 | 16.68 | ~5% |
70
  | Parameters (B) | 9.41 | 8.95 | ~5% |
 
120
 
121
  > Qwen3.5 thinks by default, generating `<think>...</think>` content before the final response. To disable thinking, pass `chat_template_kwargs={"enable_thinking": False}` in your generation call or API request.
122
 
123
+ ## Contributing
124
 
125
+ Contributions are welcome! Whether it's getting these checkpoints running on vLLM, SGLang, llama.cpp, Ollama, or something else entirely β€” we'd love your help. Bug reports, compatibility notes, and PRs are all appreciated. Open a discussion or community post and let us know what you find.
 
 
 
126
 
127
+ ## License
128
 
129
+ These checkpoints are released under the [Apache 2.0 License](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/LICENSE), consistent with the original Qwen3.5 models.
130
+
131
+ ---
132
+
133
+ *Made with love from [Principled Intelligence](https://huggingface.co/principled-intelligence)* ❀️