majentik commited on
Commit
c1c928b
Β·
verified Β·
1 Parent(s): 3224d35

feat: publish majentik model garden hub

Browse files
Files changed (1) hide show
  1. README.md +110 -0
README.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - model-garden
5
+ - quantization
6
+ - rotorquant
7
+ - turboquant
8
+ - gguf
9
+ - mlx
10
+ - awq
11
+ - kv-cache-quantization
12
+ library_name: transformers
13
+ pipeline_tag: text-generation
14
+ language:
15
+ - en
16
+ ---
17
+
18
+ # majentik β€” Model Garden
19
+
20
+ A curated collection of quantized open-weight models with inference-time
21
+ KV-cache compression. Every model keeps upstream tokenizers and
22
+ architectures; the only thing we change is how the weights and KV cache
23
+ are stored during generation.
24
+
25
+ > 307 repositories Β· 12 families Β· 6 quantization lanes
26
+
27
+ ## What this garden is for
28
+
29
+ Running bigger models on the laptop you already have. Every release
30
+ combines a standard weight-quantization format (GGUF, MLX, or AWQ) with
31
+ one of two KV-cache compressors:
32
+
33
+ | Compressor | What it does | When to use |
34
+ | --- | --- | --- |
35
+ | **RotorQuant** | Rotational isotropic KV-cache compression | Long-context work; 2–4Γ— KV memory savings with minimal drift |
36
+ | **TurboQuant** | Turbo-variant targeted at throughput | Short-context, high-throughput serving |
37
+
38
+ Both compressors are applied at **inference time**. They compose with
39
+ any weight-quantized file in this garden β€” you mix and match.
40
+
41
+ ## Families
42
+
43
+ | Family | Repos | Notes |
44
+ | --- | ---: | --- |
45
+ | Gemma 4 | 127 | E2B / E4B / 26B-A4B / 31B, base + instruct |
46
+ | Nemotron | 41 | Nano 4B + Super (Thinking + Base) variants |
47
+ | Qwen 3.5 | 28 | 27B dense + 397B-A17B MoE |
48
+ | GPT-OSS | 28 | 20B and 120B |
49
+ | Voxtral | 24 | ASR + voice chat, 3 sub-families |
50
+ | MERaLiON | 30 | 2 (20 repos) and 3 (10 repos) β€” ASR + multimodal |
51
+ | MiniMax M2.7 | 9 | Mixed quantization lanes |
52
+ | Mistral Small 4 | 8 | Instruct + reasoning |
53
+ | Leanstral | 8 | Distilled Mistral reasoning variant |
54
+ | DeepSeek V3.2 | 2 | Mostly upstream, KV-quant wrappers |
55
+
56
+ ## Quantization lanes
57
+
58
+ Every model lands in one or more of these lanes (the README for each
59
+ repo specifies which):
60
+
61
+ - **GGUF** β€” Q2_K, Q3_K_M, IQ4_XS, Q4_K_M, Q5_K_M, Q8_0. Load with
62
+ `llama.cpp`, `ollama`, `lmstudio`, or any GGUF-compatible runtime.
63
+ - **MLX** β€” 2-bit, 4-bit, 8-bit. Targets Apple Silicon. Pip install
64
+ `mlx-lm`, point it at the repo, done.
65
+ - **AWQ** β€” 4-bit and 8-bit. Targets CUDA GPUs with vLLM or autoawq.
66
+
67
+ ## Pick a starting point
68
+
69
+ - **I have a MacBook with 16 GB RAM** β†’ try a 4-bit MLX variant of
70
+ Gemma 4 E4B.
71
+ - **I have a 24 GB GPU** β†’ try an AWQ-4bit Qwen3.5-27B with RotorQuant.
72
+ - **I need 128k context on modest hardware** β†’ any GGUF + RotorQuant.
73
+ - **I want to compare** β†’ each repo card links to the corresponding
74
+ upstream model, so perplexity drift is one `eval_hf.py` run away.
75
+
76
+ ## What is *not* in each repo
77
+
78
+ - **Training data.** These are quantization-only releases. The base
79
+ model's training data is upstream's concern; we inherit the upstream
80
+ license and disclaimers.
81
+ - **Benchmarks for every axis.** We publish per-lane perplexity on
82
+ WikiText-2 plus family-specific evals (MMLU for reasoning, LibriSpeech
83
+ for ASR). If you want an axis we haven't measured yet, open a
84
+ discussion on the relevant repo.
85
+
86
+ ## Who we are
87
+
88
+ majentik publishes these as a side-project to keep our own fleet running
89
+ cheaply on commodity hardware, and to close the gap between research
90
+ releases and "can I actually run this tonight". Issues, quant requests,
91
+ and benchmark PRs are welcome.
92
+
93
+ ## Contact
94
+
95
+ - Discussions: use the **Community** tab on the specific model repo.
96
+ - Hardware donations / compute partnerships: majentik on Hugging Face.
97
+ - Everything else: open a discussion on the closest repo, we'll see it.
98
+
99
+ ## Versioning
100
+
101
+ Each repo version tracks `upstream@base-model-revision Γ— quant-lane`.
102
+ When upstream ships a new base revision, we re-run the quant lane and
103
+ bump the repo version. Card changes (docs, benchmarks) do not bump the
104
+ version.
105
+
106
+ ## License
107
+
108
+ Each repo inherits the **base model's license**, not this
109
+ organization-level license. Check the `license` field in the
110
+ repository's card before deploying.