Text Generation
Transformers
MLX
GGUF
English
model-garden
quantization
apple-silicon
asr
embeddings
kv-cache-quantization
Instructions to use majentik/garden with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use majentik/garden with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="majentik/garden")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("majentik/garden", device_map="auto") - MLX
How to use majentik/garden with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # if on a CUDA device, also pip install mlx[cuda] # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("majentik/garden") prompt = "Once upon a time in" text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- vLLM
How to use majentik/garden with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "majentik/garden" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "majentik/garden", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/majentik/garden
- SGLang
How to use majentik/garden with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "majentik/garden" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "majentik/garden", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "majentik/garden" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "majentik/garden", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - MLX LM
How to use majentik/garden with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Generate some text mlx_lm.generate --model "majentik/garden" --prompt "Once upon a time"
- Docker Model Runner
How to use majentik/garden with Docker Model Runner:
docker model run hf.co/majentik/garden
- Atomic Chat
feat: publish majentik model garden hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- model-garden
|
| 5 |
+
- quantization
|
| 6 |
+
- rotorquant
|
| 7 |
+
- turboquant
|
| 8 |
+
- gguf
|
| 9 |
+
- mlx
|
| 10 |
+
- awq
|
| 11 |
+
- kv-cache-quantization
|
| 12 |
+
library_name: transformers
|
| 13 |
+
pipeline_tag: text-generation
|
| 14 |
+
language:
|
| 15 |
+
- en
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# majentik β Model Garden
|
| 19 |
+
|
| 20 |
+
A curated collection of quantized open-weight models with inference-time
|
| 21 |
+
KV-cache compression. Every model keeps upstream tokenizers and
|
| 22 |
+
architectures; the only thing we change is how the weights and KV cache
|
| 23 |
+
are stored during generation.
|
| 24 |
+
|
| 25 |
+
> 307 repositories Β· 12 families Β· 6 quantization lanes
|
| 26 |
+
|
| 27 |
+
## What this garden is for
|
| 28 |
+
|
| 29 |
+
Running bigger models on the laptop you already have. Every release
|
| 30 |
+
combines a standard weight-quantization format (GGUF, MLX, or AWQ) with
|
| 31 |
+
one of two KV-cache compressors:
|
| 32 |
+
|
| 33 |
+
| Compressor | What it does | When to use |
|
| 34 |
+
| --- | --- | --- |
|
| 35 |
+
| **RotorQuant** | Rotational isotropic KV-cache compression | Long-context work; 2β4Γ KV memory savings with minimal drift |
|
| 36 |
+
| **TurboQuant** | Turbo-variant targeted at throughput | Short-context, high-throughput serving |
|
| 37 |
+
|
| 38 |
+
Both compressors are applied at **inference time**. They compose with
|
| 39 |
+
any weight-quantized file in this garden β you mix and match.
|
| 40 |
+
|
| 41 |
+
## Families
|
| 42 |
+
|
| 43 |
+
| Family | Repos | Notes |
|
| 44 |
+
| --- | ---: | --- |
|
| 45 |
+
| Gemma 4 | 127 | E2B / E4B / 26B-A4B / 31B, base + instruct |
|
| 46 |
+
| Nemotron | 41 | Nano 4B + Super (Thinking + Base) variants |
|
| 47 |
+
| Qwen 3.5 | 28 | 27B dense + 397B-A17B MoE |
|
| 48 |
+
| GPT-OSS | 28 | 20B and 120B |
|
| 49 |
+
| Voxtral | 24 | ASR + voice chat, 3 sub-families |
|
| 50 |
+
| MERaLiON | 30 | 2 (20 repos) and 3 (10 repos) β ASR + multimodal |
|
| 51 |
+
| MiniMax M2.7 | 9 | Mixed quantization lanes |
|
| 52 |
+
| Mistral Small 4 | 8 | Instruct + reasoning |
|
| 53 |
+
| Leanstral | 8 | Distilled Mistral reasoning variant |
|
| 54 |
+
| DeepSeek V3.2 | 2 | Mostly upstream, KV-quant wrappers |
|
| 55 |
+
|
| 56 |
+
## Quantization lanes
|
| 57 |
+
|
| 58 |
+
Every model lands in one or more of these lanes (the README for each
|
| 59 |
+
repo specifies which):
|
| 60 |
+
|
| 61 |
+
- **GGUF** β Q2_K, Q3_K_M, IQ4_XS, Q4_K_M, Q5_K_M, Q8_0. Load with
|
| 62 |
+
`llama.cpp`, `ollama`, `lmstudio`, or any GGUF-compatible runtime.
|
| 63 |
+
- **MLX** β 2-bit, 4-bit, 8-bit. Targets Apple Silicon. Pip install
|
| 64 |
+
`mlx-lm`, point it at the repo, done.
|
| 65 |
+
- **AWQ** β 4-bit and 8-bit. Targets CUDA GPUs with vLLM or autoawq.
|
| 66 |
+
|
| 67 |
+
## Pick a starting point
|
| 68 |
+
|
| 69 |
+
- **I have a MacBook with 16 GB RAM** β try a 4-bit MLX variant of
|
| 70 |
+
Gemma 4 E4B.
|
| 71 |
+
- **I have a 24 GB GPU** β try an AWQ-4bit Qwen3.5-27B with RotorQuant.
|
| 72 |
+
- **I need 128k context on modest hardware** β any GGUF + RotorQuant.
|
| 73 |
+
- **I want to compare** β each repo card links to the corresponding
|
| 74 |
+
upstream model, so perplexity drift is one `eval_hf.py` run away.
|
| 75 |
+
|
| 76 |
+
## What is *not* in each repo
|
| 77 |
+
|
| 78 |
+
- **Training data.** These are quantization-only releases. The base
|
| 79 |
+
model's training data is upstream's concern; we inherit the upstream
|
| 80 |
+
license and disclaimers.
|
| 81 |
+
- **Benchmarks for every axis.** We publish per-lane perplexity on
|
| 82 |
+
WikiText-2 plus family-specific evals (MMLU for reasoning, LibriSpeech
|
| 83 |
+
for ASR). If you want an axis we haven't measured yet, open a
|
| 84 |
+
discussion on the relevant repo.
|
| 85 |
+
|
| 86 |
+
## Who we are
|
| 87 |
+
|
| 88 |
+
majentik publishes these as a side-project to keep our own fleet running
|
| 89 |
+
cheaply on commodity hardware, and to close the gap between research
|
| 90 |
+
releases and "can I actually run this tonight". Issues, quant requests,
|
| 91 |
+
and benchmark PRs are welcome.
|
| 92 |
+
|
| 93 |
+
## Contact
|
| 94 |
+
|
| 95 |
+
- Discussions: use the **Community** tab on the specific model repo.
|
| 96 |
+
- Hardware donations / compute partnerships: majentik on Hugging Face.
|
| 97 |
+
- Everything else: open a discussion on the closest repo, we'll see it.
|
| 98 |
+
|
| 99 |
+
## Versioning
|
| 100 |
+
|
| 101 |
+
Each repo version tracks `upstream@base-model-revision Γ quant-lane`.
|
| 102 |
+
When upstream ships a new base revision, we re-run the quant lane and
|
| 103 |
+
bump the repo version. Card changes (docs, benchmarks) do not bump the
|
| 104 |
+
version.
|
| 105 |
+
|
| 106 |
+
## License
|
| 107 |
+
|
| 108 |
+
Each repo inherits the **base model's license**, not this
|
| 109 |
+
organization-level license. Check the `license` field in the
|
| 110 |
+
repository's card before deploying.
|