| --- |
| library_name: mlx |
| license: mit |
| base_model: inclusionAI/Ling-3.0-flash |
| base_model_relation: quantized |
| pipeline_tag: text-generation |
| language: |
| - en |
| - zh |
| tags: |
| - mlx |
| - mlx-lm |
| - omlx |
| - bailing_hybrid |
| - mixture-of-experts |
| - linear-attention |
| - quantized |
| - apple-silicon |
| - custom_code |
| - oq5 |
| --- |
| |
| <p align="center"> |
| <a href="https://tensorfold.dev"> |
| <img src="https://huggingface.co/spaces/TensorFold/README/resolve/main/tensorfold-logo.png" alt="TensorFold" width="160"> |
| </a> |
| </p> |
| |
| <p align="center"> |
| <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100" alt="Ling logo"> |
| </p> |
|
|
|
|
| <p align="center"> |
| <img src="https://img.shields.io/badge/InclusionAI-Ling-1677FF?style=for-the-badge" alt="InclusionAI Ling"> |
| <img src="https://img.shields.io/badge/Apple_Silicon-MLX-000000?style=for-the-badge&logo=apple&logoColor=white" alt="Apple silicon MLX"> |
| <img src="https://img.shields.io/badge/TensorFold-oMLX-6E56CF?style=for-the-badge&logo=huggingface&logoColor=white" alt="TensorFold oMLX"> |
| </p> |
|
|
|
|
| <h1 align="center">Ling 3.0 Flash — oQ5</h1> |
|
|
|
|
| <p align="center"> |
| A native Apple-silicon conversion of <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash">inclusionAI/Ling-3.0-flash</a>, quantized with oMLX's optimized mixed-precision oQ5 recipe and packaged for oMLX and compatible MLX-LM runtimes. |
| </p> |
|
|
|
|
| <p align="center"> |
| <a href="https://huggingface.co/inclusionAI/Ling-3.0-flash">Original model</a> · |
| <a href="https://modelscope.cn/organization/inclusionAI">InclusionAI on ModelScope</a> · |
| <a href="https://openrouter.ai/inclusionai/ling-3.0-flash:free">OpenRouter</a> · |
| <a href="https://github.com/ml-explore/mlx-lm">MLX-LM</a> |
| </p> |
|
|
|
|
| ## About this conversion |
|
|
|
|
| This repository contains an **oMLX-optimized mixed-precision oQ5 conversion** of Ling 3.0 Flash. Ling is a 124B-total / 5.1B-active hybrid-linear mixture-of-experts model. The conversion preserves the upstream tokenizer and chat template and includes the custom MLX architecture adapter required for Kimi Delta Attention, gated MLA, and sparse MoE layers. |
|
|
|
|
| | Item | Value | |
| | --- | --- | |
| | Base model | [`inclusionAI/Ling-3.0-flash`](https://huggingface.co/inclusionAI/Ling-3.0-flash) | |
| | Format | MLX safetensors | |
| | Quantization | oQ5 affine mixed precision, group size 64 | |
| | Base precision | 5-bit | |
| | Protected modules | 130 at 6-bit and 325 at 8-bit | |
| | Conversion/runtime stack | oMLX with MLX-LM 0.31.3 / MLX 0.32.0 | |
| | Weight shards | 16 | |
| | Weight size | 86.62 GB (80.67 GiB) | |
| | Maximum configured context | 262,144 tokens | |
| | Architecture | `bailing_hybrid` | |
|
|
|
|
| > [!IMPORTANT] |
| > This model uses the included `bailing_hybrid.py` custom MLX adapter. In oMLX, enable **Trust Remote Code** for this model before loading it. Standalone MLX-LM requires a build whose `mlx_lm.load` supports the `trust_remote_code` argument and repository-provided `model_file` adapters; stock PyPI `mlx-lm 0.31.3` does not provide that loader path. |
| |
| |
| ## Apple-silicon performance |
| |
| |
| This checkpoint was load-tested, generation-tested, and benchmarked on: |
| |
| |
| | Hardware | Configuration | |
| | --- | --- | |
| | Host | Mac Studio | |
| | Chip | Apple M3 Ultra | |
| | CPU | 32 cores (24 performance + 8 efficiency) | |
| | Unified memory | 256 GB | |
| | Runtime | oMLX-bundled MLX-LM 0.31.3 / MLX 0.32.0 | |
| |
| |
| A warmed local test produced: |
| |
| |
| | Measurement | Result | |
| | --- | ---: | |
| | Decode (median) | **73.10 tokens/s** | |
| | Individual decode runs | 73.23 / 73.10 / 73.09 tokens/s | |
| | Reported peak memory | **87.44 GB** | |
| | Timed runs | 3 × 256 generated tokens | |
| | Warm-up | 256 generated tokens | |
| | Prompt | 39 tokens after chat templating | |
| |
| |
| The decode figure is the median of three greedy 256-token runs after a full 256-token Metal-kernel warm-up. It is a practical local reference, not a controlled cross-platform benchmark. Prompt length, context growth, sampler settings, memory pressure, thermal state, and runtime versions can materially change performance. |
| |
| |
| ## Recommended use with oMLX |
| |
| |
| 1. Download the model into the oMLX model directory: |
| |
| |
| ```bash |
| hf download TensorFold/Ling-3.0-flash-oQ5 \ |
| --local-dir ~/.omlx/models/TensorFold/Ling-3.0-flash-oQ5 |
| ``` |
| |
| |
| 2. Refresh the oMLX model registry. |
| 3. Open the model settings and enable **Trust Remote Code**. |
| 4. Load `Ling-3.0-flash-oQ5` and use the normal chat UI or OpenAI-compatible endpoint. |
| |
| |
| Example request: |
| |
| |
| ```bash |
| curl http://localhost:8000/v1/chat/completions \ |
| -H "Content-Type: application/json" \ |
| -H "Authorization: Bearer $OMLX_API_KEY" \ |
| -d '{ |
| "model": "Ling-3.0-flash-oQ5", |
| "messages": [{"role": "user", "content": "Explain why hybrid linear attention is useful."}], |
| "temperature": 0.6, |
| "top_p": 0.95, |
| "max_tokens": 512 |
| }' |
| ``` |
| |
|
|
| Thinking mode is enabled by the upstream chat template by default. Disable it with the model's chat-template configuration when a direct answer is preferred. |
|
|
|
|
| ## Compatible standalone MLX-LM builds |
|
|
|
|
| With an MLX-LM build that supports repository-provided model adapters: |
|
|
|
|
| ```bash |
| mlx_lm.generate \ |
| --model TensorFold/Ling-3.0-flash-oQ5 \ |
| --trust-remote-code \ |
| --prompt "Explain why hybrid linear attention is useful." \ |
| --max-tokens 512 \ |
| --temp 0.6 \ |
| --top-p 0.95 \ |
| --top-k 20 |
| ``` |
|
|
|
|
| Python usage with the same capability: |
|
|
|
|
| ```python |
| from mlx_lm import load, generate |
| |
| |
| model, tokenizer = load( |
| "TensorFold/Ling-3.0-flash-oQ5", |
| trust_remote_code=True, |
| tokenizer_config={"trust_remote_code": True}, |
| ) |
| messages = [ |
| {"role": "user", "content": "Explain sparse mixture-of-experts routing."} |
| ] |
| prompt = tokenizer.apply_chat_template( |
| messages, |
| add_generation_prompt=True, |
| tokenize=False, |
| enable_thinking=False, |
| ) |
| print(generate(model, tokenizer, prompt=prompt, max_tokens=512)) |
| ``` |
|
|
|
|
| Only enable remote code after reviewing the included adapter files. |
|
|
|
|
| ## Architecture |
|
|
|
|
| Ling 3.0 Flash alternates Kimi Delta Attention (KDA) and gated Multi-head Latent Attention (MLA) in a 5:1 ratio and uses highly sparse routed experts. |
|
|
|
|
| | Architecture detail | Upstream value | |
| | --- | ---: | |
| | Total / active parameters | 124B / 5.1B | |
| | Transformer layers | 35 KDA + 7 gated MLA | |
| | Dense layers | 2 | |
| | Routed / shared experts | 512 / 1 | |
| | Active routed experts | 8 | |
| | Attention heads | 32 | |
| | Hidden size | 2,560 | |
| | Expert intermediate size | 768 | |
| | Dense intermediate size | 6,144 | |
| | Vocabulary size | 157,184 | |
| | Context training schedule | 8K → 32K → 256K | |
|
|
|
|
| The included MLX adapter uses MLX-LM primitives for delta attention, absorbed MLA projections, RoPE, and quantized SwitchGLU experts. The auxiliary MTP training head is excluded from ordinary causal generation; this release does **not** claim MTP or DSpark speculative-decoding support. |
|
|
|
|
| ## Upstream model highlights |
|
|
|
|
| InclusionAI describes Ling 3.0 Flash as a hybrid reasoning model for software-engineering agents, tool use, deep research, general knowledge, mathematical reasoning, instruction following, and long-context understanding. |
|
|
|
|
| The upstream defaults are: |
|
|
|
|
| ```text |
| thinking: enabled |
| temperature: 0.6 |
| top_p: 0.95 |
| top_k: 20 |
| ``` |
|
|
|
|
| For benchmark methodology, scores, intended use, limitations, and framework-specific deployment instructions, see the [original InclusionAI model card](https://huggingface.co/inclusionAI/Ling-3.0-flash). |
|
|
|
|
| ## Conversion and validation notes |
|
|
|
|
| - Source weights: upstream BF16 checkpoint. |
| - Quantization mode: affine, group size 64. |
| - oQ5 uses a 5-bit base with 455 sensitive modules kept at higher precision. |
| - Both `quantization` and `quantization_config` preserve the per-module recipe. |
| - The upstream chat template is included unchanged. |
| - All 2,067 converted tensors and all 16 indexed shards were checked locally. |
| - The custom adapter was loaded with explicit trust and exercised through end-to-end generation. |
| - Quantization can reduce output quality relative to BF16; use a higher-precision variant when quality matters more than memory use. |
|
|
|
|
| This is a community conversion, not an official InclusionAI release. Validate quality and numerical behavior on representative workloads before production use. |
|
|
|
|
| ## License and attribution |
|
|
|
|
| The upstream model declares the **MIT License** in its Hugging Face metadata. A standard MIT licence copy is included in this repository. |
|
|
|
|
| All model design, training, benchmark, and upstream documentation credit belongs to InclusionAI and the original contributors. The mixed-precision conversion, Apple-silicon validation, compatibility packaging, and model card are provided by [TensorFold](https://huggingface.co/TensorFold). |
|
|
|
|
|
|
| <!-- TensorFold-chooser-start --> |
| ## Choose for your Mac |
|
|
| [64GB Macs](https://huggingface.co/collections/TensorFold/mlx-models-for-64gb-macs-6a9fefda17932216ec9ab457) · [128GB Macs](https://huggingface.co/collections/TensorFold/mlx-models-for-128gb-macs-6a9ff0abd31bc9abbe7922d7) · [256GB Macs](https://huggingface.co/collections/TensorFold/mlx-models-for-256gb-macs-6a9ff0ef9fed7c5bdca15e9b) |
|
|
| Published peak memory: **87.44 GB**; estimated starting tier: **128GB**, leaving about **40 GB** nominal headroom. The collections use published M3 Studio peaks with at least 25% nominal headroom; fit on other Macs is an estimate, and full context is not guaranteed. Start with short context and one request. |
|
|
| ### Runtime and evidence |
|
|
| The exact tested oMLX application version is not recorded here; a library version is not an app version. The original performance tables retain their benchmark conditions and speed figures; this documentation update adds no new test results. |
|
|
| ### Quick start and demo prompt |
|
|
| ```bash |
| hf download TensorFold/Ling-3.0-flash-oQ5 --local-dir ./models/Ling-3.0-flash-oQ5 |
| ``` |
|
|
| Add the downloaded folder to oMLX model directories, refresh the list, and follow this card's architecture and MTP compatibility requirements before loading. |
|
|
| Try this in a new chat with a 128-token output limit: |
|
|
| ```text |
| Explain why the sky looks blue in three short sentences. |
| ``` |
|
|
| This is a demo prompt to try, not a recorded successful run; a captured demonstration for this documentation update is not yet available. |
|
|
| [Follow TensorFold for new Apple Silicon releases and fixes.](https://huggingface.co/TensorFold) |
| <!-- TensorFold-chooser-end --> |
|
|
