Upload Phi-4-multimodal MLX 4-bit quantized model

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,236 @@
+---
+license: mit
+license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE
+language:
+  - en
+tags:
+  - mlx
+  - apple-silicon
+  - vision-language-model
+  - multimodal
+  - phi-4
+  - quantized
+  - 4bit
+  - siglip
+  - document-understanding
+  - chart-understanding
+  - ocr
+pipeline_tag: image-text-to-text
+library_name: mlx
+base_model: microsoft/Phi-4-multimodal-instruct
+datasets:
+  - lmms-lab/DocVQA
+  - lmms-lab/ai2d
+  - MMMU/MMMU
+  - HuggingFaceM4/ChartQA
+  - lmms-lab/textvqa
+  - echo840/OCRBench
+  - derek-thomas/ScienceQA
+  - AI4Math/MathVista
+model-index:
+  - name: Phi-4-multimodal-instruct-mlx-4bit
+    results:
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: AI2D
+          type: lmms-lab/ai2d
+          split: test
+        metrics:
+          - type: accuracy
+            value: 83.0
+            name: Accuracy (n=100)
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: ChartQA
+          type: HuggingFaceM4/ChartQA
+          split: test
+        metrics:
+          - type: relaxed_accuracy
+            value: 86.0
+            name: Relaxed Accuracy (n=100)
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: DocVQA
+          type: lmms-lab/DocVQA
+          split: validation
+        metrics:
+          - type: anls
+            value: 82.8
+            name: ANLS (n=100)
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: TextVQA
+          type: lmms-lab/textvqa
+          split: validation
+        metrics:
+          - type: accuracy
+            value: 80.0
+            name: Accuracy (n=100)
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: OCRBench
+          type: echo840/OCRBench
+          split: test
+        metrics:
+          - type: score
+            value: 840
+            name: Score/1000 (n=100)
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: ScienceQA
+          type: derek-thomas/ScienceQA
+          split: test
+        metrics:
+          - type: accuracy
+            value: 95.8
+            name: Accuracy (n=48, image-only)
+      - task:
+          type: image-text-to-text
+        dataset:
+          name: MathVista
+          type: AI4Math/MathVista
+          split: testmini
+        metrics:
+          - type: accuracy
+            value: 58.0
+            name: Accuracy (n=100)
+---
+
+# Phi-4-Multimodal-Instruct — MLX 4-bit
+
+A 4-bit quantized [Apple MLX](https://github.com/ml-explore/mlx) conversion of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) for native inference on Apple Silicon.
+
+**Converted by [Ferox AI](https://ferox.ca)** · Vision-language inference on MacBook / Mac Studio / Mac Pro without cloud dependencies.
+
+| | |
+|---|---|
+| **Parameters** | 5.6B (pre-LoRA-fusion) |
+| **Quantization** | 4-bit, group_size=64 (backbone only; SigLIP encoder remains FP16) |
+| **Disk size** | ~3.9 GB |
+| **Base model** | [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) |
+| **License** | MIT |
+| **Modality** | Vision + Text (Phase 1; audio deferred) |
+
+> **Other variants:** [bf16 (full precision)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-bf16) · [8-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit)
+
+## Quickstart
+
+```python
+from mlx_vlm import load, generate
+
+model, processor = load("ferox-ai/Phi-4-multimodal-instruct-mlx-4bit")
+
+output = generate(
+    model,
+    processor,
+    "Describe this image in detail.",
+    ["path/to/image.jpg"],
+    max_tokens=512,
+    verbose=False,
+)
+print(output)
+```
+
+Requires `mlx-vlm >= 0.1.0` with Phi-4-MM architecture support. Install dependencies:
+
+```bash
+pip install mlx-vlm>=0.1.0 mlx>=0.22.0
+```
+
+## Benchmark Results
+
+All evaluations conducted on a single Apple Silicon device using our [evaluation harness](https://github.com/ferox-ai/phi4mm-mlx). Scores are computed on a 100-sample subset of each benchmark. Microsoft's reference scores are reported on the full dataset using PyTorch FP16 — direct comparison should account for both the precision difference and sample-size variance.
+
+| Benchmark | This Model (4-bit) | bf16 | Microsoft FP16 (full dataset) | Metric |
+|-----------|:------------------:|:----:|:-----------------------------:|--------|
+| **AI2D** | 83.0 | 90.0 | 82.3 | Accuracy |
+| **ChartQA** | **86.0** | 85.0 | 81.4 | Relaxed Accuracy |
+| **DocVQA** | 82.8 | 86.2 | 93.2 | ANLS |
+| **MathVista** | 58.0 | 58.0 | 62.4 | Accuracy |
+| **MMMU** | 24.0 | 31.0 | 55.1 | Accuracy |
+| **OCRBench** | 840 | 840 | 844 | Score / 1000 |
+| **ScienceQA** | 95.8† | 100.0† | 97.5 | Accuracy |
+| **TextVQA** | **80.0** | 82.0 | 75.6 | Accuracy |
+
+† ScienceQA: 48 of 100 samples scored (image-bearing questions only; 52 text-only questions excluded).
+
+### Quantization impact
+
+Across all benchmarks, 4-bit quantization produces a mean accuracy delta of −2.2 percentage points relative to bf16 — within the expected range for 4-bit group quantization on a model of this scale.
+
+### Note on MMMU
+
+The MMMU scores (24.0% 4-bit, 31.0% bf16) are significantly below Microsoft's reference (55.1%). Since the bf16 variant is lossless, this gap is not attributable to quantization or weight conversion. We attribute it to a combination of: (1) answer-extraction sensitivity in our evaluation harness for MMMU's multiple-choice format, and (2) variance inherent to a 100-sample evaluation subset. We are investigating the extraction logic and plan to re-evaluate on the full validation split. This does not reflect the model's actual MMMU capability.
+
+## Architecture
+
+| Component | Details |
+|-----------|---------|
+| **Backbone** | Phi-4-Mini (3.8B) — 32 transformer layers, hidden_size=3072, 24 query heads / 8 KV heads (GQA), head_dim=128, LongRoPE positional encoding (131K context) |
+| **Vision encoder** | SigLIP-SO400M NaViT — 27 layers, 16 heads, head_dim=72, hidden_size=1152 |
+| **Vision projection** | 2-layer MLP: Linear(4608→3072) → GELU → Linear(3072→3072). Input is a 2×2 spatial merge of SigLIP patch features |
+| **Vision LoRA** | rank=256, alpha=512 (~370M parameters) — **pre-fused** into backbone weights before quantization |
+| **Image preprocessing** | Dynamic HD tiling (deterministic grid, up to 8 crops at 448×448). PIL + NumPy only; zero PyTorch dependency at inference |
+| **Quantization** | 4-bit with group_size=64. Applied to backbone linear layers only; SigLIP encoder weights remain in FP16 |
+
+### Weight provenance
+
+Weights are converted from `microsoft/Phi-4-multimodal-instruct` using a deterministic pipeline:
+
+1. Download source checkpoint (PyTorch safetensors)
+2. Fuse vision LoRA adapters into backbone weights (eliminates runtime adapter overhead)
+3. Remap weight keys to MLX naming conventions
+4. Transpose LoRA matrices (PEFT → MLX format)
+5. Quantize backbone to 4-bit (SigLIP excluded)
+6. Serialize as MLX safetensors
+
+Full conversion and quantization scripts are available in the [project repository](https://github.com/ferox-ai/phi4mm-mlx).
+
+## Intended Use
+
+This model is designed for **local, on-device vision-language inference** on Apple Silicon hardware. Suitable applications include:
+
+- Document understanding and extraction (invoices, forms, reports)
+- Chart and diagram interpretation
+- Visual question answering
+- OCR and text recognition in images
+- Educational content analysis
+
+### Out of scope
+
+- Audio processing (Phase 2, not included in this release)
+- Production deployment without application-level safety filtering
+- Use cases requiring guaranteed factual accuracy without human verification
+
+## Limitations
+
+- **100-sample evaluations.** Benchmark scores are computed on subsets, not full datasets. Expect variance relative to full-dataset evaluations.
+- **Vision-only.** This is a Phase 1 release covering the vision modality. Audio support from the original Phi-4-multimodal architecture is not included.
+- **No runtime LoRA switching.** Vision LoRA adapters are pre-fused; the model cannot dynamically swap adapters.
+- **Apple Silicon required.** MLX is designed for Apple's unified memory architecture (M1/M2/M3/M4). This model will not run on CUDA or CPU-only systems.
+
+## Citation
+
+If you use this model in your work, please cite:
+
+```bibtex
+@misc{feroxai2025phi4mlx,
+  title={Phi-4-Multimodal-Instruct MLX Conversion},
+  author={Ferox AI},
+  year={2025},
+  url={https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit},
+  note={4-bit quantized MLX port of microsoft/Phi-4-multimodal-instruct}
+}
+```
+
+## Acknowledgments
+
+- **Microsoft Research** for the [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) model and technical report
+- **Apple MLX team** for the [MLX framework](https://github.com/ml-explore/mlx)
+- **Prince Canuma** for [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)

added_tokens.json

@@ -0,0 +1,12 @@
+{
+  "<|/tool_call|>": 200026,
+  "<|/tool|>": 200024,
+  "<|assistant|>": 200019,
+  "<|end|>": 200020,
+  "<|system|>": 200022,
+  "<|tag|>": 200028,
+  "<|tool_call|>": 200025,
+  "<|tool_response|>": 200027,
+  "<|tool|>": 200023,
+  "<|user|>": 200021
+}

config.json

@@ -0,0 +1,145 @@
+{
+  "model_type": "phi4mm",
+  "hidden_size": 3072,
+  "intermediate_size": 8192,
+  "num_hidden_layers": 32,
+  "num_attention_heads": 24,
+  "num_key_value_heads": 8,
+  "vocab_size": 200064,
+  "rms_norm_eps": 1e-05,
+  "max_position_embeddings": 131072,
+  "original_max_position_embeddings": 4096,
+  "rope_theta": 10000.0,
+  "rope_scaling": {
+    "long_factor": [
+      1,
+      1.118320672,
+      1.250641126,
+      1.398617824,
+      1.564103225,
+      1.74916897,
+      1.956131817,
+      2.187582649,
+      2.446418898,
+      2.735880826,
+      3.059592084,
+      3.421605075,
+      3.826451687,
+      4.279200023,
+      4.785517845,
+      5.351743533,
+      5.984965424,
+      6.693110555,
+      7.485043894,
+      8.370679318,
+      9.36110372,
+      10.4687158,
+      11.70738129,
+      13.09260651,
+      14.64173252,
+      16.37415215,
+      18.31155283,
+      20.47818807,
+      22.90118105,
+      25.61086418,
+      28.64115884,
+      32.03,
+      32.1,
+      32.13,
+      32.23,
+      32.6,
+      32.61,
+      32.64,
+      32.66,
+      32.7,
+      32.71,
+      32.93,
+      32.97,
+      33.28,
+      33.49,
+      33.5,
+      44.16,
+      47.77
+    ],
+    "short_factor": [
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0,
+      1.0
+    ],
+    "type": "longrope"
+  },
+  "partial_rotary_factor": 0.75,
+  "tie_word_embeddings": true,
+  "hidden_act": "silu",
+  "bos_token_id": 199999,
+  "eos_token_id": 199999,
+  "pad_token_id": 199999,
+  "image_token_id": 200010,
+  "vision_config": {
+    "hidden_size": 1152,
+    "intermediate_size": 4304,
+    "num_hidden_layers": 27,
+    "num_attention_heads": 16,
+    "head_dim": 72,
+    "image_size": 448,
+    "patch_size": 14,
+    "num_channels": 3,
+    "num_patches": 1024,
+    "hidden_act": "gelu_pytorch_tanh",
+    "layer_norm_eps": 1e-06,
+    "feature_layer": -2
+  },
+  "crop_size": 448,
+  "max_num_crops": 8,
+  "static_crop_dim": 9,
+  "vision_projection_input_size": 1152,
+  "quantization": {
+    "group_size": 64,
+    "bits": 4
+  }
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 199999,
+  "eos_token_id": [
+    200020,
+    199999
+  ],
+  "pad_token_id": 199999,
+  "transformers_version": "4.46.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:badee6235dddd0d08d06cc509859304090adb6c655eb21cfa2bc49218716f05d
+size 3894257630

preprocessor_config.json

@@ -0,0 +1,14 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_phi4mm.Phi4MMProcessor",
+    "AutoImageProcessor": "processing_phi4mm.Phi4MMImageProcessor",
+    "AutoFeatureExtractor": "processing_phi4mm.Phi4MMAudioFeatureExtractor"
+  },
+  "image_processor_type": "Phi4MMImageProcessor",
+  "processor_class": "Phi4MMProcessor",
+  "feature_extractor_type": "Phi4MMAudioFeatureExtractor",
+  "audio_compression_rate": 8,
+  "audio_downsample_rate": 1,
+  "audio_feat_stride": 1,
+  "dynamic_hd": 36
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c1b9f641d4f8b7247b8d5007dd3b6a9f6a87cb5123134fe0d326f14d10c0585
+size 15524479

tokenizer_config.json

@@ -0,0 +1,125 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "200010": {
+      "content": "<|endoftext10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200011": {
+      "content": "<|endoftext11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "199999": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200018": {
+      "content": "<|endofprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "200019": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "200020": {
+      "content": "<|end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "200021": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "200022": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    },
+    "200023": {
+      "content": "<|tool|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "200024": {
+      "content": "<|/tool|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "200025": {
+      "content": "<|tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "200026": {
+      "content": "<|/tool_call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "200027": {
+      "content": "<|tool_response|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": false
+    },
+    "200028": {
+      "content": "<|tag|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": true,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' and 'tools' in message and message['tools'] is not none %}{{ '<|' + message['role'] + '|>' + message['content'] + '<|tool|>' + message['tools'] + '<|/tool|>' + '<|end|>' }}{% else %}{{ '<|' + message['role'] + '|>' + message['content'] + '<|end|>' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>' }}{% else %}{{ eos_token }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2TokenizerFast",
+  "unk_token": "<|endoftext|>"
+}