llama_3.1_8b-PATCH-35Sparse

This checkpoint (LLaMA-3.1 8B, PATCH-Tile, 35% sparsity): 55.28% average zero-shot accuracy, 7.89 WikiText2 perplexity.

This repository hosts a mask only release for the paper PATCH: Learnable Tile-level Hybrid Sparsity for LLMs. PATCH (Pruning with a Learnable Tile-level Configuration for Hybrid Sparsity) learns a structured mask on frozen pretrained weights, assigning each tile as dense (0% sparsity) or 2:4 sparse (50% sparsity) to hit a flexible global sparsity target while staying hardware-friendly.

Because PATCH/MaskLLM keep the base weights frozen, we distribute only the binary keep/prune mask (bit-packed in mask.npz) - no weight values. You recover the sparse model by downloading the original base model and applying the mask.

• Base model: meta-llama/Llama-3.1-8B
• Method: PATCH-Tile  |  Target sparsity: 35%  |  Pattern: Dense / 2:4 tiles
• Measured mask sparsity: 35.00% over 224 Linear layers (6,979,321,856 weights).

Results (LLaMA-3.1 8B)

Sparsity	Method	Pattern	Avg Acc (% ↑)	WikiText2 PPL (↓)
0%	Dense	-	60.31	5.84
50%	Magnitude	2:4	35.93	765.92
50%	Wanda	2:4	41.77	21.29
50%	SparseGPT	2:4	45.53	15.11
50%	Thanos	2:4	45.72	16.09
50%	ProxSparse	2:4	45.14	15.17
50%	MaskLLM	2:4	52.80	8.58
45%	PATCH-Tile	Dense/2:4	53.60	8.20
35%	PATCH-Tile ⭐	Dense/2:4	55.28	7.89
25%	PATCH-Tile	Dense/2:4	56.48	7.34

Per-task zero-shot accuracy (%) for this checkpoint:

MMLU	PIQA	ARC-E	ARC-C	WinoG.	OBQA	RACE	HellaS.	Average
51.15	77.97	76.14	42.41	69.46	31.40	38.18	55.54	55.28

All numbers are from the PATCH paper (arXiv:2509.23410); accuracy is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA, RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.

Training hyper-parameters

Hyper-parameter	Value
Fine-tuning dataset	SlimPajama (2B tokens)
Training steps	2000
Global batch size	256
Sequence length	4096
Mask tile size	128 x 128 (hardware tiles: 128x128 / 128x64 / 64x128 / 64x64)
Logits init.	N(0, 0.014)
Tile-logit prior	SparseGPT (strength 3)
Regularization scope	Global (single target density)
Evaluation	LM-Eval-Harness (8 zero-shot tasks) + WikiText2 PPL @ seqlen 4096
Hardware	1 node x 4 GPUs, data parallel (HuggingFace Trainer)
Optimizer	Adam
Learning rate	1e-4
Gumbel scaling (kappa)	100 -> 500
Gumbel temp (tau)	2 -> 0.05
Sparsity reg. (lambda1)	3
Weight reg. (lambda2)	0.1

How to use

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
from load_patch_mask import apply_patch_mask  # shipped in this repo

npz = hf_hub_download(repo_id="mohammad-mozaffari/llama_3.1_8b-PATCH-35Sparse", filename="mask.npz")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B", torch_dtype=torch.bfloat16)
apply_patch_mask(model, npz)   # zeroes the pruned weights in place

Or from the command line:

python load_patch_mask.py --base_model meta-llama/Llama-3.1-8B --mask_repo mohammad-mozaffari/llama_3.1_8b-PATCH-35Sparse

Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the GitHub repository and STOICC.

License

The released mask is a derivative of the base model and is distributed under the base model's license (llama3.1). You must comply with that license and obtain access to the base model separately.

Built with Llama. Use governed by the Llama 3.1 Community License.

The mask-generation code is released under the MIT license (see the PATCH repository).

Citation

@article{hourri2025patch,
    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
    year   = 2025,
    journal = {arXiv preprint arXiv:2509.23410}
}