Sonexa-Beta

Sonexa-Beta is a lightweight, bilingual (English / Russian) language model powered by a custom Mixture of Experts (MoE) architecture. Built from scratch for ultra-low latency inference, edge devices, and consumer hardware.

• Developer / Organization: Sonexa Artificial Intelligence
• Model Architecture: Causal Language Model with Mixture of Experts (MoE)
• License: Apache-2.0

---

Architectural Specifications

The following metrics are dynamically extracted directly from the model weights:

Metric	Measured Value
Total Parameters	140,933,632 (140.93M)
Active Parameters (All-Experts / Soft MoE)	140,933,632 (140.93M)
Active Parameters (Top-1 MoE Mode)	70,154,752 (70.15M)
Tied Embedding Weights	8,192,000 (8.19M)
Attention Layers (GQA)	6,558,720 (6.56M)
Shared Expert Capacity	23,592,960 (23.59M)
Routed Experts Capacity (4x)	94,371,840 (94.37M)
Hidden Size	512
Intermediate Size (SwiGLU)	1536
Attention Mechanism	Grouped Query Attention (8 Query Heads / 2 KV Heads)
Positional Encoding	Rotary Position Embeddings (Real Cos/Sin RoPE)
Vocabulary Size	16,000 (Byte-Level BPE)
Context Window	256 tokens
Memory Footprint (FP16)	~268.81 MB VRAM
Release Artifact Size	537.64 MB

---

Key Characteristics

1. Bilingual Dialogue: Trained on a balanced bilingual corpus combining human-curated English dialogues (HuggingFaceH4/no_robots) and filtered Russian conversational datasets.
2. Anti-Refusal Calibration: Calibrated via output projection damping to eliminate generic AI assistant disclaimers ('As an AI language model...', 'I do not have personal opinions...').
3. Soft MoE Routing: Concurrent all-expert evaluation dynamic weighting for consistent knowledge routing and response quality.
4. Extreme Portability: Fits comfortably in under 300 MB of memory in FP16, allowing fast CPU and consumer GPU execution.

---

Prompt Template

Sonexa-Beta expects inputs formatted as follows:

<|user|>
Your message here
<|bot|>
Model response here<|endoftext|>

---

Citation

@misc{sonexa2026,
  author = {Sonexa Artificial Intelligence},
  title = {Sonexa-Beta: Lightweight Bilingual Mixture-of-Experts Language Model},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{[https://huggingface.co/](https://huggingface.co/)Cartik/Sonexa-Beta}}
}