---
pipeline_tag: text-generation
library_name: pytorch
language:
  - en
license: cc-by-4.0
tags:
  - research
  - semsimula
  - conservative-language-model
  - scalar-potential
  - lagrangian-mechanics
  - energy-based-model
  - physics-informed
  - splm
  - multi-channel-xi
  - structured-vtheta
  - analytical-gradient
  - interpretable
  - gaussian-mixture-potential
  - tinystories
  - non-transformer
  - attention-free
  - constant-memory-inference
  - riemannian-geometry
  - riemannian-geodesics
datasets:
  - roneneldan/TinyStories
metrics:
  - perplexity
model-index:
  - name: semsimula-splm-multixi-structured-vtheta
    results:
      - task:
          type: text-generation
        dataset:
          type: roneneldan/TinyStories
          name: TinyStories
          split: validation
        metrics:
          - type: perplexity
            value: 13.33
            name: Validation Perplexity (SQ3 K=8)
---

# Multi-Xi SPLM with Structured V_theta (SQ3 Mixture of Quadratic Wells)

A **structured variant** of the [Multi-Xi SPLM](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) conservative language model in which the MLP scalar potential \\(V_\theta\\) is replaced by a **mixture of K=8 diagonal quadratic wells** (SQ3). This replacement yields:

- **Full analytical gradients** — no `torch.autograd.grad` needed for the conservative force, eliminating the second-order computation graph (~2x speedup on force evaluation)
- **Explicit attractor centres** — the 8 semantic attractors \\(\mu_k(\xi)\\) are readable directly from the model parameters, with no gradient-descent extraction required
- **Interpretable basin structure** — attractor decoding reveals semantic specialisation: story-opening basins, family-dialogue basins, punctuation-in-quotes basins, and function-word basins

The trade-off is a modest expressivity ceiling: **13.33 PPL** vs the MLP baseline's **11.51 PPL** (5.5% excess cross-entropy), reflecting the fundamental limitation of Gaussian/quadratic potential wells compared to an unrestricted MLP.

This model is from the [Semantic Simulation](https://doi.org/10.5281/zenodo.19712427) framework.

## Table of Contents

- [When to Use This Model](#when-to-use-this-model)
- [Architecture](#architecture)
- [Structured V_theta: How It Works](#structured-v_theta-how-it-works)
- [How to Get Started](#how-to-get-started)
- [Training Details](#training-details)
- [Evaluation Results](#evaluation-results)
- [Attractor Basin Analysis](#attractor-basin-analysis)
- [Expressivity Ceiling](#expressivity-ceiling)
- [SPLM Family Overview](#splm-family-overview)
- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
- [Citation](#citation)

## When to Use This Model

Choose this structured variant over the [MLP-based Multi-Xi SPLM](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) when:

| Priority | Structured V_theta (this model) | MLP V_theta ([baseline](https://huggingface.co/dimitarpg13/semsimula-splm-multixi)) |
|---|---|---|
| **Interpretability** | 8 explicit attractor centres, zero-cost basin readout | Black-box; requires 1,500-step GD extraction per prompt |
| **Inference speed** | ~2x faster force computation (analytical gradient) | Standard (autograd) |
| **Raw PPL** | 13.33 | **11.51** |
| **Memory** | No second-order graph for V_theta | Full graph retained |

**Rule of thumb:** if you need to inspect what the model has learned (basin structure, attractor semantics, force-field geometry) or you need fast inference, use this model. If you need the best possible perplexity, use the MLP baseline.

## Architecture

```
Input tokens x_1, ..., x_T
       |
   Embedding E[x] + positional encoding
       |
   For each of L=8 integration steps:
       |
       +-- K-EMA channels: xi^(k)_t = causal_ema(h, alpha_k)   [K_xi=4 channels]
       |
       +-- Structured V_theta:
       |     xi_flat = flatten(xi_1..xi_K)                      [K_xi * d = 1024]
       |     V = -tau * logsumexp_k(-E_k/tau + log pi_k)        [K_mix=8 wells]
       |     where E_k = 0.5 * a_k(xi)^T (h - mu_k(xi))^2
       |
       +-- Analytical force: f = -grad_h V                      [closed-form, no autograd]
       |
       +-- Damped Euler step: v += dt*f/m; v /= (1 + dt*gamma); h += dt*v
       |
       +-- LayerNorm(h)
       |
   Logits = h @ E^T                                            [tied embeddings]
```

| Parameter | Value |
|---|---|
| Hidden dim (d) | 256 |
| Layers (L) | 8 |
| V_theta kind | SQ3 (mixture of K quadratic wells) |
| Mixture components (K_mix) | 8 |
| Temperature (tau) | 1.0 |
| Xi channels (K_xi) | 4 |
| V_theta input dim | (K_xi + 1) * d = 1280 |
| V_theta parameters | 4,207,625 |
| Mass model | logfreq (frozen surprisal lookup) |
| Damping gamma | 0.30 nominal, ~0.033 effective (LayerNorm prevents compounding; see note below) |
| lambda_V (output regularisation) | 0.01 |
| Total parameters | **17,335,568** |

**Effective damping.** The nominal gamma = 0.30 overstates the true dissipation. The LayerNorm applied after each integration step rescales the hidden state, absorbing most of the velocity decay. The dynamics are therefore **heavily underdamped** even at this nominal value. Gamma-sweep experiments on the OpenWebText-scale Fock-PARFLM variant confirm that the effective damping is much smaller than the nominal coefficient.

## Structured V_theta: How It Works

The standard SPLM uses an MLP to compute \\(V_\theta(\xi, h) \to \mathbb{R}\\) and obtains the conservative force via `torch.autograd.grad`. This model replaces the MLP with a **mixture of 8 diagonal quadratic energy wells**:

$$E_k(\xi, h) = \frac{1}{2} a_k(\xi)^\top (h - \mu_k(\xi))^2$$

$$V_\theta(\xi, h) = -\tau \log \sum_{k=1}^{K} \pi_k(\xi) \exp(-E_k(\xi, h) / \tau)$$

where \\(\mu_k(\xi)\\) are the **attractor centres**, \\(a_k(\xi) > 0\\) are the **diagonal precisions**, and \\(\pi_k(\xi)\\) are **mixing weights**. All are linear projections of the flattened xi context.

The **force** is computed in closed form:

$$f = -\nabla_h V_\theta = -\sum_{k=1}^{K} q_k(\xi, h) \cdot a_k(\xi) \odot (h - \mu_k(\xi))$$

where \\(q_k\\) are the softmax responsibilities. No `autograd.grad` call is needed — parameter gradients still flow through the normal `loss.backward()` chain.

At \\(\tau = 1\\), this potential is the negative log marginal likelihood of a K-component Gaussian mixture, connecting the SPLM framework directly to the Gaussian well motivation of the paper (Section 4).

For full derivations, all four structured variants (SQ1--SQ4), cost analysis, attractor basin analysis, and hyperparameter selection strategies, see the companion note: [`Structured_VTheta_Design_and_Theory.md`](https://github.com/dimitarpg13/semsimula-paper/blob/main/companion_notes/Structured_VTheta_Design_and_Theory.md).

## How to Get Started

```python
import torch, sys
sys.path.insert(0, "multixi")
sys.path.insert(0, "parf")
sys.path.insert(0, "energetic_minima")
sys.path.insert(0, "sarf_mass_variant")

from multixi.model_multixi import (
    ScalarPotentialLMSARFMassLNMultiXi,
    SPLMSARFMassLNMultiXiConfig,
)
from parf.model_structured_vtheta import MixtureQuadraticVTheta
from parf.model_structured_vtheta_multixi import StructuredVThetaMultiXiAdapter

# -- Build base model --
config = SPLMSARFMassLNMultiXiConfig(
    vocab_size=50257, d=256, n_layers=8,
    v_hidden=1024, v_depth=3,
    max_len=1024, block_size=512,
    gamma=0.30, xi_channels=4,
    xi_alpha_inits=[0.25, 0.5, 0.75, 0.95],
    xi_learnable=True, mass_mode="logfreq",
    logfreq_path="logfreq_surprisal_tinystories.npy",
)
model = ScalarPotentialLMSARFMassLNMultiXi(config)

# -- Swap in structured V_theta --
K_xi, d = 4, 256
inner = MixtureQuadraticVTheta(d=K_xi * d, K=8, tau=1.0)
model.V_theta = StructuredVThetaMultiXiAdapter(inner, K=K_xi, d=d)

# -- Load checkpoint --
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
    repo_id="dimitarpg13/semsimula-splm-multixi-structured-vtheta",
    filename="checkpoint/ckpt_best.pt",
)
state = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(state["model_state_dict"])
model.eval()

print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")

# -- Read attractor centres directly --
# (no gradient descent needed!)
x = torch.randint(0, 50257, (1, 64))
with torch.no_grad():
    h = model.embedding(x) + model.pos_enc[:, :64]
    xis = model._compute_xis(h)                        # (1, 64, 4, 256)
    centres = model.V_theta.attractor_centres(xis)      # (1, 64, 8, 1024)
    print(f"Attractor centres shape: {centres.shape}")   # 8 basins per token
```

### Available Artifacts

| File | Description |
|---|---|
| `checkpoint/ckpt_best.pt` | Best checkpoint (A2 arm, 13.33 PPL) |
| `training_log.jsonl` | Per-step training metrics |
| `training_curve_A2.png` | Training/validation loss curves |
| `v_theta_hist_A2.png` | V_theta output distribution histogram |
| `landscape_stats_A2.json` | V_theta landscape statistics (mean, std, range) |
| `attractors_A2.json` | Decoded attractor centres for 5 prompt types |
| `summary_A2.md` | Hyperparameters and final metrics |
| `model_structured_vtheta.py` | Structured V_theta classes (SQ1--SQ4) |
| `model_structured_vtheta_multixi.py` | Multi-Xi adapter |
| `config.json` | Model configuration |

## Training Details

### Training Data

[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) — a synthetic corpus of short children's stories generated by GPT-3.5/4, tokenized with GPT-2 BPE (vocab size 50,257). Training cap: 5M tokens; validation: ~140k tokens.

### Training Procedure

The base model architecture is identical to the [Multi-Xi SPLM](https://huggingface.co/dimitarpg13/semsimula-splm-multixi). The only modification is the \\(V_\theta\\) replacement: the 3-layer MLP is swapped for the SQ3 mixture at model construction time, before training begins from scratch.

| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning rate | 5e-4 (cosine decay) |
| Warmup steps | 400 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Batch size | 16 |
| Block size | 512 |
| Training steps | 16,000 |
| lambda_V (V_theta regularisation) | 0.01 |
| Hardware | NVIDIA A100 40GB (Google Colab) |

### Sweep Arms

Six arms were trained. This checkpoint is from **A2**, the best-performing arm:

| Arm | K_mix | tau | K_xi | Best PPL | V_theta params |
|---|---|---|---|---|---|
| A1 | 4 | 1.0 | 4 | 14.10 | 2.1M |
| **A2** | **8** | **1.0** | **4** | **13.33** | **4.2M** |
| A3 | 4 | 0.5 | 4 | 14.10 | 2.1M |
| A4 | 4 | 1.0 | 8 | 14.16 | 4.2M |
| A5 | 8 | 1.0 | 8 | 14.25 | 8.4M |
| B1 (MLP ref.) | — | — | 4 | 11.51 | — |

### Training Script

[`notebooks/conservative_arch/scaleup/colab_splm_multixi_structured_vtheta.ipynb`](https://github.com/dimitarpg13/semsimula-paper/blob/main/notebooks/conservative_arch/scaleup/colab_splm_multixi_structured_vtheta.ipynb) — Colab notebook with all 6 arms, GDrive output, checkpointing, and live progress display.

## Evaluation Results

### TinyStories Validation Perplexity

| Model | PPL | Params | Analytical grad | Explicit attractors |
|---|---|---|---|---|
| Matched Attention (baseline) | **7.81** | 19.5M | — | — |
| [Fock-PARFLM v2.1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) | **9.70** | 17.4M | No | No |
| [Multi-Xi SPLM (MLP)](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) | **11.51** | 16.5M | No | No |
| [Multi-Xi PARFLM](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) | **12.06** | 17.6M | No | No |
| **Multi-Xi SPLM (SQ3, this model)** | **13.33** | 17.3M | **Yes** | **Yes (8 basins)** |

### V_theta Landscape Statistics

| Metric | Value |
|---|---|
| Mean V_theta | 99.8 |
| Std V_theta | 26.3 |
| Range | 644.9 |
| Min | 9.0 |
| Max | 653.9 |

## Attractor Basin Analysis

The 8 mixture components develop **semantic specialisation** during training. Projecting each attractor centre \\(\mu_k(\xi)\\) through the LM head on five TinyStories prompt types reveals:

- **Story-opening basin** (narrative, basin 8): "lived" at 53.7% probability, followed by "wanted", "asked", "said"
- **Family-dialogue basin** (dialogue, basin 1): "dad" at 19.3%, "mom" at 15.0%, "it" at 6.6%
- **Punctuation-in-quotes basin** (dialogue, basin 7): ',"' at 19.9%, '!"' at 3.6%, '."' at 3.2%
- **Function-word basins**: several basins decode to "of", "to", "and" at 6–9% each
- **Unused basins** (~3 of 8): decode to near-uniform distributions (all top-5 probabilities less than 5e-5), indicating the model self-selects \\(K_{\text{eff}} \approx 5\\)

This basin structure is readable **directly from the model parameters** — no gradient-descent extraction is needed.

## Expressivity Ceiling

The 1.82 PPL gap between this model (13.33) and the MLP baseline (11.51) reflects a **fundamental expressivity ceiling** of the quadratic-well family:

1. **Gaussian equivalence.** Each quadratic energy well corresponds, via the Boltzmann factor, to a Gaussian distribution. The mixture of K wells is therefore equivalent to a K-component GMM, which requires K growing exponentially with intrinsic complexity.

2. **Force-field linearity.** Within each basin, the restoring force is linear in h. An MLP can generate arbitrarily nonlinear force fields. Specifically, \\(-\nabla_h V_k = -A_k(h - \mu_k)\\) is a linear function of h, while an MLP V_theta admits arbitrary nonlinearities.

3. **No inter-basin gradients.** Outside all basins (transition regions), the SQ3 force is a responsibility-weighted average of linear forces — still fundamentally linear in h within each local region. The MLP captures sharp transitions that the soft mixture cannot.

The gap is small enough (5.5% excess cross-entropy) that structured \\(V_\theta\\) remains attractive for applications that value interpretability or computational efficiency over the last few PPL points.

## SPLM Family Overview

This model is part of the **Semantic Simulation SPLM family**:

| Model | Design | Corpus | PPL | HuggingFace |
|---|---|---|---|---|
| Multi-Xi SPLM (MLP) | Pure scalar potential | TinyStories | 11.51 | [semsimula-splm-multixi](https://huggingface.co/dimitarpg13/semsimula-splm-multixi) |
| **Multi-Xi SPLM (SQ3)** | **Structured scalar potential** | **TinyStories** | **13.33** | **this model** |
| Multi-Xi PARFLM (MLP) | Scalar + pairwise forces | TinyStories | 12.06 | [semsimula-parflm-multixi](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi) |
| Multi-Xi PARFLM (SQ3) | Structured scalar + pairwise | TinyStories | 12.27 | [semsimula-parflm-multixi-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-parflm-multixi-structured-vtheta) |
| Fock-PARFLM v2.1 (MLP) | PARFLM + Fock registers | TinyStories | 9.70 | [semsimula-fock-parflm](https://huggingface.co/dimitarpg13/semsimula-fock-parflm) |
| Fock-PARFLM v2.1 (SQ3) | Structured + pairwise + Fock | TinyStories | 10.90 | [semsimula-fock-parflm-structured-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-structured-vtheta) |
| Fock-PARFLM v2.1 (depth-cond. isotropic Gaussian) | Bounded multi-context + pairwise + Fock | TinyStories | 16.33 | [semsimula-fock-parflm-depthcond-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-depthcond-vtheta) |
| Fock-PARFLM v2.1 (depth-cond. anisotropic Gaussian + fock-reg) | Bounded, ellipsoidal multi-context + pairwise + Fock | TinyStories | 9.04 | [semsimula-fock-parflm-anisogaussian-vtheta](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, first-order/Fock-G1) | Same as above, gradient-flow integrator | TinyStories | 8.95 | [semsimula-fock-parflm-anisogaussian-vtheta-fock-g1](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-fock-g1) |
| Fock Attention (MLP V_theta) | Fock + attention | TinyStories | 9.42 | [semsimula-fock-attention](https://huggingface.co/dimitarpg13/semsimula-fock-attention) |
| Hybrid SPLM+Attn | Attention + SPLM refinement | TinyStories | 8.50 | [semsimula-hybrid-splm](https://huggingface.co/dimitarpg13/semsimula-hybrid-splm) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=384) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 278.27 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=768) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 326.97 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d768-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, gamma sweep, d=1024) | Bounded, ellipsoidal multi-context + pairwise + Fock — 8-way gamma sweep + geodesic analysis | OpenWebText | 244.23 (best of 8, 3K-step sweep) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d1024-gammasweep) |
| Fock-PARFLM v2.1 (aniso-Gaussian + fock-reg, Verlet instability, d=384) | Same architecture, two full-run attempts — SCAF stiffness audit identifies structural Verlet instability | OpenWebText | 184.11 / 211.63 (both runs stalled, not final) | [semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability](https://huggingface.co/dimitarpg13/semsimula-fock-parflm-anisogaussian-vtheta-owt-d384-verlet-instability) |

**Collection:** [Semantic Simulation SPLM Model Family](https://huggingface.co/collections/dimitarpg13/semantic-simulation-splm-model-family)

## Bias, Risks, and Limitations

- **Research checkpoint only.** This model is a proof-of-concept for structured scalar potentials, not a production system.
- **TinyStories only.** Trained exclusively on synthetic children's stories (~5M tokens). Not suitable for general-purpose language generation.
- **English only.** No multilingual capability.
- **Small scale.** 17.3M parameters, 256-dim hidden states.
- **No safety training.** No RLHF, DPO, or safety filtering has been applied.
- **Expressivity ceiling.** The structured \\(V_\theta\\) has a demonstrated 5.5% excess cross-entropy vs the MLP baseline. Applications requiring maximum PPL should use the MLP variant.

## Citation

```bibtex
@misc{Gueorguiev2026SemSim,
  author    = {Gueorguiev, Dimitar P.},
  title     = {Semantic Simulation: A Prescriptive Lagrangian Framework
               for Efficient Semantic Inference --- A Conservative-by-
               Construction Language Model and the Shared-Potential
               Separator, with a Correspondence to Joint Embedding
               Predictive Architectures},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19712427},
  url       = {https://doi.org/10.5281/zenodo.19712427},
  note      = {Version v15 (Jun 7, 2026).
               Companion code repository (DOI 10.5281/zenodo.20579561):
               \url{https://github.com/dimitarpg13/semsimula-paper}}
}
```

## Environmental Impact

- **Hardware:** NVIDIA A100 40GB (Google Colab)
- **Training time:** ~1.5 hours (16,000 steps, A2 arm)
- **Carbon footprint:** Estimated less than 1 kg CO2