gdm32569/ternary-c64-688k

ternary.c64: a ternary transformer on an unexpanded Commodore 64

A 688,128-weight llama2 transformer with ternary weights (−1, 0, +1), running on a stock Commodore 64 with no RAM expansion. Everything on the machine is int16 fixed point. The weights live in a bank-switched EasyFlash cartridge and never enter RAM.

llama2 ternario, 34 token

 Once upon a time, there was a little girl
 named Lily. She loved to play outside in
 the sunshine.

That is the same text the model produces in floating point on a PC, token for token. It takes 24 minutes and 5 seconds on the C64.

Source code, build instructions and the full write-up: github.com/demartinogiuseppe/ternary.c64

Files

file	what it is
gen.crt	the cartridge: model, tables, vocabulary and code in one 361 KB EasyFlash image (22 banks). Same file as GitHub release v1.1.
model.safetensors	the trained weights in float32, as they come out of training (latent weights, before ternarisation)
config.json	llama2.c ModelArgs
model_ternary.bin	the ternarised model in this project's TC64 format, 242 KB: what the cartridge is built from
tok512.model	SentencePiece tokenizer, 512 tokens, trained on TinyStories
vocab.txt	the same vocabulary as plain text, used by the on-machine tokenizer

model_ternary.bin is derived from model.safetensors with export_ternary.py, and the result is byte-identical.

Run it on a Commodore 64

Download gen.crt and attach it in any emulator with EasyFlash support:

x64sc -cartcrt gen.crt

Press RETURN on an empty line and it writes the sentence above. Or type a beginning and it continues from there, tokenised on the machine:

> One day
, a little boy named Tim went to the park with his mom...

It writes about one word every forty seconds: 39.2 s at the start, 45.9 s at full context. Every prompt token costs a full forward pass too. Press Alt+W in VICE for warp mode.

It has only been run in VICE (3.10, x64sc, cycle-accurate). It has not yet been tested on real hardware. The design does not depend on exact timing, but until it has run on a real machine, treat that claim as unverified.

Run it in PyTorch

The ternary forward pass needs llama2.c's model.py and this project's bitpatch.py, which swaps the attention and feed-forward linear layers for BitLinear:

git clone https://github.com/karpathy/llama2.c
cd llama2.c
curl -O https://raw.githubusercontent.com/demartinogiuseppe/ternary.c64/main/src/bitpatch.py
pip install torch safetensors sentencepiece

import json, os, torch, sentencepiece as spm
from safetensors.torch import load_file
from model import Transformer, ModelArgs
import bitpatch

os.environ["TERNARY"] = "1"          # without it bitpatch leaves the model in float
model = Transformer(ModelArgs(**json.load(open("config.json"))))
bitpatch.apply(model)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

sp = spm.SentencePieceProcessor(model_file="tok512.model")
idx = torch.tensor([[1] + sp.encode("One day")])    # 1 = BOS
out = model.generate(idx, 34, temperature=0.0)
print(sp.decode(out[0].tolist()))
# One day, a little boy named Tim went to the park with his mom. ...

With an empty prompt (BOS only) it writes the reference sentence above.

Model

architecture	llama2 (llama2.c), RMSNorm, RoPE, SwiGLU
dim / layers / heads	128 / 4 / 8
KV heads	1 (multi-query attention)
FFN hidden	352
context	128 tokens
vocabulary	512 (SentencePiece)
ternary weights	688,128 (all attention and FFN matrices)
embedding	65,536, tied with the output head, int8 on the C64
norms	1,152

Training

Trained from scratch on TinyStories with llama2.c's train.py, patched by patch_train.py to use BitLinear: ternary weights with a per-tensor scale on the median of |W|, straight-through estimator, quantised on every forward pass.

iterations	20,000
batch	128 × 4 gradient accumulation, 128 tokens
learning rate	5e-4, cosine decay, 1,000 warmup
weight decay	0.1
optimizer	AdamW, β = (0.9, 0.95), grad clip 1.0
precision	float16
validation loss	1.4390

At dim 64 the ternary model was unusable (val loss 2.25 against 1.81 for the float baseline). At dim 128 it beats the float baseline one size down.

How it runs on the C64

• No floating point. Everything is int16 Q5.10. SiLU, exp, inverse square root and the RoPE rotations are lookup tables, 4,096 bytes in total. Activations are quantised to int8 with a shift.
• No multiplication in the ternary kernel. A table of the nine possible sums ±xa ±xb turns each byte of packed weights into two table reads: 21 cycles per multiply-accumulate.
• Attention uses a quarter-square MAC in assembly, f(a+b) − f(a−b) with f(x) = x²/4: 81 cycles per multiply-accumulate.
• Memory was the wall, not compute: 22 banks of 16 KB in the cartridge, 256 bytes of RAM left.

Every component was checked bit-exact against a C reference implementation on the PC before the next one was built: 22 checks, 22 passed.

Limitations

• It knows only children's stories. Simple English tokenises into whole words; anything else falls apart into single letters and produces nonsense.
• The C64's fixed-point pipeline is bit-exact against its own C reference, not against float. On the reference prompt it gives the same tokens as float; on an arbitrary prompt it agrees about a third of the time, usually diverging after fifteen tokens or so (measured over 150 prompts).
• Real hardware not yet tested (see above).

Credits

Built on karpathy/llama2.c (MIT). Trained on TinyStories by Ronen Eldan and Yuanzhi Li (CDLA-Sharing-1.0). Compiled with oscar64. Prior work by Maciej Witkowiak (llama2.c64) and Matt Beton (bitnet-6502) defined the gap this aims at. See NOTICE.

MIT, © 2026 Giuseppe De Martino.