1386.ai.rocm — Plasma 1.1 (500M)

A 500M-parameter transformer trained from scratch on a GMKtec Evo X2 mini PC (AMD Strix Halo APU, Radeon 8060S, 128 GB unified memory) with ROCm/PyTorch. No pretrained weights, ~3 weeks wall-clock at ~4,750 tok/s on the integrated GPU. Code: a ROCm port of eb1386/1386.ai.

What's in this repo

File	Checkpoint	Notes
plasma-1.1-ft-f16.gguf	instruction-tuned final (step 30k)	GGUF f16, llama.cpp-ready
plasma-1.1-pretrain-f16.gguf	pretrain final (step 200k)	GGUF f16, llama.cpp-ready
model.safetensors	instruction-tuned final	HF LlamaForCausalLM layout, f16, tied embeddings
config.json	—	1.1 architecture (26×1280, GQA 20/4, vocab 48k)
tokenizer.model	—	SentencePiece, 48k, trained on the 1.1 corpus

Architecture: 26 layers, hidden 1280, 20 query / 4 KV heads (GQA), head_dim 64, SwiGLU intermediate 3584, RMSNorm, RoPE θ=10000, context 1024, vocab 48000. ~575M parameters counting the tied embed/output matrix.

Training: pretrain 200k steps (FineWeb-Edu-hq, Wikipedia, code, books, arXiv shards), then instruction-tune 30k steps on synthetic User: …\nAssistant: … data with loss masked to assistant tokens.

Perplexity (llama.cpp, instruction-prose slice): ft 3.0, pretrain 13.4.

Honest limitations

• Format > facts. The ft model writes well-shaped markdown answers but is frequently wrong or hallucinated (it will "reverse a string" with a try/except + JSON example), and it likes to emit EOS early on instruct prompts. A 30k-step tune at 1.5e-5 LR mostly buys style.
• 1024-token context. Chat format is plain text — User: question\nAssistant: — no chat template, no special tokens beyond the SentencePiece defaults.
• The pretrain GGUF is base-model continuation only; use the ft one for instruct-style prompts.

RoPE caveat (important for conversions)

The training code rotates RoPE pairs as (2i, 2i+1) (GPT-J/interleaved style), while llama.cpp's llama arch expects split-half (i, i+32) pairing. These GGUFs have q/k_proj rows permuted to compensate — so llama.cpp's default rotary settings work correctly, and --rope-* overrides are neither needed nor wanted. If you re-convert a checkpoint with a pre-fix convert.py, the file loads but emits control-token garbage (PPL ~3000). Repo commit 4c32c49 contains the fix; verify any conversion with llama-perplexity.

Usage

llama-cli -m plasma-1.1-ft-f16.gguf -p "User: What is the capital of France?\nAssistant:" -n 100

Works with any llama.cpp build ≥ b5000. f16 → q8_0 is ~600 MB, q4_k_m ~330 MB, both fine at this size.

Trained May–June 2026 on a Strix Halo, in a docker container, on integrated graphics, because someone had to.