ftrproger/KazTok-BPE-100k-v5

KazTok BPE 100k v5

KazTok BPE 100k v5 is an open Kazakh-oriented ByteLevel BPE tokenizer with a 100,000-token vocabulary. It was trained on a hash-verified Kazakh training split. Its byte-level representation supports lossless encoding and decoding of arbitrary UTF-8 text, including mixed Kazakh, Russian, English, punctuation and emoji.

This repository contains a tokenizer, not a chat model or trained LLM.

Why 100k?

The KazTok study compared 50k, 100k, 150k and 300k vocabularies in a controlled, approximately parameter-matched small decoder proxy. At approximately equal raw-text exposure (about 8M characters; three seeds), 100k and 150k had effectively identical mean BPC: 2.994455 and 2.993932. The 100k condition used 2.653 GiB rather than 3.761 GiB peak VRAM and averaged 45.77 rather than 56.47 seconds. We release 100k as a practical default for this compute regime, not as a claim of universal or statistically proven superiority.

Intended use

• Kazakh-centric tokenizer research and small-model experiments.
• Training a new model after checking compatibility with its embeddings and architecture.
• Same-text compression comparisons with other tokenizers.

Training and evaluation record

Item	Record
Algorithm	ByteLevel BPE
Vocabulary size	100,000
Training source	Verified Leipzig Kazakh split
Training split SHA-256	b399db1387bcb0e2d55fdc79a793f3b322ac2adc38d045fa78cf0a62365e1a66
Tokenizer SHA-256	de97870bfdcd3a265418708734a5933a414d32588fc079741882e77e9e02b3f7
Unicode round-trip	100% on the project regression suite

On a frozen 1,010-sentence UD Kazakh-KTB external test after exact full-row Leipzig overlap removal, 100k used 193.23 tokens per 1,000 characters and achieved 100% exact decode. This is a tokenization-efficiency measurement, not a ranking of LLM quality.

Limitations

• The source is primarily one corpus family and does not cover all Kazakh domains or dialects.
• Compression does not demonstrate better QA, reasoning, translation, safety or general language quality.
• A tokenizer cannot be swapped into an already-trained LLM without retraining or a carefully validated embedding-transfer method.

Citation

@software{kaztok_2026,
  title = {KazTok BPE 100k v5},
  author = {Zhanbyrbay, Nuraly},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/ftrproger/KazTok-BPE-100k-v5}
}

Maintainer: Nuraly Zhanbyrbay