The problem
The token tax
Language models do not read words directly. They read tokens. When one tokenizer needs more tokens than another to represent the same sentence, it spends more context and compute on the same meaning. That extra fragmentation is the token tax.
For low-resource languages, the cost compounds. There is already less high-quality text to learn from; inefficient segmentation then carries fewer complete ideas through each context window. Nepali also spans Devanagari, Romanized spelling, English code-mix, school terminology, dates, rupees, forms, and noisy web text.
Token efficiency is a compression property. It does not by itself make a language model more capable.
Method
Experimental design
We held the corpus constant and changed only the tokenizer family and vocabulary size. BPE and Unigram candidates at 24k and 32k were trained on the same shuffled five-million-row sample. Evaluation text was excluded before training.
Every candidate saw the same source mix, row count, and seed.
Fertility was measured on 24,500 excluded rows containing 9.81 million words.
Unknown tokens and content-changing round trips were blocking failures, not secondary metrics.
Candidate study
The 5M bakeoff
Vocabulary size helped both families. The 32k Unigram candidate produced the lowest held-out fertility and retained the high throughput measured in the tokenizer speed test.
Robustness
Evidence beyond the training sample
A good result on one held-out split was not enough. We added progressively harder views that were not used to select training rows.
24,500 rows and 9.81 million words established the primary matched comparison.
201,150 documents and 86 million words: Unigram 32k used 4.39% fewer tokens than BPE 32k.
89,242 records and 17.78 million words: the advantage over BPE 32k was 4.68%.
Release gate passed. Zero exact train/evaluation overlaps, unknown tokens, or content-changing round-trip failures were detected.
Public controls
General tokenizer comparison
The same frozen held-out text was encoded as raw text with Muna, OpenAI o200k_base, Qwen3, and OpenAI cl100k_base. The broad tokenizers are controls for the reading cost paid by this corpus, not opponents in a universal ranking.
आज काठमाडौंमा मौसम सफा छ र विद्यार्थीहरू विद्यालय जाँदैछन्।Interpretation
Where specialization helps
The aggregate result hides the main trade. Muna is materially more efficient on Nepali, Wikipedia, Romanized, and code-mixed text, while paying some cost on English and structured forms. The vocabulary is allocated around the intended product surface rather than broad global coverage.
Allocation
Corpus distribution is part of the design
Tokenizer quality did not come from choosing a family alone. We reserved explicit space for writing styles that matter to the intended model instead of allowing the largest web source to erase them.
Inspection
What the tokenizer learned to keep together
Aggregate scores can hide brittle segmentations. These examples show the actual pieces emitted by the selected tokenizer.
आज काठमाडौंमा मौसम सफा छ र विद्यार्थीहरू विद्यालय जाँदैछन्।Common words remain mostly whole. Eleven pieces represent the sentence.
Conclusion
Efficiency is an allocation decision
The strongest result is not that one algorithm always wins. It is that a small-language tokenizer improves when its finite vocabulary and training sample are allocated around real use: Devanagari, Romanized Nepali, education, code-mix, and public-language formats.
The selected tokenizer makes this corpus cheaper to read while preserving its text. Whether that advantage improves a language model must still be established through pretraining and downstream evaluation.
Artifact
Download the tokenizer
The exact 32k SentencePiece Unigram model used in this study is available directly. It is a tokenizer artifact, not a trained language model.