Research objective
Large multilingual systems distribute their capacity across many languages and tasks. Muna explores a narrower question: can a compact decoder-only model become useful for Nepali, Romanized Nepali, and education-oriented text when its data and tokenizer are designed around that surface?
This is a real pretrained checkpoint, not an API wrapper. It is also not a production-ready general assistant. The work is a measured step toward a model that can eventually run closer to the user, including offline and constrained-device settings.
Model architecture
The architecture favors depth and parameter efficiency over a wide hidden state. Grouped-query attention reduces key-value state, tied embeddings avoid duplicating the vocabulary matrix, and the 32k Muna tokenizer keeps the input and output vocabulary compact.
Gated feed-forward layers improve expressiveness within a limited parameter budget.
Pre-normalized blocks and rotary position encoding keep the decoder design simple.
Three key-value heads and one shared vocabulary matrix reduce memory cost.
Pretraining
The checkpoint records 3,305,373,696 training tokens. At 161,801,280 parameters, that is approximately 20.4 training tokens per parameter: a deliberate balance between model capacity and available corpus scale.
BF16 checkpoint weights occupy about 617 MB on disk.
The training data includes Devanagari Nepali, Romanized language, Wikipedia, education material, English anchors, and structured forms.
The selected 32k Unigram tokenizer was evaluated separately for fertility, Unicode safety, and unseen-text behavior.
The parameter count and token budget are factual properties of this run. They are not evidence that the checkpoint matches larger general-purpose models; capability must be demonstrated task by task.
Adaptation evaluation
A small supervised adaptation experiment used 3,627 source rows, expanded to 5,635 effective examples, with 304 validation rows. The run completed 705 steps without paid API generation.
सन्दर्भ: नेपालको संविधान वि.सं. २०७२ असोज ३ गते जारी भएको हो।
प्रश्न: नेपालको संविधान कहिले जारी भएको हो?
नेपालको संविधान वि.सं. २०७२ असोज ३ गते जारी भएको हो।
The example above is copied from the recorded evaluation output. The adaptation remains a research checkpoint while translation quality is improved.
Try Muna
This restricted preview runs the adapted 161.8M-parameter model. Choose a verified textbook question below or type a close paraphrase. Questions outside the verified set are declined before the model runs.
Select a question to see the checkpoint generate an answer token by token.
The retrieved record supplies verified evidence; every visible answer is generated by Muna. This is a narrow model demonstration, not a general-purpose assistant.
CPU inference
The native inference path runs on CUDA or CPU, with no hosted model or external generation API. Runtime memory includes the 617 MB weight file together with activations and the key-value cache.
The mobile deployment study covers quantization, runtime integration, latency, memory use, and thermal behavior on physical devices.
Limitations and next steps
Muna establishes the core pieces of a Nepali-centered language-model stack: a measured tokenizer, a completed multi-billion-token pretraining run, native local inference, and evaluation gates that can reject an adaptation even when its loss improves.
The next work is less glamorous and more important: improve translation and instruction behavior, evaluate knowledge and safety beyond templated tasks, quantify contamination, build retrieval for factual use, and test compressed variants on real phones. Until those gates pass, the pretrained model remains a research checkpoint.