Research objective
Large multilingual systems distribute their capacity across many languages and tasks. Muna explores a narrower question: can a compact decoder-only model become useful for Nepali, Romanized Nepali, and education-oriented text when its data and tokenizer are designed around that surface?
This is a real pretrained checkpoint, not an API wrapper. It is also not a production-ready general assistant. The work is a measured step toward a model that can eventually run closer to the user, including offline and constrained-device settings.
Model architecture
The architecture favors depth and parameter efficiency over a wide hidden state. Grouped-query attention reduces key-value state, tied embeddings avoid duplicating the vocabulary matrix, and the 32k Muna tokenizer keeps the input and output vocabulary compact.
Gated feed-forward layers improve expressiveness within a limited parameter budget.
Pre-normalized blocks and rotary position encoding keep the decoder design simple.
Three key-value heads and one shared vocabulary matrix reduce memory cost.
Pretraining
The checkpoint records 3,305,373,696 training tokens. At 161,801,280 parameters, that is approximately 20.4 training tokens per parameter: a deliberate balance between model capacity and available corpus scale.
BF16 checkpoint weights occupy about 617 MB on disk.
The training data includes Devanagari Nepali, Romanized language, Wikipedia, education material, English anchors, and structured forms.
The selected 32k Unigram tokenizer was evaluated separately for fertility, Unicode safety, and unseen-text behavior.
The parameter count and token budget are factual properties of this run. They are not evidence that the checkpoint matches larger general-purpose models; capability must be demonstrated task by task.
Adaptation evaluation
A small supervised adaptation experiment used 3,627 source rows, expanded to 5,635 effective examples, with 304 validation rows. The run completed 705 steps without paid API generation.
सन्दर्भ: नेपालको संविधान वि.सं. २०७२ असोज ३ गते जारी भएको हो।
प्रश्न: नेपालको संविधान कहिले जारी भएको हो?
नेपालको संविधान वि.सं. २०७२ असोज ३ गते जारी भएको हो।
The example above is copied from the recorded evaluation output. The adaptation remains a research checkpoint while translation quality is improved.
Try Muna
Muna runs locally in this browser, not on a hosted inference server. The first visit downloads the 361 MB model once; supported browsers keep that copy on the device so later sessions can work without Wi-Fi. Browser acceleration is used when available, with CPU inference as the fallback.
Choose a textbook question to observe the model generate an answer on your device.
Muna generates each visible answer locally from the supplied textbook context. Questions and answers do not leave the device.
Local and offline inference
Muna has a native CPU inference path and a browser build that runs entirely on the reader's device. The browser chooses the fastest local path it supports and falls back to CPU execution without sending prompts to an inference API.
After the model has been prepared once, the browser can reuse its local copy without Wi-Fi. The next deployment study focuses on a smaller quantized build for faster startup, lower memory use, and practical phone performance.
Limitations and next steps
Muna establishes the core pieces of a Nepali-centered language-model stack: a measured tokenizer, a completed multi-billion-token pretraining run, native local inference, and evaluation gates that can reject an adaptation even when its loss improves.
The next work is less glamorous and more important: improve translation and instruction behavior, evaluate knowledge and safety beyond templated tasks, quantify contamination, build retrieval for factual use, and test compressed variants on real phones. Until those gates pass, the pretrained model remains a research checkpoint.