Metraipolis › archive › dc_ca47e84e463b
Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics — arXiv:2611.12628
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.12628
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-10T18:44:13.288Z
- SHA-256 of the archived text
773381eba6c20acc1c362dc14137c85de8ec21c31be18abb7c1e6f7b053a56e1- Stable URL of this page
https://metraipolis.com/documento/dc_ca47e84e463b
How to cite
Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics — arXiv:2611.12628 — arXiv (en), https://arxiv.org/abs/2611.12628 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca47e84e463b (Metraipolis archive dc_ca47e84e463b, sha256 773381eba6c20acc1c362dc14137c85de8ec21c31be18abb7c1e6f7b053a56e1)
BibTeX
@misc{dcca47e84e463b,
title = {Adapting Small Language Models to Romanized Nepali: Tokenizer Fertility, Reference Likelihood, and Generation Metrics — arXiv:2611.12628},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.12628},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca47e84e463b; sha256 773381eba6c20acc1c362dc14137c85de8ec21c31be18abb7c1e6f7b053a56e1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.12628. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Romanized Nepali, Nepali written in Latin script, is common in digital communication but has no standard spelling and few resources, and it is unclear how reliably small open language models can be adapted to it. We adapt three 2-3B base models from different families, Llama-3.2-3B, Qwen2.5-3B, and Gemma-2-2B, with QLoRA on 9,000 instruction examples produced by deterministic transliteration of Nepali instruction data. Each model is trained with one shared recipe and three seeds and evaluated on 1,000 held-out examples with reference-overlap metrics, byte-normalized reference likelihood (bits per byte, BPB), and an output audit, against zero-shot baselines under two prompts. Fine-tuning raises chrF++ from 10-14 to 29.0-30.6 and lowers BPB from 2.23-2.41 to 0.78-0.84; most zero-shot outputs (54-92%) are English-like. Likelihood and overlap disagree on how far apart the models are. Llama-3.2-3B has the lowest BPB in every seed, 0.042 below Qwen2.5-3B (95% CI 0.039-0.046), yet the two cannot be separated on chrF++ (30.51 and 30.61) or BLEU, even with paired item-level tests. Gemma-2-2B, whose tokenizer splits Romanized Nepali words least (2.72 tokens per word), scores worst on every measure. Which model gains most over its zero-shot baseline depends on the zero-shot prompt. In a supplementary comparison with an identical tokenizer, Llama-3.1-8B lowers BPB by 0.053 (0.050-0.056) relative to Llama-3.2-3B, while chrF++ and BLEU rise by only 0.68 and 0.58 points. Lower fertility and higher reference likelihood therefore did not identify the model whose outputs were closest to the references. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.12628 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-10T18:37:37.701Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).