Metraipolis › archive › dc_65e4f0eafd67
Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining — arXiv:2610.33016
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.33016
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-27T07:59:24.557Z
- SHA-256 of the archived text
111cf6d4d6649823e1ec13e3390a8b2b2c954b92ffdbf6406c9e179a8091e394- Stable URL of this page
https://metraipolis.com/documento/dc_65e4f0eafd67
How to cite
Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining — arXiv:2610.33016 — arXiv (en), https://arxiv.org/abs/2610.33016 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_65e4f0eafd67 (Metraipolis archive dc_65e4f0eafd67, sha256 111cf6d4d6649823e1ec13e3390a8b2b2c954b92ffdbf6406c9e179a8091e394)
BibTeX
@misc{dc65e4f0eafd67,
title = {Merge-Based Vocabulary Extension Stalls at the Pre-Tokenizer: An English-Preserving Repair for Nepali, Tested in Continued Pretraining — arXiv:2610.33016},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.33016},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_65e4f0eafd67; sha256 111cf6d4d6649823e1ec13e3390a8b2b2c954b92ffdbf6406c9e179a8091e394},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33016. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Llama-3, Qwen3 and many other open LLMs pre-tokenize text with a regular expression whose word class admits only Unicode letters. Devanagari vowel signs are combining marks, so this regex splits Nepali words at every vowel sign, and models that use it spend 2.6 to 5.8 times as many tokens on Nepali as on the same English text. Merge-based vocabulary extension cannot join the split pieces: for Llama-3 and Qwen3 no number of new merges brings this premium below about 2.2, and ten released extensions already end within 4% of this pre-token floor. We give a small edit to the deployed regex that provably keeps every token id of text without combining marks or Devanagari characters. With 32,000 new merges it brings Nepali to 1.01 times English, about 54% fewer Nepali tokens than standard extension. In small-budget continued pretraining of Llama-3.2-1B and Qwen3-0.6B, the original tokenizer gave the lowest held-out Nepali bits per byte under every recipe we tested. Matching the repair's training steps to standard extension, with about 35% of the data repeated, reversed their order on Qwen but not on Llama. Whether the token savings give a favourable latency and quality trade-off in generation remains untested. We release the tokenizers and code. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.33016 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-27T07:55:30.786Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).