Metraipolis › archive › dc_9738683165cc
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali — arXiv:2609.33395
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33395
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:52:59.664Z
- SHA-256 of the archived text
5134c9ce2046a141587a443a713fa671caf21aaffb702ed281a306168e672e74- Stable URL of this page
https://metraipolis.com/documento/dc_9738683165cc
How to cite
Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali — arXiv:2609.33395 — arXiv (en), https://arxiv.org/abs/2609.33395 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9738683165cc (Metraipolis archive dc_9738683165cc, sha256 5134c9ce2046a141587a443a713fa671caf21aaffb702ed281a306168e672e74)
BibTeX
@misc{dc9738683165cc,
title = {Preserving Morphemes: Morphology-Guided Pre-Tokenization for Nepali — arXiv:2609.33395},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33395},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9738683165cc; sha256 5134c9ce2046a141587a443a713fa671caf21aaffb702ed281a306168e672e74},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33395. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A byte-level BPE vocabulary learns each inflected form of a Nepali word as a separate string, so a noun stem is spelled differently in each of its case-marked forms. We test whether splitting words into stem and affixes before BPE helps, with the corpus, vocabulary size, model and number of training steps held fixed. Our pre-tokenizer, Papaya, uses a finite-state transducer built from a published grammar of Nepali, falls back to regular expressions, and leaves the BPE trainer unchanged. On 607 words annotated by seven native speakers its segmenter reaches 0.96 boundary F1, and the resulting tokens keep stems intact far more often than plain BPE does. In a 17M-parameter language model it lowers bits per byte by about 1% at equal training steps; most of the larger gain seen at equal epochs comes from the extra steps that longer token sequences buy, and an unsupervised Morfessor segmentation gives the same improvement. Downstream the effect is small: NER improves only on entities that contain words unseen in training, POS tagging and news classification do not change, and published Nepali tokenizers perform about as well. We release the annotated boundary set, a 556-affix dataset and the code. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.33395 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.514Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).