Metraipolis › archive › dc_e987b4783d26
Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition — arXiv:2609.12960
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.12960
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:08:26.613Z
- SHA-256 of the archived text
8495098830c38202afa72bd095db20575de35d38b3343dc206c5a5abd1bbd8ea- Stable URL of this page
https://metraipolis.com/documento/dc_e987b4783d26
How to cite
Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition — arXiv:2609.12960 — arXiv (en), https://arxiv.org/abs/2609.12960 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e987b4783d26 (Metraipolis archive dc_e987b4783d26, sha256 8495098830c38202afa72bd095db20575de35d38b3343dc206c5a5abd1bbd8ea)
BibTeX
@misc{dce987b4783d26,
title = {Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition — arXiv:2609.12960},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.12960},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e987b4783d26; sha256 8495098830c38202afa72bd095db20575de35d38b3343dc206c5a5abd1bbd8ea},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12960. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sanskrit fuses case, number, person and tense into word endings and chains clauses into compounds, so it is information-dense per word. Whether that density survives subword tokenization is a separate question, to be asked per unit of meaning rather than per word. On identical FLORES-200 devtest content, Sanskrit costs 1.774-2.187 times the English tokens under deployed tokenizers with vocabularies of 200,019 ids or more, but only 1.325-1.353 times the Hindi tokens. Against a deployed English tokenizer, Sanskrit-trained BPE arms then look cheaper per proposition than English on contemporary prose (0.887). Against a matched English control, the same algorithm and vocabulary trained on the English side of the same corpus, that flip disappears: at 32,000 and 64,000 pieces all 8 matched pairs, each size-matched arm against both a pair-matched and a byte-matched control, sit above 1.0 on prose with 95% intervals excluding it. The gap closes as the vocabulary grows: at 128,000 pieces the BPE pair reads 0.983 in domain while staying above parity out of domain (1.025) and on FLORES (1.116). The ratio factorises into a character-length ratio and a tokens-per-character ratio, the second near 1 throughout: what survives matched tokenization is character-level length, which Sanskrit prose lacks over English in SLP1 (1.028) and Sanskrit verse has (0.596). The robust statement is about deployed practice: on contemporary prose and on FLORES, with the Sanskrit side in SLP1 against the deployed o200k English pivot, Sanskrit costs 1.831-2.899 English tokens per proposition under the tokenizers people actually ship. Code, the results snapshot and every table here are public. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.12960 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T03:02:47.552Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).