Metraipolis › archive › dc_d53efb119d4d
Type-Driven Tokenization for Brahmic Scripts — arXiv:2609.22125
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.22125
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:48:52.803Z
- SHA-256 of the archived text
c8c2d55e2dd630ef5c0be347f93ddf852ccfdbd4c00f6137099f7dff9334f3fb- Stable URL of this page
https://metraipolis.com/documento/dc_d53efb119d4d
How to cite
Type-Driven Tokenization for Brahmic Scripts — arXiv:2609.22125 — arXiv (en), https://arxiv.org/abs/2609.22125 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d53efb119d4d (Metraipolis archive dc_d53efb119d4d, sha256 c8c2d55e2dd630ef5c0be347f93ddf852ccfdbd4c00f6137099f7dff9334f3fb)
BibTeX
@misc{dcd53efb119d4d,
title = {Type-Driven Tokenization for Brahmic Scripts — arXiv:2609.22125},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.22125},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d53efb119d4d; sha256 c8c2d55e2dd630ef5c0be347f93ddf852ccfdbd4c00f6137099f7dff9334f3fb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22125. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Standard tokenizers used in large language models produce malformed text when applied to Brahmic scripts. They are a family of abugidas, writing systems whose consonants carry an inherent vowel that dependent marks can modify. They include Devanagari, Telugu, Tamil, Kannada, and others. The underlying issue is that these tokenizers violate orthographic constraints that do not arise in alphabetic scripts like English. We observe that while English orthography forms a \emph{semigroup} (any two valid tokens can be freely concatenated), Brahmic orthography forms a \emph{partial semigroup}: not every concatenation yields a valid string. We formalise this distinction in Agda, model valid Brahmic tokens as chains in a transition system, and derive a provably correct \texttt{fixToken} function that extends any candidate token to respect orthographic boundaries. We then show how this formal derivation translates into a practical patch for SentencePiece as well as a standalone Rust-based pre-tokenizer library, eliminating the observed errors across Indic scripts.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Formal Languages and Automata Theory (cs.FL).
Pagina del lavoro: https://arxiv.org/abs/2609.22125 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:44.834Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).