Metraipolis › archive › dc_f933654826bb

SuTRA : Structurally-Unified Tokenization with Root Awareness — arXiv:2608.18087

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.18087
License (as archived)
CC0-1.0
Archived
2026-10-18T13:44:44.648Z
SHA-256 of the archived text
3c49334da38899593ecfc2a66ed703ce1aaa64b827ce4d26fdec7d677eb3ae7a
Stable URL of this page
https://metraipolis.com/documento/dc_f933654826bb

How to cite

SuTRA : Structurally-Unified Tokenization with Root Awareness — arXiv:2608.18087 — arXiv (en), https://arxiv.org/abs/2608.18087 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f933654826bb (Metraipolis archive dc_f933654826bb, sha256 3c49334da38899593ecfc2a66ed703ce1aaa64b827ce4d26fdec7d677eb3ae7a)
BibTeX
@misc{dcf933654826bb,
  title = {SuTRA : Structurally-Unified Tokenization with Root Awareness — arXiv:2608.18087},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18087},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f933654826bb; sha256 3c49334da38899593ecfc2a66ed703ce1aaa64b827ce4d26fdec7d677eb3ae7a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18087. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Existing subword tokenizers optimize statistical compression but ignore morphological structure, particularly the relationship between roots and affixes. This is harmful for morphologically rich Indic languages, where basic units are complex orthographic syllables (aksharas) rather than letters. Frequency-based methods over-fragment words, arbitrarily splitting roots and affixes - a phenomenon we term Morphological Shattering. We propose SuTRA (Structurally-Unified Tokenization with Root Awareness), a morphology-aware algorithm that preserves akshara indivisibility and penalizes merges crossing morphological boundaries. We also release a new morphological segmentation dataset for Hindi, Marathi, and Gujarati. SuTRA reduces shattering, achieving peak gains of +14.7% in morphological alignment (Boundary F1) and +34% in semantic recoverability (Hindi) over BPE. These structural gains yield an average improvement of +8.08 chrF2 in machine translation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.18087 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.895Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).