Metraipolis › archive › dc_9dfedd73d66a

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog — arXiv:2610.20859

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20859
License (as archived)
CC0-1.0
Archived
2026-10-16T09:55:48.156Z
SHA-256 of the archived text
1fb4fc252eced486395fd625d92b3ec5b3b783ef1ec0ea54704ffef531bd68ce
Stable URL of this page
https://metraipolis.com/documento/dc_9dfedd73d66a

How to cite

Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog — arXiv:2610.20859 — arXiv (en), https://arxiv.org/abs/2610.20859 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9dfedd73d66a (Metraipolis archive dc_9dfedd73d66a, sha256 1fb4fc252eced486395fd625d92b3ec5b3b783ef1ec0ea54704ffef531bd68ce)
BibTeX
@misc{dc9dfedd73d66a,
  title = {Does Morphological Correctness Matter for Tokenization? A Scrambled Control Study in Tagalog — arXiv:2610.20859},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20859},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9dfedd73d66a; sha256 1fb4fc252eced486395fd625d92b3ec5b3b783ef1ec0ea54704ffef531bd68ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20859. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Typology-aware tokenizers are motivated by the claim that segmentation which respects morpheme boundaries preserves information that frequency-driven segmentation destroys. Studies routinely establish that it helps while assuming the boundaries must be linguistically correct. We separate them. Tagalog marks voice with infixes (s-um-ulat 'wrote', s-in-ulat 'was written'), so the root is not contiguous and no sequence of BPE merges can recover it. We build MorphAustro-Tok, a tokenizer that emits the root as a contiguous unit followed by a positional infix token, and compare it against a corpus-matched BPE baseline and, at two of three fertility levels, against a scrambled-morpheme control that preserves morph count, invertibility and per-type consistency while randomising the pseudo-infix. Across 33 runs, MorphAustro-Tok improves Tagalog machine translation over matched BPE by 0.84, 1.49 and 3.40 mean sentence-level chrF++ points. Token count does not explain the gain, since it grows as segmentation coarsens, and neither does capacity: at the coarsest level the scrambled arm carries more parameters and scores lower. Scrambling the analysis moves the gain by less than between-seed dispersion at both controlled levels, so we detect no contribution from morphological correctness at this scale.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.20859 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:47.613Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).