Metraipolis › archive › dc_d4fd71a76339
Latent Core Tokenizer: Compress, but Meaningfully — arXiv:2610.12376
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.12376
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:13:30.242Z
- SHA-256 of the archived text
b0190f27ece8c94be7be5f18de8c01b19a87f505f32d84a3c88824bc14e96257- Stable URL of this page
https://metraipolis.com/documento/dc_d4fd71a76339
How to cite
Latent Core Tokenizer: Compress, but Meaningfully — arXiv:2610.12376 — arXiv (en), https://arxiv.org/abs/2610.12376 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d4fd71a76339 (Metraipolis archive dc_d4fd71a76339, sha256 b0190f27ece8c94be7be5f18de8c01b19a87f505f32d84a3c88824bc14e96257)
BibTeX
@misc{dcd4fd71a76339,
title = {Latent Core Tokenizer: Compress, but Meaningfully — arXiv:2610.12376},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.12376},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d4fd71a76339; sha256 b0190f27ece8c94be7be5f18de8c01b19a87f505f32d84a3c88824bc14e96257},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.12376. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary. Across 104 languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost. Across four multilingual downstream benchmarks, LCT improves aggregate score by 1.48, 1.83, and 2.00 points over BPE, Unigram, and parity-aware BPE, respectively. Our findings show that compression alone does not predict representation quality and highlight the importance of morphology-driven structural discovery and how frequency is used to allocate the final vocabulary across languages. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.12376 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:15.405Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).