Metraipolis › archive › dc_646aeeb27561
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining — arXiv:2609.21362
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.21362
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T01:27:22.665Z
- SHA-256 of the archived text
9d238e41571bea69489784b9ccccde21f5933c237fb5f0701a9861a3a16dc7ea- Stable URL of this page
https://metraipolis.com/documento/dc_646aeeb27561
How to cite
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining — arXiv:2609.21362 — arXiv (en), https://arxiv.org/abs/2609.21362 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_646aeeb27561 (Metraipolis archive dc_646aeeb27561, sha256 9d238e41571bea69489784b9ccccde21f5933c237fb5f0701a9861a3a16dc7ea)
BibTeX
@misc{dc646aeeb27561,
title = {Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining — arXiv:2609.21362},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.21362},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_646aeeb27561; sha256 9d238e41571bea69489784b9ccccde21f5933c237fb5f0701a9861a3a16dc7ea},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.21362. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies. We introduce \textbf{Phonemic Tokenizer}, a linguistically motivated tokenizer for Vietnamese and Chinese that converts each syllable into IPA and factorizes it into three phonological components: onset, rime, and tone. The three components jointly occupy one contextual position, preserving syllable-level sequence length while enabling representation sharing across phonologically related syllables. Non-phonological and unsupported units are handled through character-level fallback. This deterministic design requires no corpus-dependent vocabulary learning and yields vocabularies of only 112 entries for Chinese and 256 for Vietnamese. Intrinsic evaluation shows that the tokenizer achieves substantially higher Rényi efficiency in both languages, represents every entry in a standard Vietnamese syllable dictionary with a Fertility of exactly one, and generally produces shorter Vietnamese sequences than existing pretrained tokenizers. We further instantiate the tokenizer in \textbf{PhonemicBERT}, which combines factorized component embeddings and reconstructs complete masked syllables using three prediction heads. Under a controlled Chinese pretraining setup, PhonemicBERT-Zh is competitive with or outperforms character, subword, and SubChar alternatives across diverse language-understanding tasks. PhonemicBERT-Vi also achieves competitive or superior results to established Vietnamese and multilingual pretrained models. These results establish phonemic factorization as a compact, efficient, and interpretable alternative to atomic and statistically segmented text representations.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.21362 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:00:30.278Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).