Metraipolis › archive › dc_8bbcb18ae545

The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction — arXiv:2610.22071

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.22071
License (as archived): CC0-1.0
Archived: 2026-10-16T10:01:59.408Z
SHA-256 of the archived text: cddbcfd970d3c08b7315f6be47fd8bc873289243c9fa1af74370865403fd231f
Stable URL of this page: https://metraipolis.com/documento/dc_8bbcb18ae545

How to cite

The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction — arXiv:2610.22071 — arXiv (en), https://arxiv.org/abs/2610.22071 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8bbcb18ae545 (Metraipolis archive dc_8bbcb18ae545, sha256 cddbcfd970d3c08b7315f6be47fd8bc873289243c9fa1af74370865403fd231f)

BibTeX

@misc{dc8bbcb18ae545,
  title = {The Right Byte First: How UTF-16 Helps Multilingual Boundary Prediction — arXiv:2610.22071},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22071},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8bbcb18ae545; sha256 cddbcfd970d3c08b7315f6be47fd8bc873289243c9fa1af74370865403fd231f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22071. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Many hierarchical byte-level language models group bytes into patches using a learned boundary predictor whose view of the next codepoint depends on encoding and lookahead. In UTF-8, codepoints of many scripts take three bytes: 8 of 24 bits mark length and continuation, and the first byte carries only 4 high codepoint bits. After the same left context, a mark extending a grapheme cluster and a letter starting the next can thus share their first byte. Universal Byte-Level Encoding (UBE) keeps one- and two-byte UTF-8 encodings but writes these codepoints as two UTF-16 bytes, low byte first, so its first symbol carries 8 low codepoint bits that can distinguish such continuations. We train small causal predictors on matched FineWeb-2 text and evaluate on FLORES-200 under UTF-8, UBE, and UBE's same-length big-endian variant. At one-symbol lookahead, balanced grapheme-boundary error at codepoint boundaries across 17 languages in 13 Brahmic scripts is 8.76% with UTF-8, 8.62% with the big-endian variant, and 0.06% with UBE. When the target instead retains a parent tokenizer's subword cuts at grapheme boundaries, UBE lowers the mean error relative to UTF-8 at the same lookahead by 72.91% in Korean, 35.29% in Chinese, and 67.32-99.55% in eight more scripts. In the seven of them whose main Unicode block fits on one 256-codepoint page, the big-endian variant stays within 3.0% of UTF-8, so the gain follows byte order, not length. In Korean and Chinese, UBE's mean error falls below UTF-8's final mean error after only 25% and 37.5% of training. Which byte comes first thus affects how much lookahead a byte-level boundary predictor needs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.22071 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:47.687Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).