Metraipolis › archive › dc_dbebed4cf51e

Toppling the Hierarchy in Byte-level Language Modeling — arXiv:2609.00463

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.00463
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:17.315Z
SHA-256 of the archived text
5d719daf9ad373d31b2ef4fd9fd6bd7629cf3c5e052c317379a7b634767bad04
Stable URL of this page
https://metraipolis.com/documento/dc_dbebed4cf51e

How to cite

Toppling the Hierarchy in Byte-level Language Modeling — arXiv:2609.00463 — arXiv (en), https://arxiv.org/abs/2609.00463 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dbebed4cf51e (Metraipolis archive dc_dbebed4cf51e, sha256 5d719daf9ad373d31b2ef4fd9fd6bd7629cf3c5e052c317379a7b634767bad04)
BibTeX
@misc{dcdbebed4cf51e,
  title = {Toppling the Hierarchy in Byte-level Language Modeling — arXiv:2609.00463},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.00463},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dbebed4cf51e; sha256 5d719daf9ad373d31b2ef4fd9fd6bd7629cf3c5e052c317379a7b634767bad04},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00463. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This work examines recent byte-level models and their failure to perfectly manipulate characters. State-of-the-art byte-level models use a hierarchical structure, starting at the byte level, downsampling to the word level, and then upsampling back to bytes. While this improves training and inference efficiency, we find that the hierarchical design itself limits character-level understanding, with pure byte-level models consistently outperforming hierarchical variants on character manipulation tasks. Ablating transformer layers into attention and feed-forward components further reveals that byte-level attention is the primary mechanism driving this behavior. Together, our results provide an explanation for the character-level failures of hierarchical byte models and establish a clear trade-off between computational efficiency and fine-grained character understanding.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.00463 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.806Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).