Metraipolis › archive › dc_aec2b053e7ba

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation — arXiv:2610.05978

Source
arXiv
Original
https://arxiv.org/abs/2610.05978
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:27.520Z
SHA-256 of the archived text
4bc9450ca8ded7acbd63e3f7cf0e0556c1c9ddbfa822b68f2e4a75854b0a1b7f
Stable URL of this page
https://metraipolis.com/documento/dc_aec2b053e7ba

How to cite

Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation — arXiv:2610.05978 — arXiv, https://arxiv.org/abs/2610.05978 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aec2b053e7ba (Metraipolis archive dc_aec2b053e7ba, sha256 4bc9450ca8ded7acbd63e3f7cf0e0556c1c9ddbfa822b68f2e4a75854b0a1b7f)
BibTeX
@misc{dcaec2b053e7ba,
  title = {Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation — arXiv:2610.05978},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05978},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aec2b053e7ba; sha256 4bc9450ca8ded7acbd63e3f7cf0e0556c1c9ddbfa822b68f2e4a75854b0a1b7f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05978. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to $25\%$ of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields $3.4\times$ more accepted tokens than in subword Transformers.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.05978 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.353Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).