Metraipolis › archive › dc_de0690cb3e3e

When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models — arXiv:2608.26319

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26319
License (as archived)
CC0-1.0
Archived
2026-10-17T15:18:44.556Z
SHA-256 of the archived text
8a699ae7fb4a8ef8936ea179bc3c97c78f74d4ebda584db2b231c866c3e9e10a
Stable URL of this page
https://metraipolis.com/documento/dc_de0690cb3e3e

How to cite

When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models — arXiv:2608.26319 — arXiv (en), https://arxiv.org/abs/2608.26319 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de0690cb3e3e (Metraipolis archive dc_de0690cb3e3e, sha256 8a699ae7fb4a8ef8936ea179bc3c97c78f74d4ebda584db2b231c866c3e9e10a)
BibTeX
@misc{dcde0690cb3e3e,
  title = {When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models — arXiv:2608.26319},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26319},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de0690cb3e3e; sha256 8a699ae7fb4a8ef8936ea179bc3c97c78f74d4ebda584db2b231c866c3e9e10a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26319. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.26319 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:15:18.044Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).