Metraipolis › archive › dc_87ea668d6dac

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs — arXiv:2609.35475

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.35475
License (as archived): CC0-1.0
Archived: 2026-10-07T21:03:39.984Z
SHA-256 of the archived text: b84740731967e5e7bb545b87ef1a580c09fc0fafad9b0405d28685a674d567ae
Stable URL of this page: https://metraipolis.com/documento/dc_87ea668d6dac

How to cite

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs — arXiv:2609.35475 — arXiv (en), https://arxiv.org/abs/2609.35475 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_87ea668d6dac (Metraipolis archive dc_87ea668d6dac, sha256 b84740731967e5e7bb545b87ef1a580c09fc0fafad9b0405d28685a674d567ae)

BibTeX

@misc{dc87ea668d6dac,
  title = {Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs — arXiv:2609.35475},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35475},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_87ea668d6dac; sha256 b84740731967e5e7bb545b87ef1a580c09fc0fafad9b0405d28685a674d567ae},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35475. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language models sometimes produce correct outputs even when their inputs are corrupted by deletion, replacement, or misspelling. We study the internal processes accompanying this behavior, which we call context restoration, in controlled attention-only transformers and five pretrained LLMs (1B-32B parameters) across arithmetic, reading comprehension, and multiple-choice reasoning tasks. In the attention-only transformers, restoration emerges spontaneously despite training exclusively on clean sequences, without corruption training or an explicit denoising objective. We find that context restoration follows a two-phase process: early layers localize effects associated with repair at corrupted positions, while later layers accumulate these effects at uncorrupted positions through the residual stream and ultimately concentrate them at the output position. Repair outcome is predictable from hidden states: cosine alignment with the clean state is highly predictive in attention-only models, while linear probes recover additional information in pretrained LLMs. A linear probe using only the corrupted prompt's first-block hidden state predicts failure with mean ROC-AUC 0.78. This enables failure triage under matched or even partially shifted deployment conditions and may reduce unnecessary verification or computation. Failed examples also show substantially greater nonlinearity along corruption directions. Moderate-corruption finetuning increases corruption tolerance while simultaneously reducing displacement-normalized linearization error, associating improved robustness with a more nearly linear response to corruption.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.35475 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:03.371Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).