Metraipolis › archive › dc_3322fdaf1850
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval — arXiv:2611.04169
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04169
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T21:39:51.040Z
- SHA-256 of the archived text
50e1db753798ccaf28bb286d26e2585716fe3e703e980d606a8be4ee3920610a- Stable URL of this page
https://metraipolis.com/documento/dc_3322fdaf1850
How to cite
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval — arXiv:2611.04169 — arXiv (en), https://arxiv.org/abs/2611.04169 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3322fdaf1850 (Metraipolis archive dc_3322fdaf1850, sha256 50e1db753798ccaf28bb286d26e2585716fe3e703e980d606a8be4ee3920610a)
BibTeX
@misc{dc3322fdaf1850,
title = {Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval — arXiv:2611.04169},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04169},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3322fdaf1850; sha256 50e1db753798ccaf28bb286d26e2585716fe3e703e980d606a8be4ee3920610a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04169. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.04169 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T21:35:49.645Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).