Metraipolis › archive › dc_451c447e6235
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context? — arXiv:2609.08574
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08574
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:49:01.989Z
- SHA-256 of the archived text
ffd49f3f424227f3ad755f3cd692c385cc36092527144819197d79d4b537fe76- Stable URL of this page
https://metraipolis.com/documento/dc_451c447e6235
How to cite
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context? — arXiv:2609.08574 — arXiv (en), https://arxiv.org/abs/2609.08574 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_451c447e6235 (Metraipolis archive dc_451c447e6235, sha256 ffd49f3f424227f3ad755f3cd692c385cc36092527144819197d79d4b537fe76)
BibTeX
@misc{dc451c447e6235,
title = {Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context? — arXiv:2609.08574},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08574},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_451c447e6235; sha256 ffd49f3f424227f3ad755f3cd692c385cc36092527144819197d79d4b537fe76},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08574. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model finds it. Gated attention cut first token attention from 46.7 percent to 4.8 percent at NeurIPS 2025, and Kimi K3 pairs that idea with Kimi Delta Attention and Attention Residuals behind a one million token window, eight times past the range where these diagnostics have been reported. This paper asks whether the fix survives that jump. We build SinkProbe, a suite that measures sink mass, massive activation, position resolved recall and the recency gap, and apply it to four small models that differ only in how they mix tokens and depth. Three results follow. The training objective produces the sink, not the architecture. Gating did not reproduce its published effect at our scale. Sink mass, activations and position bias moved independently. Code, data and the measurement protocol are released at this https URL Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.08574 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:25.188Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).