Metraipolis › archive › dc_3bacdcf2847e

Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention — arXiv:2609.22005

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22005
License (as archived)
CC0-1.0
Archived
2026-10-10T01:27:54.596Z
SHA-256 of the archived text
2bcd88124366d467b0c4241818444f043c45a34eb8c4f4631ec4d38bab96a6df
Stable URL of this page
https://metraipolis.com/documento/dc_3bacdcf2847e

How to cite

Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention — arXiv:2609.22005 — arXiv (en), https://arxiv.org/abs/2609.22005 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3bacdcf2847e (Metraipolis archive dc_3bacdcf2847e, sha256 2bcd88124366d467b0c4241818444f043c45a34eb8c4f4631ec4d38bab96a6df)
BibTeX
@misc{dc3bacdcf2847e,
  title = {Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention — arXiv:2609.22005},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22005},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3bacdcf2847e; sha256 2bcd88124366d467b0c4241818444f043c45a34eb8c4f4631ec4d38bab96a6df},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22005. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Softmax attention has two structural gaps. A head cannot abstain, because its weights sum to one, so it outputs something even when nothing is relevant. Nor can it filter what it reads, because its output is a weighted average of value vectors, passing interference as faithfully as signal. We call these missing primitives abstention and noise filtering. Recent studies report that gating the value pathway improves pretraining but attribute the gain to different causes. We show that a value gate partly supplies both primitives, which unifies the reported causes as views of one gain. We give each primitive its own mechanism in matched models of 10M to 350M parameters and measure what each contributes. The gain from gating is almost entirely abstention at 10M, whereas by 350M filtering contributes as much as abstention, so what a study observes depends on its scale. The two benefits are largely additive, with a small overlap. A gate determined by each value alone leaves the attention sink in place, whereas a query-controlled mechanism removes it. Injecting interference into the value reads shows that abstention and filtering protect against it in distinguishable ways. The same patterns appear in pretrained models up to 20B parameters.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.22005 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:00:30.330Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).