Metraipolis › archive › dc_a7164f3cb58d
First Passage Attention — arXiv:2610.22714
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.22714
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:43:33.370Z
- SHA-256 of the archived text
2dc764b327c481617708944b6639244d16af8770c291a921e6aef192a4ad6438- Stable URL of this page
https://metraipolis.com/documento/dc_a7164f3cb58d
How to cite
First Passage Attention — arXiv:2610.22714 — arXiv (en), https://arxiv.org/abs/2610.22714 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a7164f3cb58d (Metraipolis archive dc_a7164f3cb58d, sha256 2dc764b327c481617708944b6639244d16af8770c291a921e6aef192a4ad6438)
BibTeX
@misc{dca7164f3cb58d,
title = {First Passage Attention — arXiv:2610.22714},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.22714},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a7164f3cb58d; sha256 2dc764b327c481617708944b6639244d16af8770c291a921e6aef192a4ad6438},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22714. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Augmenting softmax attention with forget gates typically relies on summing decay contributions along the key--query path. Because such accumulation is monotonic, these mechanisms cannot selectively erase intermediate context while preserving older, structurally relevant tokens. We propose First Passage Attention (FPA), which shifts from path summation to path extrema by lowering a key's logit proportionally to the deepest dip of a learned scalar field below that key's level over the intervening span. FPA encompasses the Forgetting Transformer (FoX) as a special case, and when the field and levels follow a shared learned walk, the alive keys with unpenalised logits naturally form a stack. We show that a constant-dimension FPA head realises any subset of keys left alive by subsequent dips, whereas softmax heads with positional biases or token-level forget gates require a head dimension linear in the number of keys. In synthetic experiments, one-layer FPA heads converge to these target sets while comparable softmax and FoX baselines fail on every seed. Scaling to a 110M-parameter language model, FPA matches FoX in perplexity and uniquely achieves perfect passkey retrieval at $8\times$ the training length, even with its key--value cache compressed to about $25\%$. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.22714 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:39:24.826Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).