Metraipolis › archive › dc_2c8a8cd38b27

The Query Knows What to Forget: A Second Erase Direction for Linear Attention — arXiv:2608.13668

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.13668
License (as archived): CC0-1.0
Archived: 2026-10-19T02:30:24.439Z
SHA-256 of the archived text: 0467f79bc325e436275834de9e1309801e40e2b6cbeca1e124be4f25ed1d8b5a
Stable URL of this page: https://metraipolis.com/documento/dc_2c8a8cd38b27

How to cite

The Query Knows What to Forget: A Second Erase Direction for Linear Attention — arXiv:2608.13668 — arXiv (en), https://arxiv.org/abs/2608.13668 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2c8a8cd38b27 (Metraipolis archive dc_2c8a8cd38b27, sha256 0467f79bc325e436275834de9e1309801e40e2b6cbeca1e124be4f25ed1d8b5a)

BibTeX

@misc{dc2c8a8cd38b27,
  title = {The Query Knows What to Forget: A Second Erase Direction for Linear Attention — arXiv:2608.13668},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13668},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2c8a8cd38b27; sha256 0467f79bc325e436275834de9e1309801e40e2b6cbeca1e124be4f25ed1d8b5a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13668. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.13668 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:26:17.770Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).