Metraipolis › archive › dc_3ae240ff002f

Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models — arXiv:2611.01909

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01909
License (as archived)
CC0-1.0
Archived
2026-11-03T10:42:27.988Z
SHA-256 of the archived text
5758c9cfdb4893a5c2529813168de2b134e58b12326d33629facfe5719b2e138
Stable URL of this page
https://metraipolis.com/documento/dc_3ae240ff002f

How to cite

Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models — arXiv:2611.01909 — arXiv (en), https://arxiv.org/abs/2611.01909 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3ae240ff002f (Metraipolis archive dc_3ae240ff002f, sha256 5758c9cfdb4893a5c2529813168de2b134e58b12326d33629facfe5719b2e138)
BibTeX
@misc{dc3ae240ff002f,
  title = {Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models — arXiv:2611.01909},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01909},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3ae240ff002f; sha256 5758c9cfdb4893a5c2529813168de2b134e58b12326d33629facfe5719b2e138},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01909. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hybrid language models interleave linear-attention layers with a small number of softmax-attention layers. Prior work largely attributes in-context retrieval to softmax attention, treating linear attention primarily as a mechanism for compressing past context. We revisit this division of labor and show that linear attention plays a direct role in retrieval by constructing the query used by softmax attention. Studying Qwen3.5 (Gated DeltaNet) and Granite4.0 (Mamba-2), we identify an Encode--Address circuit. An early Encode block preserves token identity, while an Address block constructs the final-position representation from which a softmax \emph{reader} forms its attention query. Transplanting this representation between prompts with different queried keys redirects retrieval in 0.97--0.99 of cases. The Address block's recurrent state carries the retrieval target, and its computation transfers across architectures: replacing Granite's Address block with Qwen's restores retrieval accuracy to 0.92--0.94 without updating either model's weights. Across six Qwen3.5 models, Address computation becomes distributed at larger scales over a chain of blocks, each guiding a successive reader, as the query no longer passes from one reader to the next through attention. These findings reveal a division of labor in hybrid retrieval: linear attention constructs the query, while softmax attention performs the lookup. Code is available at this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01909 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:39:50.246Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).