Metraipolis › archive › dc_32522aba9014

What Attention Recalls and Recurrence Controls in Hybrid Language Models — arXiv:2609.04434

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.04434
License (as archived)
CC0-1.0
Archived
2026-10-16T22:31:25.855Z
SHA-256 of the archived text
89efee4f2635349e4c4d31c1092565881306ed6ee7f0024b917bcffcafee704d
Stable URL of this page
https://metraipolis.com/documento/dc_32522aba9014

How to cite

What Attention Recalls and Recurrence Controls in Hybrid Language Models — arXiv:2609.04434 — arXiv (en), https://arxiv.org/abs/2609.04434 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_32522aba9014 (Metraipolis archive dc_32522aba9014, sha256 89efee4f2635349e4c4d31c1092565881306ed6ee7f0024b917bcffcafee704d)
BibTeX
@misc{dc32522aba9014,
  title = {What Attention Recalls and Recurrence Controls in Hybrid Language Models — arXiv:2609.04434},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.04434},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_32522aba9014; sha256 89efee4f2635349e4c4d31c1092565881306ed6ee7f0024b917bcffcafee704d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04434. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.04434 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T22:21:16.263Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).