Metraipolis › archive › dc_32522aba9014
What Attention Recalls and Recurrence Controls in Hybrid Language Models — arXiv:2609.04434
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04434
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:31:25.855Z
- SHA-256 of the archived text
89efee4f2635349e4c4d31c1092565881306ed6ee7f0024b917bcffcafee704d- Stable URL of this page
https://metraipolis.com/documento/dc_32522aba9014
How to cite
What Attention Recalls and Recurrence Controls in Hybrid Language Models — arXiv:2609.04434 — arXiv (en), https://arxiv.org/abs/2609.04434 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_32522aba9014 (Metraipolis archive dc_32522aba9014, sha256 89efee4f2635349e4c4d31c1092565881306ed6ee7f0024b917bcffcafee704d)
BibTeX
@misc{dc32522aba9014,
title = {What Attention Recalls and Recurrence Controls in Hybrid Language Models — arXiv:2609.04434},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04434},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_32522aba9014; sha256 89efee4f2635349e4c4d31c1092565881306ed6ee7f0024b917bcffcafee704d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04434. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Hybrid language models combine attention with a fixed-size recurrent state, but the role of each channel remains unclear. We introduce two cache-level interventions. Split-prefill keeps only the KV cache or only the recurrent state from a prefilled context, then generates an answer. State-swap pairs the KV cache from one context with the recurrent state from another in a single forward pass. On Qwen3.5 and Falcon-H1, the two channels split sharply by function. Exact retrieval survives only through attention (64-98% of full accuracy) and collapses to zero through recurrence. Output language and persona reverse the pattern: both survive recurrence (70-80% and 3-5x) while KV-only drops to ~1% language accuracy. State-swap confirms this causally: the answer takes its value from the KV side and its language from the recurrent side. Recurrent-only generation also accepts words that were never in the context but share meaning or parts with seen items. Attention provides a lookup over what was said; the recurrent state shapes how the model says it next. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.04434 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:21:16.263Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).