Metraipolis › archive › dc_c77d022384ea
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models — arXiv:2609.16540
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.16540
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T18:35:00.030Z
- SHA-256 of the archived text
918c80773df3dfd558f25cc0281db3d82f94e49fcc5eeb1e210e94b215131a27- Stable URL of this page
https://metraipolis.com/documento/dc_c77d022384ea
How to cite
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models — arXiv:2609.16540 — arXiv (en), https://arxiv.org/abs/2609.16540 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c77d022384ea (Metraipolis archive dc_c77d022384ea, sha256 918c80773df3dfd558f25cc0281db3d82f94e49fcc5eeb1e210e94b215131a27)
BibTeX
@misc{dcc77d022384ea,
title = {On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models — arXiv:2609.16540},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.16540},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c77d022384ea; sha256 918c80773df3dfd558f25cc0281db3d82f94e49fcc5eeb1e210e94b215131a27},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.16540. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent networks. Specifically, we show through theory and experiments that this gating mechanism causes SSMs to first learn an in-weights "memorization" solution, while delaying, or even preventing, convergence to a correct in-context learning solution. Importantly, this happens even in cases where there are no fundamental limitations due to the architecture or its memory capacity. On the other hand, we find that gating is often beneficial for improving generalization to long sequence lengths. Our results illuminate the crucial role of the gating mechanism in shaping both the training dynamics and generalization of SSMs, and provide a basis for understanding and improving linear-time models. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.16540 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T18:03:09.659Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).