Metraipolis › archive › dc_34fc73d1f68f
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position — arXiv:2610.10114
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.10114
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:26:18.728Z
- SHA-256 of the archived text
f6cb6547fdc73c710a9f81b14aa636ccb5e25dbd4bd5856c1bfe05661183ec1a- Stable URL of this page
https://metraipolis.com/documento/dc_34fc73d1f68f
How to cite
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position — arXiv:2610.10114 — arXiv (en), https://arxiv.org/abs/2610.10114 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_34fc73d1f68f (Metraipolis archive dc_34fc73d1f68f, sha256 f6cb6547fdc73c710a9f81b14aa636ccb5e25dbd4bd5856c1bfe05661183ec1a)
BibTeX
@misc{dc34fc73d1f68f,
title = {Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position — arXiv:2610.10114},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.10114},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_34fc73d1f68f; sha256 f6cb6547fdc73c710a9f81b14aa636ccb5e25dbd4bd5856c1bfe05661183ec1a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10114. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performance in length extrapolation and context extension. To explain why hybrid models work and how to design them better, we propose Mechanics of Long-Context Hybrid Models. As Part 1.1 of this series, we begin with hybrids of full attention and either sliding-window attention (SWA) or gated variants of linear attention (LA), represented by GLA and GDN. We first observe a Seesaw Effect in Context Extension: LA hybrids benefit more from long-context continual pretraining, whereas SWA hybrids perform better under length extrapolation. We attribute this behavior to differences in the positional inductive biases induced by these attention mechanisms. We find that SWA hybrids suffer from a Short-Context Learning Trap, Short-Window Weariness, and Long-Window Laziness, and require extended windows to enhance performance in continual long-context pretraining. For LA hybrids, we summarize the Matthew Effect of Hybrid Position Extrapolation and propose Sliding-Window Linear Attention, achieving 16$\times$ training-free length extrapolation while maintaining 100\% accuracy on NIAH-SK1 in 64k context length. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.10114 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:24:06.344Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).