Metraipolis › archive › dc_0820bb30e4df
Multilinguality in Hybrid Attention LLMs — arXiv:2609.35378
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.35378
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:03:32.454Z
- SHA-256 of the archived text
57bf23d174591124d18c4f90e62651ad3573efa770965d9d9ed5eee096da1dde- Stable URL of this page
https://metraipolis.com/documento/dc_0820bb30e4df
How to cite
Multilinguality in Hybrid Attention LLMs — arXiv:2609.35378 — arXiv (en), https://arxiv.org/abs/2609.35378 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0820bb30e4df (Metraipolis archive dc_0820bb30e4df, sha256 57bf23d174591124d18c4f90e62651ad3573efa770965d9d9ed5eee096da1dde)
BibTeX
@misc{dc0820bb30e4df,
title = {Multilinguality in Hybrid Attention LLMs — arXiv:2609.35378},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.35378},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0820bb30e4df; sha256 57bf23d174591124d18c4f90e62651ad3573efa770965d9d9ed5eee096da1dde},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35378. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In response to the growing demand for long sequences in agentic and reasoning use cases, many state-of-the-art LLMs combine multiple variants of attention to mitigate the quadratic complexity of traditional softmax attention. These hybrid attention LLMs aim to balance the strengths and limitations of full attention and alternatives based on recurrence. This work presents a first study of how hybrid attention impacts the multilinguality of LLMs. Beyond the impact on long sequences in poorly tokenized languages, our study is motivated by the possibility that the inductive biases of the recurrent state alter linguistic processing. Our interpretability analysis confirms this, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-attention layers. Across diverse models, we notably observe a pronounced spike in cross-lingual alignment around the first full-attention layer. These findings lead us to question the conventional ordering of attention layers. In distillation experiments on multilingual data, all alternative layer orderings outperform the standard throughout training, learning up to 2.5X faster. These stark, replicable results prompt our theory that multilingual models would benefit from starting with a full-attention layer rather than recurrent layers. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.35378 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:03.340Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).