Metraipolis › archive › dc_90dea4e8ed3c
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution — arXiv:2609.35664
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.35664
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:03:52.544Z
- SHA-256 of the archived text
164a220dc2e35cad4a19fb5fca27c216cfa0fdc0be1b450bd0fd9ebac9fa241e- Stable URL of this page
https://metraipolis.com/documento/dc_90dea4e8ed3c
How to cite
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution — arXiv:2609.35664 — arXiv (en), https://arxiv.org/abs/2609.35664 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_90dea4e8ed3c (Metraipolis archive dc_90dea4e8ed3c, sha256 164a220dc2e35cad4a19fb5fca27c216cfa0fdc0be1b450bd0fd9ebac9fa241e)
BibTeX
@misc{dc90dea4e8ed3c,
title = {MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution — arXiv:2609.35664},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.35664},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_90dea4e8ed3c; sha256 164a220dc2e35cad4a19fb5fca27c216cfa0fdc0be1b450bd0fd9ebac9fa241e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35664. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Gated Linear Attention (GLA) Transformers advance linear recurrent models through data-dependent gating, but face a core limitation: the fixed-capacity memory matrices across all heads operate at a single temporal resolution, where each token is processed individually, forcing them to simultaneously encode local syntactic patterns and long-range semantic structure, creating a representational bottleneck that gating alone is insufficient to resolve. We introduce Multi-Scale Gated Linear Attention (MS-GLA), which addresses this by distributing attention heads across multiple temporal resolutions. Coarser resolutions pool longer token spans naturally specializing toward long-range dependencies, while finer head groups retain sensitivity to local syntactic structure. A learnable, input-dependent fusion layer dynamically recombines head group outputs at each timestep, expanding effective memory capacity without increasing per-head state size. This multi-resolution decomposition draws on principles from Multi-Scale State-Space Models (MS-SSM), adapting them to the gated linear attention setting. We evaluate MS-GLA on language modeling, recall-intensive tasks, and long-context generalization. Across all settings, MS-GLA consistently achieves higher accuracy and lower perplexity than GLA at matched parameter counts, with up to 18.9% improvement on recall-intensive tasks and 9.5% lower average perplexity on language modeling benchmarks, validating multi-temporal resolution decomposition as a principled and effective extension of Gated Linear Attention. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.35664 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:03.422Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).