Metraipolis › archive › dc_26d2c7bf0fb7

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing — arXiv:2610.05842

Source
arXiv
Original
https://arxiv.org/abs/2610.05842
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:26.197Z
SHA-256 of the archived text
2f4e706c5a9d58ab64b18fbeec32fe0c4daac72905995bb0c7cc18c011fbba15
Stable URL of this page
https://metraipolis.com/documento/dc_26d2c7bf0fb7

How to cite

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing — arXiv:2610.05842 — arXiv, https://arxiv.org/abs/2610.05842 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_26d2c7bf0fb7 (Metraipolis archive dc_26d2c7bf0fb7, sha256 2f4e706c5a9d58ab64b18fbeec32fe0c4daac72905995bb0c7cc18c011fbba15)
BibTeX
@misc{dc26d2c7bf0fb7,
  title = {HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing — arXiv:2610.05842},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05842},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_26d2c7bf0fb7; sha256 2f4e706c5a9d58ab64b18fbeec32fe0c4daac72905995bb0c7cc18c011fbba15},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05842. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fixed with respect to input content and therefore cannot adapt historical access to each query. We introduce \emph{Hybrid Linear Attention} (HLA), a query-dependent chunk-level attention mechanism for Gated DeltaNet (GDN). HLA represents each completed chunk as an exact affine state transition and computes content-dependent routing gates from compact, self-attentively pooled representatives. Each gate interpolates the corresponding historical transition with the identity map, controlling both the chunk's additive memory and its transformation of earlier states. Effective-support regularization further encourages concentrated routing for sparse inference. We evaluate HLA under both pretrained adaptation and from-scratch training. Across Qwen3.5 models from 0.8B to 9B, HLA consistently improves over native GDN and fixed chunk mixing, with gains of up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER. In a controlled from-scratch 1.3B setting trained for 100B tokens with a 4K context, HLA also improves RULER performance from 4K to 32K, with gains increasing from 0.83 points at 4K to 4.22 points at 32K. These results demonstrate that query-dependent composition of recurrent memory improves long-context modeling and remains effective beyond the training context while using compact per-chunk affine summaries. Project page: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.05842 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.348Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).