Metraipolis › archive › dc_58ee770252a6
Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention — arXiv:2608.19203
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.19203
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T11:39:03.591Z
- SHA-256 of the archived text
4b86c31b6311562fb35d081d760d3e8a949d8804bed1d1d5243e4df570fba25d- Stable URL of this page
https://metraipolis.com/documento/dc_58ee770252a6
How to cite
Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention — arXiv:2608.19203 — arXiv (en), https://arxiv.org/abs/2608.19203 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_58ee770252a6 (Metraipolis archive dc_58ee770252a6, sha256 4b86c31b6311562fb35d081d760d3e8a949d8804bed1d1d5243e4df570fba25d)
BibTeX
@misc{dc58ee770252a6,
title = {Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention — arXiv:2608.19203},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.19203},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_58ee770252a6; sha256 4b86c31b6311562fb35d081d760d3e8a949d8804bed1d1d5243e4df570fba25d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19203. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric Attention Heads (AAH), a head-wise context- allocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface. In 4096- token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy. We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.19203 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T11:38:27.880Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).