Metraipolis › archive › dc_2a22f808d570
CoWindow Attention: Full Causal Coverage Is a Collective Property — arXiv:2609.32704
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.32704
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:55:58.659Z
- SHA-256 of the archived text
39dfd2fdc881f1bf04f3f854d04bf53cc6973e02264f03fa3a34292cf48655d9- Stable URL of this page
https://metraipolis.com/documento/dc_2a22f808d570
How to cite
CoWindow Attention: Full Causal Coverage Is a Collective Property — arXiv:2609.32704 — arXiv (en), https://arxiv.org/abs/2609.32704 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2a22f808d570 (Metraipolis archive dc_2a22f808d570, sha256 39dfd2fdc881f1bf04f3f854d04bf53cc6973e02264f03fa3a34292cf48655d9)
BibTeX
@misc{dc2a22f808d570,
title = {CoWindow Attention: Full Causal Coverage Is a Collective Property — arXiv:2609.32704},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.32704},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2a22f808d570; sha256 39dfd2fdc881f1bf04f3f854d04bf53cc6973e02264f03fa3a34292cf48655d9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32704. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
FullAttn repeatedly exposes the complete causal history to every attention head, creating substantial redundant computation and memory traffic even with IO-efficient dense kernels. We introduce CoWA, a structured attention architecture that distributes access to the causal history across KV heads. All heads share near-diagonal and prefix-sink windows, while complementary long-range windows partition the remaining history. Their union provides full causal coverage although each head attends sparsely to distant tokens. This position-defined attention pattern requires no learned router or indexer, is used consistently during training and inference, and aligns with KV-head tensor parallelism. A window-matched ablation at 8K isolates the effect of complementary long-range allocation: CoWA with 100% collective coverage reaches 89.73% accuracy, compared with 89.97% for FullAttn, while duplicated long-range windows perform substantially worse. Across a broader controlled associative-recall comparison with matched token budgets, CoWA closely tracks FullAttn as the context grows, whereas other sparse patterns lose a substantial fraction of the associations. In an attention-operator benchmark at 128K tokens with tensor parallelism, CoWA reduces forward and backward latency during training by 7.4x and 8.6x and decoding latency during inference by 3.0x over FullAttn. Its per-rank peak operator memory matches FullAttn during training and is 7.6x lower during decoding. Across scaling-law training from 0.6B to 14B parameters, CoWA closely tracks FullAttn in perplexity while reducing total training FLOPs. The resulting 14B models and 32B models from separate continued training achieve comparable knowledge, reasoning, and long-context retrieval scores to FullAttn. These results show that full causal coverage can be a collective property of the head ensemble rather than a duplicated property of every head. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.32704 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:00.942Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).