Metraipolis › archive › dc_ff189144949e
LoGo: Token-Level Dynamic Local-Global Attention — arXiv:2608.29539
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29539
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T08:07:01.872Z
- SHA-256 of the archived text
94be43bbb7c860d53f34743314d830d163c4ceef107365443ff62eb6b0f9724e- Stable URL of this page
https://metraipolis.com/documento/dc_ff189144949e
How to cite
LoGo: Token-Level Dynamic Local-Global Attention — arXiv:2608.29539 — arXiv (en), https://arxiv.org/abs/2608.29539 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff189144949e (Metraipolis archive dc_ff189144949e, sha256 94be43bbb7c860d53f34743314d830d163c4ceef107365443ff62eb6b0f9724e)
BibTeX
@misc{dcff189144949e,
title = {LoGo: Token-Level Dynamic Local-Global Attention — arXiv:2608.29539},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29539},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff189144949e; sha256 94be43bbb7c860d53f34743314d830d163c4ceef107365443ff62eb6b0f9724e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29539. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
As context lengths scale, attention increasingly becomes a primary computational bottleneck in large language models. Standard Transformers remain powerful but computationally inefficient, as they allocate the same attention budget to every token regardless of its contextual demand. Existing local-global hybrids provide a more efficient alternative by mixing restricted- and full-context attention, but they typically allocate span statically across layers or heads. To address these limitations, we propose LoGo, a token-level dynamic local-global attention mechanism that uses attention span as a direct proxy for attention budget allocation. Each LoGo layer contains coupled local and global branches: all tokens receive efficient local attention over a restricted context window, while a learned gate activates global attention with full-context access only for tokens requiring long-range information. A threshold-based budget controller maintains a target global ratio without auxiliary losses, and a progressive masking schedule stabilizes training before sparse routing takes effect. We further implement query-sparse Triton kernels that convert reduced global-attention computation into practical speedups. Extensive experiments validate LoGo's effectiveness, showing that it preserves the scaling behavior of full-attention Transformers across model sizes. In controlled comparisons, LoGo improves over the full-attention Transformer and matched-budget static local-global hybrids, with clear gains on long-range retrieval. Analysis further shows that LoGo learns interpretable span allocation patterns. These results suggest that learned token-level span allocation is an effective and scalable way to improve the long-context performance-compute trade-off. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.29539 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:47:52.707Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).