Metraipolis › archive › dc_5d9370dcb86e
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings — arXiv:2608.03994
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03994
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:31:19.029Z
- SHA-256 of the archived text
e2b909108fa7cc4cfc92bc7303bb8853a9566eab3f2b64ee17ce88d3ede31225- Stable URL of this page
https://metraipolis.com/documento/dc_5d9370dcb86e
How to cite
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings — arXiv:2608.03994 — arXiv (en), https://arxiv.org/abs/2608.03994 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5d9370dcb86e (Metraipolis archive dc_5d9370dcb86e, sha256 e2b909108fa7cc4cfc92bc7303bb8853a9566eab3f2b64ee17ce88d3ede31225)
BibTeX
@misc{dc5d9370dcb86e,
title = {When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings — arXiv:2608.03994},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03994},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5d9370dcb86e; sha256 e2b909108fa7cc4cfc92bc7303bb8853a9566eab3f2b64ee17ce88d3ede31225},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03994. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We identify a previously overlooked failure mode of ALiBi positional encoding: its linear bias scaling underflows floating-point precision, which zeroes out a large fraction of attention weights and renders the affected attention heads partially blind. We analyze this failure mode, characterize its impact, and examine four mitigation strategies. We further demonstrate its occurrence in state-of-the-art pretrained models based on ALiBi. Comprehensive pretraining experiments with 148M-parameter decoder models help us to disentangle its effects from out-of-context degradation. We find that ALiBi's failure mode can substantially impair token retrieval while having only a minor effect on standard decoder benchmarks. We propose four training-time mitigation strategies and evaluate them individually and in combinations, finding that log-scaled distances yield the most consistent improvements in passkey retrieval. Despite this problem, default ALiBi slopes remain a surprisingly strong baseline, particularly for needle-in-a-haystack retrieval. Based on these findings we provide concrete recommendations on how to train models with ALiBi. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.03994 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:17:37.789Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).