Metraipolis › archive › dc_fd99db9e8577
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality — arXiv:2609.28448
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.28448
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T10:58:25.036Z
- SHA-256 of the archived text
d31559ca28bc34433de53db146bcaabb5bc9ec8c1cab677a0227ec274ec50adb- Stable URL of this page
https://metraipolis.com/documento/dc_fd99db9e8577
How to cite
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality — arXiv:2609.28448 — arXiv (en), https://arxiv.org/abs/2609.28448 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd99db9e8577 (Metraipolis archive dc_fd99db9e8577, sha256 d31559ca28bc34433de53db146bcaabb5bc9ec8c1cab677a0227ec274ec50adb)
BibTeX
@misc{dcfd99db9e8577,
title = {Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality — arXiv:2609.28448},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.28448},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd99db9e8577; sha256 d31559ca28bc34433de53db146bcaabb5bc9ec8c1cab677a0227ec274ec50adb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.28448. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $d=2$, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength $\gamma$ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as $N\to\infty$ at finite fixed softmax sharpness $\beta$. Attention condensation instead emerges in the scaling regime $\beta\sim N^2$. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For $d=N\to\infty$, simulations from Gaussian initial conditions provide evidence for a condensation transition at $\beta=O(1)$, driven by dynamically generated finite overlap gaps. Depending on $\gamma$, the resulting phases include diffuse simplex-like states, consensus flips, condensed active routing with signatures of chaos, and fragmented cluster flips. These results establish temporal activity, attention condensation, and geometric clustering as distinct collective phenomena, and show that sparse attention can sustain persistent dynamics rather than freeze it. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG). Categorie arXiv: Disordered Systems and Neural Networks (cond-mat.dis-nn) ; Statistical Mechanics (cond-mat.stat-mech); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.28448 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:06:16.851Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).