Metraipolis › archive › dc_3490ea79dd1b
Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering — arXiv:2609.24202
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.24202
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T07:07:35.847Z
- SHA-256 of the archived text
89ffb7a3b63a469053d765703ada195367c4bdff88b10c3d854f391bd8d28be6- Stable URL of this page
https://metraipolis.com/documento/dc_3490ea79dd1b
How to cite
Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering — arXiv:2609.24202 — arXiv (en), https://arxiv.org/abs/2609.24202 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3490ea79dd1b (Metraipolis archive dc_3490ea79dd1b, sha256 89ffb7a3b63a469053d765703ada195367c4bdff88b10c3d854f391bd8d28be6)
BibTeX
@misc{dc3490ea79dd1b,
title = {Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering — arXiv:2609.24202},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.24202},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3490ea79dd1b; sha256 89ffb7a3b63a469053d765703ada195367c4bdff88b10c3d854f391bd8d28be6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24202. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sparse attention reduces the quadratic cost of global self-attention while retaining strong empirical performance, but how its restricted interactions shape the evolution of token representations remains theoretically underexplored. Modeling tokens as particles on the unit sphere, we introduce opinion leader dynamics, a framework that identifies two mechanisms through which token groups converge internally while maintaining distinct limiting directions. In the explicit model, fixed representatives induce a potential that attracts tokens toward distinct local maxima. In the implicit model, disconnected interaction groups evolve toward separate consensus directions. We formulate both models as reverse Wasserstein gradient flows and establish exponential convergence under suitable conditions. We further connect these theoretical predictions to token evolution in frontier sparse-attention LLMs that motivate our framework. Across four benchmarks, Kimi-K3, MiniMax-M3, and DeepSeek-V4-Flash consistently exhibit clearer cluster separation and higher clustering scores than the dense-attention model GLM-4.7-Flash in projected token representations. These observations support the relevance of the predicted multiple-group structure to trained frontier LLMs, while finite-particle simulations illustrate the theoretical convergence behavior. Together, our results connect restricted token interactions to distinct group-level attractors, providing a dynamical account of how sparse attention can support alignment within groups while preserving separation between them. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.24202 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T06:58:09.381Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).