Metraipolis › archive › dc_fa2abf709446
Leaner Transformers Can Easily Learn to Cluster — arXiv:2610.09760
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09760
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:59:10.686Z
- SHA-256 of the archived text
cb1ef409a37430b6e8ccfcc38f880527b08c8aae5a551163d388d413e3e096ea- Stable URL of this page
https://metraipolis.com/documento/dc_fa2abf709446
How to cite
Leaner Transformers Can Easily Learn to Cluster — arXiv:2610.09760 — arXiv (en), https://arxiv.org/abs/2610.09760 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa2abf709446 (Metraipolis archive dc_fa2abf709446, sha256 cb1ef409a37430b6e8ccfcc38f880527b08c8aae5a551163d388d413e3e096ea)
BibTeX
@misc{dcfa2abf709446,
title = {Leaner Transformers Can Easily Learn to Cluster — arXiv:2610.09760},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09760},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa2abf709446; sha256 cb1ef409a37430b6e8ccfcc38f880527b08c8aae5a551163d388d413e3e096ea},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09760. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformers have in-context learning capabilities, where some known learning algorithms can be executed in the forward pass through the model. Recent work shows that transformers can exactly perform Lloyd's algorithm for $k$-means clustering with $n$ points in $d$ dimensions with an embedding size $d_{\textsf{emb}} = d+k$ (thus, requiring attention projection matrices of size $(d+k)^2$). In this work, we build upon this result in the following ways: First, we present an equally expressive but smaller transformer that executes Lloyd's algorithm with embedding size $d_{\textsf{emb}} = (d + \lceil \log_2 k \rceil)$. Next, we train these transformers to learn the clustering algorithms given a distribution of clustering tasks, and theoretically characterize and empirically validate the factors affecting the convergence and in-distribution generalization of learning algorithms based on stochastic gradients. Finally, we probe the general clustering abilities of these learned algorithms (in the form of transformers), and try to understand situations where they succeed and fail.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.09760 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:50:37.713Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).