Metraipolis › archive › dc_fa2abf709446

Leaner Transformers Can Easily Learn to Cluster — arXiv:2610.09760

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09760
License (as archived)
CC0-1.0
Archived
2026-10-08T11:59:10.686Z
SHA-256 of the archived text
cb1ef409a37430b6e8ccfcc38f880527b08c8aae5a551163d388d413e3e096ea
Stable URL of this page
https://metraipolis.com/documento/dc_fa2abf709446

How to cite

Leaner Transformers Can Easily Learn to Cluster — arXiv:2610.09760 — arXiv (en), https://arxiv.org/abs/2610.09760 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa2abf709446 (Metraipolis archive dc_fa2abf709446, sha256 cb1ef409a37430b6e8ccfcc38f880527b08c8aae5a551163d388d413e3e096ea)
BibTeX
@misc{dcfa2abf709446,
  title = {Leaner Transformers Can Easily Learn to Cluster — arXiv:2610.09760},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09760},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa2abf709446; sha256 cb1ef409a37430b6e8ccfcc38f880527b08c8aae5a551163d388d413e3e096ea},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09760. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers have in-context learning capabilities, where some known learning algorithms can be executed in the forward pass through the model. Recent work shows that transformers can exactly perform Lloyd's algorithm for $k$-means clustering with $n$ points in $d$ dimensions with an embedding size $d_{\textsf{emb}} = d+k$ (thus, requiring attention projection matrices of size $(d+k)^2$). In this work, we build upon this result in the following ways: First, we present an equally expressive but smaller transformer that executes Lloyd's algorithm with embedding size $d_{\textsf{emb}} = (d + \lceil \log_2 k \rceil)$. Next, we train these transformers to learn the clustering algorithms given a distribution of clustering tasks, and theoretically characterize and empirically validate the factors affecting the convergence and in-distribution generalization of learning algorithms based on stochastic gradients. Finally, we probe the general clustering abilities of these learned algorithms (in the form of transformers), and try to understand situations where they succeed and fail.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.09760 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:50:37.713Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).