Metraipolis › archive › dc_115d74611bfb
Generalization properties of the token perceptron: the role of embedding and vocabulary — arXiv:2610.32788
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.32788
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-26T20:14:53.263Z
- SHA-256 of the archived text
e9712a94c4efbbc00dd2ab3d8eed6bfd9d7301c0008fe4fdc6c4590fd0ca73db- Stable URL of this page
https://metraipolis.com/documento/dc_115d74611bfb
How to cite
Generalization properties of the token perceptron: the role of embedding and vocabulary — arXiv:2610.32788 — arXiv (en), https://arxiv.org/abs/2610.32788 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_115d74611bfb (Metraipolis archive dc_115d74611bfb, sha256 e9712a94c4efbbc00dd2ab3d8eed6bfd9d7301c0008fe4fdc6c4590fd0ca73db)
BibTeX
@misc{dc115d74611bfb,
title = {Generalization properties of the token perceptron: the role of embedding and vocabulary — arXiv:2610.32788},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.32788},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_115d74611bfb; sha256 e9712a94c4efbbc00dd2ab3d8eed6bfd9d7301c0008fe4fdc6c4590fd0ca73db},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32788. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models are trained to predict tokens from a finite vocabulary, each represented by a vector in a $d$-dimensional embedding space. Therefore, token prediction is a classification problem whose classes have a geometry determined by the embedding. We study how this geometry affects learning in a minimal solvable model: a linear predictor that maps a sequence of $N$ embedded tokens to a $d$-dimensional output through one $d\times d$ matrix per position, trained in the teacher--student setting. We call this the \textit{token perceptron}. We compute its generalization error with the replica method, taking $N\to\infty$ at fixed number of examples per position $\alpha=P/N$ and keeping $d$ and the vocabulary size $D$ finite. As a first reference case, we consider continuous embeddings on the sphere, which isolate the role of the embedding dimension. The natural load is then $\alpha/d$, and for $d>1$ a normalized label fixes the direction of the teacher output, so the problem becomes a noiseless regression: the teacher is recovered exactly above $\alpha_{\rm rec}=d^2/(d-1)$, and spherical cross-entropy reaches this threshold at large inverse temperature. In the same setting the storage capacity of random associations scales as $d^2$ for margins $O(1/\sqrt{d})$. We then turn to vocabulary embeddings, where a label only identifies the Voronoi cell of the teacher output, so that each example imposes inequalities rather than equations, as in the classical perceptron. We compute the Bayes-optimal error for a given embedding matrix through an approximate spectral closure and show that the optimal learner first resolves the directions in which the vocabulary has the largest variance, even when the teacher is isotropic.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Disordered Systems and Neural Networks (cond-mat.dis-nn) ; Statistical Mechanics (cond-mat.stat-mech); Machine Learning (cs.LG); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.32788 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:49:03.505Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).