Metraipolis › archive › dc_2646951ee000
Solvable dynamics model of in-context learning — arXiv:2611.00921
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00921
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:36:42.011Z
- SHA-256 of the archived text
34be42e606af259e33d52e542cfcebaf5832f7382c6348a26b0792d22dd71a97- Stable URL of this page
https://metraipolis.com/documento/dc_2646951ee000
How to cite
Solvable dynamics model of in-context learning — arXiv:2611.00921 — arXiv (en), https://arxiv.org/abs/2611.00921 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2646951ee000 (Metraipolis archive dc_2646951ee000, sha256 34be42e606af259e33d52e542cfcebaf5832f7382c6348a26b0792d22dd71a97)
BibTeX
@misc{dc2646951ee000,
title = {Solvable dynamics model of in-context learning — arXiv:2611.00921},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00921},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2646951ee000; sha256 34be42e606af259e33d52e542cfcebaf5832f7382c6348a26b0792d22dd71a97},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00921. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pretrained Transformer architectures display a surprising in-context learning ability without additional training. How the learning process guides the network to achieve this intriguing capability remains unsolved. In this study, we focus on a single-layer, single-head linear attention model, where empirical risk minimization is defined for learning. Given a finite amount of training examples, we determine the exact solution of network parameter dynamics, which is linked to the training error and generalization error, characterizing how memorization and generalization emerge from learning and how these transitions are related to curvatures of the empirical risk landscape. Furthermore, we calculate a data-dependent free energy, from which the training and generalization errors in the long-time limit of learning dynamics can be well predicted. Our work adds insights toward understanding the mechanism of in-context learning, especially from a learning dynamics perspective. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Disordered Systems and Neural Networks (cond-mat.dis-nn). Pagina del lavoro: https://arxiv.org/abs/2611.00921 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:12:54.465Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).