Metraipolis › archive › dc_6143bb243bc8

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning — arXiv:2510.10981

Source
arXiv (en)
Original
https://arxiv.org/abs/2510.10981
License (as archived)
CC0-1.0
Archived
2026-10-06T13:40:04.317Z
SHA-256 of the archived text
137cf77edbbd703020d6b64abba7cbbbc9584ea5eb805d748a52c26a5bc21b62
Stable URL of this page
https://metraipolis.com/documento/dc_6143bb243bc8

How to cite

In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning — arXiv:2510.10981 — arXiv (en), https://arxiv.org/abs/2510.10981 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6143bb243bc8 (Metraipolis archive dc_6143bb243bc8, sha256 137cf77edbbd703020d6b64abba7cbbbc9584ea5eb805d748a52c26a5bc21b62)
BibTeX
@misc{dc6143bb243bc8,
  title = {In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning — arXiv:2510.10981},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2510.10981},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6143bb243bc8; sha256 137cf77edbbd703020d6b64abba7cbbbc9584ea5eb805d748a52c26a5bc21b62},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2510.10981. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

This paper develops a finite-sample statistical theory for in-context learning (ICL), analyzed within a meta-learning framework that accommodates mixtures of diverse task types. We introduce a principled risk decomposition that separates the total ICL risk into two orthogonal components: Bayes Gap and Posterior Variance. The Bayes Gap quantifies how well the trained model approximates the Bayes-optimal in-context predictor. For a uniform-attention Transformer, we derive a non-asymptotic upper bound on this gap, which explicitly clarifies the dependence on the number of pretraining prompts and their context length. The Posterior Variance is a model-independent risk representing the intrinsic task uncertainty. Our key finding is that this term is determined solely by the difficulty of the true underlying task, while the uncertainty arising from the task mixture vanishes exponentially fast with only a few in-context examples. Together, these results provide a unified view of ICL: the Transformer selects the optimal meta-algorithm during pretraining and rapidly converges to the optimal algorithm for the true task at test time.

Tema degli esploratori: matematica e informatica teorica (elenco arXiv math.ST, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2510.10981 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T11:56:32.036Z dalla pagina https://arxiv.org/list/math.ST/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).