Metraipolis › archive › dc_d617cdcff6b5
Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning — arXiv:2609.29060
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.29060
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T05:26:28.842Z
- SHA-256 of the archived text
8027a08f61bf5a7c3227256c297e7b696820d5da2f7f693667546f27570d1571- Stable URL of this page
https://metraipolis.com/documento/dc_d617cdcff6b5
How to cite
Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning — arXiv:2609.29060 — arXiv (en), https://arxiv.org/abs/2609.29060 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d617cdcff6b5 (Metraipolis archive dc_d617cdcff6b5, sha256 8027a08f61bf5a7c3227256c297e7b696820d5da2f7f693667546f27570d1571)
BibTeX
@misc{dcd617cdcff6b5,
title = {Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning — arXiv:2609.29060},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.29060},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d617cdcff6b5; sha256 8027a08f61bf5a7c3227256c297e7b696820d5da2f7f693667546f27570d1571},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29060. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformers achieve remarkable performance by jointly learning broad families of tasks during pretraining and adapting to unseen tasks from only a short prompt. Yet a rigorous mathematical and statistical understanding of this phenomenon remains limited. This paper aims to study how Transformers exploit shared cross-task structure and how this structure affects the sample complexity of in-context learning (ICL). Specifically, we characterize task-space complexity through covering numbers under a prescribed metric, thereby quantifying the low-dimensional cross-task structure without requiring an explicit parametric representation. The resulting cover provides a set of anchor functions, which we use to introduce a task-identification-and-evaluation procedure: context observations localize an unseen task among the anchor functions, and the response at a query is predicted by aggregating the corresponding anchor function query evaluations. For approximation, we explicitly construct a Transformer with Softmax attention to approximate this procedure. For generalization, we derive an error bound that separates the effects of the number of pretraining tasks and the prompt length. The scaling with respect to the number of pretraining tasks is governed by the intrinsic dimensions of the task space and input domain; once sufficiently many tasks are available, the dependence on the prompt context length becomes dimension-free. To the best of our knowledge, this is the first work to quantify cross-task complexity for general nonlinear task families and explicitly construct a Transformer that exploits their low-dimensional structure to perform ICL. Our theory provides a quantitative explanation of how joint pretraining across related tasks improves in-context generalization. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.LG). Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.29060 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T04:50:39.581Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).