Metraipolis › archive › dc_173dd2abbea1
Theory on Attention Dynamics for Out-of-Distribution In-Context Learning — arXiv:2609.36448
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36448
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:29:03.556Z
- SHA-256 of the archived text
61568e64efdcfdc3efbaa80f0de97a7ab9e0ffc79d737d50d3add401a2d3a52b- Stable URL of this page
https://metraipolis.com/documento/dc_173dd2abbea1
How to cite
Theory on Attention Dynamics for Out-of-Distribution In-Context Learning — arXiv:2609.36448 — arXiv (en), https://arxiv.org/abs/2609.36448 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_173dd2abbea1 (Metraipolis archive dc_173dd2abbea1, sha256 61568e64efdcfdc3efbaa80f0de97a7ab9e0ffc79d737d50d3add401a2d3a52b)
BibTeX
@misc{dc173dd2abbea1,
title = {Theory on Attention Dynamics for Out-of-Distribution In-Context Learning — arXiv:2609.36448},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36448},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_173dd2abbea1; sha256 61568e64efdcfdc3efbaa80f0de97a7ab9e0ffc79d737d50d3add401a2d3a52b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36448. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Transformers have demonstrated remarkable in-context learning (ICL) capabilities, enabling them to perform new tasks without additional fine-tuning. However, their performance often deteriorates when encountering out-of-distribution (OOD) inputs that deviate from the training distribution, and the underlying theory remains poorly understood. To fill this gap, we characterize the OOD error under the input distribution shift through the interplay between the dynamics of the so-called $\alpha$-type and $\beta$-type attention weights, which represent the transformer's confidence in identifying the correct and incorrect features, respectively. Our results indicate that the OOD error for each feature depends on all pairwise interactions between the training features and OOD features, and under certain cases the transformer performs no better than random guessing. To improve the OOD generalization performance, we next investigate the impact of model finetuning with the OOD data, and particularly, characterize the model forgetting performance on the source domain. Interestingly, the performance on the source domain may not always degrade after finetuning, which highly depends on the nature of the feature shift: finetuning on OOD domain keeps enhancing the confidence of identifying correct features from the original distribution, while the interference from other incorrect features may either increase or decrease. Extensive experiments on both synthetic and real data are conducted to corroborate the theoretical insights. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36448 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.164Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).