Metraipolis › archive › dc_3427200cf2bb
In-context learning of solution to simple linear regression task using transformer with masked linear self-attention — arXiv:2611.02817
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02817
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:38:42.385Z
- SHA-256 of the archived text
334f97b940c714085a8344809967d5b5b9051a29914d708a370ff20bf4a1998f- Stable URL of this page
https://metraipolis.com/documento/dc_3427200cf2bb
How to cite
In-context learning of solution to simple linear regression task using transformer with masked linear self-attention — arXiv:2611.02817 — arXiv (en), https://arxiv.org/abs/2611.02817 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3427200cf2bb (Metraipolis archive dc_3427200cf2bb, sha256 334f97b940c714085a8344809967d5b5b9051a29914d708a370ff20bf4a1998f)
BibTeX
@misc{dc3427200cf2bb,
title = {In-context learning of solution to simple linear regression task using transformer with masked linear self-attention — arXiv:2611.02817},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02817},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3427200cf2bb; sha256 334f97b940c714085a8344809967d5b5b9051a29914d708a370ff20bf4a1998f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02817. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
In-context learning is a remarkable property of transformers and has recently received a lot of interest. In many studies of in-context learning, it has been shown that transformers are capable of implementing solvers for linear and non-linear regression problems, in which the most of them implement gradient descent algorithm. However, it is still unclear whether those implementations have actually been acquired through training. In this paper, we constructed a transformer with masked linear self-attention, which in-context learns the least squares estimate in a simple regression task. The point here is that the closed form (analytical) solution is obtained by using layer normalization rather than an approximate solution based on the gradient descent algorithm. Then, we showed an experimental example, in which, by taking the non-uniqueness of the transformer representation into account, our implementation for a closed form solution is used in the transformer trained with the l1 regularization when the target output is the least squares estimate. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.02817 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:12:54.593Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).