Metraipolis › archive › dc_40cc31ed925a

From Copying to Composition: How Language Models Develop In-Context Learning — arXiv:2610.19113

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.19113
License (as archived)
CC0-1.0
Archived
2026-10-15T14:23:17.149Z
SHA-256 of the archived text
81bf84acd04c5e25d4458a222742e7fa1ee89dc37a0486f8c1941c51ac5a1325
Stable URL of this page
https://metraipolis.com/documento/dc_40cc31ed925a

How to cite

From Copying to Composition: How Language Models Develop In-Context Learning — arXiv:2610.19113 — arXiv (en), https://arxiv.org/abs/2610.19113 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_40cc31ed925a (Metraipolis archive dc_40cc31ed925a, sha256 81bf84acd04c5e25d4458a222742e7fa1ee89dc37a0486f8c1941c51ac5a1325)
BibTeX
@misc{dc40cc31ed925a,
  title = {From Copying to Composition: How Language Models Develop In-Context Learning — arXiv:2610.19113},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19113},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_40cc31ed925a; sha256 81bf84acd04c5e25d4458a222742e7fa1ee89dc37a0486f8c1941c51ac5a1325},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19113. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In-context learning (ICL) allows language models to adapt their behavior from instructions and demonstrations in a prompt, but accuracy alone does not reveal how contextual information controls their predictions. We study the acquisition and trainability of ICL with a controlled diagnostic that separates copying a nearby demonstration from integrating information across the prompt. Across 17 models, about a third prefer the copied answer, a few are undecided, and the rest prefer the rule-consistent answer, which goes with higher accuracy and with control by the requested value rather than the closest conflicting example. Checkpoint analyses show that models begin prioritizing task-relevant prompt tokens over conflicting examples long before their accuracy improves beyond chance level. In matched training experiments, synthetic documents that define endings and then query them shift models toward compositional responses on the controlled diagnostic, whereas fine-tuning on definition exercises built from rewritten real documents improves accuracy when prompts reassign label meanings. The results suggest that ICL is not acquired as a single, uniform capability. Instead, prompt control, answer preference, construction without candidates, and transfer can develop along distinct trajectories.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.19113 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.831Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).