Metraipolis › archive › dc_0ca2cda3435f

Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models — arXiv:2610.39016

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.39016
License (as archived)
CC0-1.0
Archived
2026-10-31T06:13:01.939Z
SHA-256 of the archived text
6905e0392741b63fb7b28ccd6426ef162e9a14edb7a15d85d4e201a3263d4dc7
Stable URL of this page
https://metraipolis.com/documento/dc_0ca2cda3435f

How to cite

Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models — arXiv:2610.39016 — arXiv (en), https://arxiv.org/abs/2610.39016 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0ca2cda3435f (Metraipolis archive dc_0ca2cda3435f, sha256 6905e0392741b63fb7b28ccd6426ef162e9a14edb7a15d85d4e201a3263d4dc7)
BibTeX
@misc{dc0ca2cda3435f,
  title = {Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models — arXiv:2610.39016},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.39016},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0ca2cda3435f; sha256 6905e0392741b63fb7b28ccd6426ef162e9a14edb7a15d85d4e201a3263d4dc7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39016. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Compositional generalization remains a persistent limitation of transformer language models: even when a model learns to combine known facts correctly in-distribution (ID), the same compositional mechanism may fail when applied to out-of-distribution (OOD) knowledge. We investigate this compositionality gap in a controlled two-hop reasoning setting using a GPT-2-like transformer model trained from scratch. Building on prior work identifying a circuit associated with compositional generalization, we test whether the failure of OOD transfer is related to how atomic knowledge is structurally exposed during training. We modify the training distribution by presenting both ID and OOD atomic facts in a format positionally aligned with the second-hop retrieval step of the learned compositional circuit, without modifying the model architecture or adding OOD compositional supervision. Under this intervention, the model successfully transfers the learned ID compositional behaviour to held-out OOD atomic facts, closing the observed compositionality gap in the controlled setting. Our results provide evidence that the observed compositionality gap is not a fundamental architecture limitation and that it can be overcome through mechanistically informed training distribution design.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.39016 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.689Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).