Metraipolis › archive › dc_3e2791e8eadd

Causal and Interpretable Structures in LLM Compositional Tasks — arXiv:2609.35970

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35970
License (as archived)
CC0-1.0
Archived
2026-10-07T16:25:08.384Z
SHA-256 of the archived text
8abade36f3a8eef7dd950f8588f56e5865de7e58c64b2cc9ef25f5ffd00fe92d
Stable URL of this page
https://metraipolis.com/documento/dc_3e2791e8eadd

How to cite

Causal and Interpretable Structures in LLM Compositional Tasks — arXiv:2609.35970 — arXiv (en), https://arxiv.org/abs/2609.35970 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e2791e8eadd (Metraipolis archive dc_3e2791e8eadd, sha256 8abade36f3a8eef7dd950f8588f56e5865de7e58c64b2cc9ef25f5ffd00fe92d)
BibTeX
@misc{dc3e2791e8eadd,
  title = {Causal and Interpretable Structures in LLM Compositional Tasks — arXiv:2609.35970},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35970},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e2791e8eadd; sha256 8abade36f3a8eef7dd950f8588f56e5865de7e58c64b2cc9ef25f5ffd00fe92d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35970. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models are able to solve tasks whose answers depend on not only individual input tokens, but also on relations among them. How is such relational information represented and processed across transformer layers? We study activations from ensembles of prompts that require inferring relationships between three tokens corresponding to a cyclic concept (months, hours, weekdays, and musical notes) to correctly predict the next token. Across model families (Llama, Qwen, Gemma, and Mistral) and cyclic concepts, we find a consistent layerwise progression in how the joint dependence among the tokens is geometrically organized and causally used: intermediate layers use a joint representation based on the inferred relationship between two tokens, while later layers use a joint representation associated with all three tokens to correctly complete the task. We also find other relationships between tokens that are geometrically structured but remain causally inert in the next-token prediction. Crucially, when taken together, these geometric and causal investigations reveal the representation-level mechanism that progressively organizes and composes the relational information to form the answer. More surprisingly, restricting the models to such causally relevant joint representations improves next-token prediction accuracy.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.35970 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.754Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).