Metraipolis › archive › dc_e79d09cc1ff1

A mechanistic study of language model introspection — arXiv:2609.35108

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35108
License (as archived)
CC0-1.0
Archived
2026-10-07T21:03:17.107Z
SHA-256 of the archived text
a0bf06f67892be3df2bb1806d618f74473738acfbbc65dd13302a87c0de932cf
Stable URL of this page
https://metraipolis.com/documento/dc_e79d09cc1ff1

How to cite

A mechanistic study of language model introspection — arXiv:2609.35108 — arXiv (en), https://arxiv.org/abs/2609.35108 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e79d09cc1ff1 (Metraipolis archive dc_e79d09cc1ff1, sha256 a0bf06f67892be3df2bb1806d618f74473738acfbbc65dd13302a87c0de932cf)
BibTeX
@misc{dce79d09cc1ff1,
  title = {A mechanistic study of language model introspection — arXiv:2609.35108},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35108},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e79d09cc1ff1; sha256 a0bf06f67892be3df2bb1806d618f74473738acfbbc65dd13302a87c0de932cf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35108. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) can sometimes report perturbations to their internal activations---even when the input provides no evidence that an intervention occurred. How do models detect and localize such internal changes? We study this question using a controlled task that keeps the input text fixed. We either inject a concept vector into the hidden state at one of ten token positions or apply no intervention. The model is asked to identify the perturbed position or report that no intervention occurred. Across three model families, we identify two small groups of attention heads with distinct roles in introspective reporting. Middle-layer gate heads influence whether the model reports a change, while router heads in a later layer help select the position to report. Interventions on gate heads can suppress position reports even when router heads supply location information. We further examine why reporting accuracy varies across concepts. Concept vectors that are localized more accurately produce stronger attention-score and output responses in gate heads, which is associated with better alignment of the induced key and value changes in their QK and OV computations. Together, these findings identify attention-head mechanisms supporting introspective detection and localization.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.35108 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:03.261Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).