Metraipolis › archive › dc_e042b34d0aef

What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track — arXiv:2609.15064

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15064
License (as archived)
CC0-1.0
Archived
2026-10-16T00:11:14.924Z
SHA-256 of the archived text
50209ec462fea81e8294076b644d73c610c72a181df5f6b87210fe9ffffa2e28
Stable URL of this page
https://metraipolis.com/documento/dc_e042b34d0aef

How to cite

What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track — arXiv:2609.15064 — arXiv (en), https://arxiv.org/abs/2609.15064 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e042b34d0aef (Metraipolis archive dc_e042b34d0aef, sha256 50209ec462fea81e8294076b644d73c610c72a181df5f6b87210fe9ffffa2e28)
BibTeX
@misc{dce042b34d0aef,
  title = {What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track — arXiv:2609.15064},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15064},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e042b34d0aef; sha256 50209ec462fea81e8294076b644d73c610c72a181df5f6b87210fe9ffffa2e28},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15064. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) is widely utilized in large language model training to improve targeted capabilities, yet how RL reshapes a model remains poorly understood. Prior attempts to explain how RL works largely offer behavioral perspectives, leaving open what RL gives a model at the representation level: can RL create genuinely novel features, and which existing features does it enhance or suppress? Recent developments in mechanistic interpretability suggest sparse autoencoders (SAEs) as a promising lens to decompose internal activations into human-interpretable features; however, they cannot be directly applied to tracking change across training. In this work, we introduce Fixed-SAE Track, a framework that trains one shared SAE per considered layer on activations pooled across the base model and all RL checkpoints, holding every feature direction fixed so that representation shifts are rigorously defined through the activations of interpretable SAE latents, including the detection of emerging novel features. Validated across multiple datasets and RL algorithms, we find that RL-induced drift is small, gradual, concept specific, and concentrated in late layers, mainly enhancing the sampling rates of a small set of ladder tokens, formatting scaffolding such as step breaks and answer delimiters, rather than reshaping problem content. Steering these features into the base model recovers around 80% of RL's performance gain, suggesting that RL primarily elicits capabilities the model already possesses, much as steering does. We further design a synthetic benchmark with features known by construction to test whether RL can instill genuinely novel features. We believe Fixed-SAE Track provides a principled approach to tracking representation shifts and offers representational evidence for understanding how reinforcement learning changes the inner representation of LLMs.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.15064 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T23:54:14.732Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).