Metraipolis › archive › dc_dddb2ef2a04a

Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors — arXiv:2609.34344

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.34344
License (as archived): CC0-1.0
Archived: 2026-10-07T21:02:24.717Z
SHA-256 of the archived text: 7c5ea130446aed818547ae3be96f1174ddf7b307615e16e162f5175f2d9f5c09
Stable URL of this page: https://metraipolis.com/documento/dc_dddb2ef2a04a

How to cite

Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors — arXiv:2609.34344 — arXiv (en), https://arxiv.org/abs/2609.34344 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dddb2ef2a04a (Metraipolis archive dc_dddb2ef2a04a, sha256 7c5ea130446aed818547ae3be96f1174ddf7b307615e16e162f5175f2d9f5c09)

BibTeX

@misc{dcdddb2ef2a04a,
  title = {Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors — arXiv:2609.34344},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34344},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dddb2ef2a04a; sha256 7c5ea130446aed818547ae3be96f1174ddf7b307615e16e162f5175f2d9f5c09},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34344. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning of large language models, yet the high dimensionality of parameter updates makes its training dynamics hard to analyze. We study reinforcement learning with verifiable rewards (RLVR) and use vector steering to identify a low-dimensional effective manifold in activation space associated with RL-induced gains. We uncover two geometric properties. (1) Effective Manifold Capacity: the capacity needed to reproduce RL gains can be very small but is not infinitely compressible; at extremely low capacity, intervention dimensionality and input-dependent expressiveness become key constraints, and this requirement varies with injection depth. (2) Control Manifold Separation: effective control directions lie mainly in the low-variance complement of the activation principal subspace. Within a task and base model, the learned geometry stays largely consistent across training configurations, and across tasks geometric alignment correlates with capability transfer. Experiments on 5 LLMs and 6 verifiable-reward tasks support these findings. We then propose Alpha-Stabler, a plug-and-play framework with a Predictor that monitors principal-subspace intrusion for early collapse warnings, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving the orthogonal complement. Alpha-Stabler stabilizes training for 2,000 steps and consistently improves RL gains, offering practical insights for robust post-training. Code: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.34344 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.943Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).