Metraipolis › archive › dc_ea6b671d7fbb

Steering Vectors as a Training Signal in LLM Post-Training — arXiv:2610.40998

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40998
License (as archived): CC0-1.0
Archived: 2026-11-02T07:00:54.577Z
SHA-256 of the archived text: 3c2215a622ca0617834173bfebf3a633bc890e389143daea97c947a87b1d2637
Stable URL of this page: https://metraipolis.com/documento/dc_ea6b671d7fbb

How to cite

Steering Vectors as a Training Signal in LLM Post-Training — arXiv:2610.40998 — arXiv (en), https://arxiv.org/abs/2610.40998 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ea6b671d7fbb (Metraipolis archive dc_ea6b671d7fbb, sha256 3c2215a622ca0617834173bfebf3a633bc890e389143daea97c947a87b1d2637)

BibTeX

@misc{dcea6b671d7fbb,
  title = {Steering Vectors as a Training Signal in LLM Post-Training — arXiv:2610.40998},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40998},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ea6b671d7fbb; sha256 3c2215a622ca0617834173bfebf3a633bc890e389143daea97c947a87b1d2637},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40998. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Steering vectors (SVs) are residual-stream directions that shift the behavior of LLMs when added to hidden states at inference time. While prior work has mainly used SVs as post-hoc controls, we ask whether the same directions can also guide parameter updates during post-training. We study this question across supervised learning, including SFT and distillation, and reinforcement learning with verifiable rewards (RLVR). Our framework considers two intervention regimes: a hidden-state hook during teacher-forced supervised training, and an advantage-repair mechanism on wrong rollouts in GRPO. For each regime, we analyze three design axes, which are the steering direction, its placement, and the operation used to combine it with training. In supervised settings, SV injection improves over plain SFT and distillation, with gains scaling with the off-policy gap and reaching up to a $12\%$ improvement when distilling from thinking-mode teachers. These gains hold across three random seeds. In RL settings, applying a correctness-contrast direction as advantage repair on wrong rollouts, including otherwise zero-gradient all-wrong groups, recovers a learning signal that transfers to held-out math benchmarks and acts as a composable auxiliary signal that adds to existing zero-variance remedies such as entropy-aware advantage shaping. We further identify three failure boundaries. In detail, code generation can break the direction, cross-family layer transfer can break the placement, and inference-time application on an SV-trained checkpoint can break the intervention regime. Together, these results establish steering vectors as a conditional but effective training signal for LLM post-training. Our code is available here.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.40998 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T06:56:58.360Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).