Metraipolis › archive › dc_8decd13ef2a5

Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3 — arXiv:2610.31624

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.31624
License (as archived): CC0-1.0
Archived: 2026-10-25T08:35:41.763Z
SHA-256 of the archived text: 3c2b2c70c9a3aafb1b9b19b966c788c403e152411dd7f9ce443811ceedcbbe72
Stable URL of this page: https://metraipolis.com/documento/dc_8decd13ef2a5

How to cite

Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3 — arXiv:2610.31624 — arXiv (en), https://arxiv.org/abs/2610.31624 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8decd13ef2a5 (Metraipolis archive dc_8decd13ef2a5, sha256 3c2b2c70c9a3aafb1b9b19b966c788c403e152411dd7f9ce443811ceedcbbe72)

BibTeX

@misc{dc8decd13ef2a5,
  title = {Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3 — arXiv:2610.31624},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31624},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8decd13ef2a5; sha256 3c2b2c70c9a3aafb1b9b19b966c788c403e152411dd7f9ce443811ceedcbbe72},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31624. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline. The result is specific to this model family and benchmark, and judge agreement is moderate, so absolute rates should be read cautiously.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.31624 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:13.276Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).