Metraipolis › archive › dc_756abc404dfc

On Language Drift during RLVR Post-Training — arXiv:2610.02015

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.02015
License (as archived): CC0-1.0
Archived: 2026-10-06T12:15:47.132Z
SHA-256 of the archived text: f7e6835447cdf267c9a86ad657758695a2119229d86bd4ec73dc727c916dc094
Stable URL of this page: https://metraipolis.com/documento/dc_756abc404dfc

How to cite

On Language Drift during RLVR Post-Training — arXiv:2610.02015 — arXiv (en), https://arxiv.org/abs/2610.02015 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_756abc404dfc (Metraipolis archive dc_756abc404dfc, sha256 f7e6835447cdf267c9a86ad657758695a2119229d86bd4ec73dc727c916dc094)

BibTeX

@misc{dc756abc404dfc,
  title = {On Language Drift during RLVR Post-Training — arXiv:2610.02015},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.02015},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_756abc404dfc; sha256 f7e6835447cdf267c9a86ad657758695a2119229d86bd4ec73dc727c916dc094},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.02015. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.02015 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.235Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).