Metraipolis › archive › dc_fcc1eed47ff2

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization — arXiv:2608.15445

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.15445
License (as archived): CC0-1.0
Archived: 2026-10-18T17:19:05.917Z
SHA-256 of the archived text: 3ed49a1c4166725638e601632db6250ececcc6be4e0894c334158665aa8301ff
Stable URL of this page: https://metraipolis.com/documento/dc_fcc1eed47ff2

How to cite

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization — arXiv:2608.15445 — arXiv (en), https://arxiv.org/abs/2608.15445 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fcc1eed47ff2 (Metraipolis archive dc_fcc1eed47ff2, sha256 3ed49a1c4166725638e601632db6250ececcc6be4e0894c334158665aa8301ff)

BibTeX

@misc{dcfcc1eed47ff2,
  title = {Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization — arXiv:2608.15445},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.15445},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fcc1eed47ff2; sha256 3ed49a1c4166725638e601632db6250ececcc6be4e0894c334158665aa8301ff},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15445. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization. Endpoint accuracy on the training distribution cannot tell the two apart, because solving the task and exploiting a surface feature can satisfy the reward equally well. We treat this as a measurement problem: what does a benchmark score measure once a model has been optimized against a correct but confounded signal? We train language models with GRPO on multiple-choice math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased answer positions. Across Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above 0.90 in smaller models and collapses unbiased accuracy toward chance, so accuracy stops measuring math ability and instead measures an answer-position policy. We further find reasoning-answer decoupling: capable models generate reasoning that reaches the correct numeric answer while still selecting A. We track this with numeric extraction and an LLM judge (GPT-4.1-mini; Qwen2.5-3B decoupling rate is about 0.66). The broken construct generalizes beyond the training domain: biased models inflate A-rates on out-of-domain MMLU and value-laden prompts. Continued training on unbiased data reverses the in-domain shift unevenly and only partially reverses the out-of-domain one, so a model can appear restored on its training distribution while remaining biased on unseen inputs. Reasoning-answer decoupling rate, together with answer distributions and out-of-domain behavior, separates capability loss from a learned, transferable shortcut.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.15445 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:13.802Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).