Metraipolis › archive › dc_7f073373ce6d

Debate Training Reduces Reward Hacking in RLAIF — arXiv:2608.17776

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.17776
License (as archived)
CC0-1.0
Archived
2026-10-18T14:48:42.012Z
SHA-256 of the archived text
d146eab60d30bb12178c93e53ded13668d9a737a120115e3de4bcbd6966ef276
Stable URL of this page
https://metraipolis.com/documento/dc_7f073373ce6d

How to cite

Debate Training Reduces Reward Hacking in RLAIF — arXiv:2608.17776 — arXiv (en), https://arxiv.org/abs/2608.17776 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7f073373ce6d (Metraipolis archive dc_7f073373ce6d, sha256 d146eab60d30bb12178c93e53ded13668d9a737a120115e3de4bcbd6966ef276)
BibTeX
@misc{dc7f073373ce6d,
  title = {Debate Training Reduces Reward Hacking in RLAIF — arXiv:2608.17776},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.17776},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7f073373ce6d; sha256 d146eab60d30bb12178c93e53ded13668d9a737a120115e3de4bcbd6966ef276},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.17776. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking compared to a reinforcement learning from AI feedback (RLAIF) baseline. Reward hacking is a central obstacle in RLAIF: as training progresses, the policy learns to exploit systematic errors in its AI judge, degrading task performance, a problem that worsens precisely when the judge is weaker than the policy, the setting most relevant to overseeing increasingly capable AI systems. We study mathematics tasks, where final-answer correctness is verifiable, allowing us to measure reward hacking dynamics. We train a Gemini~2.5 Flash-class policy with a frozen, weaker Gemini~2.5 Flash Lite judge, comparing a single-player RLAIF baseline against debate. While the baseline quickly hacks the judge, debate maintains judge performance throughout training, leading to a higher peak validation accuracy (45\% performance gap recovered) that persists through many RL steps. Additional experiments show that: 1) further weakening the judge leads to faster hacking, but this can be compensated by adding an additional debate round; 2) debate incentives override prompted misalignment; 3) RL using an LLM judge has a smaller train/validation reward gap than RL from verifiable rewards; 4) learning to critique to convince the judge using ground truth labels is possible but slow. Taken together, our results are a positive update on the feasibility of debate, while highlighting that balancing multi-agent training is critical: without player constraints, adversarial training risks defaulting to critic judge-hacking. We show that critique word limits (effective up to 150 words) successfully balance the game and avoid judge hacking, though this introduces a trade-off by restricting critic expressive clarity.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-19 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.17776 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T14:47:59.412Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-19?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).