Metraipolis › archive › dc_4c48bcd3c413

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback — arXiv:2609.33989

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33989
License (as archived)
CC0-1.0
Archived
2026-10-07T21:53:17.130Z
SHA-256 of the archived text
a4023bcf450fbc3800df874b4f808e20382bf780ed146cbd5eab07af8eb55bd7
Stable URL of this page
https://metraipolis.com/documento/dc_4c48bcd3c413

How to cite

RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback — arXiv:2609.33989 — arXiv (en), https://arxiv.org/abs/2609.33989 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4c48bcd3c413 (Metraipolis archive dc_4c48bcd3c413, sha256 a4023bcf450fbc3800df874b4f808e20382bf780ed146cbd5eab07af8eb55bd7)
BibTeX
@misc{dc4c48bcd3c413,
  title = {RewardExplainer: Learning Reward Model Explanations from Counterfactual Preference Feedback — arXiv:2609.33989},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33989},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4c48bcd3c413; sha256 a4023bcf450fbc3800df874b4f808e20382bf780ed146cbd5eab07af8eb55bd7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33989. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reward models (RMs) are a key component of large language model post-training, providing reward signals for subsequent reinforcement learning. However, conventional discriminative RMs typically output only scalar scores, making it difficult to identify the response behaviors associated with their scoring decisions. Existing interpretation methods often rely on predefined high-level attributes and require repeated counterfactual interventions for each response pair to validate candidate explanations, lacking a closed-loop mechanism that uses RMs' feedback to train a reusable explainer. To address this, we propose RewardExplainer, a framework that obtains feedback from the target reward model through counterfactual rewriting and uses this feedback to further optimize the explainer. RewardExplainer generates open-ended, atomic, and intervenable natural-language scoring mechanisms, making explanations more concrete, readable, and actionable. It further converts counterfactual feedback into preference supervision, enabling the explainer to more faithfully capture the target RM's scoring preferences and sensitive behaviors than single-pass generation. Extensive experiments across multiple target RMs and explainer backbones show consistent improvements. Beyond interpretation, we use the generated mechanisms to identify potential bias patterns and construct targeted debiasing data for fine-tuning the reward model, improving robustness on reward-hacking benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.33989 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.580Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).