Metraipolis › archive › dc_ffbeaf9eb2c9

Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation — arXiv:2608.13564

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13564
License (as archived)
CC0-1.0
Archived
2026-10-19T02:20:13.413Z
SHA-256 of the archived text
b95e593078355196b9f6b5098f38117378812925693654d586badd18b2c585a5
Stable URL of this page
https://metraipolis.com/documento/dc_ffbeaf9eb2c9

How to cite

Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation — arXiv:2608.13564 — arXiv (en), https://arxiv.org/abs/2608.13564 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ffbeaf9eb2c9 (Metraipolis archive dc_ffbeaf9eb2c9, sha256 b95e593078355196b9f6b5098f38117378812925693654d586badd18b2c585a5)
BibTeX
@misc{dcffbeaf9eb2c9,
  title = {Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation — arXiv:2608.13564},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13564},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ffbeaf9eb2c9; sha256 b95e593078355196b9f6b5098f38117378812925693654d586badd18b2c585a5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13564. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time. Such a judge is a reward-free proxy whose value depends on whether it can be trusted, yet existing judges either hand-write the scoring rubric, as in G-Eval, or fine-tune the judge's weights, and both tend to credit fluent but unsuccessful trajectories as successes. We instead induce the text of an agent-judging rubric from a small set of ground-truth-labeled trajectories, grounding it in true outcomes. We present RubricForge, which evolves a judge rubric by reflective evolution against labeled trajectories to maximize agreement with the environment reward, freezes it, and applies it to held-out trajectories in one model call with no environment access. The optimized artifact is human-readable text, so every verdict is attributable to named criteria. Using one frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160), the principal gain is faithfulness rather than raw agreement. The edge over a generic G-Eval judge is not statistically significant (McNemar p = 0.248), and absolute-score calibration marginally favors the generic judge (|err| difference -0.048, p = 2x10^-4). Yet RubricForge over-credits failed trajectories roughly half as often (0.115 vs. 0.173 false-pass rate on tau-bench, with three over-credit catches and zero reversals) and ranks graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370). For a reward-free evaluator the false-pass rate, not aggregate agreement, is the deployment-relevant quantity, since a false pass ships a broken agent whereas a false fail merely costs a retry.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.13564 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.709Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).