Metraipolis › archive › dc_a9b092fb26af

RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation — arXiv:2610.05308

Source
arXiv
Original
https://arxiv.org/abs/2610.05308
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:19.157Z
SHA-256 of the archived text
826987af2af49fb8835228159124549982248d9489ded8874b975b552b62524a
Stable URL of this page
https://metraipolis.com/documento/dc_a9b092fb26af

How to cite

RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation — arXiv:2610.05308 — arXiv, https://arxiv.org/abs/2610.05308 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a9b092fb26af (Metraipolis archive dc_a9b092fb26af, sha256 826987af2af49fb8835228159124549982248d9489ded8874b975b552b62524a)
BibTeX
@misc{dca9b092fb26af,
  title = {RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation — arXiv:2610.05308},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05308},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a9b092fb26af; sha256 826987af2af49fb8835228159124549982248d9489ded8874b975b552b62524a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05308. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Rubric-based reinforcement learning (RL) provides interpretable rewards for aligning large language models (LLMs) by evaluating responses against query-specific evaluation criteria. To construct rubrics at scale, a straightforward approach to LLM-based rubric generation is to prompt an LLM to generate a rubric directly from the query. However, rubrics directly generated by LLMs are vulnerable to reward hacking, since omitted or underspecified criteria allow the policy to obtain high rubric rewards with low-quality responses. Existing LLM-based rubric generation methods improve the granularity and coverage of the generated criteria but do not proactively guard against reward hacking. To address this limitation, we propose RubricArmor, an adversarial framework that exposes and mitigates potential reward hacking at the rubric generation stage before it occurs in subsequent RL. Specifically, RubricArmor performs adversarial evolution, in which an attack step and a repair step alternate over multiple rounds. The attack step simulates the reward hacking of the policy by constructing adversarial responses that satisfy the current rubric but fail to properly complete the task. The repair step then revises the rubric to detect the response defects exposed by the attack step while preserving other valid criteria. Extensive experiments demonstrate that RubricArmor outperforms competitive rubric generation baselines and translates into more effective downstream rubric-based RL.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.05308 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.324Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).