Metraipolis › archive › dc_6ddda4ccd084

Evaluating Rubric Generation with Interventional Transfer — arXiv:2610.10809

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.10809
License (as archived)
CC0-1.0
Archived
2026-10-09T06:20:59.470Z
SHA-256 of the archived text
1af5deeb95c5d1478e58e90e0cd0190a2a40b5ab6b1fecda651051aa71c9c579
Stable URL of this page
https://metraipolis.com/documento/dc_6ddda4ccd084

How to cite

Evaluating Rubric Generation with Interventional Transfer — arXiv:2610.10809 — arXiv (en), https://arxiv.org/abs/2610.10809 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6ddda4ccd084 (Metraipolis archive dc_6ddda4ccd084, sha256 1af5deeb95c5d1478e58e90e0cd0190a2a40b5ab6b1fecda651051aa71c9c579)
BibTeX
@misc{dc6ddda4ccd084,
  title = {Evaluating Rubric Generation with Interventional Transfer — arXiv:2610.10809},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10809},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6ddda4ccd084; sha256 1af5deeb95c5d1478e58e90e0cd0190a2a40b5ab6b1fecda651051aa71c9c579},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10809. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Instance-specific rubrics are common in AI benchmarks where reliable evaluation requires specific expert knowledge. This approach is difficult to scale, prompting research into the generation of rubrics with large language models (LLMs). However, even when expert rubrics are available as references, it is unclear how to productively evaluate the quality of generated rubrics at scale. In this paper, we introduce a method for the evaluation of rubric generation, which we call Interventional Transfer (IT), based on the idea that two rubrics are similar if they move together when a response is perturbed to pass/fail one of them. In contrast to existing approaches for evaluation of rubric generation, we argue that different forms of interventional transfer can be used to evaluate the utility of generated rubrics for different tasks. For instance, we apply this approach in a case study on HealthBench, where we demonstrate an asymmetry in rubrics generated by Qwen3.8-27B, Deepseek-V4-Flash, and Opus-5, used to evaluate responses from GPT-5.6-Terra. Perturbations that degrade responses according to the generated/expert rubric transfer into lower scores on the corresponding expert/generated rubric, but perturbations that improve on one rubric do not reliably transfer into higher scores on the other. We argue that this finding has implications for the usage of LLM-generated rubrics for performance monitoring and hill-climbing. We contrast our approach with existing approaches for rubric evaluation, which do not surface the same asymmetry that we observe.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.10809 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:50:30.985Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).