Metraipolis › archive › dc_a6599ea598d7

RubricCraft: Where Solutions Diverge, Criteria Emerge — arXiv:2610.31707

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.31707
License (as archived)
CC0-1.0
Archived
2026-10-26T19:50:54.408Z
SHA-256 of the archived text
5794e7df1ce49d9c179a3bc941928f02a96d6f10500f2824ad290a95cc17e163
Stable URL of this page
https://metraipolis.com/documento/dc_a6599ea598d7

How to cite

RubricCraft: Where Solutions Diverge, Criteria Emerge — arXiv:2610.31707 — arXiv (en), https://arxiv.org/abs/2610.31707 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a6599ea598d7 (Metraipolis archive dc_a6599ea598d7, sha256 5794e7df1ce49d9c179a3bc941928f02a96d6f10500f2824ad290a95cc17e163)
BibTeX
@misc{dca6599ea598d7,
  title = {RubricCraft: Where Solutions Diverge, Criteria Emerge — arXiv:2610.31707},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31707},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a6599ea598d7; sha256 5794e7df1ce49d9c179a3bc941928f02a96d6f10500f2824ad290a95cc17e163},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31707. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical. Textual rubrics make evaluation criteria explicit, but writing them requires domain expertise. Rubrics generated from task descriptions alone can miss requirements and failure modes that become apparent in attempted solutions. We introduce RubricCraft, which treats rubric construction as the generation of natural-language specifications guided by candidate solutions. Given a task description and independently sampled solver trajectories, RubricCraft compares how each attempt satisfies or violates the task requirements and uses these differences to construct rubric criteria. It applies the criteria to the collected solutions and revises checks that overlook errors or penalize valid alternatives. The resulting rubrics capture partial correctness without requiring any single solution to serve as a complete reference. On SpreadsheetBench v2, against a prompt-only rubric written and applied by the same model, RubricCraft rubrics pass fewer wrong outputs (29.5% against 42.6%), fail none of the outputs the benchmark's own checker accepts, and rank outputs by correctness more often, while separating models as often. Repair on a single attempt leaves grading unchanged, but repair driven by attempts from three model families acts on different tasks, and an evidence audit judges most of those repairs improvements, many on tasks whose reference solution is itself wrong.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.31707 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:45:33.348Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).