Metraipolis › archive › dc_d7d5769e8a48

RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection — arXiv:2609.39229

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39229
License (as archived)
CC0-1.0
Archived
2026-10-07T12:45:04.555Z
SHA-256 of the archived text
2a9afba0d015864b29ebcdf4521eb89c7b27f58d669e91c1a06d9a19685ea507
Stable URL of this page
https://metraipolis.com/documento/dc_d7d5769e8a48

How to cite

RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection — arXiv:2609.39229 — arXiv (en), https://arxiv.org/abs/2609.39229 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d7d5769e8a48 (Metraipolis archive dc_d7d5769e8a48, sha256 2a9afba0d015864b29ebcdf4521eb89c7b27f58d669e91c1a06d9a19685ea507)
BibTeX
@misc{dcd7d5769e8a48,
  title = {RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection — arXiv:2609.39229},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39229},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d7d5769e8a48; sha256 2a9afba0d015864b29ebcdf4521eb89c7b27f58d669e91c1a06d9a19685ea507},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39229. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary frontier models, costly and ill-suited to high-throughput monitoring. We investigate whether a panel of cheap open-weight judges (4--9B) can be aggregated to stand in for a frontier one, what the substitution sacrifices, and when it is worth making. We propose RAIM, an aggregation scheme robust to the members' correlated errors, coupling a cross-fitted stacked logistic regression with an admissibility test that, read from the members' own outputs, identifies when aggregating them improves on their best member and stays within reach of the frontier judge. We instantiate RAIM with ten judges from disjoint families across eight faithfulness benchmarks. Against Claude Sonnet, the panel retains a median 93% of its Cohen's $\kappa$ and gives up only 2.9 points of balanced accuracy on average; read as paired differences, it clearly improves on one benchmark and clearly worsens on three (only two by a non-negligible margin), leaving four unresolved. At a sixty-fourth of the frontier's inference price, the operative expense is a one-time in-domain calibration on 50--100 labelled records. The panel is also competitive with purpose-trained detectors on their home benchmarks (within 1.3 accuracy points of GPT-4o and 1.9 of the LLM-AggreFact leader), and beats the strongest one we reran by 6 points on our grounded sets. Whether aggregation pays depends on the members themselves: where several capable members err on different items, the panel improves on its best judge and approaches the frontier; where one dominates, the stacker recovers the leader, and only there does the frontier remain materially ahead. Both conditions are read off the calibration set at no further cost, so a cheap panel can stand in for a frontier one wherever this audit admits it.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.39229 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:14.037Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).