Metraipolis › archive › dc_858df11eb3c3

DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditing — arXiv:2608.14950

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.14950
License (as archived): CC0-1.0
Archived: 2026-10-18T18:56:38.786Z
SHA-256 of the archived text: 84473af7e5be55b61862133973c1d9bf8df5416d634c1de9136141b6aded97af
Stable URL of this page: https://metraipolis.com/documento/dc_858df11eb3c3

How to cite

DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditing — arXiv:2608.14950 — arXiv (en), https://arxiv.org/abs/2608.14950 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_858df11eb3c3 (Metraipolis archive dc_858df11eb3c3, sha256 84473af7e5be55b61862133973c1d9bf8df5416d634c1de9136141b6aded97af)

BibTeX

@misc{dc858df11eb3c3,
  title = {DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditing — arXiv:2608.14950},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14950},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_858df11eb3c3; sha256 84473af7e5be55b61862133973c1d9bf8df5416d634c1de9136141b6aded97af},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14950. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Generative AI systems are increasingly producing real-world artifacts, however their efficacy and validity are often evaluated via context-free LLM-scoring. These judges can be miscalibrated by irrelevant in-context reference examples, creating false confidence and allowing low-quality or harmful outputs to pass evaluation. We study this failure mode as context-induced miscalibration and introduce DA-RAC, a distance-aware reference-anchored calibration method for LLM judges. DA-RAC retrieves semantically and structurally similar labeled anchors for each judgement scenario, weights them by distance, and exposes neighborhood difficulty as a calibration and triage signal. On multi-run LLM-judge evaluation benchmarks, it improves calibration and reduces false-pass risk relative to zero-shot, chain-of-thought evaluation, and static-anchor baselines. Mechanistic analysis shows that judge scores vary systematically with anchor distance, while static references can induce misleading decision boundaries. Thus LLM-judgement requires not only better models, but also calibrated, auditable reference selection, especially when automated evaluation is used to support high-impact AI generated artifacts. Judgments should be grounded in relevant, inspectable, and contestable interpretive artifacts.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.14950 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T18:06:57.838Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).