Metraipolis › archive › dc_e97c526eae80

What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency — arXiv:2611.01311

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01311
License (as archived)
CC0-1.0
Archived
2026-11-03T11:01:40.784Z
SHA-256 of the archived text
938e926edca973d526b8ad7b0eed784a1f00bddf4e2c867d224194f48e424ab1
Stable URL of this page
https://metraipolis.com/documento/dc_e97c526eae80

How to cite

What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency — arXiv:2611.01311 — arXiv (en), https://arxiv.org/abs/2611.01311 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e97c526eae80 (Metraipolis archive dc_e97c526eae80, sha256 938e926edca973d526b8ad7b0eed784a1f00bddf4e2c867d224194f48e424ab1)
BibTeX
@misc{dce97c526eae80,
  title = {What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency — arXiv:2611.01311},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01311},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e97c526eae80; sha256 938e926edca973d526b8ad7b0eed784a1f00bddf4e2c867d224194f48e424ab1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01311. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM judges often check, against an amendment record, whether the documents an answer cites are still in force. Such records are rarely complete. From a table of replacements, our test generates matched answers citing a replaced document, its successor, the replaced document before its replacement, and an older document that a newer one updated without replacing; it removes links from the record and scores each finding by its stated reason. On 48 replacements from the IETF RFC index, judges given a correct record caught nearly every replaced citation, as a lookup would. On the 24 faults whose link we removed, every judge caught more from the texts alone than with the record: Gemini 3.1 Pro 23/24 against 1/24, GPT-5.5 23/24 against 9/24, Claude Opus 4.5 17/24 against 9/24 and Gemma 4 26B 13/24 against 0/24. The drop recurred on a second draw of removed links. Most misses carried no finding; in an exploratory run with a conflict code, only Gemini used it often. Rewording the gap as "no replacement recorded" did not help; presenting the record as an unverified note, or warning that it was incomplete, restored most lost catches for Gemini, GPT-5.5 and Claude (significant after correction for the first two) but at most 5 of 13 for Gemma; where catches returned, so did false alarms on updates. With both texts, judges mostly told replacements from updates; every update flagged had a near-identical title. A descriptive case study on Indian rooftop-solar rules is appended; we release the harness.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.01311 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:45.765Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).