Metraipolis › archive › dc_8919381e8def

Reliability of LLM Judges for Evaluating Entity Alignment — arXiv:2610.09554

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09554
License (as archived)
CC0-1.0
Archived
2026-10-08T11:09:23.233Z
SHA-256 of the archived text
a46fa45abe3b51d72607c2fdef6f363c4fb4e481d490e8a7453994c57f849770
Stable URL of this page
https://metraipolis.com/documento/dc_8919381e8def

How to cite

Reliability of LLM Judges for Evaluating Entity Alignment — arXiv:2610.09554 — arXiv (en), https://arxiv.org/abs/2610.09554 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8919381e8def (Metraipolis archive dc_8919381e8def, sha256 a46fa45abe3b51d72607c2fdef6f363c4fb4e481d490e8a7453994c57f849770)
BibTeX
@misc{dc8919381e8def,
  title = {Reliability of LLM Judges for Evaluating Entity Alignment — arXiv:2610.09554},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09554},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8919381e8def; sha256 a46fa45abe3b51d72607c2fdef6f363c4fb4e481d490e8a7453994c57f849770},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09554. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12-0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93-1.00) and significant recovery on biomedical pairs (0.93-0.95). Counterfactual experiments confirm causality (FSR 53-99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen's kappa=0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH-SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.09554 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:05.942Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).