Metraipolis › archive › dc_7b259fc0c5b0

Where Is the Truth When AI Agents Disagree? Dependence-Aware Belief Propagation for Unsupervised Truth Discovery (Extended Version) — arXiv:2611.04805

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04805
License (as archived)
CC0-1.0
Archived
2026-11-04T18:34:46.132Z
SHA-256 of the archived text
409f6b845bd2157340cf18b018dc52719969c2d0e05495b0ba6d516b55a6c74e
Stable URL of this page
https://metraipolis.com/documento/dc_7b259fc0c5b0

How to cite

Where Is the Truth When AI Agents Disagree? Dependence-Aware Belief Propagation for Unsupervised Truth Discovery (Extended Version) — arXiv:2611.04805 — arXiv (en), https://arxiv.org/abs/2611.04805 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7b259fc0c5b0 (Metraipolis archive dc_7b259fc0c5b0, sha256 409f6b845bd2157340cf18b018dc52719969c2d0e05495b0ba6d516b55a6c74e)
BibTeX
@misc{dc7b259fc0c5b0,
  title = {Where Is the Truth When AI Agents Disagree? Dependence-Aware Belief Propagation for Unsupervised Truth Discovery (Extended Version) — arXiv:2611.04805},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04805},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7b259fc0c5b0; sha256 409f6b845bd2157340cf18b018dc52719969c2d0e05495b0ba6d516b55a6c74e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04805. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLM agents often give conflicting answers to the same question, and many deployed agents are variants of one base model, so their errors are correlated. We study truth discovery among such agents: inferring the true answer and each agent's reliability without labels. Classical truth-discovery and crowdsourcing estimators reward an agreeing bloc and can fall below majority vote. We propose BP-TD, which adds per-agent decorrelation weights to mean-field variational inference on a one-coin Dawid-Skene model, so that agents sharing a base model count together as about one vote. Its first iteration is exactly a weighted vote; later iterations estimate agent reliability. With 13 local agents on MMLU, BP-TD beats majority vote without any labels (0.716 vs. 0.624 on 160 questions; 0.707 vs. 0.674 on 10,000) and ranks agents by their true accuracy, which classical estimators invert. The gain depends on the pool: over 2,852 sub-pools of four benchmarks, decorrelation helps when the largest correlated bloc is weaker than the other agents and hurts when it is the strongest, and we found no label-free statistic that tells the two cases apart. Labels change the picture quickly: with 20 labelled questions, selecting the single best agent is more accurate than every aggregator we tested on MMLU, ARC and TruthfulQA.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Databases (cs.DB); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2611.04805 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:32:37.336Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).