Metraipolis › archive › dc_e7650a464253

Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion — arXiv:2608.14617

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14617
License (as archived)
CC0-1.0
Archived
2026-10-18T17:47:51.202Z
SHA-256 of the archived text
5d3010c99d719b261c8334a1bf8637cc539af76a442a8c17c447712c5145cf4f
Stable URL of this page
https://metraipolis.com/documento/dc_e7650a464253

How to cite

Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion — arXiv:2608.14617 — arXiv (en), https://arxiv.org/abs/2608.14617 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e7650a464253 (Metraipolis archive dc_e7650a464253, sha256 5d3010c99d719b261c8334a1bf8637cc539af76a442a8c17c447712c5145cf4f)
BibTeX
@misc{dce7650a464253,
  title = {Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion — arXiv:2608.14617},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14617},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e7650a464253; sha256 5d3010c99d719b261c8334a1bf8637cc539af76a442a8c17c447712c5145cf4f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14617. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline. We test this on 1,000 real European Court of Human Rights cases from LexGLUE and FairLex, predicting whether the Court found a Convention violation from the case's fact paragraphs. We compare three families across two frontier LLMs (Claude Opus 4.8 and GPT-5.5) as per-fact evidence estimators: (A) the raw LLM, (B) the LLM routed through the fusion pipeline, and (C) a term-frequency baseline through the same pipeline. Across roughly 4,750 tests we find: (1) on discrimination (AUROC around 0.83) the pipeline yields no improvement over either the raw LLM or the baseline; a frontier LLM used directly is the strongest single discriminator. (2) Naively composing an LLM with Bayesian-odds and Dempster-Shafer fusion more than doubles calibration error (ECE from about 0.16 to 0.46) via a prior-mismatch mechanism that replicates across both models. (3) Dempster-Shafer fusion is actively unsafe on long chains, committing confidently to wrong labels at below-chance accuracy; we recommend removing it. (4) The pipeline's genuine value is operational: routed through a conformal selective-prediction layer, the system decides which cases to automate and which to escalate. After removing Dempster-Shafer, recalibrating, and applying class-conditional risk control on the full 1,000-case set, the tuned engine auto-clears at 96.8 percent accuracy with 0.5 percent errors escaping and 96.3 percent caught for review, versus 85.9 / 3.8 / 72.1 for an untuned baseline. The contribution of such pipelines in law is calibrated trust, not sharper prediction.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.14617 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:13.911Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).