Metraipolis › archive › dc_c818fc098a3c

XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages — arXiv:2611.01370

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01370
License (as archived)
CC0-1.0
Archived
2026-11-03T11:09:56.374Z
SHA-256 of the archived text
b8781313f090baa5bc83c644128584097d40ac0263c5854ccc3d10cf8f4e9447
Stable URL of this page
https://metraipolis.com/documento/dc_c818fc098a3c

How to cite

XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages — arXiv:2611.01370 — arXiv (en), https://arxiv.org/abs/2611.01370 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c818fc098a3c (Metraipolis archive dc_c818fc098a3c, sha256 b8781313f090baa5bc83c644128584097d40ac0263c5854ccc3d10cf8f4e9447)
BibTeX
@misc{dcc818fc098a3c,
  title = {XDecideBench: Evaluating Typed Decision Models and Confidence Policies Across Languages — arXiv:2611.01370},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01370},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c818fc098a3c; sha256 b8781313f090baa5bc83c644128584097d40ac0263c5854ccc3d10cf8f4e9447},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01370. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Applications use Typed Decision Models to route and moderate text in many languages, and they often act only on decisions above a confidence threshold. We introduce XDecideBench, a benchmark of nine suites in 122 languages with 88,425 texts and 153,875 typed questions. All suites share one scoring contract, and we report aligned, locally labelled and native items separately. Model rankings change from suite to suite. The model that is best in English, chosen in hindsight, loses significantly in about a quarter of the suite-language cells, and on Burmese Belebele it falls far behind the best model. Three open models fine-tuned for typed decisions beat the Qwen models they were built from in all twelve development languages. When the Qwen models are run locally, a pre-registered held-out cohort confirms that the accuracy gain is larger in the four languages with the least web text than in the six with the most. In both sets of languages, however, fine-tuning fixes a similar share of the base model's errors, so the larger gain may reflect weaker base models. A confidence threshold set on English data to allow 5% errors exceeds that error rate on XNLI, even in German and French.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.01370 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:52.297Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).