Metraipolis › archive › dc_14828cd99089
Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score — arXiv:2610.03825
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.03825
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:27:57.562Z
- SHA-256 of the archived text
a10012b83dde18aafab92fc7a7b50487e46ca1692247d939ddc6f46fce1b34b3- Stable URL of this page
https://metraipolis.com/documento/dc_14828cd99089
How to cite
Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score — arXiv:2610.03825 — arXiv, https://arxiv.org/abs/2610.03825 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_14828cd99089 (Metraipolis archive dc_14828cd99089, sha256 a10012b83dde18aafab92fc7a7b50487e46ca1692247d939ddc6f46fce1b34b3)
BibTeX
@misc{dc14828cd99089,
title = {Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score — arXiv:2610.03825},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.03825},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_14828cd99089; sha256 a10012b83dde18aafab92fc7a7b50487e46ca1692247d939ddc6f46fce1b34b3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03825. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the second. The retained annotation record of a six-language benchmark for personally identifiable information contains two independent annotator labellings, the aggregate shipped as gold, and a reviewer gold from independent expert re-annotation of a sample. Using it, we hold the scored output fixed and exchange the reference. The score moves by 4.95 F1 points for one output and 2.00 for the other (95% CIs [3.23, 6.27] and [0.26, 3.37]), and by 7.55 in the worst language. The comparison between two outputs moves as well: the paired interaction between reference and system is +2.95 points (CI [+1.97, +3.87]), survives correction for multiple testing, and changes one language's margin outright. The cause is an undocumented aggregation default that usually kept one annotator when adjudication did not fire, making the shipped reference a partial copy of an output being scored. We report the resulting reference-sensitivity band, show how to compute one from any retained annotation record, and argue that the quantity belongs beside the score. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.03825 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:20.236Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).