Metraipolis › archive › dc_35b257f554f7

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — arXiv:2611.03377

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03377
License (as archived)
CC0-1.0
Archived
2026-11-03T10:54:35.028Z
SHA-256 of the archived text
9da963347326a69861a15dc00f3e41f193c44b23701f9a9560b6b0fe6da9bc94
Stable URL of this page
https://metraipolis.com/documento/dc_35b257f554f7

How to cite

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — arXiv:2611.03377 — arXiv (en), https://arxiv.org/abs/2611.03377 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_35b257f554f7 (Metraipolis archive dc_35b257f554f7, sha256 9da963347326a69861a15dc00f3e41f193c44b23701f9a9560b6b0fe6da9bc94)
BibTeX
@misc{dc35b257f554f7,
  title = {Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — arXiv:2611.03377},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03377},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_35b257f554f7; sha256 9da963347326a69861a15dc00f3e41f193c44b23701f9a9560b6b0fe6da9bc94},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03377. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.03377 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.424Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).