Metraipolis › archive › dc_e33effcf082c

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — arXiv:2611.03377

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.03377
License (as archived): CC0-1.0
Archived: 2026-11-03T18:15:40.543Z
SHA-256 of the archived text: 479c9e50ac541acc6a0ef99c67e816449b5eba4cb12cb45a112bef68b5176454
Stable URL of this page: https://metraipolis.com/documento/dc_e33effcf082c

How to cite

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — arXiv:2611.03377 — arXiv (en), https://arxiv.org/abs/2611.03377 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e33effcf082c (Metraipolis archive dc_e33effcf082c, sha256 479c9e50ac541acc6a0ef99c67e816449b5eba4cb12cb45a112bef68b5176454)

BibTeX

@misc{dce33effcf082c,
  title = {Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks — arXiv:2611.03377},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03377},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e33effcf082c; sha256 479c9e50ac541acc6a0ef99c67e816449b5eba4cb12cb45a112bef68b5176454},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03377. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.03377 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.455Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).