Metraipolis › archive › dc_390998623b3d
Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation — arXiv:2610.38511
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.38511
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T06:17:16.305Z
- SHA-256 of the archived text
2cdcd354c1324c5f2d36c0f7441edc93946524cbde36492be92870ec3bea689c- Stable URL of this page
https://metraipolis.com/documento/dc_390998623b3d
How to cite
Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation — arXiv:2610.38511 — arXiv (en), https://arxiv.org/abs/2610.38511 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_390998623b3d (Metraipolis archive dc_390998623b3d, sha256 2cdcd354c1324c5f2d36c0f7441edc93946524cbde36492be92870ec3bea689c)
BibTeX
@misc{dc390998623b3d,
title = {Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation — arXiv:2610.38511},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.38511},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_390998623b3d; sha256 2cdcd354c1324c5f2d36c0f7441edc93946524cbde36492be92870ec3bea689c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38511. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions. We test whether those metrics recover human judgement on dense, multi-window video description, and find that they do not. Six frontier models captioned 196 short clips. Trained annotators post-edited every caption against a failure taxonomy fixed before annotation, once as a whole-clip overview and once window by window, and a quality-control reviewer checked each annotation, yielding 1,144 overview verdicts and 4,672 window verdicts. Reference-overlap metrics do not separate the captions reviewers accepted from those they rejected: BLEU-4 reaches an AUC of 0.510 and a CIDEr-style tf-idf cosine 0.528, and restricting references to reviewer corrections leaves BLEU-4 at 0.479 (95% CI 0.431-0.529), indistinguishable from chance. Review granularity changes what is detected. The same annotators record camera and shot errors on 2.7% of captions when reading the overview and on 12.8% when reading window by window, a 4.7x gap that holds across every robustness check we ran, on the mode most likely to need a near-total rewrite. Correct-overview rates span 54.1-72.7%, but only 4 of 15 pairwise gaps survive Bonferroni correction (5 after adjusting for reviewer and clip effects), and accuracy and derived severity order the models differently. Models with statistically indistinguishable accuracy have sharply different failure profiles, which no single scalar shows. We argue that caption evaluation should record what was wrong rather than only how often, and we report the taxonomy, the per-model profiles, the corpus properties that predict difficulty, and the code that regenerates every number in the paper. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.38511 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T06:07:23.393Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).