Metraipolis › archive › dc_4ebf401c14cf

What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis — arXiv:2611.05181

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05181
License (as archived)
CC0-1.0
Archived
2026-11-04T12:33:43.460Z
SHA-256 of the archived text
51f376b2adcbe3d9bcba7d6d8c86b53853d50c6c8c502636bf8624aaf101aeca
Stable URL of this page
https://metraipolis.com/documento/dc_4ebf401c14cf

How to cite

What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis — arXiv:2611.05181 — arXiv (en), https://arxiv.org/abs/2611.05181 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4ebf401c14cf (Metraipolis archive dc_4ebf401c14cf, sha256 51f376b2adcbe3d9bcba7d6d8c86b53853d50c6c8c502636bf8624aaf101aeca)
BibTeX
@misc{dc4ebf401c14cf,
  title = {What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis — arXiv:2611.05181},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05181},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4ebf401c14cf; sha256 51f376b2adcbe3d9bcba7d6d8c86b53853d50c6c8c502636bf8624aaf101aeca},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05181. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation validity to examine how text, audio, video, and combined inputs affect measurement and subsequent analysis. We compare input representations across six open-weight MLLMs and three human-annotated video datasets. No representation consistently performs best. On seven news-related constructs, native-video performance varies little across models (macro-F1 = .70-.77), while extracted-text performance ranges from .52 to .80. Three models perform better with text and three with video. Combining inputs rarely improves performance. Higher classification accuracy also does not reliably produce estimates closer to those based on human codes. The highest-F1 representation best reproduces human-coded regression coefficients in 17 of 42 comparisons. Even when each model uses its highest-F1 deployable representation, 45% of sample-prevalence estimates differ from the human-coded benchmark by more than ten percentage points. Input representation therefore requires validation for both the construct being measured and the quantity estimated from the resulting labels.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.05181 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:27:02.075Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).