Metraipolis › archive › dc_7bd05eb9fea2

Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams — arXiv:2611.08998

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.08998
License (as archived): CC0-1.0
Archived: 2026-11-09T22:34:43.319Z
SHA-256 of the archived text: a7e55ecaeec4ff67eb61c1fdb1f353d656a38a9549b4eb47331fb9a40d82ddba
Stable URL of this page: https://metraipolis.com/documento/dc_7bd05eb9fea2

How to cite

Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams — arXiv:2611.08998 — arXiv (en), https://arxiv.org/abs/2611.08998 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7bd05eb9fea2 (Metraipolis archive dc_7bd05eb9fea2, sha256 a7e55ecaeec4ff67eb61c1fdb1f353d656a38a9549b4eb47331fb9a40d82ddba)

BibTeX

@misc{dc7bd05eb9fea2,
  title = {Seeing vs. Counting: Complexity Twins Show Where Multimodal LLMs Fail on Diagrams — arXiv:2611.08998},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08998},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7bd05eb9fea2; sha256 a7e55ecaeec4ff67eb61c1fdb1f353d656a38a9549b4eb47331fb9a40d82ddba},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08998. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a multimodal LLM miscounts in a diagram, does it fail at seeing or at counting? Existing visual graph and puzzle benchmarks vary visual density and algorithmic hardness together, confounding the two. We pair counting questions about the same drawing that sit in different counting-complexity classes, such as its spanning trees (a determinant) and its simple paths (#P-complete), and use a transcription probe and a text-only control to separate reading from computing. We test Gemini, Gemma and Qwen3-VL models; for the Gemini models, the answer depends on how the model computes. In chain of thought, most of them read small drawings correctly and then fail at counting, at a rate set far more by the size of the answer than by its complexity class (for the primary model, the odds of an exact answer fall 2.38-fold per decade of the answer). Given code and an exact transcription, the primary model counts correctly even on 10x10 grids, and reading becomes the bottleneck: a single view misses horizontal edges, and a second reading of a rotated copy with crop checks reads 86/96 large grids exactly, against 57/96 when the upright reading is simply repeated.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.08998 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.313Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).