Metraipolis › archive › dc_22db67d4f34d

MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents — arXiv:2608.31022

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.31022
License (as archived)
CC0-1.0
Archived
2026-10-17T07:46:57.441Z
SHA-256 of the archived text
d7753c3cd65db066ee2a7d61f27dd3333e1cd8e4932c3370ff2cc205782b35d9
Stable URL of this page
https://metraipolis.com/documento/dc_22db67d4f34d

How to cite

MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents — arXiv:2608.31022 — arXiv (en), https://arxiv.org/abs/2608.31022 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_22db67d4f34d (Metraipolis archive dc_22db67d4f34d, sha256 d7753c3cd65db066ee2a7d61f27dd3333e1cd8e4932c3370ff2cc205782b35d9)
BibTeX
@misc{dc22db67d4f34d,
  title = {MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents — arXiv:2608.31022},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.31022},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_22db67d4f34d; sha256 d7753c3cd65db066ee2a7d61f27dd3333e1cd8e4932c3370ff2cc205782b35d9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.31022. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.31022 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.635Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).