Metraipolis › archive › dc_d9a7fc8a2bec
Can Video World Models Track Unobserved World States? — arXiv:2608.30692
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.30692
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T09:59:37.285Z
- SHA-256 of the archived text
0c2cf99f1a02f684a351d2501acf4b89a34bf7e96e07639756c8782ac7bc5b7c- Stable URL of this page
https://metraipolis.com/documento/dc_d9a7fc8a2bec
How to cite
Can Video World Models Track Unobserved World States? — arXiv:2608.30692 — arXiv (en), https://arxiv.org/abs/2608.30692 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d9a7fc8a2bec (Metraipolis archive dc_d9a7fc8a2bec, sha256 0c2cf99f1a02f684a351d2501acf4b89a34bf7e96e07639756c8782ac7bc5b7c)
BibTeX
@misc{dcd9a7fc8a2bec,
title = {Can Video World Models Track Unobserved World States? — arXiv:2608.30692},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.30692},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d9a7fc8a2bec; sha256 0c2cf99f1a02f684a351d2501acf4b89a34bf7e96e07639756c8782ac7bc5b7c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30692. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Video world models are increasingly used as simulators, but visual fidelity alone does not show that a model maintains the hidden state of the world. We examine this difference with an action-conditioned video Shell Game, a visual analogue of $S_5$ state tracking that separates visual rendering from compositing the unobserved world state. Trained on 5-swap chains, standard backbones (e.g., bidirectional and autoregressive Transformers, Mamba, and linear attention) render plausible videos and predict the correct ball location up to 5 swaps. However, they fail to learn the rule and generalize to longer swap chains, even with more denoising steps. As the pixel-based diffusion loss does not force the generated frames to hold the unseen ball position, output tokens cannot carry it, and the state has to live within the architecture. In a causal Transformer, this implicit state is an append-only KV cache, which is written once and never revised, so the model must re-compose the swaps at every chunk. Tracking $S_5$ this way requires depth to grow with sequence length, which no fixed-depth Transformer provides. We study what enables learning the rule, and find that length generalization requires a revisable state carried across chunks and an update expressive enough to apply a swap. Linear attention can achieve this by allowing negative transition eigenvalues, and autoregressive Transformers can do so with nonlinear TTT fast weights (e.g., SwiGLU) whose online updates change the feature map used to read their state. We further examine Memory Maze and Block World, where the state is not fixed by the input action stream alone and must be corrected from observations or keeps changing out of view, and discuss the implications for building stateful video world models. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CV). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2608.30692 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T09:28:48.171Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).