Metraipolis › archive › dc_5178da2bc819

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning — arXiv:2610.30724

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30724
License (as archived)
CC0-1.0
Archived
2026-10-23T15:15:05.867Z
SHA-256 of the archived text
f9fe665ab6a1960719bd720f15b207c4c71a4547ad0cb0223ef55ad69d837f4a
Stable URL of this page
https://metraipolis.com/documento/dc_5178da2bc819

How to cite

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning — arXiv:2610.30724 — arXiv (en), https://arxiv.org/abs/2610.30724 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5178da2bc819 (Metraipolis archive dc_5178da2bc819, sha256 f9fe665ab6a1960719bd720f15b207c4c71a4547ad0cb0223ef55ad69d837f4a)
BibTeX
@misc{dc5178da2bc819,
  title = {Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning — arXiv:2610.30724},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30724},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5178da2bc819; sha256 f9fe665ab6a1960719bd720f15b207c4c71a4547ad0cb0223ef55ad69d837f4a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30724. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.30724 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:09:08.551Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).