Metraipolis › archive › dc_285a0af37eca
Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model — arXiv:2608.29904
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29904
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T09:59:11.329Z
- SHA-256 of the archived text
effe0dd8184c177e9d2dca7f69b7e1a79faa4b330a4e518102ec6f7cb46aeba1- Stable URL of this page
https://metraipolis.com/documento/dc_285a0af37eca
How to cite
Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model — arXiv:2608.29904 — arXiv (en), https://arxiv.org/abs/2608.29904 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_285a0af37eca (Metraipolis archive dc_285a0af37eca, sha256 effe0dd8184c177e9d2dca7f69b7e1a79faa4b330a4e518102ec6f7cb46aeba1)
BibTeX
@misc{dc285a0af37eca,
title = {Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model — arXiv:2608.29904},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29904},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_285a0af37eca; sha256 effe0dd8184c177e9d2dca7f69b7e1a79faa4b330a4e518102ec6f7cb46aeba1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29904. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Modern video generators routinely fail at physical dynamics: objects float, trajectories violate gravity, contacts vanish. Standard denoising and flow-matching objectives fit visual data distributions but do not explicitly penalize such physical violations. Existing remedies can improve physical consistency, but typically add substantial inference or training cost. Candidate-selection methods generate and score multiple videos, while gradient-based world-model guidance repeatedly decodes and re-encodes intermediate estimates. Generator-internal refinement adds perturbation and re-denoising loops, whereas post-training requires curated data and additional optimization. We propose Off-Manifold Refinement (OMR), an inference-time method that instead injects world-model feedback directly into a single sampling trajectory. During scheduled middle ODE steps, we augment the generator velocity with the gradient of an adapter-space V-JEPA 2.1 surprise energy. This external correction can move the latent away from the uncorrected sampling trajectory and toward regions ranked as more physically plausible by the frozen predictor, after which the generator continues rendering from the corrected state. A small trained latent-to-embedding adapter keeps the gradient tractable at inference, and both the video generator and the world model remain frozen. On our fixed 400-prompt VideoPhy-2 detailed subset, OMR lifts the joint Semantic-Adherence-and-Physical-Commonsense metric from 47.0% to 52.0% (+5.0pp absolute, +10.6% relative) over the base Wan2.2-T2V-A14B sampler. On a separate fixed 50-prompt efficiency subset, it requires $1.71 \times$ the base runtime rather than the multiplicative cost of reward/search alternatives. Project page: this https URL . Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CV). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2608.29904 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T09:28:48.081Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).