Metraipolis › archive › dc_f70daf1de017

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation — arXiv:2608.04902

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04902
License (as archived)
CC0-1.0
Archived
2026-10-31T11:35:46.050Z
SHA-256 of the archived text
7760a1a3cc61976cab4a698f9c7b2e72634e8d6ffa15f09eea30d8fe31b7f737
Stable URL of this page
https://metraipolis.com/documento/dc_f70daf1de017

How to cite

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation — arXiv:2608.04902 — arXiv (en), https://arxiv.org/abs/2608.04902 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f70daf1de017 (Metraipolis archive dc_f70daf1de017, sha256 7760a1a3cc61976cab4a698f9c7b2e72634e8d6ffa15f09eea30d8fe31b7f737)
BibTeX
@misc{dcf70daf1de017,
  title = {Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation — arXiv:2608.04902},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04902},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f70daf1de017; sha256 7760a1a3cc61976cab4a698f9c7b2e72634e8d6ffa15f09eea30d8fe31b7f737},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04902. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.LG).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG); Sound (cs.SD).
Pagina del lavoro: https://arxiv.org/abs/2608.04902 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:24:48.842Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).