Metraipolis › archive › dc_04528f4bedf5
Keyframes First, Motion Second: Revisiting Compressed-Domain Video Captioning under Limited Supervision — arXiv:2611.00048
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00048
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:52:54.796Z
- SHA-256 of the archived text
e421edb4f4dd9c136cb78fb529174e6bc941f3eb9222937774f976506a4cb6aa- Stable URL of this page
https://metraipolis.com/documento/dc_04528f4bedf5
How to cite
Keyframes First, Motion Second: Revisiting Compressed-Domain Video Captioning under Limited Supervision — arXiv:2611.00048 — arXiv (en), https://arxiv.org/abs/2611.00048 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_04528f4bedf5 (Metraipolis archive dc_04528f4bedf5, sha256 e421edb4f4dd9c136cb78fb529174e6bc941f3eb9222937774f976506a4cb6aa)
BibTeX
@misc{dc04528f4bedf5,
title = {Keyframes First, Motion Second: Revisiting Compressed-Domain Video Captioning under Limited Supervision — arXiv:2611.00048},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00048},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_04528f4bedf5; sha256 e421edb4f4dd9c136cb78fb529174e6bc941f3eb9222937774f976506a4cb6aa},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00048. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Compressed-domain video captioners such as CoCap read the H.264 bitstream directly: a pretrained encoder reads the keyframes (I-frames), while motion and residual encoders, trained from scratch, turn the codec's motion vectors and residuals into action tokens. This motion branch costs decoding time and supervision, both scarce on edge devices. We ask when it pays off under limited supervision. In a controlled study on 2,500 VATEX training clips, we fix a frozen SigLIP 2 keyframe encoder, a GPT-2 caption decoder and the training schedule, vary only the visual tokens the decoder receives (34 configurations), and report held-out test CIDEr with paired bootstrap intervals and three seeds. Motion encoders trained from scratch bring no measurable benefit: CoCap-style action tokens change CIDEr by -0.55 (95% CI [-2.3, +1.2]), and six other uses of motion do not help. Warm-started from a model trained on 48x the caption samples, they add +1.3 CIDEr on average, though not significantly across seeds, and only while the keyframe tokens stay frozen. Reading the keyframes better matters more: fine-tuning SigLIP 2's own attention-pooling head adds 5.4 CIDEr, and one to eight such tokens beat the 16 CoCap-style tokens on every seed while decoding only I-frames (7.3 ms instead of 44-114 ms per clip). The same holds on MSVD's full split in CoCap's own video encoding, where the readout leads by 14.2 CIDEr and action tokens bring no reliable gain, also with clean single-reference motion vectors. Under limited supervision, keyframes come first and motion second. Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2611.00048 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:16:52.839Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).