Metraipolis › archive › dc_efa1fa4a5b94

Content Based Video Narration of Gameplay with Vision Language Models — arXiv:2608.14016

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14016
License (as archived)
CC0-1.0
Archived
2026-10-19T02:21:15.224Z
SHA-256 of the archived text
034fc8127dd2db013f4533e58397ec98f43ec13b6d896cfa75270d574c4e5b30
Stable URL of this page
https://metraipolis.com/documento/dc_efa1fa4a5b94

How to cite

Content Based Video Narration of Gameplay with Vision Language Models — arXiv:2608.14016 — arXiv (en), https://arxiv.org/abs/2608.14016 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_efa1fa4a5b94 (Metraipolis archive dc_efa1fa4a5b94, sha256 034fc8127dd2db013f4533e58397ec98f43ec13b6d896cfa75270d574c4e5b30)
BibTeX
@misc{dcefa1fa4a5b94,
  title = {Content Based Video Narration of Gameplay with Vision Language Models — arXiv:2608.14016},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14016},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_efa1fa4a5b94; sha256 034fc8127dd2db013f4533e58397ec98f43ec13b6d896cfa75270d574c4e5b30},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14016. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Live game commentary is scarce: it exists for professional esports broadcasts and almost nowhere else. We present a content-based video narration system that produces spoken, esports-style commentary for arbitrary gameplay recordings using a general-purpose vision-language model (VLM) and a text-to-speech back end, with no game-specific instrumentation, no engine telemetry, and no task-specific training. Three mechanisms carry the system. Temporal mosaic packing arranges nine uniformly sampled frames into a single 3x3 image, letting an image-native VLM reason about motion while consuming one image payload per segment instead of nine. Context-conditioned prompting replays the K most recent narrations as assistant-role history, suppressing the repetition that dominates per-segment captioning of static scenes. Duration-conditioned generation and elastic alignment constrain narration length in the prompt, then time-scale or symmetrically pad the synthesized audio so each utterance fills its segment slot exactly, giving frame-accurate muxing without a forced aligner. The implementation supports either cloud TTS or a 6-bit quantized 4B-parameter on-device TTS model on Apple silicon, making the speech stage fully local. We report a qualitative case study on real-time strategy footage, a cost model showing the mosaic reduces per-minute image payloads by 9x, and a candid account of observed failure modes - hallucinated game state, resolution loss from mosaicking, and prosody artifacts from time-scaling. We release the system as a reproducible baseline, with an evaluation protocol for the quantitative study a full version will report.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Graphics (cs.GR).
Pagina del lavoro: https://arxiv.org/abs/2608.14016 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.892Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).