Metraipolis › archive › dc_a22e56db2c8c

Pixels to Keys: Exploring Spatial and Motion Cues in Gameplay Inverse Dynamics — arXiv:2609.37907

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37907
License (as archived)
CC0-1.0
Archived
2026-10-07T16:24:16.995Z
SHA-256 of the archived text
a29f137dadccfa5f1bfdf55b94122def93dad9a9e918c35bbed45df82817c6c5
Stable URL of this page
https://metraipolis.com/documento/dc_a22e56db2c8c

How to cite

Pixels to Keys: Exploring Spatial and Motion Cues in Gameplay Inverse Dynamics — arXiv:2609.37907 — arXiv (en), https://arxiv.org/abs/2609.37907 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a22e56db2c8c (Metraipolis archive dc_a22e56db2c8c, sha256 a29f137dadccfa5f1bfdf55b94122def93dad9a9e918c35bbed45df82817c6c5)
BibTeX
@misc{dca22e56db2c8c,
  title = {Pixels to Keys: Exploring Spatial and Motion Cues in Gameplay Inverse Dynamics — arXiv:2609.37907},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37907},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a22e56db2c8c; sha256 a29f137dadccfa5f1bfdf55b94122def93dad9a9e918c35bbed45df82817c6c5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37907. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Video games offer scalable environments for studying perception and control in embodied agents. Abundant online gameplay videos could supply demonstrations, but they rarely include player inputs for training. Inverse Dynamics Models (IDMs) have thus been proposed to infer inputs from frames. Large (up to 1B parameters) IDMs trained on $\sim$1K-2K gameplay hours demonstrate feasibility and cross-environment generalization at this scale, but researchers do not clarify what the key components are to recover individual actions and often report only aggregate accuracy that can mask rare-action failures. We study the problem in a data-constrained scenario to evaluate how spatial motion features, model architectures, and training objectives affect an IDM's outcome and we analyse our models on per-key and balanced metrics such as $F_1^{macro}$. Our experiments on Trackmania highlight the importance of factors like the model architecture and motion flow extraction in preprocessing, while also showing the limits of evaluation through unbalanced metrics. The application of the same architecture and training recipe to Cyberpunk 2077 reveals uneven performance across game mechanics. Our per-action evaluation and failure analysis highlight ambiguities from camera motion, delayed effects and imbalanced key-press frequencies that call for explicit modeling of 3D scene structure, long-term state and the adoption of proper losses in future implementations.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.37907 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:53.669Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).