Metraipolis › archive › dc_4c072cf88bb4
RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning — arXiv:2610.09455
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09455
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T12:47:20.914Z
- SHA-256 of the archived text
021c24588e3d2cc9e180819dc3ccdf598a1a73cda4b6547c7cf8d44149d0833e- Stable URL of this page
https://metraipolis.com/documento/dc_4c072cf88bb4
How to cite
RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning — arXiv:2610.09455 — arXiv (en), https://arxiv.org/abs/2610.09455 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4c072cf88bb4 (Metraipolis archive dc_4c072cf88bb4, sha256 021c24588e3d2cc9e180819dc3ccdf598a1a73cda4b6547c7cf8d44149d0833e)
BibTeX
@misc{dc4c072cf88bb4,
title = {RLHND: Video Foundation Models as Physically Grounded Hand Trackers for Robot Learning — arXiv:2610.09455},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09455},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4c072cf88bb4; sha256 021c24588e3d2cc9e180819dc3ccdf598a1a73cda4b6547c7cf8d44149d0833e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09455. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recently, approaches that leverage human video datasets for robot policy training have become increasingly prevalent. However, most existing hand trackers regress pose from cropped frames with limited priors on hand motion and object interaction, resulting in inaccurate and physically inconsistent estimates. Moreover, the lack of physical cues, e.g., contact and force, limits the use of human videos for robot policy training. To this end, we propose RLHND, a video foundation model-based hand tracking model that jointly estimates hand pose and realistic tactile information from monocular egocentric videos. RLHND turns the pre-trained Cosmos 3 video diffusion backbone into a deterministic clip-level feature extractor via clean-latent conditioning, carrying its learned priors on hand motion and hand-object interaction into tracking. For pose estimation, RLHND (i) predicts hand poses with anatomically plausible joint angles and (ii) enables optional conditioning on the shape parameter to maintain consistent hand shape within the same video and even across videos recorded by the same actor. For tactile estimation, a separate tactile expert stream, trained with the pose stream frozen, predicts dense contact and force over the hand surface. We further adopt LBS-based feature spreading to enable vertex-wise feature extraction without costly per-vertex attention. RLHND achieves state-of-the-art performance across various benchmark datasets for pose estimation, while also achieving state-of-the-art performance in contact and force estimation. Moreover, we demonstrate the utility of RLHND for robot learning through retargeting results and real-world robot experiments. The code will be publicly available at this https URL . Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Robotics (cs.RO). Pagina del lavoro: https://arxiv.org/abs/2610.09455 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T12:18:15.370Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).