Metraipolis › archive › dc_3e4331d197e2

Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations — arXiv:2609.23486

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.23486
License (as archived): CC0-1.0
Archived: 2026-10-09T22:06:35.280Z
SHA-256 of the archived text: 6927fad8de8ae8a8868a902030f6b0558ed47b7ff492e64dd210048b3225e3ab
Stable URL of this page: https://metraipolis.com/documento/dc_3e4331d197e2

How to cite

Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations — arXiv:2609.23486 — arXiv (en), https://arxiv.org/abs/2609.23486 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e4331d197e2 (Metraipolis archive dc_3e4331d197e2, sha256 6927fad8de8ae8a8868a902030f6b0558ed47b7ff492e64dd210048b3225e3ab)

BibTeX

@misc{dc3e4331d197e2,
  title = {Cognitive Action Reasoning for Proactive Robots from Human-Centered Multimodal Observations — arXiv:2609.23486},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23486},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e4331d197e2; sha256 6927fad8de8ae8a8868a902030f6b0558ed47b7ff492e64dd210048b3225e3ab},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23486. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Robots operating in human-centered environments are typically designed to execute explicit instructions, and most robot-learning datasets likewise pair observations with task instructions or low-level actions. Although recent work has begun to explore proactive embodied assistance, existing resources target different settings and action levels, leaving real-world human-centered multimodal decision-making underexplored. We formulate this problem as \textit{Proactive Robot Action Reasoning} (\textit{ProRobo}), an upstream cognitive decision problem in which a robot must determine which action to take based on multimodal human and environmental cues without explicit action instructions. To support ProRobo, we introduce \textit{ProAction}, a real-world multimodal dataset containing 10K samples of visual observations, audio signals, and text inputs across 12 daily-life scenarios in five common scenes. To construct cognitively grounded high-level action supervision, we develop a two-stage human-in-the-loop pipeline that combines appraisal-guided candidate generation with Affective Theory-of-Mind-guided human refinement, explicitly incorporating contextual judgment about human states, urgency, feasibility, and potential risk into action annotation. Based on this supervision, we benchmark representative Multimodal Large Language Models (MLLMs) and introduce \textit{MMC2Act}, a reference model that implicitly learns the mapping from multimodal observations to cognitively grounded high-level actions. Experiments across modality settings, subject-disjoint generalization, cross-dataset transfer, and human evaluation show that general-purpose MLLMs struggle with proactively reasoning high-level actions from multimodal cues, whereas training on \textit{ProAction} substantially improves performance.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CV).
Categorie arXiv: Robotics (cs.RO) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.23486 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T21:45:21.136Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).