Metraipolis › archive › dc_0b91d457ea09

Do Embodied VLMs Actually Use Their Scene-Graph Memory? — arXiv:2611.02056

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.02056
License (as archived)
CC0-1.0
Archived
2026-11-03T11:07:27.749Z
SHA-256 of the archived text
5fc3e2c1f1b16d18cf449e7f517ca9c38e8ef772a66d7d6eb955258cc79e4c40
Stable URL of this page
https://metraipolis.com/documento/dc_0b91d457ea09

How to cite

Do Embodied VLMs Actually Use Their Scene-Graph Memory? — arXiv:2611.02056 — arXiv (en), https://arxiv.org/abs/2611.02056 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0b91d457ea09 (Metraipolis archive dc_0b91d457ea09, sha256 5fc3e2c1f1b16d18cf449e7f517ca9c38e8ef772a66d7d6eb955258cc79e4c40)
BibTeX
@misc{dc0b91d457ea09,
  title = {Do Embodied VLMs Actually Use Their Scene-Graph Memory? — arXiv:2611.02056},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02056},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0b91d457ea09; sha256 5fc3e2c1f1b16d18cf449e7f517ca9c38e8ef772a66d7d6eb955258cc79e4c40},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02056. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Embodied agents that answer questions about their route need a memory of what they saw, and scene graphs are a natural form for it. A vision-language model (VLM) can receive such a graph as text or through a graph encoder, but accuracy alone cannot show whether the graph holds the episode's facts, reaches the model, or is actually relied on. In this diagnostic study, we introduce a graph-swap test for the last of these: each episode is given another episode's graph, and we count how many answers change. On an embodied question-answering benchmark with routes through two to seven rooms, we use it to compare ways of delivering scene-graph memory to a fine-tuned VLM. As text, the memory is used strongly, adding 30 points on route questions. As a graph, it is not, and we trace why: the encoder preserves the information, but standard pooled and late fusion discard it. Giving each node its own input token recovers part of it, a gain of 8.3 points on route questions that the swap test confirms comes from the graph. The test also exposes a shortcut that accuracy hides: when the same facts appear in the text, the model ignores its graph. The test thus offers a cheap check of whether a VLM uses its scene-graph memory, and our analysis points to how that memory should be delivered.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO) ; Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.02056 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:59:46.072Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).