Metraipolis › archive › dc_ebd85ee720dc

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems — arXiv:2610.23372

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.23372
License (as archived): CC0-1.0
Archived: 2026-10-19T09:21:26.702Z
SHA-256 of the archived text: ec843893655fa4d39267ed261fe2a030288ff3c45d3ab45d7f363b9b5cbec7ea
Stable URL of this page: https://metraipolis.com/documento/dc_ebd85ee720dc

How to cite

Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems — arXiv:2610.23372 — arXiv (en), https://arxiv.org/abs/2610.23372 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ebd85ee720dc (Metraipolis archive dc_ebd85ee720dc, sha256 ec843893655fa4d39267ed261fe2a030288ff3c45d3ab45d7f363b9b5cbec7ea)

BibTeX

@misc{dcebd85ee720dc,
  title = {Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems — arXiv:2610.23372},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23372},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ebd85ee720dc; sha256 ec843893655fa4d39267ed261fe2a030288ff3c45d3ab45d7f363b9b5cbec7ea},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23372. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Robotics (cs.RO) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.23372 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:17:41.718Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).