Metraipolis › archive › dc_c19b45fa5196

BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents — arXiv:2610.41076

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41076
License (as archived): CC0-1.0
Archived: 2026-11-03T07:10:41.658Z
SHA-256 of the archived text: 6640244d07b3e49ac6f49e0a0761dba9803afde0abaa17837848a978ba4ba2ff
Stable URL of this page: https://metraipolis.com/documento/dc_c19b45fa5196

How to cite

BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents — arXiv:2610.41076 — arXiv (en), https://arxiv.org/abs/2610.41076 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c19b45fa5196 (Metraipolis archive dc_c19b45fa5196, sha256 6640244d07b3e49ac6f49e0a0761dba9803afde0abaa17837848a978ba4ba2ff)

BibTeX

@misc{dcc19b45fa5196,
  title = {BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents — arXiv:2610.41076},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41076},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c19b45fa5196; sha256 6640244d07b3e49ac6f49e0a0761dba9803afde0abaa17837848a978ba4ba2ff},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41076. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An LLM agent that helps a patient get care must do two things. It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records. We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed. In each world, the agent must find hidden information and coordinate insurers, clinics, suppliers, pharmacies, and community organizations. The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited. We evaluated 14 frontier models with three runs on each world, for a total of 1,974 runs. Agents often reached the correct outcome by an invalid route. Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks. When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41076 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:09:31.733Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).