Metraipolis › archive › dc_bf722df6a075

BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents — arXiv:2610.41076

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41076
License (as archived)
CC0-1.0
Archived
2026-11-02T04:42:41.620Z
SHA-256 of the archived text
56e67f2d1ac870b68347ef8972ea239828559000813fe848604d375a37497449
Stable URL of this page
https://metraipolis.com/documento/dc_bf722df6a075

How to cite

BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents — arXiv:2610.41076 — arXiv (en), https://arxiv.org/abs/2610.41076 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf722df6a075 (Metraipolis archive dc_bf722df6a075, sha256 56e67f2d1ac870b68347ef8972ea239828559000813fe848604d375a37497449)
BibTeX
@misc{dcbf722df6a075,
  title = {BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents — arXiv:2610.41076},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41076},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf722df6a075; sha256 56e67f2d1ac870b68347ef8972ea239828559000813fe848604d375a37497449},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41076. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An LLM agent that helps a patient get care must do two things. It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records. We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed. In each world, the agent must find hidden information and coordinate insurers, clinics, suppliers, pharmacies, and community organizations. The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited. We evaluated 14 frontier models with three runs on each world, for a total of 1,974 runs. Agents often reached the correct outcome by an invalid route. Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks. When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41076 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T04:42:12.339Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).