Metraipolis › archive › dc_b8929f1871ff
Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories — arXiv:2610.01491
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.01491
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T13:55:38.529Z
- SHA-256 of the archived text
240c170efb2272c5fe3f46ab48978a29b305a2fe4be7d0569538ad087d6f8c3f- Stable URL of this page
https://metraipolis.com/documento/dc_b8929f1871ff
How to cite
Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories — arXiv:2610.01491 — arXiv (en), https://arxiv.org/abs/2610.01491 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b8929f1871ff (Metraipolis archive dc_b8929f1871ff, sha256 240c170efb2272c5fe3f46ab48978a29b305a2fe4be7d0569538ad087d6f8c3f)
BibTeX
@misc{dcb8929f1871ff,
title = {Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories — arXiv:2610.01491},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.01491},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b8929f1871ff; sha256 240c170efb2272c5fe3f46ab48978a29b305a2fe4be7d0569538ad087d6f8c3f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01491. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Web agents are an important application of large language models, yet their evaluation often depends on rule based or language model evaluators that inspect only the final outcome. Human verification of task completion and detailed analysis of failed trajectories remain limited. We audit all 165 WebArena Lite tasks under six evaluation conditions built from GPT 5.5 and an untrained Qwen3.5 9B model. The audit retains the original score, corrects false negatives from the automatic evaluator, identifies the first consequential error, and examines progress across the trajectory. We also study a Memory and Analysis Support Mechanism (MASM), which maintains explicit execution state, and Guide Text, which provides task relevant procedural guidance. Across four GPT 5.5 settings, human review recovers 5.45 to 8.49 percentage points of success missed by the evaluator. With a 25 step budget, Guide Text raises corrected success with MASM from 34.55% to 38.18%. On the untrained Qwen3.5 9B model, MASM raises the evaluator score from 13.90% to 18.80%. Review of 102 failed GPT 5.5 trajectories reveals frequent scrolling loops, unfinished exploration, premature answers, invalid actions, and incomplete form workflows. Step level evidence further shows that substantial early progress can coexist with a final failure. These results show why final scores alone provide an incomplete account of web agent behavior and motivate human grounded, trajectory aware verification. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.01491 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T12:15:34.722Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).