Metraipolis › archive › dc_771c82297276
Benchmarking LLM Judges for Mobile Agent Evaluation — arXiv:2608.11434
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.11434
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T05:15:22.044Z
- SHA-256 of the archived text
dfffcae204097df053793e271b143283cd6556732d0ec67f15c8b3dd79d2c361- Stable URL of this page
https://metraipolis.com/documento/dc_771c82297276
How to cite
Benchmarking LLM Judges for Mobile Agent Evaluation — arXiv:2608.11434 — arXiv (en), https://arxiv.org/abs/2608.11434 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_771c82297276 (Metraipolis archive dc_771c82297276, sha256 dfffcae204097df053793e271b143283cd6556732d0ec67f15c8b3dd79d2c361)
BibTeX
@misc{dc771c82297276,
title = {Benchmarking LLM Judges for Mobile Agent Evaluation — arXiv:2608.11434},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.11434},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_771c82297276; sha256 dfffcae204097df053793e271b143283cd6556732d0ec67f15c8b3dd79d2c361},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11434. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined. We introduce MobileJudgeBench, a benchmark for systematically evaluating LLM-as-judge methods on mobile agent trajectories. Our benchmark comprises 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps. We evaluate 6 judge methods (five adapted from SPA-Bench, A3 with two modes, AndroidArena, and AgentRewardBench, plus a simple baseline we design) across multiple LLM backends. Our experiments reveal three key findings. First, a simple baseline judge with sampled screenshots is competitive with, and often exceeds, purpose-built methods, indicating that more elaborate judge pipelines do not consistently improve judge quality; among competitive methods, the LLM backbone is the primary driver. Second, benchmark quality metrics reliably predict real-world judge utility: they correlate with both agent ranking fidelity for evaluation and downstream performance when judges serve as reward signals for on-policy reinforcement learning. Third, failure analysis across two LLM backends uncovers qualitatively opposite failure profiles, one conservative and the other permissive, linked to the backbone's precision-recall characteristics. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2608.11434 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:08:46.625Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).