Metraipolis › archive › dc_771c82297276

Benchmarking LLM Judges for Mobile Agent Evaluation — arXiv:2608.11434

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11434
License (as archived)
CC0-1.0
Archived
2026-10-19T05:15:22.044Z
SHA-256 of the archived text
dfffcae204097df053793e271b143283cd6556732d0ec67f15c8b3dd79d2c361
Stable URL of this page
https://metraipolis.com/documento/dc_771c82297276

How to cite

Benchmarking LLM Judges for Mobile Agent Evaluation — arXiv:2608.11434 — arXiv (en), https://arxiv.org/abs/2608.11434 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_771c82297276 (Metraipolis archive dc_771c82297276, sha256 dfffcae204097df053793e271b143283cd6556732d0ec67f15c8b3dd79d2c361)
BibTeX
@misc{dc771c82297276,
  title = {Benchmarking LLM Judges for Mobile Agent Evaluation — arXiv:2608.11434},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11434},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_771c82297276; sha256 dfffcae204097df053793e271b143283cd6556732d0ec67f15c8b3dd79d2c361},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11434. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined. We introduce MobileJudgeBench, a benchmark for systematically evaluating LLM-as-judge methods on mobile agent trajectories. Our benchmark comprises 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps. We evaluate 6 judge methods (five adapted from SPA-Bench, A3 with two modes, AndroidArena, and AgentRewardBench, plus a simple baseline we design) across multiple LLM backends. Our experiments reveal three key findings. First, a simple baseline judge with sampled screenshots is competitive with, and often exceeds, purpose-built methods, indicating that more elaborate judge pipelines do not consistently improve judge quality; among competitive methods, the LLM backbone is the primary driver. Second, benchmark quality metrics reliably predict real-world judge utility: they correlate with both agent ranking fidelity for evaluation and downstream performance when judges serve as reward signals for on-policy reinforcement learning. Third, failure analysis across two LLM backends uncovers qualitatively opposite failure profiles, one conservative and the other permissive, linked to the backbone's precision-recall characteristics.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.11434 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:08:46.625Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).