Metraipolis › archive › dc_c2fae317ccc6
CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time — arXiv:2609.14239
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.14239
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T00:10:38.608Z
- SHA-256 of the archived text
f0b8f7cd327cde8bd76c6f9eb30e73b7c338700d3417aa4bc8592b747da41a7b- Stable URL of this page
https://metraipolis.com/documento/dc_c2fae317ccc6
How to cite
CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time — arXiv:2609.14239 — arXiv (en), https://arxiv.org/abs/2609.14239 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c2fae317ccc6 (Metraipolis archive dc_c2fae317ccc6, sha256 f0b8f7cd327cde8bd76c6f9eb30e73b7c338700d3417aa4bc8592b747da41a7b)
BibTeX
@misc{dcc2fae317ccc6,
title = {CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time — arXiv:2609.14239},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.14239},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c2fae317ccc6; sha256 f0b8f7cd327cde8bd76c6f9eb30e73b7c338700d3417aa4bc8592b747da41a7b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14239. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Static benchmarks for computer-use agents fix a task set at release and score every system against it once. That makes them reproducible, and it lets them drift from what they should measure: a fixed task set ages, leaks into training corpora, and cannot follow how people actually use agents from week to week. CoArena measures use directly. Real users submit tasks; two systems, each a single model or a multi-agent pipeline behind the same tool interface, execute the same task concurrently in identical sandboxed desktops; users judge the two outcomes without knowing which system produced them; and a public leaderboard is refit from those judgments. The central contribution is a formal account of what makes such an evaluation real-time. We define real-time as five measurable properties, each with an equation and a worked example: continuous task arrival, live concurrent execution, online rating updates, freshness with contamination resistance, and bounded feedback latency from a failed run to a reusable training environment. The rating methodology follows in full: the Bradley-Terry pairwise model, its likelihood with weighted observations and ties, the penalized maximum-likelihood estimator, and the streaming update applied when a single vote arrives (a stochastic-gradient step on the same likelihood, recovering Elo). It gives confidence intervals from the observed information and a cluster-robust sandwich, rank bands from a parametric bootstrap, the rule by which a new system enters the board, and the convergence rate of the estimate. Vote quality is treated with inter-judge agreement statistics, redundant judging, and explicit handling of ties and abstentions. A five-system example with 211 votes is carried from the vote matrix to ratings, intervals, and rank bands. Every number is derived from stated inputs or labeled illustrative; none is a measurement of a deployed system. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.14239 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T23:54:14.683Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).