Metraipolis › archive › dc_d13a473fc70d

CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks — arXiv:2608.18554

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.18554
License (as archived)
CC0-1.0
Archived
2026-10-18T13:43:18.788Z
SHA-256 of the archived text
56c3300532d9cc6fa064f10ceb7d3975818bc5787deb63bb612ac4d66c1cddeb
Stable URL of this page
https://metraipolis.com/documento/dc_d13a473fc70d

How to cite

CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks — arXiv:2608.18554 — arXiv (en), https://arxiv.org/abs/2608.18554 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d13a473fc70d (Metraipolis archive dc_d13a473fc70d, sha256 56c3300532d9cc6fa064f10ceb7d3975818bc5787deb63bb612ac4d66c1cddeb)
BibTeX
@misc{dcd13a473fc70d,
  title = {CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks — arXiv:2608.18554},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.18554},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d13a473fc70d; sha256 56c3300532d9cc6fa064f10ceb7d3975818bc5787deb63bb612ac4d66c1cddeb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.18554. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.MA).
Categorie arXiv: Computers and Society (cs.CY) ; Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); General Economics (econ.GN).
Pagina del lavoro: https://arxiv.org/abs/2608.18554 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:40:43.017Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).