Metraipolis › archive › dc_cbc2895b97d6

When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis — arXiv:2609.15309

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15309
License (as archived)
CC0-1.0
Archived
2026-10-15T23:16:03.383Z
SHA-256 of the archived text
25961b46e1b024671b9749f3a5ce5cdd3272a8f9b51d772fb554520425094d4d
Stable URL of this page
https://metraipolis.com/documento/dc_cbc2895b97d6

How to cite

When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis — arXiv:2609.15309 — arXiv (en), https://arxiv.org/abs/2609.15309 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cbc2895b97d6 (Metraipolis archive dc_cbc2895b97d6, sha256 25961b46e1b024671b9749f3a5ce5cdd3272a8f9b51d772fb554520425094d4d)
BibTeX
@misc{dccbc2895b97d6,
  title = {When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis — arXiv:2609.15309},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15309},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cbc2895b97d6; sha256 25961b46e1b024671b9749f3a5ce5cdd3272a8f9b51d772fb554520425094d4d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15309. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for intermediate submissions, making progress observable throughout long trajectories. We propose Elo-per-token analysis, which tracks the best solution found at each token budget and uses a Bradley-Terry model to aggregate within-task orderings into Elo ratings across tasks with different score scales. We apply it to four general-purpose agents on four open-ended benchmarks, with sessions of up to 100M tokens, and to three feedback-driven LLM optimization harnesses in controlled single-task interventions. Independent sampling provides a theoretically characterized reference, for which Elo grows linearly with log compute. Against this reference, agents can initially convert tokens into Elo faster than independent sampling, but their marginal gains diminish and eventually fall below the reference. In contrast, the strongest historical human contestants improve superlinearly over contest time on shared AtCoder Heuristic Contest tasks, providing evidence of continual learning and substantial headroom after agents slow down. We define the scaling inflection point as the per-session budget where marginal Elo gains match the independent-sampling reference. Using this point as the per-session budget, we split 100M tokens across parallel sessions on FrontierCS Polyomino Packing, gaining +264 Elo over one long session and +355 over ten short sessions.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.15309 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:51:09.942Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).