Metraipolis › archive › dc_bb0bfb40173a

Coding-Agent Benchmarks Should Match Their Users' Task Flows — arXiv:2610.09633

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.09633
License (as archived): CC0-1.0
Archived: 2026-10-08T11:11:17.671Z
SHA-256 of the archived text: 5a0d8097f07b447122b5892a3259b5a689c1ff2a5ee9a10c30f402b2254b647f
Stable URL of this page: https://metraipolis.com/documento/dc_bb0bfb40173a

How to cite

Coding-Agent Benchmarks Should Match Their Users' Task Flows — arXiv:2610.09633 — arXiv (en), https://arxiv.org/abs/2610.09633 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bb0bfb40173a (Metraipolis archive dc_bb0bfb40173a, sha256 5a0d8097f07b447122b5892a3259b5a689c1ff2a5ee9a10c30f402b2254b647f)

BibTeX

@misc{dcbb0bfb40173a,
  title = {Coding-Agent Benchmarks Should Match Their Users' Task Flows — arXiv:2610.09633},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09633},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bb0bfb40173a; sha256 5a0d8097f07b447122b5892a3259b5a689c1ff2a5ee9a10c30f402b2254b647f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09633. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.09633 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:14:06.110Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).