Metraipolis › archive › dc_9253fed45811

Seq@K: Evaluating Language Models Under Sequential Feedback — arXiv:2610.37859

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.37859
License (as archived)
CC0-1.0
Archived
2026-10-31T06:07:28.949Z
SHA-256 of the archived text
625c78a45d22046c65749f4a02b2f82a20f0d3c2505cf7e85e3165cdf4c8498e
Stable URL of this page
https://metraipolis.com/documento/dc_9253fed45811

How to cite

Seq@K: Evaluating Language Models Under Sequential Feedback — arXiv:2610.37859 — arXiv (en), https://arxiv.org/abs/2610.37859 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9253fed45811 (Metraipolis archive dc_9253fed45811, sha256 625c78a45d22046c65749f4a02b2f82a20f0d3c2505cf7e85e3165cdf4c8498e)
BibTeX
@misc{dc9253fed45811,
  title = {Seq@K: Evaluating Language Models Under Sequential Feedback — arXiv:2610.37859},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.37859},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9253fed45811; sha256 625c78a45d22046c65749f4a02b2f82a20f0d3c2505cf7e85e3165cdf4c8498e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.37859. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that analyzing model trajectories under Seq@K helps characterize iterative refinement behavior, such as the rate at which models address issues raised in feedback, or how rigidly they hold onto their own answers. Finally, we study how the nature of feedback affects models' iterative refinement, showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems. Together, these results establish how Seq@K offers a more faithful metric for intelligent systems that typically operate in such feedback loop, and the necessity to consider Seq@K in evaluating models for these systems.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.37859 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.430Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).