Metraipolis › archive › dc_c866812f93c2

Measuring Cross-Task Behavioral Consistency in Language Model Agents — arXiv:2608.13598

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.13598
License (as archived)
CC0-1.0
Archived
2026-10-19T02:20:16.332Z
SHA-256 of the archived text
b41a01adbcbf6c4905fa9719c6ff77e93ba994ea754e743c7b0a4bf76c15ca22
Stable URL of this page
https://metraipolis.com/documento/dc_c866812f93c2

How to cite

Measuring Cross-Task Behavioral Consistency in Language Model Agents — arXiv:2608.13598 — arXiv (en), https://arxiv.org/abs/2608.13598 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c866812f93c2 (Metraipolis archive dc_c866812f93c2, sha256 b41a01adbcbf6c4905fa9719c6ff77e93ba994ea754e743c7b0a4bf76c15ca22)
BibTeX
@misc{dcc866812f93c2,
  title = {Measuring Cross-Task Behavioral Consistency in Language Model Agents — arXiv:2608.13598},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.13598},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c866812f93c2; sha256 b41a01adbcbf6c4905fa9719c6ff77e93ba994ea754e743c7b0a4bf76c15ca22},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.13598. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it behaves. We argue that behavioral consistency across tasks is a distinct and measurable property, and we introduce the Behavioral Consistency Metric (BCM) to quantify it. BCM trains a model to predict task success from behavioral features of agent execution traces, derives a per-trajectory feature-attribution vector, and measures the mean pairwise similarity of these vectors within an agent system. Across roughly 9,000 trajectories from six language model agents on software engineering tasks, our central finding is that cross-task and within-task consistency are distinct axes that can diverge: some systems are locally reproducible, behaving similarly on repeated attempts at one task, yet globally fragmented, with no stable strategy across different tasks, while others are consistent at both scales. Prior work measures only same-task reproducibility and so cannot observe this separation. We further find that consistency is not reducible to success rate, since systems with comparable success can differ sharply in consistency, and that the frontier-versus-open-source consistency gap persists under a within-task control that holds task difficulty constant. We position BCM as a process-level reliability signal that complements outcome metrics, and we are explicit about the conditions under which it is meaningful.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-17 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.13598 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T02:02:02.716Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).