Metraipolis › archive › dc_ebff8ec6d71d

CTE-Bench: Counterfactual Trace Evaluation for Stateful Software Simulators — arXiv:2609.36647

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.36647
License (as archived): CC0-1.0
Archived: 2026-10-07T17:59:01.951Z
SHA-256 of the archived text: 13f0a73e7e934c9c66574a61b5f2fe1f7fa1799452c29bd6341463cdca91ad11
Stable URL of this page: https://metraipolis.com/documento/dc_ebff8ec6d71d

How to cite

CTE-Bench: Counterfactual Trace Evaluation for Stateful Software Simulators — arXiv:2609.36647 — arXiv (en), https://arxiv.org/abs/2609.36647 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ebff8ec6d71d (Metraipolis archive dc_ebff8ec6d71d, sha256 13f0a73e7e934c9c66574a61b5f2fe1f7fa1799452c29bd6341463cdca91ad11)

BibTeX

@misc{dcebff8ec6d71d,
  title = {CTE-Bench: Counterfactual Trace Evaluation for Stateful Software Simulators — arXiv:2609.36647},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36647},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ebff8ec6d71d; sha256 13f0a73e7e934c9c66574a61b5f2fe1f7fa1799452c29bd6341463cdca91ad11},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36647. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Coding agents change running software: they patch a service's code or overwrite its stored state, and then act on their own expectation of how the service will respond afterwards. A wrong expectation may surface only several calls later. Function-level code-execution benchmarks omit persistent service state, and agent benchmarks score the actions an agent takes or the final state it reaches. We introduce CTE-Bench, which measures whether a model can predict how an intervention changes a stateful service's future behavior, without asking it to choose actions. Each scenario gives the model Python service code, the calls and responses observed before the intervention, the intervention itself (a source edit or a state overwrite), and 40 fixed future calls; the model predicts every future response, and predictions are checked by executing the service. Three memory protocols control whether the model sees the correct earlier responses, none of them, or its own earlier predictions. CTE-Bench-Core-v1 contains 255 scenarios over six deterministic Python services, giving 10,200 predictions per model. The main score is effect-step value match (VM): exact response equality on the 2,476 future calls whose response the intervention changes. With correct earlier responses revealed, four API-hosted models (DeepSeek V4-Flash, Kimi K2.5, Qwen3.6-35B-A3B, and Claude Sonnet 4.6) reach 54.3%-61.5% effect-step VM. Hiding those responses lowers effect-step VM to 23.2%-28.9%; conditioning on self-generated predictions gives 24.8%-33.2%, and at most 1.2% of scenarios are predicted exactly end to end. Current models thus track intervention effects mainly when correct feedback is supplied, and their errors compound over a rollout. We release CTE-Bench-Core-v1 with its executable oracle, evaluation scripts, and an evaluation card mapping each claim to its protocol.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-30 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.36647 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T17:31:52.001Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).