Metraipolis › archive › dc_b95fdd682893

Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints — arXiv:2608.26197

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.26197
License (as archived)
CC0-1.0
Archived
2026-10-17T16:16:06.717Z
SHA-256 of the archived text
78a2ba4e61d691b0c74bc1a5bbba21233f6dad8df33108b69814bee4253d442b
Stable URL of this page
https://metraipolis.com/documento/dc_b95fdd682893

How to cite

Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints — arXiv:2608.26197 — arXiv (en), https://arxiv.org/abs/2608.26197 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b95fdd682893 (Metraipolis archive dc_b95fdd682893, sha256 78a2ba4e61d691b0c74bc1a5bbba21233f6dad8df33108b69814bee4253d442b)
BibTeX
@misc{dcb95fdd682893,
  title = {Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints — arXiv:2608.26197},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26197},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b95fdd682893; sha256 78a2ba4e61d691b0c74bc1a5bbba21233f6dad8df33108b69814bee4253d442b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26197. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Model (LLM) based agents exhibit substantial run-to-run execution variance even when given identical tasks and tools -- acceptable for exploratory use but unacceptable in regulated domains such as finance and compliance. We study harness engineering: wrapping an agent in a deterministic execution layer (finite-state control, forced tool selection, output validation, bounded retry, and structured planning) and measuring its effect on execution determinism and task success. Across two synthetic tasks (finance and legal) and two open-weight models (Qwen-2.5-7B-Instruct, Gemma-3-27B), a first-pass harness produces a mixed result: it significantly improves reproducibility in one of four model-task cells, significantly degrades it in two, and has no effect in the fourth. A trace-level diagnostic finds the cause: once tool sequence, state sequence, and output are already highly consistent, an unconstrained free-text planning step becomes the dominant remaining source of variance. Adding Structured Planning -- validating the plan against a fixed schema before any tool is invoked -- eliminates the effect entirely: three of four cells reach a Reproducibility Rate and Determinism Index of 1.000 at N=100, and task success rises to 100% in three of four cells. The gain is not free: token cost falls in every cell, but latency shows a genuine, sample-size-robust split by model -- one becomes faster under the constraint, the other markedly slower. We argue harness engineering is a distinct, effective discipline for agent reliability, but its cost is model-dependent and must be measured, not assumed.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-28 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.26197 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:36:41.148Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).