Metraipolis › archive › dc_fa7b1ebeca86

Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills — arXiv:2610.01833

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.01833
License (as archived): CC0-1.0
Archived: 2026-10-06T12:06:18.584Z
SHA-256 of the archived text: dcf1e66f2b3b9caa724229db9092e3822eaf2a3b47b04f2fe19b00b25de36c41
Stable URL of this page: https://metraipolis.com/documento/dc_fa7b1ebeca86

How to cite

Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills — arXiv:2610.01833 — arXiv (en), https://arxiv.org/abs/2610.01833 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa7b1ebeca86 (Metraipolis archive dc_fa7b1ebeca86, sha256 dcf1e66f2b3b9caa724229db9092e3822eaf2a3b47b04f2fe19b00b25de36c41)

BibTeX

@misc{dcfa7b1ebeca86,
  title = {Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills — arXiv:2610.01833},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01833},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa7b1ebeca86; sha256 dcf1e66f2b3b9caa724229db9092e3822eaf2a3b47b04f2fe19b00b25de36c41},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01833. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Enterprise AI agent skills evolve as tool APIs, models, and specifications change, yet final-output evaluation can miss process-level behavioral drift. We present a continuous evaluation framework combining outcome-level and process-level checks, applied to Revenue and Productivity variants of a Business Value Determination skill in an enterprise Value Aware Resiliency system. The framework independently computes per-run ground truth, materializes reusable template tests, and evaluates tool selection, arguments, execution order, and database integrity through programmatic checks and a narrowly scoped LLM judge. We evaluate 240 trials across two skills, two specification variants, two agent harnesses, and three models. Of 175 trials passing all applicable final numerical checks, 162 (92.6 percent; Wilson 95 percent CI: 87.7-95.6 percent) contained another evaluator-detected deviation. Under a broader seven-check final-state definition, 151 of 164 passing runs (92.1 percent; 95 percent CI: 86.9-95.3 percent) still violated a trajectory check. Dependency attribution reduced a mean of 6.34 failed checks per run to 2.65 roots. Specification sensitivity varied by model and harness, with exploratory bootstrap interaction intervals excluding zero for all three Revenue comparisons and one of three Productivity comparisons. Runtime-resolved templates provided reusable regression coverage across the evaluated configurations; longitudinal validation under actual API evolution remains future work.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.01833 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:27.979Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).