Metraipolis › archive › dc_ae8086fd9529

On Evaluating and Improving Conversational Agents in Production — arXiv:2609.32092

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32092
License (as archived)
CC0-1.0
Archived
2026-10-07T20:53:52.982Z
SHA-256 of the archived text
9309c0e782a2d72e6229fb61dc8a5a1b04fbe7bc9aa4e7bd760e87a7a6e044d1
Stable URL of this page
https://metraipolis.com/documento/dc_ae8086fd9529

How to cite

On Evaluating and Improving Conversational Agents in Production — arXiv:2609.32092 — arXiv (en), https://arxiv.org/abs/2609.32092 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ae8086fd9529 (Metraipolis archive dc_ae8086fd9529, sha256 9309c0e782a2d72e6229fb61dc8a5a1b04fbe7bc9aa4e7bd760e87a7a6e044d1)
BibTeX
@misc{dcae8086fd9529,
  title = {On Evaluating and Improving Conversational Agents in Production — arXiv:2609.32092},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32092},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ae8086fd9529; sha256 9309c0e782a2d72e6229fb61dc8a5a1b04fbe7bc9aa4e7bd760e87a7a6e044d1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32092. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We present a framework for evaluating and improving a large-scale, multi-agent shopping assistant in production, and report lessons from its use. Offline evaluation of such a system faces three obstacles. (i) A logged conversation cannot be replayed against a modified system, because a different response changes every turn that follows. (ii) The unchanged system itself varies from run to run. Its LLM components are stochastic, and in product search the available products, their prices, and the customer's personalization signals change. (iii) Aggregate quality scores combine distinct behaviors, so they show that quality has changed but not which behavior caused the change. Our framework addresses each obstacle in turn. For a reported behavior, an Evaluation Harness generates targeted assertions and a fixed cohort of customer scenarios. It then reproduces the behavior in a local instance of the assistant through grounded user simulation. Instead of replaying the log, the simulator writes new customer turns conditioned on the recorded messages and context. Repeated runs of the unchanged system form a stored baseline. An Improvement Orchestrator turns the assertion results into hypotheses, implements each as an isolated modification, and compares it with the baseline using paired percentile bootstrap intervals over scenario-level differences. When an investigation ends, the harness may propose revisions to future evaluations, subject to human approval and without altering past decisions. We report production investigations with this framework. Assertion profiles showed which positions of a product carousel a failure affected, and repeated runs distinguished a real improvement from run-to-run fluctuation. Audits of the evaluation itself found a judge that lacked the evidence it needed and a model setting that was configured but not applied.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.MA).
Categorie arXiv: Multiagent Systems (cs.MA) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Emerging Technologies (cs.ET); Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2609.32092 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:24:41.606Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).