Metraipolis › archive › dc_2feae5230e27
AdaT$^2$: Adaptive Test Transformations for Black-Box Boundary Testing of Conversational Agents — arXiv:2610.10141
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.10141
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T12:37:55.403Z
- SHA-256 of the archived text
44b379f7170bee4142082829c64c5b872a750c589679ddaadb308182196fe69f- Stable URL of this page
https://metraipolis.com/documento/dc_2feae5230e27
How to cite
AdaT$^2$: Adaptive Test Transformations for Black-Box Boundary Testing of Conversational Agents — arXiv:2610.10141 — arXiv (en), https://arxiv.org/abs/2610.10141 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2feae5230e27 (Metraipolis archive dc_2feae5230e27, sha256 44b379f7170bee4142082829c64c5b872a750c589679ddaadb308182196fe69f)
BibTeX
@misc{dc2feae5230e27,
title = {AdaT$^2$: Adaptive Test Transformations for Black-Box Boundary Testing of Conversational Agents — arXiv:2610.10141},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.10141},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2feae5230e27; sha256 44b379f7170bee4142082829c64c5b872a750c589679ddaadb308182196fe69f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10141. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Conversational agents based on large language models (LLMs) must comply with policies. Each condition in a policy draws a boundary between user requests, and the agent must behave differently on its two sides. We present AdaT$^2$, which extracts statements from the agent's replies in exploratory conversations with an LLM acting as the user, and uses the statements to guide boundary test generation. Each statement describes one condition and the behavior expected when the condition holds. Besides plain tests guided by single statements, AdaT$^2$ writes transformed tests guided by pairs of a statement and a test transformation instruction, such as "omit one required input". The instruction of a pair can move a test to the other side of the statement's boundary or to another boundary. The statements and instructions form far more pairs than a run can try, and many pairs are not applicable. Adaptive pair selection therefore chooses the statement of each pair by novelty and the instruction with the bandit algorithm Bayes-UCB, which learns from whether earlier pairs yielded a test and whether the agent passed it according to an LLM judge. Our benchmark counts the two sides of each boundary separately and distinguishes boundaries explicitly defined by the agent's prompt, tool code, or knowledge base from boundaries that the agent's LLM infers from domain knowledge. On four domains of $\tau^3$-bench, 62.7% to 83.3% of AdaT$^2$'s tests are valid boundary tests whose expected behavior is explicitly defined, higher in every domain than AgentEval's (47.7% to 68.1%). Transformed tests add 13 to 46 explicitly defined boundaries that plain tests miss. As regression tests, AdaT$^2$'s test suites detect all eight seeded policy faults in the airline domain and four of eight in the retail domain, and AgentEval's test suites, with fewer than a third as many tests, detect five and two, respectively. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2610.10141 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T12:14:30.252Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).