Metraipolis › archive › dc_b6ebe68e09c8
SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research — arXiv:2611.10329
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.10329
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-10T18:40:53.599Z
- SHA-256 of the archived text
3a2ddd7db8dc45a89e995d6718b837c1b1ec1c7ee1a72429e802f7db6ac471c2- Stable URL of this page
https://metraipolis.com/documento/dc_b6ebe68e09c8
How to cite
SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research — arXiv:2611.10329 — arXiv (en), https://arxiv.org/abs/2611.10329 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b6ebe68e09c8 (Metraipolis archive dc_b6ebe68e09c8, sha256 3a2ddd7db8dc45a89e995d6718b837c1b1ec1c7ee1a72429e802f7db6ac471c2)
BibTeX
@misc{dcb6ebe68e09c8,
title = {SIRIS: Can LLMs Tell When Simulation Errors Matter in AI-Driven Systems Research — arXiv:2611.10329},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.10329},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b6ebe68e09c8; sha256 3a2ddd7db8dc45a89e995d6718b837c1b1ec1c7ee1a72429e802f7db6ac471c2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.10329. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
LLM agents increasingly drive systems research and rely on simulators to evaluate candidate designs at scale. Simulators are imperfect by design: their modeling limitations can distort predictions yet leave the recommended design correct, or change which design should be selected. We introduce Siris to test whether LLMs can tell these cases apart, judging from their systems understanding when a limitation invalidates the simulator's recommendation. By perturbing a simulator, Siris constructs workload pairs with the same candidates, limitation, and simulator recommendation. The perturbation changes predictions on both workloads, yet a reference simulator supports the recommendation on only one. Evaluating thirteen frontier LLMs on LLM serving design problems, we find that identifying a limitation does not ensure a correct choice, and even the strongest models sometimes abandon valid recommendations. Self-correction through a review turn helps these models restore valid choices but introduces new errors in several others. These findings motivate evaluating systems understanding through design decisions under imperfect simulators. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.10329 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-10T18:37:37.486Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).