Metraipolis › archive › dc_28fea017d3e9
One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL — arXiv:2608.12253
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12253
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T05:16:35.517Z
- SHA-256 of the archived text
ad8c2d54a00964ac1ae927a6a06a9ad21b9d56206abb3d7f752e83dfac9e9ded- Stable URL of this page
https://metraipolis.com/documento/dc_28fea017d3e9
How to cite
One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL — arXiv:2608.12253 — arXiv (en), https://arxiv.org/abs/2608.12253 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_28fea017d3e9 (Metraipolis archive dc_28fea017d3e9, sha256 ad8c2d54a00964ac1ae927a6a06a9ad21b9d56206abb3d7f752e83dfac9e9ded)
BibTeX
@misc{dc28fea017d3e9,
title = {One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL — arXiv:2608.12253},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12253},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_28fea017d3e9; sha256 ad8c2d54a00964ac1ae927a6a06a9ad21b9d56206abb3d7f752e83dfac9e9ded},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12253. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.12253 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:08:46.781Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).