Metraipolis › archive › dc_28fea017d3e9

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL — arXiv:2608.12253

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12253
License (as archived)
CC0-1.0
Archived
2026-10-19T05:16:35.517Z
SHA-256 of the archived text
ad8c2d54a00964ac1ae927a6a06a9ad21b9d56206abb3d7f752e83dfac9e9ded
Stable URL of this page
https://metraipolis.com/documento/dc_28fea017d3e9

How to cite

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL — arXiv:2608.12253 — arXiv (en), https://arxiv.org/abs/2608.12253 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_28fea017d3e9 (Metraipolis archive dc_28fea017d3e9, sha256 ad8c2d54a00964ac1ae927a6a06a9ad21b9d56206abb3d7f752e83dfac9e9ded)
BibTeX
@misc{dc28fea017d3e9,
  title = {One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL — arXiv:2608.12253},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12253},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_28fea017d3e9; sha256 ad8c2d54a00964ac1ae927a6a06a9ad21b9d56206abb3d7f752e83dfac9e9ded},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12253. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.12253 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:08:46.781Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).