Metraipolis › archive › dc_d1d61819e4bc
Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy — arXiv:2609.19820
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.19820
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T03:20:30.259Z
- SHA-256 of the archived text
c84b3f9dc8bc1c13477760a399db919fe521492d137ff27bf096e458828014dd- Stable URL of this page
https://metraipolis.com/documento/dc_d1d61819e4bc
How to cite
Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy — arXiv:2609.19820 — arXiv (en), https://arxiv.org/abs/2609.19820 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d1d61819e4bc (Metraipolis archive dc_d1d61819e4bc, sha256 c84b3f9dc8bc1c13477760a399db919fe521492d137ff27bf096e458828014dd)
BibTeX
@misc{dcd1d61819e4bc,
title = {Steering Equilibrium Selection in Regularized Self-Play via the Reference Policy — arXiv:2609.19820},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.19820},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d1d61819e4bc; sha256 c84b3f9dc8bc1c13477760a399db919fe521492d137ff27bf096e458828014dd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19820. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Regularized self-play -- the family behind DeepNash's Stratego play -- drives a two-player zero-sum policy to a Nash equilibrium by best-responding to a slowly moving, entropy-regularized reference policy $\rho$. When the game has a polytope of value-equivalent equilibria, the regularizer silently breaks the tie: with a uniform reference it selects the maximum-entropy member, the I-projection of $\rho$ onto the Nash set. Can the reference be used to choose the equilibrium on purpose? On five exactly solvable games plus a 2-D polytope, with exact best responses and equivalence tests over independent seeds, anchoring the reference at a target member and refining steers self-play to that member with mean coordinate error 0.007 at median exploitability $5\times10^{-5}$, TOST-equivalent to the request within $\pm0.05$; the anchoring persists through refinement and follows the reference, not the initialization. Selection follows the reach-weighted I-projection (slope 0.969 [0.950, 0.987]). We report with equal emphasis where the story breaks: fixed off-manifold references cost 0.08-0.25 exploitability; stiff or flat families require a smaller mirror step, set by a pre-registered rule; boundary targets undershoot; curvature predicts where boundary saturation bites (rank correlation 0.90, p=0.037) while interior precision is curvature-independent. Table and MLP steering maps are equivalent within $\pm0.03$ at every target (30 seeds); matched control arms show attention's robust signature is excess seed variance, any systematic shift bounded at 0.018 and not significant. Against a best response the selection-robustness trade-off is degenerate: steering matters only against fixed, non-equilibrium opponents. The recipe -- anchor the reference at the desired member and refine -- reinterprets the KL anchor of RLHF-style RL as a selection knob, not only a stability leash.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.19820 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:04:13.647Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).