Metraipolis › archive › dc_977dd22dee51

Multi-LLM Collaborative Alignment via Stackelberg Games — arXiv:2609.39076

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39076
License (as archived)
CC0-1.0
Archived
2026-10-07T12:42:27.411Z
SHA-256 of the archived text
c06d7085e87789c8bca6593170f7206685010fc3b9519caebbf2aa2a0a60140d
Stable URL of this page
https://metraipolis.com/documento/dc_977dd22dee51

How to cite

Multi-LLM Collaborative Alignment via Stackelberg Games — arXiv:2609.39076 — arXiv (en), https://arxiv.org/abs/2609.39076 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_977dd22dee51 (Metraipolis archive dc_977dd22dee51, sha256 c06d7085e87789c8bca6593170f7206685010fc3b9519caebbf2aa2a0a60140d)
BibTeX
@misc{dc977dd22dee51,
  title = {Multi-LLM Collaborative Alignment via Stackelberg Games — arXiv:2609.39076},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39076},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_977dd22dee51; sha256 c06d7085e87789c8bca6593170f7206685010fc3b9519caebbf2aa2a0a60140d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39076. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A pool of language models can collaborate and improve collectively by learning from one another's responses. These interactions depend on the instructions used during training. Existing methods typically sample instructions uniformly, even though their usefulness may change as the models improve: an instruction on which models' responses once differed in quality may later be answered equally well, while a previously difficult instruction may begin to provide a useful learning signal. We propose Stackelberg Alignment, a game-theory-inspired leader-follower framework that turns instruction selection into an adaptive curriculum. An EXP3 bandit acts as the leader, allocating a fixed sampling budget across instructions and updating its sampling distribution using a reward that combines instruction difficulty and response discriminability. The language models act as followers: they respond to the selected instructions, evaluate one another's responses, and learn from the resulting preference signals through DPO or GRPO. The framework uses Elo-style reputation-weighted peer judgment and reputation-based opponent matching to support reliable and competitive model interactions. Experiments across three heterogeneous model pools and 12 benchmarks spanning scientific discovery, reasoning, code, instruction following, and knowledge show that Stackelberg Alignment achieves the highest macro-average across three diverse model pools, outperforming the strongest training-time baseline by up to 7.4% and the best static inference baseline by 12-25%. Analysis confirms that the adaptive leader concentrates duels on the most informative instructions, and ablations show that both reputation-weighted judgment and reputation-based matching improve the effectiveness of multi-LLM evolution.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.39076 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.729Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).