Metraipolis › archive › dc_682d363bb39b

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning — arXiv:2609.14896

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.14896
License (as archived): CC0-1.0
Archived: 2026-10-15T22:41:34.584Z
SHA-256 of the archived text: 46d33d228a15d62946c9bdc99e3b1bc39d17a5b4ba8ae8f3bb29bc0d9bd1a180
Stable URL of this page: https://metraipolis.com/documento/dc_682d363bb39b

How to cite

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning — arXiv:2609.14896 — arXiv (en), https://arxiv.org/abs/2609.14896 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_682d363bb39b (Metraipolis archive dc_682d363bb39b, sha256 46d33d228a15d62946c9bdc99e3b1bc39d17a5b4ba8ae8f3bb29bc0d9bd1a180)

BibTeX

@misc{dc682d363bb39b,
  title = {Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning — arXiv:2609.14896},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.14896},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_682d363bb39b; sha256 46d33d228a15d62946c9bdc99e3b1bc39d17a5b4ba8ae8f3bb29bc0d9bd1a180},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14896. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.MA).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2609.14896 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:21:04.239Z dalla pagina https://arxiv.org/catchup/cs.MA/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).