Metraipolis › archive › dc_d232e93609db

How Does Post-Training Shape LLM Value Preferences? — arXiv:2610.33371

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.33371
License (as archived): CC0-1.0
Archived: 2026-10-27T08:11:17.460Z
SHA-256 of the archived text: 3746409e5900e2feda05cb2b5d26d6893ac10f44e9c5652e0e919d226d955eb0
Stable URL of this page: https://metraipolis.com/documento/dc_d232e93609db

How to cite

How Does Post-Training Shape LLM Value Preferences? — arXiv:2610.33371 — arXiv (en), https://arxiv.org/abs/2610.33371 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d232e93609db (Metraipolis archive dc_d232e93609db, sha256 3746409e5900e2feda05cb2b5d26d6893ac10f44e9c5652e0e919d226d955eb0)

BibTeX

@misc{dcd232e93609db,
  title = {How Does Post-Training Shape LLM Value Preferences? — arXiv:2610.33371},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.33371},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d232e93609db; sha256 3746409e5900e2feda05cb2b5d26d6893ac10f44e9c5652e0e919d226d955eb0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33371. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs. Our code and data are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.33371 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:56:00.192Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).