Metraipolis › archive › dc_8d13a175d4cd

BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment — arXiv:2609.35044

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35044
License (as archived)
CC0-1.0
Archived
2026-10-07T21:03:12.937Z
SHA-256 of the archived text
68deba9db5e37f2df85eecfcdef3012edccbfacc64958f5a2fa7f95556e5ab27
Stable URL of this page
https://metraipolis.com/documento/dc_8d13a175d4cd

How to cite

BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment — arXiv:2609.35044 — arXiv (en), https://arxiv.org/abs/2609.35044 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d13a175d4cd (Metraipolis archive dc_8d13a175d4cd, sha256 68deba9db5e37f2df85eecfcdef3012edccbfacc64958f5a2fa7f95556e5ab27)
BibTeX
@misc{dc8d13a175d4cd,
  title = {BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment — arXiv:2609.35044},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35044},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d13a175d4cd; sha256 68deba9db5e37f2df85eecfcdef3012edccbfacc64958f5a2fa7f95556e5ab27},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35044. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator's bias up to a shared constant. The remaining constant is what fixes the aligned model's attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability 0.96 and BA-DPO removes 81 to 95\% of that shift; on MultiPref with real annotators it removes about half of DPO's lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.35044 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:03.241Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).