Metraipolis › archive › dc_5b4f9b341bcc

Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning — arXiv:2610.41588

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41588
License (as archived): CC0-1.0
Archived: 2026-11-02T07:51:19.369Z
SHA-256 of the archived text: 377e7f239f4bd3fe93ef100e8665bde1578b1d0d7352467419ed19307b0549f6
Stable URL of this page: https://metraipolis.com/documento/dc_5b4f9b341bcc

How to cite

Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning — arXiv:2610.41588 — arXiv (en), https://arxiv.org/abs/2610.41588 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5b4f9b341bcc (Metraipolis archive dc_5b4f9b341bcc, sha256 377e7f239f4bd3fe93ef100e8665bde1578b1d0d7352467419ed19307b0549f6)

BibTeX

@misc{dc5b4f9b341bcc,
  title = {Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning — arXiv:2610.41588},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41588},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5b4f9b341bcc; sha256 377e7f239f4bd3fe93ef100e8665bde1578b1d0d7352467419ed19307b0549f6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41588. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$, a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$. The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Methodology (stat.ME).
Pagina del lavoro: https://arxiv.org/abs/2610.41588 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:40:00.259Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).