Metraipolis › archive › dc_1caf60efda45

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation — arXiv:2609.06893

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.06893
License (as archived)
CC0-1.0
Archived
2026-10-16T12:45:22.348Z
SHA-256 of the archived text
7607c3a8da58fe794d68a5c7361042d2e4b4a856b286350d33c9ad2cdaf171f4
Stable URL of this page
https://metraipolis.com/documento/dc_1caf60efda45

How to cite

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation — arXiv:2609.06893 — arXiv (en), https://arxiv.org/abs/2609.06893 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1caf60efda45 (Metraipolis archive dc_1caf60efda45, sha256 7607c3a8da58fe794d68a5c7361042d2e4b4a856b286350d33c9ad2cdaf171f4)
BibTeX
@misc{dc1caf60efda45,
  title = {Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation — arXiv:2609.06893},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06893},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1caf60efda45; sha256 7607c3a8da58fe794d68a5c7361042d2e4b4a856b286350d33c9ad2cdaf171f4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06893. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about $42\%$, demonstrating both its effectiveness and efficiency.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.06893 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.074Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).