Metraipolis › archive › dc_1caf60efda45
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation — arXiv:2609.06893
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.06893
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:45:22.348Z
- SHA-256 of the archived text
7607c3a8da58fe794d68a5c7361042d2e4b4a856b286350d33c9ad2cdaf171f4- Stable URL of this page
https://metraipolis.com/documento/dc_1caf60efda45
How to cite
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation — arXiv:2609.06893 — arXiv (en), https://arxiv.org/abs/2609.06893 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1caf60efda45 (Metraipolis archive dc_1caf60efda45, sha256 7607c3a8da58fe794d68a5c7361042d2e4b4a856b286350d33c9ad2cdaf171f4)
BibTeX
@misc{dc1caf60efda45,
title = {Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation — arXiv:2609.06893},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.06893},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1caf60efda45; sha256 7607c3a8da58fe794d68a5c7361042d2e4b4a856b286350d33c9ad2cdaf171f4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06893. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii) Can their advantages be incorporated into offline alignment? To answer the first question, our controlled experiments reveal that the explicit preference model, additionally introduced in the iterative procedure, is a key factor behind its superiority over offline methods. This insight leads us to answer the second question affirmatively and propose Distilled Preference Probability Policy Optimization (DP3O), an effective and efficient offline alignment algorithm. DP3O first learns an explicit preference model using a helper class of LLMs and then distills its knowledge into policy optimization. Theoretically, we show that explicit preference modeling admits better estimation error control than implicit formulations, and that DP3O achieves a tighter generalization bound than hard-label DPO through variance reduction. Empirically, we evaluate DP3O on a wide range of chat-based and downstream tasks and show that it outperforms state-of-the-art offline methods, achieves performance comparable to iterative DPO, and reduces training time by about $42\%$, demonstrating both its effectiveness and efficiency. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.06893 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:25.074Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).