Metraipolis › archive › dc_bec558296737
Private Direct Preference Optimization for LLM Alignment — arXiv:2608.05040
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.05040
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T11:27:34.176Z
- SHA-256 of the archived text
5e3ccd0131a33bdd06a9ff30f2656f2cc767298be45ff43a669e51f7179a4819- Stable URL of this page
https://metraipolis.com/documento/dc_bec558296737
How to cite
Private Direct Preference Optimization for LLM Alignment — arXiv:2608.05040 — arXiv (en), https://arxiv.org/abs/2608.05040 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bec558296737 (Metraipolis archive dc_bec558296737, sha256 5e3ccd0131a33bdd06a9ff30f2656f2cc767298be45ff43a669e51f7179a4819)
BibTeX
@misc{dcbec558296737,
title = {Private Direct Preference Optimization for LLM Alignment — arXiv:2608.05040},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.05040},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bec558296737; sha256 5e3ccd0131a33bdd06a9ff30f2656f2cc767298be45ff43a669e51f7179a4819},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05040. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines. Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-08-06 in cs.CR). Categorie arXiv: Cryptography and Security (cs.CR). Pagina del lavoro: https://arxiv.org/abs/2608.05040 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T11:17:19.454Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).