Metraipolis › archive › dc_22366fbdcbdb

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization — arXiv:2610.07592

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.07592
License (as archived): CC0-1.0
Archived: 2026-10-07T11:18:52.685Z
SHA-256 of the archived text: 01f39ee5c63b324bebf5d3fca1ef60101dae4b58e17e83b1e4dc69285c169617
Stable URL of this page: https://metraipolis.com/documento/dc_22366fbdcbdb

How to cite

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization — arXiv:2610.07592 — arXiv (en), https://arxiv.org/abs/2610.07592 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_22366fbdcbdb (Metraipolis archive dc_22366fbdcbdb, sha256 01f39ee5c63b324bebf5d3fca1ef60101dae4b58e17e83b1e4dc69285c169617)

BibTeX

@misc{dc22366fbdcbdb,
  title = {LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization — arXiv:2610.07592},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.07592},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_22366fbdcbdb; sha256 01f39ee5c63b324bebf5d3fca1ef60101dae4b58e17e83b1e4dc69285c169617},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07592. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.07592 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:24.529Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).