Metraipolis › archive › dc_4d80f3ecf2b3
Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures — arXiv:2610.04328
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.04328
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T05:24:28.475Z
- SHA-256 of the archived text
49bb0081c642ff17e51753cfd602f782a0aa9d529535d0524e6271c18916dddc- Stable URL of this page
https://metraipolis.com/documento/dc_4d80f3ecf2b3
How to cite
Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures — arXiv:2610.04328 — arXiv, https://arxiv.org/abs/2610.04328 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4d80f3ecf2b3 (Metraipolis archive dc_4d80f3ecf2b3, sha256 49bb0081c642ff17e51753cfd602f782a0aa9d529535d0524e6271c18916dddc)
BibTeX
@misc{dc4d80f3ecf2b3,
title = {Bidirectional Preference Synthesis: Learning Prompt-Conditioned Preferences from Boundary Failures — arXiv:2610.04328},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.04328},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4d80f3ecf2b3; sha256 49bb0081c642ff17e51753cfd602f782a0aa9d529535d0524e6271c18916dddc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04328. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Correction-based offline preference pipelines commonly treat model failures only as rejected responses under the original prompt. This supervision is incomplete for boundary failures: responses that violate the given instruction yet coherently satisfy a nearby intent or constraint setting. We introduce Bidirectional Preference Synthesis (BPS), a data-construction method for standard Direct Preference Optimization (DPO) that makes this missing prompt dependence explicit. For each validated boundary failure, BPS keeps the conventional forward pair under the original prompt and adds a reverse pair under a synthesized achieved prompt, so the same response is rejected where it is wrong and chosen where it is right, without changing the DPO objective, training a reward model, or requiring online sampling. On Qwen3-4B-Instruct-2507, BPS preserves original-side pairwise ranking while raising achieved-side ranking accuracy from 6.8% to 62.3% on held-out crossed anchors, with a similar shift under a Kimi-K2.6 cross-teacher probe. A blind human audit supports the intended reverse preference direction, and downstream evaluations show the clearest separation from Forward-DPO in multilingual multi-turn instruction following, with consistent capability-retention patterns on agentic, tool-use, and code checks. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.04328 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T05:24:05.457Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).