Metraipolis › archive › dc_08a34b595f72
CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning — arXiv:2610.09219
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09219
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:09:07.977Z
- SHA-256 of the archived text
b0ad5e66b82a98cfcbcae34ab7fddc9534d71e1135f05b48d5ddd5e7acf09005- Stable URL of this page
https://metraipolis.com/documento/dc_08a34b595f72
How to cite
CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning — arXiv:2610.09219 — arXiv (en), https://arxiv.org/abs/2610.09219 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_08a34b595f72 (Metraipolis archive dc_08a34b595f72, sha256 b0ad5e66b82a98cfcbcae34ab7fddc9534d71e1135f05b48d5ddd5e7acf09005)
BibTeX
@misc{dc08a34b595f72,
title = {CM-DPO: Constraint-Margin Direct Preference Optimization for LLM Planning — arXiv:2610.09219},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09219},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_08a34b595f72; sha256 b0ad5e66b82a98cfcbcae34ab7fddc9534d71e1135f05b48d5ddd5e7acf09005},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09219. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Direct Preference Optimization (DPO) treats all constraint violations equally: a $1 budget overshoot and a $1,000 overshoot induce the same training signal. It is also susceptible to length and style bias when preference pairs come from different model families. We introduce Constraint-Margin DPO (CM-DPO), which replaces DPO's binary preference signal with a continuous margin derived from a deterministic symbolic verifier and scaled by violation severity. Hard and soft constraints are separated through a lexicographic objective, ensuring hard constraints are never traded off against preferences. To supply CM-DPO with bias-reduced training pairs, we generate preference data through procedurally generated constraint profiles (DCCG) and minimal-edit distillation from a reasoning teacher (RT-MED), within a framework we call SynPlan-R. On TravelPlanner, NaturalPlan, and out-of-distribution PlanBench, an 8B model fine-tuned with CM-DPO achieves 89.2% pass rate and 93.4% solve rate, matching multi-agent systems at 13x lower latency while outperforming GPT-4o on unseen Blocksworld by 9.2 points. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.09219 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:14:05.919Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).