Metraipolis › archive › dc_072ba7780f8f
Direct Preference Density Alignment for Conversational Audio Equalization — arXiv:2609.12607
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.12607
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:03:31.740Z
- SHA-256 of the archived text
73089627852a2ddb9f01ef15d4de3da070eff53ddee0ef0b1956465e46a69172- Stable URL of this page
https://metraipolis.com/documento/dc_072ba7780f8f
How to cite
Direct Preference Density Alignment for Conversational Audio Equalization — arXiv:2609.12607 — arXiv (en), https://arxiv.org/abs/2609.12607 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_072ba7780f8f (Metraipolis archive dc_072ba7780f8f, sha256 73089627852a2ddb9f01ef15d4de3da070eff53ddee0ef0b1956465e46a69172)
BibTeX
@misc{dc072ba7780f8f,
title = {Direct Preference Density Alignment for Conversational Audio Equalization — arXiv:2609.12607},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.12607},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_072ba7780f8f; sha256 73089627852a2ddb9f01ef15d4de3da070eff53ddee0ef0b1956465e46a69172},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12607. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.AI). Categorie arXiv: Sound (cs.SD) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.12607 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T02:51:27.929Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).