Metraipolis › archive › dc_35c8e718c962

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights — arXiv:2610.01511

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01511
License (as archived)
CC0-1.0
Archived
2026-10-06T13:55:41.446Z
SHA-256 of the archived text
c5cf84fc0d2f559771a805979e6ddc5b8a8e0108ad95cb26e64c2b9bbbf81a92
Stable URL of this page
https://metraipolis.com/documento/dc_35c8e718c962

How to cite

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights — arXiv:2610.01511 — arXiv (en), https://arxiv.org/abs/2610.01511 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_35c8e718c962 (Metraipolis archive dc_35c8e718c962, sha256 c5cf84fc0d2f559771a805979e6ddc5b8a8e0108ad95cb26e64c2b9bbbf81a92)
BibTeX
@misc{dc35c8e718c962,
  title = {GAW-PO: Preference Optimization with Gradient-Aligned Token Weights — arXiv:2610.01511},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01511},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_35c8e718c962; sha256 c5cf84fc0d2f559771a805979e6ddc5b8a8e0108ad95cb26e64c2b9bbbf81a92},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01511. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter $\beta$ decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.01511 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.725Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).