Metraipolis › archive › dc_54cc0032f87d

Axiom Satisfiability of Linear Rewards in Alignment — arXiv:2610.06892

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.06892
License (as archived)
CC0-1.0
Archived
2026-10-07T11:17:13.625Z
SHA-256 of the archived text
70c06de4735165bdedfaeb60e602028da6a85149ab2220bffdc0abe21faa7d2d
Stable URL of this page
https://metraipolis.com/documento/dc_54cc0032f87d

How to cite

Axiom Satisfiability of Linear Rewards in Alignment — arXiv:2610.06892 — arXiv (en), https://arxiv.org/abs/2610.06892 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_54cc0032f87d (Metraipolis archive dc_54cc0032f87d, sha256 70c06de4735165bdedfaeb60e602028da6a85149ab2220bffdc0abe21faa7d2d)
BibTeX
@misc{dc54cc0032f87d,
  title = {Axiom Satisfiability of Linear Rewards in Alignment — arXiv:2610.06892},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.06892},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_54cc0032f87d; sha256 70c06de4735165bdedfaeb60e602028da6a85149ab2220bffdc0abe21faa7d2d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.06892. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Learning from human preference data is the dominant route to aligning language models with human values. In linear social choice, where rewards are linear in a fixed feature representation of prompt-response pairs, Ge et al.[2024] show that fitting such a reward by minimizing any non-decreasing convex loss, including BTL, fails PO and PMC. Moreover, no rule that reads only the majority relation can satisfy PO once the output is required to be linearly induced. We ask what it costs to enforce these axioms anyway. To this end, we relax the linear model to allow per-candidate slack. We compute the relaxed linear reward with the smallest total slack that satisfies the axioms with a margin $\eta$, the minimum required difference between two reward values. Our solution satisfies the axioms under no assumptions about the voters or how comparisons were collected. We bound the optimal total slack by $O(1)$ when $\eta$ is at most $O(\frac{1}{m^2})$ for $m$ candidates. Furthermore, we exhibit an instance that forces this bound, concluding that the rate is tight up to constants. In practice, the no. of candidates far exceeds the feature dimension, and only a linear reward can be evaluated on unseen responses. We therefore introduce a new method that charges the linear part for each comparison it gets wrong. It simultaneously minimizes the total slack and the no. of violations, with a parameter $\lambda$ trading off between them. We show that the total slack is monotone but saturating in $\lambda$: raising it reduces the violations of the deployed linear reward and increases the slack, yet the slack stays below $(\eta+\Delta\sqrt{d})\lfloor m^2/4\rfloor$, where $\Delta$ and $d$ are the diameter and dimension of the features, respectively. Experiments on both synthetic and real-life preference data corroborate our theory and show that the linear reward output by our method beats linear BTL.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.MA, nuovi lavori del giorno).
Categorie arXiv: Computer Science and Game Theory (cs.GT) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA).
Pagina del lavoro: https://arxiv.org/abs/2610.06892 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:09.463Z dalla pagina https://arxiv.org/list/cs.MA/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).