Metraipolis › archive › dc_76ba25ef3477

From Constitutions to Control: Interpretable Rewards for Aligning Language Models — arXiv:2609.33086

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33086
License (as archived)
CC0-1.0
Archived
2026-10-07T22:37:09.680Z
SHA-256 of the archived text
735ab67a630a1dbb5cdf55f6c802734efa05c0fcdb7184a2e95fd724ece24328
Stable URL of this page
https://metraipolis.com/documento/dc_76ba25ef3477

How to cite

From Constitutions to Control: Interpretable Rewards for Aligning Language Models — arXiv:2609.33086 — arXiv (en), https://arxiv.org/abs/2609.33086 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_76ba25ef3477 (Metraipolis archive dc_76ba25ef3477, sha256 735ab67a630a1dbb5cdf55f6c802734efa05c0fcdb7184a2e95fd724ece24328)
BibTeX
@misc{dc76ba25ef3477,
  title = {From Constitutions to Control: Interpretable Rewards for Aligning Language Models — arXiv:2609.33086},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33086},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_76ba25ef3477; sha256 735ab67a630a1dbb5cdf55f6c802734efa05c0fcdb7184a2e95fd724ece24328},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33086. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Current approaches to aligning language models often make it hard to know what behavior is being rewarded or to change that reward in a targeted way. In particular, standard preference-based methods collapse multiple considerations into aggregate human judgments, obscuring what drives the resulting reward, while principle-based methods specify high-level values without fully operationalizing them. To address this gap, we develop a rubric-based framework to transform a general-purpose constitution into an interpretable and tunable reward model, using constitution-guided AI feedback to estimate initial weights for the constituent rubric items. We then reweight those dimensions to construct modified rewards for training. Across experiments on political alignment and safety-helpfulness tradeoffs, reweighting individual dimensions predictably changes targeted behaviors largely independently while navigating tradeoffs between conflicting alignment objectives. We show that the same framework can mitigate label bias encoded in preference judgments -- including sycophancy and demographic bias -- by reducing their influence on the training reward. Our results demonstrate that constitution-derived, interpretable rewards can translate high-level alignment principles into more transparent and controllable model behavior.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.33086 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.860Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).