Metraipolis › archive › dc_fd65cedee959

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback — arXiv:2608.15530

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.15530
License (as archived): CC0-1.0
Archived: 2026-10-18T18:58:44.251Z
SHA-256 of the archived text: ec520c292988ea8457038db2e755c5ef910f08640b2845915026bc7eb9e8f72d
Stable URL of this page: https://metraipolis.com/documento/dc_fd65cedee959

How to cite

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback — arXiv:2608.15530 — arXiv (en), https://arxiv.org/abs/2608.15530 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd65cedee959 (Metraipolis archive dc_fd65cedee959, sha256 ec520c292988ea8457038db2e755c5ef910f08640b2845915026bc7eb9e8f72d)

BibTeX

@misc{dcfd65cedee959,
  title = {Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback — arXiv:2608.15530},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.15530},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd65cedee959; sha256 ec520c292988ea8457038db2e755c5ef910f08640b2845915026bc7eb9e8f72d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15530. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with human feedback (RLHF) aligns LLMs with human preferences, improving summarization fluency and safety, but causes sentiment drift: overly neutral summaries stripped of emotional nuance. We diagnose why RL acts as a sentiment neutralizer and present Policy Attribution, a framework using gradient and logit decomposition to trace drift to reward model (RM) signals and KL (Kullback-Leibler) penalty. Sentiment drift reflects a strategic bias toward "low-risk" tokens maximizing expected rewards under preference uncertainty (Stiennon et al., 2020; Gao, Schulman, and Hilton, 2023). On Reddit TL;DR and CNN/DailyMail, RLHF summaries get higher rewards but show 30-40% lower sentiment variance. Cross-lingual analysis across eight languages shows language-independent drift, with morphologically richer languages more suppressed (Krasitskii et al., 2026). We propose and validate a sentiment-aware regularization technique reducing drift by 18-22% without harming summary quality. The code and toolkit will be public.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.15530 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T18:06:57.856Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).