Metraipolis › archive › dc_88f2aa29ec67

When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models — arXiv:2608.31035

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.31035
License (as archived)
CC0-1.0
Archived
2026-10-17T08:07:52.501Z
SHA-256 of the archived text
e332a8291affb1bdc2a5e1804f381405ca78e187ca6a2ba38527a6318ba54f47
Stable URL of this page
https://metraipolis.com/documento/dc_88f2aa29ec67

How to cite

When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models — arXiv:2608.31035 — arXiv (en), https://arxiv.org/abs/2608.31035 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_88f2aa29ec67 (Metraipolis archive dc_88f2aa29ec67, sha256 e332a8291affb1bdc2a5e1804f381405ca78e187ca6a2ba38527a6318ba54f47)
BibTeX
@misc{dc88f2aa29ec67,
  title = {When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models — arXiv:2608.31035},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.31035},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_88f2aa29ec67; sha256 e332a8291affb1bdc2a5e1804f381405ca78e187ca6a2ba38527a6318ba54f47},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.31035. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Sound (cs.SD); Audio and Speech Processing (eess.AS).
Pagina del lavoro: https://arxiv.org/abs/2608.31035 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:47:52.876Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).