Metraipolis › archive › dc_9def51070fb2
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO — arXiv:2610.20089
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.20089
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T21:56:53.977Z
- SHA-256 of the archived text
eeacfbd7650606e1a1896b78d9ee89b290c7ef5c6cd94bbd0330dcc75fb70cae- Stable URL of this page
https://metraipolis.com/documento/dc_9def51070fb2
How to cite
When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO — arXiv:2610.20089 — arXiv (en), https://arxiv.org/abs/2610.20089 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9def51070fb2 (Metraipolis archive dc_9def51070fb2, sha256 eeacfbd7650606e1a1896b78d9ee89b290c7ef5c6cd94bbd0330dcc75fb70cae)
BibTeX
@misc{dc9def51070fb2,
title = {When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO — arXiv:2610.20089},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.20089},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9def51070fb2; sha256 eeacfbd7650606e1a1896b78d9ee89b290c7ef5c6cd94bbd0330dcc75fb70cae},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20089. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pairwise preference feedback has become a widely adopted supervision signal for reinforcement learning in video generation models, as pairwise comparisons provide a more reliable alternative than pointwise scalar ratings for evaluating the quality of generated videos. Specifically, we define a preference graph for each GRPO group under a pairwise evaluator, where nodes represent sampled rollouts and edges encode pairwise preferences. Auditing 6,976 on-policy comparison graphs, we find that cyclic preferences are prevalent and persist throughout training. Naive pairwise reward aggregation based on average win rates discards the underlying comparison structures and treats each rollout independently, which may introduce unreliable advantage estimates for certain rollouts and consequently affect reinforcement learning optimization. Therefore, we propose the \emph{Reliability-Anchored Preference Walk} (\ours{}), a graph-aware advantage aggregation method that strengthens the global preference signal from win rates with reliable relational evidence from cyclic preference graphs through a reliability-guided preference walk. Under matched evaluator, comparison protocol, group size, prompt schedule, and query budget, \ours{} requires no additional evaluator queries beyond the observed comparisons within each rollout group and improves over Pairwise-GRPO across controlled comparisons, transfer settings, and blind human evaluations.
Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2610.20089 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T21:38:29.518Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).