Metraipolis › archive › dc_4848004cc87a

Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL — arXiv:2610.20898

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20898
License (as archived)
CC0-1.0
Archived
2026-10-16T09:06:35.486Z
SHA-256 of the archived text
34b833c8fa2b1e0e26689179209bc359b6e681444f87e0a93b776751214b0781
Stable URL of this page
https://metraipolis.com/documento/dc_4848004cc87a

How to cite

Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL — arXiv:2610.20898 — arXiv (en), https://arxiv.org/abs/2610.20898 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4848004cc87a (Metraipolis archive dc_4848004cc87a, sha256 34b833c8fa2b1e0e26689179209bc359b6e681444f87e0a93b776751214b0781)
BibTeX
@misc{dc4848004cc87a,
  title = {Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL — arXiv:2610.20898},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20898},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4848004cc87a; sha256 34b833c8fa2b1e0e26689179209bc359b6e681444f87e0a93b776751214b0781},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20898. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards. This is puzzling from the usual prompt-level view, in which majority vote only returns a model's modal answer and cannot add information about correctness. We argue that the relevant unit is the batch-level update. Because parameters are shared, a prompt's own gradient accounts for less than 1\% of the change in that prompt's output at practical batch sizes, so a wrong label matters only to the extent that label errors share a direction across prompts. We decompose the majority-vote update into shrinkage and misdirection relative to the gold update, and show that misdirection vanishes with batch size whenever errors are incoherent across prompts, however often the labels are wrong. Measuring per-response gradients on five models, we find that majority-vote errors are as incoherent as a random-reward null, that the majority-vote update agrees with the gold update as closely as two independent gold updates agree with each other, and that its noise-corrected cosine with the gold update is 1 within error. Randomly flipping up to 30\% of gold labels leaves the update direction intact, while the same fraction flipped by a shared rule destroys it, and rewards that favor a kind of response on every prompt, such as a learned reward model, fall measurably short. The equivalence holds at every checkpoint we examine but is local: gold-reward training still ends ahead on held-out majority accuracy. Error coherence, not error rate, determines when self-generated labels can stand in for verification.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.20898 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:17.881Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).