Metraipolis › archive › dc_705258983056

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards — arXiv:2610.11519

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11519
License (as archived)
CC0-1.0
Archived
2026-10-09T06:06:29.763Z
SHA-256 of the archived text
ee54211a816f6d69079f447f35c8497a4a7f68e795a198cfa511654cfc678786
Stable URL of this page
https://metraipolis.com/documento/dc_705258983056

How to cite

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards — arXiv:2610.11519 — arXiv (en), https://arxiv.org/abs/2610.11519 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_705258983056 (Metraipolis archive dc_705258983056, sha256 ee54211a816f6d69079f447f35c8497a4a7f68e795a198cfa511654cfc678786)
BibTeX
@misc{dc705258983056,
  title = {Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards — arXiv:2610.11519},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11519},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_705258983056; sha256 ee54211a816f6d69079f447f35c8497a4a7f68e795a198cfa511654cfc678786},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11519. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have become two main paradigms for post-training reasoning models. RLVR gives each response a single outcome label, leaving the steps inside it without separate credit. OPD provides token-level guidance at student-visited prefixes, but its pointwise signal does not directly reflect the pattern of teacher--student disagreement across the vocabulary. Dense, unbounded log-ratio supervision can amplify the teacher's influence, yet a strong solver is not necessarily a suitable guide when the student's solution paths depart from the teacher's. We propose Residual Advantage (\RA{}), which treats the teacher--student probability residual as a bounded one-step reward, subtracts the corresponding state value under the student policy to form a standard advantage, and centers the result within each response before adding it to the verifier advantage. The guidance term has zero mean within each response, so the verifier advantage remains the response's mean label and the teacher only redistributes credit among the steps within it. \CoRA{} further updates a teacher LoRA with verifier advantages on the same scored student batch and uses the updated teacher in the next iteration's residual, adapting guidance to the student's attempts. With Qwen3-1.7B-Base and Qwen3-4B-Base students and a Qwen3-8B teacher, \RA{} combined with GRPO or REINFORCE++ improves the underlying sequence-advantage algorithm in all 24 comparisons on three mathematical benchmarks, raising macro Avg@8 by 1.7--3.6 points and Pass@8 by 3.9--6.3 points. Both combinations surpass teacher-only OPD, and \CoRA{} adds a further 1.0--1.5 Avg@8 points.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.11519 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:15.210Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).