Metraipolis › archive › dc_cd47905ae51d

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR — arXiv:2609.37868

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37868
License (as archived)
CC0-1.0
Archived
2026-10-07T16:27:50.457Z
SHA-256 of the archived text
de711dc4c3415c59f2226785f83184b792720388b8cf75568aa9bd61f84acfb7
Stable URL of this page
https://metraipolis.com/documento/dc_cd47905ae51d

How to cite

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR — arXiv:2609.37868 — arXiv (en), https://arxiv.org/abs/2609.37868 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd47905ae51d (Metraipolis archive dc_cd47905ae51d, sha256 de711dc4c3415c59f2226785f83184b792720388b8cf75568aa9bd61f84acfb7)
BibTeX
@misc{dccd47905ae51d,
  title = {Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR — arXiv:2609.37868},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37868},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd47905ae51d; sha256 de711dc4c3415c59f2226785f83184b792720388b8cf75568aa9bd61f84acfb7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37868. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already have been discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most of the gains, improving over GRPO by 1.8 points on average without simultaneous co-training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.37868 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T15:54:54.042Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).