Metraipolis › archive › dc_e79e1da02ccd

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards — arXiv:2609.03342

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.03342
License (as archived): CC0-1.0
Archived: 2026-10-17T02:07:50.104Z
SHA-256 of the archived text: 5afee49f735fc6367211a456962735e78133e327d894429e897fef39f6549938
Stable URL of this page: https://metraipolis.com/documento/dc_e79e1da02ccd

How to cite

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards — arXiv:2609.03342 — arXiv (en), https://arxiv.org/abs/2609.03342 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e79e1da02ccd (Metraipolis archive dc_e79e1da02ccd, sha256 5afee49f735fc6367211a456962735e78133e327d894429e897fef39f6549938)

BibTeX

@misc{dce79e1da02ccd,
  title = {Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards — arXiv:2609.03342},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03342},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e79e1da02ccd; sha256 5afee49f735fc6367211a456962735e78133e327d894429e897fef39f6549938},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03342. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.03342 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T01:47:25.540Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).