Metraipolis › archive › dc_2a1ae83a7367

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents — arXiv:2610.23148

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.23148
License (as archived)
CC0-1.0
Archived
2026-10-19T08:48:37.725Z
SHA-256 of the archived text
a59c96bf01a3257948c0ffc6b256d342aa85a2df58b345f21ffbea5d12c5403c
Stable URL of this page
https://metraipolis.com/documento/dc_2a1ae83a7367

How to cite

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents — arXiv:2610.23148 — arXiv (en), https://arxiv.org/abs/2610.23148 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2a1ae83a7367 (Metraipolis archive dc_2a1ae83a7367, sha256 a59c96bf01a3257948c0ffc6b256d342aa85a2df58b345f21ffbea5d12c5403c)
BibTeX
@misc{dc2a1ae83a7367,
  title = {Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents — arXiv:2610.23148},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.23148},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2a1ae83a7367; sha256 a59c96bf01a3257948c0ffc6b256d342aa85a2df58b345f21ffbea5d12c5403c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23148. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it. We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the evidence-supported share of correct answers falls from 84.0% to 42.9%. We attribute this to GRPO's credit assignment: rollouts that retrieve answer-bearing evidence are favoured in aggregate only in proportion to their accuracy gain over the rest, a grounding gap that answering from memory or misusing evidence shrinks or reverses. To address this, we propose Grounded Credit Assignment (GCA), which uses verifiable sub-outcomes (evidence retrieval and answer correctness) to correct search and answer credit separately: a wrong answer is blamed more than the search that found the evidence, and a correct answer that bypasses retrievable evidence is credited less. Never negative on evidence-finding search, the correction does not shrink with the grounding gap; capped at 10% of each token's advantage, it never overrides the outcome signal. Without extra models, rollouts, or training stages, GCA improves average EM over Search-R1 by 5.7 (Qwen2.5-3B) and 6.3 points (Qwen3-30B-A3B) and raises this share from 76.9% to 93.4%.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.23148 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:42:40.261Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).