Metraipolis › archive › dc_d17cdb1e754f
Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents — arXiv:2609.01245
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.01245
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:34:02.133Z
- SHA-256 of the archived text
d30c4380b374e92009084e926c01892e6ab69fa659bfa662d26e86f5171d358a- Stable URL of this page
https://metraipolis.com/documento/dc_d17cdb1e754f
How to cite
Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents — arXiv:2609.01245 — arXiv (en), https://arxiv.org/abs/2609.01245 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d17cdb1e754f (Metraipolis archive dc_d17cdb1e754f, sha256 d30c4380b374e92009084e926c01892e6ab69fa659bfa662d26e86f5171d358a)
BibTeX
@misc{dcd17cdb1e754f,
title = {Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents — arXiv:2609.01245},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.01245},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d17cdb1e754f; sha256 d30c4380b374e92009084e926c01892e6ab69fa659bfa662d26e86f5171d358a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01245. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.01245 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:15:12.876Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).