Metraipolis › archive › dc_8fd141e4981e

Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks — arXiv:2609.08404

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.08404
License (as archived)
CC0-1.0
Archived
2026-10-16T12:58:08.568Z
SHA-256 of the archived text
549c5b25cfb57daf0bd1e6825e1b5d28e45e0c4aa2cb75baec0543163ba7d805
Stable URL of this page
https://metraipolis.com/documento/dc_8fd141e4981e

How to cite

Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks — arXiv:2609.08404 — arXiv (en), https://arxiv.org/abs/2609.08404 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8fd141e4981e (Metraipolis archive dc_8fd141e4981e, sha256 549c5b25cfb57daf0bd1e6825e1b5d28e45e0c4aa2cb75baec0543163ba7d805)
BibTeX
@misc{dc8fd141e4981e,
  title = {Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks — arXiv:2609.08404},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.08404},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8fd141e4981e; sha256 549c5b25cfb57daf0bd1e6825e1b5d28e45e0c4aa2cb75baec0543163ba7d805},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08404. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional \textit{agent-side warming} up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to \textit{environment-side adaptation} by constructing \textbf{F}eedback-\textbf{E}nriched \textbf{E}nvironments (\textbf{FEEs}). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs \textbf{(1)} stabilizes training dynamics by reducing entropy volatility, \textbf{(2)} facilitates proactive state-space exploration in difficult tasks, \textbf{(3) }ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and \textbf{(4) }identifies intra-group feedback consistency as a critical boundary for stable optimization.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.08404 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.355Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).