Metraipolis › archive › dc_87d964ba474f

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL — arXiv:2609.20715

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.20715
License (as archived)
CC0-1.0
Archived
2026-10-10T03:24:23.971Z
SHA-256 of the archived text
d30752e90191b38f9c854ae5a8f90c43f9e511001cb4896966a9c21194708c85
Stable URL of this page
https://metraipolis.com/documento/dc_87d964ba474f

How to cite

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL — arXiv:2609.20715 — arXiv (en), https://arxiv.org/abs/2609.20715 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_87d964ba474f (Metraipolis archive dc_87d964ba474f, sha256 d30752e90191b38f9c854ae5a8f90c43f9e511001cb4896966a9c21194708c85)
BibTeX
@misc{dc87d964ba474f,
  title = {Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL — arXiv:2609.20715},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.20715},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_87d964ba474f; sha256 d30752e90191b38f9c854ae5a8f90c43f9e511001cb4896966a9c21194708c85},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.20715. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.20715 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:04:13.983Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).