Metraipolis › archive › dc_aee461845f62
Mitigating Exploration Bias in RL for Multi-Instruction Following — arXiv:2608.23830
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.23830
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:17:55.839Z
- SHA-256 of the archived text
16847ea71aeba34aaf1aa94cca436945909f6b28f4e7fc21ff4477978593c185- Stable URL of this page
https://metraipolis.com/documento/dc_aee461845f62
How to cite
Mitigating Exploration Bias in RL for Multi-Instruction Following — arXiv:2608.23830 — arXiv (en), https://arxiv.org/abs/2608.23830 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aee461845f62 (Metraipolis archive dc_aee461845f62, sha256 16847ea71aeba34aaf1aa94cca436945909f6b28f4e7fc21ff4477978593c185)
BibTeX
@misc{dcaee461845f62,
title = {Mitigating Exploration Bias in RL for Multi-Instruction Following — arXiv:2608.23830},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.23830},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aee461845f62; sha256 16847ea71aeba34aaf1aa94cca436945909f6b28f4e7fc21ff4477978593c185},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23830. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
RL has emerged as a powerful paradigm for enhancing the instruction following capabilities of LLMs. While existing training recipes achieve substantial gains, we find that they suffer from exploration bias towards easy instructions when the training data has multiple instructions in a prompt. This bias is caused by two main reasons: 1) the policy model's initial ability to satisfy hard instructions is too low to trigger successful exploration during RL training, so the optimization is biased towards easy instructions; and 2) canonical RL training recipes typically employ a cumulative reward (the number of instructions fulfilled), treating all instructions equally, which biases the policy model towards fulfilling easy instructions to obtain the same amount of reward. To address these issues, we first propose two metrics to measure the exploration bias in instruction following and then introduce a two-stage framework to alleviate it: 1) Behavioral Bootstrapping, a lightweight rejection sampling fine-tuning stage before RL to activate hard instructions; and 2) Scarcity-Aware Rewards, a new RL reward function that assigns rewards to instructions based on their empirical scarcity. Experiments show that the proposed metrics are highly correlated with model performance, and our methods unleash the potential of RL training: our best models outperform the baselines by a significant margin across three verifiable instruction following benchmarks. We release codes at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.23830 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:17:29.346Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).