Metraipolis › archive › dc_f2dfd0726e3c

The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning — arXiv:2611.03260

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.03260
License (as archived): CC0-1.0
Archived: 2026-11-03T11:23:09.810Z
SHA-256 of the archived text: b2efeab04d1486f99946d182ad45ef0f0ea222f233a2f21afe4c289a75d046cf
Stable URL of this page: https://metraipolis.com/documento/dc_f2dfd0726e3c

How to cite

The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning — arXiv:2611.03260 — arXiv (en), https://arxiv.org/abs/2611.03260 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f2dfd0726e3c (Metraipolis archive dc_f2dfd0726e3c, sha256 b2efeab04d1486f99946d182ad45ef0f0ea222f233a2f21afe4c289a75d046cf)

BibTeX

@misc{dcf2dfd0726e3c,
  title = {The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning — arXiv:2611.03260},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03260},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f2dfd0726e3c; sha256 b2efeab04d1486f99946d182ad45ef0f0ea222f233a2f21afe4c289a75d046cf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03260. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training, yet its reliable application to reinforcement learning with verifiable rewards (RLVR) remains largely unexplored. Motivated by the distinct data regimes of on-policy RL and fixed-data optimization, we develop a generic variance-theoretic framework to investigate what limits the reliability of ZO gradient estimation. Our analysis reveals that estimation reliability is structurally determined by whether the two finite-difference evaluations are performed on matched data. On-policy reinforcement learning (RL) under the mismatched-data regime produces a $1/\mu^2$ oracle-noise amplification that sets an irreducible noise floor as the perturbation scale $\mu$ decreases, whereas Boltzmann Targeted SFT (BOLT) under the matched-data regime eliminates this amplification via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. These results naturally lead to ZO-BOLT, which applies ZO optimization to BOLT's RLVR-aligned weighted-SFT objective. Controlled perturbation-scale sweeps and matched-versus-mismatched end-to-end comparisons validate the predicted effect of data matching. Experiments across three model variants and two mathematical reasoning benchmarks further show that ZO-BOLT consistently improves over the pretrained baseline while retaining a near-inference memory footprint. The code is publicly available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.03260 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.686Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).