Metraipolis › archive › dc_610df2b455de

$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource — arXiv:2609.22041

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22041
License (as archived)
CC0-1.0
Archived
2026-10-10T01:45:20.073Z
SHA-256 of the archived text
0267fb4b27df921f9bc164f67c215e2081f450f5f628a6c59dd59e931d986fd5
Stable URL of this page
https://metraipolis.com/documento/dc_610df2b455de

How to cite

$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource — arXiv:2609.22041 — arXiv (en), https://arxiv.org/abs/2609.22041 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_610df2b455de (Metraipolis archive dc_610df2b455de, sha256 0267fb4b27df921f9bc164f67c215e2081f450f5f628a6c59dd59e931d986fd5)
BibTeX
@misc{dc610df2b455de,
  title = {$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource — arXiv:2609.22041},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22041},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_610df2b455de; sha256 0267fb4b27df921f9bc164f67c215e2081f450f5f628a6c59dd59e931d986fd5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22041. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a resource that can be measured and budgeted rather than a collection of symptoms to repair. Our method, $\lambda$-Controlled GRPO, calibrates importance-ratio behavior from this predicted law rather than from noisy empirical statistics, and allocates gradient effort across denoising steps according to their predicted cost. The two scales governing the update are fixed by standard policy choices rather than introduced as free tuning parameters. On a text-to-image model under two reward settings, rendering difficult target text scored by optical character recognition and matching human preferences scored by a preference model, $\lambda$-Controlled GRPO improves both text accuracy and preference reward over the strongest empirical stabilizer. It also keeps late-step path variance within its intended budget, precisely where the baseline systematically overshoots. The result is a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-21 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.22041 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T01:14:16.670Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).