Metraipolis › archive › dc_75d2c06ea63a
FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience — arXiv:2609.03241
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.03241
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T01:11:46.876Z
- SHA-256 of the archived text
2f1a4ede7c3c1a5f71ef48a875588fc05ce7d702d28df9ba36ff01a3d939d4a1- Stable URL of this page
https://metraipolis.com/documento/dc_75d2c06ea63a
How to cite
FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience — arXiv:2609.03241 — arXiv (en), https://arxiv.org/abs/2609.03241 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_75d2c06ea63a (Metraipolis archive dc_75d2c06ea63a, sha256 2f1a4ede7c3c1a5f71ef48a875588fc05ce7d702d28df9ba36ff01a3d939d4a1)
BibTeX
@misc{dc75d2c06ea63a,
title = {FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience — arXiv:2609.03241},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.03241},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_75d2c06ea63a; sha256 2f1a4ede7c3c1a5f71ef48a875588fc05ce7d702d28df9ba36ff01a3d939d4a1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03241. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.03241 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T00:28:02.946Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).