Metraipolis › archive › dc_99b7dd095cbf

Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery — arXiv:2609.19164

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.19164
License (as archived)
CC0-1.0
Archived
2026-10-10T03:47:58.510Z
SHA-256 of the archived text
0ca0a4469ce42bc5f86e08e8fabed74623d3a4400fae1282db1dd0c0750211c6
Stable URL of this page
https://metraipolis.com/documento/dc_99b7dd095cbf

How to cite

Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery — arXiv:2609.19164 — arXiv (en), https://arxiv.org/abs/2609.19164 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_99b7dd095cbf (Metraipolis archive dc_99b7dd095cbf, sha256 0ca0a4469ce42bc5f86e08e8fabed74623d3a4400fae1282db1dd0c0750211c6)
BibTeX
@misc{dc99b7dd095cbf,
  title = {Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery — arXiv:2609.19164},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.19164},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_99b7dd095cbf; sha256 0ca0a4469ce42bc5f86e08e8fabed74623d3a4400fae1282db1dd0c0750211c6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19164. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In verifier-style RLVR, group-relative optimization often treats advantage scale as an implementation detail. This paper separates two low-variance cases: sub-resolution jitter that should not become a preference signal, and credible but small cardinal gaps that should be learned without distorting KL calibration. We propose an advantage-scale three-way calibration interface: the same within-group scale denominator simultaneously determines the reward-branch strength, prompt-level batch weight, and the effective KL calibration induced when the reward branch is re-expressed on the original cardinal scale. This interface explains why RLOO / this http URL can let credible small gaps become KL dominated, whereas GRPO's standard-deviation denominator can amplify tiny gaps without bound. Based on this interface, we further introduce the Reward-Resolution Protocol and MaxNorm-AC, respectively filtering sub-resolution gaps and providing bounded cardinal recovery on credible nonzero gaps. Across dense / MoE architectures and math / code reasoning, MaxNorm-AC improves over the strongest robust-scale baseline while truncating the low-variance inverse-scale tail.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.19164 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-10T03:24:08.116Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).