Metraipolis › archive › dc_fa2b0609707d

HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments — arXiv:2609.15471

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15471
License (as archived)
CC0-1.0
Archived
2026-10-15T22:47:59.234Z
SHA-256 of the archived text
2ad64813392bc0db76cd5b3f9eed92c76c2cdd552554e714b0cff969a7cc56c6
Stable URL of this page
https://metraipolis.com/documento/dc_fa2b0609707d

How to cite

HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments — arXiv:2609.15471 — arXiv (en), https://arxiv.org/abs/2609.15471 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa2b0609707d (Metraipolis archive dc_fa2b0609707d, sha256 2ad64813392bc0db76cd5b3f9eed92c76c2cdd552554e714b0cff969a7cc56c6)
BibTeX
@misc{dcfa2b0609707d,
  title = {HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments — arXiv:2609.15471},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15471},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa2b0609707d; sha256 2ad64813392bc0db76cd5b3f9eed92c76c2cdd552554e714b0cff969a7cc56c6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15471. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models, but long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness. Existing policyoptimization objectives for RLVR commonly apply importance-sampling correction at either the token level (GRPO, DAPO) or the sequence level (GSPO), imposing different granularities for assigning credit across a response. We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments, assigns soft entropy-based saliency weights, and applies clipped importance-sampling correction at the segment granularity. This provides an intermediate correction unit between token-level GRPO/DAPO and sequence-level GSPO. We evaluate HISPO by fine-tuning Qwen3-1.7B-Base on mathematical reasoning tasks. Across six benchmarks, HISPO improves Pass@8 over the strongest baseline on all benchmarks and matches or exceeds the strongest baseline in Acc@8 on five of them. On AIME25, HISPO improves over GRPO by +3.75 Acc@8 and +3.78 Pass@8, and over GSPO by +2.50 Acc@8 and +1.27 Pass@8. These results suggest that segment-level correction is a promising granularity for RLVR in long-form mathematical reasoning.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.15471 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.321Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).