Metraipolis › archive › dc_5eb52f301eb5
Metropolis-Hastings Dominates Importance Resampling for Policy Composition — arXiv:2610.03480
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.03480
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T23:10:19.320Z
- SHA-256 of the archived text
753eecd66ca7532e1fe9693f50324f4c40ebdb0cae04ee1b5c805a32cc91e46f- Stable URL of this page
https://metraipolis.com/documento/dc_5eb52f301eb5
How to cite
Metropolis-Hastings Dominates Importance Resampling for Policy Composition — arXiv:2610.03480 — arXiv, https://arxiv.org/abs/2610.03480 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5eb52f301eb5 (Metraipolis archive dc_5eb52f301eb5, sha256 753eecd66ca7532e1fe9693f50324f4c40ebdb0cae04ee1b5c805a32cc91e46f)
BibTeX
@misc{dc5eb52f301eb5,
title = {Metropolis-Hastings Dominates Importance Resampling for Policy Composition — arXiv:2610.03480},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.03480},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5eb52f301eb5; sha256 753eecd66ca7532e1fe9693f50324f4c40ebdb0cae04ee1b5c805a32cc91e46f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03480. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Post-training a large language model (LLM) often requires exploring trade-offs between multiple rewards, but retraining for each trade-off is expensive. Decoding-time policy composition allows these trade-offs to be adjusted by combining reward-specific policies at inference time. This composition targets a weighted product of the policies' probabilities over complete responses, but standard implementations combine their next-token probabilities, generally introducing sampling bias. We analyze a known iterative correction based on independence Metropolis-Hastings (MH). Our main result shows that, for every rollout budget, MH produces an output distribution at least as close to the target as sampling-importance-resampling (SIR) with the same budget, as measured by every convex f-divergence. We also derive a lower bound on MH's improvement over the uncorrected decoder in a consensus objective measuring agreement with the supplied policies. We further characterize the correction's sampling error in two asymptotic regimes: when the reward-specific policies approach agreement, and when the log ratio between target and uncorrected-decoder probabilities fluctuates increasingly widely, as can happen for long responses. We complement our analysis with experiments in enumerable and LLM-scale settings. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.03480 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T23:01:38.698Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).