Metraipolis › archive › dc_d55f1c444875

Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO — arXiv:2608.24488

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.24488
License (as archived)
CC0-1.0
Archived
2026-10-18T06:36:15.145Z
SHA-256 of the archived text
e5ea4e24dea81fda2e46c303616a23e3e6a7e24313d3a33f219a16b43616d42a
Stable URL of this page
https://metraipolis.com/documento/dc_d55f1c444875

How to cite

Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO — arXiv:2608.24488 — arXiv (en), https://arxiv.org/abs/2608.24488 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d55f1c444875 (Metraipolis archive dc_d55f1c444875, sha256 e5ea4e24dea81fda2e46c303616a23e3e6a7e24313d3a33f219a16b43616d42a)
BibTeX
@misc{dcd55f1c444875,
  title = {Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO — arXiv:2608.24488},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.24488},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d55f1c444875; sha256 e5ea4e24dea81fda2e46c303616a23e3e6a7e24313d3a33f219a16b43616d42a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.24488. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A same-state decomposition shows that this is not due to variance alone: setting variance to zero still leaves 83.83% of actions near a bound, while 82.12% of state-conditioned means lie outside the executable interval. Replacing clipping with a tanh map does not remove the high-variance regime. For latent Gaussian entropy H(u), the entropy loss has zero gradient with respect to the mean and a constant variance-increasing gradient. For executed-action entropy H(a), the transform Jacobian adds an inward gradient on the mean. Across three matched MyoLeg seeds, near-boundary occupancy is 71.42%, 29.76%, and 18.83% under latent entropy, no entropy, and executed-action entropy. A 38-dimensional Dog-Stand replication with an independent CleanRL-based PPO implementation reproduces the ordering in mean geometry, which also survives shared-state evaluation and boundary margins from 1% to 10%. Direct mean penalties can match or exceed the centering produced by H(a), showing that interior means are not unique to executed entropy. However, matched mean geometry can coexist with substantially different variance and return. Entropy measurement space is therefore a coupled mean-variance design choice, and task return alone does not characterize bounded-policy geometry.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.24488 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T06:28:52.251Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).