Metraipolis › archive › dc_be36d43e59e7

The Low-Rank Structure of VLA Reinforcement Learning — arXiv:2609.34599

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34599
License (as archived)
CC0-1.0
Archived
2026-10-07T22:38:26.466Z
SHA-256 of the archived text
27bba0c3b9e0be080389b2ab3716dbbf813946a584216660ce5d2db18a4472f5
Stable URL of this page
https://metraipolis.com/documento/dc_be36d43e59e7

How to cite

The Low-Rank Structure of VLA Reinforcement Learning — arXiv:2609.34599 — arXiv (en), https://arxiv.org/abs/2609.34599 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_be36d43e59e7 (Metraipolis archive dc_be36d43e59e7, sha256 27bba0c3b9e0be080389b2ab3716dbbf813946a584216660ce5d2db18a4472f5)
BibTeX
@misc{dcbe36d43e59e7,
  title = {The Low-Rank Structure of VLA Reinforcement Learning — arXiv:2609.34599},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34599},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_be36d43e59e7; sha256 27bba0c3b9e0be080389b2ab3716dbbf813946a584216660ce5d2db18a4472f5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34599. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, yet how RL reshapes these policies remains poorly understood. We find that RL across widely used flow-based VLA models, including $\pi_{0.5}$ and GR00T~N1.5/N1.6, on LIBERO, ManiSkill, MetaWorld, and CALVIN induces substantially lower-rank parameter updates that are highly concentrated in the action expert's Timestep Modules, a small and previously overlooked component. Through systematic module-replacement experiments, we further show that these modules capture a disproportionate share of the performance gains from RL. We then characterize what is encoded in these Timestep Modules. First, we show that RL specializes them to the discrete denoising timesteps used during rollouts, and that this discrete-timestep training underlies the low-rank updates. Second, we find that among their outputs, the shift vector changes most distinctly under RL, and through probing, we show that shift update directions strongly predict task success (ROC-AUC up to $99.6\%$). Third, we find that the geometry of shift updates reflects task relationships, as their pairwise similarity correlates with cross-task transfer patterns. Building on these findings, we show that steering along shift update directions further improves RL-trained policies without additional RL training. Overall, we provide a systematic understanding of how RL reshapes VLA policies by studying how learned signals are encoded in parameter space, offering insights into more efficient and interpretable VLA post-training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.34599 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.237Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).