Metraipolis › archive › dc_030aa605111f

Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning — arXiv:2608.03875

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.03875
License (as archived): CC0-1.0
Archived: 2026-10-31T12:19:39.362Z
SHA-256 of the archived text: 92e152638bf498fcf1c6b4e37214c86735dacc1771883c9279b1b3ec1086f378
Stable URL of this page: https://metraipolis.com/documento/dc_030aa605111f

How to cite

Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning — arXiv:2608.03875 — arXiv (en), https://arxiv.org/abs/2608.03875 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_030aa605111f (Metraipolis archive dc_030aa605111f, sha256 92e152638bf498fcf1c6b4e37214c86735dacc1771883c9279b1b3ec1086f378)

BibTeX

@misc{dc030aa605111f,
  title = {Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning — arXiv:2608.03875},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03875},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_030aa605111f; sha256 92e152638bf498fcf1c6b4e37214c86735dacc1771883c9279b1b3ec1086f378},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03875. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, limiting their direct utility during deployment. We present Structure-Aware Fine-Tuning (SAFT), a simple, self-supervised method that refines these imperfect reward signals online without access to ground-truth supervision. SAFT leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. We rigorously evaluate SAFT across a spectrum of base model capabilities to demonstrate its versatility. Our results show that SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model, suggesting that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for stabilizing text-conditioned RL and underscores the broader value of incorporating task structure as a general inductive bias.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.03875 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:53.094Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).