Metraipolis › archive › dc_0db265ce73a5

Perturb, Repair, Verify: Self-Play Vision-Language Verifiers for Compositional Understanding — arXiv:2611.05158

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05158
License (as archived)
CC0-1.0
Archived
2026-11-04T12:42:51.549Z
SHA-256 of the archived text
1a9ece8a227db8913594b55e7f37245fc605f818f22d73d22ded155a9f73cb14
Stable URL of this page
https://metraipolis.com/documento/dc_0db265ce73a5

How to cite

Perturb, Repair, Verify: Self-Play Vision-Language Verifiers for Compositional Understanding — arXiv:2611.05158 — arXiv (en), https://arxiv.org/abs/2611.05158 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0db265ce73a5 (Metraipolis archive dc_0db265ce73a5, sha256 1a9ece8a227db8913594b55e7f37245fc605f818f22d73d22ded155a9f73cb14)
BibTeX
@misc{dc0db265ce73a5,
  title = {Perturb, Repair, Verify: Self-Play Vision-Language Verifiers for Compositional Understanding — arXiv:2611.05158},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05158},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0db265ce73a5; sha256 1a9ece8a227db8913594b55e7f37245fc605f818f22d73d22ded155a9f73cb14},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05158. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Vision-language verifiers judge whether an image faithfully matches a textual description, yet existing training pipelines typically rely on static supervision from stronger external judges or curated hard-negative data. Such supervision requires external generation or annotation, is teacher-bounded, and is difficult to adapt as model failures evolve. We introduce PROVE (Perturb-and-Repair Optimization for Vision-Language Evaluation), a self-play framework that trains verifiers through a closed perturb-verify-repair-verify loop. Starting from a known-correct image-caption pair, a Mutator introduces a typed perturbation over objects, attributes, counts, or relations. The verifier then judges the perturbed caption and provides textual feedback, which a Repairer uses to recover a faithful caption for re-verification. These two verification rounds produce structured rewards for informative mismatches and visually grounded repairs. Since perturbation, verification, and repair are role-conditioned behaviors of the same evolving multimodal backbone, updating the shared parameters also improves the verifier itself. Experiments show that PROVE improves fine-grained compositional verification across model scales and families and strengthens visual-detail robustness on the reference backbone, with limited, benchmark-dependent changes on broader multimodal tasks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.05158 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:29:47.406Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).