Metraipolis › archive › dc_debfdc2e2458
Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment — arXiv:2609.08188
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.08188
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T15:05:29.935Z
- SHA-256 of the archived text
8ab7541840335eea473fe61890a40a449c4412d31f5bbcae6a4a39e282891525- Stable URL of this page
https://metraipolis.com/documento/dc_debfdc2e2458
How to cite
Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment — arXiv:2609.08188 — arXiv (en), https://arxiv.org/abs/2609.08188 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_debfdc2e2458 (Metraipolis archive dc_debfdc2e2458, sha256 8ab7541840335eea473fe61890a40a449c4412d31f5bbcae6a4a39e282891525)
BibTeX
@misc{dcdebfdc2e2458,
title = {Bridging the Semantic-Utility Gap in Multimodal RAG via Generator-in-the-Loop Alignment — arXiv:2609.08188},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.08188},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_debfdc2e2458; sha256 8ab7541840335eea473fe61890a40a449c4412d31f5bbcae6a4a39e282891525},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.08188. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Vision-language models (VLMs) augmented with retrieval-augmented generation (RAG) benefit from access to external evidence. However, standard retrievers and rerankers optimize for semantic similarity rather than answer utility, creating a preference gap: documents that appear relevant may not help the generator produce a correct answer. Motivated by this, we propose a two-stage generator-in-the-loop alignment framework that closes this gap without human document-level relevance annotations. Our framework consists of two stages: in Stage 1, a VLM generates a hypothetical text passage from the image-query pair, which is used as the retrieval query for dense text search, bridging the image-to-text modality gap. In Stage 2, a cross-encoder reranker adapted with low-rank adaptation (LoRA) is fine-tuned using answer-supervised preference pairs mined from the frozen VLM: given the dataset answer label, a candidate document is labeled positive if the VLM produces the correct answer when given that document as context, and negative otherwise. This generator-guided signal is compatible with multiple alignment loss functions, including contrastive (triplet) loss, pairwise direct preference optimization (DPO), and supervised fine-tuning (SFT), and supports periodic re-mining to refresh preference pairs as the reranker improves. Experiments on VQA-X and A-OKVQA with Qwen3.5-2B and Qwen3-VL-4B-Instruct show that our proposed framework consistently outperforms rank-order, random, and REPLUG-style likelihood baselines under various alignment losses and pool size settings, suggesting that answer-level generator feedback is an effective supervision signal for preference alignment. Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-09 in cs.IR). Categorie arXiv: Artificial Intelligence (cs.AI) ; Information Retrieval (cs.IR). Pagina del lavoro: https://arxiv.org/abs/2609.08188 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T14:49:33.982Z dalla pagina https://arxiv.org/catchup/cs.IR/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).