Metraipolis › archive › dc_1995cc9600ed
Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting — arXiv:2609.31169
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.31169
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T01:39:47.544Z
- SHA-256 of the archived text
2ab2e9bbae963e2ad3eaaf820e78aa017b92172e19adea613a78519686cb00a5- Stable URL of this page
https://metraipolis.com/documento/dc_1995cc9600ed
How to cite
Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting — arXiv:2609.31169 — arXiv (en), https://arxiv.org/abs/2609.31169 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1995cc9600ed (Metraipolis archive dc_1995cc9600ed, sha256 2ab2e9bbae963e2ad3eaaf820e78aa017b92172e19adea613a78519686cb00a5)
BibTeX
@misc{dc1995cc9600ed,
title = {Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting — arXiv:2609.31169},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.31169},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1995cc9600ed; sha256 2ab2e9bbae963e2ad3eaaf820e78aa017b92172e19adea613a78519686cb00a5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31169. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multimodal Machine Translation aims to incorporate additional signal from non-textual modalities to improve translations by resolving ambiguities. While models, through multimodal fusion, are able to accept images related to the source text, they can ignore this information. Therefore, increasing their visual sensitivity remains an active research area. In this work, we introduce a training method, Metric-based Loss Weighting, that improves visual grounding of translations by increasing the loss function for tokens that benefit from the accompanying image. We identify these tokens using the Point-wise Cross-mutual Information (PCXMI) metric, which compares the model's output probabilities with and without visual context. We introduce a Congruency-based PCXMI metric and experimentally show that both metrics working in combination yield the best results. We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models on the task of Image-guided Machine Translation for three language directions. Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset, improving accuracy by up to more than 7 percentage points compared to standard fine-tuning, while maintaining strong general translation performance. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.31169 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T01:12:14.550Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).