Metraipolis › archive › dc_634fbce0f3f2
Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs — arXiv:2610.19613
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.19613
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T20:24:05.092Z
- SHA-256 of the archived text
43e9ae0ccc9241640b9b8a296c2845ccffc532033f19d208595d347feedb4677- Stable URL of this page
https://metraipolis.com/documento/dc_634fbce0f3f2
How to cite
Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs — arXiv:2610.19613 — arXiv (en), https://arxiv.org/abs/2610.19613 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_634fbce0f3f2 (Metraipolis archive dc_634fbce0f3f2, sha256 43e9ae0ccc9241640b9b8a296c2845ccffc532033f19d208595d347feedb4677)
BibTeX
@misc{dc634fbce0f3f2,
title = {Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs — arXiv:2610.19613},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.19613},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_634fbce0f3f2; sha256 43e9ae0ccc9241640b9b8a296c2845ccffc532033f19d208595d347feedb4677},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19613. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
As machine unlearning for multimodal large language models (MLLMs) rapidly advances, rigorous evaluation of its effectiveness becomes increasingly important. However, existing benchmarks largely inherit evaluation protocols from text-only unlearning and overlook the unique challenges of unlearning in multimodal settings. In this work, we identify three major pitfalls that may lead to misleading conclusions: the lack of heldout evaluation to distinguish genuine concept-wise forgetting from instance-wise overfitting, the reliance on synthetic, fine-tuned knowledge rather than real-world knowledge acquired during pre-training, and unfair method comparisons caused by applying the same training settings to different unlearning methods. To address these challenges, we introduce GEMMU (Generalizable Evaluation for Multimodal Machine Unlearning), a benchmark built on real-world knowledge and a principled evaluation protocol. Re-evaluating representative unlearning methods under GEMMU reveals substantial generalization gaps and challenges conclusions drawn from previous benchmarks. Our findings highlight the importance of rigorous evaluation for advancing multimodal machine unlearning and provide guidelines for future benchmark design. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.19613 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T19:36:18.205Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).