Metraipolis › archive › dc_94cf59d77bee
Multimodal Model Diffing for Feature Discovery and Control — arXiv:2608.09928
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.09928
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:51:01.897Z
- SHA-256 of the archived text
4464681ef520191b2e1057c59fabd31f10084ce7d5b8cfc10f082bcb2ba53a22- Stable URL of this page
https://metraipolis.com/documento/dc_94cf59d77bee
How to cite
Multimodal Model Diffing for Feature Discovery and Control — arXiv:2608.09928 — arXiv (en), https://arxiv.org/abs/2608.09928 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_94cf59d77bee (Metraipolis archive dc_94cf59d77bee, sha256 4464681ef520191b2e1057c59fabd31f10084ce7d5b8cfc10f082bcb2ba53a22)
BibTeX
@misc{dc94cf59d77bee,
title = {Multimodal Model Diffing for Feature Discovery and Control — arXiv:2608.09928},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.09928},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_94cf59d77bee; sha256 4464681ef520191b2e1057c59fabd31f10084ce7d5b8cfc10f082bcb2ba53a22},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09928. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training; (ii) task-specific feature detection, via per-token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.09928 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:49:42.014Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).