Metraipolis › archive › dc_8a6b0eb8f1dc

Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models — arXiv:2609.15671

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.15671
License (as archived)
CC0-1.0
Archived
2026-10-15T22:54:00.501Z
SHA-256 of the archived text
29a7a11c59cb388ab4d78a7e92be7019c0a24282654ec7e9e542d1e873672428
Stable URL of this page
https://metraipolis.com/documento/dc_8a6b0eb8f1dc

How to cite

Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models — arXiv:2609.15671 — arXiv (en), https://arxiv.org/abs/2609.15671 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8a6b0eb8f1dc (Metraipolis archive dc_8a6b0eb8f1dc, sha256 29a7a11c59cb388ab4d78a7e92be7019c0a24282654ec7e9e542d1e873672428)
BibTeX
@misc{dc8a6b0eb8f1dc,
  title = {Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models — arXiv:2609.15671},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15671},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8a6b0eb8f1dc; sha256 29a7a11c59cb388ab4d78a7e92be7019c0a24282654ec7e9e542d1e873672428},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15671. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Visual Question Answering (VQA) with Vision-Language Models (VLMs) is increasingly used in privacy-sensitive and bandwidth-constrained settings. Federated Learning (FL), Split Learning (SL), and U-Shaped Split Learning (USL) keep raw data local, but transmitting all visual tokens across a model partition remains costly and can expose private information. We propose QPriv-VL, a question-guided, privacy-aware token-pruning framework for FL, SL, and USL that prunes visual tokens before transmission based on task utility and privacy sensitivity. Its core component is a lightweight Dynamic Threshold Predictor (DTP) that jointly estimates a sample-specific pruning ratio and a token-level retention mask in one forward pass. DTP combines question relevance, computed from cross-modal similarity between visual patches and the pooled question embedding, with a sensitivity signal derived from frozen DINOv2 features. This allows the model to suppress potentially sensitive regions while preserving patches useful for answering the question, without requiring sensitivity labels. We evaluate QPriv-VL on GQA, OK-VQA, VQAv2, SLAKE, VQA-RAD, and PathVQA against four privacy attack families: FSHA, FORA, iDLG, and attribute-inference membership inference attacks. DTP matches or outperforms fixed-ratio pruning while using substantially fewer transmitted tokens. On VQA-RAD, it reduces membership-inference attack success from 0.99 to 0.76-0.79, lowers FSHA and FORA reconstruction PSNR relative to fixed-ratio pruning, and preserves competitive VQA accuracy using about 40% of the original visual-token budget. A sensitivity exclusion ratio of 1.20 +/- 0.18 indicates preferential removal of privacy-sensitive patches, while explainability analysis shows that retention adapts to question semantics rather than generic visual saliency.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV) ; Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2609.15671 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T22:29:03.911Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).