Metraipolis › archive › dc_dbb9d2eec2d7

When Do Frozen VLMs Respond to Image-Free Object-Token Edits? An Answer-Key-Free Protocol and What It Reveals — arXiv:2609.03429

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03429
License (as archived)
CC0-1.0
Archived
2026-10-17T02:56:17.739Z
SHA-256 of the archived text
0f5965c46f3a92c886153fee38bbcbd85ce9ca531a84d7884d8bd6208100a959
Stable URL of this page
https://metraipolis.com/documento/dc_dbb9d2eec2d7

How to cite

When Do Frozen VLMs Respond to Image-Free Object-Token Edits? An Answer-Key-Free Protocol and What It Reveals — arXiv:2609.03429 — arXiv (en), https://arxiv.org/abs/2609.03429 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dbb9d2eec2d7 (Metraipolis archive dc_dbb9d2eec2d7, sha256 0f5965c46f3a92c886153fee38bbcbd85ce9ca531a84d7884d8bd6208100a959)
BibTeX
@misc{dcdbb9d2eec2d7,
  title = {When Do Frozen VLMs Respond to Image-Free Object-Token Edits? An Answer-Key-Free Protocol and What It Reveals — arXiv:2609.03429},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03429},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dbb9d2eec2d7; sha256 0f5965c46f3a92c886153fee38bbcbd85ce9ca531a84d7884d8bd6208100a959},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03429. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Answering what-if queries about a scene with a VLM usually means injecting the assumption as text or repainting the scene with a generative model. We instead move the edit to the representation level, before the model input. The image is abstracted into a set of object-level tokens, and the original image never enters the VLM. This design rests on an open question: when do frozen VLMs actually respond to such token edits? We introduce an answer-key-free protocol: no post-edit answer is annotated. It scores edits whose answers are logically determined, and audits itself by reversing each scoreable choice. The protocol reveals three structures. The response is not free: explicit edit teaching, not ordinary VQA training, produces it in dense scenes and multiplies it in sparse ones, on all three operations. Once on, it is governed by token cleanliness and density, with deployable detector+segmenter tokens competitive with the oracle and outperforming it on VRSBench. And reading is a separable axis: the image-free token route preserves 92-96% of a matched patch-token baseline's free-text VQA, and the answers measurably depend on the tokens. The response, cleanliness, and reading structures are sign-preserved across two remote-sensing datasets (iSAID, VRSBench) and three frozen LM backbones. We release the probe generator, records, judge logs, and code.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.03429 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T02:27:18.959Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).