Metraipolis › archive › dc_b3928a2ed688

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning — arXiv:2608.04726

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.04726
License (as archived)
CC0-1.0
Archived
2026-10-31T11:12:40.979Z
SHA-256 of the archived text
d46cd4d35d1fefec208c7e42def980ca9971a55b94c3586441d5b23d035deb1a
Stable URL of this page
https://metraipolis.com/documento/dc_b3928a2ed688

How to cite

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning — arXiv:2608.04726 — arXiv (en), https://arxiv.org/abs/2608.04726 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b3928a2ed688 (Metraipolis archive dc_b3928a2ed688, sha256 d46cd4d35d1fefec208c7e42def980ca9971a55b94c3586441d5b23d035deb1a)
BibTeX
@misc{dcb3928a2ed688,
  title = {When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning — arXiv:2608.04726},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04726},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b3928a2ed688; sha256 d46cd4d35d1fefec208c7e42def980ca9971a55b94c3586441d5b23d035deb1a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04726. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multimodal large language models increasingly reason over screenshots and documents where the task itself may be written in pixels. Yet benchmarks usually place questions in text, leaving it unclear whether models use the same instruction equally well across channels. We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed. Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average. Models often transcribe the visual question correctly yet fail to use it, exposing a semantic channel gap beyond OCR. To reduce this gap, we present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view. At matched training cost, our method raises four-benchmark VTS accuracy from 58.0 to 66.3 while preserving accuracy on the original interface, and requires no OCR or region metadata at inference. Reading task-bearing text and grounding it as an instruction for reasoning are distinct capabilities.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.04726 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:02:13.424Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).