Metraipolis › archive › dc_67a20431ff8a

Sketch-Omni: Instance-Level Sketch-Based Visual Prompting for Perception MLLMs — arXiv:2611.04262

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.04262
License (as archived)
CC0-1.0
Archived
2026-11-04T19:03:07.477Z
SHA-256 of the archived text
7659973f56552eccc454edba588f1de5af9cc89542759d0711f5a0f6e8897802
Stable URL of this page
https://metraipolis.com/documento/dc_67a20431ff8a

How to cite

Sketch-Omni: Instance-Level Sketch-Based Visual Prompting for Perception MLLMs — arXiv:2611.04262 — arXiv (en), https://arxiv.org/abs/2611.04262 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_67a20431ff8a (Metraipolis archive dc_67a20431ff8a, sha256 7659973f56552eccc454edba588f1de5af9cc89542759d0711f5a0f6e8897802)
BibTeX
@misc{dc67a20431ff8a,
  title = {Sketch-Omni: Instance-Level Sketch-Based Visual Prompting for Perception MLLMs — arXiv:2611.04262},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04262},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_67a20431ff8a; sha256 7659973f56552eccc454edba588f1de5af9cc89542759d0711f5a0f6e8897802},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04262. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Promptable perception, from GroundingDINO to Rex-Omni, puts one model behind one prompt interface and returns coordinates for any localisation task. Every model in that line was built for photographs; none has been given a sketch. We present Sketch-Omni, to our knowledge the first perception MLLM built to work on sketches and from them. On a sketch, the sketch is the image, and text prompts detect, point, resolve object and spatial referring expressions, and ground affordances in it. From a sketch, the sketch is the prompt, and a drawing of one object localises that object in a photograph, even among several of the same category, a query that neither a category name nor a photo exemplar can pose. Two obstacles stand in the way. A photo-trained model's attention is tuned to colour and texture, which a sketch does not have, and no dataset pairs a drawing with the one instance it depicts among others of its kind. We build the training data instead of collecting it. An edge detector turns every labelled photograph into a sketch-like drawing that keeps the photograph's labels, and two constructions produce sketch-to-photo instance pairs from existing data. On top of this, photo-guided relation alignment passes each photograph and its edge sketch through the same model and trains the sketch pass to reproduce how the photograph pass relates phrases to patches and patches to each other. Not one training label is drawn by hand. Sketch-Omni reaches 91.4 F1 on sketch-prompted localisation against 81.8 for the strongest general-purpose MLLM, with a lead that doubles when the photograph holds several instances of the queried category, and it is best in 39 of 40 evaluation cells across six tasks. Code, model, and a hand-annotated benchmark of 2754 sketch-to-photo links will be released.

Tema degli esploratori: statistica e scienza dei dati (elenco arXiv cs.CV, nuovi lavori del giorno).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.04262 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:39:37.483Z dalla pagina https://arxiv.org/list/cs.CV/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).