Metraipolis › archive › dc_6a2e0decf555

Instruction Distillation: Text Instructions as Visual Examples — arXiv:2608.28696

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28696
License (as archived)
CC0-1.0
Archived
2026-10-17T09:53:15.944Z
SHA-256 of the archived text
4e09f65cb21170aa7721df3751d47b6bf02708ebc7bb4221d5d330de2fba39ac
Stable URL of this page
https://metraipolis.com/documento/dc_6a2e0decf555

How to cite

Instruction Distillation: Text Instructions as Visual Examples — arXiv:2608.28696 — arXiv (en), https://arxiv.org/abs/2608.28696 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6a2e0decf555 (Metraipolis archive dc_6a2e0decf555, sha256 4e09f65cb21170aa7721df3751d47b6bf02708ebc7bb4221d5d330de2fba39ac)
BibTeX
@misc{dc6a2e0decf555,
  title = {Instruction Distillation: Text Instructions as Visual Examples — arXiv:2608.28696},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28696},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6a2e0decf555; sha256 4e09f65cb21170aa7721df3751d47b6bf02708ebc7bb4221d5d330de2fba39ac},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28696. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved image example consumes several hundred context tokens, making large-$K$ settings prohibitively expensive at inference scale. We propose Instruction Distillation: an offline procedure in which the MLLM itself generates, for each individual training image, a structured identification instruction encoding general appearance cues, features that differentiate the class from visually similar ones, and a common confusion point. Unlike prior work that produces a single description per class, our instructions are generated per training image, preserving the intra-class visual diversity that per-class descriptions collapse. At inference time, we study five configurations sharing a single CLIP retrieval index: zero-shot, image ICL, instruction-only ICL, and two hybrid variants in which retrieved neighbors are split between images and instructions. Across seven fine-grained benchmarks and two MLLM backbones, instruction based pipelines match, or exceeds image ICL at $K{=}1$ and reduces per-query tokens by $2.9\times$ and inference latency by $3.3\times$ at $K{=}5$. Hybrid configurations further show that visual and textual ICL signals are complementary, images give visual patterns to learn and see, while instructions give explicit rules and logic. When both of these are provided, the quality of context improves, which is noticeable in the performance.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-01 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2608.28696 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T09:28:47.939Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).