Metraipolis › archive › dc_afa528a33684

Seeing What Matters, Verifying What Pays: Load-Adaptive Speculative Decoding for Vision-Language Models — arXiv:2611.01617

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01617
License (as archived): CC0-1.0
Archived: 2026-11-03T10:31:49.948Z
SHA-256 of the archived text: a8de26969af13dd41977ad249409781e5698c613cef82d30a1b7eb7b89282c6c
Stable URL of this page: https://metraipolis.com/documento/dc_afa528a33684

How to cite

Seeing What Matters, Verifying What Pays: Load-Adaptive Speculative Decoding for Vision-Language Models — arXiv:2611.01617 — arXiv (en), https://arxiv.org/abs/2611.01617 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_afa528a33684 (Metraipolis archive dc_afa528a33684, sha256 a8de26969af13dd41977ad249409781e5698c613cef82d30a1b7eb7b89282c6c)

BibTeX

@misc{dcafa528a33684,
  title = {Seeing What Matters, Verifying What Pays: Load-Adaptive Speculative Decoding for Vision-Language Models — arXiv:2611.01617},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01617},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_afa528a33684; sha256 a8de26969af13dd41977ad249409781e5698c613cef82d30a1b7eb7b89282c6c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01617. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Speculative decoding accelerates vision-language model (VLM) inference without altering the output distribution, yet two factors limit its gains. Existing drafters either lack the image evidence required for visually grounded tokens or pay to condition on every visual token, although a short draft block typically depends on only a few image regions. Moreover, the value of verifying additional candidates varies with GPU load, so no fixed verification structure is efficient across serving conditions. We present Uni-LaSpec, a block-parallel speculative decoding framework that addresses both factors. To improve draft quality, Uni-LaSpec reuses the verifier's own attention: the query retained from the preceding verification ranks the cached, int8-quantized visual keys, and a cross-layer vote selects a few visual tokens that the drafter fuses through gated cross-attention. Acceptance-aware training further weights each draft position by its contribution to the accepted prefix. To allocate verification, a marginal-cost criterion selects the structure per request and per round: draft trees exploit idle verifier capacity under light load, whereas a dynamic head truncates linear drafts under heavy load. Across six VLMs from three model families and 17 text and multimodal benchmarks, selecting eight visual tokens retains at least 98.6% of the acceptance obtained from the full image at sub-millisecond selection cost. With greedy decoding at concurrency one, Uni-LaSpec achieves up to 5.11x text and 3.24x multimodal throughput over autoregressive decoding. Across six load levels and three GPU types, it outperforms autoregressive decoding in every setting and matches or exceeds the best fixed verification strategy in 123 of 144 settings; under bursty arrivals, it reduces mean end-to-end latency by 47.6% relative to linear verification.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2611.01617 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:29:48.512Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).