Metraipolis › archive › dc_cd650ddd4834

How Far Are We From On-Device VLM Inference? A Measurement Study on Edge NPUs — arXiv:2610.40039

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40039
License (as archived): CC0-1.0
Archived: 2026-10-31T06:21:51.206Z
SHA-256 of the archived text: 9b1a0bf02fa8332d515b4a7bccbd3ca2407db253f566e0e8b133512cfffb28fe
Stable URL of this page: https://metraipolis.com/documento/dc_cd650ddd4834

How to cite

How Far Are We From On-Device VLM Inference? A Measurement Study on Edge NPUs — arXiv:2610.40039 — arXiv (en), https://arxiv.org/abs/2610.40039 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd650ddd4834 (Metraipolis archive dc_cd650ddd4834, sha256 9b1a0bf02fa8332d515b4a7bccbd3ca2407db253f566e0e8b133512cfffb28fe)

BibTeX

@misc{dccd650ddd4834,
  title = {How Far Are We From On-Device VLM Inference? A Measurement Study on Edge NPUs — arXiv:2610.40039},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40039},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd650ddd4834; sha256 9b1a0bf02fa8332d515b4a7bccbd3ca2407db253f566e0e8b133512cfffb28fe},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40039. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diverse optimizations spanning model architecture, inference runtime, and hardware acceleration have been proposed for on-device vision-language model (VLM) deployment on edge NPUs. Still, cross-layer coordination of VLM execution on edge NPUs remains largely underexplored. This paper presents an empirical measurement study across six representative VLMs, three real-world edge platforms, and ten multimodal benchmarks. Our evaluation drills down from end-to-end application-level latency and thermal traces down to low-level kernel-scale execution behaviors, challenging several widely held community assumptions. First, softmax dominates attention costs; even lookup-table-based softmax optimizations cannot eliminate pipeline bubbles that severely underutilize NPU matrix units. Second, theoretical speedups from linear attention vanish under real-NPU execution due to prohibitive tensor-rearrangement overhead. Third, no hardware backend universally dominates autoregressive decoding: attention-tensor-shaped parallelism dictates NPU efficiency and shifts the CPU-NPU performance balance. We further find task-specific Sweet Resolutions for VLMs, which deliver meaningful accuracy gains at only modest latency overhead. Complementing these findings, we provide actionable cross-layer recommendations for hardware architects, runtime developers, and model designers.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.AR, nuovi lavori del giorno).
Categorie arXiv: Performance (cs.PF) ; Hardware Architecture (cs.AR).
Pagina del lavoro: https://arxiv.org/abs/2610.40039 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:08:08.316Z dalla pagina https://arxiv.org/list/cs.AR/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).