Metraipolis › archive › dc_600e4e09aa8f

GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU — arXiv:2610.39761

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.39761
License (as archived)
CC0-1.0
Archived
2026-10-31T06:21:59.630Z
SHA-256 of the archived text
db9d366fc46baba9249ccad0abba203a88773e0794162706f47021e00e47dbfb
Stable URL of this page
https://metraipolis.com/documento/dc_600e4e09aa8f

How to cite

GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU — arXiv:2610.39761 — arXiv (en), https://arxiv.org/abs/2610.39761 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_600e4e09aa8f (Metraipolis archive dc_600e4e09aa8f, sha256 db9d366fc46baba9249ccad0abba203a88773e0794162706f47021e00e47dbfb)
BibTeX
@misc{dc600e4e09aa8f,
  title = {GPU-Resident Speculative Decoding for In-Browser LLM Inference with WebGPU — arXiv:2610.39761},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.39761},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_600e4e09aa8f; sha256 db9d366fc46baba9249ccad0abba203a88773e0794162706f47021e00e47dbfb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39761. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

WebGPU enables private, offline LLM inference inside web pages, but browser dispatch, inter-process communication, synchronous readback and single-token kernels slow decoding and make conventional speculative decoding inefficient. We present a WebGPU runtime that addresses these browser-specific costs. Deterministic small-batch integer kernels accelerate multi-token verification while preserving exact agreement with greedy decoding in the same quantized runtime. A GPU-resident loop performs drafting, verification, acceptance and KV-cache maintenance without a per-token CPU decision. A draft-free context-mixing drafter reuses page content, conversation history and session memory, while a device-aware controller adapts the speculation width and the submission window to measured costs. Across seven models from 0.5B to 8B parameters, 11 tasks and three client GPUs, our runtime decodes 3.5 to 5.4$\times$ faster than WebLLM on a laptop GPU, 4.8 to 6.2$\times$ faster on an integrated GPU and 4.8 to 8.6$\times$ faster on a workstation GPU. The GPU-resident loop and speculation contribute a further 1.8 to 3.2$\times$ over the optimized kernels on the workstation. Outputs match the quantized greedy decoding of the runtime exactly, and the mean GSM8K accuracy matches that of WebLLM. Optional draft heads raise the peak speedup to 10.2$\times$, while session memory provides additional gains without downloading extra weights.

Tema degli esploratori: governance digitale e beni comuni (elenco arXiv cs.CY, nuovi lavori del giorno).
Categorie arXiv: Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2610.39761 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:08:38.260Z dalla pagina https://arxiv.org/list/cs.CY/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).