Metraipolis › archive › dc_e6511a3485fc

NebulaSD: Many-for-Many Speculative Decoding — arXiv:2609.29364

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.29364
License (as archived)
CC0-1.0
Archived
2026-10-08T06:02:22.419Z
SHA-256 of the archived text
445013fa748e1dcb09638892837831571dcb680b9244f095e84ac07ef31a02ba
Stable URL of this page
https://metraipolis.com/documento/dc_e6511a3485fc

How to cite

NebulaSD: Many-for-Many Speculative Decoding — arXiv:2609.29364 — arXiv (en), https://arxiv.org/abs/2609.29364 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e6511a3485fc (Metraipolis archive dc_e6511a3485fc, sha256 445013fa748e1dcb09638892837831571dcb680b9244f095e84ac07ef31a02ba)
BibTeX
@misc{dce6511a3485fc,
  title = {NebulaSD: Many-for-Many Speculative Decoding — arXiv:2609.29364},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.29364},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e6511a3485fc; sha256 445013fa748e1dcb09638892837831571dcb680b9244f095e84ac07ef31a02ba},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29364. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Speculative decoding accelerates Large Language Model (LLM) inference by using a lightweight draft model to propose candidate tokens for parallel verification by a target model. Drafting and verification, however, exhibit different service characteristics and favor different batch configurations, making fixed draft-target coupling inefficient under concurrent workloads. Existing distributed designs can physically separate the two stages, but often retain request or batch affinities that prevent their capacities from being shared globally. We present NebulaSD, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools. Such dynamic reassignment removes fixed worker locality, requiring request states to be made available at newly selected workers without introducing migration stalls. NebulaSD addresses this challenge through worker-triggered batch reconstruction and asynchronous KV-state preparation overlapped with model execution. We evaluate NebulaSD from both system and scaling perspectives, showing that dynamic pooling improves request-round processing rate by 50.4% over a physically disaggregated baseline and 72.6% over co-located execution on a four-GPU deployment while substantially increasing effective GPU utilization. Profile-driven simulations further show approximately proportional compute-side capacity scaling under idealized state movement.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-25 in cs.NI).
Categorie arXiv: Networking and Internet Architecture (cs.NI).
Pagina del lavoro: https://arxiv.org/abs/2609.29364 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T05:26:39.514Z dalla pagina https://arxiv.org/catchup/cs.NI/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).