Metraipolis › archive › dc_7d79ad56012a

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets — arXiv:2610.40287

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40287
License (as archived)
CC0-1.0
Archived
2026-10-31T06:14:50.034Z
SHA-256 of the archived text
45b9132fed82e103d32c0ca1cc87767236e9fad551811fc4caca4b48d50da1a5
Stable URL of this page
https://metraipolis.com/documento/dc_7d79ad56012a

How to cite

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets — arXiv:2610.40287 — arXiv (en), https://arxiv.org/abs/2610.40287 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d79ad56012a (Metraipolis archive dc_7d79ad56012a, sha256 45b9132fed82e103d32c0ca1cc87767236e9fad551811fc4caca4b48d50da1a5)
BibTeX
@misc{dc7d79ad56012a,
  title = {Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets — arXiv:2610.40287},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40287},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d79ad56012a; sha256 45b9132fed82e103d32c0ca1cc87767236e9fad551811fc4caca4b48d50da1a5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40287. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2610.40287 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T06:07:08.762Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).