Metraipolis › archive › dc_d0cbbce38d18

Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding — arXiv:2609.02897

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.02897
License (as archived)
CC0-1.0
Archived
2026-10-17T01:37:17.619Z
SHA-256 of the archived text
536e7d96f5da2b9a44787d573b46092768ac4c68d5766ffbecbcd14bbfa6db6d
Stable URL of this page
https://metraipolis.com/documento/dc_d0cbbce38d18

How to cite

Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding — arXiv:2609.02897 — arXiv (en), https://arxiv.org/abs/2609.02897 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d0cbbce38d18 (Metraipolis archive dc_d0cbbce38d18, sha256 536e7d96f5da2b9a44787d573b46092768ac4c68d5766ffbecbcd14bbfa6db6d)
BibTeX
@misc{dcd0cbbce38d18,
  title = {Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding — arXiv:2609.02897},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.02897},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d0cbbce38d18; sha256 536e7d96f5da2b9a44787d573b46092768ac4c68d5766ffbecbcd14bbfa6db6d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02897. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Speculative decoding accelerates LLM inference by drafting candidate tokens and verifying them in parallel. Tree-attention drafters such as EAGLE-3 are widely adopted, yet typically hold two decisions fixed: (1) a strict token-match verification rule and (2) a static draft-tree shape. Prior work relaxes each in isolation under limiting assumptions: long draft chains for training-free lossy verification, and adaptive tree shaping under a fixed token budget. We introduce AdaptiveSpec, a training-free per-step speculative decoding method that adapts both decisions from internal signals already produced during decoding. A per-step margin rule promotes a mismatched draft-proposed token when the ratio of the target's probability on the drafted token to its top-1 probability exceeds a threshold with no dependence on draft length or underlying drafter architecture. A per-step tree policy adjusts the draft tree's depth, width, and node count directly from a fused signal of draft top-1 confidence and a rolling acceptance history capturing recent draft-target agreement, allowing the total draft count to vary rather than only be redistributed. The two adaptations operate on orthogonal axes and compound in effect. Implemented on the SGLang production-grade serving engine, AdaptiveSpec improves throughput over the state-of-the-art autoregressive speculative decoding method EAGLE-3 by up to 56%, recovering 93% to fully lossless task accuracy across GSM8K, MATH-500, and HumanEval on three target models (DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.02897 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T01:11:43.486Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).