Metraipolis › archive › dc_eac4bf28b4fe
CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters — arXiv:2610.00321
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.00321
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-06T13:53:51.475Z
- SHA-256 of the archived text
00091aa281789cee8d5345f8d92725f48895af536f23a94e899a29e50e6045d1- Stable URL of this page
https://metraipolis.com/documento/dc_eac4bf28b4fe
How to cite
CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters — arXiv:2610.00321 — arXiv (en), https://arxiv.org/abs/2610.00321 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eac4bf28b4fe (Metraipolis archive dc_eac4bf28b4fe, sha256 00091aa281789cee8d5345f8d92725f48895af536f23a94e899a29e50e6045d1)
BibTeX
@misc{dceac4bf28b4fe,
title = {CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters — arXiv:2610.00321},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.00321},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eac4bf28b4fe; sha256 00091aa281789cee8d5345f8d92725f48895af536f23a94e899a29e50e6045d1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00321. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Speculative decoding accelerates large language model inference by drafting future tokens cheaply and verifying them with the target model in parallel. Block drafters score a whole block of future tokens in one forward pass, yet standard decoding verifies only the top-scoring chain and discards the other candidates. Because these candidates are already scored, verifying more of them adds target computation but no extra drafting. We introduce CAST (Cost-Aware Speculative Trees), which packs these candidates into a tree and verifies it in a single target pass, leaving the target model, drafter weights, and decoding rule untouched. To decide how wide the tree should be, CAST adds candidates while the expected gain from the next one outweighs the verification time it adds. The width therefore adapts to each deployment from a latency measurement, without sweeping over widths. We evaluate CAST across five domains on three GPU generations and two model families. At its predicted width, CAST is faster than the standard chain in all eight settings, by up to 43%. We also find that the best width depends strongly on the deployment. Where verification cost jumps at a kernel boundary, a 128-token tree is only 2% faster than the standard chain, whereas the tree at the predicted width is 20% faster. Furthermore, we prove that CAST leaves the target output distribution unchanged under both greedy and sampled decoding. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.00321 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-06T12:15:34.613Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).