Metraipolis › archive › dc_af295fd596a8
Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs — arXiv:2611.04380
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04380
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T18:34:35.348Z
- SHA-256 of the archived text
ed050cba3d7601579e35370ea480db80e886213298738215a116828f97ecf9a2- Stable URL of this page
https://metraipolis.com/documento/dc_af295fd596a8
How to cite
Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs — arXiv:2611.04380 — arXiv (en), https://arxiv.org/abs/2611.04380 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_af295fd596a8 (Metraipolis archive dc_af295fd596a8, sha256 ed050cba3d7601579e35370ea480db80e886213298738215a116828f97ecf9a2)
BibTeX
@misc{dcaf295fd596a8,
title = {Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs — arXiv:2611.04380},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04380},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_af295fd596a8; sha256 ed050cba3d7601579e35370ea480db80e886213298738215a116828f97ecf9a2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04380. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A team serving a 9B-20B open-weight language model on vLLM or SGLang must choose a quantization recipe, and needs two numbers: the speedup over BF16 and the accuracy given up. We assemble every public measurement of these numbers that we could verify and find that the record supplies them only in pieces. Only two checkpoints, both INT4 builds of Gemma 3 12B, have both a vLLM timing and an accuracy score, and that score covers a single MMLU subject, one that the AWQ build was calibrated on. The batch-1 timings predate current engine releases, AMD and Intel Gaudi have no timing in this band at all, and FP4 has none against BF16. The timings that do exist follow one rule. Decoding a small batch is limited by memory bandwidth, so 4-bit weights roughly double speed (2.12x for Qwen2.5-14B on an A100), but the gain vanishes once a large batch makes the GPU compute-bound (1.02x for Gemma 3 12B at batch size 256), where FP8 is the recipe that can still help. Many published accuracy deltas, including every HumanEval delta, lie within benchmark sampling noise. We close with a seven-item reporting checklist for future measurements. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.04380 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T18:32:37.301Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).