Metraipolis › archive › dc_30c5705c8647

BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design — arXiv:2611.09825

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.09825
License (as archived)
CC0-1.0
Archived
2026-11-09T22:34:18.320Z
SHA-256 of the archived text
1414ca31c8a768f1fa2cdb1f478a2e7a76737b64506a826f8c264ccf45447f12
Stable URL of this page
https://metraipolis.com/documento/dc_30c5705c8647

How to cite

BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design — arXiv:2611.09825 — arXiv (en), https://arxiv.org/abs/2611.09825 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_30c5705c8647 (Metraipolis archive dc_30c5705c8647, sha256 1414ca31c8a768f1fa2cdb1f478a2e7a76737b64506a826f8c264ccf45447f12)
BibTeX
@misc{dc30c5705c8647,
  title = {BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design — arXiv:2611.09825},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.09825},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_30c5705c8647; sha256 1414ca31c8a768f1fa2cdb1f478a2e7a76737b64506a826f8c264ccf45447f12},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.09825. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving. While 4-bit microscaling formats (NVFP4, MXFP4) shrink the cache and run attention on FP4 Tensor Cores, a fundamental gap remains between algorithmic quantization and system-aware acceleration: existing FP4 KV systems use vanilla quantization that causes severe accuracy drops, whereas accuracy-oriented 4-bit algorithms rely on vector codebooks whose lookup overhead degrades decoding throughput. We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding. We mitigate three error sources: Dual-Residual Query Quantization splits queries into two FP4 tensors, cutting query logit error from 60% to 1%; Underflow-Aware Probability Scaling prevents E4M3 scales from rounding long-context probabilities to zero; and Outlier-Aware Key Quantization combines static smoothing, Hadamard rotation, and scale search to suppress key outliers under token-wise scaling. At the system level, we remove the softmax bottleneck via transposed softmax with fused probability quantization and stacked query MMA on paged KV caches. Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within 0.5 points of BF16 KV accuracy with NVFP4 (1 point with MXFP4), and tracks BF16 KV in RL training rollouts. In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers 2.4-3.0$\times$ higher peak serving throughput than BF16 KV. The implementation is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Hardware Architecture (cs.AR); Computation and Language (cs.CL); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2611.09825 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.294Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).