Metraipolis › archive › dc_6f2ef6780ce0
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training — arXiv:2611.04323
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04323
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T21:37:52.967Z
- SHA-256 of the archived text
e385fadee246a2dd07b3aa89ee359cec71bb09d38da5ed57ee2330c5b33fd952- Stable URL of this page
https://metraipolis.com/documento/dc_6f2ef6780ce0
How to cite
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training — arXiv:2611.04323 — arXiv (en), https://arxiv.org/abs/2611.04323 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f2ef6780ce0 (Metraipolis archive dc_6f2ef6780ce0, sha256 e385fadee246a2dd07b3aa89ee359cec71bb09d38da5ed57ee2330c5b33fd952)
BibTeX
@misc{dc6f2ef6780ce0,
title = {FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training — arXiv:2611.04323},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04323},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f2ef6780ce0; sha256 e385fadee246a2dd07b3aa89ee359cec71bb09d38da5ed57ee2330c5b33fd952},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04323. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The rapid scaling of large language models (LLMs) has made communication overhead a critical bottleneck in distributed model-parallel training. Although existing low-precision compression methods can effectively reduce communication costs, further reducing the communication footprint to ultra-low-bit formats such as FP4 introduces new challenges, including limited numerical range, sub-byte storage overhead, and increased computational cost from fine-grained quantization. In this paper, we present \textbf{FP4Quant}, a micro-grouped FP4 compression framework for efficient model-parallel LLM training. FP4Quant integrates a micro-grouped dual-scale FP4 quantization mechanism to preserve local tensor distribution characteristics, a warp-cooperative FP4 packing scheme to enable efficient sub-byte data representation, and a hybrid two-level FWHT pipeline to mitigate quantization errors with GPU-aware execution optimization. Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56$\times$ end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.04323 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:35:49.534Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).