Metraipolis › archive › dc_efefd1f8164c
SoftWater: Class-Aware Rate Allocation for Softmax Quantization — arXiv:2608.12026
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12026
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T05:40:02.123Z
- SHA-256 of the archived text
b06582f01503b78993adf455220b75c313e98c6dd49f4f0805779426bebc5fb9- Stable URL of this page
https://metraipolis.com/documento/dc_efefd1f8164c
How to cite
SoftWater: Class-Aware Rate Allocation for Softmax Quantization — arXiv:2608.12026 — arXiv (en), https://arxiv.org/abs/2608.12026 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_efefd1f8164c (Metraipolis archive dc_efefd1f8164c, sha256 b06582f01503b78993adf455220b75c313e98c6dd49f4f0805779426bebc5fb9)
BibTeX
@misc{dcefefd1f8164c,
title = {SoftWater: Class-Aware Rate Allocation for Softmax Quantization — arXiv:2608.12026},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12026},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_efefd1f8164c; sha256 b06582f01503b78993adf455220b75c313e98c6dd49f4f0805779426bebc5fb9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12026. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Post-training quantization pipelines routinely leave the softmax output layer in high precision. Yet in small LLMs with modern vocabularies, the head holds 15--30\% of all parameters, so a nominal ``2-bit'' model with an fp16 head can store several times as many bits per weight. We pose softmax-layer quantization as a rate-distortion problem under the KL divergence between the original and quantized output distributions. A second-order analysis reveals a class-aware geometry: quantization error is weighted jointly by feature covariance and class-specific softmax curvature. A separability approximation replaces the $Kn\times Kn$ Cholesky with one $n\times n$ factorization rescaled per class, making the lattice encodable by successive interference cancellation, with both statistics from a single forward pass. The resulting method, SoftWater, gives fine grids to frequent, low-variance classes and coarse grids to rare ones, a large gap under Zipfian token distributions. Across five models from 1B to 32B, SoftWater outperforms the released WaterSIC quantizer (near-optimal under linear-layer WMSE but not output KL) at matched head rates on 59 of 60 test points, using none of that pipeline's refinements and cutting head-induced KL by $6.5\times$--$8.3\times$ at 2 bits. On Llama-3.2-1B-Instruct with quantized bodies, a 2-bit head removes 45--60\% of stored bytes for a $2.9$--$3.7\%$ perplexity increase. Because the class-side statistic comes from calibration data, matching calibration to the deployment domain gives the lowest KL on that domain throughout. On a tied model, a 4-bit head is near-lossless and a 2-bit head costs under 4\% perplexity, making head quantization of such models practical. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.12026 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:18:32.502Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).