Metraipolis › archive › dc_e29fbb0591ef

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention — arXiv:2609.15810

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.15810
License (as archived): CC0-1.0
Archived: 2026-10-16T01:53:36.392Z
SHA-256 of the archived text: aa8ca7c5dddd35f04f4e17dfd444354f2e525c675c4219c0ad4b9fd9b4c8f47c
Stable URL of this page: https://metraipolis.com/documento/dc_e29fbb0591ef

How to cite

VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention — arXiv:2609.15810 — arXiv (en), https://arxiv.org/abs/2609.15810 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e29fbb0591ef (Metraipolis archive dc_e29fbb0591ef, sha256 aa8ca7c5dddd35f04f4e17dfd444354f2e525c675c4219c0ad4b9fd9b4c8f47c)

BibTeX

@misc{dce29fbb0591ef,
  title = {VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention — arXiv:2609.15810},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.15810},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e29fbb0591ef; sha256 aa8ca7c5dddd35f04f4e17dfd444354f2e525c675c4219c0ad4b9fd9b4c8f47c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15810. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values. Prior work smooths queries and keys, but value outliers follow no fixed channel or spatiotemporal structure and remain the dominant source of output error. Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. We propose VC-Attention, a training-free low-bit attention framework that addresses both by pairing Value smoothing with a fused probability Cast. V-Smooth reorders value tokens by lightweight online clustering, so the tokens in a hardware block quantize well together. It quantizes only the residual after subtracting the block mean, and restores that mean from the row sum the online softmax already maintains. ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion. We implement VC-Attention for B200, B300, H200, RTX PRO 6000, and RTX 5090. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, VC-Attention improves fidelity over low-bit baselines, speeds up the attention kernel over BF16 FlashAttention-4 by 1.46-1.59x on datacenter Blackwell and Hopper and by 2.3-3.6x on workstation cards, and generates a clip 1.13-1.19x and 1.36-1.70x faster end to end.

Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CV).
Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.15810 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T01:25:23.704Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).