Metraipolis › archive › dc_04ea20dfe829
Shift-Accumulate Attention: Multiplier-Free Query--Key Products for Transformer Decoding — arXiv:2609.09208
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.09208
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T07:54:06.192Z
- SHA-256 of the archived text
40753c328d7d7ca1414857eb4b79367a002c07f1d71da526e6c657e3e2e4aed7- Stable URL of this page
https://metraipolis.com/documento/dc_04ea20dfe829
How to cite
Shift-Accumulate Attention: Multiplier-Free Query--Key Products for Transformer Decoding — arXiv:2609.09208 — arXiv (en), https://arxiv.org/abs/2609.09208 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_04ea20dfe829 (Metraipolis archive dc_04ea20dfe829, sha256 40753c328d7d7ca1414857eb4b79367a002c07f1d71da526e6c657e3e2e4aed7)
BibTeX
@misc{dc04ea20dfe829,
title = {Shift-Accumulate Attention: Multiplier-Free Query--Key Products for Transformer Decoding — arXiv:2609.09208},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.09208},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_04ea20dfe829; sha256 40753c328d7d7ca1414857eb4b79367a002c07f1d71da526e6c657e3e2e4aed7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09208. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Power-of-two (PoT) quantisation turns a multiplication into a bit shift, so far only for the post-softmax attention--value product. The earlier and larger product, S=QK^T, has not been reformulated the same way. We quantise the key cache to a signed power-of-two fixed-point code, so that every scalar multiplication in QK^T becomes a sign flip, a bit shift and an integer accumulation. A fixed-point head-room F>= e_max turns every shift index r=F-e into a non-negative left shift, making the accumulation exact in integer arithmetic. We add a sub-power mantissa extension that halves the score error for one extra shift-add, a shift-exact online softmax whose running maximum lives in an integer log2 domain so that every rescaling is itself an exact shift, and a log-quantised AV product. Fused CUDA kernels in a 1.1B Llama decoder on an RTX 4090: at B=8, T=32k the 4-bit kernel is 4.60x faster than FP16 scaled dot-product attention with a 2.5x smaller KV cache, and batched decoding crosses over at B=64 to 1.22x FP16 throughput at 11.02 against 13.77 GB peak memory. A matched INT8 multiply--accumulate kernel on the same tiling reaches 5.29x: on a GPU with a hardware 4-way INT8 dot product, shift-accumulate is not faster than the MAC it replaces. An iso-storage control--an 8-bit shift code of exactly one byte per key, run through the identical kernel--prices the arithmetic substitution alone at 2.02x at comparable accuracy. The advantage of the PoT representation is therefore its density and the removal of the multiplier, not raw GPU throughput. A sweep over exponent width, mantissa levels, granularity and rounding gives the design lesson: nearest-PoT relative error is scale-free, so extra exponent bits buy nothing (eps_S=0.1302 at 3, 4 and 5 bits) and accuracy must come from mantissa terms. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AR). Categorie arXiv: Hardware Architecture (cs.AR). Pagina del lavoro: https://arxiv.org/abs/2609.09208 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T07:25:33.726Z dalla pagina https://arxiv.org/catchup/cs.AR/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).