Metraipolis › archive › dc_62c5cae71bb5

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention — arXiv:2608.08081

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.08081
License (as archived)
CC0-1.0
Archived
2026-10-19T09:15:01.289Z
SHA-256 of the archived text
ba0f1c967a0f173936814cb54aa88994037c0d44f7168bb8c04819510ea91834
Stable URL of this page
https://metraipolis.com/documento/dc_62c5cae71bb5

How to cite

RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention — arXiv:2608.08081 — arXiv (en), https://arxiv.org/abs/2608.08081 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_62c5cae71bb5 (Metraipolis archive dc_62c5cae71bb5, sha256 ba0f1c967a0f173936814cb54aa88994037c0d44f7168bb8c04819510ea91834)
BibTeX
@misc{dc62c5cae71bb5,
  title = {RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention — arXiv:2608.08081},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.08081},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_62c5cae71bb5; sha256 ba0f1c967a0f173936814cb54aa88994037c0d44f7168bb8c04819510ea91834},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.08081. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand. We present RotaryQuant, a three-axis compression system that addresses all three. Mixed-precision weight quantization assigns bit-widths by architectural role: 4-bit for dense layers, 2-bit for routed experts, and 8-bit for the shared expert whose high activation kurtosis resists aggressive compression. LRU expert offloading pages non-resident experts to disk under genuine memory pressure. The novel axis is IsoQuant, a KV cache compression method that applies a Walsh--Hadamard transform followed by block-diagonal SO(4) rotations to isotropize activation distributions before 3-bit scalar quantization, requiring $O(d \log d)$ operations and 256 stored parameters per head versus $O(d^2)$ and 16{,}384 for dense rotation methods. A fused four-kernel Metal GPU pipeline performs attention directly on packed 3-bit tensors without materializing full-precision KV state---a different execution model, not just a quantization scheme. The combined system fits Gemma 4-26B-A4B and Qwen3-30B-A3B within a 16\,GB budget and Nemotron-H 120B within 32\,GB, running interactively at 9--19 tok/s with near-zero perplexity degradation ($\Delta$PPL $\leq +0.0012$) and 100\% retrieval accuracy at 32K context.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.NE).
Categorie arXiv: Neural and Evolutionary Computing (cs.NE) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.08081 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:12:57.238Z dalla pagina https://arxiv.org/catchup/cs.NE/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).