Metraipolis › archive › dc_762bc5c685d7

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter — arXiv:2609.35465

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35465
License (as archived)
CC0-1.0
Archived
2026-10-07T22:39:08.705Z
SHA-256 of the archived text
f3d7546de89e638cfabfe5addeced4246e337f7fd5316e2b0042375c58bb2788
Stable URL of this page
https://metraipolis.com/documento/dc_762bc5c685d7

How to cite

Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter — arXiv:2609.35465 — arXiv (en), https://arxiv.org/abs/2609.35465 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_762bc5c685d7 (Metraipolis archive dc_762bc5c685d7, sha256 f3d7546de89e638cfabfe5addeced4246e337f7fd5316e2b0042375c58bb2788)
BibTeX
@misc{dc762bc5c685d7,
  title = {Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter — arXiv:2609.35465},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35465},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_762bc5c685d7; sha256 f3d7546de89e638cfabfe5addeced4246e337f7fd5316e2b0042375c58bb2788},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35465. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Leech-lattice quantization gives good quality at two bits per weight, but its codebooks hold more than 10^14 points, too many for a lookup table. Our earlier kernel expanded the codes at load time and read 4.804 bits per weight from GPU memory for 2 bits of code. We present Tetra, a new codebook on the same lattice. A 24-weight block still takes 48 bits, most of which index a 64-state trellis of the Golay code and one shared 16 KiB table. The kernel decodes a block with six table loads and two small lookups inside the matrix-vector product, and reads 2.148 bits per weight. For full models, we retrain one scale per matrix row, store the matrices that lose the most as 4-bit integers, and pay for them with 4-bit embedding tables. Our Qwen3-4B, 8B and 14B files hold 2.73, 2.70 and 2.73 bits per parameter over the whole model. They score 63.37, 69.58 and 75.66 on the full MMLU test set, 4.76, 4.21 and 2.46 points below 4-bit AWQ at 5.3 to 6.0 bits per parameter. They generate 113.8, 95.0 and 57.2 tokens per second in our engine. On GSM8K, through the served kernel, they lose 9.63, 4.62 and 3.26 points to FP16. At 4B our file scores 23.6 points above this http URL 's IQ2_XXS (2.48 bits per parameter). Every number we measured for a table or figure comes from one NVIDIA L40S GPU. We preregistered the main experiments.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.35465 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.446Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).