Metraipolis › archive › dc_eea9898c8ebd

Low-Bit Recurrent States in Hybrid Language Models — arXiv:2609.30950

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.30950
License (as archived): CC0-1.0
Archived: 2026-10-08T02:11:01.935Z
SHA-256 of the archived text: a0110ef3723dfcdc81aee840b391c7bbed596a52e34baf4c76d3e68b9b5c3af0
Stable URL of this page: https://metraipolis.com/documento/dc_eea9898c8ebd

How to cite

Low-Bit Recurrent States in Hybrid Language Models — arXiv:2609.30950 — arXiv (en), https://arxiv.org/abs/2609.30950 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eea9898c8ebd (Metraipolis archive dc_eea9898c8ebd, sha256 a0110ef3723dfcdc81aee840b391c7bbed596a52e34baf4c76d3e68b9b5c3af0)

BibTeX

@misc{dceea9898c8ebd,
  title = {Low-Bit Recurrent States in Hybrid Language Models — arXiv:2609.30950},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.30950},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eea9898c8ebd; sha256 a0110ef3723dfcdc81aee840b391c7bbed596a52e34baf4c76d3e68b9b5c3af0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.30950. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hybrid language models maintain fixed-size recurrent states, but existing quantizers typically use eight bits or more. Quantization errors persist according to channel decay rates. We derive distortion weights from the observability Gramian and combine them with normalized state ranges for mixed-precision bit allocation, without calibration data, rotation, or training. We also quantize decay rates logarithmically. With per-token state quantization, a four-bit mean payload reduces excess negative log-likelihood by factors of 3.3--27.9 relative to the best of seven baselines across three hybrid models; metadata costs vary. At six bits, negative log-likelihood differs from the FP32-state baseline by less than 0.005 nats. Ablations separate gains from variable bit widths, decay weighting, and range normalization. With less frequent write-backs, gains diminish and depend on the model and budget.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.30950 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:43:06.715Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).