Metraipolis › archive › dc_e8ea675236eb

Low-Bit N-Gram Embeddings for Language Models — arXiv:2610.31646

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.31646
License (as archived): CC0-1.0
Archived: 2026-10-25T08:35:42.694Z
SHA-256 of the archived text: b37363d85fff22d5f8f96dcfca9a17b186bb335ee805fb21e968e005de01ecad
Stable URL of this page: https://metraipolis.com/documento/dc_e8ea675236eb

How to cite

Low-Bit N-Gram Embeddings for Language Models — arXiv:2610.31646 — arXiv (en), https://arxiv.org/abs/2610.31646 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e8ea675236eb (Metraipolis archive dc_e8ea675236eb, sha256 b37363d85fff22d5f8f96dcfca9a17b186bb335ee805fb21e968e005de01ecad)

BibTeX

@misc{dce8ea675236eb,
  title = {Low-Bit N-Gram Embeddings for Language Models — arXiv:2610.31646},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31646},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e8ea675236eb; sha256 b37363d85fff22d5f8f96dcfca9a17b186bb335ee805fb21e968e005de01ecad},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31646. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or triples of consecutive tokens, which the model retrieves during processing. However, they can require substantial memory: DeepSeek-V4.1-Flash, for example, allocates 196 billion parameters to its Engram n-gram memory. We therefore investigate whether this memory cost can be reduced while preserving the performance improvement that the tables provide. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables. Using fewer bits leads to a larger drop in performance, although we find that a short fine-tuning phase recovers a significant part of this loss. In addition, reinvesting the saved memory in larger tables can improve performance beyond that of the original full-precision tables while still using less memory. These table memory savings also extend to large existing models: on DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, low-bit tables save 47-55% of table memory, with perplexity increases of less than 1.2% and largely preserved downstream task performance. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.31646 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:13.278Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).