Metraipolis › archive › dc_41e6ded75e3e

The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation — arXiv:2610.20456

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.20456
License (as archived)
CC0-1.0
Archived
2026-10-15T14:23:52.983Z
SHA-256 of the archived text
a3a08e0bae05dab877ad1d02bb7cc0c48964d465cba55517af248d94954c1e79
Stable URL of this page
https://metraipolis.com/documento/dc_41e6ded75e3e

How to cite

The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation — arXiv:2610.20456 — arXiv (en), https://arxiv.org/abs/2610.20456 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_41e6ded75e3e (Metraipolis archive dc_41e6ded75e3e, sha256 a3a08e0bae05dab877ad1d02bb7cc0c48964d465cba55517af248d94954c1e79)
BibTeX
@misc{dc41e6ded75e3e,
  title = {The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation — arXiv:2610.20456},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.20456},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_41e6ded75e3e; sha256 a3a08e0bae05dab877ad1d02bb7cc0c48964d465cba55517af248d94954c1e79},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.20456. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The scaling relationship of compute and performance for language models (LMs) has been extensively studied; more recently, models are being designed to scale memory use, as well. In particular, adding embeddings for n-grams scales memory separately from compute use, because each pass only processes individual embedding rows, which are much smaller than the storage used for the whole embedding table. However, existing methods train embeddings from scratch, which needs expensive GPU memory for fast reading and writing. Meanwhile, the ordinary memory of the host system is cheap and underutilized by these training methods. To scale on host memory, we introduce COUNT (Counting Offline Unlocks N-gram Transformers), which builds n-gram embeddings without adding learned parameters by storing precomputed n-gram counts on host. We use the corpus frequencies to weight together the LM's own unembeddings of the next tokens most often following the current n-gram history. At a fixed compute budget, COUNT can be added as an additional input embedding for pretraining 1B-parameter LMs to improve validation loss as much as an LM with 57% more compute. If GPU memory is limited, using COUNT to let backbone parameters fill GPU memory gives better performance than splitting memory with additional learned n-gram embeddings. Beyond efficiency, COUNT allows switching in a different table of n-gram counts from a domain-specific corpus at inference time to adapt to a new task without any training.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.20456 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.970Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).