Metraipolis › archive › dc_8d3053080ecd
Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory — arXiv:2610.31125
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.31125
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-23T15:20:38.214Z
- SHA-256 of the archived text
838a3e4b60d93d4581367eacdc38397e487c5dc1fdd371b9a329e9d127341496- Stable URL of this page
https://metraipolis.com/documento/dc_8d3053080ecd
How to cite
Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory — arXiv:2610.31125 — arXiv (en), https://arxiv.org/abs/2610.31125 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d3053080ecd (Metraipolis archive dc_8d3053080ecd, sha256 838a3e4b60d93d4581367eacdc38397e487c5dc1fdd371b9a329e9d127341496)
BibTeX
@misc{dc8d3053080ecd,
title = {Delta N-gram: Forward-Only Continual Writing into a Language Model's Native N-gram Memory — arXiv:2610.31125},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.31125},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d3053080ecd; sha256 838a3e4b60d93d4581367eacdc38397e487c5dc1fdd371b9a329e9d127341496},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31125. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Our previous analysis found that a model's native hashed N-gram table is where continual updates disturb the base least, but its writes needed backpropagation and changed the pretrained table. We present Delta N-gram, a forward-only write path into that table. The pretrained table T^0 is never modified. Evidence is written into a sparse patch \Delta on the rows it addresses: a 4.1M-parameter writer emits a target and a gate per (position, hash head) event, and a block-averaged delta rule keeps each row a convex combination of its old value and the gated targets. The writer is trained offline on raw text with the pretraining objective; at use time writing takes forward passes only, and clearing \Delta restores the original model. A 0.6B prototype trained to answer questions keeps 16/16 facts after a second write; online gradient writes keep 2--4/16. We study two writers: a plain writer that encodes evidence with the T^0-only model, and a delta writer that reads its current memory while encoding. On Qwen3.8-Flash-Next, the plain writer's memory is document-specific: on 64 held-out news events it lowers continuation NLL by 0.068 nats/token against empty memory and by 0.024 against another document's memory, and 0.063 and 0.013 after 16 intervening articles. The delta writer's memory is more document-specific and keeps more of it under distraction (70% vs 54%). Over 35,000 writes on our previous seven-task chain both writers stay stable and the delta writer drifts less (layer-2 math drift 176% vs 188%); later accuracy changes by -0.26/-0.52pp (-7.29pp for gradient updates) and small-k_i drift is 17.9/17.5% (39.2%). Acquisition is negative, and general-input drift reaches 32.8% (20.75%): format rows read by every example grow to 4--12 times their pretrained norm. Zeroing eight such rows removes over 98% of layer-2 drift on math inputs, together with their NLL gain. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.31125 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-23T15:09:38.352Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).