Metraipolis › archive › dc_19a38280ee44
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling — arXiv:2608.16844
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.16844
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T18:07:41.512Z
- SHA-256 of the archived text
9b046883207f3bb3bf2d6769104dd63c542587110ad8908b9ea22fda93ae45cd- Stable URL of this page
https://metraipolis.com/documento/dc_19a38280ee44
How to cite
Proteus: Incremental Memory Activation for Long-Context Sequence Modeling — arXiv:2608.16844 — arXiv (en), https://arxiv.org/abs/2608.16844 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_19a38280ee44 (Metraipolis archive dc_19a38280ee44, sha256 9b046883207f3bb3bf2d6769104dd63c542587110ad8908b9ea22fda93ae45cd)
BibTeX
@misc{dc19a38280ee44,
title = {Proteus: Incremental Memory Activation for Long-Context Sequence Modeling — arXiv:2608.16844},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.16844},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_19a38280ee44; sha256 9b046883207f3bb3bf2d6769104dd63c542587110ad8908b9ea22fda93ae45cd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16844. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state. However, most existing memory models expose a static memory throughout the entire sequence. Because early tokens face no compression pressure, they occupy too many degrees of freedom and "pollute" the memory state, leaving little capacity for later context and increasing interference between what is stored and what arrives next. We study a new paradigm of incremental memory activation, where the effective capacity of memory is progressively expanded as the context grows. Imposing an early bottleneck forces the model to compress history more effectively, while unlocking fresh capacity over time reduces interference and improves retention of later context. We instantiate this paradigm in Proteus, a straightforward mechanism that can be incorporated into a broad class of neural memory architectures at no additional cost. We apply Proteus to state-of-the-art models, including SWLA, Comba, Titans, and Hope-Attention, and observe consistent improvements on standard language modeling and reasoning, as well as on long-context retrieval and understanding, with gains that grow at longer context lengths. Overall, our results show that static memory is suboptimal and that scheduling effective capacity is a simple and broadly applicable tool for sequence modeling. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.16844 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T16:29:14.164Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).