Metraipolis › archive › dc_c433a46ae07f

The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models — arXiv:2610.08816

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.08816
License (as archived): CC0-1.0
Archived: 2026-10-08T11:56:06.359Z
SHA-256 of the archived text: 57e7d098593ea1b7a86bc7be66fbbc983edacc044feb7373cfcc8e93fb90b22f
Stable URL of this page: https://metraipolis.com/documento/dc_c433a46ae07f

How to cite

The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models — arXiv:2610.08816 — arXiv (en), https://arxiv.org/abs/2610.08816 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c433a46ae07f (Metraipolis archive dc_c433a46ae07f, sha256 57e7d098593ea1b7a86bc7be66fbbc983edacc044feb7373cfcc8e93fb90b22f)

BibTeX

@misc{dcc433a46ae07f,
  title = {The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models — arXiv:2610.08816},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08816},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c433a46ae07f; sha256 57e7d098593ea1b7a86bc7be66fbbc983edacc044feb7373cfcc8e93fb90b22f},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08816. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately? We study this question directly in forecasting risk. For algebraically decaying predictive memory, we prove matching upper and lower approximation bounds for exponential and finite-state modes. The best $r$-mode forecast error decays as $e^{-\Theta(\sqrt r)}$, so reaching forecast error $\tau$ needs $r=\Theta(\log^2(1/\tau))$ states or modes. Earlier curse-of-memory results establish broad limitations of stable recurrent models under different approximation notions; here both sides match for one canonical predictive target in forecast risk, which fixes the optimal resource exponent for that target. We then show that genuine fractional long memory changes the geometry itself. In particular, forecast error is measured after fractional integration, prediction from a finite context of length $L$ has an exact $1/L$ leading order, and a fixed fractional strength $d$ keeps the square-log state-complexity law. Near the short-memory boundary, we identify the relevant $d^2$ and $d^4$ scales and give a uniform constructive law in the intermediate regime. For nonlinear contextual recurrences with uniformly contractive state dynamics, we derive an exponential first-chaos envelope and an explicit necessary condition that relates forecast accuracy to the contraction margin. Vanishing forecasting error on an algebraic target forces the recurrence quantitatively toward criticality, a condition that is necessary and not by itself sufficient. Finite-sample Kullback--Leibler calculations further connect the predictive geometry to statistical information. Theorem-matched experiments with contractive state-space, gated recurrent, and attention models reproduce the state and stability predictions.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.08816 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:50:37.625Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).