Metraipolis › archive › dc_bedef48b773d

SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models — arXiv:2608.22354

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.22354
License (as archived)
CC0-1.0
Archived
2026-10-18T07:44:48.733Z
SHA-256 of the archived text
774cd730c499e251b2e8edb0c9e393cdab62a6ff4251f6ba2983dc32c6afc050
Stable URL of this page
https://metraipolis.com/documento/dc_bedef48b773d

How to cite

SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models — arXiv:2608.22354 — arXiv (en), https://arxiv.org/abs/2608.22354 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bedef48b773d (Metraipolis archive dc_bedef48b773d, sha256 774cd730c499e251b2e8edb0c9e393cdab62a6ff4251f6ba2983dc32c6afc050)
BibTeX
@misc{dcbedef48b773d,
  title = {SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models — arXiv:2608.22354},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.22354},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bedef48b773d; sha256 774cd730c499e251b2e8edb0c9e393cdab62a6ff4251f6ba2983dc32c6afc050},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22354. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: \textbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose \textbf{State Anomaly Neutralization (SANE)}, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3 \le \alpha \le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$, SANE retains functional reasoning ($33.46$--$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($\alpha \ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-25 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.22354 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T07:37:52.873Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).