Metraipolis › archive › dc_39fd2380cdeb

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory — arXiv:2610.08553

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.08553
License (as archived): CC0-1.0
Archived: 2026-10-07T11:21:55.864Z
SHA-256 of the archived text: 8be44061ab14e7858ef81ef45b7f819f7de92f20b2610e0ca4f9db4fdd02de30
Stable URL of this page: https://metraipolis.com/documento/dc_39fd2380cdeb

How to cite

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory — arXiv:2610.08553 — arXiv (en), https://arxiv.org/abs/2610.08553 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_39fd2380cdeb (Metraipolis archive dc_39fd2380cdeb, sha256 8be44061ab14e7858ef81ef45b7f819f7de92f20b2610e0ca4f9db4fdd02de30)

BibTeX

@misc{dc39fd2380cdeb,
  title = {DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory — arXiv:2610.08553},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08553},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_39fd2380cdeb; sha256 8be44061ab14e7858ef81ef45b7f819f7de92f20b2610e0ca4f9db4fdd02de30},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08553. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, we find that this expected advantage does not consistently materialize in nonlinear memories: a fixed-base parallel TTT baseline outperforms its serial counterpart. Our exploratory experiments point to a key underlying difficulty: nonlinear memories can be harder to optimize than linear ones within a single pass over the sequence. To alleviate this optimization difficulty, we introduce DeltaTTT, which replaces joint inner-loop optimization of a two-layer memory network with layerwise learning. Each layer is assigned a local prediction target and updated through a state-dependent delta rule. This formulation retains a nonlinear readout while enabling chunkwise parallel computation. Experiments on DeltaNet and LaCT backbones show improvements in language modeling and retrieval over their recurrent baselines.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.08553 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:24.842Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).