Metraipolis › archive › dc_ecfaeb5e7b36

Gradient Descent with Stochastic Subspaces via Persistence of Memory — arXiv:2609.18416

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.18416
License (as archived): CC0-1.0
Archived: 2026-10-15T15:07:59.544Z
SHA-256 of the archived text: ff23a30e58dc62d6627efbddf040bc8bce8b2ba6ced690efce6be7bde138a49d
Stable URL of this page: https://metraipolis.com/documento/dc_ecfaeb5e7b36

How to cite

Gradient Descent with Stochastic Subspaces via Persistence of Memory — arXiv:2609.18416 — arXiv (en), https://arxiv.org/abs/2609.18416 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ecfaeb5e7b36 (Metraipolis archive dc_ecfaeb5e7b36, sha256 ff23a30e58dc62d6627efbddf040bc8bce8b2ba6ced690efce6be7bde138a49d)

BibTeX

@misc{dcecfaeb5e7b36,
  title = {Gradient Descent with Stochastic Subspaces via Persistence of Memory — arXiv:2609.18416},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18416},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ecfaeb5e7b36; sha256 ff23a30e58dc62d6627efbddf040bc8bce8b2ba6ced690efce6be7bde138a49d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18416. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Stochastic subspace methods have gained popularity as gradient descent based techniques for large scale optimisation problems, especially in distributed settings. In this paper, we introduce the technique of "persistence of memory" to greatly extend and improve the random subspace methods. To this end, we leverage a vector that is only weakly correlated with the gradient in order to provide a guiding structure to the generative process of the random subspace along which the descent is going to take place. This guidance vector may be fixed for a large number of iterations, only to be refreshed at wide intervals (on whose size we can provide guarantees in terms of problem parameters). In important machine learning settings, such as optimisation problems embodying sparsity or a minibatch structure, we show that the guidance vector can be obtained in an effective and computationally inexpensive manner by leveraging the structured properties of the problem. En route, we establish to our knowledge the first theoretical analysis of classical SSD methods for sparse functions. In a local neighbourhood of the optimum, we demonstrate an alignment phenomenon of our gradient estimates with a low-lying eigenvector of the Hessian, allowing a once-for-all computation of the guidance vector which renders the method computationally favourable even in scenarios with unstructured objectives.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.LG).
Categorie arXiv: Optimization and Control (math.OC) ; Machine Learning (cs.LG); Signal Processing (eess.SP); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.18416 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T14:08:19.246Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).