Metraipolis › archive › dc_32af40e40921
Gradient-Free Optimization for Matrix functions — arXiv:2609.03170
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.03170
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T02:26:42.862Z
- SHA-256 of the archived text
e6985fa7a8763514298778c9e531680b8ebc2e1bfc80dd2f9e387cb1c7965acb- Stable URL of this page
https://metraipolis.com/documento/dc_32af40e40921
How to cite
Gradient-Free Optimization for Matrix functions — arXiv:2609.03170 — arXiv (en), https://arxiv.org/abs/2609.03170 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_32af40e40921 (Metraipolis archive dc_32af40e40921, sha256 e6985fa7a8763514298778c9e531680b8ebc2e1bfc80dd2f9e387cb1c7965acb)
BibTeX
@misc{dc32af40e40921,
title = {Gradient-Free Optimization for Matrix functions — arXiv:2609.03170},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.03170},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_32af40e40921; sha256 e6985fa7a8763514298778c9e531680b8ebc2e1bfc80dd2f9e387cb1c7965acb},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03170. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We consider the task of optimizing smooth, possibly non-convex functions of a matrix variable given access only to directional derivatives rather than full gradients. This setting arises when fine-tuning large neural networks on consumer-grade hardware: the network's weights are matrices, memory constraints rule out backward-mode automatic differentiation, but directional derivatives remain available through forward mode. We frame gradient estimation in this setting as a structured recovery problem, in the spirit of signal processing. From this perspective we provide three contributions. First, we introduce an alternative to the standard random gradient estimator; the difference corresponds to replacing the adjoint of the sampling operator with its pseudoinverse. Second, when the gradient satisfies an approximate low-rank condition, techniques from matrix sensing yield a family of highly accurate gradient estimators that drop into any first-order method. Third, we note that while the computational cost of such estimators is high, this can be amortized by combining them with a matrix-aware optimizer such as spectral descent. Specifically, the gradient estimator computes a factorization of the gradient, allowing for the projection step of spectral descent to be done at no extra cost. We demonstrate our findings with two careful numerical experiments on synthetic functions with approximately low-rank gradients. We show that by exploiting this low-rank property one obtains much faster convergence to good approximate solutions. Tema degli esploratori: matematica e informatica teorica (retro-catalogo arXiv: annunci del 2026-09-04 in math.OC). Categorie arXiv: Optimization and Control (math.OC). Pagina del lavoro: https://arxiv.org/abs/2609.03170 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T02:07:43.324Z dalla pagina https://arxiv.org/catchup/math.OC/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).