Metraipolis › archive › dc_33cd7aacdcc7

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces — arXiv:2608.15665

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.15665
License (as archived)
CC0-1.0
Archived
2026-10-18T19:40:47.482Z
SHA-256 of the archived text
9d6180436842aa96747175f829c1e476ffe194d12b88cd2e5e6721ab8733da52
Stable URL of this page
https://metraipolis.com/documento/dc_33cd7aacdcc7

How to cite

SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces — arXiv:2608.15665 — arXiv (en), https://arxiv.org/abs/2608.15665 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33cd7aacdcc7 (Metraipolis archive dc_33cd7aacdcc7, sha256 9d6180436842aa96747175f829c1e476ffe194d12b88cd2e5e6721ab8733da52)
BibTeX
@misc{dc33cd7aacdcc7,
  title = {SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces — arXiv:2608.15665},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.15665},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33cd7aacdcc7; sha256 9d6180436842aa96747175f829c1e476ffe194d12b88cd2e5e6721ab8733da52},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15665. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Zeroth-order (ZO) optimization with SGD in random subspaces enables memory-efficient fine-tuning of large language models without backpropagation. However, high gradient estimation noise fundamentally undermines adaptive optimizers like Adam. We propose SubZero+, which achieves practical adaptive ZO optimization through a carefully designed dual low-dimensionality strategy: (i) multi-query forward-difference gradient estimation in periodically refreshed random subspaces to mitigate noise amplification in moment buffers, and (ii) Adam updates with periodic restarts performed directly in low-dimensional space rather than full-parameter space. In experiments, this dual design retains memory overhead comparable to momentum-free ZO methods while achieving stronger optimization performance than the evaluated ZO baselines. Theoretically, in the exact-directional limit, $K$-query averaging preserves conditional unbiasedness, while the coefficient estimator's covariance and mean-squared error, as well as query-induced second-moment inflation, scale exactly as $1/K$. Extensive experiments across SuperGLUE with models from 1.3B to 32B parameters under both full fine-tuning and LoRA schemes demonstrate consistent improvements over competing ZO methods. SubZero+ significantly narrows the performance gap with first-order optimization while preserving ZO's inference-time memory efficiency.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.15665 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T19:02:44.114Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).