Metraipolis › archive › dc_d074b5eb802b
MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models — arXiv:2609.07666
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.07666
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T13:57:53.613Z
- SHA-256 of the archived text
a70b3fd25267591c2a9d8d3be11e1d5859c7214ae8c6fd489c96f9f8d7d911d7- Stable URL of this page
https://metraipolis.com/documento/dc_d074b5eb802b
How to cite
MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models — arXiv:2609.07666 — arXiv (en), https://arxiv.org/abs/2609.07666 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d074b5eb802b (Metraipolis archive dc_d074b5eb802b, sha256 a70b3fd25267591c2a9d8d3be11e1d5859c7214ae8c6fd489c96f9f8d7d911d7)
BibTeX
@misc{dcd074b5eb802b,
title = {MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models — arXiv:2609.07666},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.07666},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d074b5eb802b; sha256 a70b3fd25267591c2a9d8d3be11e1d5859c7214ae8c6fd489c96f9f8d7d911d7},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.07666. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Full-parameter fine-tuning of large language models has substantial memory costs because backpropagation stores activations and gradients. Zeroth-order optimization avoids this by estimating update directions from loss evaluations, but existing methods require tuning a sensitive learning rate for each model and task. We propose the momentum $p$-dimensional subspace trust-region method (MpSub). At each iteration, MpSub searches within a $p$-dimensional subspace: one direction preserves historical momentum from the most recent accepted step, while the remaining directions explore via fresh random sampling. The subspace gradient is estimated by central differences, a trial step is computed from a linear trust-region model, and the trust-region radius adapts according to the agreement between predicted and observed loss reduction, eliminating the learning rate. For LLM fine-tuning, evaluations within an iteration share a minibatch, and directions are regenerated in place from seeds, using forward passes alone. For smooth deterministic objectives under unorthogonalized Gaussian directions, we bound the finite-difference error, quantify gradient energy captured by the subspace, and prove that $\lim_{k\to\infty} \|\nabla f(x_k)\|_2 = 0$ almost surely under a safeguarded radius update. Under a matched budget of 8,400 training-objective forward passes, we fine-tune OPT-125M and OPT-350M on CommitmentBank. With the same preset parameters at both model sizes, MpSub attains mean test accuracies of 0.673 and 0.690 over three seeds, matching tuned MeZO (0.685) without any learning-rate search.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2609.07666 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:42.979Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).