Metraipolis › archive › dc_d008c8e80e96
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models — arXiv:2608.03277
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.03277
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-31T12:34:31.669Z
- SHA-256 of the archived text
fe1ecbb88cf4431790fb975e88d334c0a576ee65dcd29583db41cb2a3d0a3897- Stable URL of this page
https://metraipolis.com/documento/dc_d008c8e80e96
How to cite
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models — arXiv:2608.03277 — arXiv (en), https://arxiv.org/abs/2608.03277 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d008c8e80e96 (Metraipolis archive dc_d008c8e80e96, sha256 fe1ecbb88cf4431790fb975e88d334c0a576ee65dcd29583db41cb2a3d0a3897)
BibTeX
@misc{dcd008c8e80e96,
title = {Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models — arXiv:2608.03277},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.03277},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d008c8e80e96; sha256 fe1ecbb88cf4431790fb975e88d334c0a576ee65dcd29583db41cb2a3d0a3897},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03277. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Differentially private zeroth-order optimization (DP-ZO) enables memory-efficient private fine-tuning of large language models using only forward evaluations. Existing aggregation-based DP-ZO methods reconstruct model updates at a fixed scale, ignoring that the strength of useful signals varies throughout training. Consequently, noise-dominated updates may receive excessive weight and degrade model utility. To address this issue, we propose SAGE, a noise-aware shrinkage method that adaptively attenuates privatized estimates according to their estimated signal quality. SAGE subtracts the known Gaussian noise variance from the observed second moment to estimate the underlying signal energy, stabilizes this estimate through temporal tracking, and compares its current signal-to-noise level with a warm-up reference to derive a bounded shrinkage factor. As pure post-processing, SAGE requires neither additional privacy budget nor model queries and introduces only constant additional state. Our theoretical analysis shows that shrinkage reduces the quadratic update-risk term faster than the linear descent term, preserving useful descent while limiting the influence of noise-dominated updates. Experiments on RoBERTa-large, OPT-1.3B, and OPT-6.7B demonstrate that SAGE outperforms existing baselines in most settings under the same privacy budgets while preserving the forward-only memory efficiency of DP-ZO. Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CR). Categorie arXiv: Machine Learning (cs.LG) ; Cryptography and Security (cs.CR). Pagina del lavoro: https://arxiv.org/abs/2608.03277 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-31T12:21:50.572Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).