Metraipolis › archive › dc_b28b38ff2de6
Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD? — arXiv:2609.40335
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.40335
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T13:47:03.570Z
- SHA-256 of the archived text
e7c04a6c77d240feb9ef9ed948e18d99f0bfa24803a2d7682f5a9a8b1ed35c46- Stable URL of this page
https://metraipolis.com/documento/dc_b28b38ff2de6
How to cite
Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD? — arXiv:2609.40335 — arXiv (en), https://arxiv.org/abs/2609.40335 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b28b38ff2de6 (Metraipolis archive dc_b28b38ff2de6, sha256 e7c04a6c77d240feb9ef9ed948e18d99f0bfa24803a2d7682f5a9a8b1ed35c46)
BibTeX
@misc{dcb28b38ff2de6,
title = {Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD? — arXiv:2609.40335},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.40335},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b28b38ff2de6; sha256 e7c04a6c77d240feb9ef9ed948e18d99f0bfa24803a2d7682f5a9a8b1ed35c46},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40335. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Differentially Private Stochastic Gradient Descent (DP-SGD) is a leading approach for privacy-preserving fine-tuning of large language models (LLMs). Many decoder-only LLMs employ weight tying between input and output embeddings, a design choice originally introduced for parameter efficiency and improved language modeling performance in the non-private setting. However, the impact of weight tying under differentially private training remains largely unexplored. In this work, we investigate the role of weight tying in the DP setting using GPT2 and DistilGPT2 as representative decoder-only architectures. Interestingly, we find that untied embeddings consistently outperform weight-tied models under DP-SGD, achieving gains of up to 4.74% points in accuracy on SST-2, QNLI, and QQP. Beyond improved utility, untying embeddings enables the use of memory-efficient ghost clipping for DP-SGD. By contrast, weight tying introduces shared-parameter interactions that complicate standard ghost norm computation and largely negate its computational advantages. As a result, untied models achieve over 60% lower memory usage while preserving the benefits of ghost clipping. Our results indicate that untied embeddings provide a more effective and scalable design for differentially private training of decoder-only LLMs and highlight the need to revisit standard LLM architectural choices in the privacy-preserving setting. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.40335 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T13:18:10.532Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).