Metraipolis › archive › dc_b28b38ff2de6

Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD? — arXiv:2609.40335

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.40335
License (as archived)
CC0-1.0
Archived
2026-10-07T13:47:03.570Z
SHA-256 of the archived text
e7c04a6c77d240feb9ef9ed948e18d99f0bfa24803a2d7682f5a9a8b1ed35c46
Stable URL of this page
https://metraipolis.com/documento/dc_b28b38ff2de6

How to cite

Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD? — arXiv:2609.40335 — arXiv (en), https://arxiv.org/abs/2609.40335 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b28b38ff2de6 (Metraipolis archive dc_b28b38ff2de6, sha256 e7c04a6c77d240feb9ef9ed948e18d99f0bfa24803a2d7682f5a9a8b1ed35c46)
BibTeX
@misc{dcb28b38ff2de6,
  title = {Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD? — arXiv:2609.40335},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.40335},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b28b38ff2de6; sha256 e7c04a6c77d240feb9ef9ed948e18d99f0bfa24803a2d7682f5a9a8b1ed35c46},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40335. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Differentially Private Stochastic Gradient Descent (DP-SGD) is a leading approach for privacy-preserving fine-tuning of large language models (LLMs). Many decoder-only LLMs employ weight tying between input and output embeddings, a design choice originally introduced for parameter efficiency and improved language modeling performance in the non-private setting. However, the impact of weight tying under differentially private training remains largely unexplored. In this work, we investigate the role of weight tying in the DP setting using GPT2 and DistilGPT2 as representative decoder-only architectures. Interestingly, we find that untied embeddings consistently outperform weight-tied models under DP-SGD, achieving gains of up to 4.74% points in accuracy on SST-2, QNLI, and QQP. Beyond improved utility, untying embeddings enables the use of memory-efficient ghost clipping for DP-SGD. By contrast, weight tying introduces shared-parameter interactions that complicate standard ghost norm computation and largely negate its computational advantages. As a result, untied models achieve over 60% lower memory usage while preserving the benefits of ghost clipping. Our results indicate that untied embeddings provide a more effective and scalable design for differentially private training of decoder-only LLMs and highlight the need to revisit standard LLM architectural choices in the privacy-preserving setting.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.40335 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T13:18:10.532Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).