Metraipolis › archive › dc_03d221f50e09

Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs — arXiv:2608.22188

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.22188
License (as archived)
CC0-1.0
Archived
2026-10-18T07:50:17.568Z
SHA-256 of the archived text
e66a4f575fb8aa8648b96dcb2b4b4ca19eb7c1f747c4a55ab7594f48d0eca57c
Stable URL of this page
https://metraipolis.com/documento/dc_03d221f50e09

How to cite

Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs — arXiv:2608.22188 — arXiv (en), https://arxiv.org/abs/2608.22188 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_03d221f50e09 (Metraipolis archive dc_03d221f50e09, sha256 e66a4f575fb8aa8648b96dcb2b4b4ca19eb7c1f747c4a55ab7594f48d0eca57c)
BibTeX
@misc{dc03d221f50e09,
  title = {Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs — arXiv:2608.22188},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.22188},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_03d221f50e09; sha256 e66a4f575fb8aa8648b96dcb2b4b4ca19eb7c1f747c4a55ab7594f48d0eca57c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.22188. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-25 in cs.DC).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2608.22188 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T07:47:32.965Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-08-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).