Metraipolis › archive › dc_c26becd820be

SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning — arXiv:2608.15360

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.15360
License (as archived)
CC0-1.0
Archived
2026-10-18T18:03:10.251Z
SHA-256 of the archived text
993f1db652158409174851ccf5f7acab2b296c24b8aa07b729f30ac19eceb3e1
Stable URL of this page
https://metraipolis.com/documento/dc_c26becd820be

How to cite

SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning — arXiv:2608.15360 — arXiv (en), https://arxiv.org/abs/2608.15360 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c26becd820be (Metraipolis archive dc_c26becd820be, sha256 993f1db652158409174851ccf5f7acab2b296c24b8aa07b729f30ac19eceb3e1)
BibTeX
@misc{dcc26becd820be,
  title = {SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning — arXiv:2608.15360},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.15360},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c26becd820be; sha256 993f1db652158409174851ccf5f7acab2b296c24b8aa07b729f30ac19eceb3e1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.15360. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

While Parameter-Efficient Fine-Tuning (PEFT) has substantially reduced the hardware cost of adapting Large Language Models (LLMs) by decreasing the number of trainable parameters, recent studies have sought to further improve PEFT through parameter sharing. However, these approaches either employ uniform parameter sharing across layers, which can delay convergence, or rely on dynamic masking strategies, which add computational overhead. The potential of sharing patterns inspired by the inherent hierarchical structure of Transformer architectures remains unexplored in PEFT. To address this gap, we introduce SAPE (Sandwich Adapters for Parameter Efficiency), a PEFT framework based on a sandwich-style hard weight-sharing topology. SAPE routes intermediate Transformer layers through balanced shared group adapters while strictly isolating the input embedding and final projection boundary transformations to prevent gradient interference. This design significantly reduces memory consumption while eliminating the computational overhead associated with dynamic parameter-sharing methods. Extensive evaluations across encoder-only and causal decoder architectures demonstrate that SAPE achieves state-of-the-art performance in low-parameter regimes. On natural language understanding, SAPE outperforms proPETL on RoBERTa-large while utilizing only 10% of the baseline's parameter budget. On natural language generation and world knowledge reasoning with LLaMA-3.2 (3B) under a strict ~0.6M parameter constraint, SAPE outperforms AdaLoRA, yielding absolute improvements of +4.85% on GSM8K and +3.11% on CommonsenseQA. Furthermore, through comprehensive topological ablations, we formalize an inherent capacity trade-off: while hard parameter sharing strongly regularizes semantic generalization, it slightly smooths the sharp layer-wise transformations required for rigid multi-step arithmetic reasoning.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.15360 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:14.010Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).