Metraipolis › archive › dc_db54be953cf8

ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning — arXiv:2610.22883

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22883
License (as archived)
CC0-1.0
Archived
2026-10-19T08:42:18.800Z
SHA-256 of the archived text
12575f4fb55c16cd47fe4693ca0c6df11351a9719329de0c73f878d66ab00f0a
Stable URL of this page
https://metraipolis.com/documento/dc_db54be953cf8

How to cite

ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning — arXiv:2610.22883 — arXiv (en), https://arxiv.org/abs/2610.22883 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db54be953cf8 (Metraipolis archive dc_db54be953cf8, sha256 12575f4fb55c16cd47fe4693ca0c6df11351a9719329de0c73f878d66ab00f0a)
BibTeX
@misc{dcdb54be953cf8,
  title = {ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning — arXiv:2610.22883},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22883},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db54be953cf8; sha256 12575f4fb55c16cd47fe4693ca0c6df11351a9719329de0c73f878d66ab00f0a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22883. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Parameter-Efficient Fine-Tuning (PEFT) has emerged as a prevalent strategy for adapting large pre-trained models. Among PEFT techniques, reparameterization-based methods, particularly those that employ low-rank or sparsity structures, have attracted significant attention. However, these approaches often underperform full fine-tuning. Specifically, low-rank methods constrain the update rank, which is misaligned with the inherently high-rank nature of full fine-tuning. In contrast, sparsity-based techniques limit the flexibility of the parameter space and compromise connectivity across weights. To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints. A straightforward approach is to update weights via a product of Toeplitz matrices. However, similar to RNNs, long multiplicative chains often lead to gradient instability. To address this issue, we propose \textbf{ToPA}, a more refined version of this idea that replaces standard Toeplitz matrices with block-wise Toeplitz, greatly increasing each factor's capacity and shortening the chain, thereby stabilizing training and improving expressiveness. We theoretically show that products of block-wise Toeplitz matrices can approximate arbitrary matrices under shorter chain length, justifying the structural design of ToPA. Furthermore, we demonstrate that ToPA offers greater expressivity than existing low-rank and sparse parameterizations. Empirical evaluations across 21 NLP and CV datasets, spanning 5 model architectures, consistently validate the effectiveness of ToPA.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.22883 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:10.042Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).