Metraipolis › archive › dc_0df037d25bbf

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning — arXiv:2609.25058

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25058
License (as archived)
CC0-1.0
Archived
2026-10-08T14:34:20.591Z
SHA-256 of the archived text
db6afaa08d2a5fef6c8df7cc4b9cb363036fb1722535da9e7651cdf6db326bf5
Stable URL of this page
https://metraipolis.com/documento/dc_0df037d25bbf

How to cite

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning — arXiv:2609.25058 — arXiv (en), https://arxiv.org/abs/2609.25058 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0df037d25bbf (Metraipolis archive dc_0df037d25bbf, sha256 db6afaa08d2a5fef6c8df7cc4b9cb363036fb1722535da9e7651cdf6db326bf5)
BibTeX
@misc{dc0df037d25bbf,
  title = {ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning — arXiv:2609.25058},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25058},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0df037d25bbf; sha256 db6afaa08d2a5fef6c8df7cc4b9cb363036fb1722535da9e7651cdf6db326bf5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25058. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Parameter-efficient fine-tuning (PEFT) adapts large language models (LLMs) to downstream tasks while updating only a small fraction of their pretrained parameters. Low-Rank Adaptation (LoRA) uses two trainable low-rank matrices, while Weight-Decomposed Low-Rank Adaptation (DoRA) further separates weight magnitude and direction but retains the dense LoRA-style factorization in its directional branch. We propose ChainDoRA, a weight-decomposed adaptation framework that constructs the directional low-rank factors from a connected Tensor-Train (TT) chain, where the adapter rank forms the boundary rank between input- and output-side TT contractions and an independent TT rank controls representation capacity and parameter cost. Under a controlled 15,119-example response-only adaptation setting with LLaMA-7B, ChainDoRA is evaluated against matched LoRA and DoRA baselines on seven commonsense reasoning benchmarks. ChainDoRA with TT rank 16 achieves a seven-task average accuracy of 72.30%, compared with 69.88% for LoRA and 69.39% for DoRA, while requiring only 5.35M trainable parameters versus 56.10M for LoRA and 56.98M for DoRA, corresponding to a 90.62% reduction relative to DoRA. Ablations over TT rank and adapter placement show controllable parameter-accuracy trade-offs, indicating that connected TT parameterization can substantially reduce the parameter cost of magnitude-direction adaptation while preserving, and in this setting improving, downstream reasoning performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Computer Vision and Pattern Recognition (cs.CV).
Pagina del lavoro: https://arxiv.org/abs/2609.25058 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:46:15.661Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).