Metraipolis › archive › dc_edf82ade27bb

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning — arXiv:2610.06243

Source: arXiv
Original: https://arxiv.org/abs/2610.06243
License (as archived): CC0-1.0
Archived: 2026-10-06T05:31:32.339Z
SHA-256 of the archived text: eca8edad08a2cf4ccc68ed130394012ce3da8215eba05b544c21992013ba3e44
Stable URL of this page: https://metraipolis.com/documento/dc_edf82ade27bb

How to cite

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning — arXiv:2610.06243 — arXiv, https://arxiv.org/abs/2610.06243 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_edf82ade27bb (Metraipolis archive dc_edf82ade27bb, sha256 eca8edad08a2cf4ccc68ed130394012ce3da8215eba05b544c21992013ba3e44)

BibTeX

@misc{dcedf82ade27bb,
  title = {RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning — arXiv:2610.06243},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06243},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_edf82ade27bb; sha256 eca8edad08a2cf4ccc68ed130394012ce3da8215eba05b544c21992013ba3e44},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06243. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Parameter-efficient fine-tuning (PEFT) reduces the cost of adapting foundation models by focusing training on a small parameter subset. Complementary to this idea, we introduce RoSA (Rotational Sparse Adaptation), which narrows adaptation to a subset of layers at a time. RoSA freezes lower layers close to the input throughout training and rotates a trainable block over later layers, progressively increasing the number of frozen layers close to the input. This design reduces optimizer-state memory, shortens backpropagation, and even forward propagation if activations at the last frozen layer are cached. Because RoSA is orthogonal to the choice of trainable parameterization, it can be combined with PEFT methods or sparse optimizers within each active block. Experiments across multiple LLM architectures and tasks show that RoSA reduces peak memory while maintaining strong fine-tuning performance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.06243 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.870Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).