Metraipolis › archive › dc_bee7b069e331

Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA — arXiv:2610.06621

Source: arXiv
Original: https://arxiv.org/abs/2610.06621
License (as archived): CC0-1.0
Archived: 2026-10-06T05:27:47.981Z
SHA-256 of the archived text: e57bdb213da5eac81929d05e63c357f1e1e0592c28ff1c1d7133fd64943e954e
Stable URL of this page: https://metraipolis.com/documento/dc_bee7b069e331

How to cite

Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA — arXiv:2610.06621 — arXiv, https://arxiv.org/abs/2610.06621 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bee7b069e331 (Metraipolis archive dc_bee7b069e331, sha256 e57bdb213da5eac81929d05e63c357f1e1e0592c28ff1c1d7133fd64943e954e)

BibTeX

@misc{dcbee7b069e331,
  title = {Frozen Factor or Spectral Band? Disentangling Two Choices in Low-Rank LoRA — arXiv:2610.06621},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06621},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bee7b069e331; sha256 e57bdb213da5eac81929d05e63c357f1e1e0592c28ff1c1d7133fd64943e954e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06621. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Spectral variants of low-rank adaptation (LoRA) choose both a subspace and which factor to freeze. We separate these choices by freezing the input factor A or output factor B on the top or bottom singular directions of pretrained weights, with learning rates selected separately. At rank 2, the same-band advantage of freezing A is larger than either within-factor band difference on all four task-model pairs with complete comparisons. Freezing B also trails comparable-budget free LoRA by 8-18 percentage points on five pairs spanning a formatting task and OpenBookQA. The A-frozen advantage persists in a single-GPU-model replication and within individual MLP module groups, including controls with equal or greater trainable counts for B frozen, and when A is frozen on a random orthonormal basis. The factor contrast weakens with rank. On OpenBookQA / Qwen2.5-1.5B at rank 16, PEFT's MiCA implementation trails comparable-budget LoRA by 3.08 points under a shared training recipe transferred from the MiCA paper. A trained oracle output subspace largely removes the low-rank deficit; partial warm-up gains recur across three direction seeds. The factor-versus-band ordering is descriptive; an approximate multiplicity audit weakens several earlier significance claims. These results extend known factor asymmetry by showing how its magnitude depends on spectral placement, rank and training conditions.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.06621 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:06.178Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).