Metraipolis › archive › dc_319495017141

Behaviorally Effective LoRA Writes Are Sparse and Structured — arXiv:2609.01374

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01374
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:38.469Z
SHA-256 of the archived text
f08007adb1e41d9fa658e28eb132ee1eb1131e1cd97add2ea5dacc33a5553c0b
Stable URL of this page
https://metraipolis.com/documento/dc_319495017141

How to cite

Behaviorally Effective LoRA Writes Are Sparse and Structured — arXiv:2609.01374 — arXiv (en), https://arxiv.org/abs/2609.01374 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_319495017141 (Metraipolis archive dc_319495017141, sha256 f08007adb1e41d9fa658e28eb132ee1eb1131e1cd97add2ea5dacc33a5553c0b)
BibTeX
@misc{dc319495017141,
  title = {Behaviorally Effective LoRA Writes Are Sparse and Structured — arXiv:2609.01374},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01374},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_319495017141; sha256 f08007adb1e41d9fa658e28eb132ee1eb1131e1cd97add2ea5dacc33a5553c0b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01374. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained write actually carry behavior. We study that question directly and show that behaviorally effective LoRA writes are sparse, structured, and far more concentrated than the raw low-rank parameterization suggests. We use Learned-Basis LoRA, a learned-basis continuation recipe, to expose that structure. The recipe warms up an unconstrained adapter, converts its learned write columns into a module-wise orthonormal basis, freezes that basis, and continues training inside the constrained parameterization. Across 14 exact switches from unconstrained to constrained form, held-out accuracy is unchanged at the conversion step and reconstructed write matrices differ by at most 0.25% relative Frobenius error. Same-state continuation then shows that the same trained checkpoint develops differently under different write subspaces, establishing write geometry as a causal state variable. A no-retraining projection test shows that useful write signal stays inside the learned write space and largely disappears from random or frozen-activation PCA controls. The concentration pattern is strong at both local and global scales. Across GSM8K, MathQA, and AQuA, per-module top-k continuation reaches its optimum at k in {2, 4} in all twelve seed-level cases we test. A stricter global ranking test shows that learned top-16 and top-32 subsets outperform matched random subsets, especially on GSM8K/Qwen and MathQA/Qwen. Single-direction ablations further reveal a sparse set of late q_proj, o_proj, and down_proj components with outsized behavioral impact.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.01374 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.863Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).