Metraipolis › archive › dc_8d7a2de2dbea
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation — arXiv:2609.00762
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00762
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:52:13.496Z
- SHA-256 of the archived text
e7a12fcf8730b79d922ea78fb7f3db84d6279f51523cb489de6a28c567177d2e- Stable URL of this page
https://metraipolis.com/documento/dc_8d7a2de2dbea
How to cite
Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation — arXiv:2609.00762 — arXiv (en), https://arxiv.org/abs/2609.00762 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d7a2de2dbea (Metraipolis archive dc_8d7a2de2dbea, sha256 e7a12fcf8730b79d922ea78fb7f3db84d6279f51523cb489de6a28c567177d2e)
BibTeX
@misc{dc8d7a2de2dbea,
title = {Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation — arXiv:2609.00762},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00762},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d7a2de2dbea; sha256 e7a12fcf8730b79d922ea78fb7f3db84d6279f51523cb489de6a28c567177d2e},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00762. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Parameter-efficient fine-tuning is usually framed as a question of how many parameters to update. Under a severe trainable-state budget, however, where those coefficients act is equally consequential. We study this choice through frozen-core adaptation: a calibration pass fixes left and right bases for each weight matrix, and fine-tuning optimizes only an $r\times r$ core. This removes the ability of trainable factors to repair a poor initial span and makes subspace quality directly observable. We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates. Under a matched $r^2$ budget, we compare eight basis constructors on 11 tasks, four model settings, and three seeds. On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks. It ranks first at all three Qwen scales and finishes within 0.13 points of the best method on Llama-3.2-1B. Controlled ablations show gains of 2.7--17.2 points from whitening and identify QR as necessary for stable core optimization in the tested regime. Finally, FCCA comes within 0.32 and 0.23 average points of LoRA and DoRA while optimizing 36.9K rather than roughly 7.4M parameters. These results show that a carefully selected fixed span can recover most of the benefit of movable low-rank factors at a much smaller trainable and optimizer-state cost. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.00762 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:41:55.819Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).