Metraipolis › archive › dc_68fea8d017eb
MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training — arXiv:2609.15037
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15037
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T23:17:12.174Z
- SHA-256 of the archived text
4162d4473733c3bb4916cae87bc6191b67297c93b0dcd1525ccf19576b377b1d- Stable URL of this page
https://metraipolis.com/documento/dc_68fea8d017eb
How to cite
MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training — arXiv:2609.15037 — arXiv (en), https://arxiv.org/abs/2609.15037 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_68fea8d017eb (Metraipolis archive dc_68fea8d017eb, sha256 4162d4473733c3bb4916cae87bc6191b67297c93b0dcd1525ccf19576b377b1d)
BibTeX
@misc{dc68fea8d017eb,
title = {MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training — arXiv:2609.15037},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15037},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_68fea8d017eb; sha256 4162d4473733c3bb4916cae87bc6191b67297c93b0dcd1525ccf19576b377b1d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15037. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Low-rank gradient projection reduces the optimizer-state memory cost of large language model (LLM) pretraining, but the steps and wall-clock time needed to reach a target quality remain a meaningful axis for improvement. We attribute this to two design choices in existing methods: the projection-rank budget is allocated uniformly across Transformer modules with heterogeneous projection sensitivity, and projecting a raw gradient attenuates its magnitude and direction jointly. We propose MoARa, which combines a static profiling-based module-aware projection-rank allocation with a block-wise magnitude-direction decomposition; the default block size is set in the neighborhood of the attention head dimension. Across five Transformer architectures spanning Llama, Qwen, and DeepSeek at 300M to 7B scales, GaLore with MoARa reaches standard GaLore's final perplexity in 37% fewer steps and 34% less wall-clock time on Llama 2 7B, with only 0.2% peak reserved memory overhead under standard graph compilation. Across the six low-rank pretraining methods we evaluate, module-aware rank allocation alone delivers directionally consistent step reductions on all six. On compatible hosts, the two-component design reaches up to 41.7% step reduction and 37.1% wall-clock reduction. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.15037 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:51:10.032Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).