Metraipolis › archive › dc_bc7ef3d5f83c
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning — arXiv:2608.23816
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.23816
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:35:38.716Z
- SHA-256 of the archived text
cec56262b780dba26ccc5e44c2166121bf92c697373fc6be068945f0fa1ed864- Stable URL of this page
https://metraipolis.com/documento/dc_bc7ef3d5f83c
How to cite
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning — arXiv:2608.23816 — arXiv (en), https://arxiv.org/abs/2608.23816 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bc7ef3d5f83c (Metraipolis archive dc_bc7ef3d5f83c, sha256 cec56262b780dba26ccc5e44c2166121bf92c697373fc6be068945f0fa1ed864)
BibTeX
@misc{dcbc7ef3d5f83c,
title = {AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning — arXiv:2608.23816},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.23816},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bc7ef3d5f83c; sha256 cec56262b780dba26ccc5e44c2166121bf92c697373fc6be068945f0fa1ed864},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23816. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We present AQLoRA (Adaptive-Quantization LoRA), a recipe that buys part of that time back. One CPU pass ranks layers by NF4 reconstruction error, with no search and no calibration data, and keeps a fixed fraction of them in fp16. Those layers skip dequantization, which is where the speed comes from. A quality setting adapts every layer. A speed setting adapts only the top blocks, so the backward pass stops early. The selection matches the accuracy of Unsloth's hand-curated dynamic-4bit list. We evaluate on Commonsense-170K across six models and four architecture families, from 1.4B to 14B. The speed setting trains 11.3 +/- 2.3% faster than well-tuned QLoRA and gives up 0.1 to 1.5 accuracy points. It was faster in all thirteen independent timing sessions, at worst by 7%. The quality setting trains 4.8 +/- 2.0% faster, matches QLoRA's accuracy on every model, and is within a point of fp16 LoRA. The memory premium depends on the architecture: 0.06 to 0.8 GiB on the 1-4B models, 0.5 GiB at 7B, 1.9 GiB at 14B. These error bars are measured between independent sessions, not within one. Timing on shared hardware needs three rules: fix the measurement duration, not the step count; measure the noise floor from a duplicated arm; and repeat whole sessions, since on one model three sessions that each passed the floor test spread over 8.2 points. We validate the recipe with controls and report the two that failed. Choosing adapter layers by weight density is no better than random. Choosing protected layers by quantization error is not either, on five models at three budgets with memory-matched controls and downstream on two. The number of protected layers sets the speed; which layers they are does not measurably change accuracy. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.23816 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:28:52.189Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).