Metraipolis › archive › dc_33f99b0914f2
Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs — arXiv:2609.04526
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04526
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:35:00.749Z
- SHA-256 of the archived text
b7fe33e5b22ddcd8fae8f852492f972c81c026d02f797780a8d2a45c542bb11f- Stable URL of this page
https://metraipolis.com/documento/dc_33f99b0914f2
How to cite
Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs — arXiv:2609.04526 — arXiv (en), https://arxiv.org/abs/2609.04526 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_33f99b0914f2 (Metraipolis archive dc_33f99b0914f2, sha256 b7fe33e5b22ddcd8fae8f852492f972c81c026d02f797780a8d2a45c542bb11f)
BibTeX
@misc{dc33f99b0914f2,
title = {Scale-QLoRA: Code-Invariant Adapter Merging for Native 4-bit Microscaling LLMs — arXiv:2609.04526},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04526},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_33f99b0914f2; sha256 b7fe33e5b22ddcd8fae8f852492f972c81c026d02f797780a8d2a45c542bb11f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04526. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Merging a LoRA adapter into its base model is standard deployment practice: it removes the runtime adapter's per-forward overhead and leaves a single standalone checkpoint any serving stack can load. On a native 4-bit microscaling checkpoint (NVFP4, MXFP4) that step stops being free. The merged weights must be written back through a quantizer, which re-derives the checkpoint's discrete E2M1 code plane (roughly 90% of the artifact's bytes), so the deployed artifact becomes coupled to one quantization convention, and every later code-touching event in its lifecycle can move it. Done naively the step is worse than fragile: it deletes the adaptation, by up to 39 pp, because against an already-on-grid base the reconstruction optimum is that base. Scale-QLoRA instead adapts only the native per-block scale field, trains those scales on the deployment grid, and freezes every E2M1 code. Within a fixed native format, scale grid, block layout and code plane, merging is then a bit-exact identity and the merged artifact is code-invariant. Across four models and four tasks, Scale-QLoRA and merge-aware QAT-LoRA are both accuracy-lossless, so we claim no accuracy ordering between them; they differ structurally, in that QAT-LoRA re-derives the code plane through a quantizer while Scale-QLoRA preserves it exactly. That difference is what the lifecycle prices: nearest-rounding implementations disagree by about a point on the measured task, and more extreme rule mismatches can drive the weight-space artifact to ~0%, which we report as a sensitivity bound rather than a deployment frequency. Preserving the code plane also drops the weight-space straight-through estimator from training (3.9x per step on the dense 8B model) and enables exact rollback, code-plane deduplication, and a ~125x faster scale-only task swap. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.04526 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:21:16.269Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).