Metraipolis › archive › dc_8e403e030f4e

QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs — arXiv:2609.39223

Source

arXiv (en)

Original

https://arxiv.org/abs/2609.39223

License (as archived)

CC0-1.0

Archived

2026-10-07T13:45:57.968Z

SHA-256 of the archived text

f756af905255924e6fd10f0efc6fb6d8abca834a6538690735e28797b9737792

Stable URL of this page

https://metraipolis.com/documento/dc_8e403e030f4e

How to cite

QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs — arXiv:2609.39223 — arXiv (en), https://arxiv.org/abs/2609.39223 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8e403e030f4e (Metraipolis archive dc_8e403e030f4e, sha256 f756af905255924e6fd10f0efc6fb6d8abca834a6538690735e28797b9737792)

@misc{dc8e403e030f4e,
  title = {QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs — arXiv:2609.39223},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39223},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8e403e030f4e; sha256 f756af905255924e6fd10f0efc6fb6d8abca834a6538690735e28797b9737792},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39223. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive post-training quantization (PTQ) can degrade model quality. We present QATFactory, an open-source framework for deployment-aligned quantization-aware distillation (QAD) and reinforcement learning (QARL). QATFactory simulates deployment-time quantization while performing matrix multiplications in BF16, allowing models to adapt to quantization noise without requiring training hardware that natively supports the target format; for example, it supports NVFP4 training on H100 GPUs, which lack FP4 Tensor Cores. The framework supports NVFP4, MXFP4, and $\text{llama}.\text{cpp}$'s Q4_K format; dense and mixture-of-experts models; and both full-parameter and LoRA-based training. It exports checkpoints directly to vLLM and $\text{llama}.\text{cpp}$ without an additional lossy conversion step or added inference overhead. With QATFactory, we conduct extensive experiments on models ranging from 8B to 230B parameters and evaluate exported checkpoints in production inference engines. Across models and formats, QAD consistently improves deployed-model quality over strong PTQ baselines. On Qwen3.5-9B, QAD achieves average benchmark accuracies of 68.9% under NVFP4 and 66.0% under MXFP4, outperforming the best PTQ results of 65.4% and 56.4%, respectively. Through our experiments, we found that although both FP4 formats quantize weights and activations at deployment, the best training strategy is format-dependent: NVFP4 generally performs better when only weights are quantized during training, whereas MXFP4 benefits from quantizing both weights and activations. At a fixed training token budget, training on fewer 32K sequences improves average accuracy by 1.9 points over training on more 4K sequences.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.39223 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T13:18:10.341Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).