Metraipolis › archive › dc_d2fa79c1311e
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers — arXiv:2609.34866
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34866
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:03:01.617Z
- SHA-256 of the archived text
2d2785612b4e99bbd954f8afa782f8811a6d151efa5277fec45a6d2b11ead5bf- Stable URL of this page
https://metraipolis.com/documento/dc_d2fa79c1311e
How to cite
From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers — arXiv:2609.34866 — arXiv (en), https://arxiv.org/abs/2609.34866 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d2fa79c1311e (Metraipolis archive dc_d2fa79c1311e, sha256 2d2785612b4e99bbd954f8afa782f8811a6d151efa5277fec45a6d2b11ead5bf)
BibTeX
@misc{dcd2fa79c1311e,
title = {From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers — arXiv:2609.34866},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34866},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d2fa79c1311e; sha256 2d2785612b4e99bbd954f8afa782f8811a6d151efa5277fec45a6d2b11ead5bf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34866. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Most post-training quantization pipelines fit each weight matrix to its pretrained counterpart, one matrix at a time. Whether that proxy tracks what an attention block actually computes, or how errors in the Q, K and V projections compound inside the softmax, is rarely checked. We write the objective on the attention output instead, over all three projections at once, and reuse it throughout the pipeline. JAB defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it twice: to fit the quantized weights (GPTQ warm start, then STE with learnable scales), and to score the block for a multiple-choice knapsack allocation. On attention-only quantization of Mistral-7B this works. At 3 bits JAB recovers 77-90% of the gap between uniform GPTQ and full precision, and its sensitivity estimate tracks an oracle costing 73 forward passes to within a fraction of a point. It stops working once MLP layers enter the allocation. A role-aware offset rule needing no sensitivity estimate at all beats JAB on GPT-2's MLP and on the full Mistral-7B model: with a 3-bit floor it quantizes 96.4% of the weights to 4.5 bits per parameter at 6.933 perplexity, within 4.4% of full precision (6.643) at 3.56x compression, against 7.158 for JAB at the same budget. Which matrix a weight sits in matters more than any sensitivity estimate we computed. Two things came out sideways. Block-local reconstruction is an unreliable proxy for end-to-end perplexity: one run improved a block's own objective 4.6x while perplexity rose 32x, which is why every allocation here is validated end-to-end. And on attention-only quantization, fine-tuning moved weights farther from their pretrained values while pulling attention outputs closer, with net gains. Post-training seems to recover attention behavior, not weights. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.34866 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:03.181Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).