Metraipolis › archive › dc_15960ca53acc

The Undetected Damage of Quantization on Retrieval and How to Fix It — arXiv:2609.24322

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.24322
License (as archived)
CC0-1.0
Archived
2026-10-09T07:07:39.814Z
SHA-256 of the archived text
38a8a58d365153ba1be3d1ce3a0802d28561dab98a1bc8edfbe29d74877529b5
Stable URL of this page
https://metraipolis.com/documento/dc_15960ca53acc

How to cite

The Undetected Damage of Quantization on Retrieval and How to Fix It — arXiv:2609.24322 — arXiv (en), https://arxiv.org/abs/2609.24322 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_15960ca53acc (Metraipolis archive dc_15960ca53acc, sha256 38a8a58d365153ba1be3d1ce3a0802d28561dab98a1bc8edfbe29d74877529b5)
BibTeX
@misc{dc15960ca53acc,
  title = {The Undetected Damage of Quantization on Retrieval and How to Fix It — arXiv:2609.24322},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.24322},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_15960ca53acc; sha256 38a8a58d365153ba1be3d1ce3a0802d28561dab98a1bc8edfbe29d74877529b5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.24322. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We show that a quantized model that keeps its classification accuracy still changes $14$ to $46\%$ of its top-1 retrieval results, and that aggregate ranking metrics reveal only part of this damage. We tie this failure to the gap between the two highest model scores and use that gap to decide when to trust a quantized answer and where additional precision should be spent. We show that the top-1 result is guaranteed to survive quantization only when this gap exceeds twice the largest rounding error. In classification, the scores are logits, and training compares the correct class against every other class, which encourages this gap. In retrieval, the scores are query-document similarities, and training compares each positive only against sampled negatives, so nothing separates the top-1 item from the second. This gap can be measured without labels. Before deployment, it predicts which models will break under quantization, and at deployment time it tells, per input, whether the quantized answer still matches the full-precision answer. Most classification inputs have a gap wide enough to trust the quantized answer, but few retrieval queries do. That gap motivates a different fix in each task. In retrieval, spending extra bit-width on the layers whose quantization moves the gap most recovers up to three-quarters of an extra bit's benefit for half its cost. In classification, routing the few low-gap inputs to full precision recovers most of the lost accuracy at a fraction of the cost.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.24322 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.395Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).