Metraipolis › archive › dc_aeec96dd35dc

Perplexity Cost Understates What Activation Quantisation Breaks — arXiv:2609.23125

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.23125
License (as archived): CC0-1.0
Archived: 2026-10-09T07:07:03.018Z
SHA-256 of the archived text: 8958da0c127b2f977b35b23ac75e40db9a72254713357441fd577f0296b82eb0
Stable URL of this page: https://metraipolis.com/documento/dc_aeec96dd35dc

How to cite

Perplexity Cost Understates What Activation Quantisation Breaks — arXiv:2609.23125 — arXiv (en), https://arxiv.org/abs/2609.23125 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_aeec96dd35dc (Metraipolis archive dc_aeec96dd35dc, sha256 8958da0c127b2f977b35b23ac75e40db9a72254713357441fd577f0296b82eb0)

BibTeX

@misc{dcaeec96dd35dc,
  title = {Perplexity Cost Understates What Activation Quantisation Breaks — arXiv:2609.23125},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23125},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_aeec96dd35dc; sha256 8958da0c127b2f977b35b23ac75e40db9a72254713357441fd577f0296b82eb0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23125. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Activation quantisation is usually evaluated with an aggregate metric, perplexity, averaged over every token a model predicts. We ask whether that average identifies which computations a quantiser damages. Perplexity turns out to be a reliable aggregate signal: across 12 models from four families and 780 within-model comparisons, the arm perplexity prefers also retains more induction and more retrieval in all but 2.1 and 4.0 percent of cases respectively. But where perplexity has risen by only a factor of 1.2 to 1.5, induction still keeps 0.959 of its intact accuracy while retrieval has already fallen to 0.554, a gap the aggregate number does not surface. This gap has structure, not just size: a matched Gaussian-noise control of the same per-channel magnitude leaves it largely intact, and randomising only the sign of the quantisation error, every magnitude held fixed, is nearly as harmless, so magnitude alone does not explain the damage. Quantising in a rotated basis, which changes coordinate alignment without changing error magnitude, restores induction from 0.001 to 0.980 at three average bits per token in a single-block intervention, though retrieval recovers less completely at the same setting (0.694); end-to-end at four average bits, induction reaches 0.968 and retrieval 0.534. The pattern holds on two further models up to 32B parameters and, in the deployed configurations we tested, under AWQ once activations are pushed to 4 bits. A perplexity target bounds the average cost of a transformation applied to the activation; it does not, by itself, show which computations survived.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.23125 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.267Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).