Metraipolis › archive › dc_264603ee24d2

Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures — arXiv:2609.23117

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.23117
License (as archived)
CC0-1.0
Archived
2026-10-09T07:07:02.459Z
SHA-256 of the archived text
3cc2fc22dba8edf1d5ab3d0bb97df4157359014d3a81d044dd26ddb885f63348
Stable URL of this page
https://metraipolis.com/documento/dc_264603ee24d2

How to cite

Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures — arXiv:2609.23117 — arXiv (en), https://arxiv.org/abs/2609.23117 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_264603ee24d2 (Metraipolis archive dc_264603ee24d2, sha256 3cc2fc22dba8edf1d5ab3d0bb97df4157359014d3a81d044dd26ddb885f63348)
BibTeX
@misc{dc264603ee24d2,
  title = {Whitening Inverts the Hierarchy: What the Norm of a Whitened Embedding Measures — arXiv:2609.23117},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.23117},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_264603ee24d2; sha256 3cc2fc22dba8edf1d5ab3d0bb97df4157359014d3a81d044dd26ddb885f63348},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.23117. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Whitening a foundation-model embedding and using its squared norm as a training-free likelihood surrogate is motivated by the observation that whitened coordinates often appear approximately standard normal. We show that this observation follows from the projection central limit theorem and therefore does not imply a Gaussian joint distribution. Across multiple encoders and three training objectives, we find systematic over-dispersion of the whitened radius relative to the Gaussian reference, including against distributional clones with identical mean and covariance. We further show that the commonly reported agreement between empirical and theoretical norm statistics is an algebraic consequence of in-sample whitening and does not constitute evidence for Gaussianity. We identify the mechanism behind this behavior: whitening reverses the encoder's spectral hierarchy, shifting the contribution to the squared norm toward near-degenerate directions that encode predominantly noise. In these directions, the dominant variability is governed by a single input-dependent scale. We estimate this scale from two moments and use it to predict, without additional free parameters, the cross-dependence between disjoint spectral halves. These results indicate that the squared whitened norm is better interpreted as a Mahalanobis measure of semantic atypicality than as a log-likelihood. This interpretation explains both its practical effectiveness and its calibration failures: the statistic can rank and detect atypical samples consistently with nonparametric density estimates and across encoders trained with different objectives, while Gaussian tail thresholds can be inaccurate by orders of magnitude. etc.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.23117 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:58:09.265Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).