Metraipolis › archive › dc_e2e91bab3516
Tracing distinguishability through transformer processing with stochastic LayerNorm — arXiv:2608.30720
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.30720
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T08:39:20.405Z
- SHA-256 of the archived text
699995441a4118bda395a8219cafa6f2d831eb1d2e4e3cdddc324a8944dada2c- Stable URL of this page
https://metraipolis.com/documento/dc_e2e91bab3516
How to cite
Tracing distinguishability through transformer processing with stochastic LayerNorm — arXiv:2608.30720 — arXiv (en), https://arxiv.org/abs/2608.30720 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e2e91bab3516 (Metraipolis archive dc_e2e91bab3516, sha256 699995441a4118bda395a8219cafa6f2d831eb1d2e4e3cdddc324a8944dada2c)
BibTeX
@misc{dce2e91bab3516,
title = {Tracing distinguishability through transformer processing with stochastic LayerNorm — arXiv:2608.30720},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.30720},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e2e91bab3516; sha256 699995441a4118bda395a8219cafa6f2d831eb1d2e4e3cdddc324a8944dada2c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30720. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied to downstream function: nearby states may produce different behaviors, while distant states may behave similarly. We instead give representations volume, turning similarity into statistical distinguishability. Overlapping stochastic representations necessarily induce overlapping downstream distributions, grounding latent comparison in model function and bringing it under information-theoretic tools such as the data-processing inequality. We realize this idea in pretrained transformers through a light-touch modification to LayerNorm: at each residual-stream read, we normalize the state, add isotropic Gaussian noise, and renormalize. During distillation fine-tuning, one learned allocation parameter per residual-stream read distributes a fixed global rate budget across the processing stack. The resulting model can be viewed as transformer blocks reading the residual stream with learned finite precision under a shared global rate budget. Using the Bhattacharyya coefficient, we trace which counterfactual distinctions are preserved through MLP blocks or selectively exposed to the query, key, and value computations of individual attention heads. Experiments on ViT-S and GPT-2 small reveal the depthwise propagation of continuous visual perturbations and head-specific sensitivity to token distinctions aligned with known attention motifs. These results establish distinguishability as a functionally grounded lens on transformer computation that complements existing interpretability approaches. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.30720 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T08:13:26.837Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).