Metraipolis › archive › dc_ebc4144e330a
Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism? — arXiv:2609.31796
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.31796
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:59:32.895Z
- SHA-256 of the archived text
bb912313578feef194aa26c9f88cce23f5573924876d34666afaae278e686e4b- Stable URL of this page
https://metraipolis.com/documento/dc_ebc4144e330a
How to cite
Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism? — arXiv:2609.31796 — arXiv (en), https://arxiv.org/abs/2609.31796 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ebc4144e330a (Metraipolis archive dc_ebc4144e330a, sha256 bb912313578feef194aa26c9f88cce23f5573924876d34666afaae278e686e4b)
BibTeX
@misc{dcebc4144e330a,
title = {Same Probe, Different Numbers: Are Activation Probes Robust to Inference-Time Numerical Non-Determinism? — arXiv:2609.31796},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.31796},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ebc4144e330a; sha256 bb912313578feef194aa26c9f88cce23f5573924876d34666afaae278e686e4b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31796. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Activation probes are increasingly used to monitor LLMs in deployment. A probe is typically trained under one inference configuration, then used under whatever batch size and numerical precision the serving stack uses. Because common GPU kernels are not batch-invariant and floating-point formats round differently, the activations seen at deployment are not the ones the probe was trained on. We measure what that costs for Llama-3.1-8B, Qwen3-8B and Gemma-3-4B across batch sizes 4, 8 and 16 and float32, bfloat16 and float16, training 768 probes on one configuration, evaluating each on every other, and comparing verdicts example by example. Probes are stable, but aggregate accuracy is the wrong instrument for showing it: it understates how many verdicts change by a factor of two to nine. At the prompt, accuracy never moves by more than 0.47 percentage points across 1,392 transfers and only 0.076% of verdicts change; under float32 with only the batch size varied, none of 201,960 verdicts change. During decoding the flip rate rises to 2.8%, but rows whose realised tokens matched flip in only 0.12-0.15% of cases, while rows whose tokens diverged flip in 12.9%: the cause is the text, not the arithmetic. A bfloat16 batch-size change flips the first generated token for 2.1% of rows and leaves 25% on different tokens by token 20. Flips are symmetric, Cohen's kappa stays above 0.94, and AUROC moves by at most 0.05 points. Underneath, activations move about as much as the format's rounding: a bfloat16 batch-size change perturbs them by a median relative L2 of 1e-2, roughly 8x the float16 figure. Probes absorb this; the model's own next-token argmax does not. Robustness evaluations of activation monitors should report per-example agreement rather than aggregate accuracy, separate representational noise from input change, and state the serving configuration. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.31796 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:02.150Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).