Metraipolis › archive › dc_fa1408258d93

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families — arXiv:2608.08029

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.08029
License (as archived)
CC0-1.0
Archived
2026-10-19T08:58:30.484Z
SHA-256 of the archived text
2f2543e15d3509f67a67918ec4ca6b5c303bc1010bd2e8cf04a32b7ea97fd2b1
Stable URL of this page
https://metraipolis.com/documento/dc_fa1408258d93

How to cite

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families — arXiv:2608.08029 — arXiv (en), https://arxiv.org/abs/2608.08029 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa1408258d93 (Metraipolis archive dc_fa1408258d93, sha256 2f2543e15d3509f67a67918ec4ca6b5c303bc1010bd2e8cf04a32b7ea97fd2b1)
BibTeX
@misc{dcfa1408258d93,
  title = {Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families — arXiv:2608.08029},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.08029},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa1408258d93; sha256 2f2543e15d3509f67a67918ec4ca6b5c303bc1010bd2e8cf04a32b7ea97fd2b1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.08029. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Khatri et al. (2026) [DOI: https://doi.org/10.1109/DSN-W70714.2026.00027 ] show that lightweight MLP probes on final-layer activations of a single 8B model (LLaMA-3.1-8B) detect harmful prompts at F1 competitive with guard models 1000x larger, using one probe per benchmark. We reproduce this pipeline end-to-end and extend it along two axes the original study leaves open. First, we test whether the result generalizes across other model architecture and scale by training identical probes on activations from models like Gemma-4-E4B, Mistral-7B-v0.3, and Qwen2-7B, using the three benchmarks (WildJailbreak, BeaverTails, AEGIS 2.0). Second, we test how much of the reported performance is affected by non-determinism during inference by repeating extraction under five random seeds and measuring the variance of F1 scores. Our results reproduce the original LLaMA model benchmarks within 0.37 percentage points of the original F1 scores (and within 0.2 points on BeaverTails). We find that the original MLP probe architecture extends to other model families with F1 scores within a point of the values reported for LLaMA-3.1-8B. Our experiments varying seed values reveal an interesting observation: final token latent vectors remained the same for all tested architectures irrespective of the seed values used.

Tema degli esploratori: fiducia digitale e crittografia (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CR).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2608.08029 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:54:24.990Z dalla pagina https://arxiv.org/catchup/cs.CR/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).