Metraipolis › archive › dc_fbe1b0303daa
Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA — arXiv:2609.01361
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.01361
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:41:35.617Z
- SHA-256 of the archived text
119f09d8300492f3e7ac6565caec6e7dd803595c5c534486755a7881b1bf1d25- Stable URL of this page
https://metraipolis.com/documento/dc_fbe1b0303daa
How to cite
Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA — arXiv:2609.01361 — arXiv (en), https://arxiv.org/abs/2609.01361 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbe1b0303daa (Metraipolis archive dc_fbe1b0303daa, sha256 119f09d8300492f3e7ac6565caec6e7dd803595c5c534486755a7881b1bf1d25)
BibTeX
@misc{dcfbe1b0303daa,
title = {Investigating Linear Probe Robustness to Linguistic Register, Medical Specialty, and Corpus Shifts in Medical QA — arXiv:2609.01361},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.01361},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbe1b0303daa; sha256 119f09d8300492f3e7ac6565caec6e7dd803595c5c534486755a7881b1bf1d25},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01361. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Linear classifiers trained on hidden states of a large language model (LLM), linear probes, can flag factual errors from a single forward pass. Geometrically, that implies that true and false statements separate along a stable direction in hidden state space, i.e., the truth direction. Prior work disagrees on whether this generalises across input shifts, but the disagreement is hard to interpret because cross-dataset probe transfer experiments confound several kinds of input change at once. We isolate three such variables in medical question-answering (QA): writing style (register), domain (medical specialty), and corpus (dataset). We build a benchmark using 500 MedQA entries, each rewritten into four styles (textbook, patient, clinical note, colloquial), annotated with clinical specialty, and grouped with two other exam corpora, MedMCQA and MMLU-medical, for cross-dataset evaluation. Probing four open-weight LLMs (2--8B), we find that the truth direction is largely robust to writing style (mean $\Delta_\text{register} \approx 0.10$ AUROC on held-out facts) and to medical specialty ($\Delta_\text{specialty} \approx 0.03$), but degrades unevenly across corpora: by $0.12$ AUROC on MMLU-medical and by $0.21$ on MedMCQA, roughly twice the register gap. The register result replicates with a second generator and carries over to human-written patient questions. The truth direction is therefore largely stable within the medical domain but breaks under some corpus shifts, and question format does not explain the break, which suggests that the signal a linear probe recovers is partly bound to dataset structure rather than to medical knowledge alone.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.01361 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.860Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).