Metraipolis › archive › dc_ae7f44ba2ce0
Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis — arXiv:2610.03176
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.03176
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T22:51:44.895Z
- SHA-256 of the archived text
b5392e9db969848d78625e3ccc2a1aad96eae5061d0c92f15447054b82cf38e2- Stable URL of this page
https://metraipolis.com/documento/dc_ae7f44ba2ce0
How to cite
Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis — arXiv:2610.03176 — arXiv, https://arxiv.org/abs/2610.03176 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ae7f44ba2ce0 (Metraipolis archive dc_ae7f44ba2ce0, sha256 b5392e9db969848d78625e3ccc2a1aad96eae5061d0c92f15447054b82cf38e2)
BibTeX
@misc{dcae7f44ba2ce0,
title = {Hindsight-Guided Rationale Distillation for Rare Disease Diagnosis — arXiv:2610.03176},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.03176},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ae7f44ba2ce0; sha256 b5392e9db969848d78625e3ccc2a1aad96eae5061d0c92f15447054b82cf38e2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03176. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We study hindsight-guided distillation for rare disease diagnosis on ZebraMap: a 1.5B student is fine-tuned on chain-of-thought traces from a 8B teacher that observes the ground-truth diagnosis during generation. Absolute accuracy remains low for all models - the task is hard at this scale - but within this ceiling a filtered variant (StudentF) achieves a small, statistically significant accuracy advantage over the teacher (p < 0.001), concentrated in better-represented diseases. The unfiltered student does not significantly outperform the teacher (p = 0.129), establishing that contamination filtering - not hindsight distillation alone - drives the gain. The gap traces to an artifact we term GT hallucination. Label-visible generation causes the teacher to embed "ground truth is X" phrases in its reasoning chain; SFT copies the pattern. At inference, the unfiltered student reproduces the phrase in 33.9% of cases, with severe accuracy degradation when the hallucinated label is wrong. A regex filter removing these slots reduces contamination to near-zero, producing the observed gain - though the effect remains small. We precisely quantify this gain-cost tradeoff, document frequency-dependent knowledge transfer absent from the RL-trained teacher, and characterize a calibration gap that SFT does not close - identifying both as directions for future work. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.03176 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T22:46:10.577Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).