Metraipolis › archive › dc_541194331e22

What Does 99% Accuracy Measure? A Reproducible Audit of Shortcut Learning in a Widely Used Fake News Corpus — arXiv:2609.25006

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.25006
License (as archived)
CC0-1.0
Archived
2026-10-08T14:34:14.470Z
SHA-256 of the archived text
e07420d445eaf9663d628b7cc86f5be299ae4ac0d11524ee537b8168cf11e98e
Stable URL of this page
https://metraipolis.com/documento/dc_541194331e22

How to cite

What Does 99% Accuracy Measure? A Reproducible Audit of Shortcut Learning in a Widely Used Fake News Corpus — arXiv:2609.25006 — arXiv (en), https://arxiv.org/abs/2609.25006 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_541194331e22 (Metraipolis archive dc_541194331e22, sha256 e07420d445eaf9663d628b7cc86f5be299ae4ac0d11524ee537b8168cf11e98e)
BibTeX
@misc{dc541194331e22,
  title = {What Does 99% Accuracy Measure? A Reproducible Audit of Shortcut Learning in a Widely Used Fake News Corpus — arXiv:2609.25006},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.25006},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_541194331e22; sha256 e07420d445eaf9663d628b7cc86f5be299ae4ac0d11524ee537b8168cf11e98e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.25006. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Text classifiers trained on the ISOT/Kaggle "Fake and Real News" corpus routinely report accuracy and F1 above 0.98, a level of performance that sits uneasily beside the difficulty of assessing veracity. Using a transparent TF-IDF and linear-classifier pipeline as a measurement instrument, we audit the corpus along three leakage channels and two distribution-shift protocols, releasing all code and derived numbers. First, the benchmark is partly degenerate: a classifier given only the subject metadata field, with the article text discarded, attains F1 = 1.000, since the two classes have disjoint subjects. Second, removing all three leakage channels, metadata, a newswire source tag present in 99.2% of real articles, and 6,251 duplicate documents contaminating 19.4% of a naive test split, lowers F1 by only 1.21 points (0.9935 to 0.9814); the residual signal is diffuse editorial style rather than a few giveaway tokens, since deleting the 1,000 highest-weight unigrams still leaves F1 = 0.926. Third, this style signal does not transfer: under a topic-disjoint protocol, average precision falls from 0.9995 to 0.9475 and deployed F1 from 0.9905 to 0.8067, with a prior-matched analysis confirming a genuine 5.2-point loss of discrimination, while temporal transfer is nearly lossless. A fine-tuned DistilBERT is stronger in-distribution (F1 = 0.9993) but degrades far more under topic shift, losing 12.9 average-precision points against the linear model's 5.2. Transferred to the independent LIAR benchmark, all three models fall to near-chance ranking (ROC-AUC 0.54-0.57), none beating a majority-class baseline. We conclude that within-corpus scores here quantify source and topic separability rather than veracity, that added capacity exploits the shortcut rather than avoiding it, and we recommend metadata-only, small-sample, and topic-disjoint baselines as inexpensive diagnostics for future work.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.25006 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T13:46:15.635Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).