Metraipolis › archive › dc_ff49b0c2c781

Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades — arXiv:2609.01345

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01345
License (as archived)
CC0-1.0
Archived
2026-10-17T06:32:26.937Z
SHA-256 of the archived text
ff53c40a77fcd8d575204d315f575ca8f819e80054ef8f7d85f94b63433e91f1
Stable URL of this page
https://metraipolis.com/documento/dc_ff49b0c2c781

How to cite

Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades — arXiv:2609.01345 — arXiv (en), https://arxiv.org/abs/2609.01345 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff49b0c2c781 (Metraipolis archive dc_ff49b0c2c781, sha256 ff53c40a77fcd8d575204d315f575ca8f819e80054ef8f7d85f94b63433e91f1)
BibTeX
@misc{dcff49b0c2c781,
  title = {Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades — arXiv:2609.01345},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01345},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff49b0c2c781; sha256 ff53c40a77fcd8d575204d315f575ca8f819e80054ef8f7d85f94b63433e91f1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01345. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Inference cascades cut cost by answering most queries with a cheap model and escalating a hard tail to a frontier model that acts as verifier. A natural extension closes the loop: fine-tune the cheap student on the verifier's rejections so the escalation rate, and cost, fall each round. We measure this loop on real LLMs and report four findings. First, the verifier's blind spot, the fraction of the student's wrong answers it accepts, is large and moves adversarially: it grows with student capability ($\beta$ from 0.12 to 0.55 as the student scales 0.5B to 32B) and shrinks with verifier capability, so it is worst in the cheap-student, cheap-verifier regime cascades exist to create. Second, buying it away returns the saving: a frontier verifier drives $\beta$ to about 0.05 but then escalates on 46% of hard-MATH queries against a 39% true error rate, paying the frontier price on nearly half of all traffic. Third, naive corrective fine-tuning on the verifier-rejected tail does not improve the small student but degrades and ultimately collapses it, across every teacher we tried (cross-family and same-family), so at this scale the self-improving loop is self-defeating. Fourth, through all of this the cascade's own dashboard, every metric computed through the verifier, reads a flat 3% error while true delivered error swings up to 32%: the system is blind to its own degradation by construction. We then give the theory that explains the blindness, a two-population conservation law, $\epsilon_\infty \lesssim q_0 \beta_0$, under which every in-loop metric improves while true quality does not, and a synthetic study that validates the mechanism. The practical conclusion: the reliability of a self-improving cascade cannot be read from any metric computed through its own verifier.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Cryptography and Security (cs.CR); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.01345 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:15:12.690Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).