Metraipolis › archive › dc_b356d4655bbb
Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves — arXiv:2609.40190
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.40190
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T13:14:29.130Z
- SHA-256 of the archived text
eac8941bb97dc1a06b7b5213b26b3cf39a99f17b401a33190001137684f74092- Stable URL of this page
https://metraipolis.com/documento/dc_b356d4655bbb
How to cite
Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves — arXiv:2609.40190 — arXiv (en), https://arxiv.org/abs/2609.40190 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b356d4655bbb (Metraipolis archive dc_b356d4655bbb, sha256 eac8941bb97dc1a06b7b5213b26b3cf39a99f17b401a33190001137684f74092)
BibTeX
@misc{dcb356d4655bbb,
title = {Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves — arXiv:2609.40190},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.40190},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b356d4655bbb; sha256 eac8941bb97dc1a06b7b5213b26b3cf39a99f17b401a33190001137684f74092},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40190. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Sampling several answers and keeping the one a verifier scores highest is one of the simplest ways to buy accuracy at test time. Its effect is reported as a scaling curve: accuracy against the number $k$ of sampled answers. The curve is cheap to draw and expensive to trust. A budget read off it is chosen after looking at every point, so only a band that covers all budgets at once protects the choice, and on a 100-question benchmark a fixed exact-binomial design needs 192,000 generated answers to certify 64 budgets to within $\pm1/32$ at 95%. Most of that cost pays for the wrong uncertainty. A benchmark is a fixed list of questions; at budget 64, about three quarters of the variance of a selected answer's correctness lies between questions, and an audit that revisits every question need not pay for it. We derive the minimax cost of certifying the whole curve, up to logarithmic factors. It has three parts: calibrating the tail of the score distribution, telling the questions apart, and within-question noise summed along the curve. At a single benchmark the last part sharpens to the variance of one answer's influence under the best allocation of answers to questions, which every valid audit pays and an audit that learns the allocation attains, up to a logarithm, as the precision grows. A paired audit built on an exponential inequality for two independent draws at the same question needs no pilot. On 185 held-out score pools it uses 0.74 times the answers of the cheapest competing certified audit at 64 budgets and 0.53 times at 1,024, and on a newly generated MMLU-Pro study it certified the curve with 79,133 answers, within 0.6% of what a cost law fitted beforehand predicted from the study's within-question variance. The same paths certify pass@$k$ and majority voting, and the bands extend to populations of questions and to answers that depend on earlier ones. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Statistics Theory (math.ST); Machine Learning (stat.ML). Pagina del lavoro: https://arxiv.org/abs/2609.40190 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:44:51.982Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).