Metraipolis › archive › dc_491502e16bf2

Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence — arXiv:2608.14903

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.14903
License (as archived)
CC0-1.0
Archived
2026-10-18T17:08:13.003Z
SHA-256 of the archived text
ced3ca43176aadda83ff6ce6fb1ed7b8fbef27d4a9ca95e6a0ee383f98ad50d3
Stable URL of this page
https://metraipolis.com/documento/dc_491502e16bf2

How to cite

Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence — arXiv:2608.14903 — arXiv (en), https://arxiv.org/abs/2608.14903 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_491502e16bf2 (Metraipolis archive dc_491502e16bf2, sha256 ced3ca43176aadda83ff6ce6fb1ed7b8fbef27d4a9ca95e6a0ee383f98ad50d3)
BibTeX
@misc{dc491502e16bf2,
  title = {Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence — arXiv:2608.14903},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.14903},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_491502e16bf2; sha256 ced3ca43176aadda83ff6ce6fb1ed7b8fbef27d4a9ca95e6a0ee383f98ad50d3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14903. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief. This paper audits whether the public measurement record supports those connections before another trend is fitted. I construct a frozen, event-centric record through 12 August 2026 with 62 selected systems, 12 versioned benchmarks, seven capability or impact criteria, 144 graded events, 27 source records, and 408 typed relations. The record is an audit sample, not a census. Only seven systems jointly observe estimated training compute and a METR 50 percent task horizon. Training compute is absent for 19 of 27 closed systems, including every selected closed release from 2026, while none of the 35 open-weight systems has a METR horizon observation. Benchmark succession creates a second break: a seven-system link from METR Time Horizon 1.0 to 1.1 has a log-scale slope of 1.206 (95 percent CI 1.021 to 1.390), whereas a six-system MMLU to MMLU-Pro comparison appears shift-like under logit and probit links but not under linear or logarithmic links. The observed bridges have about 80 percent power only for slope departures near 25 percent. Provenance is concentrated: 52 of 71 substantive quantitative events, or 73.2 percent, come from one measurement programme, and 76.1 percent are laboratory releases. A review of 56 methodological and empirical sources identifies 16 complementary measurement directions spanning resources, inference budgets, reliability, agentic work, safety, human preference, field outcomes, and forecast backtesting. No direction supplies a replacement scalar. The result is not that frontier AI forecasting is impossible, but that a defensible dated forecast is a claim about a versioned measurement system with explicit joins, protocols, links, and source dependence, not merely a fitted curve or calendar date.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.14903 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:13.745Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).