Metraipolis › archive › dc_581e82fe0b46

Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems — arXiv:2608.19140

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.19140
License (as archived)
CC0-1.0
Archived
2026-10-18T13:44:43.742Z
SHA-256 of the archived text
7989c41ab5d6001ede549b3a6abb4c4af829f021b8fa9d33182d89bf4a32969c
Stable URL of this page
https://metraipolis.com/documento/dc_581e82fe0b46

How to cite

Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems — arXiv:2608.19140 — arXiv (en), https://arxiv.org/abs/2608.19140 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_581e82fe0b46 (Metraipolis archive dc_581e82fe0b46, sha256 7989c41ab5d6001ede549b3a6abb4c4af829f021b8fa9d33182d89bf4a32969c)
BibTeX
@misc{dc581e82fe0b46,
  title = {Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems — arXiv:2608.19140},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.19140},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_581e82fe0b46; sha256 7989c41ab5d6001ede549b3a6abb4c4af829f021b8fa9d33182d89bf4a32969c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.19140. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Frontier language models are compared, marketed, and benchmarked on capability -- what their best or average output can achieve. I argue this measures the wrong axis. The models have saturated accuracy: their mean output lands on the target. What now separates one system from another in practice is precision: how tightly concentrated their outputs are around that target across repeated, identical requests. Borrowing the marksman's distinction, capability is where the average shot lands; reliability is the size of the group. I make three claims. First, precision, not capability, is the frontier differentiator between systems, and benchmark culture systematically fails to measure it, reporting central tendency rather than spread. Second, precision is measurable, cheaply and without circularity, by running a fixed suite of deterministically scored tasks many times at fixed temperature and computing the per-task consistency of outcomes -- no model-in-the-loop grader required. Third, the measurement is not merely descriptive but decision-guiding: it separates consistent failures (a tight group off-centre, correctable by the operating discipline of Paper 1 -- a sight adjustment) from scattered failures (a wide group, correctable only by changing the model or its sampling -- a rifle problem). I define a grouping metric, specify a harness, and show how tracking a human-AI pair's grouping over time yields the compounding signal that Paper 1's field study requires. A first real run, since replicated, illustrates both the method and its most important limit: one measured gap was closed completely by a single rule (0/5 -> 5/5), while a suite of tasks authored from the rules themselves found no value, because a frontier model already embodies explicit good practice -- establishing that a discipline's worth is found by measurement on real work, not constructed from its own rulebook.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-20 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computers and Society (cs.CY); Machine Learning (cs.LG); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2608.19140 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T13:41:17.893Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-20?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).