Metraipolis › archive › dc_809728acb8dc
When Pair Count Is Not the Sample Size: What All-Pairs Agent Comparisons Estimate — arXiv:2609.33012
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33012
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:56:21.292Z
- SHA-256 of the archived text
ed295b3521c22b87e20ad6de008be5593e662e9c0d7e442a287444454c6c7895- Stable URL of this page
https://metraipolis.com/documento/dc_809728acb8dc
How to cite
When Pair Count Is Not the Sample Size: What All-Pairs Agent Comparisons Estimate — arXiv:2609.33012 — arXiv (en), https://arxiv.org/abs/2609.33012 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_809728acb8dc (Metraipolis archive dc_809728acb8dc, sha256 ed295b3521c22b87e20ad6de008be5593e662e9c0d7e442a287444454c6c7895)
BibTeX
@misc{dc809728acb8dc,
title = {When Pair Count Is Not the Sample Size: What All-Pairs Agent Comparisons Estimate — arXiv:2609.33012},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33012},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_809728acb8dc; sha256 ed295b3521c22b87e20ad6de008be5593e662e9c0d7e442a287444454c6c7895},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33012. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
When an agent benchmark compares every pair of leaderboard entries, the number of comparisons can look much larger than the independent evidence behind them: A versus B and A versus C both reuse A. Whether this reuse affects inference depends on what the analysis is meant to describe. If the board and its outcomes are fixed, the all-pairs mean is an exact summary of those entries, and any interval must come from another declared source of randomness. If the entries are instead treated as iid draws from a population of future configurations and the pair rule is regular and nondegenerate, the same mean is an order-two U-statistic whose first-order uncertainty depends on the number of configurations, not the number of pairs. We use near ties as the running example, but the distinction extends to other symmetric pair summaries when their regularity conditions hold. We examine both interpretations using a fixed SWE-bench Verified snapshot and an exact binary model with known truth. On SWE-bench, intervals that accounted for shared configurations were more than twice as wide as a pair-iid reference that treated the pairs as independent. In the exact model, pair-iid coverage fell far below the nominal level when edges shared endpoints but remained near nominal for matched independent edges. Results on two other fixed leaderboards show that exact summaries also depend on which pairs are included and how they are weighted. An all-pairs analysis must therefore state what is fixed, what is sampled, and how it handles shared entries and pair aggregation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.33012 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.131Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).