Metraipolis › archive › dc_c1e189bb6a8d

From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science — arXiv:2610.30495

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30495
License (as archived)
CC0-1.0
Archived
2026-10-25T08:32:30.606Z
SHA-256 of the archived text
a1d501aa84623fb2453264b20e86d4c85f178813e4fca591fbc211c19cd8bc58
Stable URL of this page
https://metraipolis.com/documento/dc_c1e189bb6a8d

How to cite

From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science — arXiv:2610.30495 — arXiv (en), https://arxiv.org/abs/2610.30495 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c1e189bb6a8d (Metraipolis archive dc_c1e189bb6a8d, sha256 a1d501aa84623fb2453264b20e86d4c85f178813e4fca591fbc211c19cd8bc58)
BibTeX
@misc{dcc1e189bb6a8d,
  title = {From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science — arXiv:2610.30495},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30495},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c1e189bb6a8d; sha256 a1d501aa84623fb2453264b20e86d4c85f178813e4fca591fbc211c19cd8bc58},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30495. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI agents now run experiments and report findings, so research produces claims faster than researchers can check them. Each empirical claim implies a property that must hold over its cases, but it states that property only in prose. Without an explicit property, no test can decide which evidence would falsify a claim, so validation cannot keep pace with the claims agents produce. To make claims testable, we approximate a claim's semantics by abstracting its property skeleton, a parameterized relation that recurs across claims. We present ARATest, a property-based testing library that checks these skeletons as executable tests. A typed declaration binds a claim to a skeleton and fixes the claim's compared quantities, cases, scope, and numerical policy. A shared case type makes every generator supply cases in the claim's statistical unit. A three-valued semantics assigns each property one of three verdicts: holds, violated, or inconclusive. Every violation carries a replayable witness. A default generator reads the evidence that agent-native research artifacts (ARA) link to each claim, and auto-research generators dispatch an agent to run released code for fresh cases. Our DissClaimer plugin supplies that agent and runs these generators as continuous testing. Of 243 claims in 33 ARA artifacts, 54 received faithful declarations bound to evidence: 32 held, eight were violated, and 14 remained inconclusive. Two violations exposed errors in published papers that their authors confirmed. ARATest detected all 553 seeded faults without a false alarm on 19 valid controls, outperforming two LLM judges. Fresh runs exposed behaviors that reported evidence could not show. These results suggest that one fixed test can check explicit claim properties against both reported and fresh evidence.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.30495 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:12.997Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).