Metraipolis › archive › dc_41b16fd71fb3

AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop? — arXiv:2609.35025

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35025
License (as archived)
CC0-1.0
Archived
2026-10-07T20:58:28.014Z
SHA-256 of the archived text
939b7ec4097abe305ca6941bc019adae0ec43cdd8e1c5657924a88b8f24c6fbb
Stable URL of this page
https://metraipolis.com/documento/dc_41b16fd71fb3

How to cite

AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop? — arXiv:2609.35025 — arXiv (en), https://arxiv.org/abs/2609.35025 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_41b16fd71fb3 (Metraipolis archive dc_41b16fd71fb3, sha256 939b7ec4097abe305ca6941bc019adae0ec43cdd8e1c5657924a88b8f24c6fbb)
BibTeX
@misc{dc41b16fd71fb3,
  title = {AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop? — arXiv:2609.35025},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35025},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_41b16fd71fb3; sha256 939b7ec4097abe305ca6941bc019adae0ec43cdd8e1c5657924a88b8f24c6fbb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35025. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent gains in language model capability have come more from data than from architecture. Frontier labs and data companies produce verifiable agentic tasks, which supervised finetuning and reinforcement learning then turn into this http URL production line still rests on human labour and on human-in-the-loop collaboration. Automating task creation would let data production scale with compute rather than with expert headcount, would extend to more domains, and would enable a key step in recursive self-improvement (RSI). Current evaluations of an agent's ability to write such tasks measure how a model performs after training on what the agent produced. That does not match common practice in the data industry, where data is delivered sample by sample and each sample is accepted against a set of criteria rather than put straight into training. No existing evaluation asks whether an individual task meets the acceptance criteria of a data pipeline. We therefore introduce AutoDataBench. Given an original benchmark task and a record of the target model attempting it, an agent must write a new task for the same suite that meets practical acceptance standards on validity, novelty, difficulty and behavioural coverage. Across three benchmarks of executable agent tasks, no agent we evaluate scores above 20 out of 100 at the default time budget of 45 minutes. Giving the strongest agent four times as long improves its score substantially, while the cost of one usable task stays almost unchanged. Current agents can write training tasks of the required quality, but not efficiently. AutoDataBench provides a direct measure of an agent's capacity for autonomous data synthesis: one artifact at a time, judged against the criteria a production pipeline would apply, and without a training run. Code and data are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.35025 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.753Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).