Metraipolis › archive › dc_ee9eb57dd30d

Iterated creation, grading, and revision of data science projects by language models — arXiv:2611.05548

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05548
License (as archived)
CC0-1.0
Archived
2026-11-05T18:16:36.347Z
SHA-256 of the archived text
732a11eea6095e4ef0030177c1a06d5e47a632fb2dc1bdceca0645623bfe8361
Stable URL of this page
https://metraipolis.com/documento/dc_ee9eb57dd30d

How to cite

Iterated creation, grading, and revision of data science projects by language models — arXiv:2611.05548 — arXiv (en), https://arxiv.org/abs/2611.05548 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ee9eb57dd30d (Metraipolis archive dc_ee9eb57dd30d, sha256 732a11eea6095e4ef0030177c1a06d5e47a632fb2dc1bdceca0645623bfe8361)
BibTeX
@misc{dcee9eb57dd30d,
  title = {Iterated creation, grading, and revision of data science projects by language models — arXiv:2611.05548},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05548},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ee9eb57dd30d; sha256 732a11eea6095e4ef0030177c1a06d5e47a632fb2dc1bdceca0645623bfe8361},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05548. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We evaluate the extent to which agentic AI models can complete an end-to-end data science project, and how this improves with feedback. A 'writer' model with specific tools creates a complete data science project including downloading, cleaning, and analyzing a dataset, and writing a short paper about it. The project is executed and a 'grader' model evaluates it. The writer can then revise its submission, and this repeats for a total of three rounds. We consider seven models from five developers: Anthropic, OpenAI, Google, DeepSeek, and this http URL . Each model is both a writer and a grader, so there are 49 writer-grader pairs and we run each pair three times. Of the 147 initial submissions, 107 run and render, and the average score increases from 59 per cent to 76 per cent after the final round of feedback. By writer the extent of the increase ranges from 3 percentage points for Anthropic's Opus 5.5 to 46 percentage points for Anthropic's Haiku 4.5, and by grader it ranges from 13 percentage points for Google's Gemini 3.8 Flash to 32 percentage points for Anthropic's Haiku 4.5. When a model grades its own writing the score tends to be about 9 percentage points higher than the average of what the other models give it. About half of the difference that we see from round to round is within the grader's own noise. Feedback generally results in the highest marks, but even just having the chance to revise, or having the additional budget, in terms of time or tool calls, without any feedback, results in considerable increases for some of the models but not for others. We find that how much such a loop helps depends on which model plays which role, but weaker models generally do not improve stronger models.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.05548 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:12:48.988Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).