Metraipolis › archive › dc_afa62bd96af2

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review — arXiv:2609.33153

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33153
License (as archived)
CC0-1.0
Archived
2026-10-07T21:01:05.198Z
SHA-256 of the archived text
2acff26885e67663b0f4204c70ec1c8b42cb2de8c5c7a3eff02ea39b22ab66c3
Stable URL of this page
https://metraipolis.com/documento/dc_afa62bd96af2

How to cite

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review — arXiv:2609.33153 — arXiv (en), https://arxiv.org/abs/2609.33153 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_afa62bd96af2 (Metraipolis archive dc_afa62bd96af2, sha256 2acff26885e67663b0f4204c70ec1c8b42cb2de8c5c7a3eff02ea39b22ab66c3)
BibTeX
@misc{dcafa62bd96af2,
  title = {What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review — arXiv:2609.33153},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33153},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_afa62bd96af2; sha256 2acff26885e67663b0f4204c70ec1c8b42cb2de8c5c7a3eff02ea39b22ab66c3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33153. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reported improvements from tools and reusable skills in large language model agents refer to different comparisons. This critical narrative review examines what these evaluations estimate and which conclusions their designs support. The review checks the roles of one hundred cited papers and extracts focal evaluation designs in detail from thirty-five studies. Targeted readings of thirty-five additional published or accepted studies broaden coverage of tool creation, memory, interactive benchmarks, reliability, and risk. Designs are characterized by treatment contrast, target population, outcome, budget constraint, summary measure, and identification assumptions. Analytic decompositions and counterexamples show that pairing runs on the same task does not itself identify an invocation effect when evaluation conditions on a trigger within the treated run. Paired gain and regression counts describe discordance under the coupling protocol rather than the share of tasks whose expected outcomes worsen. Total effects of deploying a module answer a different question from efficiency under a common budget. Comparisons across studies distinguish curated skill provision from retriever replacement, task populations from triggered subsets, and preparation costs from marginal usage costs. Publication status and reading depth are recorded. The review provides a methodological synthesis and a reporting checklist to help align claims about tools and skills with the comparisons their evaluation designs support.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.33153 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.560Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).