Metraipolis › archive › dc_6f4aa447481e

Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study — arXiv:2609.31640

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.31640
License (as archived): CC0-1.0
Archived: 2026-10-07T20:59:08.719Z
SHA-256 of the archived text: a7ca214edb6681b58f9c28ff4f693e070f588461822a9c00020eb482bbac4891
Stable URL of this page: https://metraipolis.com/documento/dc_6f4aa447481e

How to cite

Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study — arXiv:2609.31640 — arXiv (en), https://arxiv.org/abs/2609.31640 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f4aa447481e (Metraipolis archive dc_6f4aa447481e, sha256 a7ca214edb6681b58f9c28ff4f693e070f588461822a9c00020eb482bbac4891)

BibTeX

@misc{dc6f4aa447481e,
  title = {Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study — arXiv:2609.31640},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31640},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f4aa447481e; sha256 a7ca214edb6681b58f9c28ff4f693e070f588461822a9c00020eb482bbac4891},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31640. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Continual Learning Bench scores learning as short-horizon gain versus a reset baseline and finds naive full-context ICL strongest among the memories it tested. We treat ICL as one learning system and score it on a longer shared-world schedule. Steady-state learning is the gap versus baseline on a pre-set late window (last 40 of 174 BIRD-SQL formula-1 questions). We split the score into exploration efficiency (SQL probes), task reward (hits), and delivery cost (API dollars and context size). On gpt-5.6-luna, late probes fall from 4.6-5.6 to 0.95 while hits rise only modestly and ICL context grows to about 95k tokens with cost roughly doubling. Short-horizon gain understates the late probe saving and misses the cost inversion, so we find that unbounded ICL is a poor candidate for the learning mechanism.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.31640 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.966Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).