Metraipolis › archive › dc_d0ac7ef0d313

Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation — arXiv:2608.29560

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.29560
License (as archived)
CC0-1.0
Archived
2026-10-17T08:36:49.718Z
SHA-256 of the archived text
a3b06ef3460461bc93a1766523ca26df271ec0381db7e924a08276239c98ecff
Stable URL of this page
https://metraipolis.com/documento/dc_d0ac7ef0d313

How to cite

Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation — arXiv:2608.29560 — arXiv (en), https://arxiv.org/abs/2608.29560 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d0ac7ef0d313 (Metraipolis archive dc_d0ac7ef0d313, sha256 a3b06ef3460461bc93a1766523ca26df271ec0381db7e924a08276239c98ecff)
BibTeX
@misc{dcd0ac7ef0d313,
  title = {Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation — arXiv:2608.29560},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.29560},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d0ac7ef0d313; sha256 a3b06ef3460461bc93a1766523ca26df271ec0381db7e924a08276239c98ecff},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29560. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A company with a fixed artificial intelligence (AI) budget must decide which large language model (LLM) handles each recurring workload. What it lacks is the quality table, how well each model performs on each workload. Given that table, the decision is a multiple-choice knapsack problem and is routine to solve, so estimating it is the difficulty, and that estimation fails in two ways. Models are rarely compared on the same work, and the recorded score is usually a proxy rather than the outcome the company values. Causal and off-policy methods repair the first but condition on the second, while evaluator-validation methods estimate the second but stop short of the decision. Worse, buying more re-evaluation cannot settle the second: randomization governs which requests are scored, not how a score is produced, so the table stays uncertain however much evaluation is purchased. Yet the deployment decision may still be determined even when the table is not. We therefore ask whether one assignment stays optimal across every quality table consistent with the evidence. For the fixed-budget problem, this admits an exact two-solve certificate: solve once at the estimated table and once at a least-favourable table. Agreement certifies the assignment; disagreement identifies the model-workload pairs where further evidence can matter. We propose CASE (causal active sequential experimentation), which targets evaluation to those pairs and repeats the test as evidence accumulates. On a production log, the measurement failure is the larger of the two: correcting assignment exactly still leaves most of the loss, and randomized re-evaluation does not remove it. In our experiments, the available evidence often does not determine the assignment. On paid software tasks, better information about model quality yields more savings than further optimization of the assignment on the same estimates.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2608.29560 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T08:13:26.752Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).