Metraipolis › archive › dc_3e751e30140f

Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness? — arXiv:2610.00025

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00025
License (as archived)
CC0-1.0
Archived
2026-10-06T12:01:17.105Z
SHA-256 of the archived text
5cb132f4237216d62bb2bd6dc77114927607cca8a7b71ff05afa11cf88c7861a
Stable URL of this page
https://metraipolis.com/documento/dc_3e751e30140f

How to cite

Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness? — arXiv:2610.00025 — arXiv (en), https://arxiv.org/abs/2610.00025 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3e751e30140f (Metraipolis archive dc_3e751e30140f, sha256 5cb132f4237216d62bb2bd6dc77114927607cca8a7b71ff05afa11cf88c7861a)
BibTeX
@misc{dc3e751e30140f,
  title = {Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness? — arXiv:2610.00025},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00025},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3e751e30140f; sha256 5cb132f4237216d62bb2bd6dc77114927607cca8a7b71ff05afa11cf88c7861a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00025. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier large language model (LLM) planner: auto-approving shell commands, writing memory, selecting tools, ranking past turns. We ask whether off-the-shelf SLMs meet practitioner-defined thresholds and, when they fail, why, and whether quantization changes the answer. We build a benchmark of 4 such microtasks with fixed prompts and automatic metrics, each with a pre-specified threshold $\tau$ anchored to a cheap non-LLM baseline and a CI-aware eligibility rule (a configuration passes only if its confidence bound clears $\tau$). Sweeping Qwen3 0.6/1.7/4/8B at their best (FP16, greedy, one frozen prompt, no tuning), we find an eligibility gap: 0 of 16 (4 tasks $\times$ 4 models) configurations pass (verified by checking the raw outputs and parser behavior). A logprob decision-threshold diagnostic (T1/T3/T4; T2 via a context-length/cascade probe) separates the failures into capability deficits and failures that can be addressed by changing the decoding threshold (4 regimes). Quantization to 4-bit (RTN/GPTQ/AWQ) does damage that depends on model size and moves no configuration into eligibility (certified on the reconstructable hard-label tasks T1/T3, diagnostic/windowed robustness on T2/T4), so the gap tracks model size more than precision; it replicates on Llama-3.x (12/12 ineligible) and is robust to the anchor choice (a $\tau$-sweep) and to prompt wording (0/112 eligible across the original plus 3 neutral paraphrases per cell). The practical implication: place SLMs behind a baseline that meets the CI-backed threshold, and use the SLM only where the baseline fails to meet the threshold; e.g. a 4B re-ranker over a BM25 shortlist beats BM25 ($+0.047$ [0.020, 0.073], without itself certifying eligibility).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.00025 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:27.837Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).