Metraipolis › archive › dc_9dedeb95777a

Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models — arXiv:2610.02142

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.02142
License (as archived)
CC0-1.0
Archived
2026-10-06T13:56:22.895Z
SHA-256 of the archived text
df9c5f95d19841e954cfb551217e139c7f7b68fe4deb1378c42884979847582e
Stable URL of this page
https://metraipolis.com/documento/dc_9dedeb95777a

How to cite

Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models — arXiv:2610.02142 — arXiv (en), https://arxiv.org/abs/2610.02142 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9dedeb95777a (Metraipolis archive dc_9dedeb95777a, sha256 df9c5f95d19841e954cfb551217e139c7f7b68fe4deb1378c42884979847582e)
BibTeX
@misc{dc9dedeb95777a,
  title = {Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models — arXiv:2610.02142},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.02142},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9dedeb95777a; sha256 df9c5f95d19841e954cfb551217e139c7f7b68fe4deb1378c42884979847582e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.02142. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A 661.6M parameter model (approx. 65% code/technical text; no dedicated SFT) and a 1,109M model (web-heavy multi-phase curriculum; 6B-token tool-SFT) share decoder, tokenizer, and special tokens, scoring almost identically on lenient tool-use metrics (B4: 0.660 vs. 0.650). Verbatim-reproduction checks on training examples separate them completely: the 600M emits valid tool calls with generalized arguments on 6/6 examples; the 1B does so on 0/6 across checkpoints. A first-token probe localizes the 1B's failure to a missing prior (prob. $10^{-4}$--$10^{-5}$ on <|tool_call|>), which was erased by its web-heavy training phase. A targeted SFT recipe (diverse corpus, 5x higher learning rate, 2,202 steps, ~3.3 GPU-hours) repairs the 1B using three orders of magnitude fewer tokens than the failed phase. On all 269 corpus rows, valid emission rises from 0.100 to 0.959 (600M: 0.926). On 238 unseen prompts, the repaired 1B passes 0.536 vs. the 600M's 0.428 ($p = 0.004$). Embedding-drift checks show the repair did not move the trigger token's tied embedding (97.7% of the bf16 table remains bit-identical), meaning changes live in the surrounding network. Both models over-trigger, rarely answering negative prompts without a call (0.09 for 600M, 0.17 for repaired 1B). Factorial analyses confirm all repair configurations install the format, though suppression benefits from a diverse corpus remain a hypothesis due to seed sensitivity. This cheap diagnostic ladder costs minutes of CPU time and should gate tool-use claims on small models.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.02142 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.765Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).