Metraipolis › archive › dc_b367f1b72cdf
JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs — arXiv:2610.21412
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.21412
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T09:03:21.368Z
- SHA-256 of the archived text
7f19004fca3bafa1220c33a77e94ea5b0ebd6201906eb064ac84b5e04a73fe10- Stable URL of this page
https://metraipolis.com/documento/dc_b367f1b72cdf
How to cite
JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs — arXiv:2610.21412 — arXiv (en), https://arxiv.org/abs/2610.21412 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b367f1b72cdf (Metraipolis archive dc_b367f1b72cdf, sha256 7f19004fca3bafa1220c33a77e94ea5b0ebd6201906eb064ac84b5e04a73fe10)
BibTeX
@misc{dcb367f1b72cdf,
title = {JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs — arXiv:2610.21412},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.21412},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b367f1b72cdf; sha256 7f19004fca3bafa1220c33a77e94ea5b0ebd6201906eb064ac84b5e04a73fe10},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21412. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.21412 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T08:44:17.741Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).