Metraipolis › archive › dc_eaa77f0479f3

Rewards for Budget-Aware Multi-Turn Agents That Adapt to Arbitrary Test-Time Constraints — arXiv:2610.28409

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.28409
License (as archived)
CC0-1.0
Archived
2026-10-21T07:08:29.104Z
SHA-256 of the archived text
2a51b546efbbf03c07b3285e26d417757ad1ce58681c54b4168a76d38eb3e8e0
Stable URL of this page
https://metraipolis.com/documento/dc_eaa77f0479f3

How to cite

Rewards for Budget-Aware Multi-Turn Agents That Adapt to Arbitrary Test-Time Constraints — arXiv:2610.28409 — arXiv (en), https://arxiv.org/abs/2610.28409 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eaa77f0479f3 (Metraipolis archive dc_eaa77f0479f3, sha256 2a51b546efbbf03c07b3285e26d417757ad1ce58681c54b4168a76d38eb3e8e0)
BibTeX
@misc{dceaa77f0479f3,
  title = {Rewards for Budget-Aware Multi-Turn Agents That Adapt to Arbitrary Test-Time Constraints — arXiv:2610.28409},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.28409},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eaa77f0479f3; sha256 2a51b546efbbf03c07b3285e26d417757ad1ce58681c54b4168a76d38eb3e8e0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.28409. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language model agents are often given an inference budget and asked to maximise success within it. A multi-turn agent has to allocate this budget across many environment interactions. An external harness can enforce the budget, but it remains unclear whether the agent itself can learn when and how much to spend. We ask whether agents can learn this allocation across dynamically specified budgets and which rewards best teach this behaviour. We adapt existing length penalties to budget constraints spanning multiple resource types, introduce new reward formulations, and compare eleven objectives in a shared training pipeline. We train Qwen3.5 models at 4B and 9B on AppWorld with budgets over tokens, API calls, and environment steps, and on the BabyAI gridworld with turn budgets. We train a separate policy for each environment and resource type, with each policy trained across budget values. Training with rewards that penalise overspending substantially improves within-budget success and natural compliance over untrained models and over training on task success alone, with agents learning to adapt their resource use to stated budgets and task difficulty. These gains extend to unseen budgets and, zero-shot, to a shifted task distribution and a new benchmark. Reward objectives trade off success within budget and natural compliance. A sigmoid penalty for overspending most often achieves the highest within-budget success at an individual budget, while a fixed negative reward for overspending generally yields the strongest natural compliance. Combining penalties for overspend frequency and magnitude gives the smallest average gap to the best method at each budget. Overall, we show that a single agent can learn to allocate resources across turns and adapt its behaviour to user-specified budgets using budget feedback without external budget-allocation controllers.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.28409 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:43.650Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).