Metraipolis › archive › dc_e5002b84a692
BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents — arXiv:2609.13149
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.13149
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T23:16:28.977Z
- SHA-256 of the archived text
3f7f6a548464524cd20732af1aeb7ed8f959b9a704fcf73babf2fdc555a519d4- Stable URL of this page
https://metraipolis.com/documento/dc_e5002b84a692
How to cite
BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents — arXiv:2609.13149 — arXiv (en), https://arxiv.org/abs/2609.13149 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e5002b84a692 (Metraipolis archive dc_e5002b84a692, sha256 3f7f6a548464524cd20732af1aeb7ed8f959b9a704fcf73babf2fdc555a519d4)
BibTeX
@misc{dce5002b84a692,
title = {BudgetBench: A Budget-Tiered Protocol and Pilot Harness for Memory Strategy Evaluation in Local Large Language Model Agents — arXiv:2609.13149},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.13149},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e5002b84a692; sha256 3f7f6a548464524cd20732af1aeb7ed8f959b9a704fcf73babf2fdc555a519d4},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.13149. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
For local large language model agents, active context is a scarce resource: memory capacity, prefill latency, cache growth, and service objectives all constrain how many input tokens each call can afford. We present BudgetBench, an active-budget protocol and reference harness that treats the per-call input-token budget as the independent variable when comparing memory strategies. Holding the model, task, sampler, and decoding fixed, it sweeps budgets over 2K, 4K, 8K, 16K, and 32K tokens and records quality, budget utilization, latency, and, as a first-class outcome, budget-violation rates. The core contribution is this reusable measurement surface: a swappable MemoryStrategy contract, explicit budget enforcement, deterministic or versioned graders, prompt-audit metadata, and reproducibility artifacts, released at this https URL . We substantiate the protocol with pilot studies rather than final rankings. Across a local qwen2.5:1.5b pilot (89 items each on SWE-bench Verified and LongBench v2), a hosted 50-item Qwen3 30B-A3B LongBench replication with exact tokenization, and a 500-item LongMemEval oracle study scored by the official GPT-4o evaluator, the harness exposes budget-compliance failures, non-monotonic quality curves, and operating points that single-budget evaluation hides. The budgeted-versus-full-context direction remains unresolved: the local slice is near-null and the hosted replication favors full context in point estimate. We report results transparently, including that the early pilot's tokenizer approximation undercounts some served-model prompts, so its violation rows are tokenizer-approximation diagnostics, not claim-bearing results; all timings are operational diagnostics. The reusable contribution is the protocol, harness, and failure-reporting discipline needed to scale fixed-budget memory-strategy evaluation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.13149 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:51:09.976Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).