Metraipolis › archive › dc_b91112c7faed
JustFit: Just-in-Time State Management for Local LLM Serving — arXiv:2609.17475
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.17475
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T18:32:28.973Z
- SHA-256 of the archived text
2a11dd323803657ce4466ba8ce5f6dd9153e6bb0ac0a576344b13e8375a39b7b- Stable URL of this page
https://metraipolis.com/documento/dc_b91112c7faed
How to cite
JustFit: Just-in-Time State Management for Local LLM Serving — arXiv:2609.17475 — arXiv (en), https://arxiv.org/abs/2609.17475 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b91112c7faed (Metraipolis archive dc_b91112c7faed, sha256 2a11dd323803657ce4466ba8ce5f6dd9153e6bb0ac0a576344b13e8375a39b7b)
BibTeX
@misc{dcb91112c7faed,
title = {JustFit: Just-in-Time State Management for Local LLM Serving — arXiv:2609.17475},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.17475},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b91112c7faed; sha256 2a11dd323803657ce4466ba8ce5f6dd9153e6bb0ac0a576344b13e8375a39b7b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.17475. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Local agents need memory for model execution and working history. We present JustFit, an MLX runtime that coordinates their overlapping allocations: KVExec executes and checkpoints four-bit KV with bounded workspace, PhaseSwap loads phase-dependent components, and StateTrans preserves history across execution modes. With Qwen3.8-27B MXFP4 on a 24 GiB M4 Pro MacBook, JustFit reaches 327,680 retained positions across two requests, 10.67 times the evaluated baseline's 30,720-position single-request record. One request completes the full 262,144-position native window at median 5.986 tokens/s. Each shape completes 16,384 outputs per request in three fresh processes: B1 uses one cold build and two prefix extensions; B2 uses three ordered prefix extensions (Section 4). Image encoding can proceed while preserving a live 196,608-input text request. A controlled, repetitive 32K+6K workload reaches median 18.284 tokens/s at 15,626 MiB; a separate AIME 2026 evaluation scores 29/30. Coordinating execution and state lifetimes makes longer histories feasible on personal hardware. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2609.17475 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T18:03:09.539Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).