Metraipolis › archive › dc_b91112c7faed

JustFit: Just-in-Time State Management for Local LLM Serving — arXiv:2609.17475

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.17475
License (as archived)
CC0-1.0
Archived
2026-10-15T18:32:28.973Z
SHA-256 of the archived text
2a11dd323803657ce4466ba8ce5f6dd9153e6bb0ac0a576344b13e8375a39b7b
Stable URL of this page
https://metraipolis.com/documento/dc_b91112c7faed

How to cite

JustFit: Just-in-Time State Management for Local LLM Serving — arXiv:2609.17475 — arXiv (en), https://arxiv.org/abs/2609.17475 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b91112c7faed (Metraipolis archive dc_b91112c7faed, sha256 2a11dd323803657ce4466ba8ce5f6dd9153e6bb0ac0a576344b13e8375a39b7b)
BibTeX
@misc{dcb91112c7faed,
  title = {JustFit: Just-in-Time State Management for Local LLM Serving — arXiv:2609.17475},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.17475},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b91112c7faed; sha256 2a11dd323803657ce4466ba8ce5f6dd9153e6bb0ac0a576344b13e8375a39b7b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.17475. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Local agents need memory for model execution and working history. We present JustFit, an MLX runtime that coordinates their overlapping allocations: KVExec executes and checkpoints four-bit KV with bounded workspace, PhaseSwap loads phase-dependent components, and StateTrans preserves history across execution modes. With Qwen3.8-27B MXFP4 on a 24 GiB M4 Pro MacBook, JustFit reaches 327,680 retained positions across two requests, 10.67 times the evaluated baseline's 30,720-position single-request record. One request completes the full 262,144-position native window at median 5.986 tokens/s. Each shape completes 16,384 outputs per request in three fresh processes: B1 uses one cold build and two prefix extensions; B2 uses three ordered prefix extensions (Section 4). Image encoding can proceed while preserving a live 196,608-input text request. A controlled, repetitive 32K+6K workload reaches median 18.284 tokens/s at 15,626 MiB; a separate AIME 2026 evaluation scores 29/30. Coordinating execution and state lifetimes makes longer histories feasible on personal hardware.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-16 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2609.17475 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T18:03:09.539Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-16?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).