Metraipolis › archive › dc_97cd255dd327

Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits — arXiv:2609.38386

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.38386
License (as archived): CC0-1.0
Archived: 2026-10-07T12:41:47.151Z
SHA-256 of the archived text: 01abd485102bff05fee17d8315d24e0e90cfca9cde75a984aeccaa30c98f4a1d
Stable URL of this page: https://metraipolis.com/documento/dc_97cd255dd327

How to cite

Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits — arXiv:2609.38386 — arXiv (en), https://arxiv.org/abs/2609.38386 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_97cd255dd327 (Metraipolis archive dc_97cd255dd327, sha256 01abd485102bff05fee17d8315d24e0e90cfca9cde75a984aeccaa30c98f4a1d)

BibTeX

@misc{dc97cd255dd327,
  title = {Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits — arXiv:2609.38386},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.38386},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_97cd255dd327; sha256 01abd485102bff05fee17d8315d24e0e90cfca9cde75a984aeccaa30c98f4a1d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38386. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Concurrent autoregressive inference creates a fundamental interference problem: prefilling a newly arrived long prompt can delay tokens for requests that are already decoding. Fixed prefill chunks reduce this interference, but the best chunk size depends on the model, hardware, load, and latency objective. We introduce Decode-Latency Feedback Prefill (DLFP), a model-free controller that changes only prefill work that overlaps active decodes. After a guarded scheduling cycle, DLFP uses the observed interval as proportional feedback to resize the next prefill chunk; isolated prefills remain unrestricted. We implement DLFP in vLLM and evaluate it with open-loop Poisson arrivals, exact token accounting, raw request traces, and NVIDIA telemetry. On Qwen3-0.6B in BF16 on one A100 80 GB GPU, three paired 100-request trials reduce P99 inter-token latency by 24.8%, 30.1%, and 28.2% (mean 27.7%, paired 95% confidence interval 21.0% to 34.3%) with exact output agreement, no failures, and unchanged SLO compliance. The benefit is not free: mean P99 time to first token increases 34.8% while remaining inside the declared SLO. Crucially, the mechanism does not generalize to Qwen3-8B, Qwen3-32B, or a two-GPU tensor-parallel configuration. We trace the failure to an asynchronous scheduler-call interval that is only a proxy for completed GPU iteration time. This negative result defines the boundary of the contribution and motivates a completion-timed controller for concurrent CPU and on-device inference. We do not claim mobile-device performance; the present work is a reproducible proof-of-concept and generalization study.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.38386 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:13.619Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).