Metraipolis › archive › dc_38ab58fe15fe

Latency-Aware Adaptive Best-of-$N$ — arXiv:2611.07453

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.07453
License (as archived)
CC0-1.0
Archived
2026-11-06T14:43:05.341Z
SHA-256 of the archived text
21d7bd750b1d65f5f58b5ba17561405cb55e3e2a859a3e620f8687c535408b64
Stable URL of this page
https://metraipolis.com/documento/dc_38ab58fe15fe

How to cite

Latency-Aware Adaptive Best-of-$N$ — arXiv:2611.07453 — arXiv (en), https://arxiv.org/abs/2611.07453 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_38ab58fe15fe (Metraipolis archive dc_38ab58fe15fe, sha256 21d7bd750b1d65f5f58b5ba17561405cb55e3e2a859a3e620f8687c535408b64)
BibTeX
@misc{dc38ab58fe15fe,
  title = {Latency-Aware Adaptive Best-of-$N$ — arXiv:2611.07453},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07453},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_38ab58fe15fe; sha256 21d7bd750b1d65f5f58b5ba17561405cb55e3e2a859a3e620f8687c535408b64},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07453. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Test-time compute (TTC) improves language model responses by spending additional computation at inference time, and the right amount of compute depends on the prompt. Adaptive TTC methods allocate compute based on signals of response quality such as observed rewards, often by stopping once the signal exceeds a fixed threshold, and generally measure cost by the number of generations or tokens. But in online serving, latency is also a cost: it grows with server load, so a method using a fixed threshold spends too little compute when generation is fast and too much when it is slow. For Best-of-$N$ sampling, a widely used TTC method, we propose a two-stage adaptive algorithm that draws an initial batch, then uses the observed rewards and the batch's latency to decide whether to draw a second batch and at what size by comparing expected improvement against predicted cost: it draws one when the first batch's best reward is below a threshold that decreases with the first batch's latency, without explicitly measuring server load. We map each reward to its value under the cumulative distribution function (CDF) of the prompt's reward distribution, so the same tradeoff parameters can be reused across tasks; a regression fitted offline on other prompts predicts a distribution for the CDF value of the first batch's best reward, giving the expected improvement in closed form. We evaluate the method by simulation on a preference-based alignment benchmark, measuring generations-equivalent cost: generations plus latency, converted to generations. The method reaches Best-of-$N$'s reward at lower generations-equivalent cost, and conditioning on the measured first-batch latency saves at least as much of that cost as fixed-threshold (latency-unaware) baselines, and more when latency is highly variable, when it grows with batch size, and when the second batch's latency is highly predictable from the first's.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.07453 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:57.584Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).