Metraipolis › archive › dc_a7f25d04478e

TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge — arXiv:2611.01174

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01174
License (as archived)
CC0-1.0
Archived
2026-11-04T12:33:38.728Z
SHA-256 of the archived text
e0b620abba5483631c502a2e14a515c7d3dbe7ff670699c933e304a5d8dcb529
Stable URL of this page
https://metraipolis.com/documento/dc_a7f25d04478e

How to cite

TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge — arXiv:2611.01174 — arXiv (en), https://arxiv.org/abs/2611.01174 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a7f25d04478e (Metraipolis archive dc_a7f25d04478e, sha256 e0b620abba5483631c502a2e14a515c7d3dbe7ff670699c933e304a5d8dcb529)
BibTeX
@misc{dca7f25d04478e,
  title = {TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge — arXiv:2611.01174},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01174},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a7f25d04478e; sha256 e0b620abba5483631c502a2e14a515c7d3dbe7ff670699c933e304a5d8dcb529},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01174. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Two-tier LLM serving escalates selected requests from a small model to a large one, but prompt-based routers cannot see whether retrieved evidence supports an answer, and they treat the GPU cost of an escalation as constant. We study TRIAGE, which conditions escalation on a post-generation reliability state and on a load-dependent GPU price, and we report both what held and what did not on one A40 serving Qwen2.5-7B and 32B (AWQ) under VLLM. Escalation cost rose 21.9x from 1 to 10 requests/s but showed no monotone dependence on KV-cache headroom, and a pre-registered claim that the reliability state detects corrupted evidence (AUROC>=0.70) was falsified at 0.571, although any prompt-only router scores exactly 0.500 by construction. Post hoc, a semantic evidence judge raised detection to 0.846 (0.926 on never-served items) and predicted wrong small-model answers at AUROC 0.90-0.96 on RGB and three RouterBench model pairs; the resulting cascade stayed within about 3 pp of GPT-4 at 79.5% lower API cost for GPT-3.5->GPT-4, yet fell below random routing for open-model pairs with 2-3x price ratios. End to end, the 7B->32B pair showed only a +2.88 pp held-out gap, so the allocator rightly declined to escalate and the headline is a null that a small pre-launch gate failed to expose.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2611.01174 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:27:02.062Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).