Metraipolis › archive › dc_d14b2d6180dd
Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs — arXiv:2609.17564
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.17564
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T15:04:18.766Z
- SHA-256 of the archived text
c3bde09af36cb607064e39677054cadf67daf3d1fccd2014b363be42a0f8fce6- Stable URL of this page
https://metraipolis.com/documento/dc_d14b2d6180dd
How to cite
Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs — arXiv:2609.17564 — arXiv (en), https://arxiv.org/abs/2609.17564 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d14b2d6180dd (Metraipolis archive dc_d14b2d6180dd, sha256 c3bde09af36cb607064e39677054cadf67daf3d1fccd2014b363be42a0f8fce6)
BibTeX
@misc{dcd14b2d6180dd,
title = {Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs — arXiv:2609.17564},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.17564},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d14b2d6180dd; sha256 c3bde09af36cb607064e39677054cadf67daf3d1fccd2014b363be42a0f8fce6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.17564. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Deploying retrieval-augmented generation (RAG) on commodity GPUs such as the NVIDIA T4 (16 GB VRAM) exposes a practical failure mode we call the Compression Paradox: neural prompt compression can add key-value (KV) cache contention and preprocessing latency that outweigh generation-time savings, while skipping compression can cause out-of-memory (OOM) failures on long contexts. We identify two distinct failure mechanisms when a vLLM-served LLM and a PyTorch-based compressor are co-deployed under tight memory budgets, and introduce the Tri-Metric Router, a deterministic, training-free policy that selects among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines. The router uses three CPU-side signals: spatial complexity ($L$), syntactic density ($\rho_{key}$), and type-token ratio (TTR). Unlike prior semantic-only adaptation, our dispatch signal is hardware-physical, based on VRAM headroom and a latency crossover point. Thresholds are calibrated from profiling on LongBench qasper, yielding an operating crossover near 4,332 words on T4; our contribution is this calibration methodology rather than a hardware-specific constant. On out-of-distribution holdouts, the method achieves 0% OOM failures, 88.5 $\pm$ 4.4% oracle alignment, and 49.3% Combined F1, improving over always-on lexical compression by 5.2 points without additional VRAM or training cost.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2609.17564 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T14:08:19.044Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).