Metraipolis › archive › dc_723d6ea2590f

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs — arXiv:2608.16391

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.16391
License (as archived): CC0-1.0
Archived: 2026-10-18T18:07:14.655Z
SHA-256 of the archived text: 33ade2042a67f9fc893f439f184ab8698beeb6d8393d8c5bf7d7413baad68ae7
Stable URL of this page: https://metraipolis.com/documento/dc_723d6ea2590f

How to cite

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs — arXiv:2608.16391 — arXiv (en), https://arxiv.org/abs/2608.16391 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_723d6ea2590f (Metraipolis archive dc_723d6ea2590f, sha256 33ade2042a67f9fc893f439f184ab8698beeb6d8393d8c5bf7d7413baad68ae7)

BibTeX

@misc{dc723d6ea2590f,
  title = {Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs — arXiv:2608.16391},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.16391},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_723d6ea2590f; sha256 33ade2042a67f9fc893f439f184ab8698beeb6d8393d8c5bf7d7413baad68ae7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.16391. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{\textbf{Ventor-QTest}}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports \emph{average fidelity loss} (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report \emph{extreme fidelity loss} (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.AI).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.16391 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T16:29:14.115Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).