Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
Summary
Ventor-QTest proposes a black-box audit method for vendor-hosted LLM APIs, using repeated and long-sequence probes to measure fidelity loss and detect degradation in long-horizon agentic tasks.
View Cached Full Text
Cached at: 08/18/26, 03:51 AM
Paper page - Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
Source: https://huggingface.co/papers/2608.16391 Published on Aug 17
·
Submitted byhttps://huggingface.co/xiangfanwu
wuon Aug 18
Abstract
Ventor-QTest audits hosted open-weight model APIs via repeated and long-sequence black-box probes, measuring average and extreme fidelity loss to detect degradation in long-horizon agentic performance.
As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \textbf{Ventor-QTest}, a compositeblack-box auditthat requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs acategorical output distributionfrom the returned text counts, and reportsaverage fidelity loss(AFL) as a null-bias-corrected, within-window meancoarsened-KLstatistic. Its long-sequence component uses independent runs to reportextreme fidelity loss(EFL) through the empirical upper tail of a run-levelreference-centered-surprisalstatistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derivedcoarsened-KLcomparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association withGPQA-Diamondaccuracy. In contrast, pronounced EFL coincides with a decline inTerminal-Benchpass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive toextreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditinglong-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.16391
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.16391 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.16391 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.16391 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
This paper presents Vera, an end-to-end automated safety testing framework for LLM agents that combines literature-driven risk discovery, combinatorial composition of safety cases, and evidence-grounded verification. Evaluations on four agent frameworks reveal substantial safety weaknesses, with average attack success rates reaching 93.9% under multi-channel attacks, and the release of Vera-Bench with 1600 executable safety cases.
LLM-as-a-Verifier: A General-Purpose Verification Framework
LLM-as-a-Verifier introduces a probabilistic verification framework that computes continuous scores from LLM logits, scaling across granularity, repeated evaluation, and criteria decomposition. It achieves state-of-the-art results on multiple agentic benchmarks and provides dense feedback for RL.
Building independent LLM drift detection - sharing the methodology, looking for feedback on the approach
The author shares a methodology for building an external LLM drift detection system that continuously probes model behavior (schema adherence, instruction-following, refusal rates, etc.) to catch silent degradations in API performance, and invites feedback on the approach, pricing, and use cases.
Faithful uncertainty in LLM agents: calibration vs utility tradeoff in practice[D]
A practitioner discusses the calibration vs. utility tradeoff in LLM agents, sharing experience with a verifier-based pipeline that reduces hallucinated tool calls by ~60% but introduces latency costs and drops easy correct answers.
The Verifier Tax: Horizon-Dependent Safety–Success Tradeoffs in Tool-Using LLM Agents [R]
This paper presents a safety evaluation framework for tool-using LLM agents, introducing the concept of the 'Verifier Tax'—a horizon-dependent tradeoff between safety and task completion. It proposes a two-tier verification architecture and uses Tau-bench scenarios to demonstrate how verification can reduce unsafe successes but also decrease task completion as task horizon increases.