Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System
Summary
This paper presents a deployment-centered evaluation of an LLM system integrated in electronic health records, training a classifier to predict query-level rejection risk using pre-response context like provider type and department, achieving an AUROC of 0.719 over 4.5 months of feedback.
View Cached Full Text
Cached at: 06/12/26, 08:53 AM
# Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System Source: [https://arxiv.org/abs/2606.12702](https://arxiv.org/abs/2606.12702) [View PDF](https://arxiv.org/pdf/2606.12702) > Abstract:Large language models \(LLMs\) are increasingly integrated into clinical systems, making it essential to evaluate the real\-world utility of these systems\. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets \-\- leading to major blind spots for evaluating clinical systems\. In this work, we perform a deployment\-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions\. Specifically, we train a pre\-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment\-specific context available before generation\. We conduct a prospective analysis of our model over 4\.5 months of user feedback, finding that our prediction model achieves an AUROC of 0\.719\. Further, we estimate the benefit of such predictions in two downstream use cases \(guardrail triggering and abstention\)\. Our key conceptual insight is that making use of deployment\-specific context \(i\.e\., the provider type, department name, language model used for response\), as opposed to only query content, improves the ability to predict whether the user will reject the system output\. Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment\-specific context, opening the door to targeted guardrails\. ## Submission history From: Alyssa Unell \[[view email](https://arxiv.org/show-email/809c42c6/2606.12702)\] **\[v1\]**Wed, 10 Jun 2026 21:44:20 UTC \(640 KB\)
Similar Articles
EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs
EHRBench is an automated and reliable benchmark for evaluating LLMs on clinical decision-making tasks using real-world electronic health records, covering nearly 1M QA items across diagnosis, treatment, and prognosis tasks.
A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models
This paper presents a multi-domain red teaming framework for evaluating safety, robustness, and fairness of medical LLMs across 690 clinically grounded scenarios. Results show that high aggregate accuracy can mask critical failures, and hybrid evaluation with clinician oversight is necessary for credible safety assessment.
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
This paper introduces a margin-based confidence ranking method for LLM-as-a-judge systems, learning a dedicated estimator to ensure monotonicity between confidence and human-disagreement risk, with generalization guarantees and improved ranking accuracy across datasets.
ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV
This paper introduces ClinicalBench and the EpiKG system, evaluating assertion-aware retrieval for clinical question answering on MIMIC-IV data across multiple LLMs. It demonstrates that handling negation and temporality in retrieval significantly improves performance over standard baselines.
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
This paper introduces PRECISE, an extension of Prediction-Powered Inference that combines a small set of human labels with a large set of LLM judgments to produce unbiased and variance-reduced estimates of ranking evaluation metrics like Precision@K. The method is validated on the ESCI benchmark and in a production A/B test, where it correctly identified the best system variant using only 100 human labels, confirmed by a +407 bps sales improvement.