PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures
Summary
Introduces PQR, a framework that automatically generates diverse and realistic user queries to uncover failures in LLM-based QA agents, achieving 23-78% more unhelpful responses compared to prior methods.
View Cached Full Text
Cached at: 05/19/26, 06:34 AM
# PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures Source: [https://arxiv.org/abs/2605.16551](https://arxiv.org/abs/2605.16551) [View PDF](https://arxiv.org/pdf/2605.16551) > Abstract:Evaluating LLM\-based agents remains challenging because identifying meaningful failure cases often requires substantial human effort to design realistic test scenarios\. Prior works primarily focus on automatically discovering agent failures induced by adversarial users, while overlooking queries with real user intents that also trigger agent failures\. We introduce PQR, a framework that not only surfaces agent failures with respect to specific objectives \(e\.g\., helpfulness, safety, etc\.\) but also resembles real users' intents\. PQR operates through an iterative interaction between two complementary modules\. The query refinement module performs rewrites to explore diverse query variations, while the prompt refinement module uses prior feedback to derive new objective\-violating strategies and realism policies for refining prompts, which in turn generate failure\-triggering yet realistic queries\. We evaluate PQR on detecting an e\-commerce QA agent's unhelpful responses\. Our method uncovers 23% \- 78% more unhelpful responses, and our generated queries are more diverse and realistic compared to previous methods\. ## Submission history From: Yunan Lu \[[view email](https://arxiv.org/show-email/5471fb8c/2605.16551)\] **\[v1\]**Fri, 15 May 2026 18:50:43 UTC \(1,288 KB\)
Similar Articles
LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes
Introduces LakeQuest, a human-validated benchmark of 9,846 QA pairs across three domains for evaluating end-to-end retrieve-and-synthesize pipelines over heterogeneous data lakes, revealing critical failure modes in modern QA systems.
Your AI Agent is one bad prompt away from ruining your brand (And why traditional QA is useless)
The article argues that traditional chatbot QA is broken because it only tests happy paths, and proposes using an AI-powered user simulator that attacks the bot with diverse personas and edge cases to find vulnerabilities before deployment.
ReportQA: QA-Based Radiology Report Evaluation
This paper proposes ReportQA, a QA-based framework for evaluating radiology reports that uses LLMs to answer clinically relevant questions, demonstrating better alignment with radiologist judgments than existing metrics.
Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting
This paper audits knowledge-based VQA benchmarks, revealing systematic violations of assumptions that make accuracy a misleading metric. It introduces a repair protocol and multi-entity augmentation to restore answer derivability and question clarity, showing that corrected settings yield markedly different model rankings.
IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering
IPQA introduces a benchmark for evaluating core intent identification in personalized question answering, addressing a gap in existing metrics that focus on response quality rather than intent understanding. The paper presents a dataset construction methodology grounded in bounded rationality and demonstrates that state-of-the-art language models struggle with identifying user-prioritized intents from answer selection patterns.