Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistants

# Adaptive Discovery of Grounding Failures in Meeting Assistants
Source: [https://arxiv.org/html/2608.20392](https://arxiv.org/html/2608.20392)
###### Abstract

LLM\-powered meeting assistants are deployed at scale, yet systematic evaluation of their grounding fidelity remains limited to static benchmarks that miss failure modes tied to specific discourse structures or reasoning demands\. We propose Evaluation\-as\-Search \(EaS\), a feedback\-driven methodology that frames quality evaluation as an adaptive search over the space of natural questions a meeting participant might ask\. Rather than sampling uniformly, EaS learns from evaluator feedback across iterations to concentrate probing effort on cognitive demands where failures are most likely, guided by a UCB\-scored coverage map and blind multi\-dimensional quality evaluation\. Using EaS, we constructMeetingProbe, a benchmark of over 3,000 annotated question–answer pairs spanning 20 transcripts from three meeting genres and three LLM assistants\. In ablations, adaptive search surfaces2\.5×2\.5\\timesmore failures than random probing \(7\.1%7\.1\\%vs\.2\.9%2\.9\\%finding rate\), with the strategic planner contributing the largest individual effect\. Across three models, we observe a clear capability gradient and identify eight recurring failure categories dominated by discourse\-pragmatic challenges rather than factual recall errors\. We further validateMeetingProbeacross multiple model families and providers, finding a clean capability gradient and a curated subset of universal failures that no model handles\.MeetingProbeis released publicly to support reproducible evaluation of meeting assistant grounding fidelity\.

## 1Introduction

LLM\-powered meeting assistants, including Microsoft Copilot, Google Gemini for Workspace, and Zoom AI Companion, now serve millions of enterprise meetings daily, answering natural\-language questions about action items, decisions, and discussed topics that organizations increasingly treat as authoritative records\. Factual accuracy in this setting is an operational requirement: a response that misattributes a commitment or fabricates a consensus can propagate errors through organizational decisions and downstream workflows\. Yet the prevailing evaluation paradigm relies on static benchmarks: fixed question sets that characterize*average\-case*performance but remain blind to systematic failure modes tied to specific reasoning demands or discourse structures\. A meeting assistant could score well on a hundred standard questions while still failing when asked about multi\-speaker disagreements or action items scattered across a long transcript; no static benchmark surfaces this, because its item pool is not designed to concentrate sampling effort in those regions\.

Two strands of prior work address complementary aspects of this problem but leave a critical gap\. Adaptive evaluation approaches, including LLM\-guided iterative refinement\(Chaoet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib6); Mehrotraet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib7)\), quality\-diversity search\(Samvelyanet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib21)\), and behavioral testing\(Ribeiroet al\.,[2020](https://arxiv.org/html/2608.20392#bib.bib29); Ribeiro and Lundberg,[2022](https://arxiv.org/html/2608.20392#bib.bib30)\), demonstrate that adaptive sampling systematically exposes model behaviors that static evaluation misses, but target safety alignment or broad NLP capability rather than multi\-dimensional quality fidelity in grounded discourse\. Meeting QA and faithfulness benchmarks, including QMSum\(Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\), MeetingBank\(Huet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib12)\), ELITR\-Bench\(Thonetet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib16)\), and RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib18)\), directly measure grounding fidelity on meeting content but evaluate on fixed question sets and do not adapt probing strategy based on observed failures\. No existing method*adaptively searches*for the conditions under which a particular meeting assistant produces unfaithful or low\-quality responses across multiple quality dimensions simultaneously, and no existing benchmark provides a curated, multi\-model collection of identified grounding failures in meeting QA\.

We address both gaps\. First, we frame quality evaluation as a feedback\-driven adaptive search over natural questions, which we call*Evaluation\-as\-Search*\(EaS\): rather than sampling uniformly, EaS learns from evaluator feedback across iterations and allocates evaluation budget strategically, concentrating effort on regions where quality failures are most likely\. Second, we use EaS to constructMeetingProbe, a benchmark of 3,009 annotated question–answer pairs comprising 1,047 identified grounding failures and 1,962 hard negatives \(probes where the assistant responds correctly despite targeting a demanding cognitive operation\), drawn from 20 transcripts across three genres \(product design, academic research, parliamentary proceedings\) and three LLM assistants of varying capability\.

Both contributions hinge on a reliable evaluator: the feedback loop is only as informative as the signal it conditions on, and MeetingProbe’s labels are only as trustworthy as the rubric behind them\. Yet evaluating meeting QA answers is harder than it looks: off\-the\-shelf LLM\-as\-a\-Judge frameworks\(Zhenget al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib8); Eset al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib18)\)work well for conventional RAG pipelines but miss meeting\-specific failure modes such as speaker misattribution, temporal conflation across agenda items, and selective omission of dissenting views, and no existing judge benchmark is purpose\-built for meeting QA\. We therefore buildMARC\(Meeting Answer Rating & Calibration\), an LLM evaluator with rubrics designed for meeting QA, andQMSumCal50, a calibration dataset of 50 stratified QMSum queries augmented with golden answers, controlled perturbations targeting six meeting\-specific failure types, and human\-validated quality scores; both are released publicly\.

Our contributions are fourfold\. \(1\) the*Evaluation\-as\-Search*methodology, which combines UCB\-guided coverage over cognitive demands, three typed search operators \(Exploration,Refinement,Mutation\), and blind multi\-dimensional evaluation, and achieves2\.5×2\.5\\timesthe finding rate of random probing under a pairedtt\-test across transcripts\. \(2\) theMeetingProbebenchmark of 3,009 annotated entries across 20 QMSum transcripts\(Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\), three meeting genres, and three LLM assistants, with MARC scores on faithfulness and completeness and failure\-category labels, released publicly for reproducible evaluation\. \(3\) a failure taxonomy of eight grounding\-error categories in which overconfident paraphrase, unjustified quantification, and speaker misattribution account for over 60% of failures, with distinct per\-model profiles\. \(4\) a cross\-model and cross\-family analysis revealing a clear capability gradient \(5\.2%, 10\.1%, 15\.3% finding rates for GPT\-5\.2\-chat, GPT\-4\.1, GPT\-4\.1\-mini;p<0\.0001p<0\.0001\), strongly asymmetric transfer between models, and a curated subset of universal failures shared across multiple model families and providers\.

## 2Related Work

#### Meeting Summarization and QA Benchmarks\.

QMSum\(Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\)provides query\-based meeting summarization over AMI, ICSI, and parliamentary transcripts; we use QMSum’s transcript collection as the source forMeetingProbe\. MeetingBank\(Huet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib12)\)aggregates city council meeting transcripts with summaries, and ELITR\-Bench\(Thonetet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib16)\)provides a long\-context meeting\-assistant QA benchmark over real meeting transcripts with human\-rated quality scores; we extend this evaluation paradigm with adaptively generated probes and a calibrated multi\-dimensional MARC evaluator\. MeetingQA\(Prasadet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib13)\)and MeeQA\(Apelet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib15)\)evaluate meeting comprehension with fixed question sets\. DialSim\(Kimet al\.,[2024a](https://arxiv.org/html/2608.20392#bib.bib22)\)simulates real\-time multi\-party dialogue to evaluate long\-context understanding, demonstrating that discourse structure affects comprehension in ways static benchmarks miss\. MESA\(Kirsteinet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib14)\)uses a multi\-LLM panel to evaluate meeting summary quality across multiple dimensions, demonstrating that multi\-evaluator agreement improves assessment reliability, a finding that informs our multi\-dimensional MARC design\. All of these benchmarks use static question pools;MeetingProbecomplements them by providing adaptively discovered failure cases with multi\-dimensional quality annotations\.

#### LLM Evaluation and Faithfulness Assessment\.

HELM\(Lianget al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib1)\)establishes holistic multi\-metric evaluation as a principled framework for model comparison\. FActScore\(Minet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib17)\)establishes the decompose\-then\-verify paradigm for factual precision; RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib18)\)provides a suite of RAG evaluation metrics including faithfulness\. Chain\-of\-Verification\(Dhuliawalaet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib19)\)shows that structured self\-verification reduces hallucination, motivating the multi\-stage verification in our pipeline\. SummEdits\(Labanet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib2)\)measures factual reasoning through fine\-grained consistency judgments in summarization, and BAMBOO\(Donget al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib3)\)benchmarks long\-context faithfulness across multiple text\-modeling tasks; both evaluate on fixed inputs, whereas our approach generates evaluation questions adaptively\. G\-Eval\(Liuet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib32)\)and UniEval\(Zhonget al\.,[2022](https://arxiv.org/html/2608.20392#bib.bib31)\)demonstrate that multi\-dimensional quality can be reliably measured by LLMs\. ARES\(Saad\-Falconet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib33)\)uses prediction\-powered inference for confidence intervals in RAG evaluation\.Zhenget al\.\([2023](https://arxiv.org/html/2608.20392#bib.bib8)\)characterize position bias and self\-enhancement bias in LLM\-as\-judge setups, andPanicksseryet al\.\([2024](https://arxiv.org/html/2608.20392#bib.bib34)\)establish a causal link between self\-recognition and self\-preference in LLM evaluators, underscoring the importance of the information barriers that our architecture enforces\. These works provide measurement tools that we deploy within our search loop, but none adaptively generates evaluation questions\.

#### Dynamic Evaluation and Adaptive Testing\.

Static benchmarks eventually saturate as direct training\-benchmark contamination distorts model comparisons across popular evaluation suites\(Sainzet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib23)\)\. DynaBench\(Kielaet al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib24)\)introduces model\-in\-the\-loop dataset creation, and Adversarial NLI\(Nieet al\.,[2020](https://arxiv.org/html/2608.20392#bib.bib25)\)applies human\-and\-model\-in\-the\-loop procedures\.Liet al\.\([2025a](https://arxiv.org/html/2608.20392#bib.bib26)\)use item\-response theory to reduce required evaluation items by roughly 90% while preserving accuracy, andLiet al\.\([2025b](https://arxiv.org/html/2608.20392#bib.bib27)\)learn an RL policy for minimal evaluation subset selection\. Our work extends this paradigm by adaptively*generating*new evaluation questions targeted at specific cognitive demands, rather than selecting from a fixed pool\.

#### Feedback\-Based Optimization of Generative AI Systems\.

A growing line of work treats natural\-language feedback as the optimization signal for generative systems\. TextGrad\(Yuksekgonulet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib40)\)backpropagates language\-model critiques through multi\-agent pipelines as textual “gradients” to improve downstream tasks; Feedback Descent\(Leeet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib41)\)performs open\-ended text optimization by iteratively refining candidates from pairwise comparisons; and AlphaEvolve\(Novikovet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib42)\)couples a code\-generation agent with evaluator feedback to discover improved algorithms\. These systems share a common pattern: a generator proposes candidates, an evaluator returns structured feedback, and the generator’s next proposals are conditioned on accumulated outcomes\. Our Planner instantiates this pattern for evaluation\-question generation: it learns from the coverage map \(per\-demand test counts, findings, near\-misses, and reflections\) and shifts allocation toward productive cognitive demands across iterations, without any model\-weight updates\. EaS differs in target and output: rather than optimizing a generated artifact, the feedback loop drives the*discovery*of natural questions that expose grounding failures, producing a curated benchmark as its output\.

#### Automated Search\-Based Evaluation\.

Automated probing exhibits a fundamental diversity\-effectiveness tradeoff\(Perezet al\.,[2022](https://arxiv.org/html/2608.20392#bib.bib5)\)\. Safety\-oriented methods including PAIR\(Chaoet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib6)\), TAP\(Mehrotraet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib7)\), and Rainbow Teaming\(Samvelyanet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib21)\)address this tradeoff through iterative refinement, tree search, and quality\-diversity algorithms, but optimize a single scalar safety objective\. CheckList\(Ribeiroet al\.,[2020](https://arxiv.org/html/2608.20392#bib.bib29)\)and AdaTest\(Ribeiro and Lundberg,[2022](https://arxiv.org/html/2608.20392#bib.bib30)\)probe capability dimensions through template\-based or perturbation\-based items\. Our approach differs in three respects: we optimize a multi\-dimensional quality vector rather than a scalar safety metric, our questions are natural rather than adversarial, and our output is a curated benchmark rather than a set of jailbreaks\. Our coverage mechanism draws on the MAP\-Elites principle\(Mouret and Clune,[2015](https://arxiv.org/html/2608.20392#bib.bib28)\)of maintaining solutions indexed by behavioral descriptors, but replaces the fixed grid with an open\-ended cognitive demand scoreboard using UCB exploration scores\(Aueret al\.,[2002](https://arxiv.org/html/2608.20392#bib.bib36)\)\. On information asymmetry,Irvinget al\.\([2018](https://arxiv.org/html/2608.20392#bib.bib20)\)show that enforcing information barriers between agents enables a weaker judge to correctly evaluate claims it could not verify alone; our architecture instantiates this through blind verification\.

## 3The Evaluation\-as\-Search Framework

TheMeetingProbebenchmark is constructed using a multi\-agent framework that reframes quality evaluation of LLM meeting assistants as a feedback\-driven adaptive search problem\. Rather than curating questions manually or sampling from a fixed pool, the EaS system learns from evaluator feedback across iterations to navigate the space of natural questions, concentrating effort on cognitive demands and transcript regions where grounding failures are most likely\. We describe the search formulation, agent architecture, search operators, coverage mechanism \(the data structures tracking which cognitive demands and transcript regions have been probed and their outcomes\), and evaluation protocol\.

Plannersearch strategistGeneratorquestion tactician\+ Self\-VerifyAssistanttarget systemMARCblind evaluatorTranscript𝒯\\mathcal\{T\}source documentOrchestratorhistory⋅\\cdotcognitive demand scoreboard⋅\\cdotPlanner reflectionstrategies \(KK\)questions \(K′K^\{\\prime\}\)answersscoreboard \+reflection𝐯j∈\{1,\.\.,5\}2\\mathbf\{v\}\_\{j\}\\\!\\in\\\!\\\{1,\.\.,5\\\}^\{2\}searchevaluationper iteration,KKcandidates in parallel

Figure 1:EaS system architecture\. The search apparatus \(Planner, Generator with Self\-Verify\) and the evaluation apparatus \(MARC\) are separated by a strict information boundary \(dashed red line\): the evaluator receives only the transcript, question, and answer, with no search context\.### 3\.1Problem Formulation

Given a meeting transcript𝒯\\mathcal\{T\}and a target assistant𝒜\\mathcal\{A\}, the goal is to discover questionsqqfor which the assistant’s responsea=𝒜​\(q,𝒯\)a=\\mathcal\{A\}\(q,\\mathcal\{T\}\)exhibits quality failures: instances whereaais unfaithful to𝒯\\mathcal\{T\}, incomplete, unclear, or otherwise deficient along measurable quality dimensions\. We define the*evaluation search space*\(Eq\.[1](https://arxiv.org/html/2608.20392#S3.E1)\) as

𝒮=Topics​\(𝒯\)×QuestionTypes×CognitiveDemands×Positions​\(𝒯\),\\mathcal\{S\}=\\text\{\{Topics\}\}\(\\mathcal\{T\}\)\\times\\text\{\{QuestionTypes\}\}\\times\\text\{\{CognitiveDemands\}\}\\times\\text\{\{Positions\}\}\(\\mathcal\{T\}\),\(1\)whereTopics​\(𝒯\)\\text\{\{Topics\}\}\(\\mathcal\{T\}\)covers distinct topical regions in the transcript,QuestionTypescovers question forms \(who, what, when, how many, list, compare\),CognitiveDemandsspecify the reasoning operations the question requires \(attribution recall, temporal ordering, exhaustive enumeration, boundary distinction, and so on\), andPositions​\(𝒯\)\\text\{\{Positions\}\}\(\\mathcal\{T\}\)captures the spatial location in the transcript\. All four dimensions are open\-ended rather than fixed taxonomies\. For each questionqqdrawn from a region of𝒮\\mathcal\{S\}, a vector\-valued fitness function \(Eq\.[2](https://arxiv.org/html/2608.20392#S3.E2)\)

𝐟​\(q\)=\(ffaith​\(q\),fcomp​\(q\)\)∈\{1,…,5\}2\\mathbf\{f\}\(q\)=\(f\_\{\\mathrm\{faith\}\}\(q\),\\;f\_\{\\mathrm\{comp\}\}\(q\)\)\\in\\\{1,\\ldots,5\\\}^\{2\}\(2\)measures the assistant’s performance on two MARC dimensions: faithfulness \(grounding in the transcript\) and completeness \(coverage of all relevant information\)\. A probe is a*finding*whenmin⁡\(ffaith​\(q\),fcomp​\(q\)\)≤θ\\min\(f\_\{\\mathrm\{faith\}\}\(q\),f\_\{\\mathrm\{comp\}\}\(q\)\)\\leq\\theta, whereθ=3\\theta=3is the finding threshold\.

EaS samples𝒮\\mathcal\{S\}adaptively: it uses outcomes of previous probes to concentrate subsequent effort on high\-failure regions while maintaining sufficient exploration to avoid missing vulnerable regions elsewhere\. The output is a collection of annotated question–answer pairs, each with MARC faithfulness and completeness scores, that together characterize the grounding fidelity profile of𝒜\\mathcal\{A\}on𝒯\\mathcal\{T\}\.

### 3\.2Agent Architecture

The system comprises five functional roles: a search apparatus \(Planner, Generator, Self\-Verify\), the Assistant under test, and the MARC evaluator, all connected by the Orchestrator\. A key design principle is that*what each agent cannot see*matters as much as what it can: information asymmetries between agents prevent the biases that arise when a single monolithic system both generates and judges evaluation questions\. Figure[1](https://arxiv.org/html/2608.20392#S3.F1)shows the complete architecture\.

#### Planner \(Search Strategist\)\.

At each iteration, the Planner receives a condensed history of past probes \(questions, MARC scores, coverage statistics\) and producesKKstrategies, each specifying a cognitive demand to target, a search operator \(Exploration,Refinement, orMutation\), and concrete guidance for the Generator\. The Planner maintains a persistent*reflection*, an evolving analysis that carries forward across iterations, providing working memory for multi\-iteration reasoning about which demands and dimensions show vulnerability\. Critically, the Planner*never sees the transcript*: it reasons purely from search history and coverage statistics, preventing anchoring on salient transcript details and encouraging higher\-level strategic reasoning\.

#### Generator \(Question Tactician\)\.

The Generator translates each Planner strategy into a natural question, receiving the transcript, strategies, a position histogram, and prior questions\. It identifies specific transcript content, grounds the question in verifiable facts, and formulates it as a meeting participant would ask\. Candidates exceeding a cosine similarity threshold of 0\.85 against prior questions are discarded\.

#### Self\-Verify \(Quality Gate\)\.

Each candidate passes five verification checks: answerability from the transcript, objectivity, information\-seeking intent, naturalness \(filtering exam\-like or multi\-conditional questions\), and concrete answerability \(rejecting speculative counterfactuals\)\. Failed candidates trigger one replacement attempt targeting a different transcript region\.

#### Assistant \(Target System\)\.

The Assistant is the system under test, responding to each question given the meeting transcript\.

#### MARC \(Blind Multi\-Dimensional Evaluator\)\.

MARC receives*only*three inputs: the transcript𝒯\\mathcal\{T\}, the questionqq, and the answeraa, with no Planner strategy, Generator rationale, or search context, and produces integer scores on faithfulness \(grounding in the transcript\) and completeness \(coverage of all relevant information\) on a\{1,…,5\}\\\{1,\\ldots,5\\\}scale\. This architectural blindness mitigates confirmation bias: observing search context would conflate measurement error with strategy\-induced expectation \(see Section[4](https://arxiv.org/html/2608.20392#S4)for MARC’s calibration and validation\)\.

### 3\.3Search Operators and Coverage

The Planner directs search using three typed operators:Explorationsamples underexplored cognitive demands \(enforced as the only operator on the first iteration\);Refinementretests a productive demand in a different transcript region to assess systematicity;Mutationpivots when a demand proves unproductive or when near\-misses persist without converting to findings\.

Search is guided by a*cognitive demand scoreboard*that assigns each demanddda UCB exploration scoreUCB​\(d\)=μ^d\+c​ln⁡N/kd\\mathrm\{UCB\}\(d\)=\\hat\{\\mu\}\_\{d\}\+c\\sqrt\{\\ln N/k\_\{d\}\}, whereμ^d\\hat\{\\mu\}\_\{d\}is the effective reward rate fordd\(near\-misses contribute half\-credit\),NNis the total probe count,kdk\_\{d\}is the probe count fordd, andc=1\.0c=1\.0\. Demand statuses \(untested, dead, saturated\) and a complementary transcript\-position histogram for spatial diversity are detailed in Appendix[A](https://arxiv.org/html/2608.20392#A1)\.

The search terminates when the iteration budgetImax=20I\_\{\\max\}=20is exhausted\.

To constructMeetingProbe, we run the EaS system on 20 transcripts with three target models, collecting all probes\. Findings \(min⁡\(ffaith​\(q\),fcomp​\(q\)\)≤θ\\min\(f\_\{\\mathrm\{faith\}\}\(q\),f\_\{\\mathrm\{comp\}\}\(q\)\)\\leq\\theta\) form the positive set; probes with both scores≥θ\+1\\geq\\theta\+1form the hard negative set\. The full algorithm pseudocode appears in Appendix[A](https://arxiv.org/html/2608.20392#A1)\.

## 4MARC: Calibrated Quality Evaluation

Because feedback\-driven search depends on a calibrated evaluator \(an uncalibrated one renders the feedback signal meaningless\), we built MARC as the blind evaluation instrument that supplies the fitness signal for EaS\. MARC is a reference\-guided, single\-prompt evaluator in the taxonomy of LLM\-as\-a\-judge systems\(Zhenget al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib8)\): given a rubric, the source transcript, the question, and the candidate answer, it produces per\-dimension scores with natural\-language justifications in a single inference call\. This design keeps evaluation cost proportional to the number of probes, which is critical when the evaluator is called inside a search loop, while remaining domain\-adaptable through rubric changes alone, without the training data requirements of fine\-tuned judges\(Kimet al\.,[2024b](https://arxiv.org/html/2608.20392#bib.bib39)\)or the latency of multi\-agent debate protocols\(Kentonet al\.,[2024](https://arxiv.org/html/2608.20392#bib.bib35)\)\.

MARC scores each answer on two source\-dependent dimensions using a\{1,…,5\}\\\{1,\\ldots,5\\\}Likert scale \(full rubric in Appendix[C](https://arxiv.org/html/2608.20392#A3)\):

- •Faithfulness: the degree to which every claim in the answer is grounded in the transcript, without fabrication, distortion, or misattribution\.
- •Completeness: the degree to which the answer covers all relevant information in the transcript, without material omission\.

Other quality aspects \(relevance, fluency, coherence\) are largely source\-independent and showed ceiling effects in initial calibration\. The full rationale appears in Appendix[K](https://arxiv.org/html/2608.20392#A11)\. The evaluator operates under a strict*information barrier*: it receives only the transcript𝒯\\mathcal\{T\}, questionqq, and answeraa, with no Planner strategy, Generator rationale, or search context, preventing confirmation bias from contaminating the fitness signal\.

We validate MARC against two purpose\-built datasets, both constructed from QMSum\(Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\)using the same stratified sampling protocol but drawn from non\-overlapping meeting sets:QMSumCal50, a 150\-variant calibration set from six meetings used to refine the scoring rubric, andQMSumHeldOut, a disjoint 90\-variant held\-out set from six different meetings spanning three genres \(product design, academic research, parliamentary\)\. Each set pairs stratified queries with three answer tiers \(golden, weak, and perturbed through six targeted degradation types\), with answers generated by GPT\-family models and expected ratings assigned by three independent Claude Sonnet 4 judges under a blind protocol, providing cross\-family validation of the ground truth\. On the held\-out set, MARC achieves Spearmanρ=0\.66\\rho=0\.66–0\.730\.73\(95 % CI via 10K bootstrap\) and verdict\-level F1=0\.83=0\.83\[\.74, \.90\] for identifying grounding failures \(precision=0\.85=0\.85, recall=0\.81=0\.81\), and a synthetic perturbation study on 239 hard negatives detects 91\.6% of injected factual errors \(Appendix[J](https://arxiv.org/html/2608.20392#A10)\)\. The authors independently audited expected ratings against the source transcripts, confirming consistent alignment with expert qualitative judgment across tiers and degradation types\. Construction details, the full dimensional\-selection rationale, the complete validation\-metric table, and the within\-tier concordance analysis appear in Appendix[K](https://arxiv.org/html/2608.20392#A11)\.

## 5Experimental Setup

### 5\.1Models and Configuration

All EaS agents \(Planner, Generator, Self\-Verify, and the MARC evaluator\) are powered by GPT\-5\.2 via the Azure OpenAI Responses API \(version 2025\-04\-01\-preview\)\. Three target assistants are evaluated: GPT\-5\.2\-chat \(frontier\), GPT\-4\.1 \(standard\), and GPT\-4\.1\-mini \(compact\)\. Each run producesK=10K\{=\}10candidate questions per iteration for up toImax=20I\_\{\\max\}\{=\}20iterations, yielding a maximum of 200 probes per transcript\. Full hyperparameters are in Appendix[B](https://arxiv.org/html/2608.20392#A2)\.

### 5\.2Transcripts

We evaluate on 20 meeting transcripts from QMSum\(Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\): six product design meetings from AMI\(Carlettaet al\.,[2005](https://arxiv.org/html/2608.20392#bib.bib10)\)\(4 speakers\), six research meetings from ICSI\(Janinet al\.,[2003](https://arxiv.org/html/2608.20392#bib.bib9)\)\(4–8 speakers\), and eight parliamentary sessions \(7–105 speakers\)\. Transcripts were selected to maximize genre and structural diversity, all exceeding 100 dialogue turns\. Table[2](https://arxiv.org/html/2608.20392#S6.T2)summarizes per\-genre finding rates; per\-transcript details and dataset characteristics appear in Appendices[E](https://arxiv.org/html/2608.20392#A5)and[D](https://arxiv.org/html/2608.20392#A4)\.

### 5\.3Metrics

The primary metric is the*finding rate*\(fraction of probes with faithfulness or completeness≤3\\leq 3\); we also report*severity*\(Sev%, fraction of findings with minimum MARC dimension≤2\\leq 2\),*tokens per finding*, and*compute time per run*\. We report means with standard deviations and 95% confidence intervals, and use pairedtt\-tests across transcripts for condition comparisons\. For the transfer experiment, we report the fraction of findings on one model that also manifest on another when the same question is posed\.

## 6Results

### 6\.1Search Configuration Analysis

Table[1](https://arxiv.org/html/2608.20392#S6.T1)compares six search configurations on five transcripts across three independent trials \(per\-transcript detail in Appendix[F](https://arxiv.org/html/2608.20392#A6)\); all share the same 200\-probe budget and MARC evaluator and differ only in how probes are selected\. Three are component ablations:−\-Plannerreplaces strategic demand selection with random sampling;−\-Coverageremoves the UCB\-scored coverage map while keeping the Planner;Single\-dimrestricts MARC to faithfulness\-only scoring\. Two are alternative regimes:Static bankgenerates all 200 questions in a single call with no iteration, andPAIR\-style\(Chaoet al\.,[2023](https://arxiv.org/html/2608.20392#bib.bib6)\)serializes the adaptive loop \(200×1200\{\\times\}1instead of Full’s batched20×1020\{\\times\}10\)\.

The full EaS system achieves a 7\.1%±\\pm1\.9% finding rate,2\.5×2\.5\\timeshigher than random demand selection \(−\-Planner, 2\.9%\)\. Three of the five alternatives produce statistically significant drops relative to Full under a pairedtt\-test across transcripts: removing the Planner costs−4\.2\-4\.2pp \(p=0\.006p\{=\}0\.006\), the largest single effect, confirming that strategic selection of cognitive demands matters; restricting MARC to faithfulness only costs−1\.9\-1\.9pp \(p=0\.004p\{=\}0\.004\), because the evaluator then misses completeness\-only failures where answers are factually correct but omit critical information; and replacing iterative search with a static bank costs−2\.6\-2\.6pp \(p=0\.001p\{=\}0\.001\), confirming that iterative feedback, not just question diversity, drives the adaptive advantage\. Removing the UCB coverage map \(−0\.9\-0\.9pp,p=0\.238p\{=\}0\.238\) and serializing the adaptive loop \(−0\.7\-0\.7pp,p=0\.379p\{=\}0\.379\) produce small, non\-significant*rate*effects: these implementation details matter less than the three drivers above for finding rate, though serialization carries a large wall\-clock cost we return to next\.

![Refer to caption](https://arxiv.org/html/2608.20392v1/x1.png)Figure 2:Cumulative findings per iteration, averaged over1515runs per condition; shaded band is±\\pmSD\.The rate gap dominates the cost equation: because per\-probe evaluation accounts for the majority of tokens per run, any search\-side overhead a faster configuration saves is outweighed by the findings it fails to surface\. Full therefore undercuts every other configuration on a per\-finding basis \(2\.14M tokens per finding,1\.41\.4–2\.5×2\.5\\timescheaper than the alternatives; Table[1](https://arxiv.org/html/2608.20392#S6.T1)\)\. Faster configurations trade rate for severity as well: the static bank finishes six times faster but its findings are consistently milder \(\+0\.21\+0\.21pts,p=0\.033p\{=\}0\.033\), and serializing the adaptive loop \(PAIR\-style\) costs a4\.4×4\.4\\timeswall\-clock slowdown without improving rate or severity\. Full therefore dominates the Pareto frontier of rate, severity, and cost\.

Where does the Planner’s advantage come from? Not an initial head start: Full and−\-Planner produce the same0\.30\.3–0\.70\.7findings at iteration 1\. The gap instead widens systematically over iterations, reaching3\.5×3\.5\\timesby iteration 15 \(Figure[2](https://arxiv.org/html/2608.20392#S6.F2)\)\. A non\-learning search would keep this ratio flat or see it shrink as easy targets were exhausted; the compounding advantage is the signature of feedback\-conditioned demand selection, consistent with prior feedback\-driven systems\(Yuksekgonulet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib40); Leeet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib41); Novikovet al\.,[2025](https://arxiv.org/html/2608.20392#bib.bib42)\)\.

Table 1:Search configuration comparison on 5 transcripts across 3 independent trials \(15 runs per condition\)\. Tok/Fnd is mean tokens\-per\-finding \(in millions\) and Time is mean wall\-clock minutes per run;boldpairedppindicatesp<0\.05p<0\.05versus Full\.
### 6\.2Cross\-Genre Generalization

Running the full system on all 20 transcripts with GPT\-5\.2\-chat yields a mean finding rate of 5\.2% \(SD 2\.4 across transcripts, 95% CI\[4\.0%,6\.3%\]\[4\.0\\%,6\.3\\%\]\), with all 20 transcripts producing at least one finding \(per\-transcript rates in Table[6](https://arxiv.org/html/2608.20392#A5.T6), Appendix[E](https://arxiv.org/html/2608.20392#A5)\)\. Rates vary by genre: product design \(7\.0%\), research \(5\.8%\), parliamentary \(3\.4%\), an ordering consistent across all three target models \(Table[2](https://arxiv.org/html/2608.20392#S6.T2)\)\. Informal multi\-party discussions with rapid topic shifts are associated with higher failure rates than structured parliamentary turn\-taking, an ordering consistent with findings in long\-context dialogue evaluation\(Kimet al\.,[2024a](https://arxiv.org/html/2608.20392#bib.bib22)\)\.

### 6\.3Cross\-Model Analysis

We run EaS independently against each of the three target assistants across all 20 transcripts, producing a separate finding set per model \(1,049 findings total, 1,047 after cross\-model deduplication\)\. EaS surfaces a clear capability gradient in both rate and severity \(Table[2](https://arxiv.org/html/2608.20392#S6.T2)\): finding rate triples from GPT\-5\.2\-chat \(5\.2%\) to GPT\-4\.1\-mini \(15\.3%\), and the fraction of severe findings rises in the same direction \(27%, 19%, 39%\), with all pairwise rate differences significant atp<0\.0001p\{<\}0\.0001\. Parliamentary transcripts show the opposite pattern to their low rate: they yield the fewest findings overall but the highest severity fraction across every model \(32–42%\), indicating that when EaS surfaces a parliamentary failure it is typically a substantive grounding error rather than a borderline miss\.

Table 2:Finding rate and severity composition by genre and target model\. Rate% is the mean finding rate across transcripts in each group; Sev% is the fraction of those findings whose minimum MARC dimension score is≤2\\leq 2\(severe faithfulness or completeness failures\)\.
### 6\.4Transfer Analysis

To test whether findings are model\-specific or universal, we re\-pose each finding’s question to the other models in our set and re\-score with MARC\. Across all 1,047 findings evaluated on 5 models from 4 providers \(GPT\-5\.2\-chat, GPT\-4\.1, GPT\-4\.1\-mini, DeepSeek\-V3\.2, Llama\-3\.3\-70B\-Instruct\), 124 \(11\.8%\) are*universal failures*: they fail on every model and represent transcript\-level grounding challenges rather than model\-specific weaknesses\. Within the OpenAI family, the directional 3×\\times5 transfer matrix \(Table[8](https://arxiv.org/html/2608.20392#A7.T8), Appendix[G](https://arxiv.org/html/2608.20392#A7)\) reveals strongly asymmetric tier\-transfer: frontier\-model findings manifest on weaker GPT models 51–54% of the time, while weaker\-model findings carry upward only 17–20%\. The frontier model’s findings are the subtle, high\-difficulty questions that no model handles; weaker models additionally fail on easier questions that the frontier answers correctly\. Cross\-family transfer rates are detailed in Section[6\.5](https://arxiv.org/html/2608.20392#S6.SS5)\.

### 6\.5Multi\-Family Benchmark Validation

To test whetherMeetingProbediscriminates outside the model family used to construct it, we pose each of the 1,047 finding questions to two open\-weights models in addition to the three GPT targets and re\-score each answer with MARC\. Table[3](https://arxiv.org/html/2608.20392#S6.T3)reports finding rate \(% withmin⁡\(faith,comp\)≤3\\min\(\\text\{faith\},\\text\{comp\}\)\\leq 3\), severe rate \(% withmin≤2\\min\\leq 2\), and mean MARC faithfulness, completeness, andmin⁡\(faith,comp\)\\min\(\\text\{faith\},\\text\{comp\}\)across 5 models from 4 providers\. The benchmark separates these models cleanly, with a 44 pp finding\-rate spread: DeepSeek\-V3\.2 sits between GPT\-5\.2\-chat and GPT\-4\.1, and Llama\-3\.3\-70B\-Instruct between GPT\-4\.1 and GPT\-4\.1\-mini\. Architectural variety \(dense GPT and Llama; MoE DeepSeek\) does not disrupt the capability gradient\. Of the 1,047 findings, 124 \(11\.8%\) fail on every one of the 5 models; these universal failures form a curated hardest subset for tracking grounding progress\.

Table 3:Per\-model performance on the 1,047MeetingProbefindings\.*Finding*\(% withmin⁡\(faith,comp\)≤3\\min\(\\text\{faith\},\\text\{comp\}\)\\leq 3\) and*Severe*\(% withmin≤2\\min\\leq 2\) are failure rates, so higher values indicate more grounding failures\.*Faith*,*Comp*, and*Min*are mean MARC faithfulness, completeness, and per\-questionmin⁡\(faith,comp\)\\min\(\\text\{faith\},\\text\{comp\}\)on the 1–5 scale, so lower values indicate weaker grounding\.
### 6\.6Failure Taxonomy

To characterize the failures EaS surfaces, we classify all 1,049 pre\-curation findings into eight categories via LLM\-based taxonomy discovery\. Failures recur across transcripts and models, tied to discourse\-pragmatic demands rather than random hallucinations, and our categories parallel prior summarization\-error taxonomies\(Maynezet al\.,[2020](https://arxiv.org/html/2608.20392#bib.bib37); Pagnoniet al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib38)\)adapted to multi\-speaker meeting discourse\. The top three categories, overconfident paraphrase \(27\.3%\), unjustified quantification \(16\.9%\), and speaker misattribution \(16\.0%\), account for over 60% of failures\. Speaker misattribution rises as model capability declines, while the frontier model’s distinctive failure is fabricated action items\. Full per\-category and per\-model breakdowns appear in Tables[9](https://arxiv.org/html/2608.20392#A8.T9)–[10](https://arxiv.org/html/2608.20392#A9.T10)\(Appendices[H](https://arxiv.org/html/2608.20392#A8),[I](https://arxiv.org/html/2608.20392#A9)\)\.

## 7Discussion

#### Why feedback\-driven search matters\.

By learning from evaluator feedback across iterations, EaS discovers2\.5×2\.5\\timesmore failures per probe than random probing, with the Planner alone accounting for a 4\.2 pp improvement, analogous to the gap between random testing and coverage\-guided fuzzing\(Manèset al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib4)\)\. Because per\-probe evaluation dominates token usage, the feedback\-driven search overhead is more than paid back by the rate improvement, and Full is the cheapest configuration per finding among rate\-comparable alternatives\.

#### UsingMeetingProbe\.

The 124 universal failures \(Section[6\.5](https://arxiv.org/html/2608.20392#S6.SS5)\) form a particularly strong benchmark subset, independent of model capability\. Practitioners score a new assistant by re\-scoring the released \(transcript, question\) pairs with MARC and comparing against Table[3](https://arxiv.org/html/2608.20392#S6.T3), with the full protocol given in Appendix[M](https://arxiv.org/html/2608.20392#A13)\.

#### Limitations\.

Since both the search agents and the frontier target assistant are from the GPT\-5\.2 family, we calibrated MARC against cross\-family Claude Sonnet 4 judges on QMSumHeldOut to mitigate same\-family evaluator bias\.MeetingProbecovers English\-language QMSum transcripts and OpenAI GPT targets, and extending to additional languages, meeting styles, and model families is natural future work\.

#### Ethical considerations\.

The failure taxonomy could in principle guide targeted question construction, but the search process requires ground\-truth transcript access, limiting the threat model to insiders\. All generated questions are natural and non\-adversarial\.

## 8Conclusion

We introduced Evaluation\-as\-Search, a feedback\-driven adaptive methodology for discovering grounding failures in LLM meeting assistants, and used it to constructMeetingProbe, a benchmark of 3,009 annotated question–answer pairs spanning 20 transcripts, three genres, and three models\. By learning from evaluator feedback across iterations, EaS surfaces2\.5×2\.5\\timesmore failures than random probing while concentrating probing effort on cognitive demands where failures are most likely\. Our analysis reveals a clear capability gradient, asymmetric cross\-model transfer, and eight recurring failure categories\.

## References

- R\. Apel, T\. Braude, A\. Kantor, and E\. Kolman \(2023\)MeeQA: natural questions in meeting transcripts\.arXiv preprint arXiv:2305\.08502\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Auer, N\. Cesa\-Bianchi, and P\. Fischer \(2002\)Finite\-time analysis of the multiarmed bandit problem\.Machine Learning47\(2–3\),pp\. 235–256\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Carletta, S\. Ashby, S\. Bourban, M\. Flynn, M\. Guillemot, T\. Hain, J\. Kadlec, V\. Karaiskos, W\. Kraaij, M\. Kronenthal, G\. Lathoud, M\. Lincoln, A\. Lisowska, I\. McCowan, W\. Post, D\. Reidsma, and P\. Wellner \(2005\)The AMI meeting corpus: a pre\-announcement\.InProceedings of the Machine Learning for Multimodal Interaction Workshop \(MLMI\),pp\. 28–39\.Cited by:[§5\.2](https://arxiv.org/html/2608.20392#S5.SS2.p1.1)\.
- P\. Chao, A\. Robey, E\. Dobriban, H\. Hassani, G\. J\. Pappas, and E\. Wong \(2023\)Jailbreaking black box large language models in twenty queries\.arXiv preprint arXiv:2310\.08419\.Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1),[§6\.1](https://arxiv.org/html/2608.20392#S6.SS1.p1.4)\.
- S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. Weston \(2023\)Chain\-of\-verification reduces hallucination in large language models\.arXiv preprint arXiv:2309\.11495\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Dong, T\. Tang, J\. Li, W\. X\. Zhao, and J\. Wen \(2024\)BAMBOO: a comprehensive benchmark for evaluating long text modeling capacities of large language models\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING\),pp\. 2086–2099\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2024\)RAGAS: automated evaluation of retrieval augmented generation\.arXiv preprint arXiv:2309\.15217\.Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§1](https://arxiv.org/html/2608.20392#S1.p4.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Hu, T\. Ganter, H\. Deilamsalehy, F\. Dernoncourt, H\. Foroosh, and F\. Liu \(2023\)MeetingBank: a benchmark dataset for meeting summarization\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16409–16423\.External Links:[Link](https://arxiv.org/abs/2305.17529)Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Irving, P\. Christiano, and D\. Amodei \(2018\)AI safety via debate\.arXiv preprint arXiv:1805\.00899\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Janin, D\. Baron, J\. Edwards, D\. Ellis, D\. Gelbart, N\. Morgan, B\. Peskin, T\. Pfau, E\. Shriberg, A\. Stolcke, and C\. Wooters \(2003\)The ICSI meeting corpus\.In2003 IEEE International Conference on Acoustics, Speech, and Signal Processing \(ICASSP\),Vol\.1,pp\. I–I\.Cited by:[§5\.2](https://arxiv.org/html/2608.20392#S5.SS2.p1.1)\.
- Z\. Kenton, N\. Y\. Siegel, J\. Kramár, J\. Brown\-Cohen, S\. Albanie, J\. Bulian, R\. Agarwal, D\. Lindner, Y\. Tang, N\. D\. Goodman, and R\. Shah \(2024\)On scalable oversight with weak LLMs judging strong LLMs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2407.04622)Cited by:[§4](https://arxiv.org/html/2608.20392#S4.p1.1)\.
- D\. Kiela, M\. Bartolo, Y\. Nie, D\. Kaushik, A\. Geiger, Z\. Wu, B\. Vidgen, G\. Prasad, A\. Singh, P\. Ringshia,et al\.\(2021\)Dynabench: rethinking benchmarking in NLP\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 4110–4124\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Kim, W\. Chay, H\. Hwang, D\. Kyung, H\. Chung, E\. Cho, Y\. Kwon, Y\. Jo, and E\. Choi \(2024a\)DialSim: a real\-time simulator for evaluating long\-term multi\-party dialogue understanding of conversational agents\.arXiv preprint arXiv:2406\.13144\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1),[§6\.2](https://arxiv.org/html/2608.20392#S6.SS2.p1.1)\.
- S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. Seo \(2024b\)Prometheus: inducing fine\-grained evaluation capability in language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2608.20392#S4.p1.1)\.
- F\. Kirstein, T\. Ruas, and B\. Gipp \(2025\)Is my meeting summary good? estimating quality with a multi\-LLM evaluator\.InProceedings of the 31st International Conference on Computational Linguistics: Industry Track,pp\. 561–574\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Laban, W\. Kryscinski, D\. Agarwal, A\. R\. Fabbri, C\. Xiong, S\. Joty, and C\. Wu \(2023\)SummEdits: measuring LLM ability at factual reasoning through the lens of summarization\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 9662–9676\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Lee, J\. Boen, and C\. Finn \(2025\)Feedback descent: open\-ended text optimization via pairwise comparison\.arXiv preprint arXiv:2511\.07919\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2608.20392#S6.SS1.p4.4)\.
- P\. Li, X\. Tang, S\. Chen, Y\. Cheng, R\. Metoyer, T\. Hua, and N\. V\. Chawla \(2025a\)Adaptive testing for LLM evaluation: a psychometric alternative to static benchmarks\.arXiv preprint arXiv:2511\.04689\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, J\. Ma, M\. Ballesteros, Y\. Benajiba, and G\. Horwood \(2025b\)Active evaluation acquisition for efficient LLM benchmarking\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2511–2522\.External Links:[Link](https://arxiv.org/abs/2303.16634)Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- V\. J\. M\. Manès, H\. Han, C\. Han, S\. K\. Cha, M\. Egele, E\. J\. Schwartz, and M\. Woo \(2021\)The art, science, and engineering of fuzzing: a survey\.IEEE Transactions on Software Engineering47\(11\),pp\. 2312–2331\.Cited by:[§7](https://arxiv.org/html/2608.20392#S7.SS0.SSS0.Px1.p1.1)\.
- J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald \(2020\)On faithfulness and factuality in abstractive summarization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 1906–1919\.External Links:[Link](https://arxiv.org/abs/2005.00661)Cited by:[§6\.6](https://arxiv.org/html/2608.20392#S6.SS6.p1.1)\.
- A\. Mehrotra, M\. Zampetakis, P\. Kassianik, B\. Nelson, H\. Anderson, Y\. Singer, and A\. Karbasi \(2024\)Tree of attacks: jailbreaking black\-box LLMs automatically\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12076–12100\.External Links:[Link](https://arxiv.org/abs/2305.14251)Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Mouret and J\. Clune \(2015\)Illuminating search spaces by mapping elites\.arXiv preprint arXiv:1504\.04909\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- Y\. Nie, A\. Williams, E\. Dinan, M\. Bansal, J\. Weston, and D\. Kiela \(2020\)Adversarial NLI: a new benchmark for natural language understanding\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4885–4901\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian,et al\.\(2025\)AlphaEvolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2608.20392#S6.SS1.p4.4)\.
- A\. Pagnoni, V\. Balachandran, and Y\. Tsvetkov \(2021\)Understanding factuality in abstractive summarization with FRANK: a benchmark for factuality metrics\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4812–4829\.Cited by:[§6\.6](https://arxiv.org/html/2608.20392#S6.SS6.p1.1)\.
- A\. Panickssery, S\. R\. Bowman, and S\. Feng \(2024\)LLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:Oral presentationExternal Links:[Link](https://arxiv.org/abs/2404.13076)Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. Irving \(2022\)Red teaming language models with language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3419–3448\.External Links:[Link](https://arxiv.org/abs/2202.03286)Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- A\. Prasad, T\. Bui, S\. Yoon, H\. Deilamsalehy, F\. Dernoncourt, and M\. Bansal \(2023\)MeetingQA: extractive question\-answering on meeting transcripts\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15000–15025\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1)\.
- M\. T\. Ribeiro and S\. Lundberg \(2022\)Adaptive testing and debugging of NLP models\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 3253–3267\.Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh \(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4902–4912\.Note:Best Paper AwardExternal Links:[Link](https://arxiv.org/abs/2005.04118)Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia \(2024\)ARES: an automated evaluation framework for retrieval\-augmented generation systems\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),pp\. 338–354\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- O\. Sainz, I\. García\-Ferrero, A\. Jacovi, J\. A\. Campos, Y\. Elazar, E\. Agirre, Y\. Goldberg, W\. Chen, J\. Chim, L\. Choshen,et al\.\(2024\)Data contamination report from the 2024 CONDA shared task\.InProceedings of the 1st Workshop on Data Contamination \(CONDA\),External Links:[Link](https://arxiv.org/abs/2407.21530)Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px3.p1.1)\.
- M\. Samvelyan, S\. C\. Raparthy, A\. Lupu, E\. Hambro, A\. H\. Markosyan, M\. Bhatt, Y\. Mao, M\. Jiang, J\. Parker\-Holder, J\. Foerster, T\. Rocktäschel, and R\. Raileanu \(2024\)Rainbow teaming: open\-ended generation of diverse adversarial prompts\.arXiv preprint arXiv:2402\.16822\.Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px5.p1.1)\.
- T\. Thonet, L\. Besacier, and J\. Rozen \(2025\)ELITR\-Bench: a meeting assistant benchmark for long\-context language models\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 407–428\.Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, P\. Lu, Z\. Huang, C\. Guestrin, and J\. Zou \(2025\)Optimizing generative AI by backpropagating language model feedback\.Nature639\(8055\),pp\. 609–616\.Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px4.p1.1),[§6\.1](https://arxiv.org/html/2608.20392#S6.SS1.p4.4)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://arxiv.org/abs/2306.05685)Cited by:[§1](https://arxiv.org/html/2608.20392#S1.p4.1),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.20392#S4.p1.1)\.
- M\. Zhong, Y\. Liu, D\. Yin, Y\. Mao, Y\. Jiao, P\. Liu, C\. Zhu, H\. Ji, and J\. Han \(2022\)Towards a unified multi\-dimensional evaluator for text generation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2983–2999\.External Links:[Link](https://arxiv.org/abs/2210.07197)Cited by:[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Zhong, D\. Yin, T\. Yu, A\. Zaidi, M\. Mutuma, R\. Jha, A\. H\. Awadallah, A\. Celikyilmaz, Y\. Liu, X\. Qiu, and D\. Radev \(2021\)QMSum: a new benchmark for query\-based multi\-domain meeting summarization\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 5905–5921\.Cited by:[§K\.1](https://arxiv.org/html/2608.20392#A11.SS1.p1.1),[Appendix D](https://arxiv.org/html/2608.20392#A4.p1.1),[§1](https://arxiv.org/html/2608.20392#S1.p2.1),[§1](https://arxiv.org/html/2608.20392#S1.p5.3),[§2](https://arxiv.org/html/2608.20392#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2608.20392#S4.p5.5),[§5\.2](https://arxiv.org/html/2608.20392#S5.SS2.p1.1)\.

## Appendix AComplete Evaluation Loop Pseudocode

Algorithm 1Evaluation\-as\-Search for Benchmark Construction1:Transcript

𝒯\\mathcal\{T\}, target assistant

𝒜\\mathcal\{A\}, candidates per iteration

K=10K\{=\}10, max iterations

Imax=20I\_\{\\max\}\{=\}20, finding threshold

θ=3\\theta\{=\}3, UCB constant

c=1\.0c\{=\}1\.0, novelty threshold

τ=0\.85\\tau\{=\}0\.85
2:

ℋ←∅\\mathcal\{H\}\\leftarrow\\emptyset;

ℱ←∅\\mathcal\{F\}\\leftarrow\\emptyset;

𝑟𝑒𝑓𝑙𝑒𝑐𝑡𝑖𝑜𝑛←∅\\mathit\{reflection\}\\leftarrow\\emptyset⊳\\trianglerightHistory, findings, Planner reflection

3:

ℋdedup←∅\\mathcal\{H\}\_\{\\text\{dedup\}\}\\leftarrow\\emptyset⊳\\trianglerightAll generated questions for deduplication

4:

ℰ←∅\\mathcal\{E\}\\leftarrow\\emptyset⊳\\trianglerightEmbedding store for novelty filtering

5:for

i=1,…,Imaxi=1,\\ldots,I\_\{\\max\}do

6:

𝑠𝑐𝑜𝑟𝑒𝑏𝑜𝑎𝑟𝑑←CogDemandScoreboard​\(ℋ,c\)\\mathit\{scoreboard\}\\leftarrow\\textsc\{CogDemandScoreboard\}\(\\mathcal\{H\},c\)⊳\\trianglerightPer\-demand MARC means \+ UCB scores

7:

𝑠𝑡𝑎𝑡𝑠←StrategyStats​\(ℋ\)\\mathit\{stats\}\\leftarrow\\textsc\{StrategyStats\}\(\\mathcal\{H\}\)⊳\\trianglerightPer\-operator finding rates

8:

𝑝𝑜𝑠𝐻𝑖𝑠𝑡←PositionHistogram​\(ℋ\)\\mathit\{posHist\}\\leftarrow\\textsc\{PositionHistogram\}\(\\mathcal\{H\}\)⊳\\trianglerightTranscript position distribution

9:

𝑛𝑜𝑣𝑒𝑙𝑡𝑦𝑇𝑟𝑒𝑛𝑑←NoveltyTrend​\(ℰ\)\\mathit\{noveltyTrend\}\\leftarrow\\textsc\{NoveltyTrend\}\(\\mathcal\{E\}\)⊳\\trianglerightAvg similarity of recent questions

10:

\{s1,…,sK\}←Planner​\(ℋ,𝑠𝑐𝑜𝑟𝑒𝑏𝑜𝑎𝑟𝑑,𝑠𝑡𝑎𝑡𝑠,𝑟𝑒𝑓𝑙𝑒𝑐𝑡𝑖𝑜𝑛,𝑛𝑜𝑣𝑒𝑙𝑡𝑦𝑇𝑟𝑒𝑛𝑑,K\)\\\{s\_\{1\},\\ldots,s\_\{K\}\\\}\\leftarrow\\textsc\{Planner\}\(\\mathcal\{H\},\\mathit\{scoreboard\},\\mathit\{stats\},\\mathit\{reflection\},\\mathit\{noveltyTrend\},K\)
11:foreach strategy

sjs\_\{j\}in paralleldo

12:

qj←Generator​\(𝒯,sj,𝑝𝑜𝑠𝐻𝑖𝑠𝑡,ℋ\)q\_\{j\}\\leftarrow\\textsc\{Generator\}\(\\mathcal\{T\},s\_\{j\},\\mathit\{posHist\},\\mathcal\{H\}\)
13:

qj←SelfVerify​\(qj,𝒯\)q\_\{j\}\\leftarrow\\textsc\{SelfVerify\}\(q\_\{j\},\\mathcal\{T\}\)⊳\\trianglerightFilter or replace

14:

ℋdedup←ℋdedup∪\{qj∣j=1,…,K\}\\mathcal\{H\}\_\{\\text\{dedup\}\}\\leftarrow\\mathcal\{H\}\_\{\\text\{dedup\}\}\\cup\\\{q\_\{j\}\\mid j=1,\\ldots,K\\\}⊳\\trianglerightRecord all generated questions \(including filtered\)

15:

\{q1,…,qK′\}←NoveltyFilter​\(\{q1,…,qK\},ℰ,τ\)\\\{q\_\{1\},\\ldots,q\_\{K^\{\\prime\}\}\\\}\\leftarrow\\textsc\{NoveltyFilter\}\(\\\{q\_\{1\},\\ldots,q\_\{K\}\\\},\\mathcal\{E\},\\tau\)⊳\\trianglerightK′≤KK^\{\\prime\}\\leq Knovel candidates

16:

ℰ←ℰ∪\{Embed​\(qj\)∣j=1,…,K′\}\\mathcal\{E\}\\leftarrow\\mathcal\{E\}\\cup\\\{\\textsc\{Embed\}\(q\_\{j\}\)\\mid j=1,\\ldots,K^\{\\prime\}\\\}⊳\\trianglerightRecord surviving embeddings

17:foreach question

qjq\_\{j\}in paralleldo

18:

aj←𝒜​\(qj,𝒯\)a\_\{j\}\\leftarrow\\mathcal\{A\}\(q\_\{j\},\\mathcal\{T\}\)⊳\\trianglerightAssistant answers

19:

𝐯j←MARC​\(𝒯,qj,aj\)\\mathbf\{v\}\_\{j\}\\leftarrow\\textsc\{MARC\}\(\\mathcal\{T\},q\_\{j\},a\_\{j\}\)⊳\\trianglerightBlind evaluation;𝐯j=𝐟​\(qj\)∈\{1,…,5\}2\\mathbf\{v\}\_\{j\}=\\mathbf\{f\}\(q\_\{j\}\)\\in\\\{1,\\ldots,5\\\}^\{2\}\(Eq\.[2](https://arxiv.org/html/2608.20392#S3.E2)\)

20:foreach

\(qj,aj,𝐯j,sj\)\(q\_\{j\},a\_\{j\},\\mathbf\{v\}\_\{j\},s\_\{j\}\)do

21:

ℋ←ℋ∪\{\(qj,aj,𝐯j,sj\)\}\\mathcal\{H\}\\leftarrow\\mathcal\{H\}\\cup\\\{\(q\_\{j\},a\_\{j\},\\mathbf\{v\}\_\{j\},s\_\{j\}\)\\\}
22:if

minm⁡\(vj,m\)≤θ\\min\_\{m\}\(v\_\{j,m\}\)\\leq\\thetathen

ℱ←ℱ∪\{\(qj,aj,𝐯j\)\}\\mathcal\{F\}\\leftarrow\\mathcal\{F\}\\cup\\\{\(q\_\{j\},a\_\{j\},\\mathbf\{v\}\_\{j\}\)\\\}⊳\\trianglerightFinding

23:

𝑟𝑒𝑓𝑙𝑒𝑐𝑡𝑖𝑜𝑛←PlannerReflect​\(\{\(qj,𝐯j\)\},ℱ,𝑟𝑒𝑓𝑙𝑒𝑐𝑡𝑖𝑜𝑛\)\\mathit\{reflection\}\\leftarrow\\textsc\{PlannerReflect\}\(\\\{\(q\_\{j\},\\mathbf\{v\}\_\{j\}\)\\\},\\mathcal\{F\},\\mathit\{reflection\}\)
24:return

\(ℋ,ℱ\)\(\\mathcal\{H\},\\mathcal\{F\}\)⊳\\trianglerightFull probe history and confirmed findings

## Appendix BHyperparameter Configuration

Table[4](https://arxiv.org/html/2608.20392#A2.T4)lists all hyperparameters used in our experiments\.

Table 4:Full hyperparameter configuration for the EaS system\.CategoryParameterValueLLMEvaluation model \(all search agents\)GPT\-5\.2Target modelsGPT\-5\.2\-chat, GPT\-4\.1, GPT\-4\.1\-miniAPI version2025\-04\-01\-previewReasoningReasoning effort \(search agents\)mediumSearchCandidates per iteration \(KK\)10Max iterations \(ImaxI\_\{\\max\}\)20Min iterations \(IminI\_\{\\min\}\)5Max context history \(Planner\)30UCB exploration constant \(cc\)1\.0Near\-miss weight0\.5Novelty threshold \(τ\\tau\)0\.85EmbeddingsEmbedding modeltext\-embedding\-3\-smallEmbedding dimension1536ThresholdsFinding threshold \(θ\\theta\)3Near\-miss threshold4TerminationDead demand threshold4 tests, 0 findingsQualityMax verify retries1Self\-verify checks5 \(answerable, objective, info\-seeking, natural, concrete\)AsyncMax concurrent API requests10Max retries \(rate limit\)5Retry base delay \(seconds\)2\.0
GPT\-5\.2 uses reasoning effort control via the Responses API; temperature and top\-p parameters are not supported\.

## Appendix CAgent Prompt Summaries

We summarize the system prompts for each agent\. Full prompts are available in the released codebase\.

#### Planner \(Search Strategist\)\.

The Planner receives condensed history with per\-cognitive\-demand and per\-strategy\-type statistics, the cognitive demand scoreboard with UCB scores, a novelty trend indicating semantic diversity of recent questions, and its own persistent reflection from prior iterations\. It outputsKKstrategies, each specifying a cognitive demand, strategy type \(Exploration/Refinement/Mutation\), a reference to the prior finding being refined \(for non\-exploration strategies\), and concrete guidance for the Generator\. The Planner never receives the transcript text\.

#### Generator \(Question Tactician\)\.

The Generator receives the transcript, strategies from the Planner, a transcript position histogram, and the history of all previously generated questions\. For each strategy, it selects a specific transcript passage and formulates a natural question that a real meeting participant would plausibly ask\. Output includes the question text, topic, question type, transcript region description, estimated transcript position \(0–100%\), and cognitive demand label\.

#### Self\-Verify \(Quality Gate\)\.

Five verification checks are applied to each candidate: \(1\) answerable from the transcript alone, \(2\) seeks factual information rather than opinions, \(3\) asks about meeting content rather than meta\-level properties, \(4\) phrased in a natural conversational style rather than as an exam\-like question, and \(5\) concretely answerable from the transcript rather than requiring speculation or counterfactual reasoning\. Failed candidates trigger a topic\-pivot replacement targeting a different transcript region\. Each candidate is allowed at most one replacement attempt\.

#### Assistant \(Target System\)\.

The target meeting assistant receives the question and transcript\. It is instructed to ground answers in the transcript content and be honest about uncertainty\.

#### MARC \(Blind Evaluator\)\.

MARC receives only the transcript, question, and answer\. It evaluates the response on faithfulness \(grounding in the transcript\) and completeness \(coverage of all relevant information\) on a 1–5 scale\. The prompt includes calibrated scoring rubrics with few\-shot examples anchoring each score level\. No search context, Planner strategies, or Generator rationale are provided\.

## Appendix DDataset Details

Table[5](https://arxiv.org/html/2608.20392#A4.T5)provides metadata for all 20 transcripts used in benchmark construction\. All transcripts are sourced from QMSum\[Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\]for consistent formatting\.

Table 5:Dataset characteristics for all 20 transcripts\. All sourced from QMSum, originally from AMI, ICSI, and parliamentary corpora\.TranscriptCorpusGenreSpeakersTurnsES2004cAMIProduct design4604IS1003bAMIProduct design4407product\_1AMIProduct design41,616product\_2AMIProduct design41,401product\_3AMIProduct design41,093product\_4AMIProduct design41,050Bed003ICSIResearch41,029Bro004ICSIResearch7748academic\_1ICSIResearch51,831academic\_2ICSIResearch61,365academic\_3ICSIResearch8985academic\_4ICSIResearch5680committee\_1ParliamentaryParliamentary71393committee\_2ParliamentaryParliamentary105370committee\_3ParliamentaryParliamentary96315committee\_4ParliamentaryParliamentary11234committee\_5ParliamentaryParliamentary11229committee\_6ParliamentaryParliamentary11212covid\_4ParliamentaryParliamentary89276education\_13ParliamentaryParliamentary7133The AMI transcripts feature scenario\-based product design meetings with four speakers in assigned roles \(Project Manager, Marketing, Industrial Designer, User Interface Designer\)\. The ICSI transcripts capture naturally occurring research group meetings with overlapping speech, technical jargon, and informal discourse\. The parliamentary transcripts follow structured turn\-taking with formal language, larger speaker counts, and procedurally constrained discourse\.

## Appendix EPer\-Transcript Detailed Results

Table 6:Finding rate \(%\) per model and transcript\. Bold indicates the highest finding rate per row\.Table[7](https://arxiv.org/html/2608.20392#A6.T7)below provides additional per\-transcript detail for the unified search\-configuration experiment\.

## Appendix FSearch Configuration Detailed Results

Each cell reports the trial\-averaged finding rate for a single transcript across three independent trials\. The Mean±\\pmSD row averages across runs \(15 runs per column\)\. All six configurations share the same 200\-probe budget and MARC evaluator; they differ only in how probes are selected\.

Table 7:Per\-transcript search\-configuration results, averaged across three independent trials\.
## Appendix GCross\-Model Transfer Matrix

To measure whether failures discovered on one model also manifest on others, we pose each model’s finding questions to every other model in our set and re\-evaluate with MARC\. Table[8](https://arxiv.org/html/2608.20392#A7.T8)reports the pairwise transfer rates\. Downward transfer within the OpenAI family \(frontier→\\toweaker\) is consistently high \(51–54%\), indicating that frontier\-model failures tend to be universally hard\. Upward transfer \(weaker→\\tofrontier\) is limited \(17–20%\), confirming that weaker models additionally fail on easier questions the frontier answers correctly\. Lateral transfer between GPT\-4\.1 and GPT\-4\.1\-mini is moderate \(52% downward, 33% upward\), consistent with their intermediate capability gap\. Cross\-family transfer to DeepSeek\-V3\.2 ranges from 44% to 54%, and to Llama\-3\.3\-70B\-Instruct from 62% to 67%, broadly tracking each target’s overall finding rate \(Table[3](https://arxiv.org/html/2608.20392#S6.T3)\)\.

Table 8:Cross\-model and cross\-family transfer matrix\. Cell\(i,j\)\(i,j\)shows the fraction of findings discovered on modeliithat also fail on modeljjwhen the same question is posed and re\-scored\. Within\-OpenAI transfer is strongly asymmetric: downward \(51–54%\) far exceeds upward \(17–20%\)\. DeepSeek\-V3\.2 and Llama\-3\.3\-70B\-Instruct were not used as discovery agents and so appear as target columns only\.
## Appendix HFailure Taxonomy Details

Table 9:Taxonomy of grounding failure modes \(NN=1,049 findings across three models\)\. Categories are derived from LLM\-based clustering and per\-finding classification; full descriptions appear below the table\. Counts reflect the 1,049 pre\-deduplication findings; the released benchmark retains 1,047\.Category descriptions\.*Overconfident paraphrase*occurs when the assistant converts vague or elliptical utterances \(e\.g\., “this,” “too great”\) into confident factual claims, over\-committing to a specific interpretation where the transcript is ambiguous\.*Unjustified quantification*arises when the assistant fabricates exact counts, exhaustive lists, or precise figures not supported by the transcript \(e\.g\., “all three constraints” when only two are mentioned\)\.*Speaker misattribution*assigns a statement, proposal, or objection to the wrong meeting participant\.*Overstated decision finality*presents an outcome as explicitly decided or agreed upon when the transcript shows only tentative discussion or unresolved deliberation\.*Fabricated action items*occur when the assistant infers concrete task ownership, commitments, or due dates from general discussion without explicit assignment in the transcript\.*False temporal precedence*involves incorrectly answering “who first” or “when” questions by inventing or reversing the order of events in the discussion\.*Mischaracterized intent*supplies or distorts the reasoning behind a participant’s stance \(e\.g\., attributing cost concerns when the speaker cited aesthetic preferences\)\.*Unsupported participant enumeration*asserts a specific set or count of speakers involved in a discussion point without transcript support\.

## Appendix IPer\-Model Failure Profile

Table[10](https://arxiv.org/html/2608.20392#A9.T10)shows the percentage of each model’s findings falling into each failure category\.

Table 10:Failure category distribution \(%\) by target model\. Bold indicates the model with the highest percentage for each category\.
## Appendix JPerturbation Sensitivity Details

We describe the synthetic perturbation pipeline used to validate MARC’s sensitivity to grounding errors\.

#### Perturbation generation\.

For each of the 20 transcripts, we sample 10–14 correct answers where both MARC faithfulness and completeness score≥4\\geq 4\(hard negatives\), yielding 239 source answers in total\. We then prompt GPT\-5\.2 to introduce exactly one targeted factual error into each answer, drawn from five error types:*speaker swap*\(misattribute a statement to a different participant\),*number change*\(alter a quantity, percentage, or count\),*claim reversal*\(negate or invert a factual claim\),*temporal error*\(reorder or shift the timing of events\), and*fabrication*\(insert a detail not present in the transcript\)\. The generation prompt requires the model to \(a\) produce the perturbed answer, \(b\) identify the error type, and \(c\) cite the exact transcript passage that contradicts the perturbation\.

#### Evidence verification\.

Each cited transcript passage is verified by automated substring matching against the source transcript text, confirming that the cited evidence actually appears in the transcript and contradicts the introduced error\. All 239 perturbations pass this verification step, ensuring that every error has transcript\-grounded proof of incorrectness\.

#### Detection criterion\.

Each perturbed answer is re\-scored by MARC\. A perturbation is considered*detected*if the perturbed score on either faithfulness or completeness drops to≤3\\leq 3\(the finding threshold\), indicating that MARC identifies the introduced error as a grounding failure\.

#### Results by error type\.

The 239 perturbations span all 20 transcripts and all three genres \(Table[11](https://arxiv.org/html/2608.20392#A10.T11)\)\. Overall, MARC detects 219 of 239 perturbations \(91\.6%\)\.

Table 11:MARC sensitivity to synthetic perturbations\. Each perturbation introduces one verified factual error into a correct answer\. Detection = perturbed score drops to≤3\\leq 3on faithfulness or completeness\.Detection rates are highest for temporal errors \(12/12, 100%\) and speaker swaps \(93/95, 97\.9%\), followed by claim reversals \(55/58, 94\.8%\), fabrications \(15/18, 83\.3%\), and number changes \(44/56, 78\.6%\)\. Detection is consistent across transcripts and genres, with no transcript producing fewer than 80% detection\.

#### Undetected perturbations\.

The 20 undetected perturbations all received perturbed scores of exactly 4 \(one point above the finding threshold\), indicating that MARC recognized a quality reduction but did not score it as a clear failure\. The majority of undetected cases \(14 of 20\) are of the*number change*type, where the error involves a single altered digit embedded in an otherwise correct and well\-structured answer, a subtle perturbation that does not disrupt the overall narrative coherence of the response\. The remaining undetected cases comprise 3 fabrications, 2 speaker swaps, and 1 claim reversal, all involving minor details peripheral to the main answer content\.

## Appendix KMARC Calibration & Validation Details

### K\.1QMSumCal50 and QMSumHeldOut Construction

QMSumCal50 is built on top of QMSum\[Zhonget al\.,[2021](https://arxiv.org/html/2608.20392#bib.bib11)\], a query\-based meeting summarization dataset whose 1,576 specific queries are human\-authored questions about real meetings paired with abstractive, human\-written reference answers\. Every query includes annotated relevant text spans linking the answer to its supporting transcript segments, enabling rigorous faithfulness evaluation\.

Using stratified sampling over QMSum’s test split, we selected 50 questions from six transcripts spanning three genres \(product design, academic research, parliamentary committee\), balancing question types \(factual, person\-specific, summary, why/how, decision, action\) to cover a broad range of cognitive demands\. For each question we generated three answer variants at controlled quality tiers using different LLMs and generation strategies\.*Golden*answers were produced by a frontier model \(GPT\-5\.1 for Cal50; GPT\-5\.2 for HeldOut\) with access to the full transcript and human\-annotated relevant spans, yielding faithful, comprehensive responses\.*Weak*answers came from a smaller model \(GPT\-4\.1\-nano\) with constrained output length and no curated spans, producing shallow but on\-topic responses that are faithful yet incomplete\.*Perturbed*answers were created by systematically degrading golden answers through six perturbation types, each targeting specific failure modes \(Table[12](https://arxiv.org/html/2608.20392#A11.T12)\)\.

Table 12:Perturbation types used to generate degraded answer variants from golden answers\.Expected ratings for Cal50 were established through multi\-agent LLM scoring \(three independent GPT\-4\.1 judges per variant, with human review\), followed by a cross\-family blind audit in which three Claude Sonnet 4 judges independently re\-scored all 150 variants against the source transcripts\. The audit corrected 78 ratings across three cumulative batches, producing the final expected ratings used for all downstream evaluation\.

Using Cal50 as a development set, we iteratively refined the MARC scoring rubric to improve alignment with the audited expected ratings\. The rubric was frozen after this calibration phase and before any held\-out scoring\.

QMSumHeldOut is a 90\-variant set drawn from six QMSum meetings disjoint from the Cal50 pool, spanning the same three genres\. For each of 30 questions we generated three answer tiers \(golden, weak, perturbed\) using the same protocol as Cal50 \(golden answers produced by GPT\-5\.2 and perturbations via the same six degradation types\)\. To mitigate same\-family bias—GPT\-5\.x is used for both answer generation and MARC scoring—expected ratings for the held\-out set were assigned by three independent Claude Sonnet 4 judges under a blind protocol\.

### K\.2Dimensional Selection Rationale

MARC evaluates only faithfulness and completeness because these are the source\-dependent dimensions: they can only be assessed by comparing an answer against the meeting transcript, and their failure modes \(fabricated claims, omitted information, speaker misattribution\) arise specifically from how the system processes a particular source document\. In contrast, relevance and quality are largely source\-independent: whether an answer addresses the question asked, or whether it is coherent and well\-formed, can be judged without reference to any specific transcript\. Because these source\-independent properties are governed by the LLM’s general instruction\-following and language generation capabilities rather than by its grounding in a particular meeting, they do not exhibit the kind of systematic, source\-specific failure patterns that a targeted search over potential questions and answers would surface\. Initial calibration on QMSumCal50 confirmed this structural expectation empirically: relevance and quality scores showed strong ceiling effects \(86% and 91% of scores≥4\\geq 4, respectively\) with low variance, offering insufficient discriminative signal for a search objective\. We therefore direct MARC’s evaluation budget toward the dimensions where source\-grounding failures concentrate and where configuration choices meaningfully affect outcomes\.

### K\.3Full Validation Metrics

Table[13](https://arxiv.org/html/2608.20392#A11.T13)reports score\-level agreement on both datasets\.

Table 13:MARC scoring agreement on the calibration set \(QMSumCal50,n=150n\{=\}150\) and held\-out set \(QMSumHeldOut,n=90n\{=\}90\)\. 95% CIs via Fisherzz\-transformation \(Pearson\) and 10K bootstrap \(Spearman\)\. Within\-1 is the fraction of scores within one point of the expected value\. All correlationsp<0\.001p<0\.001\.Since Likert scores are ordinal, we foreground rank\-based measures\. Spearman correlations reachρ≥0\.66\\rho\\geq 0\.66across all conditions, with held\-out completeness atρ=0\.73\\rho=0\.73\[\.61, \.81\]\. Held\-out faithfulness \(ρ=0\.66\\rho=0\.66\[\.52, \.76\]\) is moderate, reflecting the difficulty of ranking subtle faithfulness errors in unseen meetings\. We additionally compute weighted Cohen’sκ=0\.74\\kappa=0\.74\(substantial agreement\) and Kendall’sτ=0\.64\\tau=0\.64–0\.700\.70across dimensions, both appropriate for ordinal data\. For comparability with prior work, we also report Pearsonrr: both dimensions achiever≥0\.78r\\geq 0\.78, with held\-out values virtually identical to calibration \(faithfulness:\.80\.80vs\.\.80\.80; completeness:\.76\.76vs\.\.78\.78\), consistent with the rubric not having overfit\. Within\-1 agreement is actually*higher*on the held\-out set \(9393–96%96\\%vs\.8686–93%93\\%\)\. Two factors likely contribute: the independent Claude Sonnet 4 judges produce smoother expected\-score distributions than the iteratively corrected Cal50 ratings, and the GPT\-5\.2 golden answers used in HeldOut \(vs\. GPT\-5\.1 in Cal50\) may produce cleaner quality gradients\. The correlation metrics, which measure rank agreement rather than absolute score level, are robust to this shift\.

### K\.4Within\-Tier Discrimination

Because the three\-tier design creates large between\-tier quality gaps, the correlations in Table[13](https://arxiv.org/html/2608.20392#A11.T13)are dominated by between\-tier variance\. To assess whether MARC discriminates*within*the quality range that matters for the search loop, we report pairwise concordance and effect sizes in Table[14](https://arxiv.org/html/2608.20392#A11.T14)\. Traditional rank correlations \(Spearman, Kendall\) are ill\-suited here: each tier spans only 1–2 points on the Likert scale by design, making the traditional correlation definitions irrelevant\. Instead, for every cross\-tier pair of answers we compute the*concordance rate*\(the fraction of pairs in which MARC assigns a higher or equal score to the higher\-tier answer\) and Cohen’sddbetween the tier\-level MARC score distributions\.

Table 14:Pairwise tier discrimination \(concordance rate / Cohen’sdd\) on QMSumCal50 \(n=150n\{=\}150\) and QMSumHeldOut \(n=90n\{=\}90\)\. Concordance is the fraction of cross\-tier pairs where MARC scores the higher\-tier answer≥\\geqthe lower\-tier answer\. Cohen’sddis computed on MARC score distributions between tiers\.Across both datasets, MARC reliably separates quality tiers\. Golden–perturbed concordance exceeds96%96\\%with very large effect sizes \(d\>1\.7d\>1\.7\); weak–perturbed concordance exceeds92%92\\%\(d\>1\.3d\>1\.3\), confirming that MARC discriminates adjacent tiers, not just extremes\. Golden–weak faithfulness concordance is lower \(7979–81%81\\%,d≈0d\\approx 0\), but this is*expected by design*: weak answers are generated without hallucination injection and are therefore faithful; faithfulness is not designed to separate these tiers\. Completeness correctly serves that role, with golden–weak concordance of9090–95%95\\%\(d=1\.0d=1\.0–1\.51\.5\)\. The two dimensions are thus complementary: faithfulness discriminates perturbed from \{golden, weak\}, while completeness discriminates golden from \{weak, perturbed\}\.

At the verdict level \(where a*finding*is defined asmin⁡\(ffaith,fcomp\)≤3\\min\(f\_\{\\text\{faith\}\},f\_\{\\text\{comp\}\}\)\\leq 3\), MARC achieves F1=0\.91=0\.91\[\.86, \.95\] on Cal50 \(precision=0\.90=0\.90, recall=0\.92=0\.92\) and F1=0\.83=0\.83\[\.74, \.90\] on the held\-out set \(precision=0\.85=0\.85, recall=0\.81=0\.81\); 95% CIs via 10K bootstrap\. The held\-out drop is concentrated in recall: some subtle grounding failures in unseen meetings evade detection, while precision remains high, ensuring that flagged findings are trustworthy\. This precision–recall profile suits the EaS search loop: high precision ensures the optimizer does not chase false positives, while the modest recall gap means some subtle failures go undetected—an acceptable trade\-off given that the search probes each configuration from multiple angles\.

## Appendix LMeetingProbe Benchmark Schema

Table 15:MeetingProbe benchmark statistics\. Failure category counts reflect the 1,047 post\-deduplication findings in the released benchmark \(1,049 pre\-deduplication; see Table[9](https://arxiv.org/html/2608.20392#A8.T9)\)\.Each entry in theMeetingProbebenchmark contains the following fields:

- •transcript\_id: Identifier linking to QMSum source transcript\.
- •model: Target assistant model \(gpt\-5\.2\-chat, gpt\-4\.1, gpt\-4\.1\-mini\)\.
- •question: The natural\-language question posed to the assistant\.
- •answer: The assistant’s response\.
- •is\_finding: Boolean indicating whether this is an identified quality failure\.
- •marc\_faithfulness: MARC faithfulness score \(1–5\)\.
- •marc\_completeness: MARC completeness score \(1–5\)\.
- •cognitive\_demand: The cognitive demand label \(e\.g\., “attribution recall”\)\.
- •failure\_category: One of eight categories \(for findings\) or null \(for hard negatives\)\.
- •genre: Meeting genre \(product\_design, research, parliamentary\)\.
- •corpus: Source corpus \(AMI, ICSI, Parliamentary\)\.

The benchmark is distributed as a single JSON file \(meetingprobe\.json, 8\.5 MB\) with accompanying metadata and documentation\.

## Appendix MPractitioner Usage Guide

A practitioner scores a new meeting assistant by loadingmeetingprobe\.json, posing each \(transcript, question\) pair to the model, re\-scoring the answer with the released MARC evaluator, and comparing finding rate, severe rate, and mean MARC faithfulness, completeness, andmin⁡\(faith,comp\)\\min\(\\text\{faith\},\\text\{comp\}\)against the references in Table[3](https://arxiv.org/html/2608.20392#S6.T3)\. The 124 universal failures form a curated hardest subset: a model that beats current frontier on these is making genuine progress on grounded meeting QA\. The per\-category breakdown in our released analysis files supports targeted regression testing, since a fix for, say, speaker misattribution can be verified by filtering to that category and re\-scoring\. The 1,962 hard negatives released alongside the benchmark remain useful as evaluator\-calibration controls when validating a new judge\.