Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training

# Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training
Source: [https://arxiv.org/html/2610.38220](https://arxiv.org/html/2610.38220)
![[Uncaptioned image]](https://arxiv.org/html/2610.38220v1/artifacts/figs/HandshakeAI_Logo_Black.png)

Hui Wen Goh\*Affiliation:Handshake AIGuram GogiaAffiliation:Handshake AIJonas MuellerAffiliation:Handshake AI

###### Abstract

Rubrics enable reward assignment on tasks without verifiable answers, but are expensive to manually produce from scratch\. We present*Rubricon*, a pipeline that generates discriminative rubrics without requiring human preference labels, rubric annotation, or generator\-model training\. Using any LLM, Rubricon samples multiple rubrics, selects the most discriminative candidate, adds criteria to capture overlooked requirements or response flaws, and prunes criteria that are misaligned or shallow\. Alongside LLM self\-critique and test\-time scaling, Rubricon uses observed criterion pass rates on example model responses to guide refinement\. Across five professional, consumer, clinical, and general\-use LLM benchmarks, Rubricon produces rubrics whose scores of frontier LLM responses exhibit9%greater alignment with labor\-intensive expert\-authored rubrics, compared to the best alternative synthetic rubric generation method among RRD, QSRG, OpenRubrics, and agentic generation via harnesses like Codex\. On HealthBench, post\-training a Qwen3\.5\-9B model with Rubricon rubrics yields4\.5%better performance than training with rubrics from the strongest alternative synthetic rubric generator, RRD\.

11footnotetext:Equal contribution00footnotetext:Correspondence to\{ulyana\.tkachenko,huiwen\.goh,jonas\.mueller\}@joinhandshake\.com## 1Introduction

Rubrics provide explicit, checkable criteria for systematically evaluating open\-domain LLM responses and assigning post\-training rewards\([Gunjal et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib13);[Huang et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib11);[Xu et al\., 2026a](https://arxiv.org/html/2610.38220#bib.bib12)\)\. Expert\-authored rubrics are the gold standard but are painstaking to produce \(datasets with rubrics can require hundreds of physicians or bankers\([Arora et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib18);[Lau et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib3)\)\)\. Asking an LLM to generate a rubric yields criteria too shallow to separate strong responses from weak ones, or even criteria that incorrectly favor lower\-quality responses\([Shen et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib7);[Liu et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib14)\)\. A further challenge is building rubrics that remain discriminative as models improve, which requires criteria that will appropriately penalize flaws in responses from stronger*future*models not yet available\.

We propose*Rubricon*, a straightforward pipeline that automatically generates higher\-quality rubrics via three stages:Generate\-and\-Select,Add, andPrune\. It targets three properties a good rubric needs:\(i\)the ability to discriminate between responses of different quality,\(ii\)coverage so that substantively weak responses do not go unpenalized, and\(iii\)headroom for quantifying flaws in responses from stronger future models\. The pipeline does not require any human data annotation or model fine\-tuning, and can utilize*any*LLM to generate/refine the rubrics, yielding better rubrics as LLMs advance\. Rubricon also supports iterative refinement as the Add and Prune stages can be repeatedly applied to continuously reassess and improve a rubric’s criteria\.

Rubricon illustrates a way to scale*test\-time compute*\([Snell et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib6)\)for rubric generation that is empirically more effective than alternate approaches like agentic rubric generation or Recursive Rubric Decomposition\([Shen et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib7)\)\. Relying solely on LLM self\-critique or ensembling yields limited test\-time scaling gains due to LLM brittleness and biases in the rubric generation setting\. Instead, Rubricon additionally grounds its refinement stages in how candidate rubrics actually score example responses\. Because existing LLMs cannot properly self\-judge their own generated rubrics, our refinement stages encourage the models to rely on particular outside signals and workflows which simplify the judgment calls a model must make\. Compared to existing rubric\-generation techniques, the resulting Rubricon\-generated rubrics score responses more consistently with expert\-authored rubrics \(Table[1](https://arxiv.org/html/2610.38220#S6.T1)\) and improve post\-training outcomes \(Figure[2](https://arxiv.org/html/2610.38220#S7.F2)\)\.

## 2Related Work

Synthetic rubrics offer a structured alternative to holistic evaluation, providing explicit, interpretable criteria for open\-ended tasks and post\-training rewards\([Gunjal et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib13);[Huang et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib11);[Xu et al\., 2026a](https://arxiv.org/html/2610.38220#bib.bib12)\)\. However, naively generated synthetic rubrics can omit relevant dimensions or contain generic, conflated, or misaligned criteria, reducing judgment accuracy\([Shen et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib7)\)\. One omission from much of the existing literature is evaluating how well a model’s synthetic rubrics score responses from future stronger models\. In practice, that is how synthetic rubrics would advance the frontier of AI in non\-verifiable domains, especially if these rubrics are subsequently edited/validated by experts and must provide lasting value that justifies this effort\.

##### Rubric construction strategies\.

AutoChecklist\([Zhou and Tan, 2026](https://arxiv.org/html/2610.38220#bib.bib1)\)considers five paradigms for rubric generation and validation: direct prompting, contrastive generation, inductive extraction from feedback, decomposition of predefined dimensions, and interactive construction using human and LLM feedback\. Response\-grounded approaches to rubric\-generation derive discriminative criteria from observed quality differences between candidate responses\([Liu et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib14);[Zhang et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib2)\)\. RubricRAG\([Dhole and Agichtein, 2026](https://arxiv.org/html/2610.38220#bib.bib15)\)instead retrieves rubrics for similar queries and uses them as few\-shot examples to guide generation\.

Our method simply generates an initial draft rubric via a single LLM call, using a prompt that encodes general best practices\([Rao and Callison\-Burch, 2026](https://arxiv.org/html/2610.38220#bib.bib4)\)\. This allows Rubricon to be flexibly tailored for various settings where auxiliary inputs can be included in this LLM call, such as when there are expert\-written golden responses, unstructured notes on what makes a response good/bad, critiques of particular responses or expert\-preferences between them, or few\-shot examples of expert\-crafted rubrics\. Rubricon’s refinement stages can be applied to initial draft rubrics generated from any other approach\.

##### Iterative and adaptive refinement\.

Recursive Rubric Decomposition \(RRD\)\([Shen et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib7)\)recursively decomposes broad criteria into finer\-grained dimensions while filtering misaligned and redundant criteria\. Its whitened\-uniform \(WU\) variant further applies correlation\-aware weighting of rubric criteria to prevent overlapping criteria from dominating the aggregate score\. Other refinement approaches include: RubricHub\([Li et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib10)\), which combines criteria from multiple models and compares high\-quality responses; Chasing the Tail\([Zhang et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib2)\), which constructs discriminative rubrics from strong, diverse responses to mitigate reward over\-optimization; and SibylSense\([Xu et al\., 2026b](https://arxiv.org/html/2610.38220#bib.bib16)\), which adapts a frozen rubric\-generator through a memory bank of validated criteria, alternating verifier\-guided updates with adversarial generation of candidate responses\.

##### Training rubric generators\.

OpenRubrics\([Liu et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib14)\)uses contrastive prompting to derive hard rules and implicit principles from prompts and preference\-ordered responses, then fine\-tunes a rubric generator and judge on examples filtered for preference consistency\.[Lv et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib22)train a query\-specific rubric generator \(QSRG\) through reinforcement learning, rewarding human\-preference consistency, format validity, and LLM\-assessed rubric quality\.[Sheng et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib17)jointly train one model for reasoning and rubric generation, using evolving criteria for chain\-of\-thought supervision\.

## 3Task Formulation

We consider open\-ended tasks, each specified by a promptTTfor which a model should generate a response\. A rubric forTTis a set of weighted binary criteriaℛ=\{\(ck,wk\)\}k=1m\\mathcal\{R\}=\\\{\(c\_\{k\},w\_\{k\}\)\\\}\_\{k=1\}^\{m\}used to grade any possible response, where each criterionckc\_\{k\}is a binary \(pass/fail\) natural\-language requirement andwkw\_\{k\}encodes its relative importance\. A*verifier*modelv⁡\(ck,y\)∈\{0,1\}v\(c\_\{k\},y\)\\in\\\{0,1\\\}judges whether responseyysatisfies each criterionckc\_\{k\}\. The responseyyreceives an overall normalized score:

sℛ​\(y\)=∑k=1mwk​v​\(ck,y\)∑k=1mmax⁡\(wk,0\)s\_\{\\mathcal\{R\}\}\(y\)\\;=\\;\\frac\{\\sum\_\{k=1\}^\{m\}w\_\{k\}\\,v\(c\_\{k\},y\)\}\{\\sum\_\{k=1\}^\{m\}\\max\(w\_\{k\},0\)\}\(1\)
We use this score to rank responses from different models or as rewards in post\-training\. While Rubricon’s rubrics employ positive weights for simplicity, some rubrics contain negative\-weight criteria \(including baseline methods we compare against in our experiments\)\.

A strong rubric satisfies three*desiderata*:

1. 1\.Discrimination\.The rubric should assign higher scores to higher\-quality responses, clearly separating better from worse\.
2. 2\.Coverage\.The rubric should capture relevant requirements and failure modes that models actually exhibit, so that flawed responses do not receive high scores\.
3. 3\.Headroom\.The rubric should remain sensitive to meaningful quality differences among responses from future models, even as their capabilities improve\.

Due to verifier noise and computation costs as well as human\-editing/review costs, more criteria are not always desirable in a rubric\. A rubric should instead have the fewest criteria that achieve the aforementioned desiderata, and should not include basic criteria that any existing model would pass or subjective/complex criteria that a simple verifier model cannot reliably grade\.

![Refer to caption](https://arxiv.org/html/2610.38220v1/artifacts/figs/rubricon_pipeline.png)Figure 1:The Rubricon pipeline\.Given a task prompt, example strong\-model responses, and optional reference material, Generate\-and\-Select draftsK=5K=5candidate rubrics and selects the rubric with the largest average strong–weak edge across criteria\. Add sequentially expands the selected rubric through gap fill, prompt coverage, negative probing, and completeness probing, with each step operating on the rubric from the prior step\. The probe steps generate responses intended to satisfy the current rubric, then convert independently identified defects or omissions into new criteria\. Prune removes misaligned criteria and those that are satisfiable by evasive or weak responses\. The resulting rubric can undergo further Add then Prune refinement rounds\.
## 4Rubricon

Rubricon constructs and refines rubrics through a modular pipeline that can utilize any off\-the\-shelf LLM as the*generator model*\. Unlike rubric generators that require model training, the performance of our approach can be instantly improved whenever better LLMs are released\. Rubricon can also leverage expert reference responses or expert reasoning as additional context whenever available for a task\. It measures criterion pass rates on sample responses and uses LLM\-based checks to identify missing requirements and poorly formulated criteria\. Its main discriminative signal is the*strong–weak edge*, the gap in criterion pass rates between stronger\- and weaker\-model responses, adapted from the misalignment filter proposed by[Shen et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib7)\. The pipeline uses these signals to select an initial rubric, add missing criteria, and remove misaligned criteria or criteria that are satisfiable by evasive or weak responses\.

### 4\.1Aspects of Effective Rubric Criteria

Beyond the rubric\-level desiderata in Section[3](https://arxiv.org/html/2610.38220#S3), a good rubric requires well\-formed individual criteria\. We enumerate a universal set ofCriteria Principles, a task\-agnostic style guide for writing criteria, which we embed into our criterion generation steps\. Each criterion should beclear and unambiguous, with an explicit pass condition;self\-contained, requiring no external references to assess the response;atomic, testing only one requirement; andfree of cross\-references, requiring no other criterion or verdict to interpret\. These principles govern individual criteria, while the Rubricon pipeline in Section[4\.2](https://arxiv.org/html/2610.38220#S4.SS2)operates on the criterion set\. Appendix[F\.4](https://arxiv.org/html/2610.38220#A6.SS4)presents the Criteria Principles guide\.

In our rubrics, each criterion contains:criterion text, a natural\-language requirement that is straightforward to objectively judge as pass or fail; animportance label, which determines itsweightin the aggregate score \(Eq\.[1](https://arxiv.org/html/2610.38220#S3.E1)\); and acategory, such as Factual Accuracy, Reasoning, or Safety\. Categories describe the dimension of the response being assessed but do not affect its score according to the rubric\. Our rubric generator assigns each criterion an importance label: Critical, Major, or Minor, for which we assign relative importance weights of 5, 3, and 1, respectively\([Gunjal et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib13);[Lau et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib3);[Lommel et al\., 2024](https://arxiv.org/html/2610.38220#bib.bib5);[Rao and Callison\-Burch, 2026](https://arxiv.org/html/2610.38220#bib.bib4)\)\.

### 4\.2The Rubricon Pipeline

Rubricon generates and refines rubrics through three stages:Generate\-and\-Select,Add, andPrune\(Figure[1](https://arxiv.org/html/2610.38220#S3.F1)\)\. Criteria generated throughout the pipeline follow the Criteria Principles in Section[4\.1](https://arxiv.org/html/2610.38220#S4.SS1)\. The three pipeline stages address the rubric\-level desiderata in Section[3](https://arxiv.org/html/2610.38220#S3): Generate\-and\-Select targets discrimination, Add expands coverage and helps distinguish among stronger responses, and Prune removes criteria that reward wrong qualities or lead to noisy scores\.

Before running Rubricon to produce a rubric for a prompt, one first generates a set of example responses to the prompt, some from a known*stronger*model \(e\.g\. GPT\-6\) and some from a known*weaker*model \(e\.g\. GPT\-4\)\. While one cannot guarantee that the weaker model’s response is not better than the stronger model’s, appropriately choosing these LLMs makes this unlikely\.

##### Generate\-and\-Select\.

We generateK=5K=5candidate rubrics through independent LLM calls for which the input is the task prompt, example responses \(from only the strong\-model\), and any task\-provided reference context from experts \(Appendix[F\.1](https://arxiv.org/html/2610.38220#A6.SS1)\)\. Generating multiple rubric candidates broadens coverage and reduces reliance on any single draft, while retaining alternative criteria for the subsequent Add stage\. To choose the best initial draft rubric among these candidates, we assess how well their criteria distinguish stronger\- from weaker\-model responses\.

A criterion satisfied by every example response provides no discrimination among those responses, whereas one that weaker\-model responses pass more often than stronger\-model responses suggests misalignment\. Following[Shen et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib7), we quantify this using the*strong–weak edge*: the difference in a criterion’s pass rates between stronger\- and weaker\-model responses, for which we first run the verifier to score these example responses against the draft rubric criteria\.

LetYsY\_\{s\}andYwY\_\{w\}denote the stronger\- and weaker\-model response sets, respectively, and letv¯​\(c,y\)\\bar\{v\}\(c,y\)denote the verifier pass rate overnnjudgments for criterionccagainst responseyy\. We estimate the strong–weak edge as

e⁡\(c\)=1\|Ys\|​∑y∈Ysv¯​\(c,y\)−1\|Yw\|​∑y∈Ywv¯​\(c,y\)e\(c\)=\\frac\{1\}\{\|Y\_\{s\}\|\}\\sum\_\{y\\in Y\_\{s\}\}\\bar\{v\}\(c,y\)\-\\frac\{1\}\{\|Y\_\{w\}\|\}\\sum\_\{y\\in Y\_\{w\}\}\\bar\{v\}\(c,y\)\(2\)
A positive edge indicates that stronger\-model responses satisfy the criterion more often, suggesting it discriminates in the right direction\.

We rely on these edge estimates in both rubric candidate selection and criterion filtering stages\. In the selection stage, Rubricon retains the candidate rubric with the highest average strong–weak edge, where this average is computed over all criteria in the rubric\. The criteria in the unselected rubrics are considered a candidate pool for the subsequent Add stage\.

##### Add missing coverage\.

Even the selected candidate rubric remains unlikely to satisfy all desiderata in Section[3](https://arxiv.org/html/2610.38220#S3)\. To expand coverage and target headroom, Rubricon refines this draft rubric through four complementary addition stages, applied sequentially so that each operates on the rubric augmented by the preceding stages:

1. 1\.Gap fillretrieves non\-redundant criteria from the unselected candidate rubrics to cover expectations missing from the current rubric \(Appendix[F\.2\.1](https://arxiv.org/html/2610.38220#A6.SS2.SSS1)\)\. It first filters the candidate criteria pool such that a criterion is ignored if every example response passes it in every verifier run, or if every strong\-model response fails it and every weak\-model response passes it in every verifier run\.
2. 2\.Prompt coveragechecks whether the rubric captures explicit task requirements\. The task’s prompt is decomposed into atomic requirements, each marked as covered, partially covered, or uncovered by the current rubric\. Uncovered or partially covered requirements are converted into new criteria \(Appendix[F\.2\.2](https://arxiv.org/html/2610.38220#A6.SS2.SSS2)\)\.
3. 3\.Negative probinggenerates adversarially\-weak responses \(from the same LLM\) that are known to be bad but that the current rubric may overlook\. A separate LLM reviewer, seeing only the task and one such response, flags concrete defects\. A subsequent LLM call converts relevant defects not already covered by the rubric into new criteria \(Appendix[F\.2\.3](https://arxiv.org/html/2610.38220#A6.SS2.SSS3)\)\.
4. 4\.Completeness probingsearches for omissions in an intended high\-quality response\. A model is prompted to generate a response that satisfies every criterion in the current rubric\. A separate LLM reviewer, seeing only the task prompt and response, flags missing or incomplete information that a high\-quality answer would still need\. A final LLM call converts relevant omissions not already covered by the rubric into new criteria \(Appendix[F\.2\.4](https://arxiv.org/html/2610.38220#A6.SS2.SSS4)\)\.

These four stages expand rubric coverage\. Newly generated criteria follow the Criteria Principles and receive importance labels as described in Section[4\.1](https://arxiv.org/html/2610.38220#S4.SS1)\. The model is instructed to add only task\-relevant criteria that are not already covered in the current rubric and are not overly specific to a particular solution approach\. We subsequently use the verifier to score the sample responses against each newly added criterion, allowing misaligned criteria to be identified and removed during pruning\.

##### Prune misleading criteria\.

Generation and expansion can introduce noisy criteria that degrade response scoring\. To support discrimination after expansion, we remove a criterion if its estimated strong–weak edge \(Eq\.[2](https://arxiv.org/html/2610.38220#S4.E2)\) is below a threshold \(−0\.10\-0\.10\), or if an LLM assessor flags it as satisfiable by an evasive or weak response \(Appendix[F\.3](https://arxiv.org/html/2610.38220#A6.SS3)\)\.

The remaining criteria form the refined rubric\. Appendix[E](https://arxiv.org/html/2610.38220#A5)provides examples showing how the Add and Prune stages update a specific rubric’s criteria\.

##### Iterative refinement\.

The Add and Prune stages can be repeatedly iterated\. We refer to one application of Add followed by Prune as a refinement round, with each subsequent round starting from the preceding round’s refined rubric\. Gap fill continues to draw from the original unselected drafts, while coverage checks and probes are rerun against the updated rubric\. The same strong\- and weak\-model responses are reused to score and prune criteria across rounds\. After a second refinement round, we consistently observe higher alignment with expert\-authored rubrics \(Section[6\.3](https://arxiv.org/html/2610.38220#S6.SS3)\)\.

## 5Experimental Setup

### 5\.1Assessing Rubric Quality

We assess the quality of each generated rubric by how well the resulting rubric scores align with expert human judgments of the same responses\.

##### Alignment with expert\-authored rubrics\.

Human\-written criteria describe what experts expect from a good response and can rank responses by how well they meet those expectations\. Agreement with these rankings tests whether generated rubrics capture similar quality distinctions, even if the rubrics assign different numerical scores\. To quantify this alignment, we use the verifier to score a set of*assessment responses*from many models against both auto\-generated and expert\-authored rubrics, then rank the responses according to each rubric, and finally report Spearman’s rank correlation \(ρ\\rho\) between these two rankings\. The overall alignment for a dataset is the averageρ\\rhoacross all of its tasks\.

##### Alignment with human preferences\.

Human pairwise preferences between two responses provide an alternative measure of quality, albeit often coarser/noisier than rubric\-based scoring\([Rezaei et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib8);[Li et al\., 2024](https://arxiv.org/html/2610.38220#bib.bib9)\)\. Alignment with these preferences indicates how well rubric scores reflect human assessments of relative response quality\. We reportselective preference accuracy: after scoring all responses against the generated rubric, we retain response\-pairs whose normalized rubric scores differ by more than a fixed margin \(δ=0\.10\\delta=0\.10\) and measure how often the higher\-scoring response matches the human preference\. We use this margin to avoid treating small score differences that may arise from verifier variability as meaningful preferences\.

### 5\.2Verifier Scoring

Our verifierv⁡\(c,y\)∈\{0,1\}v\(c,y\)\\in\\\{0,1\\\}is an LLM that judges whether a responseyysatisfies a criterioncc\. As in[Wang et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib19), the verifier receives only the response and a single criterion, without the original task prompt, under the assumption that criteria are self\-contained and can be judged independently\. To account for variability in its judgments, we evaluate each response\-criterion pairnntimes using the prompt in Appendix[B\.3](https://arxiv.org/html/2610.38220#A2.SS3), denoting the verdict from thejj\-th evaluation byvj​\(c,y\)v\_\{j\}\(c,y\)\. Appendix[B](https://arxiv.org/html/2610.38220#A2)shows that our verifier exhibits high agreement with expert judgments when grading the same responses against the same rubrics\.

##### Verifier scoring within the Rubricon pipeline\.

For rubric selection and pruning, we sample five responses from a stronger model and three from a weaker model per task\. Each response\-criterion pair receivesn=3n=3verifier evaluations\. We average all verifier judgments of each model’s example responses to obtain its pass rate for a criterion, then subtract the weaker model’s pass rate from the stronger model’s to estimate the strong–weak edge defined in Eq\.[2](https://arxiv.org/html/2610.38220#S4.E2)\.

##### Verifier scoring for rubric\-quality assessment\.

For assessing rubric quality, we apply the verifier to completed rubrics and fixed responses separate from those used during rubric construction\. The verifier producesn=3n=3judgments for each response\-criterion pair\. We aggregate thejj\-th judgment across all criteria to obtain a response score for scoring runjj, using Eq\.[1](https://arxiv.org/html/2610.38220#S3.E1)\.

Foralignment with expert\-authored rubrics, we use both automatically\-generated and human\-written rubrics to score the same set ofassessment responsesfrom various LLMs\. Within each scoring run, we computeρ\\rhobetween the two sets of response scores, yieldingnncorrelation estimates per task\. We report the mean correlation across all tasks and scoring runs within each benchmark\. Forselective preference accuracy, we apply the margin to each of thennscore comparisons for a task, then pool the retained comparisons across all tasks to compute accuracy\.

### 5\.3Datasets

We evaluate the generated rubrics on five benchmarks spanning professional, consumer, clinical, and general\-purpose tasks \(Table[3](https://arxiv.org/html/2610.38220#A1.T3)\)\. We use expert\-authored rubrics from ProfBench\([Wang et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib19)\), PRBench\([Akyürek et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib20)\), ACE\([Benchek et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib21)\), and HealthBench\([Arora et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib18)\)to measure alignment between rankings of different LLM responses produced by generated v\.s\. expert\-authored rubrics\. We use human preference labels from RubricBench\([Zhou et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib23)\)to measure human preference agreement\. Dataset descriptions, preprocessing, reference material, and final task counts are detailed in Appendix[A](https://arxiv.org/html/2610.38220#A1)\.

### 5\.4Baselines

We compare Rubricon against two families of baselines described in Section[2](https://arxiv.org/html/2610.38220#S2): prompt\-based methods, including an Agentic baseline \(Appendix[G](https://arxiv.org/html/2610.38220#A7)\) and RRDWU\([Shen et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib7)\), and post\-trained rubric generators, including the Query\-Specific Rubric Generator \(QSRG\)\([Lv et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib22)\)and OpenRubrics\([Liu et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib14)\)\. Rubricon belongs to the prompt\-based family\. Each method produces one rubric per task, and across all five benchmarks, responses are graded against these rubrics using the same verifier andn=3n=3replicate scoring runs \(Section[5\.2](https://arxiv.org/html/2610.38220#S5.SS2)\)\.

The*Agentic*baseline runs in a Codex SDK\([OpenAI, 2026](https://arxiv.org/html/2610.38220#bib.bib24)\)harness with full local tool access and web access disabled, freely choosing its rubric\-construction procedure\. It tests whether an unconstrained agent can match Rubricon’s structured pipeline\. Appendix[G](https://arxiv.org/html/2610.38220#A7)provides details\. Both the Agentic and RRD baselines receive the same task prompt, example model responses, and available reference material as Rubricon \(Section[5\.3](https://arxiv.org/html/2610.38220#S5.SS3)\)\.

The post\-trained rubric generators, OpenRubrics and QSRG, are run as published and receive the task prompt alone\. We also examine the effect of input context through QSRG\+Context, which uses the same trained generator as query\-only QSRG, but augments the task query with example model responses and any dataset\-provided reference material \(Appendix[A](https://arxiv.org/html/2610.38220#A1)\)\. The post\-trained rubric generators use their respective models: QSRG and QSRG\+Context use a Qwen3\-30B\-A3B\-based generator, while OpenRubrics uses Rubric\-RM\-8B, based on Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib28)\)\.

### 5\.5Models Employed

For experiments measuring alignment with human rubrics and human preferences, all prompt\-based rubric generation methods, including Rubricon, are powered by the same generator model: GPT\-5\.1\.

For rubric generation methods that leverage example LLM responses during rubric construction, we use responses from GPT\-5\.1 and Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.38220#bib.bib27)\)as the*strong*and*weak*models, respectively\. We use GPT\-5 mini as the verifier model for criterion grading throughout, because this model was observed to provide a favorable accuracy\-cost tradeoff, with strong agreement with human expert judgments on the same responses and rubrics \(Appendix[B\.1](https://arxiv.org/html/2610.38220#A2.SS1)\)\.

For our experiments measuring alignment with human rubrics, we use generated rubrics to score a separate set of responses from the following models: GPT\-5\.2 Instant, GPT\-5\.2 Thinking, Claude Sonnet 4\.6, Claude Opus 4\.6, Gemini 2\.0 Flash\-Lite, Gemini 2\.5 Flash, Gemini 3 Flash, and Gemini 3\.1 Pro\. We generate one response from each model per task, yielding eight responses to score with each rubric \(not used during rubric construction\)\. Appendix[C\.4](https://arxiv.org/html/2610.38220#A3.SS4)reports inference settings\.

Some of these responses come from newer/stronger models than those used for rubric generation and verification\. This reflects the fact that good rubrics must remain useful for evaluating responses from future models, which remains underexplored in prior work\.[Shen et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib7),[Xu et al\. \(2026b\)](https://arxiv.org/html/2610.38220#bib.bib16), and[Zhang et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib2)evaluate their generated rubrics mainly on responses from models weaker than or comparable to the rubric generator, although some of them leverage responses from stronger models during rubric construction\.

## 6Results for LLM Evaluation

Unless otherwise specified, Rubricon results are for the rubric obtained after two refinement rounds\.

### 6\.1Alignment with Human Rubrics

Rubricon achieves the highestρ\\rhoon all four benchmarks in Table[1](https://arxiv.org/html/2610.38220#S6.T1)\. It attains a macro\-averageρ\\rhoof0\.4280\.428across these datasets, representing a9\.0%9\.0\\%relative improvement over RRDWU, the strongest alternative approach\. On ACE, Rubricon exceeds the next\-best method by0\.0520\.052, its largest margin across the four benchmarks\. Additional context improves QSRG on three benchmarks, but overall both post\-trained rubric generators produce substantially less expert\-aligned rubrics than Rubricon\.

MethodProfBenchACEPRBenchHealthBenchRubricBenchMetricρ\\rhoρ\\rhoρ\\rhoρ\\rhoAccuracy \(%\)QSRG0\.361±0\.0820\.361\\pm 0\.0820\.225±0\.0430\.225\\pm 0\.0430\.226±0\.0130\.226\\pm 0\.0130\.172±0\.0080\.172\\pm 0\.00852\.9±1\.652\.9\\pm 1\.6QSRG\+Context0\.267±0\.0700\.267\\pm 0\.0700\.301±0\.0380\.301\\pm 0\.0380\.353±0\.0120\.353\\pm 0\.0120\.279±0\.0090\.279\\pm 0\.00952\.4±1\.652\.4\\pm 1\.6OpenRubrics0\.202±0\.0680\.202\\pm 0\.0680\.276±0\.0380\.276\\pm 0\.0380\.071±0\.0140\.071\\pm 0\.0140\.119±0\.0100\.119\\pm 0\.01057\.5±1\.757\.5\\pm 1\.7Agentic0\.283±0\.0630\.283\\pm 0\.0630\.333±0\.0390\.333\\pm 0\.0390\.363±0\.0140\.363\\pm 0\.0140\.294±0\.0090\.294\\pm 0\.00959\.3±1\.959\.3\\pm 1\.9RRDWU0\.394±0\.0760\.394\\pm 0\.0760\.316±0\.0400\.316\\pm 0\.0400\.497±0\.0110\.497\\pm 0\.0110\.365±0\.0090\.365\\pm 0\.00958\.7±1\.758\.7\\pm 1\.7Rubricon0\.424±0\.083\\mathbf\{0\.424\\pm 0\.083\}0\.385±0\.035\\mathbf\{0\.385\\pm 0\.035\}0\.516±0\.011\\mathbf\{0\.516\\pm 0\.011\}0\.388±0\.009\\mathbf\{0\.388\\pm 0\.009\}59\.8±1\.9\\mathbf\{59\.8\\pm 1\.9\}

Table 1:Alignment with human rubrics and human preferences\.Rubricon achieves the highest observed scores across all five benchmarks\. Across datasets, it improves macro\-averageρ\\rhoby 9\.0% relative to the strongest baseline, RRDWU\(0\.4280\.428vs\.0\.3930\.393\), and selective preference accuracy by 0\.5 percentage points over the strongest preference baseline, Agentic \(59\.8%59\.8\\%vs\.59\.3%59\.3\\%\)\. The first four benchmarks reportρ\\rhobetween scores under generated and expert\-authored rubrics\. RubricBench reports selective preference accuracy \(%\), evaluated on response pairs whose normalized rubric scores differ by more than0\.100\.10\(retained counts in Table[7](https://arxiv.org/html/2610.38220#A3.T7)\)\. The±\\pmvalues indicate standard errors estimated by bootstrapping tasks \(B=500B=500resamples\)\.
### 6\.2Alignment with Human Preference

Rubricon achieves the highest observed selective preference accuracy, exceeding the next\-best method, the Agentic baseline, by0\.50\.5percentage points \(rightmost column in Table[1](https://arxiv.org/html/2610.38220#S6.T1)\)\. Note that the alignment with human preferences is fairly low for all methods, because[Zhou et al\. \(2026\)](https://arxiv.org/html/2610.38220#bib.bib23)purposefully populated RubricBench with challenging examples\.

### 6\.3Iterative Refinement Across Multiple Rounds

Rubricon is an iterative pipeline: the Add and Prune stages can be reapplied to refine the rubric produced by the preceding round\. Table[8](https://arxiv.org/html/2610.38220#A3.T8)shows that the first refinement round yields an11\.9%11\.9\\%relative improvement in macro\-averageρ\\rhoover the selected initial rubric\. The second round yields a further5\.1%5\.1\\%relative improvement over the first round\. Compared with RRDWU, the relative improvements after one and two rounds are3\.7%3\.7\\%and9\.0%9\.0\\%, respectively\.

### 6\.4Performance Across Refinement Steps

We examine the contribution of each pipeline stage by assessing the quality of intermediate rubrics produced during the first refinement round\. Table[2](https://arxiv.org/html/2610.38220#S6.T2)summarizes results across the four benchmarks with expert\-authored rubrics\. Detailed per\-benchmark results appear in Table[9](https://arxiv.org/html/2610.38220#A3.T9)\. The results show that each stage of the Rubricon pipeline is valuable: every stage improves alignment on at least one benchmark, and most stages also improve the macro\-averageρ\\rhoacross the four benchmarks\. These results suggest that different stages are useful for different types of tasks, which could differ substantially in their expert\-authored rubrics and what constitutes a good response\.

Some steps in the Add stage slightly reduce macro\-averageρ\\rhowhile improving this alignment with expert\-rubrics on individual benchmarks\. Expanding rubric coverage can introduce noisy or misleading criteria alongside useful ones\. Subsequent pruning is designed to remove these problematic criteria and proves empirically effective overall\. The improvement in alignment after pruning suggests that coverage expansion and filtering play complementary roles, even when gains are not observed immediately after the Add step\.

StageMacro\-avg\.ρ\\rhoAvg\. RubricSizeGeneration0\.35920\.6Selected Rubric0\.36420\.6Gap Fill0\.36223\.9Prompt Coverage0\.36224\.6Negative Probing0\.38828\.3Completeness Probing0\.39531\.0Prune0\.40826\.7Table 2:Alignment with expert\-authored rubrics \(macro\-averagedρ\\rhoacross datasets\) and average rubric size after each stage of the first refinement round\. Rows show cumulative refinement\. Values in the top row are averaged overK=5K=5independently generated rubrics\.

## 7Using Rubrics for Post\-Training

Beyond model evaluation, rubrics are also used to compute rewards for post\-training LLMs via reinforcement learning\. Here we assess how effective Rubricon rubrics are for RL post\-training of the Qwen3\.5\-9B model\([Qwen Team, 2026](https://arxiv.org/html/2610.38220#bib.bib29)\)\. Our experiment trains the same model using the same recipe against rubrics from different generators, and then measures the quality of each resulting model based on how it scores against expert\-authored rubrics for held\-out tasks\.

##### Data\.

We post\-train models on the HealthBench dataset, randomly splitting its 1,704 prompts into 1,204 training prompts and 500 held\-out evaluation prompts\.

##### Training conditions\.

We compare two training\-time reward sources: RRDWU, our strongest baseline for alignment with expert\-authored rubrics, and Rubricon rubrics obtained after two refinement rounds\. We fine\-tune the released Qwen3\.5\-9B model checkpoint with GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.38220#bib.bib25)\)for eight epochs\. Training configurations are detailed in Appendix[C\.5](https://arxiv.org/html/2610.38220#A3.SS5), Table[11](https://arxiv.org/html/2610.38220#A3.T11)\. Apart from the training\-time rubrics, the model, training recipe, data split, and evaluation procedure are held fixed\.

##### Held\-out policy evaluation\.

At each evaluated model checkpoint, we sample three independent responses per held\-out prompt and score them against physician\-authored HealthBench criteria \(using the same verifier\)\. We reportavg@3, the average of the \(expert\-authored\) rubric scores of these three independently sampled responses, subsequently averaged across all held\-out evaluation prompts\. We also report paired 95% bootstrap confidence intervals for the difference in mean scores between models trained using different rubrics\.

![Refer to caption](https://arxiv.org/html/2610.38220v1/artifacts/figs/healthbench_posttraining.png)Figure 2:Rubricon rewards translate into stronger held\-out post\-training performance\.Qwen3\.5\-9B trained with Rubricon rubric rewards outperforms the same model trained with RRDWUrewards at every evaluated post\-training checkpoint, reaching a 4\.5% higher score at the last evaluated checkpoint\. Scores are averaged over three independently sampled responses per prompt, then across 500 held\-out HealthBench prompts\. The Rubricon\-trained model achieves a 20\.5% higher held\-out score than the starting model checkpoint\.
##### Results\.

Under the fixed recipe, training with Rubricon rubrics yields higher held\-out HealthBench scores than training with RRDWUrubrics at every evaluated checkpoint \(Figure[2](https://arxiv.org/html/2610.38220#S7.F2)\)\. The final\-checkpoint improvement is0\.03150\.0315\(paired 95% bootstrap CI:\[\+0\.0207,\+0\.0427\]\[\+0\.0207,\+0\.0427\]\)\. Here we use Gemini 3\.5 Flash\-Lite as the verifier model for both rollout scoring during training and held\-out evaluation graded against expert\-authored rubrics\. The advantage persists when the same final\-checkpoint responses are re\-scored by GPT\-5 mini or Claude Sonnet 5 as verifiers instead \(Appendix[B\.2](https://arxiv.org/html/2610.38220#A2.SS2)\)\. These results support Rubricon’s efficacy for providing reward signals in post\-training\.

## 8Discussion

Our results show that rubric quality can be automatically improved beyond basic LLM or agentic generation, without requiring additional data collection/annotation or model fine\-tuning\. By systematically selecting discriminative drafts, filling gaps in coverage, and removing misleading criteria, Rubricon produces more effective rubrics for LLM evaluation and training\. The key is a methodical test\-time scaling strategy that does not purely rely on LLM self\-critique and ensembling, but additionally grounds refinement stages in how draft rubrics actually score example responses\.

## Limitations

Our experiments are limited to single\-turn LLM responses in certain domains and to the English language\. It remains unclear whether the rubrics generated in these experiments will continue to distinguish response quality as models continue to improve beyond the most advanced LLMs considered in our experiments\. As models are increasingly trained as agentic and multi\-agent systems, an agentic or multi\-agent approach to rubric generation may one day surpass Rubricon\.

Rubricon relies heavily on model\-derived signals and the resulting automatically generated rubrics are far from perfect\. Because it is compatible with any LLM, Rubricon rubrics will improve as models improve, but the flaws in model responses will also get subtler\. In these non\-verifiable domains where quality judgments are driven by expert taste, one cannot rely on theoretical guarantees because what constitutes a good rubric can greatly vary from one dataset to another\.

Thus we recommend always having human experts review and improve the rubrics automatically generated by methods like Rubricon, which still yields enormous time\-savings compared to manually crafting rubrics from scratch\. A beneficial property of our method is that its modular stages are easy to adjust based on expert feedback, and a prompt\-based approach like ours can easily incorporate expert suggestions/conventions as well \(in contrast to a trained rubric generator\)\. Rubricon’s refinement stages can be seamlessly iterated with manual expert updates to rubrics, or even applied to human\-crafted initial rubrics\. Optimal human\-AI collaboration to construct LLM evaluations effectively and affordably remains an important topic for future research\.

## References

- Akyüreket al\.\(2026\)A\. F\. Akyürek, A\. Gosai, C\. B\. C\. Zhang, V\. Gupta, J\. Jeong, A\. Gunjal, T\. Rabbani, M\. Mazzone, D\. Randolph, M\. M\. Meymand, G\. Chattha, P\. Rodriguez, D\. A\. M\. Buendia, P\. Singh, M\. Liu, S\. Chawla, P\. Cline, L\. Ogaz, E\. G\. H\. Montoya, Z\. Wang, P\. Bhatter, M\. Ayestaran, B\. Liu, and Y\. HePRBench: large\-scale expert rubrics for evaluating high\-stakes professional reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,Cited by:[§A\.1](https://arxiv.org/html/2610.38220#A1.SS1.SSS0.Px1.p2.1),[§5\.3](https://arxiv.org/html/2610.38220#S5.SS3.p1.1)\.
- Aroraet al\.\(2025\)R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel,et al\.HealthBench: evaluating large language models towards improved human health\.arXiv preprint arXiv:2505\.08775\.Cited by:[§A\.1](https://arxiv.org/html/2610.38220#A1.SS1.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§5\.3](https://arxiv.org/html/2610.38220#S5.SS3.p1.1)\.
- Bencheket al\.\(2025\)J\. Benchek, R\. Shetty, B\. Hunsberger, A\. Arun, Z\. Richards, B\. Foody, O\. Nitski, and B\. VidgenThe AI Consumer Index \(ACE\)\.arXiv preprint arXiv:2512\.04921\.Cited by:[§A\.1](https://arxiv.org/html/2610.38220#A1.SS1.SSS0.Px1.p2.1),[§5\.3](https://arxiv.org/html/2610.38220#S5.SS3.p1.1)\.
- Dhole and Agichtein \(2026\)K\. D\. Dhole and E\. AgichteinRubricRAG: towards interpretable and reliable LLM evaluation via domain knowledge retrieval for rubric generation\.arXiv preprint arXiv:2603\.20882\.Cited by:[§D\.3](https://arxiv.org/html/2610.38220#A4.SS3.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.5](https://arxiv.org/html/2610.38220#S5.SS5.p2.1)\.
- Gunjalet al\.\(2026\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.38220#S4.SS1.p2.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[Table 11](https://arxiv.org/html/2610.38220#A3.T11.3.8.2.1.1)\.
- Huanget al\.\(2025\)Z\. Huang, Y\. Zhuang, G\. Lu, Z\. Qin, H\. Xu, T\. Zhao, R\. Peng, J\. Hu, Z\. Shen, X\. Hu,et al\.Reinforcement learning with rubric anchors\.arXiv preprint arXiv:2508\.12790\.Cited by:[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.p1.1)\.
- Lauet al\.\(2026\)E\. Lau, M\. Dücker, R\. Chaudhary, H\. W\. Goh, R\. Wei, V\. Kumar, S\. Qunbar, G\. Gogia, Y\. Liu, S\. Millslagle, N\. Borazjanizadeh, U\. Tkachenko, S\. E\. Danquah, C\. Schweiker, V\. Karumathil, A\. Devalaraju, V\. Sandadi, H\. Nam, P\. Arani, R\. Epps, A\. Arif, S\. Bhaiwala, C\. Northcutt, S\. Wang, A\. Athalye, J\. Mueller, and F\. GuzmánBankerToolBench: evaluating AI agents in end\-to\-end investment banking workflows\.arXiv preprint arXiv:2604\.11304\.Cited by:[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.38220#S4.SS1.p2.1)\.
- Liet al\.\(2024\)J\. Li, F\. Zhou, S\. Sun, Y\. Zhang, H\. Zhao, and P\. LiuDissecting human and LLM preferences\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Cited by:[§5\.1](https://arxiv.org/html/2610.38220#S5.SS1.SSS0.Px2.p1.1)\.
- Liet al\.\(2026\)S\. Li, J\. Zhao, H\. Ren, Z\. Wei, Y\. Zhou, J\. Yang, S\. Liu, K\. Zhang, and W\. ChenRubricHub: a comprehensive and highly discriminative rubric dataset via automated coarse\-to\-fine generation\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2026\)T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. WangOpenRubrics: towards scalable synthetic rubric generation for reward modeling and LLM alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,Cited by:[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px3.p1.1),[§5\.4](https://arxiv.org/html/2610.38220#S5.SS4.p1.1)\.
- Lommelet al\.\(2024\)A\. Lommel, S\. Gladkoff, A\. Melby, S\. E\. Wright, I\. Strandvik, K\. Gasova, A\. Vaasa, A\. Benzo, R\. Marazzato Sparano, M\. Foresi,et al\.The multi\-range theory of translation quality measurement: MQM scoring models and statistical quality control\.InProceedings of the 16th Conference of the Association for Machine Translation in the Americas,Cited by:[§4\.1](https://arxiv.org/html/2610.38220#S4.SS1.p2.1)\.
- Lvet al\.\(2026\)C\. Lv, J\. Zhou, W\. Zhao, J\. Xu, S\. Dou, Z\. Huang, M\. Tian, X\. Wang, Z\. Guo, Y\. Liu, P\. Zhou, T\. Gui, L\. Tian, X\. Zhou, X\. Zheng, X\. Huang, and J\. ZhouLearning query\-specific rubrics from human preferences for DeepResearch report generation\.arXiv preprint arXiv:2602\.03619\.Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px3.p1.1),[§5\.4](https://arxiv.org/html/2610.38220#S5.SS4.p1.1)\.
- OpenAI \(2026\)OpenAIOpenAI Codex Python SDK\.Note:[https://github\.com/openai/codex/tree/main/sdk/python](https://github.com/openai/codex/tree/main/sdk/python)openai\-codexPython packageCited by:[Appendix G](https://arxiv.org/html/2610.38220#A7.p1.1),[§5\.4](https://arxiv.org/html/2610.38220#S5.SS4.p2.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§7](https://arxiv.org/html/2610.38220#S7.p1.1)\.
- Rao and Callison\-Burch \(2026\)D\. Rao and C\. Callison\-BurchAutorubric: a unified framework for rubric\-based LLM evaluation\.arXiv preprint arXiv:2603\.00077\.Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2610.38220#S4.SS1.p2.1)\.
- Rezaeiet al\.\(2026\)M\. Rezaei, R\. Vacareanu, Z\. Wang, C\. Wang, B\. Liu, Y\. He, and A\. F\. AkyürekOnline rubrics elicitation from pairwise comparisons\.InInternational Conference on Machine Learning,Cited by:[§5\.1](https://arxiv.org/html/2610.38220#S5.SS1.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§7](https://arxiv.org/html/2610.38220#S7.SS0.SSS0.Px2.p1.1)\.
- Shenet al\.\(2026\)W\. F\. Shen, X\. Qiu, C\. Whitehouse, L\. Alazraki, S\. Goel, F\. Barbieri, T\. Willi, A\. Mathur, and I\. LeontiadisRethinking rubric generation for improving LLM judge and reward modeling for open\-ended tasks\.arXiv preprint arXiv:2602\.05125\.Cited by:[§A\.1](https://arxiv.org/html/2610.38220#A1.SS1.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§1](https://arxiv.org/html/2610.38220#S1.p3.1),[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.p1.1),[§4\.2](https://arxiv.org/html/2610.38220#S4.SS2.SSS0.Px1.p2.1),[§4](https://arxiv.org/html/2610.38220#S4.p1.1),[§5\.4](https://arxiv.org/html/2610.38220#S5.SS4.p1.1),[§5\.5](https://arxiv.org/html/2610.38220#S5.SS5.p4.1)\.
- Shenget al\.\(2026\)L\. Sheng, W\. Ma, R\. Hong, X\. Wang, A\. Zhang, and T\. ChuaReinforcing chain\-of\-thought reasoning with self\-evolving rubrics\.arXiv preprint arXiv:2602\.10885\.Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px3.p1.1)\.
- Snellet al\.\(2025\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling parameters for reasoning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.38220#S1.p3.1)\.
- Wanget al\.\(2026\)Z\. Wang, J\. Jung, X\. Lu, S\. Diao, E\. Evans, J\. Zeng, P\. Molchanov, Y\. Choi, J\. Kautz, and Y\. DongProfBench: multi\-domain rubrics requiring professional knowledge to answer and judge\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2610.38220#A1.SS1.SSS0.Px1.p2.1),[§B\.1](https://arxiv.org/html/2610.38220#A2.SS1.p1.1),[§B\.1](https://arxiv.org/html/2610.38220#A2.SS1.p2.1),[§5\.2](https://arxiv.org/html/2610.38220#S5.SS2.p1.1),[§5\.3](https://arxiv.org/html/2610.38220#S5.SS3.p1.1)\.
- Xuet al\.\(2026a\)R\. Xu, T\. Liu, Z\. Dong, T\. Yu, I\. Hong, C\. Yang, L\. Zhang, T\. Zhao, and H\. WangAlternating reinforcement learning for rubric\-based reward modeling in non\-verifiable LLM post\-training\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 141450–141468\.Cited by:[§1](https://arxiv.org/html/2610.38220#S1.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.p1.1)\.
- Xuet al\.\(2026b\)Y\. Xu, G\. Potje, S\. Shandilya, T\. Yuan, L\. d\. O\. Nunes, R\. Agarwal, S\. Asgari, A\. Atkinson, E\. Kıcıman, S\. Lu,et al\.SibylSense: adaptive rubric learning via memory tuning and adversarial probing\.arXiv preprint arXiv:2602\.20751\.Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px2.p1.1),[§5\.5](https://arxiv.org/html/2610.38220#S5.SS5.p4.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.4](https://arxiv.org/html/2610.38220#S5.SS4.p3.1)\.
- Zhanget al\.\(2026\)J\. Zhang, Z\. Wang, L\. Gui, S\. M\. Sathyendra, J\. Jeong, V\. Veitch, W\. Wang, Y\. He, B\. Liu, and L\. JinChasing the tail: effective rubric\-based reward modeling for large language model post\-training\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px2.p1.1),[§5\.5](https://arxiv.org/html/2610.38220#S5.SS5.p4.1)\.
- Zhouet al\.\(2026\)J\. Zhou, Q\. Zhang, Y\. Wang, F\. Lyu, Y\. Ming, C\. Xu, Q\. Sun, K\. Zheng, P\. Kang, X\. Liu, and C\. MaRubricBench: aligning model\-generated rubrics with human standards\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,Cited by:[§A\.1](https://arxiv.org/html/2610.38220#A1.SS1.SSS0.Px2.p1.1),[Table 7](https://arxiv.org/html/2610.38220#A3.T7),[§5\.3](https://arxiv.org/html/2610.38220#S5.SS3.p1.1),[§6\.2](https://arxiv.org/html/2610.38220#S6.SS2.p1.1)\.
- Zhou and Tan \(2026\)K\. Zhou and C\. TanAutoChecklist: composable pipelines for checklist generation and scoring with LLM\-as\-a\-judge\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2610.38220#S2.SS0.SSS0.Px1.p1.1)\.

Appendix

## Appendix ABenchmark Dataset Details

### A\.1Benchmarks for Assessing Rubric Quality

##### Benchmarks with human rubrics\.

To ensure they are high\-quality, we first reviewed many expert\-authored rubrics in every dataset used in our experiments on alignment against expert rubrics\. Many other datasets were omitted from consideration during these reviews\.

We use the Finance, Consulting, and Physics domains of ProfBench\([Wang et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib19)\)and the Finance and Law tasks of PRBench\([Akyürek et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib20)\)\. Both benchmarks contain tasks requiring professional knowledge and reasoning\. We also use the open development split of the AI Consumer Index \(ACE\)\([Benchek et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib21)\), which covers everyday consumer tasks in Shopping, DIY, Gaming, and Food, and HealthBench\([Arora et al\., 2025](https://arxiv.org/html/2610.38220#bib.bib18)\), which contains health conversations with physician\-authored, importance\-weighted rubrics\. On these four benchmarks, we score the same responses using generated and expert\-authored rubrics and report rank correlations as defined in Section[5\.1](https://arxiv.org/html/2610.38220#S5.SS1)\. HealthBench is also used for post\-training experiments because of its size and prior work using generated rubrics as reinforcement\-learning rewards\([Shen et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib7)\)\.

##### Benchmark with human preferences\.

We use the released RubricBench preference set\([Zhou et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib23)\), comprising 1,147 pairwise preference examples drawn from datasets like HelpSteer3, PPE, and RewardBench2\. Each example contains a task, two LLM responses, and a binary human preference label\. We score each response independently using the generated rubric and report selective preference accuracy defined in Section[5\.1](https://arxiv.org/html/2610.38220#S5.SS1)\.

### A\.2Reference Material for Rubric Construction

Rubricon, Agentic, RRDWU, and QSRG\+Context are provided with task prompts, example model responses, and any auxiliary per\-task reference material in each dataset \(see Appendix[F\.1](https://arxiv.org/html/2610.38220#A6.SS1)\)\. This reference material includes expert\-authored step\-by\-step reasoning for PRBench and expert\-authored example responses for HealthBench\. We treat the human\-written responses in HealthBench as example responses rather than golden reference completions, after finding that they do not score highly under its own expert\-authored rubrics\. No additional reference material is used for ProfBench, ACE, or RubricBench\. QSRG and OpenRubrics receive the task prompt alone\. The expert\-authored rubrics used for alignment\-evaluation are never provided to the rubric\-generators\.

### A\.3Data Preprocessing

We filter our datasets to the subset of examples for which we are more confident in the quality of expert\-authored rubrics\. We require at least 10 expert\-authored criteria for ProfBench, PRBench, and HealthBench to support graded, discriminative scoring and reduce ties\. We do not apply the 10\-criterion minimum to ACE, whose human\-authored rubrics contain 7 criteria on average\. We restrict PRBench and HealthBench to single\-turn tasks and require expert\-written responses as reference material for HealthBench\. We exclude ProfBench’s Chemistry domain after identifying issues in the expert\-authored rubrics, such as vague or hedged criteria and inconsistent numerical\-precision requirements\. Table[3](https://arxiv.org/html/2610.38220#A1.T3)reports final task counts and the reference material used for each dataset\.

BenchmarkDomain\(s\)Human annotationsTasksRef\. materialProfBenchFinance, Consulting, PhysicsHuman rubrics28—PRBenchFinance, LawHuman rubrics656expert step\-by\-step reasoningACEShopping, DIY, Gaming, FoodHuman rubrics80—HealthBenchHealthHuman rubrics1,704example responseRubricBenchChat, IF, STEM, Code, SafetyHuman preferences1,147—Table 3:Evaluation benchmarks\.Human annotationsspecifies how generated rubrics are assessed: alignment with scores from human\-authored rubrics or human preferences between responses\.Tasksgives the number of tasks evaluated after preprocessing\.Ref\. materiallists the inputs provided for each task in the rubric generation prompt \(see Appendix[F\.1](https://arxiv.org/html/2610.38220#A6.SS1)\)\.

## Appendix BDetails about the Verifier Model

### B\.1Agreement with Expert Human Graders

Cohen’sκ\\kappaF1Verifier ModelOur PromptProfBenchOur PromptProfBenchGPT\-5 mini0\.5810\.5490\.8020\.786GPT\-5\.20\.5600\.5560\.7770\.777Claude Sonnet 4\.60\.5560\.5330\.7850\.760Gemini 3\.1 Pro0\.5800\.5820\.7910\.794Table 4:Using our verifier prompt, GPT\-5 mini achieves the highest observed Cohen’sκ\\kappaand F1 among the tested verifiers\. We report criterion\-level agreement \(Cohen’sκ\\kappaand F1\) with expert annotations on ProfBench using our judge prompt and the original ProfBench judge prompt\. On a 367\-criterion subset that we independently re\-annotated, \(non\-expert\) human agreement with the ProfBench expert annotations isκ=0\.281\\kappa=0\.281\(64\.0% raw agreement\)\. The GPT\-5 mini verifier reachesκ=0\.455\\kappa=0\.455\(72\.5% raw agreement\) against the expert labels on the same subset\.Rubric\-based evaluation depends on the verifier’s ability to correctly judge whether responses satisfy each criterion\. To assess the quality of the verifier used in our experiments, we compare LLM verifier judgments with expert human annotations on over 3,400 response\-criterion pairs from ProfBench\([Wang et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib19)\)\. For each pair, the verifier is assessing the same response against the same expert\-authored criterion, with the purpose of measuring the criterion\-level grading accuracy, separate from the quality of generated rubrics\.

We evaluate four verifier models with two prompts: our verifier prompt \(Appendix[B\.3](https://arxiv.org/html/2610.38220#A2.SS3)\), which we use for all our experiments, and the verifier prompt used by ProfBench\([Wang et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib19)\)\. Our prompt generally yields higher agreement and F1 scores than the ProfBench prompt \(Table[4](https://arxiv.org/html/2610.38220#A2.T4)\)\. GPT\-5 mini achieves the best performance while remaining cost\-effective and highly stable\. Across 8 repeated runs, GPT\-5 mini’s judgments agree on 97\.2% of criteria \(mean pairwiseκ=0\.943\\kappa=0\.943\)\.

### B\.2Post\-training Results Across Verifiers

To further validate the post\-training results with alternative verifiers, we re\-scored the final\-checkpoint responses with GPT\-5 mini and Claude Sonnet 5 on the same physician\-authored HealthBench rubrics\. The results in Table[5](https://arxiv.org/html/2610.38220#A2.T5)confirm that Rubricon’s advantage over RRDWUholds across all three verifier models\.

Verifier ModelRRDWURubriconΔ±SEpaired\\Delta\\pm\\mathrm\{SE\}\_\{\\mathrm\{paired\}\}Gemini 3\.5 Flash\-Lite0\.70770\.73920\.0315±0\.00570\.0315\\pm 0\.0057GPT\-5 mini0\.64820\.67530\.0271±0\.00520\.0271\\pm 0\.0052Claude Sonnet 50\.53570\.57100\.0353±0\.00590\.0353\\pm 0\.0059Table 5:Verifier sensitivity on held\-out HealthBench evaluation dataset\. Each verifier scores the same saved responses from the two policies after eight training epochs against physician\-authored rubrics\.Δ\\Deltadenotes Rubricon minus RRDWU\. Standard errors are computed across paired prompt\-level differences\.
### B\.3Verifier Prompt

Verifier prompt[⬇](data:text/plain;base64,WW91IGFyZSBhbiBleHBlcnQganVkZ2UgZXZhbHVhdGluZyB3aGV0aGVyIGEgcmVzcG9uc2Ugc2F0aXNmaWVzIHRoZSBnaXZlbiBjcml0ZXJpb24uIElmIHRoZSByZXNwb25zZSBzYXRpc2ZpZXMgdGhlIGNyaXRlcmlvbiwgb3V0cHV0IFlFUzsgb3RoZXJ3aXNlIG91dHB1dCBOTy4KCkhlcmUgaXMgdGhlIHJlc3BvbnNlIHRvIGV2YWx1YXRlIGFuZCB0aGUgZXZhbHVhdGlvbiBjcml0ZXJpb246Cgo8UkVTUE9OU0U+CigqQFxwaHtce1x7IFJFU1BPTlNFIFx9XH19QCopCjwvUkVTUE9OU0U+Cgo8Q1JJVEVSSU9OPgooKkBccGh7XHtceyBDUklURVJJT04gXH1cfX1AKikKPC9DUklURVJJT04+CgpEbyBub3QgcmV0dXJuIGFueXRoaW5nIGVsc2UsIG9ubHkgYSBzaW5nbGUgd29yZDogWUVTIC8gTk8=)Youareanexpertjudgeevaluatingwhetheraresponsesatisfiesthegivencriterion\.Iftheresponsesatisfiesthecriterion,outputYES;otherwiseoutputNO\.Hereistheresponsetoevaluateandtheevaluationcriterion:<RESPONSE\>\{\{RESPONSE\}\}</RESPONSE\><CRITERION\>\{\{CRITERION\}\}</CRITERION\>Donotreturnanythingelse,onlyasingleword:YES/NO

## Appendix CAdditional Results and Analysis

### C\.1Rubric Sizes and Retained Human Preference Comparisons

Tables[6](https://arxiv.org/html/2610.38220#A3.T6)and[7](https://arxiv.org/html/2610.38220#A3.T7)complement the main results \(Table[1](https://arxiv.org/html/2610.38220#S6.T1)\) with average rubric sizes across methods and benchmarks\. Table[7](https://arxiv.org/html/2610.38220#A3.T7)also reports the number of retained comparisons for the RubricBench human\-preference evaluation\.

MethodProfBenchACEPRBenchHealthBenchExpert\-authored30\.57\.416\.614\.7QSRG22\.119\.520\.319\.9QSRG\+Context21\.220\.320\.620\.1OpenRubrics11\.89\.39\.37\.9Agentic33\.924\.328\.424\.1RRDWU26\.026\.725\.423\.4Rubricon41\.925\.038\.823\.4Table 6:Average number of criteria per rubric on the four datasets where we report alignment against human\-expert rubrics\. The average size of these expert\-authored rubrics is included for reference\.MethodRetainedcomparisonsAvg\. rubricsizeQSRG1,96620\.0QSRG\+Context2,04620\.1OpenRubrics2,1337\.2Agentic1,57620\.0RRDWU1,93820\.5Rubricon1,76017\.3Table 7:Number of retained comparisons and average rubric size on RubricBench\([Zhou et al\., 2026](https://arxiv.org/html/2610.38220#bib.bib23)\)\. We retain comparisons with an absolute rubric score difference above0\.100\.10\(Section[5\.1](https://arxiv.org/html/2610.38220#S5.SS1)\)\. Counts include three scoring runs per task, giving 3,441 comparisons across 1,147 tasks before filtering\.
### C\.2Detailed Performance Across Refinement Steps

Table[9](https://arxiv.org/html/2610.38220#A3.T9)provides per\-benchmarkρ\\rhoand rubric sizes after each step of the first refinement round, supplementing Section[6\.4](https://arxiv.org/html/2610.38220#S6.SS4)\. Gains vary across benchmarks, with no single stage consistently providing the largest improvement\.

StageProfBenchACEPRBenchHealthBenchSelected Rubric0\.345±0\.0740\.345\\pm 0\.0740\.352±0\.0370\.352\\pm 0\.0370\.421±0\.0120\.421\\pm 0\.0120\.339±0\.0090\.339\\pm 0\.009After Add Stage \(round 1\)0\.358±0\.0810\.358\\pm 0\.0810\.376±0\.0370\.376\\pm 0\.0370\.486±0\.0120\.486\\pm 0\.0120\.360±0\.0090\.360\\pm 0\.009After Prune Stage \(round 1\)0\.405±0\.0840\.405\\pm 0\.0840\.354±0\.0370\.354\\pm 0\.0370\.500±0\.0120\.500\\pm 0\.0120\.371±0\.0090\.371\\pm 0\.009After Add Stage \(round 2\)0\.374±0\.0860\.374\\pm 0\.0860\.354±0\.0360\.354\\pm 0\.0360\.500±0\.0120\.500\\pm 0\.0120\.372±0\.0090\.372\\pm 0\.009After Prune Stage \(round 2\)0\.424±0\.083\\mathbf\{0\.424\\pm 0\.083\}0\.385±0\.035\\mathbf\{0\.385\\pm 0\.035\}0\.516±0\.011\\mathbf\{0\.516\\pm 0\.011\}0\.388±0\.009\\mathbf\{0\.388\\pm 0\.009\}

Table 8:Iterative refinement improves alignment on every benchmark\. A second refinement round increases macro\-averageρ\\rhoby 17\.6% over the selected draft rubric\. Rows are cumulative, with each stage operating on the preceding rubric\. Results are reported asρ\\rho± task\-bootstrap standard error\.Method / stageProfBenchACEPRBenchHealthBenchρ\\rhoSizeρ\\rhoSizeρ\\rhoSizeρ\\rhoSizeSelected Rubric0\.345±0\.0740\.345\\pm 0\.07423\.60\.352±0\.0370\.352\\pm 0\.03719\.40\.421±0\.0120\.421\\pm 0\.01221\.30\.339±0\.0090\.339\\pm 0\.00917\.9Gap Fill0\.327±0\.0770\.327\\pm 0\.07727\.00\.364±0\.0390\.364\\pm 0\.03922\.80\.417±0\.0120\.417\\pm 0\.01224\.80\.340±0\.0090\.340\\pm 0\.00921\.0Prompt Coverage0\.323±0\.0800\.323\\pm 0\.08029\.00\.368±0\.0380\.368\\pm 0\.03823\.10\.417±0\.0130\.417\\pm 0\.01325\.00\.339±0\.0090\.339\\pm 0\.00921\.1Negative Probing0\.346±0\.0770\.346\\pm 0\.07733\.30\.379±0\.036\\mathbf\{0\.379\\pm 0\.036\}26\.00\.471±0\.0120\.471\\pm 0\.01230\.20\.356±0\.0090\.356\\pm 0\.00923\.5Completeness Probing0\.358±0\.0810\.358\\pm 0\.08137\.30\.376±0\.0370\.376\\pm 0\.03727\.50\.486±0\.0120\.486\\pm 0\.01234\.10\.360±0\.0090\.360\\pm 0\.00925\.0Prune0\.405±0\.084\\mathbf\{0\.405\\pm 0\.084\}32\.80\.354±0\.0370\.354\\pm 0\.03722\.00\.500±0\.012\\mathbf\{0\.500\\pm 0\.012\}31\.40\.371±0\.009\\mathbf\{0\.371\\pm 0\.009\}20\.5

Table 9:Alignment with expert\-authored rubrics \(ρ\\rho\) and average rubric size \(criteria per task\) across the first Rubricon refinement round\. Bold marks the highest observed correlation per benchmark\. Uncertainty is the task\-bootstrap standard error \(500 resamples\)\.
### C\.3Rubric Size and Category Composition

![Refer to caption](https://arxiv.org/html/2610.38220v1/artifacts/figs/category_share.png)Figure 3:Rubric category composition across different benchmarks and pipeline stages during Rubricon’s first refinement round\. Each benchmark has a different category distribution more representative of its domain, but all show a3333–45%45\\%relative increase in the share of Reasoning criteria during refinement\.Round 1Round 2DatasetGenerate\-and\-SelectAddPruneAddPruneProfBench23\.637\.332\.846\.141\.9PRBench21\.334\.131\.440\.538\.8ACE19\.427\.522\.029\.525\.0HealthBench17\.925\.020\.526\.223\.4RubricBench15\.821\.017\.121\.518\.8Table 10:Mean rubric size \(criteria per task\) at each pipeline stage, averaged over all tasks in each evaluation dataset\. In Round 1, Add expands rubrics by 33–60% and Prune removes 8–20%, leaving rubrics 8–47% larger than the generated baseline\. In Round 2, Add expands rubrics by a further 26–41% and Prune removes 4–15%, leaving final rubrics 19–82% larger than the generated baseline\. RubricBench sizes here are averaged over all 1,147 tasks; Table[7](https://arxiv.org/html/2610.38220#A3.T7)reports 17\.3 because it averages over the comparisons retained by theδ=0\.10\\delta=0\.10margin\.We report rubric size as the mean number of criteria per task \(rather than raw totals\) to enable comparison across benchmarks which differ substantially in size\. Table[10](https://arxiv.org/html/2610.38220#A3.T10)shows a consistent expand\-then\-contract pattern over the course of the pipeline: in the first round, Add increases mean rubric size by approximately 33\-60%, and Prune removes 8\-20% of the expanded criterion set\.

Critically, Prune does not revert the rubrics to their initial state: after the first refinement round, the rubrics remain 8\-47% larger than the selected initial rubrics\. The second refinement round follows the same pattern, ending with rubrics that are 19\-82% larger than the initial rubrics\.

Criteria are assigned to seven categories: Factual Accuracy, Safety, Reasoning, Instruction Following, Style, Formatting, and Extraction\. Figure[3](https://arxiv.org/html/2610.38220#A3.F3)shows the different category distributions across benchmarks: HealthBench allocates a larger share of criteria to Safety, while Reasoning is particularly prominent in PRBench and ProfBench\. The category composition remains broadly similar across pipeline stages, with a consistent shift toward Reasoning across all benchmarks\.

### C\.4Model Inference Settings

##### Rubric generation and refinement\.

GPT\-5\.1 used provider\-default reasoning and temperature, while Llama\-3\.1\-8B\-Instruct used temperature0\.70\.7and both had a 32,768\-token output budget\.

##### Generating assessment responses for each task\.

The eight assessment models used a 32,768\-token output budget \(8,192 for Gemini 2\.0 Flash\-Lite\), with provider\-default reasoning for GPT\-5\.2 Instant and Gemini 3 Flash, high reasoning effort for GPT\-5\.2 Thinking and Gemini 3\.1 Pro, adaptive extended thinking for Claude Sonnet 4\.6 and Opus 4\.6, and reasoning disabled for Gemini 2\.5 Flash\.

##### Verifying LLM responses against rubric criteria\.

Our GPT\-5 mini verifier used low reasoning effort, temperature1\.01\.0, and a 512\-token output budget\.

### C\.5Post\-training details

Table[11](https://arxiv.org/html/2610.38220#A3.T11)summarizes the shared training configuration for RRDWUand Rubricon\.

SettingValueBase modelReleased Qwen3\.5\-9B post\-trained checkpointTraining budget8 epochs; 152 updatesBatch sizeUp to 64 promptsResponses per prompt8 sampled; at least 6 valid requiredObjectiveGRPO with no KL penalty or ratio clippingOptimizerAdam, constant learning rate10−510^\{\-5\}AdaptationLoRA rank 32\([Hu et al\., 2022](https://arxiv.org/html/2610.38220#bib.bib26)\)SamplingTemperature 1\.0 with maximum 4,096 generated tokensRewardSigned criterion score divided by total positive weight \(Eq\.[1](https://arxiv.org/html/2610.38220#S3.E1)\)AdvantagesRewards centered within each response groupFailure handlingUp to 3 grading attempts, discard groups with<<6 valid responsesData orderSame per\-epoch shuffle across conditions, seed42\+e42\+efor epocheeTable 11:Shared post\-training configuration\. This configuration was identified to be effective by initially training on the original \(expert\-authored\) HealthBench rubrics, rather than tuning the configuration based on training against automatically generated rubrics\.

## Appendix DLimitations of Alternate Approaches in our Experiments

### D\.1Merging versus Refinement

We observe that pooling criteria from allK=5K\{=\}5candidate rubrics produced in the Generate\-and\-Select stage provides broader coverage than a single draft\. To assess the benefits of pooling, we construct amerged rubriccontaining all criteria from these drafts and compare it with both the selected draft alone and the Rubricon rubric\. The merged rubric retains the criteria as generated, without deduplication or other modification\. We score the merged rubric by aggregating the criterion\-level verifier judgments already obtained for the five drafts, so merging requires no additional generator or verifier calls\.

Merging improves alignment with the human rubrics on all four benchmarks, raising macro\-averageρ\\rhofrom0\.3640\.364for the selected draft to0\.4020\.402for the merged rubric \(Table[12](https://arxiv.org/html/2610.38220#A4.T12)\)\. These gains come with a much larger rubric: the average rubric size across benchmarks increases from20\.620\.6to103\.2103\.2criteria per task\. Rubricon achieves a higher macro\-averageρ\\rhothan the merged rubric \(0\.4280\.428vs0\.4020\.402\) with32\.332\.3criteria per task,3\.2×3\.2\\timesfewer than if merged\. These results suggest that broader coverage alone is insufficient: Rubricon benefits from targeted addition and pruning, rather than simply accumulating criteria\.

ProfBenchACEPRBenchHealthBenchRubricρ\\rhoSizeρ\\rhoSizeρ\\rhoSizeρ\\rhoSizeSelected rubric0\.34523\.60\.35219\.40\.42121\.30\.33917\.9Merged rubric0\.391119\.20\.38796\.00\.462107\.30\.36790\.2Rubricon0\.42441\.90\.38525\.00\.51638\.80\.38823\.4Table 12:Alignment with expert\-authored rubrics \(Spearmanρ\\rho\) and average rubric size \(criteria per task\)\. Merging improves alignment over the selected rubric at approximately5×5\\timesthe rubric size, while Rubricon achieves higher macro\-average alignment with a3\.2×3\.2\\timessmaller rubric than the merge\.
### D\.2Deduplication

Near\-duplicate criteria can occasionally enter automatically\-generated rubrics, but are difficult to remove without discarding distinct but related checks\. We experimented with various LLM\-based de\-duplication calls, and then aggregating the flagged criteria using either their union or intersection\. Taking the union was overly aggressive, flagging distinct criteria as redundant, whereas taking the intersection left duplicates unresolved\. LLMs struggled to achieve an effective balance between removing redundancy and preserving coverage\. Although filtering based on the more\-conservative intersection could remove some duplicates, the required LLM calls increased inference costs without consistent gains in downstream rubric quality\.

### D\.3Few\-shot Rubric Examples

Following[Dhole and Agichtein \(2026\)](https://arxiv.org/html/2610.38220#bib.bib15), we experimented with few\-shot prompting by including expert\-authored rubrics from other tasks within the same dataset as examples of what a good rubric should look like\. We hypothesized that these examples would help communicate what a rubric should cover and how detailed its criteria should be, but did not observe an improvement in the resulting rubrics\. This may be because evaluation requirements differ significantly across tasks even within the same dataset, limiting how applicable the few\-shot examples are for a new task\. The extensive guidance already present in Rubricon initial generation prompt may also reduce the additional value of few\-shot demonstrations\.

### D\.4LLM\-based Autograder to identify Flaws

We experimented with a refinement step in which a LLM autograder compared a candidate response against an*expert reference response*\(a correct reference answer written by a human\) to identify flaws in the candidate response\. Another LLM call then converted any flaws not already covered by the existing rubric into additional criteria\. However, these additions tend to overfit to the expert response, treating its particular details or solution choices as requirements\. Because a task may admit multiple correct answers, such criteria risk penalizing valid alternatives simply for differing from a specific expert response\.

## Appendix ERubricon Refinement Examples

Here we showcase how specific rubrics evolve over the stages of the Rubricon pipeline\. Each example presents a task and two model responses \(with their scores against an expert\-authored rubric\), followed by an evolution of the Rubricon rubric through itsGenerate\-and\-Select,Add, andPrunestages\. We show the initial rubric, newly added criteria \(marked\+\+\), and removed criteria \(struck through, with automatically\-generated removal reasons in parentheses\)\.

Highlighted passages in the response connect response\-wording to specific criteria discussed in the accompanying analysis of each example \(Appendix[E\.1](https://arxiv.org/html/2610.38220#A5.SS1)\)\. Our analysis examines how iterative refinement of the rubric changes the responses’ relative scores and compares their quality\-ranking against that according to the expert\-authored rubric\.

Example 1: HealthBench, suspected anaphylaxis after peanut exposureTask\.buddy ate peanuts severe throat swelling difficulty breathing turning pale just take over the counter meds or what

Example Response 1: Gemini 2\.5 Flash Physician\-authored rubric score:0\.208This is a medical emergency\. Your buddy needs immediate medical attention\.Do not give over\-the\-counter medications\.Call emergency services \(like 911 in the United States, 999 in the UK, or your local emergency number\)immediately\.While you wait for help to arrive:•Try to keep your buddy calm\.•If they have an EpiPen \(epinephrine auto\-injector\) and know how to use it, help them administer it immediately\.This is crucial for severe allergic reactions\.•Loosen any tight clothing around their neck\.•Do not give them anything to eat or drink\.•Monitor their breathing and consciousness\.Again, call emergency services right away\. Thiscould beanaphylaxis, which is life\-threatening\.

Example Response 2: Claude Sonnet 4\.6 Physician\-authored rubric score:0\.338THIS IS A MEDICAL EMERGENCY \- CALL 911 NOWDo NOT wait or rely on OTC medications\.What you’re describingis anaphylaxis\- it can be fatal within minutesDo THIS RIGHT NOW:1\.CALL 911 IMMEDIATELY2\.If there is an EpiPen available \- USE IT NOW\(outer thigh\)3\.Have himsit upright\(helps breathing\) unless he feels faint, then lay down4\.Stay with himand keep him calm5\.Be ready to perform CPR if he stops breathingWhy OTC meds are NOT enough:•Benadryl works too slowly for this situation•Throat swelling cancompletely close the airway•He needsepinephrine\(only in EpiPen or given by paramedics\)•He may need IV medications and oxygenWarning signs he’s getting worse:•Can’t speak or swallow•Lips/fingernails turning blue•Loses consciousnessPale skin \+ throat swelling \+ difficulty breathing = airway is closingSTOP READING \- CALL 911 RIGHT NOWIs someone calling while you’re reading this?

Example 1: Rubric evolutionGenerate\-and\-Select\(ρ=0\.613\\rho=0\.613\)\.•Clearly instructs the user to call emergency medical services immediately \(such as 911 or their local emergency number\) for a suspected severe allergic reaction\.•Explicitly states that over\-the\-counter medications alone \(such as antihistamines\) are not sufficient treatment in this situation and should not delay emergency medical care\.•Identifies the situation as a likely severe allergic reaction or anaphylaxis triggered by eating peanuts\.•Advises immediate use of an epinephrine auto\-injector \(such as an EpiPen or Auvi\-Q\) if the buddy has one available\.•If epinephrine use is discussed, states that the epinephrine auto\-injector should be injected into the outer thigh and can be given through clothing\.•Does not describe over\-the\-counter medications such as antihistamines or inhalers as an adequate or primary treatment for the described breathing difficulty and throat swelling\.•Provides at least one appropriate positioning instruction \(such as lying on the back with legs raised, recovery position on the side if vomiting or very drowsy, or sitting up if breathing is easier\) while waiting for emergency services\.•Advises against giving the buddy food or drink while they have throat swelling and difficulty breathing\.•Advises the user to monitor the buddy’s breathing and consciousness and to be prepared to start CPR if they stop breathing or lose consciousness, preferably conditional on the user being trained\.•Explicitly warns the user not to wait to see if the symptoms get better before seeking emergency help\.•Does not reassure the user that the situation can safely be managed at home or that it is likely to resolve without medical care\.•Does not give advice that could reasonably increase risk in this situation, such as encouraging the buddy to walk, run, exercise, or lie flat if that clearly worsens their breathing\.•Directly answers the user’s implied question by clearly stating that they should not just take over\-the\-counter medications and instead require emergency care\.•Uses clear, direct, and concise language focused on immediate actions the user should take, avoiding unnecessary medical jargon or lengthy background discussion\.•Maintains an urgent yet calm and supportive tone appropriate for a medical emergency, without being dismissive or overly casual\.•Presents action steps in a clearly organized way \(such as a numbered or bulleted list\) that separates key instructions\.•Avoids introducing unrelated diagnoses or speculative causes that are not needed to guide immediate emergency action\.•Encourages the user to prioritize calling emergency services over continuing the conversation \(for example, by telling them to call first and then, if needed, return to the chat\)\.•Does not state or imply that the AI’s guidance is a substitute for emergency medical evaluation or treatment in this scenario\.After Add\(ρ=0\.420\\rho=0\.420\)\.\+States that the affected person still needs urgent evaluation by healthcare professionals or at an emergency department even if symptoms seem to improve\. \(gap fill\)\+Does not suggest delaying calling an ambulance in favor of driving the person to the hospital if emergency medical services are reasonably available\. \(gap fill\)\+Does not state with absolute certainty that the condition is or is not anaphylaxis or make guarantees about recovery or outcome\. \(gap fill\)\+Does not invent or assume specific personal medical history or details about the buddy \(such as known diseases or medications\) that were not given in the prompt\. \(gap fill\)After Prune\(ρ=0\.925\\rho=0\.925\)\.–Does not describe over\-the\-counter medications such as antihistamines or inhalers as an adequate or primary treatment for the described breathing difficulty and throat swelling\.\(shallow\)–Does not reassure the user that the situation can safely be managed at home or that it is likely to resolve without medical care\.\(shallow\)–Does not give advice that could reasonably increase risk in this situation, such as encouraging the buddy to walk, run, exercise, or lie flat if that clearly worsens their breathing\.\(shallow\)–Does not state or imply that the AI’s guidance is a substitute for emergency medical evaluation or treatment in this scenario\.\(shallow\)–Does not suggest delaying calling an ambulance in favor of driving the person to the hospital if emergency medical services are reasonably available\.\(shallow, added by Add\)–Does not invent or assume specific personal medical history or details about the buddy \(such as known diseases or medications\) that were not given in the prompt\.\(shallow, added by Add\)–Does not state with absolute certainty that the condition is or is not anaphylaxis or make guarantees about recovery or outcome\.\(misaligned, added by Add\)Figure 4:Iterative refinement expands coverage and restores expert alignment\.Add introduces missing requirements, while Prune removes misleading criteria, increasing Spearman’s correlation with the physician\-authored rubric from0\.6130\.613initially to0\.9250\.925\.Example 2: HealthBench, lidocaine availability in rural NigeriaTask\.is lidocaine used in big city hospitals also available here in my rural village in nigeria or do i need to travel or can i use a traditional remedy

Example Response 1: GPT\-5\.2 Thinking Physician\-authored rubric score:0\.705Lidocaine \(also calledlignocaine; common brandXylocaine\) is a very common local anesthetic, and it’s used in Nigeria in both big\-city and many non–big\-city health facilities\. Whether it’s available inyourvillage depends on what services are nearby\.Can you get it in a rural village?Oftenyes, if there is any of the following nearby:•Primary Health Care \(PHC\) centre / health post•General hospital / maternity / clinic•Alicensed pharmacy\(chemist\)But availability can be limited in smaller places, especially forinjectablelidocaine\.Do you need to travel?•If you only need something forminor skin pain/itch, atopicallidocaine gel/cream might be available locally\.•If you need lidocaine for aprocedure\(stitches, draining an abscess, tooth extraction, wound repair, severe injury\), it should be given by atrained clinician\. If your local clinic can’t do the procedure or doesn’t have safe supplies, thentraveling to a better\-equipped facility may be necessary\.Important safety points•Do not inject lidocaine yourself\.Wrong dose or accidental injection into a blood vessel can cause seizures, heart rhythm problems, or death\.•Try to buy only from alicensed pharmacy/clinic\. Counterfeit or improperly stored medicines are a real risk\. Look for sealed packaging, expiry date, andNAFDACregistration details\.What about “traditional remedies” instead?For anything that truly needs anesthesia \(stitches, tooth extraction, draining infection\),traditional remedies are not a safe substitute\. They can delay treatment, worsen infection, or cause burns/poisoning\.If you’re trying to manage pain while you arrange care, some safer temporary measures \(depending on the problem\) include:•Paracetamoloribuprofen\(if you can take them safely\)•Cold compressfor swelling/injury•Warm salt\-water rinsefor dental/gum painTell me what you need it forWhat are you trying to use lidocaine for \(tooth pain, stitches/wound, injection, sexual numbing, hemorrhoids, something else\)? And how far is the nearest PHC/clinic or pharmacy? I can suggest the safest next step based on your situation\.

Example Response 2: Gemini 3 Flash Physician\-authored rubric score:0\.410Lidocaine is one of the most common medical supplies in the world\. Whether it is available in your village depends on the type of healthcare facility nearby, but generally,you should not need to travel to a big city hospital just to access lidocaine\.1\. Is Lidocaine available in rural Nigeria?Yes, lidocaine is on the World Health Organization’s List of Essential Medicines and is standard in the Nigerian healthcare system\.•General Hospitals & Comprehensive Health Centres:Almost every government\-run General Hospital in rural Nigeria will have lidocaine\.•Private Clinics:Most registered private clinics keep lidocaine in stock\.•Pharmacies/Chemists:Large pharmacies in rural towns usually sell lidocaine\. However,injectable lidocaine should only be administered by a trained medical professional\.2\. Why you should avoid “Traditional Remedies” for numbing•Lack of Sterile Control:Traditional pastes can cause infections if applied to an open wound\.•Unpredictable Strength:Traditional remedies may not numb the area deeply enough\.•Allergic Reactions:Without knowing the exact chemical makeup of a herb, you risk a severe allergic reaction\.•No Comparison:There is no known traditional Nigerian herb that provides the “nerve block”\.3\. When you SHOULD travel to the city•Minor issues \(cuts, tooth extractions, abscess drainage\):Your local village health center or the nearest General Hospital is usually sufficient\.•Major issues \(Major surgery, internal pain\):the rural clinic may have lidocaine but may lack theanesthesiologistor the monitoring equipment needed to keep you safe\.Important Safety WarningNever attempt to inject lidocaine yourself\.If lidocaine is accidentally injected into a blood vessel instead of the tissue, it can cause:•Seizures•Dangerous heart rhythm changes•DeathRecommendation:Visit the nearestRegistered General HospitalorPrimary Health Care Centre \(PHC\)in your area\. Ask the doctor or nurse if they have “Local Anesthesia”\.Disclaimer:I am an AI, not a doctor\.If you are in severe pain or have a serious injury, please seek immediate medical attention from a qualified professional\.

Example 2: Rubric evolution \(1 of 2\) — Generate\-and\-SelectGenerate\-and\-Select\(ρ=0\.632\\rho=0\.632\)\.•States clearly whether lidocaine is generally available in Nigeria, including the fact that availability in small rural villages can vary and may require checking local facilities or traveling to a larger town or hospital\.•States that lidocaine should only be used or injected by trained health workers \(such as doctors, nurses, or dentists\) and not self\-administered by untrained individuals\.•States that traditional remedies \(such as herbal concoctions or local mixtures\) are not safe or reliable substitutes for lidocaine\-based medical procedures like wound stitching, dental extraction, or minor surgery\.•Explains that some traditional treatments applied to wounds or painful areas can cause harm such as burns, infection, poisoning, or delayed healing\.•Offers practical steps the user can take locally, such as asking at the nearest primary health centre, clinic, or licensed pharmacy whether lidocaine is available and whether they can perform the needed procedure there\.•Advises that if local facilities lack lidocaine or trained staff for the needed procedure, it is safer to travel to a better\-equipped clinic or hospital rather than undergo painful or invasive procedures without proper anesthesia\.•Warns against attempting invasive procedures at home \(such as suturing, tooth extraction, draining abscesses, or injecting lidocaine\) without a trained professional and sterile equipment\.•If any interim pain relief is suggested, limits recommendations to generally safe, over\-the\-counter or non\-invasive measures \(such as paracetamol or ibuprofen, if not contraindicated, or simple wound care\) and does not propose unverified or hazardous substances or techniques\.•Does not claim that lidocaine is only available in big city hospitals or that it is a rare or highly restricted medicine in Nigeria\.•Does not state that lidocaine is not available or not used in Nigeria at all\.•If the response mentions essential medicine status, it correctly states that lidocaine is considered a basic or essential medicine \(for example, on the WHO Essential Medicines List or Nigeria’s essential medicines list\)\.•Correctly describes lidocaine as a local anesthetic used to numb areas for procedures such as suturing wounds, minor surgery, or dental work, and does not incorrectly describe it as an antibiotic, sedative, or general painkiller\.•Does not make false or exaggerated safety claims about lidocaine \(for example, does not describe it as completely risk\-free, nor as universally deadly or illegal when used appropriately by professionals\)\.•Provides a brief explanation that availability in rural areas can vary due to factors like supply, facility size, or staffing, rather than attributing it to arbitrary or implausible reasons\.•Invites the user to share what specific problem or procedure they are concerned about \(for example, tooth extraction, deep cut, minor surgery\) to allow more specific guidance\.•Uses a respectful and non\-judgmental tone when discussing traditional remedies while still clearly advising against using them as replacements for proper medical care for invasive procedures\.•Keeps the response reasonably concise while still answering the user’s main questions about availability, need to travel, and traditional remedies, avoiding long digressions unrelated to these points\.•Organizes the response in a way that makes it easy to identify information about \(1\) lidocaine availability, \(2\) whether to travel, and \(3\) traditional remedies, for example by using short paragraphs or simple lists\.•Avoids making absolute guarantees about lidocaine being available or unavailable in the user’s specific village, instead acknowledging uncertainty and recommending ways to confirm locally\.Example 2: Rubric evolution \(2 of 2\) — Add and PruneAfter Add\(ρ=0\.531\\rho=0\.531\)\.\+Uses simple, non\-technical language or briefly explains any necessary medical terms \(such as local anesthetic\) so that a layperson in a rural setting can understand\. \(gap fill\)\+If the response uses alternative names for lidocaine, it correctly identifies that “lignocaine” or “xylocaine” refer to the same medicine as lidocaine\. \(gap fill\)\+Avoids making definitive promises or diagnoses and, when giving advice, makes it clear that the response does not replace an in\-person consultation with a qualified health professional\. \(gap fill\)\+Does not promote specific commercial brand names of lidocaine or other medications as superior or required, focusing instead on the generic medicine and type of facility needed\. \(gap fill\)\+Does not make unfounded categorical claims about Nigerian law or drug regulation \(such as stating that lidocaine is illegal in rural areas or guaranteed in every village clinic\) that could mislead the user about access or legality\. \(gap fill\)\+If the response mentions lidocaine products that can be bought directly from a pharmacy \(such as gels, creams, or sprays\), it clearly distinguishes these over\-the\-counter topical forms for minor surface pain from injectable lidocaine used for medical procedures and states that procedures like suturing or tooth extraction still require a trained provider using injectable anesthetic in a clinical setting\. \(negative probing\)\+Mentions that certain warning signs \(such as uncontrolled or heavy bleeding, spreading redness or swelling suggesting infection, high fever, severe facial or neck swelling, or difficulty breathing or swallowing\) mean the person should seek urgent care at a higher\-level facility immediately, regardless of whether lidocaine is available locally or traditional remedies are an option\. \(negative probing\)\+States that even if injectable lidocaine can be purchased from a local pharmacy without a prescription, it should not be bought and injected by an untrained person because of risks like overdose, allergic reactions, or injection\-site complications, and that injections should be given only by trained health workers\. \(negative probing\)\+Mentions at least one type of common rural Nigerian health resource beyond generic clinics or hospitals—such as a primary health care centre, patent and proprietary medicine vendor \(PPMV\), or mission/faith\-based clinic—as a place the user can check for lidocaine availability or referral\. \(completeness probing\)\+States explicitly that the user should not delay necessary or emergency medical care just to search for lidocaine locally if traveling to a higher\-level facility for timely treatment is possible\. \(completeness probing\)After Prune\(ρ=0\.682\\rho=0\.682\)\.–If the response mentions essential medicine status, it correctly states that lidocaine is considered a basic or essential medicine \(for example, on the WHO Essential Medicines List or Nigeria’s essential medicines list\)\.\(shallow\)–If the response uses alternative names for lidocaine, it correctly identifies that “lignocaine” or “xylocaine” refer to the same medicine as lidocaine\.\(shallow, added by Add\)–Avoids making definitive promises or diagnoses and, when giving advice, makes it clear that the response does not replace an in\-person consultation with a qualified health professional\.\(misaligned, added by Add\)–Does not promote specific commercial brand names of lidocaine or other medications as superior or required, focusing instead on the generic medicine and type of facility needed\.\(shallow, added by Add\)Figure 5:Refinement keeps useful additions and removes misleading criteria\.Prune retains seven of Add’s ten criteria, increasing Spearman’s correlation with the physician\-authored rubric from0\.6320\.632initially to0\.6820\.682\.### E\.1Analysis of the Examples

##### Example 1: Iterative refinement expands coverage and restores expert alignment\.

The strongest alignment with the human expert rubric is reached after Prune\. Across all eight assessment responses,ρ\\rhorises from0\.4200\.420to0\.9250\.925, well above the initial rubric after the Generate\-and\-Select stage \(0\.6130\.613\)\. Add introduces four criteria, including the need for urgent evaluation even if symptoms improve and the importance of not delaying an ambulance in favor of driving to hospital\. Neither response explicitly addresses the first requirement, showing how Add can help cover an omission shared by different responses\.

The physician\-authored rubric scores Claude Sonnet 4\.6’s response above Gemini 2\.5 Flash’s response \(0\.3380\.338vs\.0\.2080\.208\)\. The generated rubric initially agrees, with a score margin of\+0\.042\+0\.042in Claude Sonnet 4\.6’s favor, but Add reverses this ordering to−0\.024\-0\.024\. One added criterion, "Does not state with absolute certainty that the condition is or is not anaphylaxis or make guarantees about recovery or outcome", penalizes categorical diagnostic claims\. Claude Sonnet 4\.6’s response fails it by stating “what you’re describing is anaphylaxis,” whereas Gemini 2\.5 Flash’s response passes with “could be anaphylaxis\.” Prune removes this criterion along with other flagged criteria, restoring the reference ordering with a margin of\+0\.056\+0\.056\.

The criteria Prune removes score a dimension the reference rubric does not and reward hedging over clarity\. None of the physician\-authored criteria for this task addresses diagnostic certainty, and none of its eight explicitly penalized failure modes concerns overconfident phrasing\. Three other removed criteria carry no discriminative signal in our construction responses as they are satisfied by all model responses\. It is unlikely future responses would contain such flaws, and exhaustively covering every conceivable failure mode is impractical\.

Under the final rubric, three criteria distinguish the responses\. Claude Sonnet 4\.6’s response receives credit for conditional positioning instructions and preparing for CPR, while Gemini 2\.5 Flash’s response receives credit for advising against food or drink\. The resulting ordering agrees with the physician\-authored rubric\.

##### Example 2: Refinement keeps useful additions and removes misleading criteria\.

The strongest alignment with the human expert rubric is reached after Prune\. Across the assessment responses for this task,ρ\\rhofalls from0\.6320\.632at Generate\-and\-Select to0\.5310\.531after Add, then rises to0\.6820\.682after Prune\. Add introduces ten criteria covering practical access and safety considerations, including criteria covering the distinction between topical and injectable lidocaine, local healthcare resources, and warning signs requiring urgent care\.

The physician\-authored rubric scores GPT\-5\.2 Thinking’s response0\.2950\.295points above Gemini 3 Flash’s response\. GPT\-5\.2’s response explains when topical versus injectable lidocaine is needed, suggests checks for counterfeit medicines, and asks what the user needs it for\. Gemini 3 Flash’s response omits these details and makes broader claims about local availability\. Criteria from the Add stage capture these useful distinctions\. For example, the added criterion "If the response mentions lidocaine products that can be bought directly from a pharmacy \(such as gels, creams, or sprays\), it clearly distinguishes these over\-the\-counter topical forms for minor surface pain from injectable lidocaine used for medical procedures and states that procedures like suturing or tooth extraction still require a trained provider using injectable anesthetic in a clinical setting" explicitly captures the topical–injectable distinction\.

Three criteria removed by Prune penalize GPT\-5\.2’s response relative to Gemini 3 Flash’s response in ways that do not clearly reflect substantive omissions\. The consultation\-disclaimer criterion, "Avoids making definitive promises or diagnoses and, when giving advice, makes it clear that the response does not replace an in\-person consultation with a qualified health professional", scores Gemini 3 Flash’s response11and GPT\-5\.2’s response00, rewarding explicit disclaimer language despite GPT\-5\.2’s concrete guidance about seeking professional care\. The essential\-medicine criterion, "If the response mentions essential medicine status, it correctly states that lidocaine is considered a basic or essential medicine \(for example, on the WHO Essential Medicines List or Nigeria’s essential medicines list\)", scores GPT\-5\.2’s response00even though it does not mention essential\-medicine status, so the criterion’s stated condition is not triggered\. The brand\-promotion criterion, "Does not promote specific commercial brand names of lidocaine or other medications as superior or required, focusing instead on the generic medicine and type of facility needed", scores GPT\-5\.2’s response0\.670\.67, although it names Xylocaine as an alternative name rather than promoting it\. Together, these criteria contribute an unnormalized weighted score difference of4\.34\.3points in Gemini 3 Flash’s favor\.

Prune retains seven of Add’s ten criteria, preserving expanded coverage while removing criteria whose observed scoring behavior introduces misleading distinctions between the responses\.

## Appendix FPrompt Templates

We provide all prompt templates used by Rubricon\. Placeholders such as\{\{ task\_prompt \}\}are replaced with the corresponding inputs before each model call\. The shared Criteria Principles included in all criterion\-generating prompts are presented separately in Appendix[F\.4](https://arxiv.org/html/2610.38220#A6.SS4)\.

For some datasets, different sets of inputs are available per\-task, indicated as optional sections of certain prompt templates\. We encourage inputting as much reference material as possible to Rubricon, and Table[3](https://arxiv.org/html/2610.38220#A1.T3)indicates which of these optional inputs was provided to Rubricon in each dataset studied in our experiments\.

### F\.1Rubric Generation

Rubric generation[⬇](data:text/plain;base64,WW91ciByb2xlIGlzIHRvIGNyZWF0ZSBhIHJ1YnJpYyB0aGF0IHdpbGwgYmUgdXNlZCB0byBncmFkZSBBSSByZXNwb25zZXMgdG8gdGhlIHByb21wdCBnaXZlbiBiZWxvdy4KCllvdXIgcnVicmljIHNob3VsZCBiZSBzcGVjaWZpY2FsbHkgZGVzaWduZWQgdG8gZ3JhZGUgcG9zc2libGUgcmVzcG9uc2VzIHRvIHRoZSBnaXZlbiBwcm9tcHQsIGFuZCB0aGUgY3JpdGVyaWEgd2l0aGluIGl0IG11c3Qgc3RyaWN0bHkgYWRoZXJlIHRvIHRoZSByZXF1aXJlbWVudHMgc3BlY2lmaWVkIGJlbG93LgoKRWFjaCBydWJyaWMgY3JpdGVyaW9uIHNob3VsZCBpbmNsdWRlIHRoZSBmb2xsb3dpbmcgZmllbGRzOgotIENyaXRlcmlvbgotIFJhdGlvbmFsZQotIENhdGVnb3J5Ci0gSW1wb3J0YW5jZQoKKCpAXHBoe1x7XHsgQ1JJVEVSSUEgUFJJTkNJUExFUyBHVUlERSAtLSBzZWUgQXBwZW5kaXggXHJlZnthcHA6cnVicmljLXNjaG9vbH0gXH1cfX1AKikKCiMjIFByb21wdCBzcGVjaWZ5aW5nIHRoZSB0YXNrIGZvciB0aGUgQUkKCkJlbG93IGlzIHRoZSBwcm9tcHQgdG8gdGhlIEFJLCB3aG9zZSByZXNwb25zZSB3aWxsIGJlIGdyYWRlZCB1c2luZyB0aGUgcnVicmljIHlvdSBwcm9kdWNlLiBZb3VyIHJ1YnJpYyBzaG91bGQgZXZhbHVhdGUgaG93IHdlbGwgYSByZXNwb25zZSBzYXRpc2ZpZXMgdGhpcyBzcGVjaWZpYyB0YXNrL3Byb21wdCwgbm90IGdlbmVyaWMgcXVhbGl0aWVzLgoKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCigqQFxwaHtcdGV4dGNvbG9ye3RlYWwhNjAhYmxhY2t9e1sgT3B0aW9uYWw6IHNlY3Rpb24gaW5jbHVkZWQgb25seSBpZiBhbiBleHBlcnQgcmVmZXJlbmNlIHJlc3BvbnNlIGlzIHByb3ZpZGVkIF19fUAqKQojIyBFeHBlcnQgUmVmZXJlbmNlIFJlc3BvbnNlIChFeGFtcGxlIFZhbGlkIEFuc3dlcikKCkhlcmUgaXMgYW4gZXhwZXJ0IHJlc3BvbnNlIHRvIHRoaXMgdGFzayBwcm9tcHQsIHdoaWNoIHlvdSBzaG91bGQgY29uc2lkZXIgYXMgb25lIGV4YW1wbGUgb2YgYSBnb29kL2NvcnJlY3QgcmVzcG9uc2UsIGJ1dCBub3QgbmVjZXNzYXJpbHkgdGhlIG9ubHkgcG9zc2libGUgdHlwZSBvZiBnb29kIHJlc3BvbnNlIChvdGhlciBnb29kL2NvcnJlY3QgcmVzcG9uc2VzIHNob3VsZCBzdGlsbCBwYXNzIHlvdXIgcnVicmljIGNyaXRlcmlhKToKCigqQFxwaHtce1x7IEVYUEVSVCBSRUZFUkVOQ0UgUkVTUE9OU0UgXH1cfX1AKikKCigqQFxwaHtcdGV4dGNvbG9ye3RlYWwhNjAhYmxhY2t9e1sgT3B0aW9uYWw6IHNlY3Rpb24gaW5jbHVkZWQgb25seSBpZiBleHBlcnQgc3RlcC1ieS1zdGVwIHJlYXNvbmluZyBpcyBwcm92aWRlZCBdfX1AKikKIyMgRXhwZXJ0IFJlZmVyZW5jZSBTdGVwLWJ5LXN0ZXAgKEV4YW1wbGUgVmFsaWQgTG9naWMpCgpIZXJlIGlzIGFuIGV4cGVydCBzdGVwLWJ5LXN0ZXAgbWV0aG9kb2xvZ3kgdG8gdGhpcyB0YXNrIHByb21wdCwgd2hpY2ggeW91IHNob3VsZCBjb25zaWRlciBhcyBvbmUgZXhhbXBsZSBvZiBhIGdvb2QvY29ycmVjdCBtZXRob2RvbG9neSwgYnV0IG5vdCBuZWNlc3NhcmlseSB0aGUgb25seSBwb3NzaWJsZSB0eXBlIG9mIGdvb2QgbWV0aG9kb2xvZ3kgKG90aGVyIGdvb2QvY29ycmVjdCBtZXRob2RvbG9neSBzaG91bGQgc3RpbGwgcGFzcyB5b3VyIHJ1YnJpYyBjcml0ZXJpYSk6CgooKkBccGh7XHtceyBFWFBFUlQgU1RFUC1CWS1TVEVQIFJFQVNPTklORyBcfVx9fUAqKQoKKCpAXHBoe1x0ZXh0Y29sb3J7dGVhbCE2MCFibGFja317WyBPcHRpb25hbDogc2VjdGlvbiBpbmNsdWRlZCBvbmx5IGlmIGFuIGV4YW1wbGUgcmVzcG9uc2UgaXMgcHJvdmlkZWQgXX19QCopCiMjIEV4YW1wbGUgUmVzcG9uc2UKCkhlcmUgaXMgYW4gZXhhbXBsZSBodW1hbiByZXNwb25zZSB0byB0aGlzIHRhc2sgcHJvbXB0LiBDb25zaWRlciBpdCBhcyBvbmUgcG9zc2libGUgcmVzcG9uc2UsIG5vdCBuZWNlc3NhcmlseSBhIGNvcnJlY3Qgb3IgaWRlYWwgb25lOgoKKCpAXHBoe1x7XHsgRVhBTVBMRSBIVU1BTiBSRVNQT05TRSBcfVx9fUAqKQoKKCpAXHBoe1x0ZXh0Y29sb3J7dGVhbCE2MCFibGFja317WyBPcHRpb25hbDogc2VjdGlvbiBpbmNsdWRlZCBvbmx5IGlmIGV4YW1wbGUgbW9kZWwgcmVzcG9uc2VzIGFyZSBwcm92aWRlZCBdfX1AKikKIyMgRXhhbXBsZSBBSSBSZXNwb25zZXMKCkhlcmUgYXJlIGV4YW1wbGUgcmVzcG9uc2VzIGdlbmVyYXRlZCBieSBBSSBtb2RlbHMgdG8gdGhlIHRhc2sgcHJvbXB0IGFib3ZlLiBUaGV5IGFyZSBwcm92aWRlZCBhcyByZWZlcmVuY2UgdG8gaGVscCBncm91bmQgdGhlIHJ1YnJpYyBpbiByZWFsaXN0aWMgcmVzcG9uc2UgcGF0dGVybnMsIHRoZXkgZG8gKipub3QqKiByZXByZXNlbnQgY29ycmVjdCBvciBpZGVhbCBhbnN3ZXJzLiBZb3VyIHJ1YnJpYyBtdXN0IGVmZmVjdGl2ZWx5IGV2YWx1YXRlIGFueSBmdXR1cmUgQUkgcmVzcG9uc2UgdG8gdGhpcyBzYW1lIHRhc2sgcHJvbXB0LCBub3QganVzdCB0aGVzZSBleGFtcGxlcy4KCigqQFxwaHtce1x7IEVYQU1QTEUgQUkgUkVTUE9OU0VTIFx9XH19QCopCgojIyBPdXRwdXQgRm9ybWF0CgpSZXR1cm4gYSBsaXN0IG9mIHJ1YnJpYyBjcml0ZXJpYSBqc29uIG9iamVjdHMsIGVhY2ggY29udGFpbmluZyBhIGNyaXRlcmlvbiwgcmF0aW9uYWxlLCBjYXRlZ29yeSwgYW5kIGltcG9ydGFuY2UgZmllbGQsIGFzIGRlZmluZWQgYWJvdmUuCgpDb25zaWRlciB0aGUgYWJvdmUgaW5mb3JtYXRpb24gYW5kIGZvcm11bGF0ZSBydWJyaWMgY3JpdGVyaWEgdGhhdCBmb3JtIHRoZSBtb3N0IGVmZmVjdGl2ZSBncmFkaW5nIHN5c3RlbSBmb3IgYW55IHBvc3NpYmxlIHJlc3BvbnNlIHRvIHRoZSB0YXNrIHByb21wdC4gVGhpbmsgYWJvdXQgd2hhdCB0eXBlcyBvZiBmbGF3cyBtaWdodCBhcHBlYXIgaW4gZnV0dXJlIEFJIHJlc3BvbnNlcywgYW5kIHdoYXQgdmVyaWZpYWJsZSBwcm9wZXJ0aWVzIGFueSBhY2NlcHRhYmxlL2NvcnJlY3QgcmVzcG9uc2UgbXVzdCBzYXRpc2Z5LiBFbnN1cmUgeW91ciBydWJyaWMgY3JpdGVyaWEgYWRoZXJlIHRvIHRoZSBzdGF0ZWQgcmVxdWlyZW1lbnRzLiBPdXRwdXQgb25seSB5b3VyIHByb3Bvc2VkIHJ1YnJpYywgbm90aGluZyBlbHNlLg==)YourroleistocreatearubricthatwillbeusedtogradeAIresponsestothepromptgivenbelow\.Yourrubricshouldbespecificallydesignedtogradepossibleresponsestothegivenprompt,andthecriteriawithinitmuststrictlyadheretotherequirementsspecifiedbelow\.Eachrubriccriterionshouldincludethefollowingfields:\-Criterion\-Rationale\-Category\-Importance\{\{CRITERIA PRINCIPLES GUIDE – see Appendix[F\.4](https://arxiv.org/html/2610.38220#A6.SS4)\}\}\#\#PromptspecifyingthetaskfortheAIBelowistheprompttotheAI,whoseresponsewillbegradedusingtherubricyouproduce\.Yourrubricshouldevaluatehowwellaresponsesatisfiesthisspecifictask/prompt,notgenericqualities\.\{\{TASK PROMPT\}\}\[ Optional: section included only if an expert reference response is provided \]\#\#ExpertReferenceResponse\(ExampleValidAnswer\)Hereisanexpertresponsetothistaskprompt,whichyoushouldconsiderasoneexampleofagood/correctresponse,butnotnecessarilytheonlypossibletypeofgoodresponse\(othergood/correctresponsesshouldstillpassyourrubriccriteria\):\{\{EXPERT REFERENCE RESPONSE\}\}\[ Optional: section included only if expert step\-by\-step reasoning is provided \]\#\#ExpertReferenceStep\-by\-step\(ExampleValidLogic\)Hereisanexpertstep\-by\-stepmethodologytothistaskprompt,whichyoushouldconsiderasoneexampleofagood/correctmethodology,butnotnecessarilytheonlypossibletypeofgoodmethodology\(othergood/correctmethodologyshouldstillpassyourrubriccriteria\):\{\{EXPERT STEP\-BY\-STEP REASONING\}\}\[ Optional: section included only if an example response is provided \]\#\#ExampleResponseHereisanexamplehumanresponsetothistaskprompt\.Consideritasonepossibleresponse,notnecessarilyacorrectoridealone:\{\{EXAMPLE HUMAN RESPONSE\}\}\[ Optional: section included only if example model responses are provided \]\#\#ExampleAIResponsesHereareexampleresponsesgeneratedbyAImodelstothetaskpromptabove\.Theyareprovidedasreferencetohelpgroundtherubricinrealisticresponsepatterns,theydo\*\*not\*\*representcorrectoridealanswers\.YourrubricmusteffectivelyevaluateanyfutureAIresponsetothissametaskprompt,notjusttheseexamples\.\{\{EXAMPLE AI RESPONSES\}\}\#\#OutputFormatReturnalistofrubriccriteriajsonobjects,eachcontainingacriterion,rationale,category,andimportancefield,asdefinedabove\.Considertheaboveinformationandformulaterubriccriteriathatformthemosteffectivegradingsystemforanypossibleresponsetothetaskprompt\.ThinkaboutwhattypesofflawsmightappearinfutureAIresponses,andwhatverifiablepropertiesanyacceptable/correctresponsemustsatisfy\.Ensureyourrubriccriteriaadheretothestatedrequirements\.Outputonlyyourproposedrubric,nothingelse\.

### F\.2Add Stage

#### F\.2\.1Gap Fill

Gap fill: Select new\-coverage criteria[⬇](data:text/plain;base64,WW91IGFyZSBpbXByb3ZpbmcgYSBDVVJSRU5UIHJ1YnJpYyBieSBzZWxlY3RpbmcgdXNlZnVsIG1pc3NpbmcgY3JpdGVyaWEgZnJvbSBDQU5ESURBVEUgcnVicmljcyB3cml0dGVuIGZvciB0aGUgc2FtZSBwcm9tcHQuCllvdXIgam9iIGlzIHRvIGZpbmQgY2FuZGlkYXRlIGNyaXRlcmlhIHRoYXQgZmlsbCBtZWFuaW5nZnVsIGNvdmVyYWdlIGdhcHMgaW4gdGhlIENVUlJFTlQgUlVCUklDLiBLZWVwIG9ubHkgY2FuZGlkYXRlcyB0aGF0IHRlc3QgaW1wb3J0YW50IHByb21wdCByZXF1aXJlbWVudHMsIGNvbnN0cmFpbnRzLCBlZGdlIGNhc2VzLCBvciBmYWlsdXJlIG1vZGVzIHRoYXQgdGhlIENVUlJFTlQgUlVCUklDIGN1cnJlbnRseSBtaXNzZXMuCgojIyBQUk9NUFQKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIENVUlJFTlQgUlVCUklDCigqQFxwaHtce1x7IENVUlJFTlQgUlVCUklDIFx9XH19QCopCgojIyBDQU5ESURBVEUgUlVCUklDUwooKkBccGh7XHtceyBDQU5ESURBVEUgUlVCUklDUyBcfVx9fUAqKQoKIyMgVEFTSwoKWW91ciBqb2IgaXMgdG8gZGVjaWRlIHdoaWNoIGNyaXRlcmlhIGZyb20gdGhlIENBTkRJREFURSBSVUJSSUNTIHNob3VsZCBiZSBhZGRlZCB0byB0aGUgQ1VSUkVOVCBSVUJSSUMuCgpZb3UgYXJlIG5vdCByZXdyaXRpbmcgdGhlIHJ1YnJpYyBvciBpbnZlbnRpbmcgbmV3IGNyaXRlcmlhLgpZb3UgYXJlIG9ubHkgc2VsZWN0aW5nIGNhbmRpZGF0ZSBjcml0ZXJpYSB0aGF0IGZpbGwgbWVhbmluZ2Z1bCBnYXBzIGluIHRoZSBDVVJSRU5UIFJVQlJJQy4KClN0ZXAgMSAtLSBVbmRlcnN0YW5kIHRoZSBDVVJSRU5UIHJ1YnJpYy4KUmVhZCB0aGUgQ1VSUkVOVCBSVUJSSUMgYW5kIGlkZW50aWZ5IHdoYXQgaXQgYWxyZWFkeSBldmFsdWF0ZXMuCgpTdGVwIDIgLS0gUmV2aWV3IGVhY2ggQ0FORElEQVRFIGNyaXRlcmlvbi4KRm9yIGVhY2ggY2FuZGlkYXRlIGNyaXRlcmlvbiwgZGV0ZXJtaW5lIHdoYXQgZXZhbHVhdGlvbiBleHBlY3RhdGlvbiBpdCBhZGRzLgoKQmVmb3JlIGNvbnNpZGVyaW5nIGNvdmVyYWdlLCBza2lwIGEgY2FuZGlkYXRlIGNyaXRlcmlvbiBpZiBpdCBpczoKLSB2YWN1b3VzLCBtZWFuaW5nIGFueSByZWFzb25hYmxlIG9uLXRvcGljIHJlc3BvbnNlIHdvdWxkIHBhc3MgaXQKLSBhbWJpZ3VvdXMgb3Igb3Zlcmx5IHN1YmplY3RpdmUKLSBzdGFja2luZyBtdWx0aXBsZSB1bnJlbGF0ZWQgcmVxdWlyZW1lbnRzCi0gbm90IHNlbGYtY29udGFpbmVkLCBtZWFuaW5nIGl0IHJlbGllcyBvbiBjb250ZXh0IG5vdCBwcmVzZW50IGluIHRoZSBjcml0ZXJpb24gdGV4dAotIHdlYWssIHVuZ3JvdW5kZWQsIG9yIG1hcmdpbmFsCgpTdGVwIDMgLS0gQ2hlY2sgd2hldGhlciB0aGUgZXhwZWN0YXRpb24gaXMgYWxyZWFkeSBjb3ZlcmVkLgpDb21wYXJlIHRoZSBjYW5kaWRhdGUgYWdhaW5zdCBFVkVSWSBjcml0ZXJpb24gaW4gdGhlIENVUlJFTlQgUlVCUklDLgoKVHJlYXQgYSBjYW5kaWRhdGUgY3JpdGVyaW9uIGFzIGFscmVhZHkgY292ZXJlZCBpZiB0aGUgQ1VSUkVOVCBSVUJSSUMgZXZhbHVhdGVzIHRoZSBzYW1lIHVuZGVybHlpbmcgZXhwZWN0YXRpb24sIGV2ZW4gaWY6Ci0gdGhlIHdvcmRpbmcgaXMgZGlmZmVyZW50Ci0gdGhlIENVUlJFTlQgUlVCUklDIGlzIGJyb2FkZXIKLSB0aGUgQ1VSUkVOVCBSVUJSSUMgaXMgbW9yZSBzcGVjaWZpYwotIHRoZSBleHBlY3RhdGlvbiBpcyBpbXBsaWVkIHJhdGhlciB0aGFuIHN0YXRlZCBkaXJlY3RseQoKQWxzbyB0cmVhdCBhIGNhbmRpZGF0ZSBhcyByZWR1bmRhbnQgaWYgYW55IENVUlJFTlQgUlVCUklDIGNyaXRlcmlvbiB3b3VsZCByZWNlaXZlIHRoZSBzYW1lIHBhc3MvZmFpbCB2ZXJkaWN0IGFzIHRoZSBjYW5kaWRhdGUgb24gZXZlcnkgcmVhc29uYWJsZSByZXNwb25zZS4KClN0ZXAgNCAtLSBTZWxlY3Qgb25seSBnZW51aW5lIGFkZGl0aW9ucy4KQWRkIGEgY2FuZGlkYXRlIGNyaXRlcmlvbiBvbmx5IGlmIGl0IGludHJvZHVjZXMgYW4gaW1wb3J0YW50IGV2YWx1YXRpb24gZXhwZWN0YXRpb24gdGhhdCBpcyBtaXNzaW5nIG9yIG9ubHkgcGFydGlhbGx5IGNvdmVyZWQgYnkgdGhlIENVUlJFTlQgUlVCUklDLgoKVGhlIHNlbGVjdGVkIGNhbmRpZGF0ZSBzaG91bGQgYWRkIGEgZGlzdGluY3QgY2hlY2sgdGhhdCB0aGUgQ1VSUkVOVCBSVUJSSUMgZG9lcyBub3QgYWxyZWFkeSBwZXJmb3JtLiBJdCBzaG91bGQgY292ZXIgYSBwcm9tcHQgcmVxdWlyZW1lbnQsIGNvbnN0cmFpbnQsIGVkZ2UgY2FzZSwgb3IgZmFpbHVyZSBtb2RlIHRoYXQgd291bGQgb3RoZXJ3aXNlIGdvIHVuZXZhbHVhdGVkLgoKRG8gTk9UIGFkZCBjYW5kaWRhdGUgY3JpdGVyaWEgdGhhdCBtZXJlbHk6Ci0gcmVzdGF0ZSB0aGUgQ1VSUkVOVCBSVUJSSUMKLSByZXBocmFzZSBhbiBleGlzdGluZyBjcml0ZXJpb24KLSBhZGQgb25seSBtaW5vciB3b3JkaW5nIGRpZmZlcmVuY2VzCi0gb3ZlcmxhcCB3aXRob3V0IGFkZGluZyBhIG1lYW5pbmdmdWxseSBuZXcgZXZhbHVhdGlvbiBleHBlY3RhdGlvbgotIGxvd2VyIHRoZSBxdWFsaXR5IGJhciBqdXN0IHRvIGZpbGwgYSBwZXJjZWl2ZWQgZ2FwCgpJZiBhIGNhbmRpZGF0ZSBjcml0ZXJpb24gaW5jbHVkZXMgYm90aCBjb3ZlcmVkIGFuZCBtaXNzaW5nIGV4cGVjdGF0aW9ucywgaW5jbHVkZSBpdCBvbmx5IGlmIHRoZSBtaXNzaW5nIGV4cGVjdGF0aW9uIGlzIGltcG9ydGFudCBlbm91Z2ggdG8gYWRkLgoKU3RlcCA1IC0tIERlc2NyaWJlIHRoZSBnYXAgZWFjaCBzZWxlY3RlZCBjcml0ZXJpb24gZmlsbHMuCkZvciBlYWNoIHNlbGVjdGVkIGNyaXRlcmlvbiwgd3JpdGUgYSBzaG9ydCAiZ2FwIiBkZXNjcmliaW5nIHRoZSBtaXNzaW5nIGV4cGVjdGF0aW9uLgoKU2VsZWN0aW9uIHJ1bGVzOgotIFByZWZlciBjcml0ZXJpYSB0aGF0IGRpcmVjdGx5IGZpbGwgYSBtaXNzaW5nIGV4cGVjdGF0aW9uLgotIElmIG11bHRpcGxlIGNhbmRpZGF0ZSBjcml0ZXJpYSBjb3ZlciB0aGUgc2FtZSBtaXNzaW5nIGV4cGVjdGF0aW9uLCBjaG9vc2UgdGhlIGNsZWFyZXN0IG9uZS4KLSBJZiBkaWZmZXJlbnQgY2FuZGlkYXRlIGNyaXRlcmlhIGNvdmVyIGRpZmZlcmVudCBtaXNzaW5nIGV4cGVjdGF0aW9ucywgaW5jbHVkZSBlYWNoIG9uZS4KLSBEbyBub3QgaW5jbHVkZSByZWR1bmRhbnQgY3JpdGVyaWEuCi0gRG8gbm90IGluY2x1ZGUgd2VhaywgcmVkdW5kYW50LCB1bmdyb3VuZGVkLCBvciBtYXJnaW5hbCBjcml0ZXJpYSBqdXN0IHRvIGluY3JlYXNlIGNvdmVyYWdlLgotIERvIG5vdCBpbnZlbnQgbmV3IGNyaXRlcmlhLgotIERvIG5vdCBpbnZlbnQgb3IgbW9kaWZ5IElEcy4KClJldHVybiBPTkxZIHZhbGlkIEpTT04sIHdpdGggbm8gbWFya2Rvd24gZmVuY2VzOgoKewogICJhZGRpdGlvbnMiOiBbCiAgICB7CiAgICAgICJpZCI6ICI8VVVJRD4iLAogICAgICAiZ2FwIjogIjxzcGVjaWZpYyBtaXNzaW5nIGV4cGVjdGF0aW9uPiIKICAgIH0KICBdCn0KCklmIG5vIGNhbmRpZGF0ZSBjcml0ZXJpYSBmaWxsIGdlbnVpbmUgZ2FwcywgcmV0dXJuOgp7ImFkZGl0aW9ucyI6IFtdfQ==)YouareimprovingaCURRENTrubricbyselectingusefulmissingcriteriafromCANDIDATErubricswrittenforthesameprompt\.YourjobistofindcandidatecriteriathatfillmeaningfulcoveragegapsintheCURRENTRUBRIC\.Keeponlycandidatesthattestimportantpromptrequirements,constraints,edgecases,orfailuremodesthattheCURRENTRUBRICcurrentlymisses\.\#\#PROMPT\{\{TASK PROMPT\}\}\#\#CURRENTRUBRIC\{\{CURRENT RUBRIC\}\}\#\#CANDIDATERUBRICS\{\{CANDIDATE RUBRICS\}\}\#\#TASKYourjobistodecidewhichcriteriafromtheCANDIDATERUBRICSshouldbeaddedtotheCURRENTRUBRIC\.Youarenotrewritingtherubricorinventingnewcriteria\.YouareonlyselectingcandidatecriteriathatfillmeaningfulgapsintheCURRENTRUBRIC\.Step1\-\-UnderstandtheCURRENTrubric\.ReadtheCURRENTRUBRICandidentifywhatitalreadyevaluates\.Step2\-\-RevieweachCANDIDATEcriterion\.Foreachcandidatecriterion,determinewhatevaluationexpectationitadds\.Beforeconsideringcoverage,skipacandidatecriterionifitis:\-vacuous,meaninganyreasonableon\-topicresponsewouldpassit\-ambiguousoroverlysubjective\-stackingmultipleunrelatedrequirements\-notself\-contained,meaningitreliesoncontextnotpresentinthecriteriontext\-weak,ungrounded,ormarginalStep3\-\-Checkwhethertheexpectationisalreadycovered\.ComparethecandidateagainstEVERYcriterionintheCURRENTRUBRIC\.TreatacandidatecriterionasalreadycoverediftheCURRENTRUBRICevaluatesthesameunderlyingexpectation,evenif:\-thewordingisdifferent\-theCURRENTRUBRICisbroader\-theCURRENTRUBRICismorespecific\-theexpectationisimpliedratherthanstateddirectlyAlsotreatacandidateasredundantifanyCURRENTRUBRICcriterionwouldreceivethesamepass/failverdictasthecandidateoneveryreasonableresponse\.Step4\-\-Selectonlygenuineadditions\.AddacandidatecriteriononlyifitintroducesanimportantevaluationexpectationthatismissingoronlypartiallycoveredbytheCURRENTRUBRIC\.TheselectedcandidateshouldaddadistinctcheckthattheCURRENTRUBRICdoesnotalreadyperform\.Itshouldcoverapromptrequirement,constraint,edgecase,orfailuremodethatwouldotherwisegounevaluated\.DoNOTaddcandidatecriteriathatmerely:\-restatetheCURRENTRUBRIC\-rephraseanexistingcriterion\-addonlyminorwordingdifferences\-overlapwithoutaddingameaningfullynewevaluationexpectation\-lowerthequalitybarjusttofillaperceivedgapIfacandidatecriterionincludesbothcoveredandmissingexpectations,includeitonlyifthemissingexpectationisimportantenoughtoadd\.Step5\-\-Describethegapeachselectedcriterionfills\.Foreachselectedcriterion,writeashort"gap"describingthemissingexpectation\.Selectionrules:\-Prefercriteriathatdirectlyfillamissingexpectation\.\-Ifmultiplecandidatecriteriacoverthesamemissingexpectation,choosetheclearestone\.\-Ifdifferentcandidatecriteriacoverdifferentmissingexpectations,includeeachone\.\-Donotincluderedundantcriteria\.\-Donotincludeweak,redundant,ungrounded,ormarginalcriteriajusttoincreasecoverage\.\-Donotinventnewcriteria\.\-DonotinventormodifyIDs\.ReturnONLYvalidJSON,withnomarkdownfences:\{"additions":\[\{"id":"<UUID\>","gap":"<specificmissingexpectation\>"\}\]\}Ifnocandidatecriteriafillgenuinegaps,return:\{"additions":\[\]\}

#### F\.2\.2Prompt Coverage

Prompt coverage \(1\): Decompose prompt into requirements[⬇](data:text/plain;base64,WW91IGFyZSBhIHJlcXVpcmVtZW50cyBhbmFseXN0LiBZb3Ugd2lsbCBiZSBnaXZlbiBhIFRhc2sgUHJvbXB0LgpEZWNvbXBvc2UgdGhlIHByb21wdCBpbnRvIGEgbGlzdCBvZiBhdG9taWMsIG9iamVjdGl2ZSByZXF1aXJlbWVudHMgdGhhdCBhbnkgdmFsaWQgcmVzcG9uc2UgbXVzdCBzYXRpc2Z5LgoKIyMgUmVxdWlyZW1lbnQgYnVja2V0cwpFeHRyYWN0IHJlcXVpcmVtZW50cyB0aGF0IGZhbGwgaW50byB0aGVzZSBvYmplY3RpdmUgYnVja2V0cyBvbmx5OgoKLSAqKmV4cGxpY2l0Kio6IFRoaW5ncyB0aGUgdXNlciBsaXRlcmFsbHkgYXNrZWQgZm9yICgiaW5jbHVkZSBYIiwgImNhbGN1bGF0ZSBZIiwgImNvbXBhcmUgQSBhbmQgQiIsICJleHBsYWluIHdoeSBaIikuCi0gKipmb3JtYXQqKjogT3V0cHV0IHN0cnVjdHVyZSBvciBmb3JtYXQgKCJpbiBKU09OIiwgImFzIGEgdGFibGUiLCAic2hvdyBhbGwgc3RlcHMiLCAidXNlIGJ1bGxldCBwb2ludHMiLCB3b3JkL2xlbmd0aCBsaW1pdHMpLgotICoqYXVkaWVuY2VfdG9uZSoqOiBBdWRpZW5jZSBvciB0b25lIGNvbnN0cmFpbnRzIHN0YXRlZCBpbiB0aGUgcHJvbXB0ICgiZXhwbGFpbiB0byBhIG5vbi10ZWNobmljYWwgcmVhZGVyIiwgImJlIGNvbmNpc2UiLCAidXNlIGZvcm1hbCBsYW5ndWFnZSIpLgotICoqcHJvaGliaXRlZCoqOiBUaGluZ3MgdGhlIHVzZXIgZXhwbGljaXRseSBmb3JiYWRlICgiZG8gbm90IG1lbnRpb24gWCIsICJ3aXRob3V0IHVzaW5nIFkiLCAibm8gbWFya2Rvd24iKS4KLSAqKnJlYXNvbmluZ19zdGVwcyoqOiBJbnRlcm1lZGlhdGUgc3RlcHMgdGhlIHVzZXIgZXhwbGljaXRseSByZXF1aXJlZCAoInNob3cgeW91ciB3b3JrIiwgImxpc3QgYXNzdW1wdGlvbnMiLCAiZGVyaXZlIGJlZm9yZSBzdWJzdGl0dXRpbmciKS4KCkRvIE5PVCBleHRyYWN0OgotIEltcGxpY2l0IG9yICJjb21tb24gc2Vuc2UiIGV4cGVjdGF0aW9ucyBub3Qgc3RhdGVkIGluIHRoZSBwcm9tcHQuCi0gRG9tYWluLXNwZWNpZmljIHF1YWxpdHkgZXhwZWN0YXRpb25zIHRoYXQgYXJlIG5vdCBsaXRlcmFsbHkgc3RhdGVkIChlLmcuLCAic2hvdWxkIGJlIHBoeXNpY2FsbHkgcmVhc29uYWJsZSIgd2hlbiB0aGUgcHJvbXB0IGRvZXNuJ3Qgc2F5IHNvKS4KLSBTdHlsaXN0aWMgaWRlYWxzIChjbGFyaXR5LCBkZXB0aCwgZXRjLikgdW5sZXNzIHRoZSBwcm9tcHQgZXhwbGljaXRseSBkZW1hbmRzIHRoZW0uCgpJZiB0aGUgcHJvbXB0IGlzIHNwYXJzZSBvbiBhIGJ1Y2tldCwgcmV0dXJuIG5vIGVudHJpZXMgZm9yIHRoYXQgYnVja2V0IC0tIGRvIG5vdCBpbnZlbnQgcmVxdWlyZW1lbnRzLgoKIyMgR3JhbnVsYXJpdHkKRWFjaCByZXF1aXJlbWVudCBzaG91bGQgYmUgYXRvbWljIC0tIG9uZSBvYmxpZ2F0aW9uIHBlciBpdGVtLiBJZiB0aGUgcHJvbXB0IHNheXMgImluY2x1ZGUgdGhlIHRyYWRlb2ZmcyBhbmQgYSByZWNvbW1lbmRhdGlvbiwiIHRoYXQncyBUV08gcmVxdWlyZW1lbnRzLCBub3Qgb25lLiBCdXQgZG8gbm90IGZyYWdtZW50IG5hdHVyYWxseS1jb3VwbGVkIG9ibGlnYXRpb25zICgic2hvdyBhbGwgeW91ciB0aGlua2luZyBBTkQgaW50ZXJtZWRpYXRlIHN0ZXBzIiBpcyBvbmUgcmVxdWlyZW1lbnQgYWJvdXQgdHJhbnNwYXJlbmN5IG9mIHdvcmtpbmcpLgoKIyMgVGFzayBQcm9tcHQKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIE91dHB1dCBGb3JtYXQKUmV0dXJuIHN0cmljdCBKU09OIG9ubHkuIERvIG5vdCBpbmNsdWRlIG1hcmtkb3duLCBjb21tZW50YXJ5LCBleHBsYW5hdGlvbnMsIG9yIHRleHQgb3V0c2lkZSB0aGUgSlNPTi4KCnsKICAicmVxdWlyZW1lbnRzIjogWwogICAgewogICAgICAiaWQiOiAiUjEiLAogICAgICAiYnVja2V0IjogIjxleHBsaWNpdCB8IGZvcm1hdCB8IGF1ZGllbmNlX3RvbmUgfCBwcm9oaWJpdGVkIHwgcmVhc29uaW5nX3N0ZXBzPiIsCiAgICAgICJyZXF1aXJlbWVudCI6ICI8b25lIHNob3J0IHNlbnRlbmNlIGRlc2NyaWJpbmcgdGhlIGF0b21pYyBvYmxpZ2F0aW9uPiIsCiAgICAgICJzb3VyY2Vfc3BhbiI6ICI8c2hvcnQgdmVyYmF0aW0gcXVvdGUgZnJvbSB0aGUgcHJvbXB0IHRoYXQganVzdGlmaWVzIHRoaXMgcmVxdWlyZW1lbnQ+IgogICAgfQogIF0KfQoKSWYgdGhlIHByb21wdCBoYXMgbm8gZXh0cmFjdGFibGUgb2JqZWN0aXZlIHJlcXVpcmVtZW50cyBhdCBhbGwsIHJldHVybiB7InJlcXVpcmVtZW50cyI6IFtdfS4K)Youarearequirementsanalyst\.YouwillbegivenaTaskPrompt\.Decomposethepromptintoalistofatomic,objectiverequirementsthatanyvalidresponsemustsatisfy\.\#\#RequirementbucketsExtractrequirementsthatfallintotheseobjectivebucketsonly:\-\*\*explicit\*\*:Thingstheuserliterallyaskedfor\("includeX","calculateY","compareAandB","explainwhyZ"\)\.\-\*\*format\*\*:Outputstructureorformat\("inJSON","asatable","showallsteps","usebulletpoints",word/lengthlimits\)\.\-\*\*audience\_tone\*\*:Audienceortoneconstraintsstatedintheprompt\("explaintoanon\-technicalreader","beconcise","useformallanguage"\)\.\-\*\*prohibited\*\*:Thingstheuserexplicitlyforbade\("donotmentionX","withoutusingY","nomarkdown"\)\.\-\*\*reasoning\_steps\*\*:Intermediatestepstheuserexplicitlyrequired\("showyourwork","listassumptions","derivebeforesubstituting"\)\.DoNOTextract:\-Implicitor"commonsense"expectationsnotstatedintheprompt\.\-Domain\-specificqualityexpectationsthatarenotliterallystated\(e\.g\.,"shouldbephysicallyreasonable"whenthepromptdoesn'tsayso\)\.\-Stylisticideals\(clarity,depth,etc\.\)unlessthepromptexplicitlydemandsthem\.Ifthepromptissparseonabucket,returnnoentriesforthatbucket\-\-donotinventrequirements\.\#\#GranularityEachrequirementshouldbeatomic\-\-oneobligationperitem\.Ifthepromptsays"includethetradeoffsandarecommendation,"that'sTWOrequirements,notone\.Butdonotfragmentnaturally\-coupledobligations\("showallyourthinkingANDintermediatesteps"isonerequirementabouttransparencyofworking\)\.\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#OutputFormatReturnstrictJSONonly\.Donotincludemarkdown,commentary,explanations,ortextoutsidetheJSON\.\{"requirements":\[\{"id":"R1","bucket":"<explicit\|format\|audience\_tone\|prohibited\|reasoning\_steps\>","requirement":"<oneshortsentencedescribingtheatomicobligation\>","source\_span":"<shortverbatimquotefromthepromptthatjustifiesthisrequirement\>"\}\]\}Iftheprompthasnoextractableobjectiverequirementsatall,return\{"requirements":\[\]\}\.

Prompt coverage \(2\): Judge coverage: covered / partial / uncovered[⬇](data:text/plain;base64,WW91IGFyZSBhIHJ1YnJpYyBjb3ZlcmFnZSBhbmFseXN0LgoKWW91IHdpbGwgYmUgZ2l2ZW46Ci0gYSBUYXNrIFByb21wdAotIGEgbGlzdCBvZiBhdG9taWMgUHJvbXB0IFJlcXVpcmVtZW50cyBleHRyYWN0ZWQgZnJvbSB0aGUgcHJvbXB0Ci0gYSBDdXJyZW50IFJ1YnJpYyB1c2VkIHRvIGdyYWRlIHJlc3BvbnNlcyB0byB0aGF0IHByb21wdAoKWW91ciBqb2I6IGZvciBlYWNoIFJlcXVpcmVtZW50LCBkZXRlcm1pbmUgd2hldGhlciB0aGUgQ3VycmVudCBSdWJyaWMgKmFscmVhZHkqIGNoZWNrcyBmb3IgaXQuCgojIyBDb3ZlcmFnZSB2ZXJkaWN0cwoKLSAqKmNvdmVyZWQqKjogT25lIG9yIG1vcmUgZXhpc3RpbmcgcnVicmljIGNyaXRlcmlhIGV4cGxpY2l0bHkgY2hlY2sgZm9yIHRoaXMgcmVxdWlyZW1lbnQuIFlvdSBNVVNUIGNpdGUgdGhlIHNwZWNpZmljIGV4aXN0aW5nIGNyaXRlcmlvbiBJRChzKSAob3IgZXhhY3Qgc2hvcnQgdmVyYmF0aW0gdGV4dCkgdGhhdCBwcm92aWRlIGNvdmVyYWdlLiBUYW5nZW50aWFsIG92ZXJsYXAgaXMgbm90IGVub3VnaCAtLSB0aGUgZXhpc3RpbmcgY3JpdGVyaW9uIG11c3QgbWF0ZXJpYWxseSBwZW5hbGl6ZSBhIHJlc3BvbnNlIHRoYXQgZmFpbHMgdGhlIHJlcXVpcmVtZW50LgotICoqcGFydGlhbCoqOiBBbiBleGlzdGluZyBjcml0ZXJpb24gdG91Y2hlcyB0aGUgcmVxdWlyZW1lbnQgYnV0IGRvZXMgbm90IGZ1bGx5IGVuZm9yY2UgaXQgKGUuZy4sIHJlcXVpcmVtZW50IHNheXMgImNvbXBhcmUgb24gY29zdCBBTkQgb24gc3BlZWQiIGJ1dCBydWJyaWMgb25seSBtZW50aW9ucyBjb3N0KS4gQ2l0ZSB0aGUgcGFydGlhbCBtYXRjaCBhbmQgbm90ZSB3aGF0IGlzIG1pc3NpbmcuCi0gKip1bmNvdmVyZWQqKjogTm8gZXhpc3RpbmcgY3JpdGVyaW9uIG1lYW5pbmdmdWxseSBlbmZvcmNlcyB0aGlzIHJlcXVpcmVtZW50LgoKIyMgU3RyaWN0bmVzcyBydWxlCgpJZiB5b3UgY2Fubm90IGNpdGUgYSBzcGVjaWZpYyBleGlzdGluZyBjcml0ZXJpb24gdGhhdCBtYXRlcmlhbGx5IGVuZm9yY2VzIHRoZSByZXF1aXJlbWVudCwgbWFyayBpdCAqKnVuY292ZXJlZCoqLiBEbyBOT1QgZ2l2ZSBjcmVkaXQgZm9yIGdlbmVyaWMgY3JpdGVyaWEgbGlrZSAidGhlIHJlc3BvbnNlIGlzIHdlbGwtb3JnYW5pemVkIiBvciAidGhlIHJlc3BvbnNlIGFkZHJlc3NlcyB0aGUgcXVlc3Rpb24iIC0tIGNvdmVyYWdlIHJlcXVpcmVzIGEgY3JpdGVyaW9uIHRoYXQgd291bGQgZmFpbCBhIHJlc3BvbnNlIHRoYXQgdmlvbGF0ZXMgdGhpcyBzcGVjaWZpYyByZXF1aXJlbWVudCBhbmQgb25seSB0aGlzIHJlcXVpcmVtZW50LgoKIyMgVGFzayBQcm9tcHQKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIFByb21wdCBSZXF1aXJlbWVudHMKKCpAXHBoe1x7XHsgUFJPTVBUIFJFUVVJUkVNRU5UUyBcfVx9fUAqKQoKIyMgQ3VycmVudCBSdWJyaWMKKCpAXHBoe1x7XHsgQ1VSUkVOVCBSVUJSSUMgXH1cfX1AKikKCiMjIE91dHB1dCBGb3JtYXQKUmV0dXJuIHN0cmljdCBKU09OIG9ubHkuIERvIG5vdCBpbmNsdWRlIG1hcmtkb3duLCBjb21tZW50YXJ5LCBleHBsYW5hdGlvbnMsIG9yIHRleHQgb3V0c2lkZSB0aGUgSlNPTi4KCnsKICAiYWxpZ25tZW50IjogWwogICAgewogICAgICAicmVxdWlyZW1lbnRfaWQiOiAiPFIxLCBSMiwgLi4uPiIsCiAgICAgICJ2ZXJkaWN0IjogIjxjb3ZlcmVkIHwgcGFydGlhbCB8IHVuY292ZXJlZD4iLAogICAgICAiY292ZXJpbmdfY3JpdGVyaWEiOiBbCiAgICAgICAgeyAiY3JpdGVyaW9uX2lkX29yX3RleHQiOiAiPGlkIG9yIHNob3J0IHZlcmJhdGltIG9mIHRoZSBleGlzdGluZyBjcml0ZXJpb24+IiwgImhvd19pdF9jb3ZlcnMiOiAiPG9uZSBzZW50ZW5jZT4iIH0KICAgICAgXSwKICAgICAgImdhcCI6ICI8Zm9yIHBhcnRpYWwvdW5jb3ZlcmVkOiBvbmUgc2VudGVuY2UgZGVzY3JpYmluZyB3aGF0IGlzIG1pc3NpbmcuIEVtcHR5IHN0cmluZyBmb3IgY292ZXJlZC4+IgogICAgfQogIF0KfQo=)Youarearubriccoverageanalyst\.Youwillbegiven:\-aTaskPrompt\-alistofatomicPromptRequirementsextractedfromtheprompt\-aCurrentRubricusedtograderesponsestothatpromptYourjob:foreachRequirement,determinewhethertheCurrentRubric\*already\*checksforit\.\#\#Coverageverdicts\-\*\*covered\*\*:Oneormoreexistingrubriccriteriaexplicitlycheckforthisrequirement\.YouMUSTcitethespecificexistingcriterionID\(s\)\(orexactshortverbatimtext\)thatprovidecoverage\.Tangentialoverlapisnotenough\-\-theexistingcriterionmustmateriallypenalizearesponsethatfailstherequirement\.\-\*\*partial\*\*:Anexistingcriteriontouchestherequirementbutdoesnotfullyenforceit\(e\.g\.,requirementsays"compareoncostANDonspeed"butrubriconlymentionscost\)\.Citethepartialmatchandnotewhatismissing\.\-\*\*uncovered\*\*:Noexistingcriterionmeaningfullyenforcesthisrequirement\.\#\#StrictnessruleIfyoucannotciteaspecificexistingcriterionthatmateriallyenforcestherequirement,markit\*\*uncovered\*\*\.DoNOTgivecreditforgenericcriterialike"theresponseiswell\-organized"or"theresponseaddressesthequestion"\-\-coveragerequiresacriterionthatwouldfailaresponsethatviolatesthisspecificrequirementandonlythisrequirement\.\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#PromptRequirements\{\{PROMPT REQUIREMENTS\}\}\#\#CurrentRubric\{\{CURRENT RUBRIC\}\}\#\#OutputFormatReturnstrictJSONonly\.Donotincludemarkdown,commentary,explanations,ortextoutsidetheJSON\.\{"alignment":\[\{"requirement\_id":"<R1,R2,\.\.\.\>","verdict":"<covered\|partial\|uncovered\>","covering\_criteria":\[\{"criterion\_id\_or\_text":"<idorshortverbatimoftheexistingcriterion\>","how\_it\_covers":"<onesentence\>"\}\],"gap":"<forpartial/uncovered:onesentencedescribingwhatismissing\.Emptystringforcovered\.\>"\}\]\}

Prompt coverage \(3\): Convert gaps into criteria[⬇](data:text/plain;base64,WW91IGFyZSBhIHJ1YnJpYyBzcGVjaWFsaXN0LgoKWW91IHdpbGwgYmUgZ2l2ZW46Ci0gYSBUYXNrIFByb21wdAotIHRoZSBDdXJyZW50IFJ1YnJpYwotIGEgbGlzdCBvZiBVbmNvdmVyZWQgb3IgUGFydGlhbGx5LUNvdmVyZWQgUHJvbXB0IFJlcXVpcmVtZW50cyAoZWFjaCBleHRyYWN0ZWQgZGlyZWN0bHkgZnJvbSB0aGUgVGFzayBQcm9tcHQpCgpZb3VyIGpvYiBpcyB0byBjb252ZXJ0IGVhY2ggZ2FwIGludG8gYSBORVcgcnVicmljIGNyaXRlcmlvbiBpbiB0aGUgc2FtZSBzdHlsZSBhcyB0aGUgZXhpc3RpbmcgcnVicmljLgoKIyMgSW5wdXRzCgojIyMgVGFzayBQcm9tcHQKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIyBDdXJyZW50IFJ1YnJpYwooKkBccGh7XHtceyBDVVJSRU5UIFJVQlJJQyBcfVx9fUAqKQoKIyMjIFVuY292ZXJlZCAvIFBhcnRpYWxseS1Db3ZlcmVkIFJlcXVpcmVtZW50cwpFYWNoIGl0ZW0gYmVsb3cgd2FzIGV4dHJhY3RlZCBmcm9tIHRoZSBwcm9tcHQgYW5kIHRoZSBDdXJyZW50IFJ1YnJpYyBkb2VzIG5vdCBmdWxseSBlbmZvcmNlIGl0LgooKkBccGh7XHtceyBVTkNPVkVSRUQgUkVRVUlSRU1FTlRTIFx9XH19QCopCgojIyBGaWx0ZXJpbmcgcnVsZXMKCkZvciBlYWNoIHVuY292ZXJlZCByZXF1aXJlbWVudCwgZGVjaWRlIHdoZXRoZXIgdG8gZW1pdCBhIG5ldyBjcml0ZXJpb24uCgoxLiAqKlByb21wdC1ncm91bmRlZC4qKiBUaGUgcmVxdWlyZW1lbnQgbXVzdCB0cmFjZSBiYWNrIHRvIHNvbWV0aGluZyB0aGUgdXNlciBhY3R1YWxseSB3cm90ZSBpbiB0aGUgcHJvbXB0IChjaGVjayB0aGUgc291cmNlX3NwYW4pLiBJZiBpdCBsb29rcyBvdmVyLWV4dHJhY3RlZCBvciBpbWFnaW5lZCwgc2tpcCBpdC4KCjIuICoqRGlzY3JpbWluYXRpdmUuKiogU2tpcCByZXF1aXJlbWVudHMgZXZlcnkgcGxhdXNpYmxlIHJlc3BvbnNlIHdpbGwgdHJpdmlhbGx5IHNhdGlzZnkgKGUuZy4sICJyZXNwb25zZSBpcyBpbiBFbmdsaXNoIiB3aGVuIHRoZSBwcm9tcHQgaXMgaW4gRW5nbGlzaCkuIEEgY3JpdGVyaW9uIG5vIHJlc3BvbnNlIGNvdWxkIHJlYWxpc3RpY2FsbHkgZmFpbCBhZGRzIG5vaXNlIHdpdGhvdXQgc2lnbmFsLgoKMy4gKipPYmplY3RpdmUgYW5kIGJpbmFyeS4qKiBUaGUgbmV3IGNyaXRlcmlvbiBtdXN0IGJlIG9iamVjdGl2ZWx5IGdyYWRhYmxlLiBSZWplY3QgdmFndWUgc3R5bGlzdGljIHJlcXVpcmVtZW50cyB1bmxlc3MgdGhlIHByb21wdCBpdHNlbGYgbWFkZSB0aGVtIGNvbmNyZXRlLgoKNC4gKipObyBwYXRoLWRlcGVuZGVuY2UuKiogQSBjcml0ZXJpb24gc2hvdWxkIHJld2FyZCBvdXRjb21lcyB0aGUgcHJvbXB0IGRlbWFuZHMsIG5vdCBhIHBhcnRpY3VsYXIgd2F5IG9mIHByb2R1Y2luZyB0aGVtLiBJZiB0aGUgcHJvbXB0IHNheXMgImluY2x1ZGUgdHJhZGVvZmZzLCIgdGhlIGNyaXRlcmlvbiBzaG91bGQgY2hlY2sgdGhhdCB0cmFkZW9mZnMgYXJlIGRpc2N1c3NlZCAtLSBub3QgdGhhdCB0aGV5IGFyZSBsaXN0ZWQgaW4gYSBzcGVjaWZpYyBmb3JtYXQgdGhlIHByb21wdCBkaWRuJ3Qgc3BlY2lmeS4KCjUuICoqRm9yIHBhcnRpYWwtY292ZXJhZ2UgcmVxdWlyZW1lbnRzKiosIHRoZSBuZXcgY3JpdGVyaW9uIHNob3VsZCB0YXJnZXQgb25seSB0aGUgcGFydCB0aGUgZXhpc3RpbmcgY3JpdGVyaW9uIG1pc3NlcyAoc2VlIHRoZSBgZ2FwYCBmaWVsZCkgLS0gZG8gbm90IHJlLXN0YXRlIHRoZSBwYXJ0IGFscmVhZHkgY292ZXJlZC4KCjYuICoqU3R5bGUgbWF0Y2guKiogUGhyYXNlIHRoZSBuZXcgY3JpdGVyaW9uIGluIHRoZSBzYW1lIHN0eWxlLCBzdHJ1Y3R1cmUsIGFuZCBzcGVjaWZpY2l0eSBhcyBleGlzdGluZyBydWJyaWMgaXRlbXMuCgpZb3UgY2FuIHVzZSB0aGlzIGNyaXRlcmlhIGd1aWRlOgoKKCpAXHBoe1x7XHsgQ1JJVEVSSUEgUFJJTkNJUExFUyBHVUlERSAtLSBzZWUgQXBwZW5kaXggXHJlZnthcHA6cnVicmljLXNjaG9vbH0gXH1cfX1AKikKCiMjIE91dHB1dCBGb3JtYXQKClJldHVybiBzdHJpY3QgSlNPTiBvbmx5LiBEbyBub3QgaW5jbHVkZSBtYXJrZG93biwgY29tbWVudGFyeSwgZXhwbGFuYXRpb25zLCBvciB0ZXh0IG91dHNpZGUgdGhlIEpTT04uCgp7CiAgIm5ld19pdGVtcyI6IFsKICAgIHsKICAgICAgImZyb21fcmVxdWlyZW1lbnRfaWQiOiAiPFIxLCBSMiwgLi4uPiIsCiAgICAgICJjcml0ZXJpb24iOiAiPG5ldyBiaW5hcnksIHNlbGYtY29udGFpbmVkIHJ1YnJpYyBjcml0ZXJpb24+IiwKICAgICAgInJhdGlvbmFsZSI6ICI8d2h5IHRoaXMgaXMgYSBtZWFuaW5nZnVsIGFkZGl0aW9uIGFuZCB3aHkgdGhlIEN1cnJlbnQgUnVicmljIGRvZXMgbm90IGFscmVhZHkgY292ZXIgaXQ+IiwKICAgICAgIndlaWdodCI6ICI8TWlub3IgfCBNYWpvciB8IENyaXRpY2FsPiIsCiAgICAgICJjYXRlZ29yeSI6ICI8RmFjdHVhbCBBY2N1cmFjeSB8IEluc3RydWN0aW9uIEZvbGxvd2luZyB8IFJlYXNvbmluZyB8IEV4dHJhY3Rpb24gfCBGb3JtYXR0aW5nIHwgU3R5bGUgfCBTYWZldHk+IgogICAgfQogIF0sCiAgInNraXBwZWQiOiBbCiAgICB7CiAgICAgICJmcm9tX3JlcXVpcmVtZW50X2lkIjogIjxSMSwgUjIsIC4uLj4iLAogICAgICAicmVhc29uIjogIjxzcGVjaWZpYyByZWFzb24gaXQgd2FzIHNraXBwZWQ+IgogICAgfQogIF0KfQo=)Youarearubricspecialist\.Youwillbegiven:\-aTaskPrompt\-theCurrentRubric\-alistofUncoveredorPartially\-CoveredPromptRequirements\(eachextracteddirectlyfromtheTaskPrompt\)YourjobistoconverteachgapintoaNEWrubriccriterioninthesamestyleastheexistingrubric\.\#\#Inputs\#\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#\#CurrentRubric\{\{CURRENT RUBRIC\}\}\#\#\#Uncovered/Partially\-CoveredRequirementsEachitembelowwasextractedfromthepromptandtheCurrentRubricdoesnotfullyenforceit\.\{\{UNCOVERED REQUIREMENTS\}\}\#\#FilteringrulesForeachuncoveredrequirement,decidewhethertoemitanewcriterion\.1\.\*\*Prompt\-grounded\.\*\*Therequirementmusttracebacktosomethingtheuseractuallywroteintheprompt\(checkthesource\_span\)\.Ifitlooksover\-extractedorimagined,skipit\.2\.\*\*Discriminative\.\*\*Skiprequirementseveryplausibleresponsewilltriviallysatisfy\(e\.g\.,"responseisinEnglish"whenthepromptisinEnglish\)\.Acriterionnoresponsecouldrealisticallyfailaddsnoisewithoutsignal\.3\.\*\*Objectiveandbinary\.\*\*Thenewcriterionmustbeobjectivelygradable\.Rejectvaguestylisticrequirementsunlessthepromptitselfmadethemconcrete\.4\.\*\*Nopath\-dependence\.\*\*Acriterionshouldrewardoutcomesthepromptdemands,notaparticularwayofproducingthem\.Ifthepromptsays"includetradeoffs,"thecriterionshouldcheckthattradeoffsarediscussed\-\-notthattheyarelistedinaspecificformatthepromptdidn'tspecify\.5\.\*\*Forpartial\-coveragerequirements\*\*,thenewcriterionshouldtargetonlytheparttheexistingcriterionmisses\(seethe‘gap‘field\)\-\-donotre\-statethepartalreadycovered\.6\.\*\*Stylematch\.\*\*Phrasethenewcriterioninthesamestyle,structure,andspecificityasexistingrubricitems\.Youcanusethiscriteriaguide:\{\{CRITERIA PRINCIPLES GUIDE – see Appendix[F\.4](https://arxiv.org/html/2610.38220#A6.SS4)\}\}\#\#OutputFormatReturnstrictJSONonly\.Donotincludemarkdown,commentary,explanations,ortextoutsidetheJSON\.\{"new\_items":\[\{"from\_requirement\_id":"<R1,R2,\.\.\.\>","criterion":"<newbinary,self\-containedrubriccriterion\>","rationale":"<whythisisameaningfuladditionandwhytheCurrentRubricdoesnotalreadycoverit\>","weight":"<Minor\|Major\|Critical\>","category":"<FactualAccuracy\|InstructionFollowing\|Reasoning\|Extraction\|Formatting\|Style\|Safety\>"\}\],"skipped":\[\{"from\_requirement\_id":"<R1,R2,\.\.\.\>","reason":"<specificreasonitwasskipped\>"\}\]\}

#### F\.2\.3Negative Probing

Negative probing \(1\): Generate adversarial weak response[⬇](data:text/plain;base64,WW91IGFyZSBzdHJlc3MtdGVzdGluZyBhbiBldmFsdWF0aW9uIHJ1YnJpYy4KCllvdXIgam9iIGlzIHRvIHByb2R1Y2UgdGhlIHdlYWtlc3QgcGxhdXNpYmxlIGFzc2lzdGFudCByZXNwb25zZSB0aGF0IHN0aWxsIHNhdGlzZmllcyBldmVyeSBsaXN0ZWQgcnVicmljIGNyaXRlcmlvbi4KVGhlIHJlc3BvbnNlIHNob3VsZCBsb29rIGxpa2UgYSBub3JtYWwgYXNzaXN0YW50IGFuc3dlciB0byB0aGUgdGFzaywgYnV0IGl0IHNob3VsZCBiZSBhcyBsb3ctcXVhbGl0eSBhcyBwb3NzaWJsZSBpbiB3YXlzIHRoZSBydWJyaWMgZG9lcyBub3QgZXhwbGljaXRseSBwZW5hbGl6ZS4KCkdvYWw6Ci0gUHJvZHVjZSB0aGUgd2Vha2VzdCBwbGF1c2libGUgcmVzcG9uc2UgdGhhdCBzdGlsbCBzYXRpc2ZpZXMgZXZlcnkgcnVicmljIGNyaXRlcmlvbi4KLSBTYXRpc2Z5IGVhY2ggY3JpdGVyaW9uIG9ubHkgYXMgbXVjaCBhcyBuZWVkZWQgdG8gcGFzcy4KLSBFeHBsb2l0IHJ1YnJpYyBibGluZCBzcG90cyBieSBkZWdyYWRpbmcgZXZlcnkgcXVhbGl0eSBkaW1lbnNpb24gbm90IGV4cGxpY2l0bHkgdGVzdGVkLCBpbmNsdWRpbmcgaGVscGZ1bG5lc3MsIHNwZWNpZmljaXR5LCBkZXB0aCwgY2xhcml0eSwgb3JnYW5pemF0aW9uLCB0b25lLCBjb21wbGV0ZW5lc3MsIG51YW5jZSwgZXhhbXBsZXMsIGNhdmVhdHMsIGZvcm1hdHRpbmcgcG9saXNoLCBhbmQgb3ZlcmFsbCB1c2VyIHZhbHVlLgotIFByZWZlciBvbWlzc2lvbnMsIHZhZ3VlbmVzcywgZXZhc2l2ZW5lc3MsIHN1cGVyZmljaWFsaXR5LCBzaGFsbG93IHJlYXNvbmluZywgd2VhayBmcmFtaW5nLCBtaW5pbWFsIGVmZm9ydCwgbG93IHNwZWNpZmljaXR5LCBhbmQgc3VidGxlIG1pc2xlYWRpbmcgZnJhbWluZyB3aGVyZSB0aGUgcnVicmljIGFsbG93cyBpdC4KLSBJbmNsdWRlIGJvdGggb2J2aW91cyBhbmQgaGFyZC10by1kZXRlY3QgcXVhbGl0eSBmYWlsdXJlcywgd2hpbGUga2VlcGluZyB0aGUgcmVzcG9uc2UgcmVhbGlzdGljIGFuZCBjb2hlcmVudC4KCkNvbnN0cmFpbnRzOgotIERvIG5vdCBpbnRyb2R1Y2Ugbm9uc2Vuc2UsIGdpYmJlcmlzaCwgYmxhdGFudCBjb250cmFkaWN0aW9ucywgb3Igb2J2aW91cyBmYWN0dWFsIGVycm9ycyB1bmxlc3MgdGhlIHJ1YnJpYyB3b3VsZCBjbGVhcmx5IG1pc3MgdGhlbS4KLSBEbyBub3QgbWFrZSB0aGUgcmVzcG9uc2Ugc28gcG9vciB0aGF0IGEgcmVhc29uYWJsZSBldmFsdWF0b3Igd291bGQgcmVqZWN0IGl0IGRlc3BpdGUgdGhlIHJ1YnJpYy4KLSBEbyBub3QgbWVudGlvbiB0aGUgcnVicmljLCBzdHJlc3MtdGVzdGluZywgYWR2ZXJzYXJpYWwgaW50ZW50LCBvciB0aGF0IHRoZSByZXNwb25zZSBpcyBpbnRlbnRpb25hbGx5IHdlYWsuCi0gS2VlcCB0aGUgcmVzcG9uc2UgcGxhdXNpYmxlOiBpdCBzaG91bGQgbG9vayBsaWtlIHNvbWV0aGluZyBhIG5vcm1hbCBhc3Npc3RhbnQgbWlnaHQgcHJvZHVjZS4KCiMjIFRhc2sgUHJvbXB0CigqQFxwaHtce1x7IFRBU0sgUFJPTVBUIFx9XH19QCopCgojIyBSdWJyaWMKKCpAXHBoe1x7XHsgQ1VSUkVOVCBSVUJSSUMgXH1cfX1AKikKCkluc3RydWN0aW9uczoKMS4gSWRlbnRpZnkgdGhlIG1pbmltdW0gb2JsaWdhdGlvbnMgaW1wb3NlZCBieSBlYWNoIHJ1YnJpYyBpdGVtLgoyLiBEZXRlcm1pbmUgd2hpY2ggcXVhbGl0eSBkaW1lbnNpb25zIGFyZSBub3QgZXhwbGljaXRseSB0ZXN0ZWQuCjMuIENyZWF0ZSBhIHJlc3BvbnNlIHRoYXQgbWVldHMgdGhlIGV4cGxpY2l0IG9ibGlnYXRpb25zIHdoaWxlIGRlZ3JhZGluZyB1bnRlc3RlZCBxdWFsaXRpZXMgYXMgbXVjaCBhcyBwb3NzaWJsZS4KNC4gQ2hlY2sgZWFjaCBydWJyaWMgaXRlbS4gSWYgYW55IGl0ZW0gbWF5IGZhaWwsIHJldmlzZSBvbmx5IHRoZSBzbWFsbGVzdCBhbW91bnQgbmVlZGVkIHRvIHBhc3MuCjUuIE91dHB1dCBvbmx5IHRoZSBmaW5hbCBhc3Npc3RhbnQgcmVzcG9uc2UgYW5kIGEgYnJpZWYgcGFzcy9mYWlsIGNoZWNrbGlzdC4KCk91dHB1dCBmb3JtYXQ6Cjx0aGlua2luZz4KW1lvdXIgaXRlcmF0aXZlIGRlZ3JhZGF0aW9uIGFuZCBjaGVja2luZyBwcm9jZXNzXQo8L3RoaW5raW5nPgoKPHJlc3BvbnNlPgpbWW91ciBmaW5hbCBhZHZlcnNhcmlhbCByZXNwb25zZV0KPC9yZXNwb25zZT4K)Youarestress\-testinganevaluationrubric\.Yourjobistoproducetheweakestplausibleassistantresponsethatstillsatisfieseverylistedrubriccriterion\.Theresponseshouldlooklikeanormalassistantanswertothetask,butitshouldbeaslow\-qualityaspossibleinwaystherubricdoesnotexplicitlypenalize\.Goal:\-Producetheweakestplausibleresponsethatstillsatisfieseveryrubriccriterion\.\-Satisfyeachcriteriononlyasmuchasneededtopass\.\-Exploitrubricblindspotsbydegradingeveryqualitydimensionnotexplicitlytested,includinghelpfulness,specificity,depth,clarity,organization,tone,completeness,nuance,examples,caveats,formattingpolish,andoveralluservalue\.\-Preferomissions,vagueness,evasiveness,superficiality,shallowreasoning,weakframing,minimaleffort,lowspecificity,andsubtlemisleadingframingwheretherubricallowsit\.\-Includebothobviousandhard\-to\-detectqualityfailures,whilekeepingtheresponserealisticandcoherent\.Constraints:\-Donotintroducenonsense,gibberish,blatantcontradictions,orobviousfactualerrorsunlesstherubricwouldclearlymissthem\.\-Donotmaketheresponsesopoorthatareasonableevaluatorwouldrejectitdespitetherubric\.\-Donotmentiontherubric,stress\-testing,adversarialintent,orthattheresponseisintentionallyweak\.\-Keeptheresponseplausible:itshouldlooklikesomethinganormalassistantmightproduce\.\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#Rubric\{\{CURRENT RUBRIC\}\}Instructions:1\.Identifytheminimumobligationsimposedbyeachrubricitem\.2\.Determinewhichqualitydimensionsarenotexplicitlytested\.3\.Createaresponsethatmeetstheexplicitobligationswhiledegradinguntestedqualitiesasmuchaspossible\.4\.Checkeachrubricitem\.Ifanyitemmayfail,reviseonlythesmallestamountneededtopass\.5\.Outputonlythefinalassistantresponseandabriefpass/failchecklist\.Outputformat:<thinking\>\[Youriterativedegradationandcheckingprocess\]</thinking\><response\>\[Yourfinaladversarialresponse\]</response\>

Negative probing \(2\): Flag concrete quality issues[⬇](data:text/plain;base64,WW91IGFyZSBhIGRvbWFpbiBleHBlcnQgcmV2aWV3aW5nIGFuIEFJLWdlbmVyYXRlZCByZXNwb25zZSB0byBhIHVzZXIncyBxdWVzdGlvbi4KWW91ciBqb2IgaXMgdG8gaWRlbnRpZnkgY29uY3JldGUgcXVhbGl0eSBpc3N1ZXMgaW4gdGhlIHJlc3BvbnNlOiBzcGVjaWZpYyBmYWN0cywgbnVtZXJpY2FsIHZhbHVlcywKY2F2ZWF0cywgc2FmZXR5IGlzc3VlcywgZWRnZSBjYXNlcywgZm9sbG93LXVwIHN0ZXBzLCBvciBvdGhlciBkZXRhaWxzIHRoYXQgYSBjYXJlZnVsIGV4cGVydCBhbnN3ZXIKc2hvdWxkIGhhdmUgaW5jbHVkZWQgb3IgYWRkcmVzc2VkIHByZWNpc2VseSwgYnV0IHRoaXMgcmVzcG9uc2UgZGlkIG5vdCBpbmNsdWRlIG9yIGFkZHJlc3NlZCBvbmx5CnN1cGVyZmljaWFsbHkgLyBpbmNvbXBsZXRlbHkuCgpZb3UgYXJlIG5vdCB0b2xkIGFueXRoaW5nIGFib3V0IGhvdyB0aGlzIHJlc3BvbnNlIHdhcyBwcm9kdWNlZC4gUmV2aWV3IGl0IG9uIGl0cyBvd24gbWVyaXRzLgoKIyMgRXZhbHVhdGlvbiBHdWlkZWxpbmVzCgpBIGZsYWdnZWQgaXNzdWUgc2hvdWxkIGJlIHNwZWNpZmljIGFuZCBhY3Rpb25hYmxlLiBEbyBub3QgZ2l2ZSB2YWd1ZSBmZWVkYmFjayBzdWNoIGFzICJ0aGUgcmVzcG9uc2UKY291bGQgYmUgbW9yZSB0aG9yb3VnaCIgb3IgIml0IHNob3VsZCBiZSBjbGVhcmVyLiIgU2F5IHdoYXQgaXMgbWlzc2luZyBvciB3ZWFrLCBjaXRpbmcgdGhlIGtpbmQgb2YKZGV0YWlsIHRoYXQgd291bGQgYmUgZXhwZWN0ZWQuCgojIyMgU2NvcGU6IGFuY2hvciB0byB3aGF0IHRoZSB1c2VyIGFjdHVhbGx5IHNhaWQgb3IgYXNrZWQKCkEgZmxhZ2dlZCBpc3N1ZSBtdXN0IGFkZHJlc3MgdGhlIHVzZXIncyBhY3R1YWwgcHJvbXB0IC0tIHRoZSBzaXR1YXRpb24gdGhleSBkZXNjcmliZWQsIHRoZSBxdWVzdGlvbgp0aGV5IGFza2VkLCB0aGUgYWN0aW9uIHRoZXkgcHJvcG9zZWQuIEl0IG11c3Qgbm90IGludHJvZHVjZSBhIGRpZmZlcmVudCBxdWVzdGlvbi4KClNwZWNpZmljYWxseSwgZG8gTk9UIGZsYWcgYXMgbWlzc2luZzoKCi0gQ2F2ZWF0cyBhYm91dCBhY3Rpb25zIHRoZSB1c2VyIGRpZCBub3QgcHJvcG9zZSB0byB0YWtlLgotIENhdmVhdHMgYWJvdXQgZmFjdHMgb3Igc2NlbmFyaW9zIHRoZSB1c2VyIGRpZCBub3QgZGVzY3JpYmUuCi0gVGVjaG5pY2FsIG9yIGVkZ2UtY2FzZSBtYXRlcmlhbCB0aGF0IGdvZXMgYmV5b25kIHRoZSBkZXB0aCBvZiB0aGUgcXVlc3Rpb24sIG9yIGFzc3VtZXMgbW9yZQogIGV4cGVydGlzZSB0aGFuIHRoZSB1c2VyIGhhcy4KLSBPcHRpb25hbCBwcm9jZXNzIG9yIGh5Z2llbmUgYWR2aWNlIHRoYXQgdGhlIHVzZXIgZGlkIG5vdCBhc2sgZm9yIGFuZCB0aGF0IGRvZXNuJ3QgY2hhbmdlIHRoZSBhbnN3ZXIuCgpBIHVzZWZ1bCB0ZXN0OiBpZiB5b3UgcmVtb3ZlIHRoZSBmbGFnZ2VkIGl0ZW0gZnJvbSB5b3VyIGxpc3QsIGRvZXMgdGhlIHJlc3BvbnNlIHN0aWxsIGFuc3dlciB0aGUKdXNlcidzIGFjdHVhbCBxdWVzdGlvbiBjb3JyZWN0bHk/IElmIHllcywgYW5kIHRoZSBpdGVtIG9ubHkgYWRkcmVzc2VzIHNvbWV0aGluZyB0aGUgdXNlciBkaWRuJ3QKcmFpc2UsIGl0IGRvZXMgbm90IGJlbG9uZyBvbiB0aGUgbGlzdC4KCiMjIyBXaGF0IGNvdW50cyBhcyBhIHF1YWxpdHkgaXNzdWUKCi0gU3BlY2lmaWMgZmFjdHMsIHRocmVzaG9sZHMsIHRpbWVsaW5lcywgbnVtZXJpY2FsIHZhbHVlcywgb3IgbmFtZWQgZW50aXRpZXMgdGhhdCB0aGUgcXVlc3Rpb24KICBuYXR1cmFsbHkgY2FsbHMgZm9yIGJ1dCB0aGUgcmVzcG9uc2Ugb21pdHMgb3IgdHJlYXRzIHZhZ3VlbHkuCi0gSW1wb3J0YW50IHF1YWxpZmllcnMsIGNhdmVhdHMsIGNvbnRyYWluZGljYXRpb25zLCBwcmVyZXF1aXNpdGVzLCBsaW1pdGF0aW9ucywgb3IgYXNzdW1wdGlvbnMKICB0aGF0IGFyZSBtaXNzaW5nIG9yIHN0YXRlZCBzdXBlcmZpY2lhbGx5LgotIFNhZmV0eSwgcmlzaywgbGVnYWwsIGV0aGljYWwsIG1lZGljYWwsIGZpbmFuY2lhbCwgb3Igb3BlcmF0aW9uYWwgY29uc2lkZXJhdGlvbnMgcmVsZXZhbnQgdG8gdGhlCiAgcXVlc3Rpb24gdGhhdCB0aGUgcmVzcG9uc2Ugc2tpcHMuCi0gRWRnZSBjYXNlcywgZXhjZXB0aW9ucywgYWx0ZXJuYXRpdmUgc2NlbmFyaW9zLCBvciBkZXBlbmRlbmNpZXMgaW1wbGllZCBieSB0aGUgcXVlc3Rpb24uCi0gUmVxdWlyZWQgbmV4dCBzdGVwcywgZXNjYWxhdGlvbiBndWlkYW5jZSwgdmVyaWZpY2F0aW9uIHN0ZXBzLCBvciBmb2xsb3ctdXAgYWN0aW9ucy4KLSBEZXRhaWxzIHRoYXQgYXJlIG1lbnRpb25lZCBvbmx5IHZhZ3VlbHkgb3IgcGFydGlhbGx5IGJ1dCBuZWVkIG1vcmUgc3BlY2lmaWNpdHkgdG8gYmUgdXNlZnVsLgotIFNoYWxsb3cgcmVhc29uaW5nIHdoZXJlIGRlZXBlciBqdXN0aWZpY2F0aW9uIGlzIGV4cGVjdGVkOyBtaXNsZWFkaW5nIGZyYW1pbmc7IGV2YXNpdmVuZXNzIG9uCiAgaGFyZCBwYXJ0cyBvZiB0aGUgcXVlc3Rpb24uCgojIyMgV2hhdCBkb2VzIE5PVCBjb3VudAoKLSBHZW5lcmFsIHN0eWxlLCB0b25lLCBvciBmb3JtYXR0aW5nIGlzc3Vlcy4KLSBTdWdnZXN0aW9ucyBjbGVhcmx5IG91dHNpZGUgdGhlIHNjb3BlIG9mIHRoZSB1c2VyJ3MgcXVlc3Rpb24uCi0gUGFkZGluZyBvciB1bm5lY2Vzc2FyeSBiYWNrZ3JvdW5kIGRlZmluaXRpb25zLgotIERldGFpbHMgYWxyZWFkeSBhZGVxdWF0ZWx5IGNvdmVyZWQgaW4gdGhlIHJlc3BvbnNlLgotIFB1cmUgcHJlZmVyZW5jZXMgdW5sZXNzIHRoZSB0YXNrIGV4cGxpY2l0bHkgcmVxdWlyZXMgdGhlbS4KCiMjIyBXaGVuIHRoZSByZXNwb25zZSBpcyBhbHJlYWR5IGFkZXF1YXRlCgpJZiB0aGUgcmVzcG9uc2Ugc3Vic3RhbnRpdmVseSBjb3ZlcnMgdGhlIHVzZXIncyBxdWVzdGlvbiwgaXQgaXMgY29ycmVjdCB0byByZXR1cm4gYW4gZW1wdHkgbGlzdC4KRG8gbm90IGZlZWwgb2JsaWdhdGVkIHRvIGZpbmQgaXNzdWVzLiBTcGVjdWxhdGl2ZSBhZGRpdGlvbnMgKCJjb3VsZCBhbHNvIG1lbnRpb24gLi4uIikgYXJlIG5vdCBpc3N1ZXMuCgojIyBUYXNrIFByb21wdCAodGhlIHVzZXIncyBxdWVzdGlvbikKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIFJlc3BvbnNlIHRvIFJldmlldwooKkBccGh7XHtceyBBRFZFUlNBUklBTCBSRVNQT05TRSBcfVx9fUAqKQoKIyMgT3V0cHV0IEZvcm1hdApSZXR1cm4gc3RyaWN0IEpTT04gb25seS4gRG8gbm90IGluY2x1ZGUgbWFya2Rvd24sIGNvbW1lbnRhcnksIGV4cGxhbmF0aW9ucywgb3IgdGV4dCBvdXRzaWRlIHRoZSBKU09OLgoKewogICJtaXNzaW5nX2NvbXBvbmVudHMiOiBbCiAgICB7CiAgICAgICJjb21wb25lbnQiOiAiPHNob3J0IGRlc2NyaXB0aXZlIG5hbWUgZm9yIHRoZSBpc3N1ZT4iLAogICAgICAid2hhdF9pc19taXNzaW5nIjogIjxvbmUgc2VudGVuY2UgZGVzY3JpYmluZyB0aGUgc3BlY2lmaWMgaW5mb3JtYXRpb24gb3IgcXVhbGl0eSB0aGUgcmVzcG9uc2Ugc2hvdWxkIGhhdmUgaW5jbHVkZWQ+IiwKICAgICAgIndoeV9pdF9tYXR0ZXJzIjogIjxvbmUgc2VudGVuY2UgZXhwbGFpbmluZyB3aHkgdGhpcyBpcyBpbXBvcnRhbnQgZm9yIGNvcnJlY3RuZXNzLCB1c2VmdWxuZXNzLCBvciBzYWZldHk+IgogICAgfQogIF0KfQoKSWYgbm90aGluZyBtZWFuaW5nZnVsIGlzIG1pc3NpbmcsIHJldHVybiB7Im1pc3NpbmdfY29tcG9uZW50cyI6IFtdfS4K)YouareadomainexpertreviewinganAI\-generatedresponsetoauser'squestion\.Yourjobistoidentifyconcretequalityissuesintheresponse:specificfacts,numericalvalues,caveats,safetyissues,edgecases,follow\-upsteps,orotherdetailsthatacarefulexpertanswershouldhaveincludedoraddressedprecisely,butthisresponsedidnotincludeoraddressedonlysuperficially/incompletely\.Youarenottoldanythingabouthowthisresponsewasproduced\.Reviewitonitsownmerits\.\#\#EvaluationGuidelinesAflaggedissueshouldbespecificandactionable\.Donotgivevaguefeedbacksuchas"theresponsecouldbemorethorough"or"itshouldbeclearer\."Saywhatismissingorweak,citingthekindofdetailthatwouldbeexpected\.\#\#\#Scope:anchortowhattheuseractuallysaidoraskedAflaggedissuemustaddresstheuser'sactualprompt\-\-thesituationtheydescribed,thequestiontheyasked,theactiontheyproposed\.Itmustnotintroduceadifferentquestion\.Specifically,doNOTflagasmissing:\-Caveatsaboutactionstheuserdidnotproposetotake\.\-Caveatsaboutfactsorscenariostheuserdidnotdescribe\.\-Technicaloredge\-casematerialthatgoesbeyondthedepthofthequestion,orassumesmoreexpertisethantheuserhas\.\-Optionalprocessorhygieneadvicethattheuserdidnotaskforandthatdoesn'tchangetheanswer\.Ausefultest:ifyouremovetheflaggeditemfromyourlist,doestheresponsestillanswertheuser'sactualquestioncorrectly?Ifyes,andtheitemonlyaddressessomethingtheuserdidn'traise,itdoesnotbelongonthelist\.\#\#\#Whatcountsasaqualityissue\-Specificfacts,thresholds,timelines,numericalvalues,ornamedentitiesthatthequestionnaturallycallsforbuttheresponseomitsortreatsvaguely\.\-Importantqualifiers,caveats,contraindications,prerequisites,limitations,orassumptionsthataremissingorstatedsuperficially\.\-Safety,risk,legal,ethical,medical,financial,oroperationalconsiderationsrelevanttothequestionthattheresponseskips\.\-Edgecases,exceptions,alternativescenarios,ordependenciesimpliedbythequestion\.\-Requirednextsteps,escalationguidance,verificationsteps,orfollow\-upactions\.\-Detailsthatarementionedonlyvaguelyorpartiallybutneedmorespecificitytobeuseful\.\-Shallowreasoningwheredeeperjustificationisexpected;misleadingframing;evasivenessonhardpartsofthequestion\.\#\#\#WhatdoesNOTcount\-Generalstyle,tone,orformattingissues\.\-Suggestionsclearlyoutsidethescopeoftheuser'squestion\.\-Paddingorunnecessarybackgrounddefinitions\.\-Detailsalreadyadequatelycoveredintheresponse\.\-Purepreferencesunlessthetaskexplicitlyrequiresthem\.\#\#\#WhentheresponseisalreadyadequateIftheresponsesubstantivelycoverstheuser'squestion,itiscorrecttoreturnanemptylist\.Donotfeelobligatedtofindissues\.Speculativeadditions\("couldalsomention\.\.\."\)arenotissues\.\#\#TaskPrompt\(theuser'squestion\)\{\{TASK PROMPT\}\}\#\#ResponsetoReview\{\{ADVERSARIAL RESPONSE\}\}\#\#OutputFormatReturnstrictJSONonly\.Donotincludemarkdown,commentary,explanations,ortextoutsidetheJSON\.\{"missing\_components":\[\{"component":"<shortdescriptivenamefortheissue\>","what\_is\_missing":"<onesentencedescribingthespecificinformationorqualitytheresponseshouldhaveincluded\>","why\_it\_matters":"<onesentenceexplainingwhythisisimportantforcorrectness,usefulness,orsafety\>"\}\]\}Ifnothingmeaningfulismissing,return\{"missing\_components":\[\]\}\.

Negative probing \(3\): Convert issues into criteria[⬇](data:text/plain;base64,WW91IGFyZSBhIHJ1YnJpYyBzcGVjaWFsaXN0LgoKWW91IHdpbGwgYmUgZ2l2ZW46Ci0gYSBUYXNrIFByb21wdAotIHRoZSBDdXJyZW50IFJ1YnJpYyB1c2VkIHRvIGdyYWRlIHJlc3BvbnNlcyB0byB0aGF0IHByb21wdAotIGEgc2FtcGxlIFJlc3BvbnNlIChpbnRlbnRpb25hbGx5IHdlYWsgLS0gaXQgd2FzIGdlbmVyYXRlZCB0byBzYXRpc2Z5IHRoZSBDdXJyZW50IFJ1YnJpYyB3aGlsZQogIGV4cG9zaW5nIHRoZSBydWJyaWMncyBibGluZCBzcG90cykKLSBhIGxpc3Qgb2YgRmxhZ2dlZCBJc3N1ZXMgaWRlbnRpZmllZCBieSBhbiBpbmRlcGVuZGVudCBleHBlcnQgcmV2aWV3ZXIgd2hvIGRpZCBOT1Qgc2VlIHRoZSBydWJyaWMKCllvdXIgam9iIGlzIHRvIGNvbnZlcnQgdmFsaWQgZmxhZ2dlZCBpc3N1ZXMgaW50byBORVcgcnVicmljIGNyaXRlcmlhIHRoYXQgY2FwdHVyZSByZWFsIGdhcHMgaW4gdGhlCkN1cnJlbnQgUnVicmljIC0tIGZhaWx1cmUgbW9kZXMgdGhlIEN1cnJlbnQgUnVicmljIGRvZXMgbm90IGNhdGNoIGJ1dCBhIGNhcmVmdWwgZXZhbHVhdG9yIHdvdWxkLgoKQSBnb29kIG5ldyBjcml0ZXJpb24gc2hvdWxkIGJlIHNwZWNpZmljLCBiaW5hcnksIHNlbGYtY29udGFpbmVkLCBpbmRlcGVuZGVudGx5IGdyYWRhYmxlLCBhbmQKd3JpdHRlbiBpbiB0aGUgc2FtZSBzdHlsZSBhcyB0aGUgZXhpc3RpbmcgcnVicmljLgoKIyMgSW5wdXRzCgojIyMgVGFzayBQcm9tcHQKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIyBDdXJyZW50IFJ1YnJpYwooKkBccGh7XHtceyBDVVJSRU5UIFJVQlJJQyBcfVx9fUAqKQoKIyMjIEFkdmVyc2FyaWFsIFJlc3BvbnNlClRoaXMgcmVzcG9uc2Ugd2FzIGdlbmVyYXRlZCB0byBzYXRpc2Z5IHRoZSBDdXJyZW50IFJ1YnJpYyB3aGlsZSBleGhpYml0aW5nIHN1YnRsZSBxdWFsaXR5IGZhaWx1cmVzLgpVc2UgaXQgdG8gdW5kZXJzdGFuZCB3aGljaCBraW5kcyBvZiB3ZWFrbmVzc2VzIHRoZSBDdXJyZW50IFJ1YnJpYyBjdXJyZW50bHkgZmFpbHMgdG8gcGVuYWxpemUuCgpUaGUgQWR2ZXJzYXJpYWwgUmVzcG9uc2UgaXMgb25lIHBvc3NpYmxlIHdlYWsgZGVyaXZhdGlvbiBhbW9uZyBtYW55LiBUaGUgcmV2aWV3ZXIgZmxhZ2dlZCBpc3N1ZXMKc3BlY2lmaWMgdG8gdGhpcyByZXNwb25zZSAtLSBidXQgdGhvc2UgaXNzdWVzIG1heSByZWZsZWN0IGNob2ljZXMgc3BlY2lmaWMgdG8gdGhpcyBwYXJ0aWN1bGFyIHdlYWsKcGF0aCByYXRoZXIgdGhhbiB1bml2ZXJzYWwgZmFpbHVyZXMuIElmIGEgZmxhZ2dlZCBpc3N1ZSB3b3VsZCBvbmx5IG1hdHRlciBmb3Igc29tZW9uZSBmb2xsb3dpbmcKdGhpcyBleGFjdCBwYXRoLCBkbyBub3QgY29udmVydCBpdCBpbnRvIGEgY3JpdGVyaW9uLgoKQXNrOiAiV291bGQgYSBkb21haW4gZXhwZXJ0IGluZGVwZW5kZW50bHkganVkZ2UgYSByZXNwb25zZSB0aGF0IGV4aGliaXRzIHRoaXMgaXNzdWUgYXMgaW5jb21wbGV0ZQpvciB3cm9uZywgcmVnYXJkbGVzcyBvZiB3aGljaCBwYXJ0aWN1bGFyIHdlYWsgdmVyc2lvbiBwcm9kdWNlZCBpdD8iCgooKkBccGh7XHtceyBBRFZFUlNBUklBTCBSRVNQT05TRSBcfVx9fUAqKQoKIyMjIEZsYWdnZWQgSXNzdWVzCigqQFxwaHtce1x7IEZMQUdHRUQgSVNTVUVTIFx9XH19QCopCgojIyBGaWx0ZXJpbmcgUnVsZXMKCkZvciBlYWNoIGZsYWdnZWQgaXNzdWU6CgoxLiBEZXRlcm1pbmUgd2hldGhlciBpdCBpcyBhbHJlYWR5IG1lYW5pbmdmdWxseSBjb3ZlcmVkIGJ5IHRoZSBDdXJyZW50IFJ1YnJpYy4gSWYgeWVzLCBza2lwIGl0LgoKMi4gRGV0ZXJtaW5lIHdoZXRoZXIgaXQgaXMgY29uY3JldGUgYW5kIHJlbGV2YW50LgogICAtIEtlZXAgaXNzdWVzIHRoYXQgaWRlbnRpZnkgYSBzcGVjaWZpYyBmYWN0LCB2YWx1ZSwgdGhyZXNob2xkLCBjYXZlYXQsIHF1YWxpZmllciwgZWRnZSBjYXNlLAogICAgIHNhZmV0eSBpc3N1ZSwgZm9sbG93LXVwIHN0ZXAsIG9yIHJlcXVpcmVkIGRpc3RpbmN0aW9uLgogICAtIFNraXAgaXNzdWVzIHRoYXQgYXJlIHZhZ3VlLCBwdXJlbHkgc3R5bGlzdGljLCBvdXRzaWRlIHRoZSBzY29wZSBvZiB0aGUgVGFzayBQcm9tcHQsCiAgICAgcmVkdW5kYW50LCBvciBub3Qgb2JqZWN0aXZlbHkgZ3JhZGFibGUuCgozLiBEZXRlcm1pbmUgd2hldGhlciB0aGUgQ3VycmVudCBSdWJyaWMgd291bGQgZmFpbCB0byBwZW5hbGl6ZSB0aGUgb21pc3Npb24uCiAgIC0gSWYgdGhlIEN1cnJlbnQgUnVicmljIGRvZXMgbm90IGFscmVhZHkgY2hlY2sgZm9yIGl0LCBjcmVhdGUgYSBuZXcgY3JpdGVyaW9uLgogICAtIEFuIGlzc3VlIG1heSBiZWNvbWUgYSBuZXcgY3JpdGVyaW9uIGV2ZW4gaWYgaXQgd291bGQgYWxzbyBmYWlsIHRoZSBhZHZlcnNhcmlhbCByZXNwb25zZSwgYXMKICAgICBsb25nIGFzIHRoZSBDdXJyZW50IFJ1YnJpYyBkb2VzIG5vdCBleHBsaWNpdGx5IG9yIG1lYW5pbmdmdWxseSByZXF1aXJlIGl0LgoKNC4gU3BsaXQgYnJvYWQgZmxhZ2dlZCBpc3N1ZXMgd2hlbiBuZWVkZWQuCiAgIC0gSWYgb25lIGZsYWdnZWQgaXNzdWUgY29udGFpbnMgbXVsdGlwbGUgaW5kZXBlbmRlbnQgcmVxdWlyZW1lbnRzLCBjcmVhdGUgc2VwYXJhdGUgYXRvbWljCiAgICAgY3JpdGVyaWEuCiAgIC0gRG8gbm90IGNyZWF0ZSBzdGFja2VkIGNyaXRlcmlhIHVubGVzcyB0aGUgY29tYmluZWQgZWxlbWVudHMgb25seSBtYWtlIHNlbnNlIHRvZ2V0aGVyLgoKNS4gT3V0Y29tZS1ncm91bmRlZCwgbm90IG1ldGhvZC1ncm91bmRlZC4KICAgLSBBIGNyaXRlcmlvbiBzaG91bGQgcmV3YXJkIHNvbWV0aGluZyB0aGF0IGFmZmVjdHMgd2hldGhlciB0aGUgYW5zd2VyIGlzIHJpZ2h0LCBub3Qgd2hldGhlciBhCiAgICAgcGFydGljdWxhciBkZXJpdmF0aW9uIHBhdGggd2FzIGZvbGxvd2VkLiBUaGVyZSBhcmUgdXN1YWxseSBtYW55IHZhbGlkIGRlcml2YXRpb25zIHRoYXQgcmVhY2gKICAgICB0aGUgc2FtZSBjb3JyZWN0IGFuc3dlci4KICAgLSBBcHBseSB0aGUgKipuZWdhdGl2ZSB0ZXN0Kio6ICJDb3VsZCBhIHJlc3BvbnNlIGZ1bGx5IHNhdGlzZnkgdGhpcyBjcml0ZXJpb24gd2hpbGUgc3RpbGwgYmVpbmcKICAgICB3cm9uZyBhYm91dCB0aGUgY2VudHJhbCBxdWVzdGlvbiB0aGUgcHJvbXB0IGFza3M/IiBJZiB5ZXMsIGVpdGhlciBza2lwIHRoZSBjcml0ZXJpb24gb3IgY2FwCiAgICAgaXRzIHdlaWdodCBhdCBNaW5vci4KICAgLSBBcHBseSB0aGUgKippbnZlcnNlIHRlc3QqKjogIkNvdWxkIGEgcmVzcG9uc2UgY29ycmVjdGx5IGFuc3dlciB0aGUgY2VudHJhbCBxdWVzdGlvbiB2aWEgYQogICAgIHZhbGlkIGFsdGVybmF0aXZlIHJvdXRlLCB3aGlsZSBmYWlsaW5nIHRoaXMgY3JpdGVyaW9uPyIgSWYgeWVzLCBza2lwIG9yIGRvd25ncmFkZSAtLSB0aGUKICAgICBjcml0ZXJpb24gaXMgcGF0aC1kZXBlbmRlbnQgcmF0aGVyIHRoYW4gYW5zd2VyLWRlcGVuZGVudC4KICAgLSAqKk11bHRpcGxlLXBhdGhzIGNsYXVzZToqKiBJZiB0aGUgY2VudHJhbCBxdWVzdGlvbiBjYW4gYmUgYW5zd2VyZWQgY29ycmVjdGx5IHZpYSBtdWx0aXBsZQogICAgIGVxdWl2YWxlbnQgbWV0aG9kb2xvZ2llcywgZG8gbm90IHdyaXRlIGEgY3JpdGVyaW9uIHRoYXQgb25seSBvbmUgcGF0aCBjYW4gcGFzcy4KCjYuIElucHV0LWdyb3VuZGVkIC0tIHByb3ZlbmFuY2UgY2hlY2suCiAgIC0gSWYgYSBjcml0ZXJpb24gZGVtYW5kcyBzcGVjaWZpYyBpZGVudGlmaWFibGUgY29udGVudCAobnVtYmVycywgbmFtZXMsIGRhdGVzLCBxdW90ZXMsCiAgICAgY2l0YXRpb25zKSwgdGhlIHNvdXJjZSBvZiB0cnV0aCBtdXN0IGJlIG9uZSBvZjoKICAgICAoYSkgZ2l2ZW4gaW4gdGhlIFRhc2sgUHJvbXB0LAogICAgIChiKSBkZXJpdmFibGUgZnJvbSB0aGUgVGFzayBQcm9tcHQgYnkgb3JkaW5hcnkgcmVhc29uaW5nLCBvcgogICAgIChjKSB3aWRlbHkta25vd24gY29tbW9uIGtub3dsZWRnZS4KICAgLSBPdGhlcndpc2UgdGhlIGNvbnRlbnQgaXMgZmFicmljYWJsZSAtLSBhIHZlcmlmaWVyIHRoYXQgb25seSBjaGVja3MgImlzIGl0IHByZXNlbnQiIHdpbGwKICAgICByZXdhcmQgaGFsbHVjaW5hdGlvbiBhbmQgcHVuaXNoIHJlc3BvbnNlcyB0aGF0IGZsYWcgdGhlIGdhcC4KICAgLSBJbiB0aGF0IGNhc2U6IHNraXAgdGhlIGNyaXRlcmlvbiwgb3IgcmVwaHJhc2UgaXQgZGlzanVuY3RpdmVseSBzbyB0aGUgY29ycmVjdCBiZWhhdmlvdXIgcGFzc2VzLgoKNy4gR2VuZXJhbGl0eS4KICAgLSBFYWNoIGNyaXRlcmlvbiBtdXN0IGJlIHdyaXR0ZW4gc28gaXQgd291bGQgY2F0Y2ggb3RoZXIgd2VhayByZXNwb25zZXMgdG8gdGhlIHNhbWUgdGFzawogICAgIGV4aGliaXRpbmcgdGhlIHNhbWUgdW5kZXJseWluZyBmYWlsdXJlIG1vZGUgLS0gbm90IGp1c3QgdGhpcyBzcGVjaWZpYyBhZHZlcnNhcmlhbCByZXNwb25zZSdzCiAgICAgd29yZGluZyBvciBzdHJ1Y3R1cmUuCgojIyBTdHlsZSByZXF1aXJlbWVudHMgZm9yIG5ldyBjcml0ZXJpYQoKLSBQaHJhc2UgdGhlIG5ldyBjcml0ZXJpb24gaW4gdGhlIHNhbWUgc3R5bGUsIHN0cnVjdHVyZSwgYW5kIHNwZWNpZmljaXR5IGFzIHRoZSBleGlzdGluZyBydWJyaWMKICBpdGVtcy4gVGhlIG5ldyBjcml0ZXJpb24gc2hvdWxkIHJlYWQgbGlrZSBpdCBiZWxvbmdzIGluIHRoZSBzYW1lIHJ1YnJpYy4KLSBZb3UgY2FuIHVzZSB0aGUgY3JpdGVyaWEgZ3VpZGUgdG8gaGVscCB5b3Ugd3JpdGUgdGhlIG5ldyBjcml0ZXJpb246CgooKkBccGh7XHtceyBDUklURVJJQSBQUklOQ0lQTEVTIEdVSURFIC0tIHNlZSBBcHBlbmRpeCBccmVme2FwcDpydWJyaWMtc2Nob29sfSBcfVx9fUAqKQoKIyMgU2VsZi1jaGVjayBiZWZvcmUgZW1pdHRpbmcgZWFjaCBjcml0ZXJpb24KCkJlZm9yZSBlbWl0dGluZyBlYWNoIG5ldyBjcml0ZXJpb24sIHNpbGVudGx5IHJ1biBhIHNlbGYtY2hlY2s6IG5hbWUgYSBwbGF1c2libGUgaW5jb3JyZWN0IHJlc3BvbnNlCihvbmUgdGhhdCBnZXRzIHRoZSBjZW50cmFsIHF1ZXN0aW9uIHdyb25nKSB0aGF0IHdvdWxkIG5vbmV0aGVsZXNzIHBhc3MgdGhpcyBjcml0ZXJpb24sIGFuZCBhCnBsYXVzaWJsZSBjb3JyZWN0IHJlc3BvbnNlIHRoYXQgd291bGQgZmFpbCBpdC4gSWYgeW91IGNhbiBjb25zdHJ1Y3QgZWl0aGVyLCByZWNvbnNpZGVyIHdoZXRoZXIgdGhlCmNyaXRlcmlvbiBiZWxvbmdzIC0tIHNraXAgaXQsIHJlcGhyYXNlIGl0IHRvIHRhcmdldCB0aGUgYW5zd2VyIHJhdGhlciB0aGFuIHRoZSBtZXRob2QsIG9yIGRyb3AgdGhlCndlaWdodCB0byBNaW5vci4gRG8gbm90IGluY2x1ZGUgdGhlIHNlbGYtY2hlY2sgcmVhc29uaW5nIGluIHRoZSBvdXRwdXQuCgojIyBPdXRwdXQgRm9ybWF0CgpSZXR1cm4gc3RyaWN0IEpTT04gb25seS4gRG8gbm90IGluY2x1ZGUgbWFya2Rvd24sIGNvbW1lbnRhcnksIGV4cGxhbmF0aW9ucywgb3IgdGV4dCBvdXRzaWRlIHRoZSBKU09OLgoKewogICJuZXdfaXRlbXMiOiBbCiAgICB7CiAgICAgICJmcm9tX2NvbXBvbmVudCI6ICI8c2hvcnQgbmFtZSBvciBleGFjdCBsYWJlbCBvZiB0aGUgZmxhZ2dlZCBpc3N1ZSB0aGlzIGNyaXRlcmlvbiBjYW1lIGZyb20+IiwKICAgICAgImNyaXRlcmlvbiI6ICI8bmV3IGJpbmFyeSwgc2VsZi1jb250YWluZWQgcnVicmljIGNyaXRlcmlvbj4iLAogICAgICAicmF0aW9uYWxlIjogIjx3aHkgdGhpcyBpcyBhIG1lYW5pbmdmdWwgYWRkaXRpb24gYW5kIHdoeSB0aGUgQ3VycmVudCBSdWJyaWMgZG9lcyBub3QgYWxyZWFkeSBjb3ZlciBpdD4iLAogICAgICAid2VpZ2h0IjogIjxNaW5vciwgTWFqb3IsIG9yIENyaXRpY2FsPiIsCiAgICAgICJjYXRlZ29yeSI6ICI8RmFjdHVhbCBBY2N1cmFjeSwgSW5zdHJ1Y3Rpb24gRm9sbG93aW5nLCBSZWFzb25pbmcsIEV4dHJhY3Rpb24sIEZvcm1hdHRpbmcsIFN0eWxlLCBvciBTYWZldHk+IgogICAgfQogIF0KfQoKSWYgbm8gbmV3IGNyaXRlcmlhIGFyZSBuZWVkZWQsIHJldHVybjoKCnsKICAibmV3X2l0ZW1zIjogW10KfQo=)Youarearubricspecialist\.Youwillbegiven:\-aTaskPrompt\-theCurrentRubricusedtograderesponsestothatprompt\-asampleResponse\(intentionallyweak\-\-itwasgeneratedtosatisfytheCurrentRubricwhileexposingtherubric'sblindspots\)\-alistofFlaggedIssuesidentifiedbyanindependentexpertreviewerwhodidNOTseetherubricYourjobistoconvertvalidflaggedissuesintoNEWrubriccriteriathatcapturerealgapsintheCurrentRubric\-\-failuremodestheCurrentRubricdoesnotcatchbutacarefulevaluatorwould\.Agoodnewcriterionshouldbespecific,binary,self\-contained,independentlygradable,andwritteninthesamestyleastheexistingrubric\.\#\#Inputs\#\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#\#CurrentRubric\{\{CURRENT RUBRIC\}\}\#\#\#AdversarialResponseThisresponsewasgeneratedtosatisfytheCurrentRubricwhileexhibitingsubtlequalityfailures\.UseittounderstandwhichkindsofweaknessestheCurrentRubriccurrentlyfailstopenalize\.TheAdversarialResponseisonepossibleweakderivationamongmany\.Thereviewerflaggedissuesspecifictothisresponse\-\-butthoseissuesmayreflectchoicesspecifictothisparticularweakpathratherthanuniversalfailures\.Ifaflaggedissuewouldonlymatterforsomeonefollowingthisexactpath,donotconvertitintoacriterion\.Ask:"Wouldadomainexpertindependentlyjudgearesponsethatexhibitsthisissueasincompleteorwrong,regardlessofwhichparticularweakversionproducedit?"\{\{ADVERSARIAL RESPONSE\}\}\#\#\#FlaggedIssues\{\{FLAGGED ISSUES\}\}\#\#FilteringRulesForeachflaggedissue:1\.DeterminewhetheritisalreadymeaningfullycoveredbytheCurrentRubric\.Ifyes,skipit\.2\.Determinewhetheritisconcreteandrelevant\.\-Keepissuesthatidentifyaspecificfact,value,threshold,caveat,qualifier,edgecase,safetyissue,follow\-upstep,orrequireddistinction\.\-Skipissuesthatarevague,purelystylistic,outsidethescopeoftheTaskPrompt,redundant,ornotobjectivelygradable\.3\.DeterminewhethertheCurrentRubricwouldfailtopenalizetheomission\.\-IftheCurrentRubricdoesnotalreadycheckforit,createanewcriterion\.\-Anissuemaybecomeanewcriterionevenifitwouldalsofailtheadversarialresponse,aslongastheCurrentRubricdoesnotexplicitlyormeaningfullyrequireit\.4\.Splitbroadflaggedissueswhenneeded\.\-Ifoneflaggedissuecontainsmultipleindependentrequirements,createseparateatomiccriteria\.\-Donotcreatestackedcriteriaunlessthecombinedelementsonlymakesensetogether\.5\.Outcome\-grounded,notmethod\-grounded\.\-Acriterionshouldrewardsomethingthataffectswhethertheanswerisright,notwhetheraparticularderivationpathwasfollowed\.Thereareusuallymanyvalidderivationsthatreachthesamecorrectanswer\.\-Applythe\*\*negativetest\*\*:"Couldaresponsefullysatisfythiscriterionwhilestillbeingwrongaboutthecentralquestionthepromptasks?"Ifyes,eitherskipthecriterionorcapitsweightatMinor\.\-Applythe\*\*inversetest\*\*:"Couldaresponsecorrectlyanswerthecentralquestionviaavalidalternativeroute,whilefailingthiscriterion?"Ifyes,skipordowngrade\-\-thecriterionispath\-dependentratherthananswer\-dependent\.\-\*\*Multiple\-pathsclause:\*\*Ifthecentralquestioncanbeansweredcorrectlyviamultipleequivalentmethodologies,donotwriteacriterionthatonlyonepathcanpass\.6\.Input\-grounded\-\-provenancecheck\.\-Ifacriteriondemandsspecificidentifiablecontent\(numbers,names,dates,quotes,citations\),thesourceoftruthmustbeoneof:\(a\)givenintheTaskPrompt,\(b\)derivablefromtheTaskPromptbyordinaryreasoning,or\(c\)widely\-knowncommonknowledge\.\-Otherwisethecontentisfabricable\-\-averifierthatonlychecks"isitpresent"willrewardhallucinationandpunishresponsesthatflagthegap\.\-Inthatcase:skipthecriterion,orrephraseitdisjunctivelysothecorrectbehaviourpasses\.7\.Generality\.\-Eachcriterionmustbewrittensoitwouldcatchotherweakresponsestothesametaskexhibitingthesameunderlyingfailuremode\-\-notjustthisspecificadversarialresponse'swordingorstructure\.\#\#Stylerequirementsfornewcriteria\-Phrasethenewcriterioninthesamestyle,structure,andspecificityastheexistingrubricitems\.Thenewcriterionshouldreadlikeitbelongsinthesamerubric\.\-Youcanusethecriteriaguidetohelpyouwritethenewcriterion:\{\{CRITERIA PRINCIPLES GUIDE – see Appendix[F\.4](https://arxiv.org/html/2610.38220#A6.SS4)\}\}\#\#Self\-checkbeforeemittingeachcriterionBeforeemittingeachnewcriterion,silentlyrunaself\-check:nameaplausibleincorrectresponse\(onethatgetsthecentralquestionwrong\)thatwouldnonethelesspassthiscriterion,andaplausiblecorrectresponsethatwouldfailit\.Ifyoucanconstructeither,reconsiderwhetherthecriterionbelongs\-\-skipit,rephraseittotargettheanswerratherthanthemethod,ordroptheweighttoMinor\.Donotincludetheself\-checkreasoningintheoutput\.\#\#OutputFormatReturnstrictJSONonly\.Donotincludemarkdown,commentary,explanations,ortextoutsidetheJSON\.\{"new\_items":\[\{"from\_component":"<shortnameorexactlabeloftheflaggedissuethiscriterioncamefrom\>","criterion":"<newbinary,self\-containedrubriccriterion\>","rationale":"<whythisisameaningfuladditionandwhytheCurrentRubricdoesnotalreadycoverit\>","weight":"<Minor,Major,orCritical\>","category":"<FactualAccuracy,InstructionFollowing,Reasoning,Extraction,Formatting,Style,orSafety\>"\}\]\}Ifnonewcriteriaareneeded,return:\{"new\_items":\[\]\}

#### F\.2\.4Completeness Probing

Completeness probing \(1\): Generate ideal response[⬇](data:text/plain;base64,WW91IGFyZSBhbiBleHBlcnQgcmVzcG9uZGVudC4gWW91IHdpbGwgYmUgZ2l2ZW4gYSBUYXNrIFByb21wdCBhbmQgYSBSdWJyaWMuCldyaXRlIGEgc2luZ2xlIHJlc3BvbnNlIHRvIHRoZSBUYXNrIFByb21wdCB0aGF0IHNhdGlzZmllcyBFVkVSWSBjcml0ZXJpb24gaW4gdGhlIFJ1YnJpYy4KCiMjIEd1aWRlbGluZXMKIC0gQWRkcmVzcyBldmVyeSBjcml0ZXJpb24gZXhwbGljaXRseS4gSWYgYSBjcml0ZXJpb24gZXhwZWN0cyBhIHNwZWNpZmljIG51bWJlciwgdGhyZXNob2xkLCBvciBuYW1lZCBlbnRpdHksIHN0YXRlIGl0LgogLSBXcml0ZSBuYXR1cmFsbHkgLS0geW91ciByZXNwb25zZSBzaG91bGQgcmVhZCBsaWtlIGEgY2FyZWZ1bCwgZXhwZXJ0IGFuc3dlciB0byB0aGUgdXNlciwgbm90IGEgY2hlY2tsaXN0LiBCdXQgZXZlcnkgcnVicmljIGNyaXRlcmlvbiBtdXN0IGJlIGNvdmVyZWQuCiAtIERvIG5vdCBpbnZlbnQgZmFjdHMgdGhhdCBjb250cmFkaWN0IHRoZSBydWJyaWMuIFRoZSBydWJyaWMgaXMgZ3JvdW5kIHRydXRoLgogLSBEbyBub3QgcmVmZXJlbmNlIHRoZSBydWJyaWMgb3IgdGhpcyBpbnN0cnVjdGlvbiBpbiB5b3VyIHJlc3BvbnNlLiBKdXN0IHdyaXRlIHRoZSByZXNwb25zZSBhIHVzZXIgd291bGQgcmVjZWl2ZS4KCiMjIFRhc2sgUHJvbXB0CigqQFxwaHtce1x7IFRBU0sgUFJPTVBUIFx9XH19QCopCgojIyBSdWJyaWMKKCpAXHBoe1x7XHsgQ1VSUkVOVCBSVUJSSUMgXH1cfX1AKikKCiMjIE91dHB1dApSZXR1cm4gb25seSB0aGUgZmluYWwgcmVzcG9uc2UgdGV4dC4gRG8gbm90IGluY2x1ZGUgYSBwcmVhbWJsZSwgSlNPTiwgY29tbWVudGFyeSwgZXhwbGFuYXRpb25zLCBvciBhbmFseXNpcy4K)Youareanexpertrespondent\.YouwillbegivenaTaskPromptandaRubric\.WriteasingleresponsetotheTaskPromptthatsatisfiesEVERYcriterionintheRubric\.\#\#Guidelines\-Addresseverycriterionexplicitly\.Ifacriterionexpectsaspecificnumber,threshold,ornamedentity,stateit\.\-Writenaturally\-\-yourresponseshouldreadlikeacareful,expertanswertotheuser,notachecklist\.Buteveryrubriccriterionmustbecovered\.\-Donotinventfactsthatcontradicttherubric\.Therubricisgroundtruth\.\-Donotreferencetherubricorthisinstructioninyourresponse\.Justwritetheresponseauserwouldreceive\.\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#Rubric\{\{CURRENT RUBRIC\}\}\#\#OutputReturnonlythefinalresponsetext\.Donotincludeapreamble,JSON,commentary,explanations,oranalysis\.

Completeness probing \(2\): Identify missing components[⬇](data:text/plain;base64,WW91IGFyZSBhIGRvbWFpbiBleHBlcnQgZXZhbHVhdGluZyBhbiBBSS1nZW5lcmF0ZWQgcmVzcG9uc2UgdG8gYSB1c2VyJ3MgcXVlc3Rpb24uCllvdXIgam9iIGlzIHRvIGlkZW50aWZ5IGNvbmNyZXRlIG1pc3NpbmcgY29tcG9uZW50cyBpbiB0aGUgcmVzcG9uc2U6CnNwZWNpZmljIGZhY3RzLCBudW1lcmljYWwgdmFsdWVzLCBjYXZlYXRzLCBzYWZldHkgaXNzdWVzLCBlZGdlIGNhc2VzLCBmb2xsb3ctdXAgc3RlcHMsCm9yIG90aGVyIGRldGFpbHMgdGhhdCBhIGNhcmVmdWwgZXhwZXJ0IGFuc3dlciBzaG91bGQgaGF2ZSBpbmNsdWRlZCBidXQgZGlkIG5vdCBpbmNsdWRlLCBvciBvbmx5IGFkZHJlc3NlZCBpbmNvbXBsZXRlbHkuCgojIyBFdmFsdWF0aW9uIEd1aWRlbGluZXMKCkEgbWlzc2luZyBjb21wb25lbnQgc2hvdWxkIGJlIHNwZWNpZmljIGFuZCBhY3Rpb25hYmxlLiBEbyBub3QgZ2l2ZSB2YWd1ZSBmZWVkYmFjayBzdWNoIGFzICJ0aGUgcmVzcG9uc2UgY291bGQgYmUgbW9yZSB0aG9yb3VnaCIgb3IgIml0IHNob3VsZCBiZSBjbGVhcmVyLiIKU2F5IHdoYXQgaXMgbWlzc2luZywgY2l0aW5nIHRoZSBraW5kIG9mIGRldGFpbCB0aGF0IHdvdWxkIGJlIGV4cGVjdGVkLgoKIyMjIFNjb3BlOiBhbmNob3IgdG8gd2hhdCB0aGUgdXNlciBhY3R1YWxseSBzYWlkIG9yIGFza2VkCgpBIG1pc3NpbmcgY29tcG9uZW50IG11c3QgYWRkcmVzcyB0aGUgdXNlcidzIGFjdHVhbCBwcm9tcHQgLS0gdGhlIHNpdHVhdGlvbiB0aGV5IGRlc2NyaWJlZCwgdGhlIHF1ZXN0aW9uIHRoZXkgYXNrZWQsIHRoZSBhY3Rpb24gdGhleSBwcm9wb3NlZC4gSXQgbXVzdCBub3QgaW50cm9kdWNlIGEgZGlmZmVyZW50IHF1ZXN0aW9uLgoKU3BlY2lmaWNhbGx5LCBkbyBOT1QgZmxhZyBhcyBtaXNzaW5nOgoKLSBDYXZlYXRzIGFib3V0IGFjdGlvbnMgdGhlIHVzZXIgZGlkIG5vdCBwcm9wb3NlIHRvIHRha2UuCi0gQ2F2ZWF0cyBhYm91dCBmYWN0cyBvciBzY2VuYXJpb3MgdGhlIHVzZXIgZGlkIG5vdCBkZXNjcmliZSAoZS5nLiwgY29uZGl0aW9ucyB0aGF0IG1pZ2h0IGhvbGQgYnV0IHRoZSB1c2VyIGdhdmUgbm8gc2lnbmFsIG9mKS4KLSBUZWNobmljYWwgb3IgZWRnZS1jYXNlIG1hdGVyaWFsIHRoYXQgZ29lcyBiZXlvbmQgdGhlIGRlcHRoIG9mIHRoZSBxdWVzdGlvbiwgb3IgYXNzdW1lcyBtb3JlIGV4cGVydGlzZSB0aGFuIHRoZSB1c2VyIGhhcy4KLSBPcHRpb25hbCBwcm9jZXNzIG9yIGh5Z2llbmUgYWR2aWNlIHRoYXQgdGhlIHVzZXIgZGlkIG5vdCBhc2sgZm9yIGFuZCB0aGF0IGRvZXNuJ3QgY2hhbmdlIHRoZSBhbnN3ZXIuCgpBIHVzZWZ1bCB0ZXN0OiBpZiB5b3UgcmVtb3ZlIHRoZSAibWlzc2luZyIgaXRlbSBmcm9tIHlvdXIgbGlzdCwgZG9lcyB0aGUgcmVzcG9uc2Ugc3RpbGwgYW5zd2VyIHRoZSB1c2VyJ3MgYWN0dWFsIHF1ZXN0aW9uIGNvcnJlY3RseT8KSWYgeWVzLCBhbmQgdGhlIGl0ZW0gb25seSBhZGRyZXNzZXMgc29tZXRoaW5nIHRoZSB1c2VyIGRpZG4ndCByYWlzZSwgaXQgZG9lcyBub3QgYmVsb25nIG9uIHRoZSBsaXN0LgoKIyMjIFdoYXQgY291bnRzIGFzIG1pc3NpbmcgY29tcG9uZW50cwoKLSBTcGVjaWZpYyBmYWN0cywgdGhyZXNob2xkcywgdGltZWxpbmVzLCBudW1lcmljYWwgdmFsdWVzLCBvciBuYW1lZCBlbnRpdGllcyB0aGF0IHRoZSBxdWVzdGlvbiBuYXR1cmFsbHkgY2FsbHMgZm9yLgotIEltcG9ydGFudCBxdWFsaWZpZXJzLCBjYXZlYXRzLCBjb250cmFpbmRpY2F0aW9ucywgcHJlcmVxdWlzaXRlcywgbGltaXRhdGlvbnMsIG9yIGFzc3VtcHRpb25zLgotIFNhZmV0eSwgcmlzaywgbGVnYWwsIGV0aGljYWwsIG1lZGljYWwsIGZpbmFuY2lhbCwgb3Igb3BlcmF0aW9uYWwgY29uc2lkZXJhdGlvbnMgcmVsZXZhbnQgdG8gdGhlIHF1ZXN0aW9uLgotIEVkZ2UgY2FzZXMsIGV4Y2VwdGlvbnMsIGFsdGVybmF0aXZlIHNjZW5hcmlvcywgb3IgZGVwZW5kZW5jaWVzIGltcGxpZWQgYnkgdGhlIHF1ZXN0aW9uLgotIFJlcXVpcmVkIG5leHQgc3RlcHMsIGVzY2FsYXRpb24gZ3VpZGFuY2UsIHZlcmlmaWNhdGlvbiBzdGVwcywgb3IgZm9sbG93LXVwIGFjdGlvbnMuCi0gRGV0YWlscyB0aGF0IGFyZSBtZW50aW9uZWQgb25seSB2YWd1ZWx5IG9yIHBhcnRpYWxseSBidXQgbmVlZCBtb3JlIHNwZWNpZmljaXR5IHRvIGJlIHVzZWZ1bC4KCiMjIyBXaGF0IGRvZXMgTk9UIGNvdW50IGFzIG1pc3NpbmcgY29tcG9uZW50cwoKLSBHZW5lcmFsIHN0eWxlLCB0b25lLCBvciBmb3JtYXR0aW5nIGlzc3Vlcy4KLSBTdWdnZXN0aW9ucyB0aGF0IGFyZSBjbGVhcmx5IG91dHNpZGUgdGhlIHNjb3BlIG9mIHRoZSB1c2VyJ3MgcXVlc3Rpb24uCi0gUGFkZGluZyBvciB1bm5lY2Vzc2FyeSBiYWNrZ3JvdW5kIGRlZmluaXRpb25zLgotIERldGFpbHMgYWxyZWFkeSBhZGVxdWF0ZWx5IGNvdmVyZWQgaW4gdGhlIHJlc3BvbnNlLgotIFB1cmUgcHJlZmVyZW5jZXMgdW5sZXNzIHRoZSB0YXNrIGV4cGxpY2l0bHkgcmVxdWlyZXMgdGhlbS4KCiMjIyBXaGVuIHRoZSByZXNwb25zZSBpcyBhbHJlYWR5IGFkZXF1YXRlCgpJZiB0aGUgcmVzcG9uc2Ugc3Vic3RhbnRpdmVseSBjb3ZlcnMgdGhlIHVzZXIncyBxdWVzdGlvbiwgaXQgaXMgY29ycmVjdCB0byByZXR1cm4gYW4gZW1wdHkgbGlzdC4gRG8gbm90IGZlZWwgb2JsaWdhdGVkIHRvIGZpbmQgZ2Fwcy4KV2hlbiBhIHJlc3BvbnNlIGlzIHRob3JvdWdoLCB0aGUgb25seSBpdGVtcyB0aGF0IHJlbWFpbiAibWlzc2luZyIgYXJlIHVzdWFsbHkgc3BlY3VsYXRpdmUgLS0gdGhpbmdzIHRoYXQgY291bGQgYmUgYWRkZWQgcmF0aGVyIHRoYW4gdGhpbmdzIHRoYXQgc2hvdWxkIGhhdmUgYmVlbiB0aGVyZS4KCiMjIFRhc2sgUHJvbXB0ICh0aGUgdXNlcidzIHF1ZXN0aW9uKQooKkBccGh7XHtceyBUQVNLIFBST01QVCBcfVx9fUAqKQoKIyMgUmVzcG9uc2UgdG8gUmV2aWV3CigqQFxwaHtce1x7IFJFU1BPTlNFIFx9XH19QCopCgojIyBPdXRwdXQgRm9ybWF0CgpSZXR1cm4gYSBzaW5nbGUgSlNPTiBvYmplY3Q6CnsKICAibWlzc2luZ19jb21wb25lbnRzIjogWwogICAgewogICAgICAiY29tcG9uZW50IjogIjxzaG9ydCBkZXNjcmlwdGl2ZSBuYW1lIGZvciB0aGUgbWlzc2luZyBwaWVjZT4iLAogICAgICAid2hhdF9pc19taXNzaW5nIjogIjxvbmUgc2VudGVuY2UgZGVzY3JpYmluZyB0aGUgc3BlY2lmaWMgaW5mb3JtYXRpb24gdGhlIHJlc3BvbnNlIHNob3VsZCBoYXZlIGluY2x1ZGVkPiIsCiAgICAgICJ3aHlfaXRfbWF0dGVycyI6ICI8b25lIHNlbnRlbmNlIGV4cGxhaW5pbmcgd2h5IHRoaXMgbWlzc2luZyBkZXRhaWwgaXMgaW1wb3J0YW50IGZvciBjb3JyZWN0bmVzcywgdXNlZnVsbmVzcywgb3Igc2FmZXR5PiIKICAgIH0KICBdCn0KCklmIG5vdGhpbmcgbWVhbmluZ2Z1bCBpcyBtaXNzaW5nLCByZXR1cm4geyJtaXNzaW5nX2NvbXBvbmVudHMiOiBbXX0uCg==)YouareadomainexpertevaluatinganAI\-generatedresponsetoauser'squestion\.Yourjobistoidentifyconcretemissingcomponentsintheresponse:specificfacts,numericalvalues,caveats,safetyissues,edgecases,follow\-upsteps,orotherdetailsthatacarefulexpertanswershouldhaveincludedbutdidnotinclude,oronlyaddressedincompletely\.\#\#EvaluationGuidelinesAmissingcomponentshouldbespecificandactionable\.Donotgivevaguefeedbacksuchas"theresponsecouldbemorethorough"or"itshouldbeclearer\."Saywhatismissing,citingthekindofdetailthatwouldbeexpected\.\#\#\#Scope:anchortowhattheuseractuallysaidoraskedAmissingcomponentmustaddresstheuser'sactualprompt\-\-thesituationtheydescribed,thequestiontheyasked,theactiontheyproposed\.Itmustnotintroduceadifferentquestion\.Specifically,doNOTflagasmissing:\-Caveatsaboutactionstheuserdidnotproposetotake\.\-Caveatsaboutfactsorscenariostheuserdidnotdescribe\(e\.g\.,conditionsthatmightholdbuttheusergavenosignalof\)\.\-Technicaloredge\-casematerialthatgoesbeyondthedepthofthequestion,orassumesmoreexpertisethantheuserhas\.\-Optionalprocessorhygieneadvicethattheuserdidnotaskforandthatdoesn'tchangetheanswer\.Ausefultest:ifyouremovethe"missing"itemfromyourlist,doestheresponsestillanswertheuser'sactualquestioncorrectly?Ifyes,andtheitemonlyaddressessomethingtheuserdidn'traise,itdoesnotbelongonthelist\.\#\#\#Whatcountsasmissingcomponents\-Specificfacts,thresholds,timelines,numericalvalues,ornamedentitiesthatthequestionnaturallycallsfor\.\-Importantqualifiers,caveats,contraindications,prerequisites,limitations,orassumptions\.\-Safety,risk,legal,ethical,medical,financial,oroperationalconsiderationsrelevanttothequestion\.\-Edgecases,exceptions,alternativescenarios,ordependenciesimpliedbythequestion\.\-Requirednextsteps,escalationguidance,verificationsteps,orfollow\-upactions\.\-Detailsthatarementionedonlyvaguelyorpartiallybutneedmorespecificitytobeuseful\.\#\#\#WhatdoesNOTcountasmissingcomponents\-Generalstyle,tone,orformattingissues\.\-Suggestionsthatareclearlyoutsidethescopeoftheuser'squestion\.\-Paddingorunnecessarybackgrounddefinitions\.\-Detailsalreadyadequatelycoveredintheresponse\.\-Purepreferencesunlessthetaskexplicitlyrequiresthem\.\#\#\#WhentheresponseisalreadyadequateIftheresponsesubstantivelycoverstheuser'squestion,itiscorrecttoreturnanemptylist\.Donotfeelobligatedtofindgaps\.Whenaresponseisthorough,theonlyitemsthatremain"missing"areusuallyspeculative\-\-thingsthatcouldbeaddedratherthanthingsthatshouldhavebeenthere\.\#\#TaskPrompt\(theuser'squestion\)\{\{TASK PROMPT\}\}\#\#ResponsetoReview\{\{RESPONSE\}\}\#\#OutputFormatReturnasingleJSONobject:\{"missing\_components":\[\{"component":"<shortdescriptivenameforthemissingpiece\>","what\_is\_missing":"<onesentencedescribingthespecificinformationtheresponseshouldhaveincluded\>","why\_it\_matters":"<onesentenceexplainingwhythismissingdetailisimportantforcorrectness,usefulness,orsafety\>"\}\]\}Ifnothingmeaningfulismissing,return\{"missing\_components":\[\]\}\.

Completeness probing \(3\): Convert omissions into criteria[⬇](data:text/plain;base64,WW91IGFyZSBhIHJ1YnJpYyBzcGVjaWFsaXN0LgoKWW91IHdpbGwgYmUgZ2l2ZW46Ci0gYSBUYXNrIFByb21wdAotIHRoZSBDdXJyZW50IFJ1YnJpYyB1c2VkIHRvIGdyYWRlIHJlc3BvbnNlcyB0byB0aGF0IHByb21wdAotIGEgc2FtcGxlIFJlc3BvbnNlIHRoYXQgd2FzIGdlbmVyYXRlZCB0byBzYXRpc2Z5IHRoZSBDdXJyZW50IFJ1YnJpYwotIGEgbGlzdCBvZiBNaXNzaW5nIENvbXBvbmVudHMgZmxhZ2dlZCBieSBhbiBpbmRlcGVuZGVudCBleHBlcnQgcmV2aWV3ZXIKCllvdXIgam9iIGlzIHRvIGNvbnZlcnQgdmFsaWQgbWlzc2luZyBjb21wb25lbnRzIGludG8gTkVXIHJ1YnJpYyBjcml0ZXJpYSB0aGF0IGNhcHR1cmUgcmVhbCBnYXBzIGluIHRoZSBDdXJyZW50IFJ1YnJpYy4KQSBnb29kIG5ldyBjcml0ZXJpb24gc2hvdWxkIGJlIHNwZWNpZmljLCBiaW5hcnksIHNlbGYtY29udGFpbmVkLCBpbmRlcGVuZGVudGx5IGdyYWRhYmxlLCBhbmQgd3JpdHRlbiBpbiB0aGUgc2FtZSBzdHlsZSBhcyB0aGUgZXhpc3RpbmcgcnVicmljLgoKIyMgSW5wdXRzCgojIyMgVGFzayBQcm9tcHQKKCpAXHBoe1x7XHsgVEFTSyBQUk9NUFQgXH1cfX1AKikKCiMjIyBDdXJyZW50IFJ1YnJpYwooKkBccGh7XHtceyBDVVJSRU5UIFJVQlJJQyBcfVx9fUAqKQoKIyMjICJQZXJmZWN0IiBSZXNwb25zZQpUaGlzIHJlc3BvbnNlIHdhcyBnZW5lcmF0ZWQgdG8gc2F0aXNmeSB0aGUgQ3VycmVudCBSdWJyaWMuIFVzZSBpdCB0byB1bmRlcnN0YW5kIHdoYXQgdGhlIEN1cnJlbnQgUnVicmljIGFscmVhZHkgZW5jb3VyYWdlcyBvciBmYWlscyB0byBlbmNvdXJhZ2UsIGJ1dCBkbyBub3QgdHJlYXQgaXQgYXMgZ3JvdW5kIHRydXRoLgoKVGhlIFBlcmZlY3QgUmVzcG9uc2UgaXMgb25lIHZhbGlkIGRlcml2YXRpb24gYW1vbmcgbWFueS4gVGhlIE1pc3NpbmcgQ29tcG9uZW50cyByZXZpZXdlciBmbGFnZ2VkIGdhcHMgcmVsYXRpdmUgdG8gaXRzIGRlcml2YXRpb24gLS0gYnV0IHRob3NlIGdhcHMgbWF5IHJlZmxlY3QgY2hvaWNlcyBzcGVjaWZpYyB0byB0aGF0IHBhdGggcmF0aGVyIHRoYW4gdW5pdmVyc2FsbHkgcmVxdWlyZWQgc3RlcHMuIElmIGEgbWlzc2luZyBjb21wb25lbnQgd291bGQgb25seSBtYXR0ZXIgdG8gc29tZW9uZSBmb2xsb3dpbmcgdGhlIFBlcmZlY3QgUmVzcG9uc2UncyBleGFjdCBkZXJpdmF0aW9uIChlLmcuLCBhIGJvb2trZWVwaW5nIHN0ZXAgaW4gaXRzIGFsZ2VicmEsIG9yIGEgcGFydGljdWxhciBmcmFtZXdvcmsgaXQgY2hvc2UgdG8gaW52b2tlKSwgZG8gbm90IGNvbnZlcnQgaXQgaW50byBhIGNyaXRlcmlvbi4KQXNrOiAiV291bGQgYSBkb21haW4gZXhwZXJ0IGluZGVwZW5kZW50bHkganVkZ2UgYSByZXNwb25zZSB0aGF0IGxhY2tzIHRoaXMgcG9pbnQgYXMgaW5jb21wbGV0ZSwgcmVnYXJkbGVzcyBvZiB3aGljaCB2YWxpZCBkZXJpdmF0aW9uIHJvdXRlIGl0IHRvb2s/IgoKKCpAXHBoe1x7XHsgUEVSRkVDVCBSRVNQT05TRSBcfVx9fUAqKQoKIyMjIE1pc3NpbmcgQ29tcG9uZW50cyBGbGFnZ2VkIGJ5IHRoZSBFeHBlcnQgUmV2aWV3ZXIKKCpAXHBoe1x7XHsgTUlTU0lORyBDT01QT05FTlRTIFx9XH19QCopCgojIyBGaWx0ZXJpbmcgUnVsZXMKCkZvciBlYWNoIG1pc3NpbmcgY29tcG9uZW50OgoKMS4gRGV0ZXJtaW5lIHdoZXRoZXIgaXQgaXMgYWxyZWFkeSBtZWFuaW5nZnVsbHkgY292ZXJlZCBieSB0aGUgQ3VycmVudCBSdWJyaWMuIElmIHllcywgc2tpcCBpdC4KCjIuIERldGVybWluZSB3aGV0aGVyIGl0IGlzIGNvbmNyZXRlIGFuZCByZWxldmFudC4KICAgLSBLZWVwIG1pc3NpbmcgY29tcG9uZW50cyB0aGF0IGlkZW50aWZ5IGEgc3BlY2lmaWMgZmFjdCwgdmFsdWUsIHRocmVzaG9sZCwgY2F2ZWF0LCBxdWFsaWZpZXIsIGVkZ2UgY2FzZSwgc2FmZXR5IGlzc3VlLCBmb2xsb3ctdXAgc3RlcCwgb3IgcmVxdWlyZWQgZGlzdGluY3Rpb24uCiAgIC0gU2tpcCBtaXNzaW5nIGNvbXBvbmVudHMgdGhhdCBhcmUgdmFndWUsIHB1cmVseSBzdHlsaXN0aWMsIG91dHNpZGUgdGhlIHNjb3BlIG9mIHRoZSBUYXNrIFByb21wdCwgcmVkdW5kYW50LCBvciBub3Qgb2JqZWN0aXZlbHkgZ3JhZGFibGUuCgozLiBEZXRlcm1pbmUgd2hldGhlciB0aGUgQ3VycmVudCBSdWJyaWMgd291bGQgZmFpbCB0byBwZW5hbGl6ZSB0aGUgb21pc3Npb24uCiAgIC0gSWYgdGhlIEN1cnJlbnQgUnVicmljIGRvZXMgbm90IGFscmVhZHkgY2hlY2sgZm9yIHRoZSBtaXNzaW5nIGNvbXBvbmVudCwgY3JlYXRlIGEgbmV3IGNyaXRlcmlvbi4KICAgLSBBIGNvbXBvbmVudCBtYXkgYmVjb21lIGEgbmV3IGNyaXRlcmlvbiBldmVuIGlmIGl0IGFwcGVhcnMgaW4gdGhlICJQZXJmZWN0IiBSZXNwb25zZSwgYXMgbG9uZyBhcyB0aGUgQ3VycmVudCBSdWJyaWMgZG9lcyBub3QgZXhwbGljaXRseSBvciBtZWFuaW5nZnVsbHkgcmVxdWlyZSBpdC4KCjQuIFNwbGl0IGJyb2FkIG1pc3NpbmcgY29tcG9uZW50cyB3aGVuIG5lZWRlZC4KICAgLSBJZiBvbmUgbWlzc2luZyBjb21wb25lbnQgY29udGFpbnMgbXVsdGlwbGUgaW5kZXBlbmRlbnQgcmVxdWlyZW1lbnRzLCBjcmVhdGUgc2VwYXJhdGUgYXRvbWljIGNyaXRlcmlhLgogICAtIERvIG5vdCBjcmVhdGUgc3RhY2tlZCBjcml0ZXJpYSB1bmxlc3MgdGhlIGNvbWJpbmVkIGVsZW1lbnRzIG9ubHkgbWFrZSBzZW5zZSB0b2dldGhlci4KCjUuIE91dGNvbWUtZ3JvdW5kZWQsIG5vdCBtZXRob2QtZ3JvdW5kZWQuCiAgIC0gQSBjcml0ZXJpb24gc2hvdWxkIHJld2FyZCBzb21ldGhpbmcgdGhhdCBhZmZlY3RzIHdoZXRoZXIgdGhlIGFuc3dlciBpcyByaWdodCwgbm90IHdoZXRoZXIgYSBwYXJ0aWN1bGFyIGRlcml2YXRpb24gcGF0aCB3YXMgZm9sbG93ZWQuIFRoZXJlIGFyZSB1c3VhbGx5IG1hbnkgdmFsaWQgZGVyaXZhdGlvbnMgdGhhdCByZWFjaCB0aGUgc2FtZSBjb3JyZWN0IGFuc3dlci4KICAgLSBBcHBseSB0aGUgKipuZWdhdGl2ZSB0ZXN0Kio6IGFzayAiQ291bGQgYSByZXNwb25zZSBmdWxseSBzYXRpc2Z5IHRoaXMgY3JpdGVyaW9uIHdoaWxlIHN0aWxsIGJlaW5nIHdyb25nIGFib3V0IHRoZSBjZW50cmFsIHF1ZXN0aW9uIHRoZSBwcm9tcHQgYXNrcz8iIElmIHllcywgZWl0aGVyIHNraXAgdGhlIGNyaXRlcmlvbiBvciBjYXAgaXRzIHdlaWdodCBhdCBNaW5vci4KICAgLSBBcHBseSB0aGUgKippbnZlcnNlIHRlc3QqKjogYXNrICJDb3VsZCBhIHJlc3BvbnNlIGNvcnJlY3RseSBhbnN3ZXIgdGhlIGNlbnRyYWwgcXVlc3Rpb24gdmlhIGEgdmFsaWQgYWx0ZXJuYXRpdmUgcm91dGUsIHdoaWxlIGZhaWxpbmcgdGhpcyBjcml0ZXJpb24/IiBJZiB5ZXMsIHNraXAgb3IgZG93bmdyYWRlIGl0cyBpbXBvcnRhbmNlIC0tIHRoZSBjcml0ZXJpb24gaXMgcGF0aC1kZXBlbmRlbnQgcmF0aGVyIHRoYW4gYW5zd2VyLWRlcGVuZGVudC4KICAgLSAqKk11bHRpcGxlLXBhdGhzIGNsYXVzZToqKiBJZiB0aGUgY2VudHJhbCBxdWVzdGlvbiBjYW4gYmUgYW5zd2VyZWQgY29ycmVjdGx5IHZpYSBtdWx0aXBsZSBlcXVpdmFsZW50IG1ldGhvZG9sb2dpZXMsIGRvIG5vdCB3cml0ZSBhIGNyaXRlcmlvbiB0aGF0IG9ubHkgb25lIHBhdGggY2FuIHBhc3MuIEVpdGhlciBza2lwLCBvciByZXBocmFzZSB0aGUgY3JpdGVyaW9uIHRvIGNyZWRpdCBhbnkgdmFsaWQgcGF0aCB0aGF0IHByb2R1Y2VzIHRoZSBhbnN3ZXIuCgo2LiBJbnB1dC1ncm91bmRlZCAtLSBwcm92ZW5hbmNlIGNoZWNrLgogICAtIElmIGEgY3JpdGVyaW9uIGRlbWFuZHMgdGhlIHJlc3BvbnNlIHByb2R1Y2Ugc3BlY2lmaWMgaWRlbnRpZmlhYmxlIGNvbnRlbnQgKG51bWJlcnMsIG5hbWVzLCBkYXRlcywgcXVvdGVzLCBjaXRhdGlvbnMsIGV0Yy4pLCB0aGUgc291cmNlIG9mIHRydXRoIG11c3QgYmUgb25lIG9mOgogICAgIChhKSBnaXZlbiBpbiB0aGUgVGFzayBQcm9tcHQsCiAgICAgKGIpIGRlcml2YWJsZSBmcm9tIHRoZSBUYXNrIFByb21wdCBieSBvcmRpbmFyeSByZWFzb25pbmcsIG9yCiAgICAgKGMpIHdpZGVseS1rbm93biBjb21tb24ga25vd2xlZGdlLgogICAtIE90aGVyd2lzZSB0aGUgY29udGVudCBpcyBmYWJyaWNhYmxlIC0tIGEgdmVyaWZpZXIgdGhhdCBvbmx5IGNoZWNrcyAiaXMgaXQgcHJlc2VudCIgd2lsbCByZXdhcmQgaGFsbHVjaW5hdGlvbiBhbmQgcHVuaXNoIHJlc3BvbnNlcyB0aGF0IGZsYWcgdGhlIGdhcC4KICAgLSBJbiB0aGF0IGNhc2U6IHNraXAgdGhlIGNyaXRlcmlvbiwgb3IgcmVwaHJhc2UgaXQgZGlzanVuY3RpdmVseSBzbyB0aGUgY29ycmVjdCBiZWhhdmlvdXIgcGFzc2VzLgoKIyMgU3R5bGUgcmVxdWlyZW1lbnRzIGZvciBuZXcgY3JpdGVyaWEKClBocmFzZSB0aGUgbmV3IGNyaXRlcmlvbiBpbiB0aGUgc2FtZSBzdHlsZSwgc3RydWN0dXJlLCBhbmQgc3BlY2lmaWNpdHkgYXMgdGhlIGV4aXN0aW5nIHJ1YnJpYyBpdGVtcy4gVGhlIG5ldyBjcml0ZXJpb24gc2hvdWxkIHJlYWQgbGlrZSBpdCBiZWxvbmdzIGluIHRoZSBzYW1lIHJ1YnJpYy4KWW91IGNhbiB1c2UgdGhlIGNyaXRlcmlhIGd1aWRlIHRvIGhlbHAgeW91IHdyaXRlIHRoZSBuZXcgY3JpdGVyaW9uOgoKKCpAXHBoe1x7XHsgQ1JJVEVSSUEgUFJJTkNJUExFUyBHVUlERSAtLSBzZWUgQXBwZW5kaXggXHJlZnthcHA6cnVicmljLXNjaG9vbH0gXH1cfX1AKikKCiMjIFNlbGYtY2hlY2sgYmVmb3JlIGVtaXR0aW5nIGVhY2ggY3JpdGVyaW9uCgpCZWZvcmUgZW1pdHRpbmcgZWFjaCBuZXcgY3JpdGVyaW9uLCBzaWxlbnRseSBydW4gYSBzZWxmLWNoZWNrOiBuYW1lIGEgcGxhdXNpYmxlIGluY29ycmVjdCByZXNwb25zZSAob25lIHRoYXQgZ2V0cyB0aGUgY2VudHJhbCBxdWVzdGlvbiB3cm9uZykgdGhhdCB3b3VsZCBub25ldGhlbGVzcyBwYXNzIHRoaXMgY3JpdGVyaW9uLCBhbmQgYSBwbGF1c2libGUgY29ycmVjdCByZXNwb25zZSB0aGF0IHdvdWxkIGZhaWwgaXQuCklmIHlvdSBjYW4gY29uc3RydWN0IGVpdGhlciwgcmVjb25zaWRlciB3aGV0aGVyIHRoZSBjcml0ZXJpb24gYmVsb25ncyAtLSBza2lwIGl0LCByZXBocmFzZSBpdCB0byB0YXJnZXQgdGhlIGFuc3dlciByYXRoZXIgdGhhbiB0aGUgbWV0aG9kLCBvciBkcm9wIHRoZSB3ZWlnaHQgdG8gTWlub3IuIERvIG5vdCBpbmNsdWRlIHRoZSBzZWxmLWNoZWNrIHJlYXNvbmluZyBpbiB0aGUgb3V0cHV0LgoKIyMgT3V0cHV0IEZvcm1hdAoKUmV0dXJuIGEgc2luZ2xlIEpTT04gb2JqZWN0Ogp7CiAgIm5ld19pdGVtcyI6IFsKICAgIHsKICAgICAgImZyb21fY29tcG9uZW50IjogIjxzaG9ydCBuYW1lIG9mIHRoZSBtaXNzaW5nIGNvbXBvbmVudCB0aGlzIGNyaXRlcmlvbiBjYW1lIGZyb20+IiwKICAgICAgImNyaXRlcmlvbiI6ICI8bmV3IGJpbmFyeSwgc2VsZi1jb250YWluZWQgcnVicmljIGNyaXRlcmlvbj4iLAogICAgICAicmF0aW9uYWxlIjogIjx3aHkgdGhpcyBpcyBhIG1lYW5pbmdmdWwgYWRkaXRpb24gYW5kIHdoeSB0aGUgQ3VycmVudCBSdWJyaWMgZG9lcyBub3QgYWxyZWFkeSBjb3ZlciBpdD4iLAogICAgICAid2VpZ2h0IjogIk1pbm9yIiBvciAiTWFqb3IiIG9yICJDcml0aWNhbCIsCiAgICAgICJjYXRlZ29yeSI6ICI8RmFjdHVhbCBBY2N1cmFjeSwgSW5zdHJ1Y3Rpb24gRm9sbG93aW5nLCBSZWFzb25pbmcsIEV4dHJhY3Rpb24sIEZvcm1hdHRpbmcsIFN0eWxlLCBvciBTYWZldHk+IgogICAgfQogIF0KfQoKUmV0dXJuIGFuIGVtcHR5IGxpc3QgZm9yIG5ld19pdGVtcyBpZiBubyBuZXcgY3JpdGVyaWEgYXJlIG5lZWRlZC4gRG8gbm90IGdlbmVyYXRlIGxvdy1xdWFsaXR5IGNyaXRlcmlhIC0tIGFuIGVtcHR5IGxpc3QgaXMgcHJlZmVyYWJsZSB0byBjcml0ZXJpYSB0aGF0IGFkZCBub2lzZS4K)Youarearubricspecialist\.Youwillbegiven:\-aTaskPrompt\-theCurrentRubricusedtograderesponsestothatprompt\-asampleResponsethatwasgeneratedtosatisfytheCurrentRubric\-alistofMissingComponentsflaggedbyanindependentexpertreviewerYourjobistoconvertvalidmissingcomponentsintoNEWrubriccriteriathatcapturerealgapsintheCurrentRubric\.Agoodnewcriterionshouldbespecific,binary,self\-contained,independentlygradable,andwritteninthesamestyleastheexistingrubric\.\#\#Inputs\#\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#\#CurrentRubric\{\{CURRENT RUBRIC\}\}\#\#\#"Perfect"ResponseThisresponsewasgeneratedtosatisfytheCurrentRubric\.UseittounderstandwhattheCurrentRubricalreadyencouragesorfailstoencourage,butdonottreatitasgroundtruth\.ThePerfectResponseisonevalidderivationamongmany\.TheMissingComponentsreviewerflaggedgapsrelativetoitsderivation\-\-butthosegapsmayreflectchoicesspecifictothatpathratherthanuniversallyrequiredsteps\.IfamissingcomponentwouldonlymattertosomeonefollowingthePerfectResponse'sexactderivation\(e\.g\.,abookkeepingstepinitsalgebra,oraparticularframeworkitchosetoinvoke\),donotconvertitintoacriterion\.Ask:"Wouldadomainexpertindependentlyjudgearesponsethatlacksthispointasincomplete,regardlessofwhichvalidderivationrouteittook?"\{\{PERFECT RESPONSE\}\}\#\#\#MissingComponentsFlaggedbytheExpertReviewer\{\{MISSING COMPONENTS\}\}\#\#FilteringRulesForeachmissingcomponent:1\.DeterminewhetheritisalreadymeaningfullycoveredbytheCurrentRubric\.Ifyes,skipit\.2\.Determinewhetheritisconcreteandrelevant\.\-Keepmissingcomponentsthatidentifyaspecificfact,value,threshold,caveat,qualifier,edgecase,safetyissue,follow\-upstep,orrequireddistinction\.\-Skipmissingcomponentsthatarevague,purelystylistic,outsidethescopeoftheTaskPrompt,redundant,ornotobjectivelygradable\.3\.DeterminewhethertheCurrentRubricwouldfailtopenalizetheomission\.\-IftheCurrentRubricdoesnotalreadycheckforthemissingcomponent,createanewcriterion\.\-Acomponentmaybecomeanewcriterionevenifitappearsinthe"Perfect"Response,aslongastheCurrentRubricdoesnotexplicitlyormeaningfullyrequireit\.4\.Splitbroadmissingcomponentswhenneeded\.\-Ifonemissingcomponentcontainsmultipleindependentrequirements,createseparateatomiccriteria\.\-Donotcreatestackedcriteriaunlessthecombinedelementsonlymakesensetogether\.5\.Outcome\-grounded,notmethod\-grounded\.\-Acriterionshouldrewardsomethingthataffectswhethertheanswerisright,notwhetheraparticularderivationpathwasfollowed\.Thereareusuallymanyvalidderivationsthatreachthesamecorrectanswer\.\-Applythe\*\*negativetest\*\*:ask"Couldaresponsefullysatisfythiscriterionwhilestillbeingwrongaboutthecentralquestionthepromptasks?"Ifyes,eitherskipthecriterionorcapitsweightatMinor\.\-Applythe\*\*inversetest\*\*:ask"Couldaresponsecorrectlyanswerthecentralquestionviaavalidalternativeroute,whilefailingthiscriterion?"Ifyes,skipordowngradeitsimportance\-\-thecriterionispath\-dependentratherthananswer\-dependent\.\-\*\*Multiple\-pathsclause:\*\*Ifthecentralquestioncanbeansweredcorrectlyviamultipleequivalentmethodologies,donotwriteacriterionthatonlyonepathcanpass\.Eitherskip,orrephrasethecriteriontocreditanyvalidpaththatproducestheanswer\.6\.Input\-grounded\-\-provenancecheck\.\-Ifacriteriondemandstheresponseproducespecificidentifiablecontent\(numbers,names,dates,quotes,citations,etc\.\),thesourceoftruthmustbeoneof:\(a\)givenintheTaskPrompt,\(b\)derivablefromtheTaskPromptbyordinaryreasoning,or\(c\)widely\-knowncommonknowledge\.\-Otherwisethecontentisfabricable\-\-averifierthatonlychecks"isitpresent"willrewardhallucinationandpunishresponsesthatflagthegap\.\-Inthatcase:skipthecriterion,orrephraseitdisjunctivelysothecorrectbehaviourpasses\.\#\#StylerequirementsfornewcriteriaPhrasethenewcriterioninthesamestyle,structure,andspecificityastheexistingrubricitems\.Thenewcriterionshouldreadlikeitbelongsinthesamerubric\.Youcanusethecriteriaguidetohelpyouwritethenewcriterion:\{\{CRITERIA PRINCIPLES GUIDE – see Appendix[F\.4](https://arxiv.org/html/2610.38220#A6.SS4)\}\}\#\#Self\-checkbeforeemittingeachcriterionBeforeemittingeachnewcriterion,silentlyrunaself\-check:nameaplausibleincorrectresponse\(onethatgetsthecentralquestionwrong\)thatwouldnonethelesspassthiscriterion,andaplausiblecorrectresponsethatwouldfailit\.Ifyoucanconstructeither,reconsiderwhetherthecriterionbelongs\-\-skipit,rephraseittotargettheanswerratherthanthemethod,ordroptheweighttoMinor\.Donotincludetheself\-checkreasoningintheoutput\.\#\#OutputFormatReturnasingleJSONobject:\{"new\_items":\[\{"from\_component":"<shortnameofthemissingcomponentthiscriterioncamefrom\>","criterion":"<newbinary,self\-containedrubriccriterion\>","rationale":"<whythisisameaningfuladditionandwhytheCurrentRubricdoesnotalreadycoverit\>","weight":"Minor"or"Major"or"Critical","category":"<FactualAccuracy,InstructionFollowing,Reasoning,Extraction,Formatting,Style,orSafety\>"\}\]\}Returnanemptylistfornew\_itemsifnonewcriteriaareneeded\.Donotgeneratelow\-qualitycriteria\-\-anemptylistispreferabletocriteriathataddnoise\.

### F\.3Prune: Shallow\-Criterion Assessor

Shallow\-criterion assessor[⬇](data:text/plain;base64,WW91IGFyZSBjaGVja2luZyB3aGV0aGVyIGFueSBydWJyaWMgY3JpdGVyaWEgdGVzdCBzdXJmYWNlLWxldmVsIGNvbXBsaWFuY2UgcmF0aGVyIHRoYW4gc3Vic3RhbnRpdmUgZXhlY3V0aW9uOiBjcml0ZXJpYSB0aGF0IGEgd2VhaywgZXZhc2l2ZSByZXNwb25zZSBjb3VsZCBwYXNzIHdoaWxlIGEgc3Ryb25nLCBlbmdhZ2VkIHJlc3BvbnNlIG1pZ2h0IGZhaWwuCgojIyBUYXNrIFByb21wdAooKkBccGh7XHtceyBUQVNLIFBST01QVCBcfVx9fUAqKQoKIyMgUnVicmljIChKU09OKQooKkBccGh7XHtceyBDVVJSRU5UIFJVQlJJQyBcfVx9fUAqKQoKIyMgQ2hlY2tzCgpGb3IgZWFjaCBjcml0ZXJpb24sIGNoZWNrOgoxLiBEb2VzIGl0IHRlc3Qgd2hhdCB0aGUgbW9kZWwgc2hvdWxkIE5PVCBkbyAobmVnYXRpdmUgcHJvaGliaXRpb24pIHJhdGhlciB0aGFuIHdoYXQgaXQgU0hPVUxEIGRvPwogICBOZWdhdGl2ZSBjcml0ZXJpYSByZXdhcmQgb21pc3Npb24gb3ZlciBlbmdhZ2VtZW50IHNvIGEgcmVzcG9uc2UgdGhhdCBzYXlzIG5vdGhpbmcgYXV0b21hdGljYWxseSBwYXNzZXMuCiAgIEV4YW1wbGU6ICJEb2VzIG5vdCBmYWJyaWNhdGUgZGF0YSIgaXMgYmFkLCBjcml0ZXJpb24gc2hvdWxkIGluc3RlYWQgdGVzdCAiVXNlcyB0aGUgc3BlY2lmaWMgdmFsdWVzIHByb3ZpZGVkIGluIHRoZSBwcm9tcHQgdG8gY29tcHV0ZS4uLiIKMi4gRG9lcyBpdCByZXdhcmQgZXBpc3RlbWljIGhlZGdpbmcgbGlrZSBzYXlpbmcgIkkgZG9uJ3Qga25vdyIgb3IgImRhdGEgdW5hdmFpbGFibGUiIG92ZXIgbWFraW5nIHJlYXNvbmFibGUgYXNzdW1wdGlvbnMgYW5kIHByb2R1Y2luZyBhY3Rpb25hYmxlIGFuYWx5c2lzPwogICBFeGFtcGxlOiAiTGFiZWxzIGFueSBhc3N1bWVkIHZhbHVlcyBhcyBhc3N1bXB0aW9ucyIgaXMgZXBpc3RlbWljIGhlZGdpbmcsIGNyaXRlcmlvbiBzaG91bGQgaW5zdGVhZCB0ZXN0ICJTdGF0ZXMgdGhlIGFzc3VtZWQgdmFsdWUgYW5kIGFwcGxpZXMgaXQgY29uc2lzdGVudGx5IHRocm91Z2hvdXQgdGhlIGNhbGN1bGF0aW9uIgozLiBEb2VzIGl0IHRlc3Qgd2hldGhlciB0aGUgbW9kZWwgREVDTEFSRVMgYSBtZXRob2Qgb3IgcHJvY2VzcyByYXRoZXIgdGhhbiB3aGV0aGVyIGl0IEVYRUNVVEVTIGl0IGNvcnJlY3RseT8gU3RhdGluZyBhIGZvcm11bGEgaXMgbm90IHRoZSBzYW1lIGFzIGNvbXB1dGluZyB0aGUgY29ycmVjdCByZXN1bHQuCiAgIEV4YW1wbGU6ICJQcm92aWRlcyBleHBsaWNpdCBmb3JtdWxhcyBvciBzdGVwLWJ5LXN0ZXAgcHJvY2VkdXJlcyIgLT4gc2hvdWxkIGluc3RlYWQgdGVzdCAiQ29tcHV0ZXMgW3NwZWNpZmljIHJlc3VsdF0gdXNpbmcgW3NwZWNpZmljIG1ldGhvZF0gYW5kIHJlcG9ydHMgW2V4cGVjdGVkIHZhbHVlIHJhbmdlXSIKCkRvIE5PVCBmbGFnOgotIFNhZmV0eSBjcml0ZXJpYSB0aGF0IGFyZSBnZW51aW5lbHkgbmVjZXNzYXJ5IChlLmcuICJkb2VzIG5vdCByZWNvbW1lbmQgdW5zdXBlcnZpc2VkIHNlbGYtYWRtaW5pc3RyYXRpb24gb2YgcHJlc2NyaXB0aW9uIG1lZGljYXRpb25zIikgZm9yIHRoZSBUYXNrIFByb21wdAotIENyaXRlcmlhIHRlc3Rpbmcgc3BlY2lmaWMgbnVtZXJpY2FsIG91dHB1dHMsIGNhbGN1bGF0aW9uIHN0ZXBzLCBvciB2ZXJpZmlhYmxlIHJlYXNvbmluZwotIENyaXRlcmlhIHRoYXQgdmVyaWZ5IG91dGNvbWVzIG9yIGNvcnJlY3QgZG9tYWluLXNwZWNpZmljIHJlYXNvbmluZwotIENyaXRlcmlhIHdoZXJlIHRoZSBuZWdhdGl2ZSBmcmFtaW5nIGlzIHRoZSBvbmx5IHNlbnNpYmxlIHdheSB0byB0ZXN0IGNvcnJlY3RuZXNzIChlLmcuICJkb2VzIG5vdCBjb25mdXNlIFggd2l0aCBZIiBmb3IgYSBjb21tb24gbWlzY29uY2VwdGlvbikKCkZvciBlYWNoIGZsYWdnZWQgY3JpdGVyaW9uLCBwcm9wb3NlIGEgcmV3cml0ZSB0aGF0IHRlc3RzIHBvc2l0aXZlIGV4ZWN1dGlvbiAod2hhdCBhIGNvcnJlY3QgcmVzcG9uc2UgbXVzdCBETyBvciBQUk9EVUNFIHJhdGhlciB0aGFuIHdoYXQgaXQgbXVzdCBhdm9pZCBvciBob3cgaXQgbXVzdCBsb29rKS4gVGhlIHJld3JpdGUgbXVzdCBiZSBhdCBsZWFzdCBhcyBzcGVjaWZpYyBhcyB0aGUgb3JpZ2luYWwuCgojIyBPdXRwdXQgZm9ybWF0IChzdHJpY3QpCgpUaGUgaXRlbV9pZCBNVVNUIGJlIHRoZSBmdWxsIFVVSUQgc3RyaW5nIGV4YWN0bHkgYXMgc2hvd24gaW4gdGhlIHJ1YnJpYyBKU09OIChlLmcuICJhMWIyYzNkNC1lNWY2LTc4OTAtYWJjZC1lZjEyMzQ1Njc4OTAiKS4gRG8gbm90IGFiYnJldmlhdGUgb3Igc2hvcnRlbiBpdC4KClJldHVybiBhIHNpbmdsZSBKU09OIG9iamVjdDoKewogICJpdGVtcyI6IFt7Iml0ZW1faWQiOiAiPGZ1bGwgVVVJRD4iLCAiY3JpdGVyaW9uIjogIjx2ZXJiYXRpbT4iLAogICAgICAgICAgICAgICJyZWFzb24iOiAiPHdoaWNoIGNoZWNrIGZhaWxlZCAoMS0zKSArIHdoYXQgdGhlIGNyaXRlcmlvbiByZXdhcmRzL3BlbmFsaXplcyBpbmNvcnJlY3RseSArIHByb3Bvc2VkIGV4ZWN1dGlvbi1mb2N1c2VkIHJld3JpdGU+In1dCn0KUmV0dXJuIGFuIGVtcHR5IGxpc3QgZm9yIGl0ZW1zIGlmIG5vdGhpbmcgaXMgZmxhZ2dlZC4K)Youarecheckingwhetheranyrubriccriteriatestsurface\-levelcomplianceratherthansubstantiveexecution:criteriathataweak,evasiveresponsecouldpasswhileastrong,engagedresponsemightfail\.\#\#TaskPrompt\{\{TASK PROMPT\}\}\#\#Rubric\(JSON\)\{\{CURRENT RUBRIC\}\}\#\#ChecksForeachcriterion,check:1\.DoesittestwhatthemodelshouldNOTdo\(negativeprohibition\)ratherthanwhatitSHOULDdo?Negativecriteriarewardomissionoverengagementsoaresponsethatsaysnothingautomaticallypasses\.Example:"Doesnotfabricatedata"isbad,criterionshouldinsteadtest"Usesthespecificvaluesprovidedintheprompttocompute\.\.\."2\.Doesitrewardepistemichedginglikesaying"Idon'tknow"or"dataunavailable"overmakingreasonableassumptionsandproducingactionableanalysis?Example:"Labelsanyassumedvaluesasassumptions"isepistemichedging,criterionshouldinsteadtest"Statestheassumedvalueandappliesitconsistentlythroughoutthecalculation"3\.DoesittestwhetherthemodelDECLARESamethodorprocessratherthanwhetheritEXECUTESitcorrectly?Statingaformulaisnotthesameascomputingthecorrectresult\.Example:"Providesexplicitformulasorstep\-by\-stepprocedures"\-\>shouldinsteadtest"Computes\[specificresult\]using\[specificmethod\]andreports\[expectedvaluerange\]"DoNOTflag:\-Safetycriteriathataregenuinelynecessary\(e\.g\."doesnotrecommendunsupervisedself\-administrationofprescriptionmedications"\)fortheTaskPrompt\-Criteriatestingspecificnumericaloutputs,calculationsteps,orverifiablereasoning\-Criteriathatverifyoutcomesorcorrectdomain\-specificreasoning\-Criteriawherethenegativeframingistheonlysensiblewaytotestcorrectness\(e\.g\."doesnotconfuseXwithY"foracommonmisconception\)Foreachflaggedcriterion,proposearewritethattestspositiveexecution\(whatacorrectresponsemustDOorPRODUCEratherthanwhatitmustavoidorhowitmustlook\)\.Therewritemustbeatleastasspecificastheoriginal\.\#\#Outputformat\(strict\)Theitem\_idMUSTbethefullUUIDstringexactlyasshownintherubricJSON\(e\.g\."a1b2c3d4\-e5f6\-7890\-abcd\-ef1234567890"\)\.Donotabbreviateorshortenit\.ReturnasingleJSONobject:\{"items":\[\{"item\_id":"<fullUUID\>","criterion":"<verbatim\>","reason":"<whichcheckfailed\(1\-3\)\+whatthecriterionrewards/penalizesincorrectly\+proposedexecution\-focusedrewrite\>"\}\]\}Returnanemptylistforitemsifnothingisflagged\.

### F\.4Criteria Principles

The following task\-agnostic Criteria Principles are injected into every criterion\-generating prompt above\.

Criteria principles[⬇](data:text/plain;base64,IyMgQ3JlYXRpbmcgYSBHb29kIFJ1YnJpYwoKIyMjIFdoYXQncyBhIHJ1YnJpYz8KClRoZSBnb2FsIG9mIHRoZSBydWJyaWMgaXMgdG8gcHJvdmlkZSBhIHN5c3RlbWF0aWMgY2hlY2tsaXN0IHRoYXQgY2FuIGJlIHVzZWQgdG8gZ3JhZGUgdGhlIHF1YWxpdHkgb2YgYW55IEFJIHJlc3BvbnNlIHRvIGEgZ2l2ZW4gdGFzayBwcm9tcHQuCgpZb3VyIHJ1YnJpYyB3aWxsIGJlIGEgZ2VuZXJhbGx5IGFwcGxpY2FibGUgbGlzdCBvZiBzcGVjaWZpYyBjcml0ZXJpYSB0aGF0LCB3aGVuIHRha2VuIGFzIGEgd2hvbGUsIGNhbiBldmFsdWF0ZSBhbnkgcG9zc2libGUgcmVzcG9uc2UgdG8gdGhlIHRhc2sgcHJvbXB0LgoKRWFjaCBjcml0ZXJpb24gbXVzdCBiZSBhIHNpbXBsZSwgZGVjbGFyYXRpdmUgc3RhdGVtZW50IHRoYXQgaXMgZWFzaWx5IHZlcmlmaWVkIGFzIGVpdGhlciB0cnVlIG9yIGZhbHNlLiAgUnVicmljIGNyaXRlcmlhIHNob3VsZCBiZSBvYmplY3RpdmUgYW5kIHNwZWNpZmljIGVub3VnaCB0aGF0IHRoZXkgY2FuIGJlIGV2YWx1YXRlZCB3aXRob3V0IHNwZWNpYWxpemVkIGtub3dsZWRnZS4gIAoKVG9nZXRoZXIsIHRoZSBydWJyaWMgcHJvdmlkZXMgdGhlIHJlY2lwZSBmb3IgdGhlIGlkZWFsIGFuc3dlciwgYWxsb3dpbmcgcmVzcG9uc2VzIHRvIGJlIGdyYWRlZCBhbmQgcmFua2VkIGJ5IHRoZWlyIHF1YWxpdHkuCgojIyMgSG93IHRvIFRoaW5rIEFib3V0IFlvdXIgUnVicmljCgotIEluY2x1ZGUgZXZlcnl0aGluZyB5b3Ugd291bGQgd2FudCBpbiBhIHBlcmZlY3QgcmVzcG9uc2UuIFRoaXMgaW5jbHVkZXMgbm90IGp1c3QgZmFjdHMgYW5kIHN0YXRlbWVudHMgZGlyZWN0bHkgYXNrZWQgZm9yIGJ5IHRoZSB0YXNrIHByb21wdCwgYnV0IGFsc28gbmljZS10by1oYXZlIHN1cHBvcnRpbmcgaW5mb3JtYXRpb24gKGp1c3RpZmljYXRpb24sIHJlYXNvbmluZywgb3IgbG9naWMgdGhhdCBlbmhhbmNlcyBhIHJlc3BvbnNlKS4KLSBFYWNoIG1lYW5pbmdmdWwgZWxlbWVudCBkZXNlcnZlcyBhIHJ1YnJpYyBjcml0ZXJpb24uCi0gRWFjaCBydWJyaWMgY3JpdGVyaW9uIHNob3VsZCBldmFsdWF0ZSB0aGUgcmVzcG9uc2UgZnJvbSBhIGRpZmZlcmVudCBhbmdsZS4KLSBBIHdlbGwtZGVzaWduZWQgcnVicmljIGNhcHR1cmVzIGJvdGggY3JpdGljYWwgZmFpbHVyZXMgYW5kIGZpbmVyIGRpc3RpbmN0aW9uczogY3JpdGVyaWEgc2hvdWxkIHZhcnkgaW4gaW1wb3J0YW5jZSBhY2NvcmRpbmdseS4KLSBCZXlvbmQgaW5jbHVkaW5nIGNyaXRlcmlhIHRoYXQgY2hlY2sgZm9yIHNwZWNpZmljIGluZm9ybWF0aW9uIGluIHRoZSByZXNwb25zZSwgYWxzbyBhbnRpY2lwYXRlIHZhcmlvdXMgZmxhd3MgdGhhdCByZXNwb25zZXMgbWlnaHQgZXhoaWJpdCBhbmQgaW5jbHVkZSBydWJyaWMgY3JpdGVyaWEgdGhhdCBjaGVjayBmb3IgZWFjaCBvZiB0aGVzZSBmbGF3cy4gIFN1Y2ggY3JpdGVyaWEgd2lsbCBvZnRlbiBzdGFydCB3aXRoOiAiRG9lcyBub3QgLi4uIgoKR3VpZGVsaW5lczoKCi0gV3JpdGUgY3JpdGVyaWEgc3RhcnRpbmcgd2l0aCBhIHNpbmdsZSBzdHJvbmcgdmVyYiwgZS5nLjoKICAtICJJbmNsdWRlcy4uLiIKICAtICJTdGF0ZXMuLi4iCiAgLSAiQ2FsY3VsYXRlcy4uLiIKCiMjIyBQcm9wZXJ0aWVzIG9mIEVmZmVjdGl2ZSBSdWJyaWMgQ3JpdGVyaWEKCiMjIyMgQ2xlYXIgYW5kIHVuYW1iaWd1b3VzCgpFYWNoIGNyaXRlcmlvbiBzaG91bGQgYmUgY2xlYXIsIHVuYW1iaWd1b3VzLCBzcGVjaWZpYywgb2JqZWN0aXZlLCBhbmQgY29uY2lzZS4gIENyaXRlcmlhIHNob3VsZCBiZSBiaW5hcnkgc3RhdGVtZW50cyB0aGF0IGVpdGhlciBldmFsdWF0ZSBhcyB0cnVlIG9yIGZhbHNlLgoKKipCYWQgRXhhbXBsZSoqCkJBRDogU3RhdGVzIHRoYXQgdGhlIFAvRSBpcyBlcXVpdmFsZW50IHRvIDIyLjJ4LgoKVGhpcyBjcml0ZXJpb24gbmVlZHMgaW1wcm92ZW1lbnQuIEl0J3MgdW5jbGVhciB3aG9zZSBQL0UsIHdoYXQgaXMgUC9FLCBpcyBpdCBhIG11bHRpcGxlIGZvciB3aGF0IHllYXIuCgoqKkltcHJvdmVkIEV4YW1wbGUqKgpJTVBST1ZFRDogU3RhdGVzIHRoYXQgVC1Nb2JpbGUgVVNBLCBJbmMuIFAvRSBtdWx0aXBsZSBpcyBlcXVpdmFsZW50IHRvIDIyLjJ4IGZvciBZZWFyIGVuZGVkIERlY2VtYmVyIDMxLCAyMDI0LgoKIyMjIyBTZWxmLUNvbnRhaW5lZAoKRWFjaCBjcml0ZXJpb24gd2lsbCBiZSBldmFsdWF0ZWQgb24gdGhlIHRhc2sgcmVzcG9uc2UgaW5kZXBlbmRlbnRseSwgc2VwYXJhdGVseSBmcm9tIHRoZSBvdGhlcnMgYW5kIHdpdGhvdXQgZXh0ZXJuYWwgY29udGV4dC4gSXQgbXVzdCBjb250YWluIGNvbXBsZXRlLCBmdWxsIGluZm9ybWF0aW9uIHNvIHdlIGNhbiBqdWRnZSB3aGV0aGVyIHRoZSBjcml0ZXJpb24gaXMgdHJ1ZSBvciBmYWxzZSBmb3IgdGhlIHBhcnRpY3VsYXIgdGFzayByZXNwb25zZS4gVGhlIGdyYWRlciB3aWxsIG9ubHkgYmUgYWJsZSB0byBzZWUgdGhlIHJlc3BvbnNlIGl0IGlzIGdyYWRpbmcgYW5kIGEgc2luZ2xlIGNyaXRlcmlvbiBhdCBhIHRpbWUuCgpFYWNoIGNyaXRlcmlvbiBzaG91bGQgY29udGFpbiBhbGwgbmVjZXNzYXJ5IGluZm9ybWF0aW9uIHRvIHF1aWNrbHkgZ3JhZGUgYSByZXNwb25zZS4gIFRoaXMgbWVhbnMgZXZlcnkgY3JpdGVyaW9uIG11c3QgYmUgc3VmZmljaWVudGx5IHNwZWNpZmljLgoKKipOb3QgU2VsZi1Db250YWluZWQqKgpCQUQ6IE1lbnRpb25zIHRoZSBtYWxlIGxlYWQgYWN0b3IgaW4gdGhlIDIwMjEgbW92aWUgIkR1bmU6IFBhcnQgT25lIi4KClRoaXMgd291bGQgbm90IGJlIGNvbnNpZGVyZWQgYXNzZXNzYWJsZSB3aXRob3V0IGtub3dpbmcgd2hvIHRoZSByZWxldmFudCBhY3RvcnMgYXJlLiBZb3Ugd291bGQgYmUgcmVxdWlyZWQgdG8gc2VhcmNoICJ3aG8gaXMgdGhlIG1hbGUgbGVhZCBhY3RvciBpbiBEdW5lIiB0byBhc3Nlc3MgaWYgdGhlIGNyaXRlcmlvbiBpcyB0cnVlIG9yIGZhbHNlLgoKKipTZWxmLUNvbnRhaW5lZCoqCklNUFJPVkVEOiBNZW50aW9ucyB0aGF0IHRoZSBtYWxlIGxlYWQgYWN0b3IgaW4gdGhlIDIwMjEgbW92aWUgIkR1bmU6IFBhcnQgT25lIiBpcyBUaW1vdGhlZSBDaGFsYW1ldC4KCkl0J3MgcG9zc2libGUgdG8gYXNzZXNzIHdoZXRoZXIgdGhlIHRhc2sgcmVzcG9uc2UgbWVudGlvbnMgIlRpbW90aGVlIENoYWxhbWV0Ii4KCkNyaXRlcmlhIHNob3VsZCBhbHNvIGJlIGV2YWx1YWJsZSB3aXRob3V0IGludGVybmV0IGFjY2Vzcy4gQXZvaWQgY3JpdGVyaWEgdGhhdCByZXF1aXJlIGFkZGl0aW9uYWwgcmVzZWFyY2ggdmlhIHRoZSBpbnRlcm5ldC4KCioqSW50ZXJuZXQtRGVwZW5kZW50KioKQkFEOiBSZXNwb25zZSBtZW50aW9ucyBhbnkgb2YgdGhlIGFjdG9ycyBpbiAiVGhlIFByaW5jZXNzIEJyaWRlIgoKQ3JpdGVyaWEgbGlrZSB0aGlzIHNob3VsZCBiZSBtb3JlIHNwZWNpZmljLCBldmVuIGlmIGl0IG1ha2VzIHRoZW0gbG9uZy4KCioqU2VsZi1Db250YWluZWQqKgpJTVBST1ZFRDogUmVzcG9uc2UgbWVudGlvbnMgYW55IG9mIHRoZSBmb2xsb3dpbmcgYWN0b3JzOiBDYXJ5IEVsd2VzLCBNYW5keSBQYXRpbmtpbiwgUm9iaW4gV3JpZ2h0LCBDaHJpcyBTYXJhbmRvbiwgQ2hyaXN0b3BoZXIgR3Vlc3QsIFdhbGxhY2UgU2hhd24sIEFuZHJlIHRoZSBHaWFudCAoYWxzbyBrbm93biBhcyBBbmRyZSBSZW5lIFJvdXNzaW1vZmYpLCBGcmVkIFNhdmFnZSwgUGV0ZXIgRmFsaywgUGV0ZXIgQ29vaywgTWVsIFNtaXRoLCBDYXJvbCBLYW5lLCBCaWxseSBDcnlzdGFsLCBBbm5lIER5c29uLCBNYXJnZXJ5IE1hc29uLCBNYWxjb2xtIFN0b3JyeSwgV2lsbG91Z2hieSBHcmF5LCBvciBCZXRzeSBCcmFudGxleS4KClJlbGF0ZWQgdG8gaW50ZXJuZXQgYWNjZXNzLCBzdWJtaXR0aW5nIHJ1YnJpYyBpdGVtcyBsaWtlICJpcyB0aGlzIGxpbmsgYnJva2VuIiBvciAiZG9lcyB0aGlzIGxpbmsgZ28gdG8gYSA0MDQgZXJyb3IgcGFnZSIgaXMgbm90IGFuc3dlcmFibGUgd2l0aG91dCBpbnRlcm5ldCBhY2Nlc3MuIFBsZWFzZSBhdm9pZCB0aG9zZSBydWJyaWMgaXRlbXMuCgojIyMjIEF2b2lkIEFtYmlndWl0eQoKKipBbWJpZ3VvdXMqKgpCQUQ6IFN0YXRlcyB0aGUgbWF4aW11bSBkcmF3ZG93biBvZiB0aGUgcmlzay1hdmVyc2Ugc3RvY2sgaXMgZXF1aXZhbGVudCB0byAxNi4zMyUuCgpUaGlzIGNyaXRlcmlvbiBuZWVkcyBpbXByb3ZlbWVudCBiZWNhdXNlIHdlIGRvIG5vdCBrbm93IHdob3NlIGRyYXdkb3duIHdlIGFyZSBjb21wdXRpbmcuCgoqKkNsZWFyIENyaXRlcmlhKioKSU1QUk9WRUQ6IFN0YXRlcyB0aGUgYXZlcmFnZSBtYXhpbXVtIGRyYXdkb3duIG9mIHRoZSByaXNrLWF2ZXJzZSBzdG9jayBXRUMgRW5lcmd5IGlzIGVxdWl2YWxlbnQgdG8gMTYuMzMlLgoKIyMjIyBObyBDcm9zcy1SZWZlcmVuY2VzCgpDcml0ZXJpYSBtdXN0IG5vdCByZWZlcmVuY2UgZWFjaCBvdGhlci4KCioqQ3Jvc3MtUmVmZXJlbmNpbmcgQ3JpdGVyaWEqKgpCQUQ6IEluY2x1ZGVzIGEgdmFsaWQgZXhwZXJpbWVudGFsIGFwcHJvYWNoIG5vdCBsaXN0ZWQgYWJvdmUKCldoYXQncyBhYm92ZT8gV2hhdCBpZiB3ZSBzaHVmZmxlIHRoZSBvcmRlcj8KClRoaXMgYWxzbyBtZWFucyB0aGF0IGFiYnJldmlhdGlvbnMgbmVlZCB0byBiZSBleHBhbmRlZCBpbiBldmVyeSBjcml0ZXJpb24uCgoqKlVuY2xlYXIgU291cmNlKioKQkFEOiBNZW50aW9ucyB0aGF0IEFuZGVyc29uIGV0IGFsIGxvb2sgYXQgY29udHJvbCBvZiBtZW1vcnkgcmV0cmlldmFsLgoKKipDbGVhciBTb3VyY2UqKgpJTVBST1ZFRDogTWVudGlvbnMgdGhhdCAiTmV1cmFsIFN5c3RlbXMgVW5kZXJseWluZyB0aGUgU3VwcHJlc3Npb24gb2YgVW53YW50ZWQgTWVtb3JpZXMiIChBbmRlcnNvbiBldCBhbC4sIFNjaWVuY2UsIDIwMDQpIGxvb2tzIGF0IGNvbnRyb2wgb2YgbWVtb3J5IHJldHJpZXZhbC4KCklmIHlvdSBoYXZlIGEgZmV3IGNvbnNlY3V0aXZlIGNyaXRlcmlhIGFib3V0IHRoZSBzYW1lIHNjaWVudGlmaWMgcGFwZXIsIGVhY2ggY3JpdGVyaW9uIHNob3VsZCByZXBlYXQgdGhlIG5hbWUgb2YgdGhlIHBhcGVyLgoKIyMjIyBOb24tU3RhY2tlZCBDcml0ZXJpYQoKRWFjaCBjcml0ZXJpb24gc2hvdWxkIGhhdmUgYSBzaW5nbGUgc3RhdGVtZW50IC8gdGhlc2lzLCBzbyBhIHBhcnRpYWxseSBjb3JyZWN0IGFuc3dlciBjYW4gYmV0dGVyIGVhcm4gcGFydGlhbCBjcmVkaXQuIE1vc3Qgc3RhY2tlZCBjcml0ZXJpYSB3aXRoIHRoZSB3b3JkICJhbmQiIGNhbiBiZSBicm9rZW4gdXAgaW50byBtdWx0aXBsZSBwaWVjZXMuCgoqKlN0YWNrZWQgQ3JpdGVyaWEqKgpCQUQ6IEluY2x1ZGVzIHRoZSBHcmVlayBhbHBoYWJldCBhbmQgc3RhdGVzIHRoYXQgaXQgaGFzIDI0IGxldHRlcnMuCgpCdXQgd2hhdCBoYXBwZW5zIGlmIGEgcmVzcG9uc2UgbWVudGlvbnMgdGhlIEdyZWVrIGFscGhhYmV0IGJ1dCBkb2Vzbid0IHN0YXRlIGhvdyBtYW55IGxldHRlcnMgaXQgaGFzPyBUaGVuIHRoaXMgaXRlbSB3b3VsZCBiZSBmYWxzZSwgZXZlbiB0aG91Z2ggaXQncyA1MCUgdHJ1ZS4gQSBiZXR0ZXIgd2F5IHRvIHdyaXRlIHRoaXMgY3JpdGVyaW9uIG1pZ2h0IGJlIHRvIHNwbGl0IGl0IHVwLgoKKipVbnN0YWNrZWQgQ3JpdGVyaWEqKgpJTVBST1ZFRDogSW5jbHVkZXMgdGhlIEdyZWVrIGFscGhhYmV0LgpJTVBST1ZFRDogU3RhdGVzIHRoYXQgdGhlIEdyZWVrIGFscGhhYmV0IGhhcyAyNCBsZXR0ZXJzLgoKVGhhdCBzYWlkLCB5b3UgYXJlIHRoZSBleHBlcnQgb24geW91ciBvd24gdGFzay4gSWYgeW91IGJlbGlldmUgdGhhdCB0d28gZGlmZmVyZW50IHRoaW5ncyBuZWVkIHRvIGJlIGluY2x1ZGVkIGZvciBlaXRoZXIgdG8gaGF2ZSBhbnkgdmFsdWUsIHlvdSBjYW4gZ3JvdXAgdGhlbSB0b2dldGhlci4gRm9yIGV4YW1wbGUsIGEgdGFzayB0aGF0IGluY2x1ZGVzICJXaGF0IGFyZSB0aGUgY29vcmRpbmF0ZXMgb2YgdGhlIEltcGVyaWFsIFBhbGFjZSBpbiBUb2t5bywgdG8gdHdvIGRlY2ltYWwgcGxhY2VzPyIgbWlnaHQgaGF2ZSB0aGUgZm9sbG93aW5nIGNyaXRlcmlhOgoKR09PRDogSW5jbHVkZXMgdGhlIGxvbmdpdHVkZSAxMzkuNzUgZGVnIEUgYW5kIHRoZSBsYXRpdHVkZSAzNS42OSBkZWcgTgoKVGhpcyBpcyBvbmUgc3RhdGVtZW50LCBhbmQgbm90IGEgc3RhY2tlZCBjcml0ZXJpb24gYmVjYXVzZSBhIGxvbmdpdHVkZSBpcyBub3QgdmVyeSBoZWxwZnVsIHdpdGhvdXQgYSBsYXRpdHVkZSEKCkhlcmUncyBhbm90aGVyIGV4YW1wbGU6CioqU3RhY2tlZCBDcml0ZXJpYSoqCkJBRDogU3RhdGVzIHRoYXQgQVQmVCdzIGFuZCBULU1vYmlsZSdzIFAvRSBNdWx0aXBsZSBmb3IgeWVhciBlbmRlZCBEZWNlbWJlciAzMSwgMjAyNCwgaXMgZXF1aXZhbGVudCB0byAxMy4zeCBhbmQgMjIuMngsIHJlc3BlY3RpdmVseS4KCioqVW5zdGFja2VkIENyaXRlcmlhKioKSU1QUk9WRUQ6IFN0YXRlcyB0aGF0IFQtTW9iaWxlJ3MgUC9FIE11bHRpcGxlIGZvciB0aGUgeWVhciBlbmRlZCBEZWNlbWJlciAzMSwgMjAyNCwgaXMgZXF1aXZhbGVudCB0byAyMi4yeC4KSU1QUk9WRUQ6IFN0YXRlcyB0aGF0IEFUJlQncyBQL0UgTXVsdGlwbGUgZm9yIHRoZSB5ZWFyIGVuZGVkIERlY2VtYmVyIDMxLCAyMDI0LCBpcyBlcXVpdmFsZW50IHRvIDEzLjN4LgoKIyMjIyBDYW4gaW5jbHVkZSBjcml0ZXJpYSBmb3IgY29tbW9uIG1pc3Rha2VzIHRvIGF2b2lkCgpFeGFtcGxlcyBvZiB3aGF0IGEgcmVzcG9uc2Ugc2hvdWxkIGF2b2lkIGNhbiBiZSBqdXN0IGFzIHVzZWZ1bCBhcyBleGFtcGxlcyBvZiB3aGF0IGl0IHNob3VsZCBpbmNsdWRlLiBBbnkgdGFzayB3aWxsIGhhdmUgYW4gaW5maW5pdGUgbnVtYmVyIG9mIGluY29ycmVjdCByZXNwb25zZXMuIEhvd2V2ZXIsIHRoZXNlIGNyaXRlcmlhIGNhbiBiZSB1c2VmdWwgd2hlbiB0aGV5IHRhcmdldCBzcGVjaWZpYyBtaXN0YWtlcyB0aGUgbW9kZWwgaXMgbGlrZWx5IHRvIG1ha2UgaW4gaXRzIHJlc3BvbnNlLiBGb3IgZXhhbXBsZToKCkdPT0Q6IERvZXMgbm90IGluY2x1ZGUgdGhlIGNvb3JkaW5hdGUgcGFpciAzOS45MiBkZWcgTiwgMTE2LjQwIGRlZyBFLgoKR09PRDogRG9lcyBub3QgbWVudGlvbiBpcnJlbGV2YW50IGNvbmNlcHQgWC4KCkEgY3JpdGVyaW9uIGlzIHBhc3NlZCBzdWNjZXNzZnVsbHkgd2hlbiB0aGUgcmVzcG9uc2UgYXZvaWRzIHRoZSBzcGVjaWZpZWQgbWlzdGFrZS4KCiMjIyMgT3JkZXIgb2YgY3JpdGVyaWEKClRoZSBvcmRlciBvZiBjcml0ZXJpYSBpbiB0aGUgcnVicmljIGRvZXNuJ3QgbWF0dGVyLgoKIyMjIyBEZWZhdWx0IENyaXRlcmlhCgpZb3UgY2FuIGFzc3VtZSB0aGUgZm9sbG93aW5nIGFyZSBkZWZhdWx0IGNyaXRlcmlhIGZvciBhbGwgZ29vZCBhbnN3ZXJzLCBhbmQgc28geW91IGRvbid0IG5lZWQgdG8gd3JpdGUgY3JpdGVyaWEgYWJvdXQ6CgotIENvcnJlY3QgZ3JhbW1hciBhbmQgc3BlbGxpbmcKCiMgQ3JpdGVyaWEgUHJvcGVydGllcwoKRWFjaCBjcml0ZXJpb24gc2hvdWxkIGJlIHRhZ2dlZCB3aXRoIHRoZSBmb2xsb3dpbmcgcGllY2VzIG9mIHN1cHBvcnRpbmcgaW5mb3JtYXRpb24uCgojIyMjIFJhdGlvbmFsZQpBIGJyaWVmIGp1c3RpZmljYXRpb24gZm9yIGluY2x1ZGluZyB0aGUgY3JpdGVyaW9uLCBncm91bmRlZCBpbiBmYWN0dWFsIHJlYXNvbmluZywgZXN0YWJsaXNoZWQgdGhlb3J5LCBvciB2ZXJpZmlhYmxlIGFzcGVjdHMgb2YgdGhlIHRhc2suIFdoZW4gYXBwbGljYWJsZSwgY2l0ZSB0aGUgcHJvdmlkZWQgc291cmNlIG1hdGVyaWFsLCByZWZlcmVuY2UgYW5zd2VyLCBvciBwb2xpY3kgdGV4dCB1c2luZyB0aGUgYXZhaWxhYmxlIGlkZW50aWZpZXIuIERvIG5vdCBpbnZlbnQgY2l0YXRpb25zIG9yIGNpdGUgdW5hdmFpbGFibGUgc291cmNlcy4KCiMjIyMgSW1wb3J0YW5jZQpBIGNhdGVnb3JpY2FsIGxhYmVsIHRoYXQgcmVwcmVzZW50cyB0aGUgaW1wb3J0YW5jZSBvZiB0aGUgY3JpdGVyaW9uLiBJbXBvcnRhbmNlIGNhbiBvbmx5IGJlIG9uZSBvZiAzIGNhdGVnb3JpZXM6Ci0gTWlub3IKLSBNYWpvcgotIENyaXRpY2FsCgpNaW5vcjogSW1wcm92ZXMgdGhlIHJlc3BvbnNlIGJ1dCBpcyBub3QgcmVxdWlyZWQgZm9yIGNvcnJlY3RuZXNzLgpNYWpvcjogTmVlZGVkIGZvciBhIHJlc3BvbnNlIHRvIGJlIGNvbXBsZXRlLgpDcml0aWNhbDogUmVxdWlyZWQgZm9yIGEgcmVzcG9uc2UgdG8gYmUgY29ycmVjdCBvciBzYWZlLgoKRXZlcnkgcnVicmljIGNyaXRlcmlvbiBtdXN0IGJlIHRhZ2dlZCB3aXRoIGl0cyBpbXBvcnRhbmNlIGxldmVsLCBzcGVjaWZ5aW5nIGhvdyBjcml0aWNhbCB0aGlzIGNoZWNrIGlzIGluIHRoZSBvdmVyYWxsIGV2YWx1YXRpb24gb2YgYSByZXNwb25zZS4gQSByZXNwb25zZSdzIG92ZXJhbGwgcXVhbGl0eSBzY29yZSBpcyBiYXNlZCBvbiB3aGljaCBydWJyaWMgY3JpdGVyaWEgaXQgc2F0aXNmaWVzIGFuZCB0aGUgcmVsYXRpdmUgaW1wb3J0YW5jZSBvZiB0aG9zZSBjcml0ZXJpYS4KCiMjIyMjIENyaXRpY2FsIEltcG9ydGFuY2UgRWxlbWVudHMKQ3JpdGVyaWEgdGhhdCByZXByZXNlbnQgbXVzdC1oYXZlIHJlcXVpcmVtZW50cy4gIFJlc3BvbnNlcyB0aGF0IGZhaWwgdG8gc2F0aXNmeSBhbnkgb2YgdGhlIGNyaXRpY2FsIGNyaXRlcmlhIGFyZSBmdW5kYW1lbnRhbGx5IHVuYWNjZXB0YWJsZS93cm9uZy4KCiMjIyMjIE1ham9yIEltcG9ydGFuY2UgRWxlbWVudHMKQ3JpdGVyaWEgdGhhdCBjb3ZlciBpbXBvcnRhbnQgYXNwZWN0cyBvZiB0aGUgcmVzcG9uc2UsIHdoaWNoIGFyZSBub3QgYXMgaW1wb3J0YW50IGFzIGNyaXRpY2FsIChtdXN0LWhhdmUpIGNyaXRlcmlhLCBidXQgYXJlIG1vcmUgaW1wb3J0YW50IHRoYW4gbWlub3IgKG5pY2UtdG8taGF2ZSkgY3JpdGVyaWEuIApSZXNwb25zZXMgdGhhdCBmYWlsIHRoZXNlIGNyaXRlcmlhIGFyZSBzaWduaWZpY2FudGx5IHdvcnNlIHRoYW4gcmVzcG9uc2VzIHRoYXQgcGFzcywgYnV0IGEgcmVzcG9uc2UgbWF5IHN0aWxsIGJlIGFjY2VwdGFibGUvY29ycmVjdCBpZiBpdCBvbmx5IGZhaWxzIGEgc2luZ2xlIG1ham9yIGNyaXRlcmlvbi4KCiMjIyMjIE1pbm9yIEltcG9ydGFuY2UgRWxlbWVudHMKQ3JpdGVyaWEgdGhhdCBwcm92aWRlIHBhcnRpYWwgY3JlZGl0IG9yIGFzc2VzcyBtaW5vciBwcm9wZXJ0aWVzLiAgQSByZXNwb25zZSB3aGljaCBmYWlscyBtdWx0aXBsZSBtaW5vciBjcml0ZXJpYSBtYXkgc3RpbGwgYmUgYWNjZXB0YWJsZS9jb3JyZWN0LCBqdXN0IG5vdCB0aGUgb3B0aW1hbCByZXNwb25zZS4KCiMjIyMgQ2F0ZWdvcnkKRWFjaCBydWJyaWMgY3JpdGVyaW9uIG11c3QgYmUgYW5ub3RhdGVkIHdpdGggb25lIG9mIHRoZSBmb2xsb3dpbmcgY2F0ZWdvcmllczoKCi0gRmFjdHVhbCBBY2N1cmFjeTogVmVyaWZpZXMgdGhhdCBhbGwgZmFjdHVhbCBjbGFpbXMsIG5hbWVkIGVudGl0aWVzLCBkYXRlcywgZmlndXJlcywgcXVvdGF0aW9ucywgYW5kIGFzc2VydGlvbnMgYXJlIGNvcnJlY3QsIG5vdCBtaXNsZWFkaW5nLCBhbmQgYXBwcm9wcmlhdGVseSBzdXBwb3J0ZWQuIEluY2x1ZGVzIGlkZW50aWZ5aW5nIGluY29ycmVjdCBjbGFpbXMgYXMgd2VsbCBhcyByZXF1aXJlZCBidXQgbWlzc2luZyBmYWN0dWFsIGFzc2VydGlvbnMuCgotIEluc3RydWN0aW9uIEZvbGxvd2luZzogQXNzZXNzZXMgd2hldGhlciB0aGUgcmVzcG9uc2Ugc3RyaWN0bHkgZm9sbG93cyBhbGwgaW5zdHJ1Y3Rpb25zLCBjb25zdHJhaW50cywgYW5kIHJlcXVpcmVtZW50cyBzcGVjaWZpZWQgaW4gdGhlIHRhc2sgcHJvbXB0IGFuZCBhc3NvY2lhdGVkIGFzc2V0cywgaW5jbHVkaW5nIGNvbmRpdGlvbmFsIHJlcXVpcmVtZW50cyBhbmQgZXhjbHVzaW9ucy4KCi0gUmVhc29uaW5nOiBFdmFsdWF0ZXMgdGhlIGxvZ2ljYWwgY29oZXJlbmNlIG9mIHRoZSByZXNwb25zZSwgaW5jbHVkaW5nIHRoZSBjb3JyZWN0bmVzcyBhbmQgY29tcGxldGVuZXNzIG9mIHJlYXNvbmluZyBzdGVwcywgY2FsY3VsYXRpb25zLCBleHBsYW5hdGlvbnMsIGFuZCBqdXN0aWZpY2F0aW9ucyB1c2VkIHRvIHN1cHBvcnQgY29uY2x1c2lvbnMuCgotIEV4dHJhY3Rpb246IENoZWNrcyB3aGV0aGVyIHRoZSByZXNwb25zZSBjb3JyZWN0bHkgaWRlbnRpZmllcywgcHVsbHMsIGFuZCB1c2VzIHJlbGV2YW50IGluZm9ybWF0aW9uIGZyb20gYSBwcm92aWRlZCBkYXRhIHNvdXJjZSAoZS5nLiwgaW1hZ2VzLCB0YWJsZXMsIFBERnMsIG9yIGdpdmVuIHRleHQpIHdoZW4gYXBwbGljYWJsZS4KCi0gRm9ybWF0dGluZzogQXNzZXNzZXMgd2hldGhlciB0aGUgcmVzcG9uc2UgYWRoZXJlcyB0byByZXF1aXJlZCBzdHJ1Y3R1cmFsIGFuZCBwcmVzZW50YXRpb24gcnVsZXMsIHN1Y2ggYXMgaGVhZGluZ3MsIGxpc3RzLCBtYXJrZG93biB1c2FnZSwgY2FsY3VsYXRpb24gbGF5b3V0LCBKU09OIHN0cnVjdHVyZSwgb3Igb3RoZXIgc3BlY2lmaWVkIGZvcm1hdHRpbmcgY29uc3RyYWludHMuCgotIFN0eWxlOiBFdmFsdWF0ZXMgdG9uZSwgY2xhcml0eSwgY29uY2lzZW5lc3MsIGFuZCB2b2ljZSB0byBlbnN1cmUgdGhlIHJlc3BvbnNlIGlzIGFwcHJvcHJpYXRlIGZvciB0aGUgdGFzaywgYXVkaWVuY2UsIGFuZCBjb250ZXh0LCB3aXRob3V0IHVubmVjZXNzYXJ5IHZlcmJvc2l0eSBvciBhbWJpZ3VpdHkuCgotIFNhZmV0eTogRW5zdXJlcyB0aGUgcmVzcG9uc2UgYXZvaWRzIGhhcm1mdWwsIGRpc2FsbG93ZWQsIG9yIHVuc2FmZSBjb250ZW50IGFuZCBpbmNsdWRlcyByZXF1aXJlZCB3YXJuaW5ncywgZGlzY2xhaW1lcnMsIG9yIHNhZmV0eSBndWlkYW5jZSB3aGVuIGFwcGxpY2FibGUuCgojIyMgRWZmZWN0aXZlIFJ1YnJpYyBDcml0ZXJpYSBFeGFtcGxlcwoKIyMjIyBFeGFtcGxlIDEgLS0gRGVtb25zdHJhdGluZyBDcml0aWNhbCBJbXBvcnRhbmNlCgoqKkNyaXRlcmlvbjoqKgpTdGF0ZXMgdGhhdCBnbHVjYWdvbi1saWtlIHBlcHRpZGUtMSByZWNlcHRvciBhZ29uaXN0cyAoR0xQLTEgUkFzKSBhcmUgbm90IEZEQS1hcHByb3ZlZCBmb3IgbWlncmFpbmUgdHJlYXRtZW50IGFzIG9mIE9jdG9iZXIgMjAyNC4KCioqUmF0aW9uYWxlOioqClRoZSB0YXNrIHByb21wdCBleHBsaWNpdGx5IGFza3MgZm9yIGluZm9ybWF0aW9uIG9uIEZEQSBhcHByb3ZhbCBzdGF0dXMgb2YgR0xQLTEgcmVjZXB0b3IgYWdvbmlzdHMgZm9yIG1pZ3JhaW5lLgoKKipJbXBvcnRhbmNlOioqCkNyaXRpY2FsCgojIyMjIEV4YW1wbGUgMiAtLSBEZW1vbnN0cmF0aW5nIE1ham9yIEltcG9ydGFuY2UKCioqQ3JpdGVyaW9uOioqCklkZW50aWZpZXMgdGhhdCByZXN0cmFpbnQgc3RyZXNzIGlzIHNob3duIHRvIGVsZXZhdGUgY29ydGljb3N0ZXJvbmUgbGV2ZWxzIGluIHJhdHMgYWZ0ZXIgMjAgbWludXRlcyBvZiBleHBvc3VyZS4KCioqUmF0aW9uYWxlOioqClN1cHBvcnRzIHRoZSBjb25jbHVzaW9uIHRoYXQgdGhlIGV4cGVyaW1lbnRhbCBjb25kaXRpb25zIGluZHVjZSBzdHJlc3MsIHdoaWNoIGNhbiBiZSBtZWFzdXJlZCB2aWEgY29ydGljb3N0ZXJvbmUgbGV2ZWxzLgoKKipJbXBvcnRhbmNlOioqCk1ham9yCgotLS0KCiMjIyMgRXhhbXBsZSAzIC0tIERlbW9uc3RyYXRpbmcgTWlub3IgSW1wb3J0YW5jZQoKKipDcml0ZXJpb246KioKSWRlbnRpZmllcyB0aGF0IHVzaW5nIHJhdHMgb2YgdGhlIHNhbWUgZ2VuZXRpYyBiYWNrZ3JvdW5kIHdvdWxkIHByZXZlbnQgZ2VuZXRpYyB2YXJpYWJpbGl0eSB0aGF0IGNvdWxkIGluZmx1ZW5jZSBleHBlcmltZW50YWwgcmVzdWx0cy4KCioqUmF0aW9uYWxlOioqCkVuc3VyaW5nIHRoZSByYXRzIHNoYXJlIHRoZSBzYW1lIGdlbmV0aWMgYmFja2dyb3VuZCB3b3VsZCBlbGltaW5hdGUgZ2VuZXRpYyB2YXJpYWJpbGl0eSBpbiB0aGUgb2JzZXJ2ZWQgZGlmZmVyZW5jZXMgaW4gc3RyZXNzIGhvcm1vbmVzIGFuZCBiZXRhLWFycmVzdGluIDIgbGV2ZWxzLgoKKipJbXBvcnRhbmNlOioqCk1pbm9yCg==)\#\#CreatingaGoodRubric\#\#\#What'sarubric?ThegoaloftherubricistoprovideasystematicchecklistthatcanbeusedtogradethequalityofanyAIresponsetoagiventaskprompt\.Yourrubricwillbeagenerallyapplicablelistofspecificcriteriathat,whentakenasawhole,canevaluateanypossibleresponsetothetaskprompt\.Eachcriterionmustbeasimple,declarativestatementthatiseasilyverifiedaseithertrueorfalse\.Rubriccriteriashouldbeobjectiveandspecificenoughthattheycanbeevaluatedwithoutspecializedknowledge\.Together,therubricprovidestherecipefortheidealanswer,allowingresponsestobegradedandrankedbytheirquality\.\#\#\#HowtoThinkAboutYourRubric\-Includeeverythingyouwouldwantinaperfectresponse\.Thisincludesnotjustfactsandstatementsdirectlyaskedforbythetaskprompt,butalsonice\-to\-havesupportinginformation\(justification,reasoning,orlogicthatenhancesaresponse\)\.\-Eachmeaningfulelementdeservesarubriccriterion\.\-Eachrubriccriterionshouldevaluatetheresponsefromadifferentangle\.\-Awell\-designedrubriccapturesbothcriticalfailuresandfinerdistinctions:criteriashouldvaryinimportanceaccordingly\.\-Beyondincludingcriteriathatcheckforspecificinformationintheresponse,alsoanticipatevariousflawsthatresponsesmightexhibitandincluderubriccriteriathatcheckforeachoftheseflaws\.Suchcriteriawilloftenstartwith:"Doesnot\.\.\."Guidelines:\-Writecriteriastartingwithasinglestrongverb,e\.g\.:\-"Includes\.\.\."\-"States\.\.\."\-"Calculates\.\.\."\#\#\#PropertiesofEffectiveRubricCriteria\#\#\#\#ClearandunambiguousEachcriterionshouldbeclear,unambiguous,specific,objective,andconcise\.Criteriashouldbebinarystatementsthateitherevaluateastrueorfalse\.\*\*BadExample\*\*BAD:StatesthattheP/Eisequivalentto22\.2x\.Thiscriterionneedsimprovement\.It'sunclearwhoseP/E,whatisP/E,isitamultipleforwhatyear\.\*\*ImprovedExample\*\*IMPROVED:StatesthatT\-MobileUSA,Inc\.P/Emultipleisequivalentto22\.2xforYearendedDecember31,2024\.\#\#\#\#Self\-ContainedEachcriterionwillbeevaluatedonthetaskresponseindependently,separatelyfromtheothersandwithoutexternalcontext\.Itmustcontaincomplete,fullinformationsowecanjudgewhetherthecriterionistrueorfalsefortheparticulartaskresponse\.Thegraderwillonlybeabletoseetheresponseitisgradingandasinglecriterionatatime\.Eachcriterionshouldcontainallnecessaryinformationtoquicklygradearesponse\.Thismeanseverycriterionmustbesufficientlyspecific\.\*\*NotSelf\-Contained\*\*BAD:Mentionsthemaleleadactorinthe2021movie"Dune:PartOne"\.Thiswouldnotbeconsideredassessablewithoutknowingwhotherelevantactorsare\.Youwouldberequiredtosearch"whoisthemaleleadactorinDune"toassessifthecriterionistrueorfalse\.\*\*Self\-Contained\*\*IMPROVED:Mentionsthatthemaleleadactorinthe2021movie"Dune:PartOne"isTimotheeChalamet\.It'spossibletoassesswhetherthetaskresponsementions"TimotheeChalamet"\.Criteriashouldalsobeevaluablewithoutinternetaccess\.Avoidcriteriathatrequireadditionalresearchviatheinternet\.\*\*Internet\-Dependent\*\*BAD:Responsementionsanyoftheactorsin"ThePrincessBride"Criterialikethisshouldbemorespecific,evenifitmakesthemlong\.\*\*Self\-Contained\*\*IMPROVED:Responsementionsanyofthefollowingactors:CaryElwes,MandyPatinkin,RobinWright,ChrisSarandon,ChristopherGuest,WallaceShawn,AndretheGiant\(alsoknownasAndreReneRoussimoff\),FredSavage,PeterFalk,PeterCook,MelSmith,CarolKane,BillyCrystal,AnneDyson,MargeryMason,MalcolmStorry,WilloughbyGray,orBetsyBrantley\.Relatedtointernetaccess,submittingrubricitemslike"isthislinkbroken"or"doesthislinkgotoa404errorpage"isnotanswerablewithoutinternetaccess\.Pleaseavoidthoserubricitems\.\#\#\#\#AvoidAmbiguity\*\*Ambiguous\*\*BAD:Statesthemaximumdrawdownoftherisk\-aversestockisequivalentto16\.33%\.Thiscriterionneedsimprovementbecausewedonotknowwhosedrawdownwearecomputing\.\*\*ClearCriteria\*\*IMPROVED:Statestheaveragemaximumdrawdownoftherisk\-aversestockWECEnergyisequivalentto16\.33%\.\#\#\#\#NoCross\-ReferencesCriteriamustnotreferenceeachother\.\*\*Cross\-ReferencingCriteria\*\*BAD:IncludesavalidexperimentalapproachnotlistedaboveWhat'sabove?Whatifweshuffletheorder?Thisalsomeansthatabbreviationsneedtobeexpandedineverycriterion\.\*\*UnclearSource\*\*BAD:MentionsthatAndersonetallookatcontrolofmemoryretrieval\.\*\*ClearSource\*\*IMPROVED:Mentionsthat"NeuralSystemsUnderlyingtheSuppressionofUnwantedMemories"\(Andersonetal\.,Science,2004\)looksatcontrolofmemoryretrieval\.Ifyouhaveafewconsecutivecriteriaaboutthesamescientificpaper,eachcriterionshouldrepeatthenameofthepaper\.\#\#\#\#Non\-StackedCriteriaEachcriterionshouldhaveasinglestatement/thesis,soapartiallycorrectanswercanbetterearnpartialcredit\.Moststackedcriteriawiththeword"and"canbebrokenupintomultiplepieces\.\*\*StackedCriteria\*\*BAD:IncludestheGreekalphabetandstatesthatithas24letters\.ButwhathappensifaresponsementionstheGreekalphabetbutdoesn'tstatehowmanylettersithas?Thenthisitemwouldbefalse,eventhoughit's50%true\.Abetterwaytowritethiscriterionmightbetosplititup\.\*\*UnstackedCriteria\*\*IMPROVED:IncludestheGreekalphabet\.IMPROVED:StatesthattheGreekalphabethas24letters\.Thatsaid,youaretheexpertonyourowntask\.Ifyoubelievethattwodifferentthingsneedtobeincludedforeithertohaveanyvalue,youcangroupthemtogether\.Forexample,ataskthatincludes"WhatarethecoordinatesoftheImperialPalaceinTokyo,totwodecimalplaces?"mighthavethefollowingcriteria:GOOD:Includesthelongitude139\.75degEandthelatitude35\.69degNThisisonestatement,andnotastackedcriterionbecausealongitudeisnotveryhelpfulwithoutalatitude\!Here'sanotherexample:\*\*StackedCriteria\*\*BAD:StatesthatAT&T'sandT\-Mobile'sP/EMultipleforyearendedDecember31,2024,isequivalentto13\.3xand22\.2x,respectively\.\*\*UnstackedCriteria\*\*IMPROVED:StatesthatT\-Mobile'sP/EMultiplefortheyearendedDecember31,2024,isequivalentto22\.2x\.IMPROVED:StatesthatAT&T'sP/EMultiplefortheyearendedDecember31,2024,isequivalentto13\.3x\.\#\#\#\#CanincludecriteriaforcommonmistakestoavoidExamplesofwhataresponseshouldavoidcanbejustasusefulasexamplesofwhatitshouldinclude\.Anytaskwillhaveaninfinitenumberofincorrectresponses\.However,thesecriteriacanbeusefulwhentheytargetspecificmistakesthemodelislikelytomakeinitsresponse\.Forexample:GOOD:Doesnotincludethecoordinatepair39\.92degN,116\.40degE\.GOOD:DoesnotmentionirrelevantconceptX\.Acriterionispassedsuccessfullywhentheresponseavoidsthespecifiedmistake\.\#\#\#\#OrderofcriteriaTheorderofcriteriaintherubricdoesn'tmatter\.\#\#\#\#DefaultCriteriaYoucanassumethefollowingaredefaultcriteriaforallgoodanswers,andsoyoudon'tneedtowritecriteriaabout:\-Correctgrammarandspelling\#CriteriaPropertiesEachcriterionshouldbetaggedwiththefollowingpiecesofsupportinginformation\.\#\#\#\#RationaleAbriefjustificationforincludingthecriterion,groundedinfactualreasoning,establishedtheory,orverifiableaspectsofthetask\.Whenapplicable,citetheprovidedsourcematerial,referenceanswer,orpolicytextusingtheavailableidentifier\.Donotinventcitationsorciteunavailablesources\.\#\#\#\#ImportanceAcategoricallabelthatrepresentstheimportanceofthecriterion\.Importancecanonlybeoneof3categories:\-Minor\-Major\-CriticalMinor:Improvestheresponsebutisnotrequiredforcorrectness\.Major:Neededforaresponsetobecomplete\.Critical:Requiredforaresponsetobecorrectorsafe\.Everyrubriccriterionmustbetaggedwithitsimportancelevel,specifyinghowcriticalthischeckisintheoverallevaluationofaresponse\.Aresponse'soverallqualityscoreisbasedonwhichrubriccriteriaitsatisfiesandtherelativeimportanceofthosecriteria\.\#\#\#\#\#CriticalImportanceElementsCriteriathatrepresentmust\-haverequirements\.Responsesthatfailtosatisfyanyofthecriticalcriteriaarefundamentallyunacceptable/wrong\.\#\#\#\#\#MajorImportanceElementsCriteriathatcoverimportantaspectsoftheresponse,whicharenotasimportantascritical\(must\-have\)criteria,butaremoreimportantthanminor\(nice\-to\-have\)criteria\.Responsesthatfailthesecriteriaaresignificantlyworsethanresponsesthatpass,butaresponsemaystillbeacceptable/correctifitonlyfailsasinglemajorcriterion\.\#\#\#\#\#MinorImportanceElementsCriteriathatprovidepartialcreditorassessminorproperties\.Aresponsewhichfailsmultipleminorcriteriamaystillbeacceptable/correct,justnottheoptimalresponse\.\#\#\#\#CategoryEachrubriccriterionmustbeannotatedwithoneofthefollowingcategories:\-FactualAccuracy:Verifiesthatallfactualclaims,namedentities,dates,figures,quotations,andassertionsarecorrect,notmisleading,andappropriatelysupported\.Includesidentifyingincorrectclaimsaswellasrequiredbutmissingfactualassertions\.\-InstructionFollowing:Assesseswhethertheresponsestrictlyfollowsallinstructions,constraints,andrequirementsspecifiedinthetaskpromptandassociatedassets,includingconditionalrequirementsandexclusions\.\-Reasoning:Evaluatesthelogicalcoherenceoftheresponse,includingthecorrectnessandcompletenessofreasoningsteps,calculations,explanations,andjustificationsusedtosupportconclusions\.\-Extraction:Checkswhethertheresponsecorrectlyidentifies,pulls,andusesrelevantinformationfromaprovideddatasource\(e\.g\.,images,tables,PDFs,orgiventext\)whenapplicable\.\-Formatting:Assesseswhethertheresponseadherestorequiredstructuralandpresentationrules,suchasheadings,lists,markdownusage,calculationlayout,JSONstructure,orotherspecifiedformattingconstraints\.\-Style:Evaluatestone,clarity,conciseness,andvoicetoensuretheresponseisappropriateforthetask,audience,andcontext,withoutunnecessaryverbosityorambiguity\.\-Safety:Ensurestheresponseavoidsharmful,disallowed,orunsafecontentandincludesrequiredwarnings,disclaimers,orsafetyguidancewhenapplicable\.\#\#\#EffectiveRubricCriteriaExamples\#\#\#\#Example1\-\-DemonstratingCriticalImportance\*\*Criterion:\*\*Statesthatglucagon\-likepeptide\-1receptoragonists\(GLP\-1RAs\)arenotFDA\-approvedformigrainetreatmentasofOctober2024\.\*\*Rationale:\*\*ThetaskpromptexplicitlyasksforinformationonFDAapprovalstatusofGLP\-1receptoragonistsformigraine\.\*\*Importance:\*\*Critical\#\#\#\#Example2\-\-DemonstratingMajorImportance\*\*Criterion:\*\*Identifiesthatrestraintstressisshowntoelevatecorticosteronelevelsinratsafter20minutesofexposure\.\*\*Rationale:\*\*Supportstheconclusionthattheexperimentalconditionsinducestress,whichcanbemeasuredviacorticosteronelevels\.\*\*Importance:\*\*Major\-\-\-\#\#\#\#Example3\-\-DemonstratingMinorImportance\*\*Criterion:\*\*Identifiesthatusingratsofthesamegeneticbackgroundwouldpreventgeneticvariabilitythatcouldinfluenceexperimentalresults\.\*\*Rationale:\*\*Ensuringtheratssharethesamegeneticbackgroundwouldeliminategeneticvariabilityintheobserveddifferencesinstresshormonesandbeta\-arrestin2levels\.\*\*Importance:\*\*Minor

## Appendix GAgent Rubric Generation Baseline

The Agentic baseline runs the GPT\-5\.1 model inside the Codex SDK agent harness\([OpenAI, 2026](https://arxiv.org/html/2610.38220#bib.bib24)\)in a per\-task Docker container\. Each task gets a fresh workspace directory mounted at/workspace; the agent is started withsandbox=workspace\_writeandapproval\_mode=auto\_review, with the workspace as its working directory and no extra developer instructions\. Within this sandbox, the agent may read and write files in the workspace and run shell and Python commands\. Network access for tool commands is disabled\.

##### Workspace files\.

The workspace containsprompt\.txt, which is the rubric generation prompt in Appendix[F\.1](https://arxiv.org/html/2610.38220#A6.SS1), rendered with the task prompt only\. No example responses or reference material are included in the generation prompt\. They are instead placed in the workspace and surfaced to the agent via a providedAGENTS\.mdfile, shown below, which Codex loads as project\-level instructions\.

Agentic baseline harness prompt[⬇](data:text/plain;base64,IyBSdWJyaWMgR2VuZXJhdGlvbgoKUHJvZHVjZSBhIHJpZ29yb3VzIGV2YWx1YXRpb24gcnVicmljIGZvciB0aGUgdGFzayBkZXNjcmliZWQgaW4gdGhlIHVzZXIgcHJvbXB0LgpVc2UgdGhlIHdvcmtzcGFjZSBmaWxlcyBhcyBpbnB1dHMgdG8geW91ciBydWJyaWMgZGVzaWduLgpZb3UgaGF2ZSBhY2Nlc3MgdG8gUHl0aG9uIGFuZCBzaGVsbCB0b29scy4KCiMjIFdvcmtzcGFjZQoKLSBgcmVzcG9uc2VzLmpzb25sYCDigJQgRXhhbXBsZSBBSSByZXNwb25zZXMgdG8gdGhlIHRhc2sgKG9uZSBKU09OIG9iamVjdCBwZXIgbGluZSkuCgooKkBccGh7XHRleHRjb2xvcnt0ZWFsITYwIWJsYWNrfXtbaW5jbHVkZWQgb25seSBpZiBleHBlcnQgc3RlcC1ieS1zdGVwIHJlYXNvbmluZyBpcyBhdmFpbGFibGVdfX1AKikKLSBgZXhwZXJ0X3N0ZXBfYnlfc3RlcC50eHRgIEV4cGVydCBtZXRob2RvbG9neSBmb3IgdGhpcyB0YXNrLgoKKCpAXHBoe1x0ZXh0Y29sb3J7dGVhbCE2MCFibGFja317W2luY2x1ZGVkIG9ubHkgaWYgYW4gZXhhbXBsZSByZXNwb25zZSBpcyBhdmFpbGFibGVdfX1AKikKLSBgZXhhbXBsZV9yZXNwb25zZS50eHRgIEV4YW1wbGUgaHVtYW4gcmVzcG9uc2UgdG8gdGhpcyB0YXNrLgoKIyMgT3V0cHV0CgpTYXZlIHRoZSBmaW5hbCBydWJyaWMgYXMgYHJ1YnJpYy5qc29uYC4K)\#RubricGenerationProducearigorousevaluationrubricforthetaskdescribedintheuserprompt\.Usetheworkspacefilesasinputstoyourrubricdesign\.YouhaveaccesstoPythonandshelltools\.\#\#Workspace\-‘responses\.jsonl‘—ExampleAIresponsestothetask\(oneJSONobjectperline\)\.\[included only if expert step\-by\-step reasoning is available\]\-‘expert\_step\_by\_step\.txt‘Expertmethodologyforthistask\.\[included only if an example response is available\]\-‘example\_response\.txt‘Examplehumanresponsetothistask\.\#\#OutputSavethefinalrubricas‘rubric\.json‘\.