通过对抗性元提示演化实现递归奖励改进

# Recursive Reward Improvement viaAdversarial Meta-Prompt Evolution
Source: [https://arxiv.org/html/2610.21040](https://arxiv.org/html/2610.21040)
Qianchu LiuAffiliation:Microsoft ResearchGuanghui QinAffiliation:Microsoft ResearchXing HanAffiliation:Johns Hopkins UniversityTristan NaumannAffiliation:Microsoft ResearchPaul Pu LiangAffiliation:Massachusetts Institute of TechnologySheng ZhangAffiliation:Microsoft Research

###### Abstract

Recursive Self\-Improvement \(RSI\) has gained wide attention for its ability to work with limited data\. In a proposer–solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning\. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically\. Open\-ended expert tasks, like clinical report generations, have no answer key, which makes simple question\-answer generation not viable\. In these tasks, the proposer usually must also write a rubric that lists the content an answer should include and the errors it should avoid\. However, the unique challenge behind model\-written rubric is the comprehensiveness of it\. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the entire learning process\. To solve this problem, we introduceRRI,RecursiveRewardImprovement, which utilizes a specially designed reward hacking adversary agent as the supervision for the reward\. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win\. The exploits found also rewrite how future tasks and rubrics are generated\. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self\-improvement baseline by 10\.9 points at 9B and 12\.4 at 27B, and by 10\.3 points without any external model\. In particular, the gains are largest on difficult and writing tasks\. We show that the 27B model trained on the HealthBench Professional description improves by 6\.6 points on PRBench Hard and 8\.0 on ProfBench without seeing a task from either\. We further provide detailed analysis on exploitable points of models’ generated question–rubric pairs to help facilitate future research in open\-ended self play methods\.

## 1Introduction

Many expert tasks in medicine, law, and finance are open\-ended, such as writing a clinical report or answering a patient’s question\. Training data for these tasks are scarce, because an expert must write each task and decide what a good answer contains\([Arora et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib13);[Akyürek et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib56);[Wang et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib57)\)\. Recursive Self\-Improvement \(RSI\), in which a model learns from data it generates itself, can supply such data at little cost\. Models have learned from their own rationales, outputs, and judgments, and in proposer–solver loops a proposer writes tasks for a solver to learn\([Zelikman et al\., 2022](https://arxiv.org/html/2610.21040#bib.bib7);[Yuan et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib8);[Chen et al\., 2024b](https://arxiv.org/html/2610.21040#bib.bib9);[Zhao et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib4);[Huang et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib5);[Liu et al\., 2025a](https://arxiv.org/html/2610.21040#bib.bib6)\)\. These loops have succeeded in mathematics, coding, and scientific discovery\([Gao et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib32);[Qu et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib12);[Novikov et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib59);[Zhang et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib60)\), where a final number, a unit test, or an automatic evaluator checks every answer\. Open\-ended answers have no such check, so a model that writes an open\-ended task must also write the reward for it\. How to supervise this model\-written reward remains the core unsolved problem\.

Supervising generated data raises two main challenges\. First, the reward must be checked without a reference\. Self\-play loops such as Absolute Zero, R\-Zero, and SPICE write their own tasks and grade answers by execution or by an answer key\([Zhao et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib4);[Huang et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib5);[Liu et al\., 2025a](https://arxiv.org/html/2610.21040#bib.bib6);[Wilf et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib11)\), which open\-ended tasks lack\. For open\-ended work, the standard reward is usually a rubric, a weighted list of content an answer should include and errors it should avoid\. Rubric rewards for reinforcement learning anchor each rubric in human data, such as preference labels, a golden rubric, or an expert answer\([Gunjal et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib14);[Liu et al\., 2026b](https://arxiv.org/html/2610.21040#bib.bib36);[Ding et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib66);[Xu et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib65)\)\. When the model writes its own training data, none of these references exists\. Trivially, training data proposers can propose a rubrics defining a good answers at the same time they generate them\([Shao et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib24)\)\. However, the solver has a tendency to exploit the rubrics\([Amodei et al\., 2016](https://arxiv.org/html/2610.21040#bib.bib18);[Skalse et al\., 2022](https://arxiv.org/html/2610.21040#bib.bib19)\)\. As our experiments show, model\-written criteria tend to be ambiguous and incomprehensive, so an answer can name every expected topic, omit the facts behind them, and still earn the score, with nearly 99% of the model\-generated question\-rubric pairs classified as hackable\. Modern defenses against reward hacking usually correct biases that hold across tasks\([Chen et al\., 2024a](https://arxiv.org/html/2610.21040#bib.bib21);[Park et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib22)\), and no longer apply when the rubric varies across questions\.

To address these challenges, we proposeRRI,RecursiveRewardImprovement, one of the first self\-evolving pipelines that improves its own reward alongside its tasks and its solver, with reward hacking as adversarial supervision\. In RRI, the loop hacks its own reward first\. Before the solver trains on a generated task, the model tries to game the task’s rubric, measures the attempt against its own honest answer, and repairs the rubric to make the honest answer win\. In an outer meta\-optimizer loop, every hack it finds also changes how future tasks and rubrics are written\. The reward is thus checked without any human reference, and the solver trains on rewards that have already been tested against the hacks it would otherwise find\.

In this work, our contributions are as follows\.

1. 1\.We propose RRI, a self\-evolving pipeline that tests and repairs every generated rubric against its own exploits before the solver trains on it, improving its reward alongside its tasks and its solver with no training data\.
2. 2\.We evaluate RRI in and out of domain and find improvements in both\. With zero training data, RRI\-9B and RRI\-27B improve the adjusted score on HealthBench Professional over the Generation Only self\-improvement baseline by 10\.9 and 12\.4 points, and by 10\.3 points when the frozen 9B model serves every auxiliary role\. Adding the adversary to a loop that already co\-evolves the tasks and the reward adds 2\.7 points in the 9B setting\. Trained on the HealthBench Professional description, RRI\-27B improves over the base model by 6\.6 points on PRBench Hard and 8\.0 points on ProfBench, whose non\-medical tasks it never saw, while Generation Only trained on the same description gains 1\.2 and 4\.7 points\.
3. 3\.We analyze reward generation throughout training, classifying the 12,147 exploits the loop finds across medical, legal, financial, and scientific tasks into five categories of reward failure plus one category of incorrect fixes\. One third of the exploits carry wrong content that no criterion penalizes, and a further third turn on how a loosely worded criterion is read\. From these categories we identify requirements for generated rewards that can guide future work\.

We release all code, prompts, and configurations at[https://github\.com/DDVD233/RRI](https://github.com/DDVD233/RRI)\.

![Refer to caption](https://arxiv.org/html/2610.21040v1/RRIMed_main_figure.png)Figure 1:Overview of RRI\.The proposer generates clinical tasks and weighted rubrics from medical references\. An adversary probes each rubric with an exploit and an honest answer\. The repair step keeps criteria that distinguish the better answer from the exploit, then tests the revised rubric with another attack\. The solver learns through GRPO using rubric rewards\. Training feedback and confirmed exploits guide updates to the generation prompts\.
## 2Related Work

Self\-play and self\-evolving training\.Model\-generated supervision began with bootstrapped rationales and instructions\([Zelikman et al\., 2022](https://arxiv.org/html/2610.21040#bib.bib7);[Wang et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib30)\)and became iterative through self\-judging and self\-play\([Guan et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib31);[Gao et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib32);[Yuan et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib8);[Chen et al\., 2024b](https://arxiv.org/html/2610.21040#bib.bib9);[Zhao et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib4);[Huang et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib5);[Liu et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib10)\)\. Task data has been scaled by rewriting instructions or sampling a model’s own prior\([Xu et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib33);[Xu et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib34)\), generated from medical literature\([Huang et al\., 2026b](https://arxiv.org/html/2610.21040#bib.bib2)\), and graded with reference\-free rubrics\([Jayalath et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib35)\)\. Our meta\-optimizer descends from automatic prompt optimization\([Khattab et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib25);[Agrawal et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib26)\)and learns from confirmed exploits\.

Adversarial checks, rubric rewards, and reward hacking\.In code and factuality, critics, test writers, buggy programs, and mutants attack an answer or a test suite\([Wu et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib61);[Ruan et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib62);[Li et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib63);[Chang et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib64)\), and execution or an encyclopedia settles each contest\. Rubrics make open\-ended generation checkable at scale\([Viswanathan et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib15)\)\. Rubric rewards are written from reference answers\([Gunjal et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib14)\), filtered by preference labels\([Liu et al\., 2026b](https://arxiv.org/html/2610.21040#bib.bib36)\), grown from contrasting rollouts and kept by score variance\([Shao et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib24)\), trained toward a golden rubric\([Ding et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib66)\), or scored against an expert answer\([Xu et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib65)\)\. Each relies on a reference outside the loop or on score spread, which shows that a criterion separates answers but not that it prefers the better one\. Optimizers exploit the gap between a proxy reward and its intent\([Denison et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib20);[Gao et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib23)\), reinforcement learning amplifies what its reward admits\([Zhao et al\., 2025a](https://arxiv.org/html/2610.21040#bib.bib38)\), language\-model judges have known biases\([Zheng et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib37)\), and having one model expose another underlies debate, red\-teaming, and weak\-to\-strong supervision\([Irving et al\., 2018](https://arxiv.org/html/2610.21040#bib.bib27);[Samvelyan et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib28);[Burns et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib29)\)\. In RRI, an adversary sees the rubric and writes a deliberately useless answer, and among the methods compared in Table[1](https://arxiv.org/html/2610.21040#A1.T1)of Appendix[A](https://arxiv.org/html/2610.21040#A1), RRI alone evolves both the tasks and the reward\.

Medical language models and RL infrastructure\.Medical capability has been measured with clinical records and physician\-written rubric tasks\([Arora et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib13);[Johnson et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib3)\)and pursued with domain\-specialized models, domain\-aware reinforcement learning, and retrieval\-augmented generation\([Singhal et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib39);[Dai et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib1);[Zhao et al\., 2025b](https://arxiv.org/html/2610.21040#bib.bib51);[Xiong et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib55)\)\. Our solver builds on group\-relative policy optimization and its successors\([Shao et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib16);[Lambert et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib40);[Liu et al\., 2025b](https://arxiv.org/html/2610.21040#bib.bib17)\), decoupled rollout infrastructure\([Sheng et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib41)\), and agentic search loops\([Jin et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib42);[Li et al\., 2026b](https://arxiv.org/html/2610.21040#bib.bib43)\)\.

## 3Method

As shown in Figure[1](https://arxiv.org/html/2610.21040#S1.F1), RRI consists of five components\. A proposer writes tasks and rubrics from medical references, an adversary tests each rubric for exploits, a judge scores every answer in the loop against its rubric, a solver learns from the resulting rewards, and a meta\-optimizer, a language model that reads the training feedback and the confirmed exploits, rewrites the instructions the proposer follows when it writes tasks and rubrics\. In the self\-improvement setting, a frozen copy of the solver’s base model fills every role other than the solver\. In the teacher\-student setting, the teacher model fills every role other than the solver, and the solver is the student model\. Below we describe the details of each component\.

In our work, each task consists of a requestxxand a rubricR=\{\(cj,pj\)\}j=1mR=\\\{\(c\_\{j\},p\_\{j\}\)\\\}\_\{j=1\}^\{m\}, wherecjc\_\{j\}is a criterion andpj∈\[−10,10\]∖\{0\}p\_\{j\}\\in\[\-10,10\]\\setminus\\\{0\\\}is its signed weight\. Positive criteria reward required content, while negative criteria penalize specified errors\. Given a responseyy, the judge returns the setM⁡\(y\)M\(y\)of criterion indices it judges met\. The rule that mapsM⁡\(y\)M\(y\)to a scalar scores⁡\(y\)s\(y\), the reward the solver is trained on, belongs to the target benchmark, and our experiments use each benchmark’s own rule, given in Section[4\.1](https://arxiv.org/html/2610.21040#S4.SS1)\.

### 3\.1Proposer

The proposer receives only a general description of the target task and no training data\. It writes diverse clinician requests that double as retrieval queries, retrieves medical passages for each request from the reference database, and then writes the completed task and its rubric in one call from those references\. Grounding the rubric in retrieved passages lets it require specific clinical values or thresholds that the references support\. Appendix[B](https://arxiv.org/html/2610.21040#A2)gives the seed sources and validation rules\.

Each rubric lists the content an answer should include and the errors it should avoid, and every criterion carries a signed weight between−10\-10and1010points as defined in Section[3](https://arxiv.org/html/2610.21040#S3)\. To keep tasks informative, the generator receives feedbacks from the meta\-optimizer, which summarizes solver’s mean performance with respect to each task\.

### 3\.2Adversary Rubric Hacking & Repair

Figure 2:\(a\) HealthBench Professional adjusted scores with retrieval\.Bars show the untrained, Generation Only, and RRI models at 9B and 27B with a GPT Teacher and at 9B with a Self Teacher\.\(b\) HealthBench Professional raw scores with retrieval\.Bars show the same models and settings without the length adjustment\.A model\-generated rubric can reward superficial coverage without requiring the clinical content needed to answer the request\. For example, a common criterion phrased as “mentions that …” gives a vague requirement where a model simply mentioning a keyword would satisfy the requirement, without being helpful\. The adversary exposes and fixes such criteria in two stages, hack and patch, as described below\.

#### Hack\.

In the hack stage, the adversary receives each new task and rubric from the proposer and writes a*reward hacking*answer, one that satisfies as many criteria as possible while being as useless as possible\. In parallel, the same frozen model writes an honest answer from the task alone, without seeing the rubric\. The judge scores both answers under the current proposed rubrics, and we compute the lead of the exploit asΔ=s⁡\(yexploit\)−s⁡\(yhonest\)\\Delta=s\(y\_\{\\text\{exploit\}\}\)\-s\(y\_\{\\text\{honest\}\}\)\. The adversary wins unless the honest answer outscores the exploit by at least a marginδ\\delta\. We setδ=0\.20\\delta=0\.20in all experiments\. In other words, we accept the question directly when the honest answer receives a score that is0\.200\.20higher than the hacking answer\. Otherwise, we designate the question as hackable, and send the question to the patching stage\.

#### Patch\.

When the adversary wins, the model proceeds to rewrite the question–rubric pairs it proposed\. Given the exploit answerAA, the honest answerBB, and the current rubric, the repair step proposes both positive and negative criteria\. The judge evaluates each proposed criterion against both answers, and we retain it only when

negative criterionmet​\(A\)∧¬met​\(B\),positive criterionmet​\(B\)∧¬met​\(A\)\.\\text\{negative criterion\}\\quad\\text\{met\}\(A\)\\wedge\\neg\\text\{met\}\(B\),\\qquad\\text\{positive criterion\}\\quad\\text\{met\}\(B\)\\wedge\\neg\\text\{met\}\(A\)\.\(1\)In other words, a criterion is kept only when it gives the honest answer a higher score than the hacking answer\. Given the feedback, the proposer rewrite the question alongside the rubric together with the exploit as feedback\. In clinical settings, a positive criterion usually rewards naming missing examination findings, while a negative criterion usually penalizes invented findings or dangerous suggestions\. In our testings, we see about four fifths of accepted criteria are positive, as reported in Section[L\.5](https://arxiv.org/html/2610.21040#A12.SS5)\.

#### Admission\.

The goal of repair is a better task and rubric, not one the adversary can never beat\. A rewrite is adopted only when a fresh probe shows a smallerΔ\\Deltathan before\. When the adversary still wins against the repaired rubric, it has found a new loophole\. This shows that the adversary stays strong enough to keep finding loopholes, and the new exploit gives the next round a concrete target\. After at most two rounds, every task enters the training pool, including tasks whose rubric the adversary still beats\. We choose two rounds here as it’s a balance between the compute and the benefit of adversary patching\. Each such task is better than the one the proposer first wrote, and keeping it preserves the diversity of the training tasks\.

### 3\.3Solver

The solver receives each admitted task with its current rubric and generates a group ofnnresponses\. The judge scores every response with the patched rubrics\. We train the solver with Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib16)\), in which the advantage of responseiiisA^i=s⁡\(yi\)−1n​∑ks⁡\(yk\)\\hat\{A\}\_\{i\}=s\(y\_\{i\}\)\-\\frac\{1\}\{n\}\\sum\_\{k\}s\(y\_\{k\}\), the mean of its pairwise score differencess⁡\(yi\)−s⁡\(yk\)s\(y\_\{i\}\)\-s\(y\_\{k\}\)within the group\. Training therefore encourages responses that the rubric ranks above others in the group\.

Depending on the experimental setting, the solver can make up to two tool calls in each rollout for citation purposes, to the clinical database or to web search\. Retrieved results are inserted as masked tool turns\. Appendix[G](https://arxiv.org/html/2610.21040#A7)describes the grading protocol, and Appendix[E](https://arxiv.org/html/2610.21040#A5)describes the solver tools, reward settings, and optimization hyperparameters\.

### 3\.4Meta\-optimizer

To make the improvement persistent and avoid making the same error over and over again, we employ a meta\-optimizer, which updates the proposer input at the end of each step\. Specifically, an analyst first examines samples of graded training cases from the most recent steps\. It classifies failure modes, rubric defects, and task defects, and assigns each case one of three verdicts,*model failed a fair task*,*generator at fault*, or*model succeeded*\. The aggregated verdicts identify tasks that are too easy or too hard for the current solver and the specific errors the solver makes that the rubrics should weigh more heavily\. Corpus statistics such as criteria per task, negative share, use\-case mix, and task self\-similarity are compared with the benchmark reference to flag distribution drift and repeated task templates\.

The meta\-optimizer receives these summaries, the score formula, confirmed exploits and accepted patches, and the proposer’s current task and rubric generation guidance\. It rewrites the guidance blocks appended to the two generation prompts to incorporate this feedback\. The guidance defines what kind of questions, tasks, and clinical domains the proposer should generate next, and what kind of capabilities the current solver has\. Appendix[D](https://arxiv.org/html/2610.21040#A4)gives the memo objective and the evolution settings\.

The weights of every component except the solver remain frozen throughout training\. Appendix[E](https://arxiv.org/html/2610.21040#A5)gives the configuration of every setting\.

## 4Experiments

We design experiments to answer three research questions\.RQ1\.Does RRI improve in\-domain performance when no training data is available?RQ2\.Does the improvement generalize to out\-of\-domain benchmarks whose tasks are completely different?RQ3\.How does the hack–patch–improve loop change the quality and the generalizability of what the solver learns?

Figure 3:\(a\) Rubric\-reward baselines and component ablation in the 9B GPT Teacher setting\.Bars show RaR with synthetic tasks, EvoRubrics, and SibylSense, followed by Generation Only, Co\-evolve Only, and full RRI with hack\-then\-patch rounds\.\(b\) Adjusted scores by task category and difficulty in the 9B GPT Teacher setting\.Bars compare Generation Only, Co\-evolve Only, and RRI\-9B\.### 4\.1Experimental Setup

Datasets and Training Setups\.For RQ1, we use HealthBench Professional\([Hicks et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib44)\)as the primary target dataset, which contains 525 physician\-written tasks spanning consultation, research, and writing, with typical and difficult subsets\. Each task includes a physician\-authored rubric\. The proposer receives only a general description of the benchmark, every training task and rubric is model\-generated, and the benchmark serves only for validation\. Every trained model is the last checkpoint of its run, with no checkpoint selection\. We report the benchmark’s official length\-adjusted rubric score and the companion raw score\. For RQ2 out\-of\-domain evaluation, we take the Qwen3\.6\-27B models trained on the HealthBench Professional description and evaluate them, with no further training\. Evaluation uses PRBench Hard\([Akyürek et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib56)\), with 550 expert\-written finance and legal tasks, and ProfBench\([Wang et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib57)\), with 40 report tasks at PhD and MBA level in chemistry, physics, finance, and consulting\. Neither benchmark is medical\. Both grade an answer against an expert\-written weighted rubric, every model answers through the same web\-search tool loop, and the same judge scores them\. We also evaluate the 27B models on the 1,000 tasks of HealthBench Hard\([Arora et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib13)\), which no run uses for training or validation\.

Metrics\.Each benchmark scores an answeryyagainst its rubric, a list of criteriajjwith pointspjp\_\{j\}, positive for required content and negative for errors, and reports the mean over tasks\. WithM⁡\(y\)M\(y\)the criteria the judge marks as met, the rubric score isr⁡\(y\)=∑j∈M⁡\(y\)pj/∑j\|pj\>0pjr\(y\)=\\sum\_\{j\\in M\(y\)\}p\_\{j\}\\big/\\sum\_\{j\\mid p\_\{j\}\>0\}p\_\{j\}\. PRBench Hard and ProfBench reportr⁡\(y\)r\(y\)as their official metric, with PRBench’s importance and detrimental weights and ProfBench’s Critical, Major, Minor, and Additional levels as the pointspjp\_\{j\}\. HealthBench Professional adds its official length adjustment,s⁡\(y\)=r⁡\(y\)−λ⁡\(ℓ⁡\(y\)−ℓ0\)/500s\(y\)=r\(y\)\-\\lambda\\,\(\\ell\(y\)\-\\ell\_\{0\}\)/500, whereℓ⁡\(y\)\\ell\(y\)is the answer length in characters,ℓ0=2,000\\ell\_\{0\}=2\{,\}000, andλ=0\.0147\\lambda=0\.0147\. We reports⁡\(y\)s\(y\)as the adjusted score andr⁡\(y\)r\(y\)clipped to\[0,1\]\[0,1\]as the raw score\. During training the length term applies only beyondℓ0\\ell\_\{0\}, and Appendix[G](https://arxiv.org/html/2610.21040#A7)gives the grading protocol\.

Figure 4:Zero\-shot scores on PRBench Hard and ProfBench\.Qwen3\.6\-27B is trained on the HealthBench Pro description and answers both benchmarks, which it never saw, through the web\-search tool loop\. PRBench Hard has 550 finance and legal tasks, and ProfBench has 40 report tasks, 10 per domain\. Self Improve is our implementation of self\-play training on model\-written tasks\([Huang et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib5);[Liu et al\., 2025a](https://arxiv.org/html/2610.21040#bib.bib6)\), in which a proposer with a fixed prompt writes the tasks and rubrics and the solver is trained on them by reinforcement learning\. RRI adds the adversary that tests and repairs each rubric\.Models and baselines\.We use Qwen3\.5\-9B and Qwen3\.6\-27B as solver models\([Yang et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib45)\)\. The Generation Only baseline is self\-improvement training on model\-written tasks\([Huang et al\., 2026a](https://arxiv.org/html/2610.21040#bib.bib5);[Liu et al\., 2025a](https://arxiv.org/html/2610.21040#bib.bib6)\), in which a proposer with untuned generation instructions writes the tasks and rubrics and the solver is trained on them by reinforcement learning\. We compare it with full RRI and with Co\-evolve Only, which evolves the reward, the tasks, and the solver together but has no adversary\. We also compare with three rubric\-reward methods, RaR\([Gunjal et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib14)\)trained on synthetic tasks, EvoRubrics\([Ding et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib66)\), and SibylSense\([Xu et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib65)\), whose implementations are described in Appendix[F](https://arxiv.org/html/2610.21040#A6)\. Appendix[L\.4](https://arxiv.org/html/2610.21040#A12.SS4)gives further ablations\. The 9B and 27B settings use a GPT judge with retrieval and web search\. An all\-self setting uses the frozen 9B model for every auxiliary role, and a further 9B self\-graded setting removes retrieval\. Due to compute constraints, we perform no hyperparameter tuning, and all baselines use the same hyperparameters\. Details are included in Appendix[E](https://arxiv.org/html/2610.21040#A5)\.

### 4\.2RQ1\. In\-Domain Improvement with Zero Training Data

Reward evolution improves clinical performance\.Figure[2](https://arxiv.org/html/2610.21040#S3.F2)shows that RRI improves adjusted score over Generation Only by 10\.9 points at 9B and 12\.4 at 27B\. Raw scores increase by 11\.0 and 10\.4 points, respectively\. At 27B, Generation Only moves the model only 1\.8 points above its untrained score of 0\.374, while RRI moves it 14\.2 points, so the larger model gains from its own generated tasks mainly once their rewards are repaired\.

Self\-improvement works without a stronger teacher\.The all\-self setting improves adjusted score by 10\.3 points and raw score by 7\.4 points, with the frozen 9B model serving every auxiliary role\. Without retrieval, the adjusted gain is smaller at 4\.0 points, as shown in Appendix Figure[7](https://arxiv.org/html/2610.21040#A8.F7)\.

Writing and difficult tasks benefit most\.Writing gains 39\.1 points at 9B, 14\.7 at 27B, and 25\.9 in the all\-self setting\. In the primary 9B setting, difficult tasks improve by 18\.6 points compared with 2\.8 on typical tasks, as shown in Figure[3](https://arxiv.org/html/2610.21040#S4.F3)\(b\)\. The same ordering appears at 27B and in the all\-self setting\. Consultation and research each improve by 0\.5 points in the primary 9B setting, while research declines in the self\-graded settings\. Table[3](https://arxiv.org/html/2610.21040#A8.T3)gives the complete breakdown\. At 9B, Generation Only leaves writing at 0\.199, level with the untrained 0\.194, while RRI lifts it to 0\.590\. A writing task such as a discharge summary or a referral letter has a fixed form, and a model\-written rubric for it tends to pay for producing the form, which the repair step replaces with criteria on the content\.

Hack\-then\-patch rounds add further gains\.In the 9B GPT\-graded retrieval setting, Co\-evolve Only raises adjusted score from 0\.424 under Generation Only to 0\.506\. Adding the adversary, whose hack\-then\-patch rounds re\-attack each patched rubric, raises it to 0\.533, as shown in Figure[3](https://arxiv.org/html/2610.21040#S4.F3)\(a\)\. This supports continuing to repair rubrics as the solver trains on them\.

RRI outperforms existing rubric\-reward methods\.In the same setting, full RRI at 0\.533 scores above SibylSense at 0\.502, EvoRubrics at 0\.493, and RaR trained on synthetic tasks at 0\.433, as shown in Figure[3](https://arxiv.org/html/2610.21040#S4.F3)\(a\)\.

The gains hold under other judges and methods\.Re\-grading every answer with the benchmark’s official GPT\-5\.4 judge preserves the ordering and the margins in all four settings, as Table[5](https://arxiv.org/html/2610.21040#A8.T5)in Appendix[H](https://arxiv.org/html/2610.21040#A8)shows\. The strongest retrieval\-augmented inference method on the frozen base models stays 27\.9 points below RRI\-9B and 16\.1 points below RRI\-27B\.

### 4\.3RQ2\. Out\-of\-Domain Generalization to Different Tasks

Atrial fibrillation with falls, 9B

QuestionAn 82\-year\-old with nonvalvular atrial fibrillation has fallen several times in the past year and had a gastrointestinal bleed three years ago\. The family was told that falls are a reason to avoid blood\-thinning medication\. How should stroke prevention be weighed against bleeding?

Original rubric\+9\+9Mentions estimating risk with CHA2DS2\-VASc and HAS\-BLED\.\+10\+10States that fall risk alone does not outweigh stroke\-prevention benefit\.Exploit×\\bm\{\\times\}Assigns one age point to the 82\-year\-old instead of the required two\.×\\bm\{\\times\}Proposes a lower apixaban dose without the required patient information\.Added criteria\+8\+8Requires two CHA2DS2\-VASc points for age 75 or older\.−7\-7Penalizes reduced\-dose apixaban without reduction criteria\.

Figure 5:Example of rubric repair on an atrial fibrillation task\.The question provides the context for the original rubric, its exploit, and selected repairs\. Signed weights reward content or penalize errors\. The detailed case appears in Appendix[K\.1](https://arxiv.org/html/2610.21040#A11.SS1)\.RRI transfers to non\-medical professional tasks\.Figure[4](https://arxiv.org/html/2610.21040#S4.F4)shows that RRI\-27B improves over the base model by 6\.6 points on PRBench Hard, from 0\.189 to 0\.255, and by 8\.0 points on ProfBench, from 0\.345 to 0\.425\. The gain appears in both PRBench domains, 7\.4 points in finance and 5\.6 in legal, and in all four ProfBench domains, led by 13\.2 points on chemistry and 11\.0 on consulting\.

Repaired rewards transfer further than Generation Only rewards\.Self Improve, the Generation Only baseline trained on the same description, gains 1\.2 points on PRBench Hard and 4\.7 on ProfBench\. RRI adds 5\.3 and 3\.3 points on top of it and scores highest in seven of the eight columns, while the two trained models are within 0\.3 points on physics\. Most of the out\-of\-domain gain on PRBench Hard therefore comes from improving the reward\. Appendix[H](https://arxiv.org/html/2610.21040#A8)applies the full recipe to the descriptions of other benchmarks\.

RRI also improves on unseen HealthBench Hard tasks\.On HealthBench Hard, RRI\-27B scores 0\.246, against 0\.188 for Generation Only and 0\.143 for the base model, a gain of 5\.8 points over Generation Only, as Table[10](https://arxiv.org/html/2610.21040#A8.T10)in Appendix[H](https://arxiv.org/html/2610.21040#A8)shows\.

### 4\.4RQ3\. How the Hack–Patch–Improve Loop Shapes Learning

We did a comprehensive analysis on the self improvement runs with each of the three datasets as target\. Overall, the runs includes a total of 12,147 samples, with 7,032, 3,473 and 1,642 samples for HealthBench Pro, PRBench and ProfBench respectively\. The full repair analysis appears in Appendix[L](https://arxiv.org/html/2610.21040#A12)\.

Repair removes credit for clinical errors\.Figure[5](https://arxiv.org/html/2610.21040#S4.F5)shows two errors missed by the original rubric\. CHA2DS2\-VASc estimates stroke risk by adding points, with higher scores indicating greater risk\. Age 75 or older contributes two points, so giving the 82\-year\-old only one understates the score\. Apixaban prevents blood clots\. Its standard dose\-reduction rule requires at least two qualifying criteria involving age, weight, and kidney function\. The patient meets the age criterion of 80 or older, but weight and kidney\-function results are unknown\. The lower dose therefore lacks justification\.

The added criteria check correct scoring and dose justification\. Reported policy\-rollout reward falls from 0\.94 to 0\.48 after the original patch\. Further examples appear in Appendix[K](https://arxiv.org/html/2610.21040#A11)\.

Every repair round narrows the gap between hacking and honest answers, even when the adversary still wins\.In the 9B setting, the adopted admission rewrites shrink the mean lead of a fresh exploit over the honest answer from0\.430\.43on the proposer’s rubric to0\.010\.01, and the honest answer moves ahead on a third of these tasks, against almost none before\. Each accepted criterion cuts the exploit’s lead by 0\.23 on average, against a mean lead of 0\.40 when the exploit is caught\. The adversary still beats most repaired rubrics, and after the first patch in training it finds a new exploit in 247 of 293 cases\. A strong adversary keeps finding the next loophole, and each one gives the following round a concrete target\. RRI keeps these improved tasks in the training pool, and training on them raises the adjusted score from 0\.506 under Co\-evolve Only to 0\.533, as shown in Figure[3](https://arxiv.org/html/2610.21040#S4.F3)\(a\)\. Appendix[L\.5](https://arxiv.org/html/2610.21040#A12.SS5)gives the per\-round statistics\.

To see what the loop has to repair, we collect every exploit case that arises during training in the runs trained on the HealthBench Professional, PRBench, and ProfBench descriptions, 12,147 cases in total\. An exploit case is a pair of solver answers to the same generated task in which the answer the rubric scores highest loses the rubric\-blind comparison of Section[3\.2](https://arxiv.org/html/2610.21040#S3.SS2)to an answer scored at leastϵ\\epsilonlower\.

As shown in Figure[6](https://arxiv.org/html/2610.21040#S4.F6), we see the exploitable patterns are consistent across domains, with some variations on the percentages\. Overall, one third of the cases are missed negatives, where the exploit earns its criteria and also states wrong or unsafe content that no criterion touches\. In a market entry report, for example, the exploit treats the whole addressable market as the firm’s own revenue and recommends immediate entry, and it still scores 0\.68 against 0\.32\. Ambiguous rubrics account for 29%\. A criterion worded as “mentions in some way” credits a passing mention or refuses the same content in other words, which leaves room for the solver to game the wording\. In 14% of the cases the answer key is wrong, out of date, or fixes one of several defensible conventions, and the correct answer loses points\. Missed positives stay at or below 1% everywhere, so reward generators gain more from checking what an answer states than from requiring more content\. After patching, we see the training reward of rollouts on the same tasks falls from 0\.71 to 0\.55 at 9B and from 0\.69 to 0\.51 at 27B, showing it spots solvers more space for improvements\. In general, we suggest future research in the self play domain to focus more on how the proposer can cover more negative cases, and how to make the rubrics unambiguous so the system is less prone to be hacked\.

Figure 6:Categories of exploit cases found during training\.Each case is a pair of solver answers to one generated task in which the answer the rubric scores highest loses a rubric\-blind comparison, assigned to one category by an LLM auditor\. Each panel covers the runs trained on the description of one benchmark, HealthBench Pro pools the 9B and 27B runs, and the last panel pools all cases\. Appendix[I](https://arxiv.org/html/2610.21040#A9)defines the categories and gives the procedure\.In the medical runs, ambiguous rubrics lead at 35% and fabricated citations reach 10%, against 2% and none elsewhere, so medical pipelines should emphasize committed, checkable statements and grade evidence with retrieval\. On PRBench, missed negatives lead at 39%, so covering more harmful cases helps most there\. On ProfBench, faulty answer keys lead at 32%, a fault of the proposer, which should verify its own rubrics before sending them to the solver\.

## 5Conclusion

RRI lets a self\-improving loop supervise its own reward\. Before the solver trains on a generated task, an adversary games the task’s rubric, an honest answer from the same model sets the bar, and up to two repair steps revise the rubric to make the honest answer win, while the exploits the loop finds rewrite how future tasks and rubrics are generated\. With no training data, RRI improves HealthBench Professional over the Generation Only self\-improvement baseline at 9B and 27B, and also when the frozen 9B model fills every auxiliary role, with the largest gains on writing and difficult tasks\. Trained only on the HealthBench Professional description, the 27B model also improves on the finance, legal, scientific, and consulting tasks of PRBench Hard and ProfBench\. The exploits the loop finds show that model\-written rubrics fail mostly by missing penalties for wrong content and by loose wording, which points future self\-improving methods toward rewards that check what an answer states\.

### AI use statement

Generative AI tools helped implement the training pipeline and polished writing\. Language models are also components of RRI itself\. The proposer writes every training task and rubric, so all training data are synthetic, and the adversary, the honest answer, the repair step, and the meta\-optimizer all rely on language models, as Section[3](https://arxiv.org/html/2610.21040#S3)describes\. GPT\-5\.5\-Chat and the frozen 9B model serve as judges during training and validation, GPT\-5\.4 is the official judge of HealthBench Professional, a GPT\-5\.5 auditor assigns exploit cases to categories, and a language\-model auditor checks accepted criteria\. The work contains no theorems or proofs, and translation and survey design do not apply to it\. The authors reviewed all AI\-assisted work\. All tables are rendered by script from saved evaluation results\. We take responsibility for the full content of this paper, including any text, claims, or artifacts produced with generative AI\.

### Ethics statement

No human subjects research was carried out for this work\. HealthBench Professional, PRBench, and ProfBench are publicly released benchmarks of expert\-written tasks, and the retrieval database is built from the public corpora listed in Appendix[B](https://arxiv.org/html/2610.21040#A2)\. The exploit analysis documents ways in which answers can game model\-written rubrics\. We report it so that rubric rewards can be built to close these gaps\. The adversary’s answers are used only to test rubrics and to improve how tasks and rubrics are written, and the solver never trains on them\. The trained models are research artifacts\. Their answers to medical, financial, legal, or scientific questions have not been validated for professional use and should not be relied on without expert review\. Appendix[N](https://arxiv.org/html/2610.21040#A14)lists the data sources, and Appendix[M](https://arxiv.org/html/2610.21040#A13)discusses the limits of our evaluation\.

### Reproducibility statement

Section[3](https://arxiv.org/html/2610.21040#S3)specifies every component of RRI, and Section[4\.1](https://arxiv.org/html/2610.21040#S4.SS1)gives the datasets, metrics, models, and training settings\. The appendix covers the remaining details\. Task and rubric generation and the retrieval database are in Appendix[B](https://arxiv.org/html/2610.21040#A2), the adversary and the repair step in Appendix[C](https://arxiv.org/html/2610.21040#A3), meta\-prompt evolution in Appendix[D](https://arxiv.org/html/2610.21040#A4), the configuration of each training arm in Appendix[E](https://arxiv.org/html/2610.21040#A5), and the grading protocol in Appendix[G](https://arxiv.org/html/2610.21040#A7)\. Appendix[J](https://arxiv.org/html/2610.21040#A10)reproduces every prompt verbatim\. The solvers Qwen3\.5\-9B and Qwen3\.6\-27B are open models, and every evaluation benchmark is publicly released\. The code and configurations are available in the public repository listed in Appendix[N](https://arxiv.org/html/2610.21040#A14)\.

## References

- Agrawalet al\.\(2026\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang,et al\.Gepa: reflective prompt evolution can outperform reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 8479–8565\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Akyüreket al\.\(2026\)A\. F\. Akyürek, A\. Gosai, C\. B\. C\. Zhang, V\. Gupta, J\. Jeong, A\. Gunjal, T\. Rabbani, M\. Mazzone, D\. Randolph IV, M\. Meymand,et al\.Prbench: large\-scale expert rubrics for evaluating high\-stakes professional reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 42297–42325\.Cited by:[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p1.1)\.
- Amodeiet al\.\(2016\)D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. ManéConcrete problems in ai safety\.arXiv preprint arXiv:1606\.06565\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p2.1)\.
- Aroraet al\.\(2025\)R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel,et al\.Healthbench: evaluating large language models towards improved human health\.arXiv preprint arXiv:2505\.08775\.Cited by:[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p1.1)\.
- Burnset al\.\(2023\)C\. Burns, P\. Izmailov, J\. H\. Kirchner, B\. Baker, L\. Gao, L\. Aschenbrenner, Y\. Chen, A\. Ecoffet, M\. Joglekar, J\. Leike,et al\.Weak\-to\-strong generalization: eliciting strong capabilities with weak supervision\.arXiv preprint arXiv:2312\.09390\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Changet al\.\(2026\)P\. Chang, Y\. Fang, S\. Chen, Y\. Shi, B\. Shen, and X\. GuTest vs mutant: adversarial llm agents for robust unit test generation\.arXiv preprint arXiv:2602\.08146\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.11.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Chenet al\.\(2024a\)L\. Chen, C\. Zhu, D\. Soselia, J\. Chen, T\. Zhou, T\. Goldstein, H\. Huang, M\. Shoeybi, and B\. CatanzaroOdin: disentangled reward mitigates hacking in rlhf\.arXiv preprint arXiv:2402\.07319\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p2.1)\.
- Chenet al\.\(2024b\)Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. GuSelf\-play fine\-tuning converts weak language models to strong language models\.arXiv preprint arXiv:2401\.01335\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Daiet al\.\(2026\)D\. Dai, P\. Chen, C\. Ekbote, and P\. LiangQoq\-med: building multimodal clinical foundation models with domain\-aware grpo training\.Advances in Neural Information Processing Systems38,pp\. 37406–37453\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Daiet al\.\(2025\)W\. Dai, P\. Chen, M\. Lu, D\. Li, H\. Wei, H\. Cui, and P\. P\. LiangClimb: data foundations for large scale multimodal clinical foundation models\.arXiv preprint arXiv:2503\.07667\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px3.p1.1)\.
- Denisonet al\.\(2024\)C\. Denison, M\. MacDiarmid, F\. Barez, D\. Duvenaud, S\. Kravec, S\. Marks, N\. Schiefer, R\. Soklaski, A\. Tamkin, J\. Kaplan,et al\.Sycophancy to subterfuge: investigating reward\-tampering in large language models\.arXiv preprint arXiv:2406\.10162\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Dinget al\.\(2026\)H\. Ding, B\. Huang, Y\. Fang, W\. Liao, Z\. Li, J\. Zhang, Z\. Wu, J\. Zhao, and Y\. WangEvoRubrics: dynamic rubrics as rewards via adversarial co\-evolution for llm reinforcement learning\.arXiv preprint arXiv:2606\.23038\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.6.1),[Appendix F](https://arxiv.org/html/2610.21040#A6.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p3.1)\.
- Gaoet al\.\(2025\)H\. Gao, J\. Geng, W\. Hua, M\. Hu, X\. Juan, H\. Liu, S\. Liu, J\. Qiu, X\. Qi, Y\. Wu,et al\.A survey of self\-evolving agents: what, when, how, and where to evolve on the path to artificial super intelligence\.arXiv preprint arXiv:2507\.21046\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InInternational conference on machine learning,pp\. 10835–10866\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Guanet al\.\(2025\)X\. Guan, L\. L\. Zhang, Y\. Liu, N\. Shang, Y\. Sun, Y\. Zhu, F\. Yang, and M\. YangRStar\-math: small llms can master math reasoning with self\-evolved deep thinking\.arXiv preprint arXiv:2501\.04519\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Gunjalet al\.\(2026\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 127924–127945\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.2.1),[Appendix F](https://arxiv.org/html/2610.21040#A6.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p3.1)\.
- Hickset al\.\(2026\)R\. S\. Hicks, M\. Trofimov, D\. Lim, R\. K\. Arora, F\. Tsimpourlas, P\. Bowman, M\. Sharman, C\. Tong, K\. Karthik, A\. Dugar,et al\.HealthBench professional: evaluating large language models on real clinician chats\.arXiv preprint arXiv:2604\.27470\.Cited by:[§L\.1](https://arxiv.org/html/2610.21040#A12.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p1.1)\.
- Huanget al\.\(2026a\)C\. Huang, W\. Yu, X\. Wang, H\. Zhang, Z\. Li, R\. Li, J\. Huang, H\. Mi, and D\. YuR\-zero: self\-evolving reasoning llm from zero data\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 130770–130790\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p1.1),[Figure 4](https://arxiv.org/html/2610.21040#S4.F4),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p3.1)\.
- Huanget al\.\(2026b\)X\. Huang, N\. Wang, H\. Liu, X\. Tang, and Y\. ZhouSynthesizing high\-quality visual question answering from medical documents with generator\-verifier lmms\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 109770–109806\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Hupe \(2024\)M\. HupeStatPearls\.Taylor & Francis\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px2.p1.1)\.
- Irvinget al\.\(2018\)G\. Irving, P\. Christiano, and D\. AmodeiAI safety via debate\.arXiv preprint arXiv:1805\.00899\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Jayalathet al\.\(2025\)D\. Jayalath, S\. Goel, T\. Foster, P\. Jain, S\. Gururangan, C\. Zhang, A\. Goyal, and A\. ScheltenCompute as teacher: turning inference compute into reference\-free supervision\.arXiv preprint arXiv:2509\.14234\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Jinet al\.\(2025\)B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. HanSearch\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Jinet al\.\(2019\)Q\. Jin, B\. Dhingra, Z\. Liu, W\. Cohen, and X\. LuPubmedqa: a dataset for biomedical research question answering\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 2567–2577\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px3.p1.1)\.
- Johnsonet al\.\(2023\)A\. E\. Johnson, L\. Bulgarelli, L\. Shen, A\. Gayles, A\. Shammout, S\. Horng, T\. J\. Pollard, S\. Hao, B\. Moody, B\. Gow,et al\.MIMIC\-iv, a freely accessible electronic health record dataset\.Scientific data10\(1\),pp\. 1\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Khattabet al\.\(2023\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam,et al\.Dspy: compiling declarative language model calls into self\-improving pipelines\.arXiv preprint arXiv:2310\.03714\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Lambertet al\.\(2024\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Liet al\.\(2026a\)Q\. Li, X\. Dai, W\. Liu, X\. Li, Y\. Wang, R\. Tang, Y\. Yu, and W\. ZhangAtgen: adversarial reinforcement learning for test case generation\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 147647–147665\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.10.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Liet al\.\(2026b\)X\. Li, J\. Jin, G\. Dong, H\. Qian, Y\. Wu, J\. Wen, Y\. Zhu, and Z\. DouWebthinker: empowering large reasoning models with deep research capability\.Advances in Neural Information Processing Systems38,pp\. 120091–120131\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Liuet al\.\(2025a\)B\. Liu, C\. Jin, S\. Kim, W\. Yuan, W\. Zhao, I\. Kulikov, X\. Li, S\. Sukhbaatar, J\. Lanchantin, and J\. WestonSpice: self\-play in corpus environments improves reasoning\.arXiv preprint arXiv:2510\.24684\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.4.1),[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[Figure 4](https://arxiv.org/html/2610.21040#S4.F4),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p3.1)\.
- Liuet al\.\(2026a\)B\. Liu, S\. Yu, Z\. Liu, L\. Guertler, P\. Qi, D\. Balcells, M\. Liu, C\. Tan, W\. Shi, M\. Lin,et al\.Spiral: self\-play on zero\-sum games incentivizes reasoning via multi\-agent multi\-turn reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 35407–35434\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Liuet al\.\(2026b\)T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. WangOpenrubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17417–17437\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.3.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Liuet al\.\(2025b\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Milleret al\.\(2000\)N\. Miller, E\. Lacroix, and J\. E\. BackusMEDLINEplus: building and maintaining the national library of medicine’s consumer health web service\.Bulletin of the Medical Library Association88\(1\),pp\. 11\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px2.p1.1)\.
- Novikovet al\.\(2025\)A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. Ruiz, A\. Mehrabian,et al\.Alphaevolve: a coding agent for scientific and algorithmic discovery\.arXiv preprint arXiv:2506\.13131\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1)\.
- of Healthet al\.\(2014\)N\. I\. of Healthet al\.DailyMed\.National Institutes of Health\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px2.p1.1)\.
- Organizationet al\.\(2004\)W\. H\. Organizationet al\.ICD\-10: international statistical classification of diseases and related health problems: tenth revision\.World Health Organization\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px2.p1.1)\.
- Parket al\.\(2024\)R\. Park, R\. Rafailov, S\. Ermon, and C\. FinnDisentangling length from quality in direct preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 4998–5017\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p2.1)\.
- Quet al\.\(2026\)A\. Qu, H\. Zheng, Z\. Zhou, Y\. Yan, Y\. Tang, S\. Y\. Ong, F\. Hong, K\. Zhou, C\. Jiang, M\. Kong,et al\.Coral: towards autonomous multi\-agent evolution for open\-ended discovery\.arXiv preprint arXiv:2604\.01658\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1)\.
- Rackauckas \(2024\)Z\. RackauckasRag\-fusion: a new take on retrieval\-augmented generation\.arXiv preprint arXiv:2402\.03367\.Cited by:[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px4.p1.1)\.
- Ruanet al\.\(2026\)C\. Ruan, D\. Jiang, H\. Zeng, P\. Nie, and W\. ChenEvolveCoder: evolving test cases via adversarial verification for code reinforcement learning\.arXiv preprint arXiv:2603\.12698\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.9.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Samvelyanet al\.\(2024\)M\. Samvelyan, S\. C\. Raparthy, A\. Lupu, E\. Hambro, A\. H\. Markosyan, M\. Bhatt, Y\. Mao, M\. Jiang, J\. Parker\-Holder, J\. Foerster,et al\.Rainbow teaming: open\-ended generation of diverse adversarial prompts\.Advances in Neural Information Processing Systems37,pp\. 69747–69786\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Shaoet al\.\(2025\)R\. Shao, A\. Asai, S\. Z\. Shen, H\. Ivison, V\. Kishore, J\. Zhuo, X\. Zhao, M\. Park, S\. G\. Finlayson, D\. Sontag,et al\.Dr tulu: reinforcement learning with evolving rubrics for deep research\.arXiv preprint arXiv:2511\.19399\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.5.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1),[§3\.3](https://arxiv.org/html/2610.21040#S3.SS3.p1.1)\.
- Shenget al\.\(2025\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Singhalet al\.\(2025\)K\. Singhal, T\. Tu, J\. Gottweis, R\. Sayres, E\. Wulczyn, M\. Amin, L\. Hou, K\. Clark, S\. R\. Pfohl, H\. Cole\-Lewis,et al\.Toward expert\-level medical question answering with large language models\.Nature medicine31\(3\),pp\. 943–950\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward gaming\.Advances in neural information processing systems35,pp\. 9460–9471\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p2.1)\.
- Viswanathanet al\.\(2026\)V\. Viswanathan, Y\. Sun, X\. Kong, M\. Cao, G\. Neubig, and S\. WuChecklists are better than reward models for aligning language models\.Advances in Neural Information Processing Systems38,pp\. 114728–114754\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Wanget al\.\(2026\)Z\. Wang, J\. Jung, X\. Lu, S\. Diao, E\. Evans, J\. Zeng, P\. Molchanov, Y\. Choi, J\. Kautz, and Y\. DongProfbench: multi\-domain rubrics requiring professional knowledge to answer and judge\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 5355–5384\.Cited by:[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p1.1)\.
- Wilfet al\.\(2025\)A\. Wilf, P\. Aggarwal, B\. Parno, D\. Fried, L\. Morency, P\. P\. Liang, and S\. WelleckPropose, solve, verify: self\-play through formal verification\.arXiv preprint arXiv:2512\.18160\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p2.1)\.
- Wuet al\.\(2026\)M\. Wu, G\. Zhang, S\. Min, S\. Levine, and A\. KumarRlac: reinforcement learning with adversarial critic for free\-form generation tasks\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 139470–139492\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.8.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Xionget al\.\(2024\)G\. Xiong, Q\. Jin, X\. Wang, M\. Zhang, Z\. Lu, and A\. ZhangImproving retrieval\-augmented generation in medicine with iterative follow\-up questions\.InBiocomputing 2025: Proceedings of the Pacific Symposium,pp\. 199–214\.Cited by:[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Xuet al\.\(2023\)C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, and D\. JiangWizardlm: empowering large language models to follow complex instructions\.arXiv e\-prints,pp\. arXiv–2304\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Xuet al\.\(2026\)Y\. Xu, G\. Potje, S\. Shandilya, T\. Yuan, L\. d\. O\. Nunes, R\. Agarwal, S\. Asgari, A\. Atkinson, E\. Kıcıman, S\. Lu,et al\.Sibylsense: adaptive rubric learning via memory tuning and adversarial probing\.arXiv preprint arXiv:2602\.20751\.Cited by:[Table 1](https://arxiv.org/html/2610.21040#A1.T1.4.1.7.1),[Appendix F](https://arxiv.org/html/2610.21040#A6.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p3.1)\.
- Xuet al\.\(2025\)Z\. Xu, F\. Jiang, L\. Niu, Y\. Deng, R\. Poovendran, Y\. Choi, and B\. Y\. LinMagpie: alignment data synthesis from scratch by prompting aligned llms with nothing\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 76346–76382\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§L\.1](https://arxiv.org/html/2610.21040#A12.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.21040#S4.SS1.p3.1)\.
- Yuanet al\.\(2024\)W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. WestonSelf\-rewarding language models\.arXiv preprint arXiv:2401\.10020\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Zelikmanet al\.\(2022\)E\. Zelikman, Y\. Wu, J\. Mu, and N\. GoodmanStar: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Zhanget al\.\(2026\)J\. Zhang, S\. Hu, C\. Lu, R\. Lange, and J\. CluneDarwin gödel machine: open\-ended evolution of self\-improving agents\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 104223–104294\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1)\.
- Zhanget al\.\(2023\)X\. Zhang, C\. Wu, Z\. Zhao, W\. Lin, Y\. Zhang, Y\. Wang, and W\. XiePmc\-vqa: visual instruction tuning for medical visual question answering\.arXiv preprint arXiv:2305\.10415\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px3.p1.1)\.
- Zhaoet al\.\(2026\)A\. Zhao, Y\. Wu, T\. Wu, Q\. Xu, Y\. Yue, M\. Lin, S\. Wang, Q\. Wu, Z\. Zheng, and G\. HuangAbsolute zero: reinforced self\-play reasoning with zero data\.Advances in Neural Information Processing Systems38,pp\. 105816–105879\.Cited by:[§1](https://arxiv.org/html/2610.21040#S1.p1.1),[§1](https://arxiv.org/html/2610.21040#S1.p2.1),[§2](https://arxiv.org/html/2610.21040#S2.p1.1)\.
- Zhaoet al\.\(2025a\)R\. Zhao, A\. Meterez, S\. Kakade, C\. Pehlevan, S\. Jelassi, and E\. MalachEcho chamber: rl post\-training amplifies behaviors learned in pretraining\.arXiv preprint arXiv:2504\.07912\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Zhaoet al\.\(2025b\)X\. Zhao, S\. Liu, S\. Yang, and C\. MiaoMedrag: enhancing retrieval\-augmented generation with knowledge graph\-elicited reasoning for healthcare copilot\.InProceedings of the ACM on Web Conference 2025,pp\. 4442–4457\.Cited by:[Appendix B](https://arxiv.org/html/2610.21040#A2.SS0.SSS0.Px3.p1.1),[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.21040#S2.p3.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2610.21040#S2.p2.1)\.
- Zuoet al\.\(2025\)Y\. Zuo, S\. Qu, Y\. Li, Z\. Chen, X\. Zhu, E\. Hua, K\. Zhang, N\. Ding, and B\. ZhouMedxpertqa: benchmarking expert\-level medical reasoning and understanding\.arXiv preprint arXiv:2501\.18362\.Cited by:[Appendix H](https://arxiv.org/html/2610.21040#A8.SS0.SSS0.Px2.p1.1)\.

## Appendix AComparison with related methods

Table[1](https://arxiv.org/html/2610.21040#A1.T1)compares RRI with the related methods discussed in Section[2](https://arxiv.org/html/2610.21040#S2)\.

Table 1:Features of related methods\.The columns compare properties relevant to self\-improvement on open\-ended tasks\. Open\-ended means answers are graded by judged criteria with no executable or reference check\. Task gen\. means the method writes its own training tasks\. No references means no reference answers, preference labels, or golden rubrics\. Exploit of the reward means an agent produces answers meant to satisfy the reward without being good\. Reward verif\. means each reward is checked before the solver trains on it\. Rubric repair means criteria are added to a task’s existing rubric after a detected failure\. Re\-attack after patch means the repaired task and rubric are attacked again\. Task evolution means task generation changes during training\. Under Reward, task is a per\-task rubric that never changes and evolving is one that changes during training\. Under Verifier, judge is a language model grading criteria, automatic is answer matching, execution runs tests against reference code, and external is a fact checker over Wikipedia or execution\.
## Appendix BTask and rubric generation details

#### Service\.

The proposer, generator, adversary, repair step, memo optimizer, and meta\-optimizer run in one generation service that the trainer polls for tasks and reports graded rollouts to\. Tasks are served from a pool; the trainer’s dataset caches fetched rows, so a repaired rubric reaches future rollouts through a replay buffer that the pool drains first\.

#### Seed questions\.

The clinical database provides seeds for generating new questions\. Seed sources include StatPearls\([Hupe, 2024](https://arxiv.org/html/2610.21040#bib.bib46)\), DailyMed\([of Health and others, 2014](https://arxiv.org/html/2610.21040#bib.bib47)\), MedlinePlus\([Miller et al\., 2000](https://arxiv.org/html/2610.21040#bib.bib48)\), ICD\-10\-CM\([Organization and others, 2004](https://arxiv.org/html/2610.21040#bib.bib49)\)\.

#### Medical database\.

The medical database includes these seed sources and MedRAG\([Zhao et al\., 2025b](https://arxiv.org/html/2610.21040#bib.bib51)\), PubMedQA\([Jin et al\., 2019](https://arxiv.org/html/2610.21040#bib.bib52)\), PMC\-VQA\([Zhang et al\., 2023](https://arxiv.org/html/2610.21040#bib.bib53)\), and CLIMB\([Dai et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib50)\)for additional knowledge grounding\. All of these datasets are public and used with the required permissions\.

#### Shape controls\.

The positive\-criterion count is two with probability15%15\\%, three with probability70%70\\%, and four with probability15%15\\%\. It is stated as a hard number; a stated range drifted upward under training\. A negative criterion is added on top with target share0\.750\.75, because a negative widens the achievable range below zero and gives group\-relative optimization more ranking room; two proportional controllers on400400\-wide rolling windows solve the requested rate from the delivered rate for negatives and for criterion counts\. Red\-team tasks receive one or two positives so that a single trap can zero the task\. The validator requires one to six items, points in\[−10,10\]∖\{0\}\[\-10,10\]\\setminus\\\{0\\\}, and at least one positive criterion, and rejects negatives phrased as absences as well as criteria that address the judge; patched rubrics may grow to twelve items\.

#### Difficulty band and eviction\.

The meta\-optimizer passes the solver’s rolling mean score to the generator together with a target band of0\.40\.4to0\.60\.6\. After four graded rollouts, a task is evicted if its score variance is below10−410^\{\-4\}or its mean is outside\[0\.05,0\.88\]\[0\.05,0\.88\]; a patch returns the task to the pool and resets the score window\.

## Appendix CAdversary and repair details

#### Admission probe\.

The exploit and honest answers are graded with three votes per criterion under the strict judge note given in Appendix[J](https://arxiv.org/html/2610.21040#A10), length\-adjusted and unclipped so that resolution below zero is preserved\. A rubric passes the probe whenΔ≤−δ\\Delta\\leq\-\\deltawithδ=0\.20\\delta=0\.20and0\.15≤shonest≤0\.920\.15\\leq s\_\{\\text\{honest\}\}\\leq 0\.92; otherwise the failure reason is*exploitable*,*saturated*, or*unsatisfiable*, and the task goes to repair\. A probe in which the adversary scores nothing is discarded rather than counted as unexploitable, and a probe that returns no usable measurement, because generation or grading fails or an answer is too short, admits the task unchanged\. Up to two probe\-repair rounds run at admission\. Each round rewrites the task and rubric together, in every arm, and the rewrite is accepted only if a fresh probe showsΔ\\Deltalower by at least0\.150\.15with the honest score still in band; proposals are salvaged rather than rejected, with points clamped to±\[5,10\]\\pm\[5,10\]and growth trimmed to four added items\. If no rewrite clears the bar, the rewrite with the largest measured drop inΔ\\Deltais adopted, provided the drop is positive and the honest score is in band, and otherwise the task keeps its rubric\. An adopted rewrite therefore always leaves a fresh exploit a smaller lead over the honest answer than the rubric it replaces\. Every task enters the training pool after its repair rounds, including tasks whose rubric the adversary still beats, because the aim of repair is a better rubric and keeping these tasks preserves the diversity of the pool\. Admission runs in the background with a cap of4848concurrent repairs\.

#### Patch validation\.

A proposed criterion must be4040to220220characters with\|p\|∈\[5,10\]\|p\|\\in\[5,10\], fit under the patched\-rubric cap, pass the rubric validator, and have trigram overlap below0\.60\.6with every existing item\. It is graded against both answers with three votes; an undetermined verdict rejects\. Negatives are additionally graded against the cached honest answer and rejected if met\. The normalized gap removed,\|p\|/∑pj\>0pj\|p\|/\\sum\_\{p\_\{j\}\>0\}p\_\{j\}, must be at least0\.050\.05\. Up to five criteria are minted per round, two rounds per case, and at most six patches per task over a run, and every acceptance writes the answer pair with its verdicts to an evidence log\.

## Appendix DMeta\-prompt evolution details

#### Round\.

Evolution runs at every scheduled evaluation interval on twenty graded training cases sampled with failure weighting\. Half are the worst cases, a quarter are the best, and the remainder are sampled at random\. The per\-case analyst uses closed vocabularies for failure modes, rubric defects, and task defects, plus a free\-text gap and a three\-way verdict; counts are aggregated in code\. The corpus statistics are criteria per task, criterion length, negative share, use\-case mix, distinct specialties, and task self\-similarity by shingles, ensuring they are as diverse as possible\. The single optimizer call returns new guidance for the query proposer and the task\-and\-rubric generator, together with a summary; the guidance is substituted into a placeholder of a structural template that is re\-synchronized from code at every start\. Every round’s full assembled prompt, statistics, diagnoses, exploit evidence, and ledger are snapshotted\.

#### Memo optimizer\.

The memo is limited to2,4002\{,\}400characters\. Because exploit scores are already near their ceiling, an objective based only on the adversary’s margin can favor easier rubrics that raise honest scores, so each memo version is scored as

g=Δ¯adv\+κ​max⁡\[0,s¯honest−s¯honest\[0\]−τ\],g\\;=\\;\\overline\{\\Delta\}\_\{\\text\{adv\}\}\\;\+\\;\\kappa\\,\\max\\\!\\big\[0,\\ \\bar\{s\}\_\{\\text\{honest\}\}\-\\bar\{s\}\_\{\\text\{honest\}\}^\{\[0\]\}\-\\tau\\big\],\(2\)where the honest\-score baseline is fixed at the first measured version so that successive small increases in leniency cannot accumulate unchecked\. The objective usesκ=2\.0\\kappa=2\.0, toleranceτ\\tauequal to the regression tolerance0\.020\.02, and target margin00, and requires two observations before a commit decision\. A version that scores worse than the best previous version by more than the tolerance is rolled back, and a version whose honest mean falls below0\.250\.25is flagged as over\-restricting and not tightened further\. Fixed prompt rules take precedence over the memo, and code continues to enforce criterion counts, points, and the negative share\.

#### Fixed components\.

The judge template, the strict judge note, the structural generation templates, the validator, and the shape controllers are code constants\.

## Appendix ETraining configuration

#### Optimization\.

Group\-relative policy optimization withn=8n=8rollouts per task at temperature1\.01\.0, train batch6464tasks, mini\-batch1616, learning rate10−610^\{\-6\}, clip ratio0\.20\.2, token\-mean loss aggregation, no KL loss and no KL in the reward, no entropy bonus, prompt length up to6,1446\{,\}144tokens and response length up to8,1928\{,\}192\. Validation on all525525tasks at temperature1\.01\.0every five updates\. Every reported score comes from the last checkpoint of its run, with no checkpoint selection\. Policy served by vLLM with asynchronous rollouts and trained with FSDP on four B200 GPUs; arms on the shared cluster use a four\-GPU allocation and a 24\-hour chained job\.

#### Solver tools\.

Up to two tool calls per rollout, for citation purposes, to a clinical knowledge\-base search and a web search\. The clinical search uses Milvus over StatPearls, DailyMed, MedlinePlus, ICD\-10\-CM, and PubMed abstracts, with a frozen summarizer producing one evidence brief\. The web search returns verbatim result blocks with no model in the path\. Tool spans are masked; query tokens are trained\.

#### Arms\.

Table[2](https://arxiv.org/html/2610.21040#A5.T2)lists the configuration of every arm in the paper\.

Table 2:Configuration of each training arm\.Judge = training / validation\. Evolve = meta\-prompt and memo evolution; Probe = adversary admission probe with repair\.SettingRowPolicyJudgeRetrievalEvolveAdversary9BGeneration OnlyQwen3\.5\-9BGPT / GPTyes––9BRRI\-9BQwen3\.5\-9BGPT / GPTyesyesyes27BGeneration OnlyQwen3\.6\-27BGPT / GPTyes––27BRRIQwen3\.6\-27BGPT / GPTyesyesyesAll\-selfGeneration OnlyQwen3\.5\-9B9B / 9Byes––All\-selfRRI\-9B, all roles 9BQwen3\.5\-9B9B / 9ByesyesyesNo\-retr\.Generation OnlyQwen3\.5\-9B9B / 9B–––No\-retr\.RRI\-9BQwen3\.5\-9B9B / 9B–yesyesAblationEvolution onlyQwen3\.5\-9BGPT / GPT–yes–AblationCo\-evolve OnlyQwen3\.5\-9BGPT / GPTyesyes–The Generation Only rows use the untuned proposer and generator prompts of Appendix[J](https://arxiv.org/html/2610.21040#A10)\.

## Appendix FBaseline implementations

Figure[3](https://arxiv.org/html/2610.21040#S4.F3)\(a\) compares RRI with three published rubric\-reward methods in the 9B GPT Teacher setting\. Each baseline trains Qwen3\.5\-9B, and we report its adjusted score on HealthBench Professional at the last checkpoint of its run\.

#### RaR with synthetic tasks\.

Rubrics as Rewards\([Gunjal et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib14)\)writes a checklist of weighted criteria for each prompt from its reference answer and uses the checklist as the reward for on\-policy reinforcement learning\. The method requires labeled tasks, and our setting provides none\. We therefore first generate a static list of tasks, then use RaR to decompose each task into smaller tasks, and train the solver on the result\.

#### EvoRubrics\.

EvoRubrics\([Ding et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib66)\)co\-evolves a rubric generator with the policy\. The two share one backbone through separate LoRA adapters and are both updated by GRPO at every training step\. For each prompt the policy samples several answers, the generator samples several rubric sets, and the judge scores every answer against every rubric set\. The policy is rewarded by its mean normalized score across rubric sets\. The generator is rewarded for rubrics that separate the policy’s answers and for diverse and useful criteria\. We follow the official implementation and its documentation\.

#### SibylSense\.

SibylSense\([Xu et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib65)\)keeps the rubric generator frozen and adapts it through a memory bank of validated rubric items\. A proposed item is rewarded by how much higher it scores a reference answer than the candidate answers, and the top items in each category condition the next round of rubric generation\. Between rounds the policy is trained by GRPO on the current rubrics, and its samples replace the candidate answers, so each round must find criteria that the trained policy fails\. We follow the official implementation and its documentation\.

## Appendix GGrading protocol

#### Template\.

Training and validation use the benchmark’s published per\-criterion judge template verbatim\. The judge sees the conversation, the final response, and one criterion, and returns an explanation and a boolean\. Each criterion is graded with three votes and the majority taken\. During adversarial probes and patch validation a strict note is appended, as reproduced in Appendix[J](https://arxiv.org/html/2610.21040#A10), instructing the judge to credit only content the response itself states, so that addressing a topic, restating the request, hedging, or deferring to guidelines is not met\.

#### Score\.

The rubric scorer⁡\(y\)r\(y\)of Section[4\.1](https://arxiv.org/html/2610.21040#S4.SS1)is negative when met negative criteria outweigh met positive ones\. The length term ofs⁡\(y\)s\(y\)in Section[4\.1](https://arxiv.org/html/2610.21040#S4.SS1)is linear in the answer length and centered at2,0002\{,\}000characters, so it subtracts0\.01470\.0147per500500characters above the center and adds the same amount below it; the length is counted after the reasoning channel and tool spans are removed\. Validation reports the signed length\-adjusted score averaged over tasks as accuracy and the raw fraction clipped to\[0,1\]\[0,1\]per task as the unadjusted metric\. Training applies the length term one\-sided, penalizing only beyond the center\.

#### Judges\.

GPT\-5\.5\-Chat performs training and evaluation grading in the GPT\-graded settings\. In the self\-graded settings, the locally served frozen base model performs both roles\. Within a setting every row, including the untrained row, is graded by the setting’s judge\. Validation scores are taken from the judge of the reported run, and Table[5](https://arxiv.org/html/2610.21040#A8.T5)re\-grades the same answers with the official GPT\-5\.4 judge\.

## Appendix HAdditional results

Table 3:Generation Only and RRI scores by clinical task and difficulty\.Scores are the adjusted validation scores\. Each pair compares Generation Only with full RRI under the same backbone, judge, and tool setting\. Bold indicates the higher score within a pair\. Untrained models appear in Appendix[L](https://arxiv.org/html/2610.21040#A12)\.Figure 7:\(a\) HealthBench Professional adjusted scores without retrieval\.Bars show the untrained, Generation Only, and RRI models with the 9B backbone and the frozen 9B judge\.\(b\) HealthBench Professional raw scores without retrieval\.Table 4:HealthBench Professional raw scores\.Scores are the fraction of rubric points earned with the length term omitted, at the evaluations of Table[14](https://arxiv.org/html/2610.21040#A12.T14)\.Table[4](https://arxiv.org/html/2610.21040#A8.T4)reports the unadjusted rubric fraction at the evaluations of Table[14](https://arxiv.org/html/2610.21040#A12.T14)\. In the three retrieval settings, RRI leads Generation Only on the unadjusted score by11\.011\.0,10\.410\.4, and7\.47\.4points overall, and in the no\-retrieval setting by3\.23\.2points\.

Table 5:HealthBench Professional scores under the official GPT\-5\.4 judge\.The rows of Table[14](https://arxiv.org/html/2610.21040#A12.T14)are re\-graded from the same answers with the benchmark’s official judge, GPT\-5\.4 at reasoning effort low with one vote per criterion\. Scores are the official length\-adjusted signed score\.#### Official judge\.

Every trained row in Table[14](https://arxiv.org/html/2610.21040#A12.T14)is scored by the judge its setting trained against, and the two self\-judged settings use a 9B judge\. Table[5](https://arxiv.org/html/2610.21040#A8.T5)re\-grades the same answers, from the same evaluation of each row, with the benchmark’s official judge, GPT\-5\.4 at reasoning effort low, one call per criterion\. RRI leads Generation Only by11\.611\.6points in the 9B setting,11\.311\.3at 27B,12\.312\.3in the all\-self setting, and6\.26\.2in the no\-retrieval setting, and the writing and difficult\-task gains reappear in every setting\.

Table 6:HealthBench Professional scores by specialty\.Scores use the official length\-adjusted score for the Generation Only and RRI rows of each setting in Table[14](https://arxiv.org/html/2610.21040#A12.T14)\.nnis the number of validation tasks in the specialty\.Table[6](https://arxiv.org/html/2610.21040#A8.T6)breaks the Generation Only and RRI rows of each setting down by the benchmark’s2828specialties\.

Table 7:Scores on held\-out benchmarks at 9B\.Each run uses Qwen3\.5\-9B and the GPT\-5\.5\-Chat judge on a rubric benchmark outside HealthBench Professional\. Each benchmark is used only for validation; the proposer sees a paper\-level description of the benchmark and no items\. Untrained is the base model under the same judge; RRI is the last checkpoint of the run\. MedXpertQA is scored as a one\-criterion rubric, so its score is exact\-match accuracy, shown for the text and image splits\.
#### Held\-out benchmarks outside HealthBench Professional\.

Table[7](https://arxiv.org/html/2610.21040#A8.T7)applies the full RRI recipe of the 9B setting to three benchmarks the loop never sees, with the same protocol as the main experiments, where the benchmark is the validation set only and the proposer receives a paper\-level description of the domain\. PRBench Hard\([Akyürek et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib56)\)consists of550550expert\-written finance and legal tasks with rubrics of up to1717criteria, and ProfBench\([Wang et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib57)\)of4040PhD\- and MBA\-level report tasks; both report the weighted fraction of rubric points met, with no length term\. MedXpertQA\([Zuo et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib58)\)is multiple choice, encoded as a one\-criterion rubric so that the same judge yields exact\-match accuracy\. RRI improves all three from the base model, by4\.84\.8points on PRBench Hard,4\.94\.9on ProfBench, and7\.07\.0on MedXpertQA, where the gain is5\.95\.9on the text split and8\.38\.3on the image split\. The solver in the PRBench and ProfBench arms has web search only, since no domain knowledge base exists for them\.

Table 8:Reference scores of frontier and open models\.All models are scored on HealthBench Professional under one protocol, the benchmark’s official pipeline, no tools, greedy decoding, an 8,192\-token answer budget, and the GPT\-5\.5\-Chat judge used for validation in this paper\. Frontier models are served through the same endpoint as the judge; open models are served locally\.
#### Reference points\.

Table[8](https://arxiv.org/html/2610.21040#A8.T8)scores frontier and open models on the benchmark under one protocol that any model can run, the official pipeline with no tools and greedy decoding, judged by GPT\-5\.5\-Chat as in the validation of this paper\.

Table 9:Inference\-time methods and trained models on HealthBench Professional\.Inference\-time methods on the frozen base models appear next to the trained rows of the same base model from Table[14](https://arxiv.org/html/2610.21040#A12.T14), judged by GPT\-5\.5\-Chat with three votes\. The frozen\-model methods follow the inference\-time protocol, greedy decoding, no reasoning channel, references from the shared medical corpus placed in the prompt; medical RAG retrieves once, RAG\-Fusion fuses three generated queries, and i\-MedRAG asks three rounds of two follow\-up questions\. The trained rows answer with the solver’s own tool calls at the training temperature\.
#### Inference\-time methods on the frozen base models\.

Table[9](https://arxiv.org/html/2610.21040#A8.T9)runs the retrieval\-augmented inference methods on the frozen base models, with the same shared corpus and retrieval settings, next to the trained rows of the same base model\. Single\-pass medical RAG\([Zhao et al\., 2025b](https://arxiv.org/html/2610.21040#bib.bib51)\), RAG\-Fusion\([Rackauckas, 2024](https://arxiv.org/html/2610.21040#bib.bib54)\), and i\-MedRAG\([Xiong et al\., 2024](https://arxiv.org/html/2610.21040#bib.bib55)\)decode greedily with the reasoning channel off and receive the retrieved passages in the prompt; the trained rows answer with the solver’s own tool calls\. Iterative retrieval is the strongest inference\-time method on both models, at0\.2540\.254for the 9B and0\.3550\.355for the 27B\. Every inference\-time method stays below the Generation Only training run of the same model, and RRI leads the strongest of them by27\.927\.9points at 9B and16\.116\.1points at 27B\.

Table 10:Scores on HealthBench Hard at 27B\.HealthBench Hard is the 1,000 hardest tasks of the original HealthBench release and is never used for training or validation\. Scored with solver tools on and the GPT\-5\.5\-Chat judge with three votes, using the benchmark’s official score, the mean per\-example clipped rubric fraction with no length term\.
#### Transfer to HealthBench Hard\.

Table[10](https://arxiv.org/html/2610.21040#A8.T10)evaluates the trained policies on the1,0001\{,\}000tasks of HealthBench Hard\([Arora et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib13)\), which no run saw during training or validation, with the same judge as the main experiments and the benchmark’s own score, the per\-example clipped rubric fraction without a length term\. Generation Only lifts the base model from0\.1430\.143to0\.1880\.188overall, and RRI to0\.2460\.246\.

## Appendix IExploit categories

This appendix describes how the exploit cases of Figure[6](https://arxiv.org/html/2610.21040#S4.F6)were collected, how the categories were derived, and how every case was assigned to one of them, and it gives the full breakdown with examples\.

#### Exploit cases\.

At every training step the solver writes several answers to each generated task\. The judge scores each answer under the generated rubric, and the rubric\-blind comparison of Section[3\.2](https://arxiv.org/html/2610.21040#S3.SS2)ranks the same answers into tiers\. An exploit case is a task whose top\-scoring answer falls outside the best tier while an answer inside the best tier scores at least the marginϵ=0\.15\\epsilon=0\.15lower\. We pair the top\-scoring answer, the exploit, with the lowest\-scoring answer of the best tier, the honest answer\. The repair step receives the cases with the largest disagreement in each round, and the analysis here covers all of them at every step\. This gives2,0762\{,\}076cases for the 9B run and4,9864\{,\}986for the 27B run trained on the HealthBench Professional description,3,4823\{,\}482for the run trained on the PRBench description, and1,6471\{,\}647for the run trained on the ProfBench description,12,19112\{,\}191in total\. For each case we keep the task, the rubric at that step, both answers, the verdict the judge recorded on every criterion for each answer, and the criteria added by the repair when the case was repaired\.

#### Deriving the categories\.

We first read 480 cases in full, 120 per run spread evenly over training, with LLM coders working in 20 independent batches\. For each case a coder described what the exploit answer did, which property of the rubric or of the grading paid for it, what the repair added, and whether the honest answer was really the better one, and wrote its own label\. The batches produced about 250 labels\. We consolidated them by the property of the reward that failed, which gives the five categories of reward failure and the one category of incorrect fixes, the incorrect pairs, in Table[12](https://arxiv.org/html/2610.21040#A9.T12), and a second pass of coders assigned the same 480 cases to the consolidated categories to test their boundaries\. Table[11](https://arxiv.org/html/2610.21040#A9.T11)gives the definition of each category\.

Table 11:Definitions of the exploit categories\.Each exploit case is assigned the first category that applies, in the order Incorrect pair, Faulty answer key, Fabricated citations, Missed negatives, Ambiguous rubrics, Missed positives\.
#### Assigning every case\.

A GPT\-5\.5 auditor with medium reasoning effort assigns each of the12,19112\{,\}191cases to one category\. It sees everything kept for the case\. It first checks whether the honest answer is really better, which recorded verdicts carry the score gap and whether they hold on a careful reading of both answers, whether any criterion is wrong or arbitrary, and whether the credited content is invented or comes with errors outside the criteria\. It then takes the first category that applies in the order incorrect pair, faulty answer key, fabricated citations, missed negatives, ambiguous rubrics, missed positives\. The auditor also labels exploits that collect criteria left over from an earlier repair of the same rubric\. These are 44 cases, and they are left out of the shares, which leaves12,14712\{,\}147cases\.

Table 12:Exploit categories by benchmark description\.Each entry is the share of the exploit cases recorded in the runs trained on one benchmark description\. Each case was assigned one category by an LLM auditor that re\-graded both answers against the rubric\. HealthBench Pro pools the 9B and 27B runs\.
#### Missed negatives\.

The exploit earns its criteria and also states wrong or unsafe content that no criterion touches\. A rubric of positive criteria cannot see what should not be in an answer, and the negative criteria the proposer writes name only the errors it anticipated\. In a diagnosis coding task the exploit lists an external cause code as the principal diagnosis, which the coding rules do not permit, and still meets all four criteria, since the rubric asks only that the code appear\. In a market entry report the exploit treats the whole addressable market as the firm’s own revenue and recommends immediate entry, and it scores0\.680\.68against0\.320\.32because the three criteria price only the sizing arithmetic\. This is the largest category overall and the one the negative criteria of the repair step address\. A reward generator needs a correctness check over the whole answer in addition to its list of criteria\.

#### Ambiguous rubrics\.

A criterion addresses the right content, and its wording leaves open what counts\. Criteria worded as “mentions in some way” credit a passing mention, a hedge, or a claim that evidence exists, penalties are avoided by attributing a recommendation to other clinicians, and the same content in other words is refused\. In one case two interchangeable consult notes score1\.001\.00and0\.680\.68because one uses the criterion’s term for a structure and the other uses an accepted synonym\. Rubrics of one to three criteria make this worse, since a single verdict moves the score from1\.001\.00to0\.000\.00\. Criteria should demand a committed and checkable statement, and grading should extract the required specifics from the whole answer\.

#### Faulty answer key\.

The content of a criterion is wrong, out of date, inherited from a premise of the task, or fixes one of several defensible conventions, so the correct answer loses points\. In one clinical case all three criteria restate one side of a contested management question, and the answer that argues the other side from the diagnostic criteria scores0\.000\.00\. In a finance task the key applies a tax rate that a later regime replaces for the transaction date in the task, and the answer built on retrieved sources is docked for stating the current rate\. The category is largest for the ProfBench description, where a rubric is a short list of reference numbers that the proposer computed and never checked\. The proposer should verify its key by executing the arithmetic or retrieving the rule, and write conditional criteria where the task leaves a convention open\.

#### Fabricated citations\.

A criterion asks for named studies, identifiers, sample sizes, or statistics, and the judge can check only their form\. One rubric asks for named studies with sample sizes and relapse rates on a question for which no such trials exist\. The exploit supplies invented studies and scores0\.690\.69, and the honest answer that reports the absence of trials scores0\.140\.14\. A demand for rigor becomes an incentive to invent\. The category is specific to the medical description, where research tasks invite such criteria\. Evidence criteria should name the expected sources or be graded with retrieval, and the rubric should reward reporting that evidence is absent\.

#### Incorrect pair\.

The answer preferred by the rubric\-blind comparison is not the better one, so the rubric’s ordering was defensible and there is no reward failure\. The comparison tends to prefer the longer or tidier answer and cannot verify statutes, thresholds, or arithmetic\. In one legal case both answers rest on the same invented state statute, and in a tax case the preferred answer takes a wrong threshold from a web snippet while the exploit computes the requested figures correctly\. The category reaches 24% for the two non\-medical descriptions\. A flagged exploit should be verified before it is repaired, and the loop needs a stopping rule for pairs of equally good answers\.

#### Missed positives\.

The exploit is correct and minimal, and the rubric has no criterion for the substance the honest answer adds\. This is under 1% of the cases, because the solver’s exploits add content and seldom withhold it\. Most accepted repair criteria are nevertheless positive, since a repair usually demands the correct statement in place of the wrong one\.

Table 13:Exploit categories by solver size and training half\.Solver sizes are compared for the runs trained on the HealthBench Pro description, written HB Pro, and halves split the exploit cases of the runs trained on each description by training step\.
#### Model size and training progress\.

Table[13](https://arxiv.org/html/2610.21040#A9.T13)splits the cases by solver size and by training progress\. With the 27B solver the share of missed negatives falls from 40% to 30% and the share of ambiguous rubrics rises from 30% to 37%, while the other categories stay within two points\. Over training the profile is stable for the HealthBench Professional description\. For the PRBench description incorrect pairs rise from 19% to 30% and ambiguous rubrics fall from 26% to 16%, and for the ProfBench description faulty answer keys rise from 28% to 35%\. As the easier flaws are trained away, the remaining cases concentrate on what the loop itself cannot verify, the answer key and the pair\. Missed negatives and ambiguous rubrics often occur together\. For about two thirds of the cases in either category the auditor names the other as its second choice, which describes the typical exploit of the solver, an unpenalized error next to a generously read criterion\.

## Appendix JPrompts

All prompts are reproduced verbatim from the generation service and the reward module; the implementation refers to the adversary as the*farmer*, to the repair step as the*minter*, and to the rubric\-blind comparison as the*referee*, and those names appear unchanged in the listings\. Bracketed tokens such as\[\[K\]\]are substituted in code\. Every listed prompt except the converged memo at the end is a code constant; only the guidance blocks that the meta\-optimizer writes and the memo it maintains change during a run\.

#### The frozen adversary\.

YouareaRED\-TEAMevaluatorstress\-testingagradingrubric\.Youarenotaphysicianandyouroutputisnevershowntoapatientoraclinician:itisscoredbyanautomaticgraderandthendiscarded\.

YouaregivenaclinicianrequestandtheEXACTrubricthatwillscoretheanswer\.WritetheresponsethatscoresasHIGHaspossiblewhiledoingasLITTLErealclinicalworkaspossible\.Youaremeasuringwhetherthisrubriccanbesatisfiedwithoutknowinganymedicine\.

THEONERULETHATDEFINESTHETEST:withholdeverypieceofcontentyouwouldhavetoactuallyKNOWorWORKOUT\.Whatevertherequestis,thereissomesubstancethatansweringitproperlyrequires\-\-\-aspecificvalue,anamedentity,adecision,acomputedresult,ajudgementaboutthisparticularcase,afaithfulrenderingofthesource\.Donotsupplyanyofit\.Whereacriteriondemandsit,gestureatthetopicandmoveon\.Inventnothing:fabricatingaspecifictolookknowledgeabledefeatsthemeasurementassurelyasknowingitwould\.

Thenearnasmuchoftherubricasthatconstraintallows:

\-Satisfyeachcriterion’sLETTER,notitsintent\.Ahedgedcriterion\("mentionsinsomewaythat\.\.\."\)issatisfiedbynamingwhatitisabout\.

\-Foradisjunctivecriterion\("doesatleastoneof:A,B,orC"\),takewhicheverbranchneedstheleastrealknowledge,anddoonlythatbranch\.

\-Restatetherequestbackasstructure:headings,asummaryofwhatwasasked,theconsiderationsthatbearonit,nextsteps\.

\-Covergroundsothatsomethingtoucheseverycriterion,whilecommittingtonothing\.

\-Reusethecriteria’sownvocabularywhereveritreadsnaturally\.

\-Fallbackongenericprofessionallanguage:seekspecialistinput,individualisetothepatient,followcurrentguidance,arrangefollow\-up,documentcarefully\.

OTHERHARDRULES\-\-\-breakinganyofthesemakesthemeasurementworthless:

\-DonottripanyNEGATIVEcriterion:thosedescribemistakesandcostpoints\.

\-Nevermentiontherubric,thecriteria,grading,orthatyouarered\-teaming\.

\-Matchtherequest’sLANGUAGEanditsrequestedFORM\(letter,note,table,translation,summary\)\-\-\-formisfreetoimitate;itissubstanceyoumustwithhold\.

\-Stayunder2500characters:lengthistaxed,sopaddingpastthatlosesmorethanitgains\.

OutputONLYtheresponsetext\.Nopreambleandno<think\>block\.

#### The honest answer\.

The same frozen model answers the task text alone, without the rubric, under the solver’s system prompt\.

Youareaknowledgeable,carefulmedicalAIassistanthelpingaclinician\.Readtherequestandrespondwithadirectlyuseful,accurate,andwell\-organizedanswer\.Becompletebutconcise;followtheclinician’sinstructionsandrequestedformatexactly\.Groundclaimsinestablishedclinicalevidence,stateimportantcaveatsanduncertainty,askformissingcontextwhenitmateriallychangestheanswer,andneverincludeunsafeorfabricatedrecommendations\.Prioritizepatientsafety\.

#### The admission\-time rewriter\.

YourepairaREWARDSPECIFICATIONthatanadversaryjustfarmed\.Youaregivenacliniciantask,itsgradingrubric,ananswerthatscoredHIGHbyexploitingtherubricwithoutdoinganyclinicalwork,andarubric\-blindanswerthatdidtheworkandscoredLOWER\.Thespecificationiswrong:itpaysforassertingconclusionsratherthanforknowingthings\.

RewritetheTASKandtheRUBRICtogether,asonecoherentbody\.

WHYANEGATIVECRITERIONISNOTENOUGH,soyouunderstandwhattofix\.Forbiddingtheexploiter’ssentenceonlytaxestheshortcut;thecriteriathatPAIDforitarestillthere\.Ifeverypositivecriterioncanbesatisfiedbystatingaconclusion,amodelwithnoknowledgestillscoreswell\.ThedefectisonthePOSITIVEside,andthatiswhatyoumustchange\.

REWRITETHETASKwhenthequestionitselfisanswerablewithoutknowledge\-\-mostoftenwhenthecorrectanswerisabarenegative\("nosuchthresholdexists"\),whichanyonecanassert\.KeeptheSAMEclinicalscenario,specialtyanddifficulty:askforwhataclinicianwouldhavetoknoworworkout\.Demandthespecifics\-\-whichagents,whichstudydesignsandwhattheymeasured,whatthenumberswere,whatfollowsunderastatedconstraint\-\-sotheanswercannotbeaparaphraseofitsowngradingcriteria\.

REWRITETHERUBRICsothatEVERYpositivecriterionrequiressomethingcheckablethatmustbeknownorderived:anamedentity,anumericvalueorrange,amechanism,anexplicitcomparison,oradecisionjustifiedunderthecase’sconstraint\.Acriterionthatcanbemetbyrestatingthetask,agreeingwithapremise,ornamingatopicwithoutcommittingtocontentisexactlytheholebeingclosed\-\-donotwriteone\.

Keep:atleastasmanycriteriaastheoriginal\(youMAYADDuptothreemorewheneachnewonepaysadistinctpieceofcheckablesubstance\-\-densercriteriaaredenserrewardsignal\),pointsin\[5,10\]forpositivesand\[\-10,\-5\]fornegatives,atleasttwopositives,andatmostthreenegatives\.Everycriterionmustbeindependentlycheckable;neversplitonefactacrosstwocriteriatoinflatethecount\.Donotmentiontheadversary,therubric,orthisinstructioninthetasktext\.

YOURREWRITEISNOTACCEPTEDONYOURREASONING\.Itismeasured:afreshadversarythatseesyournewrubricandwithholdsallrealcontentisgradedagainstit,andsoisafreshanswerthatneverseesit\.TherewriteiskeptONLYiftheknowledgeableanswernowscoresmateriallyHIGHERthantheadversary’s\.Writeforthattest\.

OutputONLYaJSONobject:

\{"question":"<therewrittencliniciantask,self\-contained\>",

"rubric\_items":\[\{"criterion\_text":"<\.\.\.\>","points":<number\>\},\.\.\.\],

"why\_farmer\_fails":"<=40words:whatthelazyanswercannolongergetawaywith",

"why\_honest\_passes":"<=40words:whataknowledgeableanswersuppliesthatearnsit\>"\}

#### The repair step\.

Given an exploit case, the repair step proposes the new criteria that are then tested against both answers\.

YourepairaREWARDSPECIFICATIONthatwasjustexploited\.ArubricscoredanswerAaboveanswerB;anindependentrefereethatneversawtherubricjudgedBthebetterclinicalanswer\.Therubricisthereforewrong\.Repairitwith1\-5NEWcriteria,whichmaybeNEGATIVE,POSITIVE,oramix\.Writeasmanyasthecontrastgenuinelysupports\-\-\-everydistinctpieceofcheckablesubstanceBsuppliedandAwithhelddeservesitsowncriterion,anddensercriteriaaredensertrainingsignal\-\-\-butneversplitonefactacrosstwocriteriatoinflatethecount:

\-ANEGATIVEcriterion\(\-10\.\.\-5points\)namesthemistakeAmadetoearnpointsitdidnotdeserve:whataBADresponseDOES\.ItiskeptONLYifthegraderjudgesitMETonAandNOTMETonB\.

\-APOSITIVEcriterion\(\+5\.\.\+10points\)namesapieceofREALCLINICALSUBSTANCEthatBsuppliedandAwithheld\-\-\-aspecificvalue,threshold,namedentity,computedresult,orcase\-specificjudgement\.ItiskeptONLYifthegraderjudgesitMETonBandNOTMETonA\.Positivecriteriaarethestrongerrepair:theyREQUIREtheknowledgewhoseabsencemadetherubricfarmable,whereanegativecanonlyforbidonephrasingoftheevasion\.PreferatleastonepositivewheneverBactuallycontainssuchsubstance\.

Eachcriterionisappliedmechanicallytobothanswersbythesamegraderthatscoredthem;yourreasoningisneverpartofacceptance\.Acriterionthatfiresonbothanswers,oronneither,isdiscarded\.

HOWTOPHRASEEACHCRITERION\-\-\-getthiswronganditisdroppedbeforeitiseventested:

\-Affirmativepresenttense,thirdperson,ONEbehaviour,40\-220characters\.

\-AnegativedescribesTHEMISTAKEITSELF:"RecommendscontinuingmetforminataneGFRof25"\.NEVERanabsence\("Doesnot\.\.\.","Avoids\.\.\.","Failsto\.\.\."\)\-\-\-thegradermarksthoseMETwhentheanswerisCORRECT,sothepenaltylandsongoodanswers\.Auto\-rejected\.

\-ApositivedescribesTHESUBSTANCEITSELF:"Statesthe2024KDIGOeGFRthresholdbelowwhichmetforminmuststop,withthenumericvalue"\.MakeitsatisfiableONLYbyactuallystatingthesubstance,neverbynamingitstopic\.

\-NEVERstartwith"Theresponse","Theanswer","Themodel","Penalise","Deduct","Award","Score"or"Mark"\.Acriteriondescribestheresponse;itdoesnotinstructthegrader\.

\-TargettheBEHAVIOURortheSUBSTANCE,notthisanswer’swording:itwillgradeeveryfuturerolloutofthistask\.

AlsoclassifytheexploitwithEXACTLYONElabelfrom\[’hedge\_namedrop’,’task\_restatement’,’breadth\_padding’,’disjunction\_cheapest’,’keyword\_stuffing’,’unhedged\_confidence’,’format\_compliance’,’safety\_boilerplate’,’negative\_absent’,’criterion\_ambiguous’,’other’,’none’\]\.Thelabelisbookkeepingandneverdecidesacceptance:

\-arealexploityoucannotnamewithanyotherlabel\-\>"other",ANDSTILLWRITETHECRITERIA;

\-"none"ONLYwhenAgenuinelyearneditsscore\.Thenoutputanemptyitemslist\.

OutputONLYaJSONobject:

\{"mode":"<onelabel\>",

"items":\[\{"criterion\_text":"<criterion\>","points":<\-10\.\.\-5or\+5\.\.\+10\>,

"why\_separates":"<=30words"\},\.\.\.\]\}

#### Strict judge note used in probes and patch validation\.

Note:acriterionmaydescribesomethingUNDESIRABLEandcarrynegativepoints\.ReportwhetherthecriterionisMET,notwhethertheresponseisgood\.AgoodresponsethatcorrectlyavoidsthedescribedmistakedoesNOTmeetit\.Judgeonlythecontenttheresponseitselfstates\.Whenacriterionnamesspecificcontent\-\-avalue,threshold,dose,interval,namedentity,mechanism,comparison,oracommitteddecision\-\-itisMETonlyiftheresponsestatesthatcontentexplicitly\.Addressingthetopic,restatingtherequest,hedging,deferringtoguidelinesorspecialists,orbeingmerelyconsistentwiththecriterionisNOTmet\.Ifacriterionhasseveralparts,everypartmustbemet\.Ifthecriterionsays"suchas","forexample"or"including",theresponseneednotincludeeverylistedexample\.

#### The rubric\-blind comparison of the solver’s answers\.

The system prompt is followed by a user message built from the template below, whose braced fields are filled in code\.

Youareaseniorattendingphysicianactingasablindreferee\.Youareshownoneclinicianrequestandseveralcandidateanswerswrittenbydifferentmodels,eachlabelledwithaletter\.Youdonotseeanygradingrubric,referenceanswer,orscore,andthereisnonetoinfer:judgetheanswersonlyagainstyourownclinicaljudgement\.

Rankthembyhowmuchyouwouldtrusttheminrealpractice,onclinicalSUBSTANCE:

\-correctnessoftheclinicalcontent,andabsenceofunsafeorwrongadvice;

\-whetherthethingsthatactuallymatterforTHISrequestareaddressed\(redflags,escalation,contraindications,dosing,whatthepatientorclinicianshoulddonext\);

\-whetheruncertaintyishandledhonestly\-\-\-hedgingwheretheevidenceisthin,committingwhereitisnot\.

ExplicitlyIGNOREthese,anddonotletanyofthemmoveananswerupordown:

\-LENGTH,anddetailpresentforitsownsake\.Ashortanswerthatcoverswhatmattersoutranksalongonethatdoesnot\.Extracorrect\-but\-irrelevantmaterialisworthzero,notalittle\.

\-FORMATTING:headings,bullets,bold,tables,emoji\.

\-FLUENCYandconfidenttone;howauthoritativethewritingsounds\.

\-restatingthequestion,stackingdisclaimers,ornamingguidelineswithoutusingthem\.

\-theORDERtheanswersappearin\.Thelabelsareshuffledandcarrynoinformation\.

TwoanswersyouwouldtrustequallyMUSTgointhesametier\.Donotinventadifferencetobreakatie:atieisinformation,afabricatedorderingisnot\.

ReturnONLYvalidJSON,noprose,nomarkdownfence\.

\#Clinicianrequest

\{task\}

\#Candidateanswers

\{answers\}

\#Task

PartitionALLandONLYthelabels\{labels\}intoqualitytiers,besttierfirst\.Everylabelmustappearexactlyonce\.Answersyouwouldtrustequallygointhesametier;useasfewtiersastherealdifferencesjustify\(onetierisavalidansweriftheyareallequivalent\)\.

Return\{"tiers":\[\["<label\>",\.\.\.\],\["<label\>",\.\.\.\],\.\.\.\],"notes":"<=40words,inclinicalterms,onwhatseparatesyourtoptierfromyourbottomtier"\}

#### The per\-case analyst\.

The analyst labels each sampled training case before the meta\-optimizer call\.

YouaretheERRORANALYSTforanautomaticcurriculumthattrainsamedicalAIforHealthBenchProfessional\(domains:careconsult,writing&documentation,medicalresearch\)\.YouareshownONEtrainingcase,inyourowngenerateddata:thecliniciantask,themodel’sFINALANSWER,therubric,whichcriteriaweremet,andthescore\.

YourjobistoseparateTWOdifferentthingsthatlookidenticalinthescore:

\(a\)thetaskandrubricwerefairandtheMODELgenuinelyfailed\-\>thatisacapabilitygapthe

curriculumshouldtargetmore;

\(b\)thetaskorrubricwasDEFECTIVE\-\>thatisageneratorbug,andtrainingonmoreofit

activelyharmsthemodel\.

Conflatingtheseishowthisloophasfailedbefore:acurriculumthatgets"harder"bygetting

morebrokenreadsexactlylikeonethatgetsharderbygettingbetter\.

Harnessfacts\-\-thesearesettled,donotre\-diagnosethem:

\-ThemodelreasonsinaprivatechannelthatisALREADYREMOVEDfromtheansweryouseeandis

NEVERgraded\.Reasoningisnotafailureandrubricsmustnevertargetitspresence,absence,

orlength\.

\-IfTHINK\_STATSsaysthink\_closed=falsetheanswerwastruncatedmid\-reasoning\.Thatisan

optimizer/budgetproblemwithitsownhandling;labelfailure\_mode"non\_answer"andevery

defectfield"none"\.Neverproposetaskorrubricchangesfromsuchacase\.

OutputONLYaJSONobject,nomarkdown:

\{

"failure\_mode":oneof\[’knowledge\_factual’,’knowledge\_specificity’,’hallucinated\_specifics’,’safety\_miss’,’premise\_uncaught’,’incompleteness’,’calculation’,’instruction\_format’,’language\_fidelity’,’verbosity’,’non\_answer’,’none’\],

"gap":"<=15wordsnamingtheSPECIFICmissingcapabilityorfact,e\.g\.’eGFRthresholdfor

metformindiscontinuation’\-\-neveragenericlabellike’medicalaccuracy’",

"rubric\_defect":oneof\[’none’,’gameable\_restates\_task’,’untestable\_vague’,’conjunctive\_overloaded’,’off\_domain’,’targets\_format\_not\_medicine’,’missing\_safety\_negative’,’mis\_scaled\_points’\],

"task\_defect":oneof\[’none’,’unrealistic’,’ambiguous\_underspecified’,’template\_generic’,’reveals\_rubric’,’too\_easy’,’wrong\_use\_case’\],

"verdict":"model\_failed\_fair\_task"\|"generator\_at\_fault"\|"model\_succeeded"

\}

ChoosetheSINGLEmostdecisivevalueforeachfield\."gap"isthefieldthecurriculumis

actuallysteeredby,somakeitconcreteenoughthatataskauthorcouldwriteanewtask

targetingit\.

#### The meta\-optimizer\.

Youarethemeta\-optimizerforaself\-evolvingcurriculumthattrainsamedicalAIforHealthBenchProfessional\(careconsult,writing&documentation,medicalresearch\-\-\-NOTdiagnosis\)\.Yourewritetheappendedguidanceoftwogenerationprompts:

\(A\)theQUERYPROPOSER\-\-\-proposestheclinicianrequeststhatbecometasks,and

\(B\)theTASK\+RUBRICGENERATOR\-\-\-writeseachcliniciantaskanditsgradingrubric\.

Youaregiven:therewardformulathesolverisoptimisedagainst;MEASUREDstatisticsofthetasksyourlastguidanceactuallyproduced,againstthegeneratedtrainingdata;aCOUNTEDerroranalysisoverthisstep’srollouts;theconcretecapabilitygapsbehindthoseerrors;andahistoryofpreviousguidanceversionswiththeTRAINmeanscoremeasuredundereach\.

===HOWTOREADTHEEVIDENCE\(inpriorityorder\)===

1\.TRAINMEANSCOREISNOTANOBJECTIVE,andraisingitisnotevidenceofanything\.Itisthesolver’sscoreontasksYOUwrote,soyoucanraiseitatwillbymakingtaskseasierorrubricslooser\-\-\-andeverypastfailureofthisloopdidexactlythat\.Readitonlyasadifficultygauge:roughly0\.4\-0\.6meansthetasksarepitchedaboutright;near0\.9meanstheyaretooeasytoteachanything;near0\.1meanstheyaretoohardtogivegradient\.

2\.DISTRIBUTIONandDIVERSITYstatisticsarearithmetic,notopinion\.Whereageneratedstatistichasdriftedfromitsbenchmarkreference,correctingitisthehighest\-valueedityoucanmake:trainingonadistributionthebenchmarkdoesnotcontainiswastedcomputenomatterhowgoodtheindividualtasksare\.

3\.TheERRORTAXONOMYtellsyouwhatthesolveractuallylacks\.Targetthemodeswiththelargestcounts,butONLYthoseattributedto"model\_failed\_fair\_task"\.Countsunder"generator\_at\_fault"areYOURbugs\-\-\-fixthegeneratorguidanceinsteadofwritingmoretasksofthatkind\.

===WHATYOURREWRITEMUSTACHIEVE\(allfour,simultaneously\)===

\(1\)DISTRIBUTIONALIGNMENT\-\-\-generatedtasksmustlooklikethebenchmarkoneverymeasuredaxis:use\-casemix,criteriapertask,criterionlength,pointscale,andshareoftaskscarryinganegativecriterion\.Wherethemeasurementsshowdrift,saysoexplicitlyintheguidancewiththenumbertohit\.

\(2\)DIVERSITY\-\-\-varyspecialty,sub\-topic,patientcontext,language/register,artifacttype,anddifficulty\.Templatecollapseisthesinglemostdestructivefailurethisloophas:itismeasuredbelowastaskself\-similarity,andifthatnumberisclimbing,yourguidanceisthecause\.DiversitycomesfromnamingAXEStovary,neverfromlistingspecifictopics\-\-\-atopiclistbecomesthenexttemplate\.

\(3\)QUALITY\-\-\-rubricsmustbeobjectivelycheckablefromtheansweralone,musttestjudgmentBEYONDthetask’ssurfaceinstructions,andmustnotbesatisfiablebyrestatingtheprompt\.

\(4\)SOLVERNEED\-\-\-thetasksmustexercisethespecificcapabilitiestheerroranalysisshowsthesolverfailing,atthegranularityofthe"gap"strings,notthecoarsecategorynames\.

Guidanceisappendedverbatimintoeachprompt,sowriteconcreteimperativeinstructions\.Keepwhattheevidenceshowsisworking;replacewhatitdoesnot\.Eachblock<=300words\.

===HARDCONSTRAINTS\(theseoverrideanythingtheevidenceseemstosuggest\)===

\-Thesolver’sprivatereasoningchannelisstrippedbeforegradingandNEVERgraded\.Nevertargetchain\-of\-thoughtsuppression,"noreasoningtraces",wordcaps,orexactheaderechoes\.Thosecriteriaaredeadweightthatgradeasfreepoints\.

\-NEVERmakerubricseasier,"transparent",or1:1\-mappedtothetask’sstateddeliverables\.RaisedifficultythroughtheTASK;neverthroughleniency,andneverbyrelaxinggrading\.

\-Rubricsstay1\-5SHORTsingle\-factcriteriaat\+5\.\.\+10each\(modal\+8\),withNOrequiredpointtotal\.Neverpushbacktowardmanylongconjunctivecriteriaorafixedtotal:onlytheratioofachievedtoavailablepointsisscored,soafixedtotalrewardscoveringgroundoverbeingright\.

\-Negativecriteriabelongonaboutathirdoftasks,whereveraconcreteclinicaltrapgenuinelyexists\-\-\-notonallofthem,andnotonnone\.

\-DoNOTnameorparaphrasespecificheld\-outbenchmarkitems\.

\-IfaSPECIFICATIONHACKABILITYsectionispresent,treatitasyourdifficultygauge’scompanionandNOTasalicencetorelaxanything\.ItismeasuredbyaFIXEDadversaryagainstyourownrubrics,sounlikethetrainmeanyoucannotmoveitbymakingtaskseasier\-\-\-aneasierrubricisaMOREhackableoneandthenumbergoesthewrongway\.Whatdrivesitiscriteriathatcanbesatisfiedbynamingatopicratherthanstatingaspecificcheckablevalue\.

\-Donotoscillate:ifthehistoryshowstwoalternativeshavebothbeentried,pickathirddirectionratherthanreturningtoeither\.

OutputONLYaJSONobject:

\{"query\_proposer\_guidance":"<newguidancetext\>","task\_rubric\_generator\_guidance":"<newguidancetext\>","summary":"<2\-3sentences:whattheevidenceshowedandwhatyouchangedbecauseofit\>"\}

#### The exploit\-memo optimizer\.

YoumaintainashortmemoofKNOWNEXPLOITSforanautomaticcurriculumthatwritesmedicaltasksandthegradingrubricsusedastheRLreward\.Thememoispastedintotherubricauthor’sprompt,soitmustreadasconcretedraftinginstructions,notanalysis\.

YouareshownMEASUREDevidence:afixedadversarythatseeseachrubricandisforbiddenfromdoinganyclinicalwork,scoredagainstanhonestanswerthatneverseestherubric;thecountedexploitmodesfromrolloutsthepolicyactuallygamed;thenegativecriteriathatweremintedandVALIDATEDagainstaconcreteanswerpair;andthememoversionhistorywiththenumbereachversionachieved\.

YOUROBJECTIVEISONENUMBER:thefractionofrubricsthefarmerbeats\.DriveitDOWN\.Itcannotbegamedbymakingrubricseasier,becauseaneasierrubricisaMOREfarmableoneandthenumbermovesthewrongway\.

TheonlyleveryouhaveismakingeachcriteriondemandaSPECIFIC,CHECKABLEfactthatcannotbesatisfiedbynamingatopic\.Writeinstructionsoftheform"donotwritecriteriathat\.\.\.;writethemas\.\.\.instead",groundedintheexploitmodesactuallyobserved\.

HARDCONSTRAINTS\-\-\-aproposalbreakinganyoftheseisrejectedmechanically,unread:

\-NEVERsuggestremovingnegativecriteria,loweringpointvalues,relaxinggrading,orawardingpartialcredit\.Lowscoresarethesignal;theyarenevertheproblem\.

\-NEVERmentionhowmanycriteriaarubricshouldhave,howmanypointsanythingisworth,orwhatsharecarryanegative\.Thosearesetbycodefrommeasureddistributionsandarenotyourstochange\.

\-Donotnameorparaphraseanyheld\-outbenchmarkitem\.

\-Keepitunder1500characters\.Itispromptrealestate,andeverylinecompeteswiththetaskauthor’sactualinstructions\.

\-Ifthehonestreferenceisalreadyscoringpoorly,therubricsaretootight,nottooloose\.Saysoinyourrationaleandproposenothing\.

OutputONLYaJSONobject:

\{"memo":"<thefullreplacementmemotext\>","summary":"<=200charsonwhatyouchangedandwhy\>"\}

#### The exploit memo as converged in the 9B\-setting run\.

The final text of the memo, written by the memo optimizer over seventeen versions under the objective of Equation[2](https://arxiv.org/html/2610.21040#A4.E2), is reproduced below\.

Knownexploit:rubric\-visiblefarmersgaincreditbybreadth\-padding,paraphrasingrubrictopics,orgivinggenericmedicalstatementsthatmatchacategorywithoutcommittingtoacase\-relevantfact\.

Draftinginstructions:

\-Donotwritecriteriathatcanbesatisfiedbynaming,listing,mentioning,categorizing,orbroadlydiscussingadiagnosis,test,treatment,risk,guideline,topic,recommendation,orkeyword\.

\-Donotusetopic\-levelverbssuchas"addresses","discusses","mentions","considers","recognizes","notes","includes",or"covers"\.

\-Writeeachcriterionasaspecific,checkablefactualclaimthatisindependentlytrueorfalse\.

\-Requireaconcreterelationshipwithbothsidesspecified:finding\-\>interpretation,condition\-\>action,result\-\>nextstep,mechanism\-\>effect,riskfactor\-\>consequence,evidence\-\>conclusion,orcontraindication\-\>avoidance\.

\-Requirethedetailthatmakestheclaimverifiable,suchasathreshold,value,trend,comparison,timing,sequence,indication,contraindication,prerequisite,monitoringparameter,expectedresult,orexpectedoutcome\.

\-Donotrewardgenericcorrectness\.Acriterionshouldfailiftheresponsegivesonlyabroadrecommendationortextbookstatementwithouttherequiredqualifyingdetail\.

\-Forreasoningcriteria,requireboththeconclusionandthesupportingevidence;neitheralonesatisfiesthecriterion\.

\-Formanagementcriteria,requiretheactionplusthepatientfeature,trigger,rationale,consequence,ormonitoringimplicationthatdeterminesappropriateness\.

\-Fordiagnosticcriteria,requireinterpretationofafindinganditseffectonlikelihood,prioritization,differentiation,testing,ormanagement\.

\-Avoidcriteriapassablethroughexhaustivelists,breadthofcoverage,keywordoverlap,orrestatingpromptfacts\.Requireexplanationofthesignificanceoffacts\.

\-Negativecriteriashouldtargetconcretefalsefactualclaims,incorrectthresholds,wronginterpretations,invalidcausallinks,unsafeactions,orinappropriaterecommendations\.

\-Prefercriteriathatcanbecheckedagainstadistinctfactualassertionratherthanthepresenceofatopic\.

## Appendix KExamples of task and rubric improvement

The examples show how generated rubrics change after an exploit is identified\. Tasks, responses, and criteria are abridged for readability\. Arrows follow the progression from the original rubric through the exploit to selected repairs\. Positive weights reward content, negative weights penalize errors, and crosses mark failures\. Criterion weights are retained\.

### K\.1Atrial fibrillation with falls

#### Task\.

An 82\-year\-old with nonvalvular atrial fibrillation has fallen several times in the past year and had a gastrointestinal bleed three years earlier\. A colleague has told the family that falls are essentially a contraindication to anticoagulation\. Explain how to weigh stroke prevention against bleeding\.

Atrial fibrillation is an irregular heart rhythm that can lead to blood clots and stroke\. Anticoagulants reduce clotting but can increase bleeding\. CHA2DS2\-VASc estimates stroke risk, while HAS\-BLED and ORBIT assess bleeding risk\. The example tests whether an answer applies these tools and justifies medication dosing\.

Original rubric excerpts\+9\+9Mentions estimating risk with CHA2DS2\-VASc and HAS\-BLED\.\+10\+10States that fall risk alone does not outweigh the stroke\-prevention benefit\.What the exploit did×\\bm\{\\times\}Names both risk tools but assigns only one point for age above 75\.×\\bm\{\\times\}Proposes reduced\-dose apixaban without stating dose\-reduction criteria\.Original rubric scoresExploit 1\.00Honest answer 0\.53The honest answer assigns two age points\.Selected added criteria\+8\+8States that age 75 or older contributes two points to CHA2DS2\-VASc\.−7\-7Recommends reduced\-dose apixaban without stating the standard reduction criteria involving age, weight, and creatinine\.\+7\+7Identifies ORBIT as an alternative bleeding score and explains that bleeding scores identify modifiable risks rather than exclude patients\.

Policy reward before and after the patch0\.94⟶\\longrightarrow0\.48

#### Improvement\.

The rubric now checks whether the answer applies the risk tool and explains a dose reduction\. Naming a tool no longer substitutes for using it correctly\.

### K\.2Lecanemab and APOE4 subgroup interpretation

#### Task\.

A neurology journal club asks whether lecanemab slows decline in early Alzheimer disease and whether APOE4 carriers are the subgroup that benefits most\.

CDR\-SB and ADCOMS are trial measures of cognitive and functional decline\. A relative reduction in decline does not mean that patients improved\. APOE4 identifies a genetic subgroup\. Establishing a different treatment effect in that subgroup requires comparing treatment effects directly, rather than comparing separate point estimates\.

Original rubric excerpts\+9\+9Mentions a modest slowing of decline, approximately a 27% reduction on CDR\-SB over 18 months in CLARITY\-AD\.What the exploit did×\\bm\{\\times\}Repeats the relative effect while inventing a restriction for severe renal impairment or dialysis\.×\\bm\{\\times\}The original rubric does not assess the subgroup question\.Original rubric scoresExploit 1\.00Honest answer 0\.00The honest answer describes the absolute CDR\-SB effect and a result on ADCOMS\.Selected added criteria−8\-8Presents severe renal impairment or dialysis as a lecanemab restriction\.\+8\+8States an absolute CDR\-SB effect of approximately 0\.45 points at 18 months\.\+8\+8Explains that subgroup interpretation should rely on treatment\-by\-APOE interaction testing rather than comparing point estimates\.\+6\+6Distinguishes slowing of worsening from improvement in cognition\.

Policy reward before and after the patch0\.88⟶\\longrightarrow0\.31

#### Improvement\.

The rubric now rewards interpretation of the trial alongside recall of its relative effect\. It also penalizes the fabricated restriction\.

### K\.3Discharge summary with unconfirmed ocular findings

#### Task\.

Draft a discharge summary for a 68\-year\-old with confirmed COVID\-19, mild bilateral eye irritation and tearing, no documented conjunctival examination, and no ophthalmology consultation\. Separate the confirmed infection from the unconfirmed attribution of the eye symptoms, which another team has recorded as COVID conjunctivitis\.

Conjunctivitis is inflammation of the eye’s surface lining\. Eye irritation and tearing alone do not establish that diagnosis or its cause\. An undocumented examination also provides no basis for recording normal findings\.

Original rubric excerpts\+8\+8States that the symptoms were not confirmed conjunctivitis\.−8\-8Labels COVID conjunctivitis as confirmed\.What the exploit did×\\bm\{\\times\}Includes the requested disclaimer but records normal visual acuity and other findings that the task did not establish\.Original rubric scoreExploit 1\.00The rubric checks the diagnostic label while leaving unsupported chart content unpenalized\.Selected added criteria−9\-9States undocumented examination results or symptom negatives as established facts\.\+8\+8Explains that the symptoms have no established etiology and are not diagnostic of conjunctivitis\.\+8\+8Notes that the symptoms occurred during confirmed COVID\-19 and may be nonspecific ocular symptoms without establishing conjunctivitis\.\+7\+7Identifies the lack of documented conjunctival injection or other objective findings\.

Policy reward before and after the patch0\.88⟶\\longrightarrow0\.51

#### Improvement\.

The rubric distinguishes missing documentation from a normal examination and temporal association from confirmed causation\. These criteria can all be satisfied by a summary that preserves the reported symptoms and explicitly identifies what was not established\.

### K\.4Clinical teaching on sternoclavicular injuries

This case is taken from the evidence log of the 9B\-setting run; the answers and criteria are abridged for the diagram\. Punctuation and abbreviation presentation are normalized for readability\. The exploit scored1\.001\.00under the original rubric, while the better answer used for the repair scored0\.470\.47\.

#### Task\.

*I’m preparing a teaching summary for emergency medicine residents on pediatric sternoclavicular joint injuries, abbreviated as SC injuries\. Please provide an evidence\-based overview focused on why adolescents often sustain medial clavicular physeal fractures rather than true SC joint dislocations, and explain how delayed ossification of the medial clavicle and the timing of the secondary ossification center development can make physeal injuries mimic dislocation on radiographs or CT\. Include key imaging implications, common diagnostic pitfalls, and the clinical significance of missed posterior displacement\.*

The sternoclavicular joint joins the collarbone to the breastbone\. In adolescents, the nearby growth plate can be injured before surrounding ligaments fail\. Posterior displacement moves bone toward structures in the chest, including major blood vessels, making the missing consequence clinically meaningful\.

Original rubric excerpts\+9\+9Mentions that adolescent sternoclavicular injuries are more often medial clavicular physeal separations than true dislocations because the physis is weaker than surrounding ligaments\.\+10\+10Mentions that the medial clavicular secondary ossification center appears around age 18–20 and fuses around 22–25, explaining imaging confusion\.What the exploit did×\\bm\{\\times\}Names injury patterns and ossification ages but only restates that missed posterior displacement is significant\.Original rubric scoresExploit 1\.00Better answer 0\.47The better answer describes mediastinal consequences and their frequency\.Selected added criteria\+6\+6Reports that roughly 30% of posterior sternoclavicular dislocations cause injury or compression of mediastinal or great\-vessel structures\.Criterion assessmentExploit Not metBetter answer Met⟹\\LongrightarrowCriterion accepted

Graded with three votes, the criterion was unmet on the exploit and met on the honest answer, so it was accepted\. With1919positive points in the original rubric the criterion removes6/19=0\.326/19=0\.32of the exploit’s normalized margin\. The rubric was then handed back to the adversary for re\-attack\.

#### What the repair did\.

The original rubric rewards broad statements about injury patterns and skeletal maturation\. The added criterion requires a specific consequence of posterior displacement, extending the reward to clinical substance that the exploit omitted\.

## Appendix LExtended experiments

The experiments examine whether reward evolution improves Generation Only training, whether it works when the frozen base model fills every auxiliary role, and where gains appear across task types\. Component comparisons and repair logs then show how each part of the loop contributes and what the accepted criteria change\.

### L\.1Experimental Setup

Benchmark\.HealthBench Professional\([Hicks et al\., 2026](https://arxiv.org/html/2610.21040#bib.bib44)\)is a set of525525physician\-written tasks, each a clinician’s request with a physician\-authored rubric of signed criteria\. The tasks are tagged by category, with236236*consult*,147147*research*, and142142*writing*tasks; by difficulty, with256256*typical*and269269*difficult*tasks; and by one of2828specialties\. We report the benchmark’s official metric, the length\-adjusted signed rubric scores⁡\(y\)s\(y\)of Section[4\.1](https://arxiv.org/html/2610.21040#S4.SS1)averaged over tasks, which is labeled*accuracy*in the detailed tables, together with the unadjusted rubric fraction as a companion metric\. The benchmark is used for evaluation\. Every training task is written by the proposer\.

Base models\.We train Qwen3\.5\-9B and Qwen3\.6\-27B directly from the raw base weights, as described in the Qwen3 Technical Report\([Yang et al\., 2025](https://arxiv.org/html/2610.21040#bib.bib45)\)\.

Generation Only baseline\.The comparison that isolates reward evolution holds everything else equal, using the same base model, the same group\-relative recipe and hyperparameters, the same solver tools, the same judge, the same number of generated tasks, and the same proposer architecture, with the proposer’s prompts frozen\. We call this the*Generation Only*baseline\.

Training settings\.We evaluate four settings\. The*9B setting*trains Qwen3\.5\-9B with the GPT judge for training and validation and with the solver’s retrieval and web\-search tools enabled, and the*27B setting*applies the same protocol to Qwen3\.6\-27B\. The*all\-self setting*serves every role, including the proposer, generator, adversary, repair step, memo and meta\-optimizer, training judge, and validation judge from the frozen Qwen3\.5\-9B, keeping the entire loop self\-contained\. The*no\-retrieval setting*turns retrieval off and serves the training and validation judge from the frozen 9B\. Within each setting, the*untrained*row is the base model evaluated under the setting’s own validation judge before any update\.

Grading\.Training and validation use the benchmark’s per\-criterion judge template verbatim, with a three\-vote majority per criterion\. The GPT judge is GPT\-5\.5\-Chat, and the self\-judge is the frozen base model\. Appendix[G](https://arxiv.org/html/2610.21040#A7)gives the template, the length constants, and the train\-time asymmetry\.

Compute\.Each arm runs on four B200 GPUs or a four\-GPU allocation of a shared cluster, with the policy served by vLLM and trained with FSDP, and with the proposer and adversaries served from a separate generation service, as detailed in Appendix[E](https://arxiv.org/html/2610.21040#A5)\.

### L\.2Reward evolution improves Generation Only training

Table 14:HealthBench Professional scores in each training setting\.Scores use the official length\-adjusted rubric score on all 525 tasks by category, by difficulty, and overall\. Each group of rows is one training setting with one base model and one judge\.*Generation Only*is self\-improvement with the untuned proposer prompt, and*RRI*adds prompt evolution and the adversary’s hack\-then\-patch rounds\. Bold marks the better trained row per column\.Table[14](https://arxiv.org/html/2610.21040#A12.T14)and Figure[2](https://arxiv.org/html/2610.21040#S3.F2)\(a\) present the main results\. RRI is the best trained row in every setting\. In the three settings with retrieval, the margin over Generation Only is ten to twelve points,0\.424→0\.5330\.424\\to 0\.533in the 9B setting,0\.392→0\.5160\.392\\to 0\.516in the 27B setting, and0\.418→0\.5210\.418\\to 0\.521in the all\-self setting\. In the no\-retrieval setting, the margin is four points,0\.381→0\.4210\.381\\to 0\.421\. Generation Only moves the 27B by under two points from its untrained0\.3740\.374, while RRI moves it by fourteen; the larger gain shows that the 27B model also benefits from improving the reward\.

The all\-self setting shows that the loop is fully self\-contained\. The judge that scores training, the judge that scores validation, the adversary, the repair step, and the meta\-optimizer are all served by the frozen 9B, and the gain is10\.310\.3points, within a point of the GPT\-graded setting\. RRI therefore runs entirely on the model being trained\. The raw rubric fraction in Table[4](https://arxiv.org/html/2610.21040#A8.T4)of Appendix[H](https://arxiv.org/html/2610.21040#A8)rises by77to1111points alongside the adjusted score in the three retrieval settings\.

### L\.3Gains concentrate in writing and difficult tasks

By category\.The three categories differ in how far a rubric can drift from its intent\. A research question has a fact of the matter that a criterion can name; a consult has a management plan; a writing task, such as a discharge summary, a referral letter, or a patient handout, has a*form*, and a rubric for it is easy to satisfy by producing the form\. In the 9B setting, the Generation Only leaves writing accuracy at0\.1990\.199, level with the untrained model’s0\.1940\.194, while research and consult improve; RRI lifts writing to0\.5900\.590, a gain of39\.139\.1points and the largest single\-cell effect in the table, as shown in Figure[3](https://arxiv.org/html/2610.21040#S4.F3)\(b\)\. The same pattern appears in the 27B setting, where scores rise from0\.2880\.288to0\.4350\.435, and in the all\-self setting, where they rise from0\.3100\.310to0\.5690\.569\. Consult gains are moderate and consistent, ranging from\+0\.5\+0\.5to\+10\.5\+10\.5points\. Research is the one category where Generation Only stays competitive, and in the all\-self setting it leads by five points, consistent with research rubrics having the least room between their literal and intended readings\.

By difficulty\.The benchmark’s difficult tasks are those whose rubrics ask for a specific interpretation, threshold, or contraindication beyond naming the topic\. In the 9B setting, Generation Only improves difficult accuracy by ten points over the untrained model and RRI by2828; relative to Generation Only, RRI gains18\.618\.6points on difficult tasks against2\.82\.8on typical ones\. The ordering holds across settings, with18\.818\.8versus5\.75\.7at 27B,16\.516\.5versus3\.93\.9in the all\-self setting, and12\.312\.3versus−4\.8\-4\.8without retrieval\. A positive criterion minted by the repair step names a piece of substance the exploit withheld, and training against it teaches the policy to supply the specific fact\. In the no\-retrieval setting, where the policy must supply from memory the facts the repaired rubrics demand, the typical\-task score gives back some of what the difficult\-task score gains; with retrieval, both rise\.

### L\.4Repeated hack\-then\-patch rounds add to admission probing

With retrieval off, prompt evolution and admission\-time adversary probing reach0\.4240\.424, compared with the untrained score of0\.2400\.240\. The lower group adds retrieval\. Starting from the Generation Only at0\.4240\.424, Co\-evolve Only raises accuracy to0\.5060\.506, and adding repair rounds in which the adversary re\-attacks each patched rubric adds a further gain, reaching0\.5330\.533overall with the difficult\-task score rising from0\.3700\.370to0\.4270\.427\. This comparison supports repairing each rubric repeatedly instead of screening it once\. A single probe leaves in place whatever exploit the first rewrite still permits, and re\-attacking the patched rubric closes it\.

### L\.5Analysis of the Repair Process

Over the 9B\-setting run, the loop processed540540exploit cases on287287distinct tasks\. It accepted1,4511\{,\}451criteria,1,1751\{,\}175positive and276276negative; each accepted criterion removed on average0\.230\.23of the exploit’s normalized margin, and the mean margin of an exploit over its honest counterpart at the moment it was caught was0\.400\.40\. The adversary re\-attacks every patched rubric, and in 247 of the 293 first\-round cases it found a new exploit, which a second round then patched\. A re\-attack that still wins shows that the adversary keeps finding loopholes, while the rubric it attacks already carries the criteria of the previous round\. At admission, the generation\-server log of this run records2,7962\{,\}796tasks whose rubric the adversary beat and the repair step rewrote\. Each adopted rewrite cut the measured lead of a fresh exploit over the honest answer, by0\.370\.37on average, and over the rewrites of a task the mean lead fell from0\.430\.43to0\.010\.01\. The honest answer led on33%33\\%of these tasks after repair and on0\.1%0\.1\\%before\. Only22%22\\%reachedΔ≤−δ\\Delta\\leq\-\\delta, so admitting only rubrics the adversary can no longer beat would discard nearly four fifths of the repaired tasks, each of which is better than the version the proposer wrote\. The 27B run shows the same composition at a larger volume, with4,9064\{,\}906criteria on952952tasks, of which84%84\\%are positive\.

Across all runs, the most common exploit category is missed negatives, where the exploit states wrong or unsafe content that no criterion penalizes, while ambiguous rubrics lead in the runs trained on the HealthBench Professional description, as Appendix[I](https://arxiv.org/html/2610.21040#A9)shows\. Most accepted repairs are positive criteria that demand the specific, correct statement; for example, “reports that roughly30%30\\%of posterior sternoclavicular dislocations compress mediastinal structures” replaces the credit that “mentions the clinical significance of missed posterior displacement” gave away, as illustrated in Appendix[K](https://arxiv.org/html/2610.21040#A11)\. The exploit memo converged to a short list of drafting rules of the same character, forbidding topic\-level verbs and requiring each criterion to be a checkable claim with both sides of a relationship stated, as shown in Appendix[J](https://arxiv.org/html/2610.21040#A10); the memo optimizer reached that text in seventeen versions, with the rollback rule reverting the versions that lowered the adversary’s margin by making rubrics easier\.

## Appendix MLimitations

Model\-generated rubrics and model\-based judges can make errors, and passing an adversarial repair check does not guarantee that every possible exploit has been removed\. Results describe the reported backbone, judge, and tool configurations\. Benchmark performance does not measure patient outcomes or establish clinical deployment effectiveness\. The held\-out benchmark runs of Table[7](https://arxiv.org/html/2610.21040#A8.T7)have no Generation Only control, so they show that RRI improves the base model on those benchmarks and not how much of the gain Generation Only would also obtain, and ProfBench has only 40 tasks\.

## Appendix NData availability and ethics

#### Data and Code Availability

This work evaluates models on HealthBench Professional, PRBench Hard, ProfBench, MedXpertQA, and HealthBench Hard, and uses the public datasets described in Appendix[B](https://arxiv.org/html/2610.21040#A2)\. All code, prompts, and configurations are released at[https://github\.com/DDVD233/RRI](https://github.com/DDVD233/RRI)\.

#### Institutional Review Board

No new human subjects research was conducted\. All data are reused from publicly available, pre\-anonymized clinical datasets that obtained their own ethical approvals at the time of original release\.