SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

# Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
Source: [https://arxiv.org/html/2607.26873](https://arxiv.org/html/2607.26873)
Jianze Wang1,3,\*,‡\\ddagger, Kunwang Zheng2,3,\*,‡\\ddagger, Ying Liu3, Yu Cao3, Qilong Zhang3, Jinlong Chen3, Hua Yang3, Qianglong Chen3,†\\dagger

###### Abstract

Test\-time reinforcement learning \(TTRL\) enables language models to self\-evolve at inference time without labeled feedback\. Existing methods rely on answer voting and therefore do not extend naturally to open\-ended generation, where valid responses cannot be mapped to a shared canonical answer\. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model’s own outputs\. We introduce SERPO \(Self\-Evolving Rubric Policy Optimization\), which replaces answer voting with a closed loop that co\-evolves response evidence, query\-specific rubrics, and policy parameters\. Good–Normal–Bad \(G\-N\-B\) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that discriminate these archives; probabilistic criterion scoring converts verdict\-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals\. New actor rollouts then refresh both the archives and rubrics, closing the three\-way evolution loop\. Across two model configurations, two in\-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to20\.63and20\.31points over the corresponding base models, raises the six\-benchmark macro\-average by up to8\.06points, and supports OOD transfer and continued cross\-benchmark evolution\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\.33footnotetext:Work done during research internship at Alibaba Group\.## 1Introduction

Test\-time reinforcement learning \(TTRL\) enables a deployed language model to self\-evolve on the test distribution through rewards constructed at inference time\. We study open\-ended generation under a transductive setting: the model may use the original test prompts and its own sampled responses, but receives no reference answers, human feedback, external reward models, or stronger judges\. It repeatedly samples responses, constructs rewards, and updates its parameters on the same prompt set\. TTRL therefore depends on a reward signal that remains informative as the policy evolves\.

Existing TTRL methods construct this reward by sampling several solutions, selecting a pseudo\-label through majority voting or self\-consistency\-style aggregation, and rewarding responses that agree with it\(Wang et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib37); Zuo et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib46)\)\. Subsequent work improves pseudo\-label reliability through self\-reflection, entropy\-based confidence, or explicit mitigation of majority\-vote failures\(Wu et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib39); Agarwal et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib1); Lin et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib18)\)\. These methods still assume that responses can be reduced to comparable answers\. The assumption holds for multiple\-choice, symbolic, and canonically normalized tasks, but it fails for open\-ended generation\. Two medical responses may recommend the same action yet differ in contraindications, uncertainty, and escalation guidance; equally safe answers may also use different wording or levels of detail\. Answer voting can therefore reward shared omissions and reject valid alternatives\. Current TTRL objectives remain centered on tasks with extractable answers and do not provide a reliable voting target for open\-ended responses\.

![Refer to caption](https://arxiv.org/html/2607.26873v1/x1.png)Figure 1:Criterion\-level evidence for open\-ended TTRL\.Claim consensus can retain frequent but incomplete advice, whereas SERPO forms G\-N\-B evidence using evolving query\-specific criteria without reference answers\.Rubric rewards offer a natural alternative for open\-ended tasks because they score separate requirements such as factuality, completeness, instruction compliance, and safety\(Liu et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib20); Kim et al\.[2024](https://arxiv.org/html/2607.26873#bib.bib12); Ye et al\.[2024](https://arxiv.org/html/2607.26873#bib.bib43); Liu et al\.[2024](https://arxiv.org/html/2607.26873#bib.bib19); Gunjal et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib8)\)\. Existing methods use benchmark rubrics, condition frozen models on rubric feedback, or co\-train policies with rubric generators or evaluators through outcome supervision, adversarial updates, temporal contrasts, meta\-verification, or response\-set conditioning\(Rezaei et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib30); Shao et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib32); Sheng et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib34); Ding et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib5); Li et al\.[2026b](https://arxiv.org/html/2607.26873#bib.bib16); Guan et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib7); Wang et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib36)\)\. Other self\-evolution methods generate tasks, hints, or curriculum variants from pre\-training text and source documents\(Huang et al\.[2026a](https://arxiv.org/html/2607.26873#bib.bib9); Kwan et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib13); Huang et al\.[2026b](https://arxiv.org/html/2607.26873#bib.bib10); Yang et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib41)\)\. These methods operate under broader post\-training budgets\. Fixed\-set TTRL must instead build rewards from the original prompts and self\-generated responses, without external data, official rubrics, stronger judges, or task generation\. This requires a judge that preserves graded confidence and a query\-specific rubric that evolves as the policy changes\.

We address this problem with SERPO, which co\-evolves query\-local G\-N\-B response archives, query\-specific rubrics, and shared actor parameters\. Figure[1](https://arxiv.org/html/2607.26873#S1.F1)contrasts this criterion\-verification loop with answer voting\. For each prompt, SERPO stores an ordered G\-N\-B response archive and derives atomic criteria from contrasts among these responses\. To our knowledge, SERPO is the first to combine post\-reasoning Boolean verdict probabilities with evolving, query\-specific rubrics for policy optimization; concurrent LLM\-as\-a\-Verifier independently studies a related fixed\-criterion interface\(Kwok et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib14)\)\. SERPO retains criteria with response\-level variance and ordered G\-N\-B separation, consolidates duplicates, and removes persistently weak criteria\. The resulting rubric supplies group\-relative rewards for actor updates\. New actor responses then refresh the archives and rubric, closing the response–rubric–policy loop\.

Across two model configurations, two in\-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20\.63 and 20\.31 points and raises the six\-benchmark macro\-average by up to 8\.06 points\. The evolved policies also transfer to unseen benchmarks and continue improving under sequential cross\-benchmark evolution\.

Contributions\.

- •We formulate TTRL for open\-ended generation under a fixed, label\-free information budget and introduce judge\-based response voting and a stronger claim\-consensus objective\.
- •We introduce SERPO, a closed\-loop framework that co\-evolves rolling G\-N\-B response archives, query\-specific rubrics, and shared actor parameters\.
- •We introduce a probabilistic reward interface for evolving rubrics, aggregating criterion\-satisfaction probabilities derived from post\-reasoning Pass/Fail distributions\.
- •Experiments across two model configurations and six benchmarks establish strong in\-domain gains, OOD transfer, and continued cross\-benchmark evolution\.

## 2Related Work

##### Test\-Time Reinforcement Learning\.

TTRL turns unlabeled test prompts into online reinforcement\-learning data by sampling solutions and using the majority answer as a pseudo\-label, following self\-consistency aggregation\(Wang et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib37); Zuo et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib46)\)\. Self\-reflection, entropy minimization, and extinction\-window mitigation improve pseudo\-label selection or confidence\(Wu et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib39); Agarwal et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib1); Lin et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib18)\), but still require answer extraction and task\-specific equivalence\. SERPO instead constructs rewards through query\-specific criterion verification, which applies when open\-ended responses have no canonical answer\.

##### Rubric Rewards and Policy–Rubric Co\-Evolution\.

Rubrics turn open\-ended criteria into rewards for self\-distillation or policy training\(Gunjal et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib8); Li et al\.[2026a](https://arxiv.org/html/2607.26873#bib.bib15); Whitehouse et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib38); Chen et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib4); Xie et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib40); Rezaei et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib30); Fang et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib6)\)\. RLCER evolves CoT rubrics around outcome\-centric RLVR; EvoRubrics adversarially trains a rubric generator; EvoLM alternates rubric and policy training using temporal checkpoint contrasts; EvoRubric shares one policy across roles with meta\-verification and rubric memory; and DynamicRubric co\-trains a response\-set\-conditioned evaluator\(Sheng et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib34); Ding et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib5); Li et al\.[2026b](https://arxiv.org/html/2607.26873#bib.bib16); Guan et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib7); Wang et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib36)\)\. These frameworks operate under broader post\-training budgets that can include outcome supervision, curated rubrics, stronger judges, or trainable rubric and evaluator modules; related systems further use external corpora or generated tasks\(Huang et al\.[2026a](https://arxiv.org/html/2607.26873#bib.bib9); Kwan et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib13); Huang et al\.[2026b](https://arxiv.org/html/2607.26873#bib.bib10)\)\. SERPO instead keeps both evaluator roles frozen and learns only from the fixed prompts and self\-generated responses\. Token distributions provide continuous evaluation or verification signals\(Liu et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib20); Kwok et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib14)\); SERPO couples them to rolling G\-N\-B evidence that evolves query\-local criteria as the policy improves\.

##### Contextual Self\-Evolution\.

Other methods improve a frozen model through test\-time sampling or search\(Wang et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib37); Yao et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib42); Bay and Yearick[2026](https://arxiv.org/html/2607.26873#bib.bib3)\), iterative self\-feedback\(Madaan et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib22); Shinn et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib35)\), or accumulated external state\. ACE stores reusable context, RGSD supplies teacher rubrics, ARBOR maintains a rubric buffer, TTCS creates curriculum variants, and recursive agents select improved configurations across rounds\(Zhang et al\.[2025b](https://arxiv.org/html/2607.26873#bib.bib45); Rezaei et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib30); Liu et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib21); Yang et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib41); Nguyen, Nguyen, and Vuong[2026](https://arxiv.org/html/2607.26873#bib.bib24)\)\. Their progress resides mainly in context, memory, or external rubric pools and may depend on auxiliary task construction\. SERPO retains evolving response and rubric states but converts their feedback into persistent actor updates on the fixed test set\.

## 3Preliminaries

### Test\-Time Reinforcement Learning

We consider fixed\-set transductive TTRL\. Let𝒟adapt=\{xn\}n=1N\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}=\\\{x\_\{n\}\\\}\_\{n=1\}^\{N\}denote the unlabeled test prompts observed before adaptation\. During adaptation, a method may use only these prompts, its sampled responses, and derived statistics; it receives no references, external task rewards, human or external evaluator feedback, auxiliary corpus, or grader\-guided prompt selection\. Auxiliary models may be queried only in a frozen, evaluation\-blind manner: they are not updated during adaptation and cannot access evaluation resources\. After adaptation, a hidden grader evaluates fresh responses on𝒟adapt\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}for reporting only, while a disjoint𝒟eval\\mathcal\{D\}\_\{\\mathrm\{eval\}\}measures held\-out transfer\.

At global adaptation steptt, the policy processes a minibatch𝒳t⊂𝒟adapt\\mathcal\{X\}\_\{t\}\\subset\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}\. For each promptx∈𝒳tx\\in\\mathcal\{X\}\_\{t\}, the current policyπθt\\pi\_\{\\theta\_\{t\}\}samples a group ofGGresponses,

𝒪t\(x\)=\{ot,i\}i=1G,ot,i∼πθt\(⋅∣x\)\.\\mathcal\{O\}\_\{t\}\(x\)=\\\{o\_\{t,i\}\\\}\_\{i=1\}^\{G\},\\qquad o\_\{t,i\}\\sim\\pi\_\{\\theta\_\{t\}\}\(\\cdot\\mid x\)\.Here𝒪t​\(x\)\\mathcal\{O\}\_\{t\}\(x\)is the sampled rollout group, andot,io\_\{t,i\}is itsii\-th response\. Because the true qualityr⋆​\(x,o\)r^\{\\star\}\(x,o\)is unavailable, the method optimizes a pseudo\-rewardr^t​\(x,o\)\\hat\{r\}\_\{t\}\(x,o\)constructed from permitted information:

maxθ⁡𝔼x∼𝒟adapt,o∼πθ\(⋅∣x\)​\[r^t​\(x,o\)\]\.\\max\_\{\\theta\}\\ \\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{adapt\}\},\\,o\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\hat\{r\}\_\{t\}\(x,o\)\\right\]\.\(1\)The reward rule may be refreshed between updates using accumulated test\-time evidence, but is fixed within each actor update\.

Answer\-level TTRL instantiatesr^t\\hat\{r\}\_\{t\}by mapping each response to a comparable answer\. LetExt\\mathrm\{Ext\}be an answer extractor andSel\\mathrm\{Sel\}a pseudo\-label selector, such as majority voting over extracted answers\(Wang et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib37); Zuo et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib46); Wu et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib39)\)\. Givenai=Ext​\(ot,i\)a\_\{i\}=\\mathrm\{Ext\}\(o\_\{t,i\}\), it selectsa^=Sel​\(\{ai\}i=1G\)\\hat\{a\}=\\mathrm\{Sel\}\(\\\{a\_\{i\}\\\}\_\{i=1\}^\{G\}\)and induces

r^ans​\(x,ot,i\):=𝕀​\[Ext​\(ot,i\)≡a^\],\\hat\{r\}\_\{\\mathrm\{ans\}\}\(x,o\_\{t,i\}\):=\\mathbb\{I\}\\\!\\left\[\\mathrm\{Ext\}\(o\_\{t,i\}\)\\equiv\\hat\{a\}\\right\],\(2\)where≡\\equivis task\-specific equivalence\. This is well defined for categorical or normalized answers, but open\-ended responses can share a coarse answer while differing in factuality, completeness, or safety\. We retain Eq\. \([1](https://arxiv.org/html/2607.26873#S3.E1)\) while replacing answer agreement with criterion verification\.

### Rubric Criteria for Open\-Ended Responses

A rubric decomposes open\-ended quality into atomic natural\-language criteria\. We writeℛ=\{\(cm,ρm,wm\)\}m=1M\\mathcal\{R\}=\\\{\(c\_\{m\},\\rho\_\{m\},w\_\{m\}\)\\\}\_\{m=1\}^\{M\}, wherecmc\_\{m\}is a criterion,ρm∈\{\+1,−1\}\\rho\_\{m\}\\in\\\{\+1,\-1\\\}is its polarity, andwm≥0w\_\{m\}\\geq 0is its weight\. A judge provides a criterion\-satisfaction probabilityqJ​\(x,o,cm\)∈\[0,1\]q\_\{J\}\(x,o,c\_\{m\}\)\\in\[0,1\]for responseooand criterioncmc\_\{m\}; positive criteria reward desirable properties, while negative criteria mark failures\. Section[4](https://arxiv.org/html/2607.26873#S4)shows how SERPO orients and calibrates these probabilities, then aggregates the resulting scores into the scalar reward used by GRPO\.

### Group\-Relative Policy Optimization

SERPO uses GRPO\(Shao et al\.[2024](https://arxiv.org/html/2607.26873#bib.bib33)\), a PPO\-style group\-relative optimizer\(Schulman et al\.[2017](https://arxiv.org/html/2607.26873#bib.bib31)\), as the actor optimizer; only the reward source changes\. For a rollout group on promptxx, GRPO converts scalar rewards\{ri\}i=1G\\\{r\_\{i\}\\\}\_\{i=1\}^\{G\}into critic\-free advantages,A^i=\(ri−r¯\)/\(σr\+εA\)\\hat\{A\}\_\{i\}=\(r\_\{i\}\-\\bar\{r\}\)/\(\\sigma\_\{r\}\+\\varepsilon\_\{A\}\), and updates the actor with the standard clipped objective and KL penalty\. Verifiable tasks may use Eq\. \([2](https://arxiv.org/html/2607.26873#S3.E2)\); SERPO suppliesrir\_\{i\}from the evolving rubric reward in Section[4](https://arxiv.org/html/2607.26873#S4.SSx4)\.

## 4Method: SERPO

SERPO is a closed\-loop TTRL framework for open\-ended prompts\. For each prompt, SERPO maintains three pieces of test\-time state: rolling G\-N\-B response archives, a query\-specific rubric pool, and a shared actor updated across prompts\. At encountertt, letℰt​\(x\)=\(𝒜tG​\(x\),𝒜tN​\(x\),𝒜tB​\(x\)\)\\mathcal\{E\}\_\{t\}\(x\)=\(\\mathcal\{A\}\_\{t\}^\{G\}\(x\),\\mathcal\{A\}\_\{t\}^\{N\}\(x\),\\mathcal\{A\}\_\{t\}^\{B\}\(x\)\)andℛt​\(x\)=\{\(cm,ρm,wm,t\)\}m∈ℳt​\(x\)\\mathcal\{R\}\_\{t\}\(x\)=\\\{\(c\_\{m\},\\rho\_\{m\},w\_\{m,t\}\)\\\}\_\{m\\in\\mathcal\{M\}\_\{t\}\(x\)\}, whereℳt​\(x\)\\mathcal\{M\}\_\{t\}\(x\)indexes the currently active criteria for promptxx; we omitxxfrom prompt\-local archive and criterion\-index sets when unambiguous\. The actor, rubric generator, and judge all start from the deployed model; only the actor is adapted, while fixed initial\-weight copies generate criteria and judge criterion satisfaction\. Each encounter samples responses, computes provisional archive scores, updates G\-N\-B archives, proposes and merges criteria when scheduled, refreshes criterion utilities and the active rubric, and updates the actor with final rewards\. Figure[2](https://arxiv.org/html/2607.26873#S4.F2)summarizes the loop; the appendix provides complete pseudocode and implementation details\.

![Refer to caption](https://arxiv.org/html/2607.26873v1/x2.png)Figure 2:Overview of SERPO\.Hereqqis the input query, whileoio\_\{i\}andrir\_\{i\}are theii\-th policy rollout and its scalar reward\.ℰt\\mathcal\{E\}\_\{t\},ℛt\\mathcal\{R\}\_\{t\}, andθt\\theta\_\{t\}denote the query\-local G\-N\-B archive state, query\-specific rubric state, and shared actor parameters at encountertt, respectively\. Criterion utility combines response\-score variancevmv\_\{m\}and G\-N\-B order agreementama\_\{m\};ΔR\\Delta\_\{R\}denotes the low\-utility elimination region\. Snowflakes mark fixed initial\-weight copies used as the rubric generator and judge\.### Criterion Scoring and Reward Interface

Under the fixed\-set budget, each judge call must preserve reward resolution\. Hard verdicts create ties and shrink GRPO advantages, so SERPO converts post\-reasoning Boolean verdict\-token probabilities into criterion\-satisfaction probabilities\. IfℓT\(m\)\\ell^\{\(m\)\}\_\{\\mathrm\{T\}\}andℓF\(m\)\\ell^\{\(m\)\}\_\{\\mathrm\{F\}\}are the log probabilities assigned to the positive and negative verdicts, define

qJ​\(x,o,cm\):=exp⁡ℓT\(m\)exp⁡ℓT\(m\)\+exp⁡ℓF\(m\)\.q\_\{J\}\(x,o,c\_\{m\}\):=\\frac\{\\exp\\ell^\{\(m\)\}\_\{\\mathrm\{T\}\}\}\{\\exp\\ell^\{\(m\)\}\_\{\\mathrm\{T\}\}\+\\exp\\ell^\{\(m\)\}\_\{\\mathrm\{F\}\}\}\.\(3\)HereqJ​\(x,o,cm\)q\_\{J\}\(x,o,c\_\{m\}\)is the criterion\-satisfaction probability for responseooon criterioncmc\_\{m\}\. This\[0,1\]\[0,1\]probability distinguishes confident from borderline satisfaction\. SERPO converts positive and negative criteria into an oriented criterion score,

zm,t​\(o\)=\{qJ​\(x,o,cm\),ρm=\+1,1−qJ​\(x,o,cm\),ρm=−1\.z\_\{m,t\}\(o\)=\\begin\{cases\}q\_\{J\}\(x,o,c\_\{m\}\),&\\rho\_\{m\}=\+1,\\\\ 1\-q\_\{J\}\(x,o,c\_\{m\}\),&\\rho\_\{m\}=\-1\.\\end\{cases\}\(4\)Herezm,t​\(o\)z\_\{m,t\}\(o\)is the polarity\-oriented criterion score at encountertt\. Largerzm,t​\(o\)z\_\{m,t\}\(o\)always indicates better behavior\. SERPO caches each judged tuple and reuses it for archive ordering, utility estimation, calibration, and reward construction; GRPO receives only the final scalar reward computed after the rubric update in Section[4](https://arxiv.org/html/2607.26873#S4.SSx4)\.

### G\-N\-B Response Evolution

For a new prompt, the fixed rubric generator first creates a nonempty initial set of query\-specific criteria, from which SERPO initializes an equal\-weight rubric\. At each later encounter, the actor samples a rollout group and the judge assigns provisional archive scores for ordering,

rt,iarc\\displaystyle r^\{\\mathrm\{arc\}\}\_\{t,i\}=1Ztarc​∑m∈ℳt​\(x\)wm,t​zm,t​\(ot,i\),\\displaystyle=\\frac\{1\}\{Z^\{\\mathrm\{arc\}\}\_\{t\}\}\\sum\_\{m\\in\\mathcal\{M\}\_\{t\}\(x\)\}w\_\{m,t\}z\_\{m,t\}\(o\_\{t,i\}\),\(5\)Ztarc\\displaystyle Z^\{\\mathrm\{arc\}\}\_\{t\}=∑m∈ℳt​\(x\)wm,t\.\\displaystyle=\\sum\_\{m\\in\\mathcal\{M\}\_\{t\}\(x\)\}w\_\{m,t\}\.Here, the superscriptarc\\mathrm\{arc\}marks this as an archive\-ordering score rather than the GRPO reward, andZtarcZ^\{\\mathrm\{arc\}\}\_\{t\}normalizes the active rubric weights\. SERPO stores the most contrastive ordered triple rather than only the absolute best and worst samples\. Let\[G\]=\{1,…,G\}\[G\]=\\\{1,\\ldots,G\\\}, and letℐt\\mathcal\{I\}\_\{t\}collect distinct ordered triples of rollout indices whose archive scores decrease from candidate Good to Normal to Bad:ℐt=\{\(i,j,k\)∈\[G\]3:i,j,k​distinct,rt,iarc≥rt,jarc≥rt,karc\}\\mathcal\{I\}\_\{t\}=\\\{\(i,j,k\)\\in\[G\]^\{3\}:i,j,k\\ \\text\{distinct\},\\ r^\{\\mathrm\{arc\}\}\_\{t,i\}\\geq r^\{\\mathrm\{arc\}\}\_\{t,j\}\\geq r^\{\\mathrm\{arc\}\}\_\{t,k\}\\\}\. SERPO selects

Dt​\(i,j,k\)\\displaystyle D\_\{t\}\(i,j,k\):=\(rt,iarc−rt,jarc\)​\(rt,jarc−rt,karc\)​\(rt,iarc−rt,karc\),\\displaystyle=\(r^\{\\mathrm\{arc\}\}\_\{t,i\}\-r^\{\\mathrm\{arc\}\}\_\{t,j\}\)\(r^\{\\mathrm\{arc\}\}\_\{t,j\}\-r^\{\\mathrm\{arc\}\}\_\{t,k\}\)\(r^\{\\mathrm\{arc\}\}\_\{t,i\}\-r^\{\\mathrm\{arc\}\}\_\{t,k\}\),\(6\)\(itG,itN,itB\)\\displaystyle\(i\_\{t\}^\{G\},i\_\{t\}^\{N\},i\_\{t\}^\{B\}\)=arg⁡max\(i,j,k\)∈ℐt⁡Dt​\(i,j,k\)\.\\displaystyle=\\arg\\max\_\{\(i,j,k\)\\in\\mathcal\{I\}\_\{t\}\}D\_\{t\}\(i,j,k\)\.HereDt​\(i,j,k\)D\_\{t\}\(i,j,k\)is the separation score of a candidate ordered triple, and\(itG,itN,itB\)\(i\_\{t\}^\{G\},i\_\{t\}^\{N\},i\_\{t\}^\{B\}\)are the selected Good, Normal, and Bad rollout indices\. If the maximumDtD\_\{t\}is zero, the archive is left unchanged\. Otherwise, the selected responses are appended to bounded FIFO*Good*,*Normal*, and*Bad*archives, producingℰt\+1​\(x\)\\mathcal\{E\}\_\{t\+1\}\(x\)\. This keeps query\-local evidence both ordered and separated for rubric refresh\.

### Rubric Evolution

When a prompt reaches its refresh interval, the rubric generator receives the prompt, current rubricℛt​\(x\)\\mathcal\{R\}\_\{t\}\(x\), and updated archivesℰt\+1​\(x\)\\mathcal\{E\}\_\{t\+1\}\(x\), then proposes atomic criteria explaining stronger–weaker contrasts\. SERPO merges candidates with the existing pool: duplicate requirements map to one canonical statement and stable identifier, related but distinct rules remain separate, retained criteria keep their counters, and new criteria start from zero\.

Letℳ~t\+1​\(x\)\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)denote the resulting criterion pool after carrying over existing criteria and applying any scheduled merge\. A useful criterion should separate updated archives in the G\-N\-B direction\. After each encounter and any scheduled merge, SERPO scores only unseen archive–criterion pairs and converts them to Eq\. \([4](https://arxiv.org/html/2607.26873#S4.E4)\)\. Letℋt\+1​\(x\)=𝒜t\+1G​\(x\)∪𝒜t\+1N​\(x\)∪𝒜t\+1B​\(x\)\\mathcal\{H\}\_\{t\+1\}\(x\)=\\mathcal\{A\}\_\{t\+1\}^\{G\}\(x\)\\cup\\mathcal\{A\}\_\{t\+1\}^\{N\}\(x\)\\cup\\mathcal\{A\}\_\{t\+1\}^\{B\}\(x\)denote the updated archive entries for the current prompt\. For eachm∈ℳ~t\+1​\(x\)m\\in\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\), SERPO assigns a discrimination utility

dm,t\+1\\displaystyle d\_\{m,t\+1\}=vm,t\+1​am,t\+1,\\displaystyle=v\_\{m,t\+1\}a\_\{m,t\+1\},\(7\)vm,t\+1\\displaystyle v\_\{m,t\+1\}=4​Varo∈ℋt\+1​\(x\)​\[zm,t\+1​\(o\)\],\\displaystyle=4\\,\\mathrm\{Var\}\_\{o\\in\\mathcal\{H\}\_\{t\+1\}\(x\)\}\\left\[z\_\{m,t\+1\}\(o\)\\right\],am,t\+1\\displaystyle a\_\{m,t\+1\}=\[Cm,t\+1−Dm,t\+1Pm,t\+1\]\+\.\\displaystyle=\\left\[\\frac\{C\_\{m,t\+1\}\-D\_\{m,t\+1\}\}\{P\_\{m,t\+1\}\}\\right\]\_\{\+\}\.Herevm,t\+1v\_\{m,t\+1\}measures response\-level score variance,am,t\+1a\_\{m,t\+1\}measures G\-N\-B order agreement, and\[⋅\]\+=max⁡\(⋅,0\)\[\\cdot\]\_\{\+\}=\\max\(\\cdot,0\); the factor44normalizes the maximum variance of a\[0,1\]\[0,1\]score to one\. The termsCm,t\+1C\_\{m,t\+1\}andDm,t\+1D\_\{m,t\+1\}count concordant and discordant cross\-bucket pairs over\(G,N\)\(G,N\),\(G,B\)\(G,B\), and\(N,B\)\(N,B\)under marginτ\\tau, andPm,t\+1P\_\{m,t\+1\}is the comparison count\. A criterion receives high utility only when it varies over archives and assigns higher oriented scores to stronger buckets; constant, tied, or order\-reversing criteria are ranked for possible removal\.

Recoverable elimination turns this ranking into a stable rubric update\. If\|ℳ~t\+1​\(x\)\|≥4\|\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)\|\\geq 4, the bottomKt\+1=max⁡\{1,⌊ζ​\|ℳ~t\+1​\(x\)\|⌋\}K\_\{t\+1\}=\\max\\\{1,\\lfloor\\zeta\|\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)\|\\rfloor\\\}criteria enter the elimination regionΔR\\Delta\_\{R\}in Figure[2](https://arxiv.org/html/2607.26873#S4.F2), whereζ\\zetais the elimination fraction\. If the pool has fewer than four criteria, the elimination region is empty\. A criterion’s counter increments only while it remains in this region, resets once it leaves, and triggers deletion after three consecutive at\-risk rounds\. Survivors formℳt\+1​\(x\)\\mathcal\{M\}\_\{t\+1\}\(x\)\.

### Policy Evolution

Unlike the provisional archive\-ordering score in Eq\. \([5](https://arxiv.org/html/2607.26873#S4.E5)\), the GRPO reward is computed after the rubric update, using the surviving criteria, utility\-derived weights, and G\-N\-B\-calibrated oriented scores:

rt,i​\(ot,i,ℛt\+1\)\\displaystyle r\_\{t,i\}\\\!\\left\(o\_\{t,i\},\\mathcal\{R\}\_\{t\\\!\+\\\!1\}\\right\)=1Zt\+1rew​∑m∈ℳt\+1wm,t\+1rew​z~m,t\+1​\(ot,i\),\\displaystyle=\\frac\{1\}\{Z^\{\\mathrm\{rew\}\}\_\{t\\\!\+\\\!1\}\}\\sum\_\{m\\in\\mathcal\{M\}\_\{t\\\!\+\\\!1\}\}w^\{\\mathrm\{rew\}\}\_\{m,t\\\!\+\\\!1\}\\tilde\{z\}\_\{m,t\\\!\+\\\!1\}\(o\_\{t,i\}\),\(8\)ηm,t\+1​\(o\)\\displaystyle\\eta\_\{m,t\\\!\+\\\!1\}\(o\)=zm,t\+1​\(o\)−μm,t\+1BΔ​μm,t\+1,\\displaystyle=\\frac\{z\_\{m,t\\\!\+\\\!1\}\(o\)\-\\mu\_\{m,t\\\!\+\\\!1\}^\{B\}\}\{\\Delta\\mu\_\{m,t\\\!\+\\\!1\}\},z~m,t\+1​\(o\)\\displaystyle\\tilde\{z\}\_\{m,t\\\!\+\\\!1\}\(o\)=\{clip\[0,1\]​\(ηm,t\+1​\(o\)\),Δ​μm,t\+1≥δ,zm,t\+1​\(o\),otherwise\.\\displaystyle=Herewm,t\+1rew=max⁡\(ϵu,dm,t\+1\)w^\{\\text\{rew\}\}\_\{m,t\+1\}=\\max\(\\epsilon\_\{u\},d\_\{m,t\+1\}\), andZt\+1rewZ^\{\\text\{rew\}\}\_\{t\+1\}normalizes these weights overℳt\+1\\mathcal\{M\}\_\{t\+1\}\. The calibration maps the Bad\-archive mean toward0and the Good\-archive mean toward11usingΔ​μm,t\+1=μm,t\+1G−μm,t\+1B\\Delta\\mu\_\{m,t\+1\}=\\mu^\{G\}\_\{m,t\+1\}\-\\mu^\{B\}\_\{m,t\+1\}; when this range is too small, SERPO falls back to the static oriented scorezm,t\+1z\_\{m,t\+1\}\. GRPO normalizes rewards within each rollout group\. The actor update changes future rollouts, archives, proposals, utilities, calibration ranges, and rewards, closing the response–rubric–policy loop while keeping the rubric generator and judge fixed\.

## 5Experiments

Our experiments answer five questions\.RQ1: how much does SERPO improve in\-domain open\-ended performance across benchmarks and model configurations?RQ2: do the gains from evolution transfer to held\-out OOD benchmarks?RQ3: how do rubric evolution and policy evolution complement each other?RQ4: which components and evaluator design choices drive SERPO’s gains?RQ5: can SERPO sustain improvement over longer horizons and after switching to a new benchmark?

Table 1:Main 30\-epoch ID/OOD results\.∗Claim\-consensus TTRL;†external judge and official rubric \(not label\-free\)\.### Experimental Setup

##### Benchmarks and evaluation\.

We evaluate two Qwen\-family models,Qwen3\-4B\-Instruct\-2507\(Qwen Team[2025](https://arxiv.org/html/2607.26873#bib.bib26)\)andQwen3\.5\-9B\(Qwen Team[2026a](https://arxiv.org/html/2607.26873#bib.bib27)\), in non\-thinking mode\. For HealthBench\(Arora et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib2)\), we use the official HealthBench\-500 subset and refer to it simply as HealthBench\. Unless otherwise noted, we use the official evaluation or validation split of each dataset\. Each actor evolves independently on HealthBench or ResearchQA\(Li S\. Yifei et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib17)\)\. Without further adaptation, we evaluate the HealthBench\-evolved policy on MedQA\(Jin et al\.[2021](https://arxiv.org/html/2607.26873#bib.bib11)\)and LLMEval\-Med\(Zhang et al\.[2025a](https://arxiv.org/html/2607.26873#bib.bib44)\), and the ResearchQA\-evolved policy on GPQA\-Diamond\(Rein et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib29)\)and RaR\-Science\(Gunjal et al\.[2025](https://arxiv.org/html/2607.26873#bib.bib8)\)\. MedQA and GPQA\-Diamond are multiple\-choice benchmarks scored by direct answer extraction; GPT\-5\.1\(OpenAI[2025](https://arxiv.org/html/2607.26873#bib.bib25)\)evaluates HealthBench, ResearchQA, LLMEval\-Med, and RaR\-Science using their official rubrics\. Except for this privileged reference, no external judge or official rubric is available during adaptation\. In Table[1](https://arxiv.org/html/2607.26873#S5.T1), subscripts denote changes from Base, light\-blue rows extend the method above, bold marks the best label\-free result, and Avg\. is the macro\-average over all six benchmark columns\.

##### Evolution protocol\.

All adapted methods in Table[1](https://arxiv.org/html/2607.26873#S5.T1)use a common 30\-epoch budget, and reported scores are averaged over three evaluation runs\. The sequential experiment evolves the same actor for 30 epochs on HealthBench and then 30 epochs on ResearchQA, evaluating both benchmarks during the second stage\. Figure[5](https://arxiv.org/html/2607.26873#S5.SSx2.SSS0.Px3)alone extends SERPO to 45 epochs to examine long\-horizon evolution\. The appendix reports full decoding, optimization, compute, and reproducibility details, plus mean±\\pmstandard deviation for all main\-table entries\.

##### Baselines\.

*Base*performs no adaptation\. To extend voting\-based TTRL to open\-ended generation, both voting baselines use a frozen copy of the initial actor as a self\-judge and rebuild their pseudo\-labels from the current rollout group at every adaptation step\. We strengthen both baselines withG=16G=16rollouts per prompt, twice the group size used by SERPO\.*Response vote*compresses each response into its principal recommendation and rewards membership in the largest word\-set Jaccard cluster\. Our stronger*claim consensus*baseline extracts atomic claims, constructs a response–claim support matrixAA, and rewards coverage of claims supported by at least a fractionκ\\kappaof the group:

pj=1G​∑ℓ=1GAℓ​j,𝒯κ=\{j:pj≥κ\},riclaim=∑j∈𝒯κAi​j\|𝒯κ\|,κ=0\.5\.\\begin\{gathered\}p\_\{j\}=\\frac\{1\}\{G\}\\sum\_\{\\ell=1\}^\{G\}A\_\{\\ell j\},\\qquad\\mathcal\{T\}\_\{\\kappa\}=\\\{j:p\_\{j\}\\geq\\kappa\\\},\\\\\[\-2\.0pt\] r\_\{i\}^\{\\mathrm\{claim\}\}=\\frac\{\\sum\_\{j\\in\\mathcal\{T\}\_\{\\kappa\}\}A\_\{ij\}\}\{\|\\mathcal\{T\}\_\{\\kappa\}\|\},\\qquad\\kappa=0\.5\.\\end\{gathered\}\(9\)This claim\-level objective follows the atomic decomposition used in long\-form factual evaluation\(Min et al\.[2023](https://arxiv.org/html/2607.26873#bib.bib23)\)and preserves partial agreement without requiring complete\-response equivalence\. The appendix gives the response\-vote equation, prompts, thresholds, canonicalization, and fallback rules\. To our knowledge, these are the first voting\-based TTRL objectives designed for open\-ended generation\.

We instantiate*RGSD \(TTS\)*\(Rezaei et al\.[2026](https://arxiv.org/html/2607.26873#bib.bib30)\), originally a rubric\-guided self\-distillation method, as rubric\-guided test\-time scaling, where the rubric acts as privileged guidance and the actor remains frozen\. We then add our rubric\-evolution mechanism while still freezing the actor, isolating gains without parameter evolution\. SERPO additionally updates the actor with GRPO\. The privileged reference usesQwen/Qwen3\.6\-27B\(Qwen Team[2026b](https://arxiv.org/html/2607.26873#bib.bib28)\)as a strong fixed external judge with the official rubric during adaptation\.

### Main Results

##### Open\-Ended Test\-Time Evolution \(RQ1\)\.

Table[1](https://arxiv.org/html/2607.26873#S5.T1)shows that claim consensus improves response voting by 13–28%, as claim decomposition recovers semantic agreement missed by whole\-response voting\. This finer pseudo\-label is still frequency\-based: highly supported claims can remain incomplete when the rollout group shares the same omission\. SERPO is the strongest label\-free method in every in\-domain setting, improving Base by 21–54% and claim consensus by 5–14%\. The remaining gap isolates the benefit of evolving quality criteria over agreement alone\. Without external supervision, Qwen3\-4B recovers roughly three quarters of the in\-domain improvement achieved by the external\-judge and official\-rubric reference\.

##### OOD Transfer \(RQ2\)\.

All eight SERPO OOD results improve over Base, by up to 11\.4%\. Across both model configurations, HealthBench evolution transfers to the medical OOD benchmarks, and ResearchQA evolution transfers to the science OOD benchmarks\. SERPO also outperforms the external\-judge and official\-rubric reference in all eight OOD settings, although that reference remains strongest in\-domain\. This ID–OOD reversal shows broader transfer from self\-evolved criteria, while privileged supervision favors the benchmark used for evolution\. Voting baselines show mixed OOD changes, confirming that stronger in\-domain consensus alone does not ensure transfer\.

##### Complementary Roles of Rubric and Policy Evolution \(RQ3\)\.

Static rubric\-guided TTS lowers the six\-benchmark average below Base, showing that fixed rubric guidance alone does not create a reliable improvement path\. Rubric self\-evolution recovers the loss without policy updates, while policy evolution improves the rubric\-only variant by up to 55%\. The two stages are complementary: rubric evolution sharpens the reward signal, and policy evolution converts it into cumulative gains\.

\\captionbox

Qwen3\-4B evolution on HealthBench \(HB\)\. The dashed line extrapolates the mean epoch\-40–45 slope toward the privileged ID reference\.\[0\.485\]![[Uncaptioned image]](https://arxiv.org/html/2607.26873v1/x3.png)\\captionboxQwen3\-4B evolution from HealthBench \(HB\) to ResearchQA \(RQA\); labels mark stage maxima\.\[0\.485\]![[Uncaptioned image]](https://arxiv.org/html/2607.26873v1/x4.png)

\\captionbox

Qwen3\-4B ablations on HealthBench \(HB\); response length is normalized by Base\.\[0\.485\]\\captionboxMatched checkpoints and stage bests; bold marks the better method at matched epochs and the better stage within each method\.\[0\.485\]

### Evolution Components and Dynamics

#### Ablation Studies \(RQ4\)

Table[5](https://arxiv.org/html/2607.26873#S5.SSx2.SSS0.Px3)evaluates individual components and evaluator design choices through controlled ablations\. Rel\. Len\. averages response length over the three reported benchmarks and normalizes it by Base\. Removing rubric evolution, freezing the G\-N\-B archives, or training either evaluator role reduces HealthBench by roughly 12–13%\. Hard verdicts lower HealthBench by 6% because binary decisions collapse borderline responses into reward ties, whereas verdict probabilities retain the ordering needed by GRPO\. Freezing the actor causes the largest loss, about 36%\. The comparable losses from disabling rubric evolution and freezing the G\-N\-B archives show that criterion quality depends on both evolving rubric definitions and up\-to\-date response evidence\.

Training either evaluator also hurts, supporting fixed evaluator roles as a stable scoring reference\. The full method attains the best average while producing 4–40% shorter responses than the actor\-updating ablations; without rubric evolution, length reaches2\.29×2\.29\{\\times\}Base with a lower score\. Appendix response\-length results confirm that the complete response–rubric–policy loop, rather than verbosity, drives the gains\.

#### Long\-Horizon and Cross\-Benchmark Evolution \(RQ5\)

Figure[5](https://arxiv.org/html/2607.26873#S5.SSx2.SSS0.Px3)tracks evolution beyond the standard budget\. Response voting returns toward the base policy, rubric\-only evolution plateaus, and claim consensus improves but remains below SERPO\. As rollouts become more homogeneous, frequency\-based rewards lose resolution; refreshed criteria continue to expose quality differences among responses\. The complete response–rubric–policy loop therefore shows the strongest sustained upward trend\.

Extending SERPO to 45 epochs adds a further 3\.6% relative gain, showing that the standard budget does not exhaust the available signal\. The continued rise indicates that the response archives and rubrics still discover useful distinctions after epoch 30\. A descriptive linear projection of the epoch\-40–45 trend reaches the privileged ID reference near epoch 99\.

Figure[5](https://arxiv.org/html/2607.26873#S5.SSx2.SSS0.Px3)and Table[5](https://arxiv.org/html/2607.26873#S5.SSx2.SSS0.Px3)test continuation on ResearchQA\. At matched checkpoints, SERPO exceeds TTRL\-claim on HealthBench and ResearchQA by 15% and 2% at epoch 30, widening to 23% and 19% at epoch 60\. Its stage\-best ResearchQA and HealthBench scores also rise by 19% and 4% after the switch\. TTRL\-claim improves only its transient HealthBench best and ends below its epoch\-30 score\. The agreement between matched checkpoints and stage bests rules out checkpoint selection as the source of the gain\. SERPO learns the new benchmark while further improving HealthBench, extending the OOD result from held\-out transfer to sequential evolution\.

## 6Limitations and Future Work

##### Limitations\.

Our experiments cover two model configurations, one 45\-epoch single\-benchmark run, and one 60\-epoch HealthBench\-to\-ResearchQA sequence\. Rubric generation and judging may inherit biases from the deployed model and increase test\-time computation\.

##### Future work\.

The long\-horizon and sequential results show that test\-time evolution can continue beyond one benchmark and one adaptation stage\. The next step is to extend SERPO to longer, cyclic benchmark streams, where the policy acquires new capabilities while revisiting earlier domains\. Replay, adaptive stopping, and rollback provide mechanisms for managing saturation, evaluator drift, and forgetting, moving TTRL toward continual, label\-free improvement on open\-ended tasks\.

## References

- Agarwal et al\. \(2025\)Agarwal, S\.; Zhang, Z\.; Yuan, L\.; Han, J\.; and Peng, H\. 2025\.The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning\.arXiv:2505\.15134\.
- Arora et al\. \(2025\)Arora, R\. K\.; Wei, J\.; Hicks, R\. S\.; Bowman, P\.; Quiñonero\-Candela, J\.; Tsimpourlas, F\.; Sharman, M\.; Shah, M\.; Vallone, A\.; Beutel, A\.; Heidecke, J\.; and Singhal, K\. 2025\.HealthBench: Evaluating Large Language Models Towards Improved Human Health\.arXiv:2505\.08775\.
- Bay and Yearick \(2026\)Bay, Y\. Y\.; and Yearick, K\. A\. 2026\.When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test\-Time Scaling\.arXiv:2606\.28661\.
- Chen et al\. \(2025\)Chen, X\.; Li, G\.; Wang, Z\.; Jin, B\.; Qian, C\.; Wang, Y\.; Wang, H\.; Zhang, Y\.; Zhang, D\.; Zhang, T\.; Tong, H\.; and Ji, H\. 2025\.RM\-R1: Reward Modeling as Reasoning\.arXiv:2505\.02387\.
- Ding et al\. \(2026\)Ding, H\.; Huang, B\.; Fang, Y\.; Liao, W\.; Li, Z\.; Zhang, J\.; Wu, Z\.; Zhao, J\.; and Wang, Y\. 2026\.EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co\-Evolution for LLM Reinforcement Learning\.arXiv:2606\.23038\.
- Fang et al\. \(2026\)Fang, J\.; Hong, Z\.; Zheng, M\.; Song, M\.; Li, G\.; Jiang, H\.; Zhang, D\.; Guo, H\.; Wang, X\.; and Chua, T\.\-S\. 2026\.Rubric\-based On\-policy Distillation\.arXiv:2605\.07396\.
- Guan et al\. \(2026\)Guan, X\.; Hu, X\.; Huang, S\.; Wang, Z\.; Zhang, B\.; Li, Z\.; Xie, P\.; Liu, B\.; and Cao, J\. 2026\.EvoRubric: Self\-Evolving Rubric\-Driven RL for Open\-Ended Generation\.arXiv:2605\.29847\.
- Gunjal et al\. \(2025\)Gunjal, A\.; Wang, A\.; Lau, E\.; Nath, V\.; He, Y\.; Liu, B\.; and Hendryx, S\. 2025\.Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains\.arXiv:2507\.17746\.
- Huang et al\. \(2026a\)Huang, C\.; Chou, S\.\-Y\.; Zhang, Z\.; and Cardie, C\. 2026a\.Bootstrapping Post\-training Signals for Open\-ended Tasks via Rubric\-based Self\-play on Pre\-training Text\.arXiv:2604\.20051\.
- Huang et al\. \(2026b\)Huang, C\.; Liu, H\.; Zheng, T\.; Dai, R\.; Huang, L\.; Li, J\.; Li, Z\.; Wei, Z\.; Meng, Y\.; and Huang, J\. 2026b\.G\-Zero: Self\-Play for Open\-Ended Generation from Zero Data\.arXiv:2605\.09959\.
- Jin et al\. \(2021\)Jin, D\.; Pan, E\.; Oufattole, N\.; Weng, W\.\-H\.; Fang, H\.; and Szolovits, P\. 2021\.What Disease Does This Patient Have? A Large\-Scale Open Domain Question Answering Dataset from Medical Exams\.*Applied Sciences*, 11\(14\): 6421\.
- Kim et al\. \(2024\)Kim, S\.; Suk, J\.; Longpre, S\.; Lin, B\. Y\.; Shin, J\.; Welleck, S\.; Neubig, G\.; Lee, M\.; Lee, K\.; and Seo, M\. 2024\.Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models\.arXiv:2405\.01535\.
- Kwan et al\. \(2026\)Kwan, W\.\-C\.; Gema, A\. P\.; Leang, J\. O\. J\.; and Minervini, P\. 2026\.SCOPE: Self\-Play via Co\-Evolving Policies for Open\-Ended Tasks\.arXiv:2605\.31433\.
- Kwok et al\. \(2026\)Kwok, J\.; Li, S\.; Atreya, P\.; Liu, Y\.; Jiang, Y\.; Finn, C\.; Pavone, M\.; Stoica, I\.; and Mirhoseini, A\. 2026\.LLM\-as\-a\-Verifier: A General\-Purpose Verification Framework\.arXiv:2607\.05391\.
- Li et al\. \(2026a\)Li, S\.; Zhao, J\.; Wei, M\.; Ren, H\.; Zhou, Y\.; Yang, J\.; Liu, S\.; Zhang, K\.; and Chen, W\. 2026a\.RubricHub: A Comprehensive and Highly Discriminative Rubric Dataset via Automated Coarse\-to\-Fine Generation\.arXiv:2601\.08430\.
- Li et al\. \(2026b\)Li, S\. S\.; Xin, R\.; Xiao, T\.; Wang, Y\.; Shao, R\.; Hao, Z\.; Sclar, M\.; Oh, S\.; Brahman, F\.; Koh, P\. W\.; and Tsvetkov, Y\. 2026b\.EvoLM: Self\-Evolving Language Models through Co\-Evolved Discriminative Rubrics\.arXiv:2605\.03871\.
- Li S\. Yifei et al\. \(2026\)Li S\. Yifei; Chang, A\.; Malaviya, C\.; and Yatskar, M\. 2026\.ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey\-Mined Questions and Rubrics\.*Transactions of the Association for Computational Linguistics*, 14: 1344–1368\.
- Lin et al\. \(2026\)Lin, H\.; Kuai, Z\.; Xue, E\.; and Wang, L\. 2026\.Detecting and Mitigating the Correct\-Answer Extinction Window in Test\-Time Reinforcement Learning with Majority Voting\.arXiv:2605\.19444\.
- Liu et al\. \(2024\)Liu, M\.; Shen, Y\.; Xu, Z\.; Cao, Y\.; Cho, E\.; Kumar, V\.; Ghanadan, R\.; and Huang, L\. 2024\.X\-Eval: Generalizable Multi\-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects\.arXiv:2311\.08788\.
- Liu et al\. \(2023\)Liu, Y\.; Iter, D\.; Xu, Y\.; Wang, S\.; Xu, R\.; and Zhu, C\. 2023\.G\-Eval: NLG Evaluation using GPT\-4 with Better Human Alignment\.arXiv:2303\.16634\.
- Liu et al\. \(2026\)Liu, Z\.; Zhang, L\.; Wang, X\.; Xu, Z\.; Zhan, S\.; Shan, X\.; Huang, W\.; Dai, T\.; Xia, S\.\-T\.; Huo, C\.; and Ding, L\. 2026\.ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents\.arXiv:2606\.03239\.
- Madaan et al\. \(2023\)Madaan, A\.; Tandon, N\.; Gupta, P\.; Hallinan, S\.; Gao, L\.; Wiegreffe, S\.; Alon, U\.; Dziri, N\.; Prabhumoye, S\.; Yang, Y\.; Gupta, S\.; Majumder, B\. P\.; Hermann, K\.; Welleck, S\.; Yazdanbakhsh, A\.; and Clark, P\. 2023\.Self\-Refine: Iterative Refinement with Self\-Feedback\.arXiv:2303\.17651\.
- Min et al\. \(2023\)Min, S\.; Krishna, K\.; Lyu, X\.; Lewis, M\.; Yih, W\.\-t\.; Koh, P\. W\.; Iyyer, M\.; Zettlemoyer, L\.; and Hajishirzi, H\. 2023\.FActScore: Fine\-grained Atomic Evaluation of Factual Precision in Long Form Text Generation\.arXiv:2305\.14251\.
- Nguyen, Nguyen, and Vuong \(2026\)Nguyen, M\.; Nguyen, Q\.; and Vuong, P\. 2026\.Recursive Self\-Evolving Agents via Held\-Out Selection\.arXiv:2606\.28374\.
- OpenAI \(2025\)OpenAI\. 2025\.GPT\-5\.1 Instant and GPT\-5\.1 Thinking System Card Addendum\.https://openai\.com/index/gpt\-5\-system\-card\-addendum\-gpt\-5\-1/\.Accessed: 2026\-07\-29\.
- Qwen Team \(2025\)Qwen Team\. 2025\.Qwen3\-4B\-Instruct\-2507 Model Card\.https://huggingface\.co/Qwen/Qwen3\-4B\-Instruct\-2507\.Accessed: 2026\-07\-29\.
- Qwen Team \(2026a\)Qwen Team\. 2026a\.Qwen3\.5\-9B Model Card\.https://huggingface\.co/Qwen/Qwen3\.5\-9B\.Accessed: 2026\-07\-29\.
- Qwen Team \(2026b\)Qwen Team\. 2026b\.Qwen3\.6\-27B Model Card\.https://huggingface\.co/Qwen/Qwen3\.6\-27B\.Accessed: 2026\-07\-29\.
- Rein et al\. \(2023\)Rein, D\.; Hou, B\. L\.; Stickland, A\. C\.; Petty, J\.; Pang, R\. Y\.; Dirani, J\.; Michael, J\.; and Bowman, S\. R\. 2023\.GPQA: A Graduate\-Level Google\-Proof Q&A Benchmark\.arXiv:2311\.12022\.
- Rezaei et al\. \(2026\)Rezaei, M\.; Mahmoud, A\.; Wang, Z\.; Tyagi, U\.; Gosai, A\.; Dumitru, R\.\-G\.; Sabharwal, A\.; Liu, B\.; and He, Y\. 2026\.Rubric\-Guided Self\-Distillation: Post\-Training Without Rubric Verifiers\.arXiv:2606\.12507\.
- Schulman et al\. \(2017\)Schulman, J\.; Wolski, F\.; Dhariwal, P\.; Radford, A\.; and Klimov, O\. 2017\.Proximal Policy Optimization Algorithms\.arXiv:1707\.06347\.
- Shao et al\. \(2025\)Shao, R\.; Asai, A\.; Shen, S\. Z\.; Ivison, H\.; Kishore, V\.; Zhuo, J\.; Zhao, X\.; Park, M\.; Finlayson, S\. G\.; Sontag, D\.; Murray, T\.; Min, S\.; Dasigi, P\.; Soldaini, L\.; Brahman, F\.; Yih, W\.\-t\.; Wu, T\.; Zettlemoyer, L\.; Kim, Y\.; Hajishirzi, H\.; and Koh, P\. W\. 2025\.DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research\.arXiv:2511\.19399\.
- Shao et al\. \(2024\)Shao, Z\.; Wang, P\.; Zhu, Q\.; Xu, R\.; Song, J\.; Bi, X\.; Zhang, H\.; Zhang, M\.; Li, Y\. K\.; Wu, Y\.; and Guo, D\. 2024\.DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv:2402\.03300\.
- Sheng et al\. \(2026\)Sheng, L\.; Ma, W\.; Hong, R\.; Wang, X\.; Zhang, A\.; and Chua, T\.\-S\. 2026\.Reinforcing Chain\-of\-Thought Reasoning with Self\-Evolving Rubrics\.arXiv:2602\.10885\.
- Shinn et al\. \(2023\)Shinn, N\.; Cassano, F\.; Berman, E\.; Gopinath, A\.; Narasimhan, K\.; and Yao, S\. 2023\.Reflexion: Language Agents with Verbal Reinforcement Learning\.arXiv:2303\.11366\.
- Wang et al\. \(2026\)Wang, B\.; Su, W\.; Tian, H\.; Kong, H\.; Yang, T\.; Yao, T\.; Pan, Q\.; Wu, Y\.; Ai, Q\.; Zhang, M\.; and Liu, Y\. 2026\.Co\-Evolving LLM Evaluators and Policies via DynamicRubric\.arXiv:2607\.20083\.
- Wang et al\. \(2023\)Wang, X\.; Wei, J\.; Schuurmans, D\.; Le, Q\.; Chi, E\.; Narang, S\.; Chowdhery, A\.; and Zhou, D\. 2023\.Self\-Consistency Improves Chain of Thought Reasoning in Language Models\.arXiv:2203\.11171\.
- Whitehouse et al\. \(2025\)Whitehouse, C\.; Wang, T\.; Yu, P\.; Li, X\.; Weston, J\.; Kulikov, I\.; and Saha, S\. 2025\.J1: Incentivizing Thinking in LLM\-as\-a\-Judge via Reinforcement Learning\.arXiv:2505\.10320\.
- Wu et al\. \(2026\)Wu, S\.; Tan, H\.; Zhang, X\.; Xia, B\.; Zhang, S\.; Qi, X\.; Yu, B\.; and Jia, J\. 2026\.Beyond Majority Voting: Self\-Reflective Test\-Time Reinforcement Learning for LLM Reasoning\.In*Proceedings of the 43rd International Conference on Machine Learning*, volume 306 of*Proceedings of Machine Learning Research*\.To appear\.
- Xie et al\. \(2026\)Xie, W\.; Zhao, H\.; Liu, W\.; Zhu, Y\.; Chen, L\.; Ye, M\.; Chen, Z\.; Xu, Y\.; Dong, S\.; Wang, Z\.; Xu, X\.; Shi, K\.; Wu, R\.; Zhang, X\.; Shao, W\.; Chang, B\.; Duan, N\.; and Wang, J\. 2026\.Step\-wise Rubric Rewards for LLM Reasoning\.arXiv:2605\.17291\.
- Yang et al\. \(2026\)Yang, C\.; Xiang, Z\.; Tang, Y\.; Teng, Z\.; Huang, C\.; Long, F\.; Liu, Y\.; and Su, J\. 2026\.TTCS: Test\-Time Curriculum Synthesis for Self\-Evolving\.arXiv:2601\.22628\.
- Yao et al\. \(2023\)Yao, S\.; Yu, D\.; Zhao, J\.; Shafran, I\.; Griffiths, T\. L\.; Cao, Y\.; and Narasimhan, K\. 2023\.Tree of Thoughts: Deliberate Problem Solving with Large Language Models\.arXiv:2305\.10601\.
- Ye et al\. \(2024\)Ye, S\.; Kim, D\.; Kim, S\.; Hwang, H\.; Kim, S\.; Jo, Y\.; Thorne, J\.; Kim, J\.; and Seo, M\. 2024\.FLASK: Fine\-grained Language Model Evaluation based on Alignment Skill Sets\.arXiv:2307\.10928\.
- Zhang et al\. \(2025a\)Zhang, M\.; Shen, Y\.; Li, Z\.; Sha, H\.; Hu, B\.; Wang, Y\.; Huang, C\.; Liu, S\.; Tong, J\.; Jiang, C\.; Chai, M\.; Xi, Z\.; Dou, S\.; Gui, T\.; Zhang, Q\.; and Huang, X\. 2025a\.LLMEval\-Med: A Real\-world Clinical Benchmark for Medical LLMs with Physician Validation\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, 4888–4914\. Association for Computational Linguistics\.
- Zhang et al\. \(2025b\)Zhang, Q\.; Hu, C\.; Upasani, S\.; Ma, B\.; Hong, F\.; Kamanuru, V\.; Rainton, J\.; Wu, C\.; Ji, M\.; Li, H\.; Thakker, U\.; Zou, J\.; and Olukotun, K\. 2025b\.Agentic Context Engineering: Evolving Contexts for Self\-Improving Language Models\.arXiv:2510\.04618\.
- Zuo et al\. \(2025\)Zuo, Y\.; Zhang, K\.; Sheng, L\.; Qu, S\.; Cui, G\.; Zhu, X\.; Li, H\.; Zhang, Y\.; Long, X\.; Hua, E\.; Qi, B\.; Sun, Y\.; Ma, Z\.; Yuan, L\.; Ding, N\.; and Zhou, B\. 2025\.TTRL: Test\-Time Reinforcement Learning\.arXiv:2504\.16084\.

## Appendix AExperimental Details

This section records the information budget, optimization settings, and method hyperparameters omitted from the main paper\.

### A\.1Information Access and Budgets

SERPO follows the fixed\-set transductive TTRL setting in the main paper\. During adaptation, the method may use only the prompts in𝒟adapt\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}, actor rollouts on these prompts, self\-generated rubrics, and statistics derived from those objects\. It receives no references, official rubric scores, hidden evaluator feedback, human feedback, external reward\-model scores, auxiliary documents, or generated new tasks\. GPT\-5\.1 and dataset\-official evaluation rubrics are used only for final reporting\. The marked privileged policy\-evolution reference instead exposes the evolution benchmark’s official rubric toQwen/Qwen3\.6\-27Bduring adaptation\.

The actor backbones are Qwen3\-4B\-Instruct\-2507 and Qwen3\.5\-9B, both run in non\-thinking mode\. The actor is updated by GRPO, while the rubric generator and probabilistic judge are frozen initial\-weight copies; SERPO maintains Good–Normal–Bad \(G\-N\-B\) response archives\. Unless stated otherwise, adapted methods use 30 evolution epochs\. The sequential experiment runs 30 epochs on HealthBench followed by 30 epochs on ResearchQA; the long\-horizon run extends only Qwen3\-4B on HealthBench to 45 epochs\. Table[2](https://arxiv.org/html/2607.26873#A1.T2)gives the shared rollout and optimization settings\.

##### Reproducibility\.

Each run uses one Ubuntu 22\.04 Linux node with eight NVIDIA H100 GPUs \(80 GB each\)\. The implementation uses Python 3\.10\.13,verl0\.9\.0\.dev, PyTorch 2\.10\.0, CUDA 12\.8, vLLM 0\.19\.1, and Transformers 5\.6\.1\. For methods with a frozen self\-judge, six GPUs serve actor training and rollout generation and two serve the initial\-weight judge; otherwise, all eight GPUs serve actor training and rollouts\. A 30\-epoch run takes about 10 hours for Qwen3\-4B and 18 hours for Qwen3\.5\-9B, corresponding to approximately 80 and 144 H100 GPU\-hours\. Reported scores use evaluation seeds 40, 41, and 42; tables report their arithmetic mean, evolution curves show the corresponding unsmoothed means, and Table[4](https://arxiv.org/html/2607.26873#A4.T4)additionally reports standard deviations\. HealthBench uses the official HealthBench\-500 subset rather than an author\-selected sample, and the remaining experiments use official evaluation or validation splits\. Upon publication, we will release the training, evaluation, configuration, and analysis code under a license permitting free research use\.

### A\.2Actor Optimization

Table 2:Actor rollout and optimization configuration\.
### A\.3SERPO Hyperparameters

We set the optimization and reward parameters from the Qwen3 training recipe and standard GRPO practice, then keep the same configuration across benchmarks\.

Table 3:SERPO archive, rubric, judge, and reward hyperparameters\.

## Appendix BComplete SERPO Implementation

Algorithm[1](https://arxiv.org/html/2607.26873#alg1)gives the complete adaptation loop\. The subsections that follow specify the query\-local state, rubric maintenance, utility estimation, and reward construction used in each prompt visit\.

Algorithm 1SERPO adaptation on a fixed unlabeled prompt set\.Input: adaptation prompts𝒟adapt\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}, actorπθ\\pi\_\{\\theta\}, frozen rubric generatorRubGen\\mathrm\{RubGen\}, frozen judgeJJ, stepsTT, rollout group sizeGG, archive widthWW, refresh intervalFF, elimination fractionζ\\zeta, tie marginτ\\tau, calibration thresholdδ\\delta, utility floorϵu\\epsilon\_\{u\}\. State: for each promptxx, active criteriaℛt​\(x\)=\{\(cm,ρm,wm,t\)\}m∈ℳt​\(x\)\\mathcal\{R\}\_\{t\}\(x\)=\\\{\(c\_\{m\},\\rho\_\{m\},w\_\{m,t\}\)\\\}\_\{m\\in\\mathcal\{M\}\_\{t\}\(x\)\}and archivesℰt​\(x\)=\(𝒜tG,𝒜tN,𝒜tB\)\\mathcal\{E\}\_\{t\}\(x\)=\(\\mathcal\{A\}\_\{t\}^\{G\},\\mathcal\{A\}\_\{t\}^\{N\},\\mathcal\{A\}\_\{t\}^\{B\}\)\. Output: adapted actorπθ\\pi\_\{\\theta\}\.

1:for all

x∈𝒟adaptx\\in\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}do

2:Initialize

ℛ0​\(x\)=RubGen​\(x\)\\mathcal\{R\}\_\{0\}\(x\)=\\mathrm\{RubGen\}\(x\)with polarities

ρm∈\{\+1,−1\}\\rho\_\{m\}\\in\\\{\+1,\-1\\\}and empty bounded FIFO archives

ℰ0​\(x\)\\mathcal\{E\}\_\{0\}\(x\)\.

3:endfor

4:for

t=1,…,Tt=1,\\ldots,Tdo

5:Draw minibatch

𝒳t⊂𝒟adapt\\mathcal\{X\}\_\{t\}\\subset\\mathcal\{D\}\_\{\\mathrm\{adapt\}\}\.

6:for all

x∈𝒳tx\\in\\mathcal\{X\}\_\{t\}do

7:Sample rollout group

𝒪t\(x\)=\{ot,i∼πθ\(⋅∣x\)\}i=1G\\mathcal\{O\}\_\{t\}\(x\)=\\\{o\_\{t,i\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\\\}\_\{i=1\}^\{G\}\.

8:for all

ot,i∈𝒪t​\(x\)o\_\{t,i\}\\in\\mathcal\{O\}\_\{t\}\(x\)and active criterion

m∈ℳt​\(x\)m\\in\\mathcal\{M\}\_\{t\}\(x\)do

9:Recover criterion\-satisfaction probability

qm,i=exp⁡ℓT\(m,i\)exp⁡ℓT\(m,i\)\+exp⁡ℓF\(m,i\)q\_\{m,i\}=\\frac\{\\exp\\ell^\{\(m,i\)\}\_\{\\mathrm\{T\}\}\}\{\\exp\\ell^\{\(m,i\)\}\_\{\\mathrm\{T\}\}\+\\exp\\ell^\{\(m,i\)\}\_\{\\mathrm\{F\}\}\}\.

10:Orient the criterion score as

zm,i=qm,iz\_\{m,i\}=q\_\{m,i\}if

ρm=\+1\\rho\_\{m\}=\+1, and

zm,i=1−qm,iz\_\{m,i\}=1\-q\_\{m,i\}if

ρm=−1\\rho\_\{m\}=\-1\.

11:endfor

12:Compute provisional archive score

si=rt,iarc=1Ztarc​∑m∈ℳt​\(x\)wm,t​zm,is\_\{i\}=r^\{\\mathrm\{arc\}\}\_\{t,i\}=\\frac\{1\}\{Z\_\{t\}^\{\\mathrm\{arc\}\}\}\\sum\_\{m\\in\\mathcal\{M\}\_\{t\}\(x\)\}w\_\{m,t\}z\_\{m,i\}\.

13:Let

ℐt=\{\(i,j,k\):i,j,k​distinct,si≥sj≥sk\}\\mathcal\{I\}\_\{t\}=\\\{\(i,j,k\):i,j,k\\ \\mathrm\{distinct\},\\ s\_\{i\}\\geq s\_\{j\}\\geq s\_\{k\}\\\}\.

14:Select

\(itG,itN,itB\)=arg⁡max\(i,j,k\)∈ℐt⁡\(si−sj\)​\(sj−sk\)​\(si−sk\)\(i\_\{t\}^\{G\},i\_\{t\}^\{N\},i\_\{t\}^\{B\}\)=\\arg\\max\_\{\(i,j,k\)\\in\\mathcal\{I\}\_\{t\}\}\(s\_\{i\}\-s\_\{j\}\)\(s\_\{j\}\-s\_\{k\}\)\(s\_\{i\}\-s\_\{k\}\)\.

15:ifthe maximum separation is positivethen

16:Append

ot,itGo\_\{t,i\_\{t\}^\{G\}\},

ot,itNo\_\{t,i\_\{t\}^\{N\}\}, and

ot,itBo\_\{t,i\_\{t\}^\{B\}\}to the length\-

WWGood, Normal, and Bad archives\.

17:endif

18:if

xxreaches a refresh encounterthen

19:Propose candidates

𝒞t=RubGen​\(x,ℛt​\(x\),ℰt\+1​\(x\)\)\\mathcal\{C\}\_\{t\}=\\mathrm\{RubGen\}\(x,\\mathcal\{R\}\_\{t\}\(x\),\\mathcal\{E\}\_\{t\+1\}\(x\)\)and merge them with the current pool to form a temporary pool

ℳ~t\+1​\(x\)\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)\.

20:else

21:Set

ℳ~t\+1​\(x\)=ℳt​\(x\)\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)=\\mathcal\{M\}\_\{t\}\(x\)\.

22:endif

23:Score missing rollout–criterion and archive–criterion pairs needed for rewards and utility estimates\.

24:for allcriterion

m∈ℳ~t\+1​\(x\)m\\in\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)do

25:Let

ℋt\+1​\(x\)=𝒜t\+1G​\(x\)∪𝒜t\+1N​\(x\)∪𝒜t\+1B​\(x\)\\mathcal\{H\}\_\{t\+1\}\(x\)=\\mathcal\{A\}\_\{t\+1\}^\{G\}\(x\)\\cup\\mathcal\{A\}\_\{t\+1\}^\{N\}\(x\)\\cup\\mathcal\{A\}\_\{t\+1\}^\{B\}\(x\)and compute

vm=4​V​a​ro∈ℋt\+1​\(x\)​\[zm​\(o\)\]v\_\{m\}=4\\mathrm\{Var\}\_\{o\\in\\mathcal\{H\}\_\{t\+1\}\(x\)\}\[z\_\{m\}\(o\)\]\.

26:Compute

am=max⁡\{0,\(Cm−Dm\)/\(Cm\+Dm\+Tm\)\}a\_\{m\}=\\max\\\{0,\(C\_\{m\}\-D\_\{m\}\)/\(C\_\{m\}\+D\_\{m\}\+T\_\{m\}\)\\\}from Good–Normal, Good–Bad, and Normal–Bad pair comparisons with margin

τ\\tau\.

27:Set utility

dm=vm​amd\_\{m\}=v\_\{m\}a\_\{m\}\.

28:endfor

29:If the temporary pool has at least four criteria, mark the bottom

max⁡\(1,⌊ζ​\|ℳ~t\+1​\(x\)\|⌋\)\\max\(1,\\lfloor\\zeta\|\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)\|\\rfloor\)by

dmd\_\{m\}as at\-risk; delete only criteria with persistent strikes\. If the temporary pool has fewer than four criteria, keep all criteria active\. Survivors form

ℳt\+1​\(x\)\\mathcal\{M\}\_\{t\+1\}\(x\)\.

30:for all

ot,i∈𝒪t​\(x\)o\_\{t,i\}\\in\\mathcal\{O\}\_\{t\}\(x\)and

m∈ℳt\+1​\(x\)m\\in\\mathcal\{M\}\_\{t\+1\}\(x\)do

31:If

μmG−μmB≥δ\\mu\_\{m\}^\{G\}\-\\mu\_\{m\}^\{B\}\\geq\\delta, set

z~m,i=clip​\(\(zm,i−μmB\)/\(μmG−μmB\),0,1\)\\tilde\{z\}\_\{m,i\}=\\mathrm\{clip\}\\\!\\left\(\(z\_\{m,i\}\-\\mu\_\{m\}^\{B\}\)/\(\\mu\_\{m\}^\{G\}\-\\mu\_\{m\}^\{B\}\),0,1\\right\); otherwise set

z~m,i=zm,i\\tilde\{z\}\_\{m,i\}=z\_\{m,i\}\.

32:endfor

33:Set

wm,t\+1=wmrew=max⁡\{ϵu,dm\}w\_\{m,t\+1\}=w\_\{m\}^\{\\mathrm\{rew\}\}=\\max\\\{\\epsilon\_\{u\},d\_\{m\}\\\}and

Zt\+1rew=∑m∈ℳt\+1​\(x\)wmrewZ\_\{t\+1\}^\{\\mathrm\{rew\}\}=\\sum\_\{m\\in\\mathcal\{M\}\_\{t\+1\}\(x\)\}w\_\{m\}^\{\\mathrm\{rew\}\}\.

34:Compute

rt,i=1Zt\+1rew​∑m∈ℳt\+1​\(x\)wmrew​z~m,ir\_\{t,i\}=\\frac\{1\}\{Z\_\{t\+1\}^\{\\mathrm\{rew\}\}\}\\sum\_\{m\\in\\mathcal\{M\}\_\{t\+1\}\(x\)\}w\_\{m\}^\{\\mathrm\{rew\}\}\\tilde\{z\}\_\{m,i\}\.

35:Normalize the group rewards into GRPO advantages

A^t,i=\(rt,i−r¯t\)/\(σr,t\+εA\)\\hat\{A\}\_\{t,i\}=\(r\_\{t,i\}\-\\bar\{r\}\_\{t\}\)/\(\\sigma\_\{r,t\}\+\\varepsilon\_\{A\}\)\.

36:endfor

37:Update the shared actor

πθ\\pi\_\{\\theta\}with GRPO on all rollout groups from

𝒳t\\mathcal\{X\}\_\{t\}\.

38:endfor

39:return

πθ\\pi\_\{\\theta\}\.

### B\.1Query\-Local State

For each promptxx, SERPO maintains an active rubric poolℛt​\(x\)\\mathcal\{R\}\_\{t\}\(x\)and a rolling G\-N\-B archiveℰt​\(x\)\\mathcal\{E\}\_\{t\}\(x\)\. The initial rubric is generated once from the prompt and fixed task\-format demonstrations, without candidate responses or external supervision, and is cached by prompt identity\. At each later prompt visit, the actor samplesG=8G=8responses\. The frozen judge scores each response against active criteria, producing a provisional score used only for archive ordering\.

Among valid rollout triples, SERPO selects Good, Normal, and Bad responses that maximize

Dt=\(sG−sN\)​\(sN−sB\)​\(sG−sB\),D\_\{t\}=\(s\_\{G\}\-s\_\{N\}\)\(s\_\{N\}\-s\_\{B\}\)\(s\_\{G\}\-s\_\{B\}\),\(10\)wheresG≥sN≥sBs\_\{G\}\\geq s\_\{N\}\\geq s\_\{B\}\. If no nondegenerate triple exists, the archive is left unchanged; any scheduled rubric refresh then uses the existing archive evidence\.

### B\.2Rubric Proposal and Pool Maintenance

When the refresh interval is reached, the frozen rubric generator receives the prompt, the current active rubric pool, and archived G\-N\-B responses\. It proposes at most five atomic positive or negative criteria that explain quality differences not already covered by the active pool\. Proposal and validation are separated: archived contrasts suggest candidates, and the updated archive plus current judge matrix validate whether those candidates still discriminate\.

SERPO exact\-deduplicates criteria by normalized text\. If semantic deduplication is enabled, a conservative duplicate judge is called twice with opposite criterion orders\. A pair is merged only when both calls identify it as duplicate, the two criteria have the same polarity, and their observed binary judgments agree on at least90%90\\%of at least three common graded responses\. This avoids collapsing broad/narrow or complementary criteria\.

### B\.3Utility and Elimination

Letqm​\(o\)q\_\{m\}\(o\)be the criterion\-satisfaction probability for criterionmmon responseoo\. SERPO orients scores so that larger is always better:

zm​\(o\)=\{qm​\(o\),ρm=\+1,1−qm​\(o\),ρm=−1\.z\_\{m\}\(o\)=\\begin\{cases\}q\_\{m\}\(o\),&\\rho\_\{m\}=\+1,\\\\ 1\-q\_\{m\}\(o\),&\\rho\_\{m\}=\-1\.\\end\{cases\}\(11\)Letℋt​\(x\)=𝒜tG​\(x\)∪𝒜tN​\(x\)∪𝒜tB​\(x\)\\mathcal\{H\}\_\{t\}\(x\)=\\mathcal\{A\}\_\{t\}^\{G\}\(x\)\\cup\\mathcal\{A\}\_\{t\}^\{N\}\(x\)\\cup\\mathcal\{A\}\_\{t\}^\{B\}\(x\)denote the archived responses for the current prompt\. Criterion utility combines archive variance with pairwise order agreement\. The variance factor is

vm=4​V​a​ro∈ℋt​\(x\)​\[zm​\(o\)\]\.v\_\{m\}=4\\mathrm\{Var\}\_\{o\\in\\mathcal\{H\}\_\{t\}\(x\)\}\[z\_\{m\}\(o\)\]\.\(12\)Pairwise order agreement compares all Good–Normal, Good–Bad, and Normal–Bad archive pairs with tie marginτ\\tau\. IfCm,Dm,TmC\_\{m\},D\_\{m\},T\_\{m\}are concordant, discordant, and tied counts, then

am=max⁡\{0,Cm−DmCm\+Dm\+Tm\},dm=vm​am\.a\_\{m\}=\\max\\left\\\{0,\\frac\{C\_\{m\}\-D\_\{m\}\}\{C\_\{m\}\+D\_\{m\}\+T\_\{m\}\}\\right\\\},\\qquad d\_\{m\}=v\_\{m\}a\_\{m\}\.\(13\)The factor44maps the largest possible variance of a\[0,1\]\[0,1\]score to one\. The utilitydmd\_\{m\}controls candidate admission, elimination ranking, and reward weighting\.

When the temporary pool has at least four criteria, the bottommax⁡\(1,⌊ζ​\|ℳ~t\+1​\(x\)\|⌋\)\\max\(1,\\lfloor\\zeta\|\\widetilde\{\\mathcal\{M\}\}\_\{t\+1\}\(x\)\|\\rfloor\)criteria by utility enter the elimination region\. A criterion is deleted after three consecutive strikes\. If the pool is full, temporary challengers are admitted only when they pass replay vetoes and exceedmax⁡\{0\.05,uweak\+0\.02\}\\max\\\{0\.05,u\_\{\\mathrm\{weak\}\}\+0\.02\\\}, whereuweaku\_\{\\mathrm\{weak\}\}is the weakest utility inside the elimination region\.

### B\.4Reward Construction

For each surviving criterion, SERPO first obtains its criterion\-satisfaction probability from top\-logprobs of the generatedtrueandfalsetokens\. If Good and Bad archive means are available andμmG−μmB≥δ\\mu\_\{m\}^\{G\}\-\\mu\_\{m\}^\{B\}\\geq\\delta, SERPO applies

z~m​\(o\)=clip​\(zm​\(o\)−μmBμmG−μmB,0,1\);\\tilde\{z\}\_\{m\}\(o\)=\\mathrm\{clip\}\\left\(\\frac\{z\_\{m\}\(o\)\-\\mu\_\{m\}^\{B\}\}\{\\mu\_\{m\}^\{G\}\-\\mu\_\{m\}^\{B\}\},0,1\\right\);\(14\)otherwisez~m​\(o\)=zm​\(o\)\\tilde\{z\}\_\{m\}\(o\)=z\_\{m\}\(o\)\. For every surviving criterion, the next archive weight and current reward weight are identical:wm,t\+1=wmrew=max⁡\{ϵu,dm\}w\_\{m,t\+1\}=w\_\{m\}^\{\\mathrm\{rew\}\}=\\max\\\{\\epsilon\_\{u\},d\_\{m\}\\\}; only the initial rubric uses equal weights\.

The final rubric reward is the weighted average of the surviving calibrated criterion scores:

Zrew\\displaystyle Z^\{\\mathrm\{rew\}\}=∑m∈ℳwmrew,\\displaystyle=\\sum\_\{m\\in\\mathcal\{M\}\}w\_\{m\}^\{\\mathrm\{rew\}\},\(15\)r​\(o\)\\displaystyle r\(o\)=1Zrew​∑m∈ℳwmrew​z~m​\(o\)\.\\displaystyle=\\frac\{1\}\{Z^\{\\mathrm\{rew\}\}\}\\sum\_\{m\\in\\mathcal\{M\}\}w\_\{m\}^\{\\mathrm\{rew\}\}\\tilde\{z\}\_\{m\}\(o\)\.Missing criterion grades are excluded from both the numerator andZrewZ^\{\\mathrm\{rew\}\}\. If the missing\-grade rate exceeds0\.200\.20or no valid criterion remains, the response receives fallback reward0\. GRPO then normalizes rewards within each rollout group\.

## Appendix CBaseline Details

### C\.1Open\-Ended TTRL Voting Baselines

Standard TTRL constructs a pseudo\-label by voting over extracted answers\. For open\-ended responses, we implement two variants using a frozen, non\-thinking copy of the initial actor\. This self\-judge receives no reference answer, official rubric, or evaluation score, and remains fixed while the actor evolves\. To strengthen the comparison, both voting baselines useG=16G=16rollouts per prompt, whereas SERPO usesG=8G=8\. For every prompt and adaptation step, the current actor samples𝒪=\{oi\}i=1G\\mathcal\{O\}=\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}and the voting signal is rebuilt from that group\.

##### Response vote\.

The frozen copy independently summarizes each rollout into its single most important recommendation in at most 15 words,

ai=Sθ0​\(x,oi\)\.a\_\{i\}=S\_\{\\theta\_\{0\}\}\(x,o\_\{i\}\)\.\(16\)We lowercase each summary, remove punctuation, and represent it by its set of wordsWiW\_\{i\}\. Summaries are processed in rollout order and greedily assigned to the most similar existing cluster when

Jac⁡\(ai,aj\)=\|Wi∩Wj\|\|Wi∪Wj\|≥τresp,τresp=0\.5\.\\operatorname\{Jac\}\(a\_\{i\},a\_\{j\}\)=\\frac\{\|W\_\{i\}\\cap W\_\{j\}\|\}\{\|W\_\{i\}\\cup W\_\{j\}\|\}\\geq\\tau\_\{\\mathrm\{resp\}\},\\qquad\\tau\_\{\\mathrm\{resp\}\}=0\.5\.\(17\)Otherwise, the summary starts a new cluster\. If𝒞k⋆\\mathcal\{C\}\_\{k^\{\\star\}\}is the largest resulting cluster, the binary reward is

riresp=𝕀​\[ai∈𝒞k⋆\],k⋆=arg⁡maxk⁡\|𝒞k\|\.r\_\{i\}^\{\\mathrm\{resp\}\}=\\mathbb\{I\}\[a\_\{i\}\\in\\mathcal\{C\}\_\{k^\{\\star\}\}\],\\qquad k^\{\\star\}=\\arg\\max\_\{k\}\|\\mathcal\{C\}\_\{k\}\|\.\(18\)Summary calls use temperature0, at most 64 generated tokens, non\-thinking decoding, concurrency 32, and up to four attempts\. If every summary call in a group fails or returns empty text, all responses receive the fallback reward0\.

##### Claim consensus\.

The strengthened baseline asks the frozen copy to decompose each response into at most 12 short, self\-contained, independently checkable claims, including facts, recommendations, warnings, and assertions\. Compound statements are split, whereas pleasantries and unsupported hedging are omitted\. Claims from allGGresponses are pooled, lowercased, stripped to alphanumeric tokens, and deduplicated by exact normalized form in their original order\. This lexical canonicalization, capped at 40 claims, avoids introducing an additional semantic\-clustering heuristic\.

Let𝒞=\{cj\}j=1M\\mathcal\{C\}=\\\{c\_\{j\}\\\}\_\{j=1\}^\{M\}be the canonical pool\. A second frozen\-model call for each response identifies the claims that the response clearly states or directly implies, producing

Ai​j=𝕀​\[oi​supports​cj\],pj=1G​∑i=1GAi​j\.A\_\{ij\}=\\mathbb\{I\}\[o\_\{i\}\\text\{ supports \}c\_\{j\}\],\\qquad p\_\{j\}=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}A\_\{ij\}\.\(19\)Omitted, contradicted, or only vaguely related claims are not counted\. Define the consensus and low\-support sets as

𝒯κ\\displaystyle\\mathcal\{T\}\_\{\\kappa\}=\{j:pj≥κ\},\\displaystyle=\\\{j:p\_\{j\}\\geq\\kappa\\\},ℒlow\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{low\}\}=\{j:pj≤τlow\},\\displaystyle=\\\{j:p\_\{j\}\\leq\\tau\_\{\\mathrm\{low\}\}\\\},\(20\)κ\\displaystyle\\kappa=0\.5,\\displaystyle=5,τlow\\displaystyle\\tau\_\{\\mathrm\{low\}\}=0\.2\.\\displaystyle=2\.Thus, with the voting baseline’sG=16G=16, a claim enters the consensus set when at least eight rollouts support it\. The implemented reward is

riclaim\\displaystyle r\_\{i\}^\{\\mathrm\{claim\}\}=clip\[0,1\]⁡\(∑j∈𝒯κAi​j−λ​∑j∈ℒlowAi​j\|𝒯κ\|\),\\displaystyle=\\operatorname\{clip\}\_\{\[0,1\]\}\\left\(\\frac\{\\sum\_\{j\\in\\mathcal\{T\}\_\{\\kappa\}\}A\_\{ij\}\-\\lambda\\sum\_\{j\\in\\mathcal\{L\}\_\{\\mathrm\{low\}\}\}A\_\{ij\}\}\{\|\\mathcal\{T\}\_\{\\kappa\}\|\}\\right\),\(21\)λ\\displaystyle\\lambda=0\.\\displaystyle=0\.The configuredλ=0\\lambda=0reduces this expression to unweighted consensus\-claim coverage\. If no claim reachesκ\\kappa, the highest\-support tier becomes the consensus set; if no canonical claim or valid reward is available, the fallback reward is0\. Claim extraction and support checks use temperature0, at most 512 generated tokens, non\-thinking decoding, concurrency 32, and up to four attempts\. The support matrix is reused directly for rewards, requiring approximately2​G2Gfrozen\-model calls per group rather than an additional grading pass\.

### C\.2Rubric\-Guided Test\-Time Scaling

We instantiate RGSD as rubric\-guided test\-time scaling: the rubric guides generation while the actor remains frozen\. The static variant keeps its initial rubric\. The strengthened rubric\-only variant uses SERPO’s rubric evolution, G\-N\-B archive update, utility estimation, and pool maintenance, but still performs no actor update\.

##### Privileged Policy\-Evolution Reference\.

This reference keeps SERPO’s actor, rollout procedure, GRPO optimizer, and 30\-epoch budget, but usesQwen/Qwen3\.6\-27Bas a fixed adaptation\-time judge and fixes the reward specification to the dataset\-official HealthBench or ResearchQA rubric\. It performs no rubric generation or evolution and constructs no G\-N\-B archives; only the actor parameters evolve\. Criterion judging uses the same interface and decoding configuration as Table[3](https://arxiv.org/html/2607.26873#A1.T3)\. GPT\-5\.1 remains the final reporting evaluator\. Because an external judge and official rubrics are exposed during adaptation, this reference is not label\-free\.

## Appendix DEvaluation Protocol and Additional Analyses

### D\.1Evaluation Protocol

We use the official HealthBench\-500 subset and refer to it as HealthBench\. HealthBench tests safety\-sensitive medical advice, while ResearchQA tests long\-form, evidence\-oriented scientific responses; the four held\-out benchmarks measure whether the resulting policies transfer beyond the prompts used for evolution\. Each actor evolves independently on HealthBench or ResearchQA using the official evaluation or validation splits\. Without further adaptation, HealthBench\-evolved policies are evaluated on MedQA and LLMEval\-Med, while ResearchQA\-evolved policies are evaluated on GPQA\-Diamond and RaR\-Science\. MedQA and GPQA\-Diamond use direct answer extraction and report accuracy on a 0–100 scale\. Thegpt\-51\-1113\-globalendpoint scores the other four benchmarks with their official rubrics or protocols, also reported on a 0–100 scale\. The six\-benchmark Avg\. is their unweighted arithmetic mean\. These reporting graders remain hidden during adaptation\.

For GPT\-5\.1 grading, each request usesmax\_completion\_tokens=4096=4096,reasoning\_effort=low, and one completion \(n=1n=1\)\. Temperature and top\-ppare not sent, so the endpoint defaults are used\. HealthBench and ResearchQA place the full official rubric of one example in a single judge request; LLMEval\-Med and RaR\-Science likewise use one judge completion per evaluated response\. Judge outputs are parsed as structured JSON\. HealthBench permits up to five attempts, while the other graders permit four; a length\-truncated output is retried with an adaptively increased token budget capped at 16,384 tokens\.

Table 1 reports epoch 30, Figure 3 through epoch 45, and Table 3 matched epoch\-30/60 checkpoints plus per\-benchmark stage bests; the epoch\-30 row uses the first joint evaluation at the switch\. Reporting scores never affect rewards, prompt selection, early stopping, or checkpoint selection\.

##### Long\-horizon projection\.

Figure 3 preserves each unsmoothed trajectory while aligning its epoch\-30 checkpoint with Table 1\. The dashed line extends SERPO’s mean epoch\-40–45 slope toward the privileged ID reference, producing the descriptive epoch\-99 prediction shown in the figure\.

### D\.2Evaluation Stability

Table[4](https://arxiv.org/html/2607.26873#A4.T4)reports the mean and standard deviation over evaluation runs with seeds 40, 41, and 42\. These runs affect reporting only and are not used for checkpoint selection or hyperparameter tuning\.

Table 4:Main results with evaluation variability\. Benchmark columns report mean±\\pmstandard deviation; Avg\. reports the macro\-average mean\. Bold marks the best label\-free entry;†uses an external judge and official rubric during adaptation\.Across repeated evaluations, SERPO remains the best label\-free method on both in\-domain benchmarks and on the six\-benchmark macro\-average for both actor backbones\. Several OOD columns have smaller margins, so they support the transfer trend rather than a broad claim of uniform dominance\.

### D\.3Response\-Length Analysis

##### Ablation\-level length\.

Figure[3](https://arxiv.org/html/2607.26873#A4.F3)separates aggregate response length from benchmark\-specific endpoints\. Among variants that update the actor, full SERPO is the most concise: its1\.38×1\.38\{\\times\}relative length is 4–40% below the other actor\-updating ablations\. The hard\-verdict variant averages1\.44×1\.44\{\\times\}Base length, with1\.50×1\.50\{\\times\}on HealthBench and1\.34×1\.34\{\\times\}on LLMEval\-Med, yet loses 6% on HealthBench\. Removing rubric evolution produces the longest responses \(2\.29×2\.29\{\\times\}Base\) while scoring below the full method\. The ablation therefore separates effective evolution from unconstrained verbosity\.

![Refer to caption](https://arxiv.org/html/2607.26873v1/x5.png)Figure 3:Response\-length ablations on Qwen3\-4B\. Both panels share the same variant rows\. Left: the three\-benchmark mean normalized by Base\. Right: endpoint lengths on HealthBench and LLMEval\-Med; MedQA uses direct answer extraction\.
##### Cross\-benchmark method comparison\.

The left panel of Figure[4](https://arxiv.org/html/2607.26873#A4.F4)plots the six\-benchmark macro\-average against mean length over the five benchmarks with recorded response lengths; MedQA is excluded because it uses direct answer extraction\. SERPO lies on the label\-free Pareto frontier, using 27% fewer tokens than response voting and 29% fewer than claim consensus while achieving a higher score\. Static RGSD is shorter but substantially less accurate\. The privileged reference gains less than one macro\-average point over SERPO while generating 32% more tokens\. The quality gain therefore cannot be explained by longer responses\.

##### Length dynamics during evolution\.

The right panel of Figure[4](https://arxiv.org/html/2607.26873#A4.F4)reports the raw 30\-epoch trajectories without smoothing\. From the first five to the last five epochs, the two voting baselines increase their mean length by 79–120% and approach22k tokens on both evolution benchmarks\. Over the same windows, SERPO grows by about 36% and ends near1\.31\.3k tokens while attaining the strongest label\-free scores\. Rubric\-only evolution shows similarly moderate length growth but lower quality\. The non\-monotonic trajectories indicate that performance does not track response length directly\.

![Refer to caption](https://arxiv.org/html/2607.26873v1/x6.png)

![Refer to caption](https://arxiv.org/html/2607.26873v1/x7.png)

Figure 4:Response\-length behavior of Qwen3\-4B\. Left: six\-benchmark macro\-average versus mean length over the five benchmarks with recorded response lengths; ellipse axes reflect evaluation variability and are enlarged for visibility\. Right: raw trajectories over 30 evolution epochs on HealthBench and ResearchQA, without smoothing\.

### D\.4Qualitative Audit Protocol

For qualitative audits, we replay stored G\-N\-B visits, active and deleted criteria, criterion utilities, calibrated Good/Bad means, and final reward contributions\. We inspect whether admitted criteria are task\-relevant, separate Good from Bad responses, and agree with the hidden reporting rubric under post\-hoc evaluation\. We also inspect rejected criteria for redundancy, low variance, reversed G\-N\-B ordering, high missing\-grade rate, near\-constant trigger rate, or response\-length correlation\. For medical prompts, the audit additionally covers safety\-critical escalation, contraindication handling, and negative criteria that merely restate missing content\.

## Appendix EQualitative Case Study of Rubric Evolution

This example shows how SERPO converts response\-level differences into a query\-local reward\. We use the HealthBench prompt from Figure 1 of the main paper\. The trace contains only the evolving rubric and G\-N\-B archive; no reference answer or official HealthBench rubric is available during adaptation\.

Prompt and abbreviated G\-N\-B archivePrompt\.“I am in my third trimester and I keep having these headaches\. They will not go away\. I am not sure if it is just normal tension headaches or something more serious\.”Good\.Recommends same\-day obstetric assessment; checks blood pressure and urine protein; lists preeclampsia red flags; treats rest and hydration as interim care\.Normal\.Recommends rest, hydration, and contacting an obstetric clinician if the headache persists, but gives no urgency or concrete workup\.Bad\.Attributes the symptom to stress or hormones, recommends a pain reliever, and delays escalation until the next prenatal visit unless symptoms become severe\.

Representative query\-local rubric evolutionInitial pool\.Broad criteria cover pregnancy\-specific risk, clinician assessment, blood\-pressure checking, red flags, interim self\-care, and false reassurance\.First refresh\.Good and Normal both mention clinician contact, but only Good makes it same\-day and pregnancy\-specific\. SERPO sharpens urgency and adds a negative criterion for routine self\-care despite persistent third\-trimester headache\.Second refresh\.Good mentions blood pressure and urine protein; Normal omits the workup; Bad substitutes symptom management for escalation\. SERPO splits the broad evaluation criterion into judgeable checks and downweights generic explanations that do not preserve the G\-N\-B order\.Later refresh\.Once clinician contact becomes common, the remaining distinction is explicit escalation guidance\. The pool refines red flags into observable symptoms such as visual disturbance, severe swelling, upper abdominal pain, confusion, breathing difficulty, and reduced consciousness\.Stable pool\.The retained criteria reward same\-day assessment, concrete preeclampsia checks, red\-flag escalation, and safe interim care, while penalizing false reassurance\.

Answer consensus can preserve frequent but incomplete advice such as rest and hydration\. SERPO instead retains criteria that separate the ordered archive\. Here, self\-care receives credit only when paired with pregnancy\-specific assessment and escalation, shifting the reward from common wording toward discriminative safety content\.

## Appendix FPrompt Templates

This section gives the prompt templates used in our implementation\. Braced names such as\{query\}and\{response\}denote runtime substitutions\. The boxes preserve the message role used by each frozen generator or judge, with non\-thinking decoding enabled\.

### F\.1Initial Rubric Generation

The initial rubric generator receives the target query and fixed task\-format demonstrations\. These examples specify only the output schema and criterion style; they contain no adaptation\-set prompts, candidate responses, reference answers, official rubrics, or evaluation feedback\. HealthBench uses the following system prompt\.

`Initial rubric generation: HealthBench system prompt ResearchQA replaces the system role with the following task profile while keeping the same JSON schema\. Initial rubric generation: ResearchQA system prompt`

`F\.2 Rubric Evolution The rubric\-evolution call proposes new criteria from the current G\-N\-B archive and active rubric pool\. Rubric evolution: system prompt Rubric evolution: system prompt \(continued\) The G\-N\-B archive user prompt is: Rubric evolution: G\-N\-B archive user prompt F\.3 Probabilistic Judge The probabilistic judge evaluates one response against one criterion\. SERPO asks the judge to identify the supporting evidence first and place the true or false verdict at the end, so the calibrated Boolean probability is recovered from a decision conditioned on the explanation\. Probabilistic judge: evidence\-first prompt The direction prefix is inserted before the criterion text: Probabilistic judge: criterion direction prefixes If JSON parsing, final verdict\-token lookup, or true/false logprob recovery fails, the cell is marked missing rather than converted to a hard score\.`