Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation

# Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation
Source: [https://arxiv.org/html/2608.14684](https://arxiv.org/html/2608.14684)
Dingyao Yu1,2Tong Zhang2Yutao Mou1,2Yunxiao Zhang1,2 Wei Ye1Shikun Zhang1

1Peking University 2Weixin Al, Tencent Inc \{yudingyao,wye,shangsk\}@pku\.edu\.cn

###### Abstract

LLM judges increasingly evaluate responses against fine\-grained rubric checklists\. When a sample requires multiple rubrics, current methods typically assess each in a separate inference call\. Evaluating all rubrics in a single pass is a natural alternative with greater efficiency, but we find that it introducesrubric interference: the verdict on one rubric shifts depending on which other rubrics are co\-present\. In a preliminary study, only one\-third of samples receive fully consistent verdicts when evaluated under rubric sets of varying composition\. We develop a measurement framework that probes interference through four controlled operations: rubric set expansion, subsetting, reordering, and noise injection\. To mitigate interference without external supervision, we proposeSelf\-Anchored Rubric Alignment \(SARA\)\. SARA uses a model’s own single\-rubric judgments as stable anchors and aligns multi\-rubric reasoning with these anchors through on\-policy self\-distillation\. We validate SARA on three datasets \(HealthBench, FLASK, ResearchQA\) and two model families \(Qwen3, Llama\-3\.1\)\. SARA consistently improves evaluation consistency while maintaining agreement with both base models and GPT\-4\.1 as a reference judge\. Furthermore, the learned consistency transfers across datasets, confirming that SARA teaches a general capability rather than fitting dataset\-specific patterns\. Our code is available at[https://anonymous\.4open\.science/r/SARA\-22E2/](https://anonymous.4open.science/r/SARA-22E2/)

Mitigating Rubric Interference in LLM Judges via On\-Policy Self\-Distillation

Dingyao Yu1,2Tong Zhang2Yutao Mou1,2Yunxiao Zhang1,2Wei Ye1Shikun Zhang11Peking University2Weixin Al, Tencent Inc\{yudingyao,wye,shangsk\}@pku\.edu\.cn

## 1Introduction

Large language models now serve as automated evaluators for open\-ended text generation, replacing or supplementing human annotation in both benchmarking and reward modelingZhenget al\.\([2023](https://arxiv.org/html/2608.14684#bib.bib28)\); Kimet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib29)\)\. Early approaches rely on holistic scoring or pairwise comparison\. Holistic scoring collapses multidimensional quality into a single number, sacrificing diagnostic granularityChiang and Lee \([2023](https://arxiv.org/html/2608.14684#bib.bib30)\)\. Pairwise comparison avoids calibration issues but still provides no per\-dimension feedback\. These limitations have driven the field toward rubric\-based evaluation, which decomposes quality into specific, verifiable items for individual assessmentLeeet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib24)\); Kimet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib29)\); Pathaket al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib22)\)\.

In tasks like HealthBenchAroraet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib26)\)and ResearchQAYifeiet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib27)\), a single sample requires evaluation against multiple rubrics\. Two strategies handle such multi\-rubric sets\.Isolation evaluationprocesses each rubric in a separate inference call, producing independent verdicts\.Joint evaluationassesses all rubrics in a single pass, generating a unified structured report\. Current practice defaults to isolationLeeet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib24)\)\. Joint evaluation is attractive for its efficiency and ability to produce coherent cross\-rubric narratives, both desirable when the judge serves as an online reward signal in RLHFGunjalet al\.\([2026](https://arxiv.org/html/2608.14684#bib.bib13)\)\. However, its reliability remains largely unexamined\.

![Refer to caption](https://arxiv.org/html/2608.14684v1/x1.png)Figure 1:Rubric interference in joint evaluation\. The same rubric receives different verdicts depending on the composition and ordering of co\-present rubrics\.For joint evaluation to reliably replace isolation calls, the verdict on each rubric must depend only on the conversation and that rubric\. We find that this invariance does not hold\. Joint evaluation introducesrubric interference: the verdict on a rubric shifts depending on how the rubric set is composed\. Figure[1](https://arxiv.org/html/2608.14684#S1.F1)illustrates one manifestation: simply reordering the rubric list flips a verdict\. Interference also emerges when we expand from isolation to joint mode: comparing each rubric’s isolation verdict against its joint verdict on HealthBench, even Qwen3\-32B achieves only 36% sample\-level exact match\. This reflects systematic interference rather than generation noise: all models achieve≥\\geq0\.98 self\-agreement across repeated evaluations with different random seeds\.

Rubric interference is systematic and reproducible\. It differs from known judge biases such as position, verbosity, and self\-enhancementWanget al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib33)\); Wu and Aji \([2025](https://arxiv.org/html/2608.14684#bib.bib34)\); Panicksseryet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib42)\), which concern how a judge ranks differentresponses\. Rubric interference concerns how the judge handles differentevaluation contextsfor the same response\.

A key observation enables our solution\. While joint evaluation is unreliable, the same model evaluating each rubricin isolationproduces stable judgments that require no external annotation\. The gap between isolation and joint verdicts reveals where interference distorts reasoning, providing a training signal without external supervision\.

We proposeSelf\-Anchored Rubric Alignment \(SARA\), which aligns joint\-mode reasoning with the model’s own isolation judgments through on\-policy self\-distillation\. SARA generates a multi\-rubric evaluation, parses it into per\-rubric segments, and distills from a slowly updated teacher that evaluates each segment in isolation context\. A divergence loss corrects analysis segments where interference distorts reasoning; a preservation loss maintains verdict structure\.

Our contributions are as follows:

1. 1\.We identify rubric interference as a systematic failure mode in multi\-rubric LLM evaluation and develop a measurement framework with four operations to quantify it across binary and scored formats\.
2. 2\.We propose SARA, a self\-distillation method that mitigates rubric interference by anchoring joint\-mode reasoning to the model’s own isolation judgments, requiring no external supervision\.
3. 3\.We validate SARA on three datasets and two model families\. SARA improves consistency while preserving evaluation quality relative to both base models and GPT\-4\.1\.

## 2Related Work

#### LLM\-as\-a\-Judge\.

LLM judges have become standard evaluation infrastructureZhenget al\.\([2023](https://arxiv.org/html/2608.14684#bib.bib28)\); Liet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib31)\)\. Early work showed that GPT\-4 matches inter\-annotator agreement on MT\-Bench, justifying LLM judges at scaleZhenget al\.\([2023](https://arxiv.org/html/2608.14684#bib.bib28)\)\. Chain\-of\-thought prompting further improves correlation with human ratingsLiuet al\.\([2023](https://arxiv.org/html/2608.14684#bib.bib44)\), and task\-specific checklists from real users have moved evaluation toward structured per\-sample criteriaLinet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib45)\)\. More recently, rubric\-based evaluation decomposes quality into fine\-grained, verifiable items and shows improved reproducibility across judge modelsLeeet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib24)\); Kimet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib29)\); Pathaket al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib22)\)\. Benchmarks such as HealthBenchAroraet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib26)\)and ResearchQAYifeiet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib27)\)adopt per\-sample rubric sets as part of the evaluation protocol\. Our work identifies rubric interference as a reliability concern specific to joint multi\-rubric assessment\.

#### Biases and reliability of LLM judges\.

LLM judges exhibit systematic biases including position bias, verbosity bias, and self\-enhancement biasWanget al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib33)\); Wu and Aji \([2025](https://arxiv.org/html/2608.14684#bib.bib34)\); Panicksseryet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib42)\)\. Position bias in particular is stable and model\-specific rather than a sampling artifact, persisting across 12 judge models and over 100K evaluation instancesZhenget al\.\([2023](https://arxiv.org/html/2608.14684#bib.bib28)\)\. Sensitivity to prompt complexity and a general leniency tendency further compromise reliabilityThakuret al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib46)\)\. Even GPT\-4o performs near chance on challenging response pairsTanet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib47)\)\. Calibration strategies have been proposed to address these issuesLiet al\.\([2025](https://arxiv.org/html/2608.14684#bib.bib5),[2026](https://arxiv.org/html/2608.14684#bib.bib9)\)\. All these findings concerninter\-responsebiases—how the judge compares different responses\. Rubric interference is orthogonal: the judgment on a single rubric shifts depending on the co\-evaluated set, with the response fixed\.

#### Prompt sensitivity and evaluation consistency\.

Beyond judge\-specific biases, LLMs exhibit broad prompt sensitivity: formatting changes alone cause up to 76\-point accuracy swingsSclaret al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib48)\), motivating multi\-prompt evaluation as a methodological defaultMizrahiet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib49)\)\. Paraphrase\-based consistency frameworks measure whether equivalently phrased queries elicit the same factual answerElazaret al\.\([2021](https://arxiv.org/html/2608.14684#bib.bib50)\)\. These works study sensitivity of LLMs astask\-solvers\. We study sensitivity of LLMs asjudges: how context changes the label assigned to another model’s output\.

#### On\-policy distillation\.

On\-policy distillation trains a student on self\-sampled trajectories with per\-token teacher guidance, mitigating distribution shiftAgarwalet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib39)\); Guet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib43)\)\. Self\-distillation removes the need for a separate teacher: distilling a model into itself can improve generalizationFurlanelloet al\.\([2018](https://arxiv.org/html/2608.14684#bib.bib36)\)\. In the LLM setting, models have been trained to distinguish their own outputs from referencesChenet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib37)\), to serve as their own reward functionYuanet al\.\([2024](https://arxiv.org/html/2608.14684#bib.bib38)\), and to teach themselves by conditioning on privileged informationZhaoet al\.\([2026](https://arxiv.org/html/2608.14684#bib.bib25)\)\. SARA shares this on\-policy self\-distillation principle but targets a different goal: aligning joint\-mode with isolation\-mode evaluation\. The privileged information is not a stronger model but the absence of interfering rubrics\.

## 3Problem Definition

### 3\.1Joint Rubric Evaluation

We consider a judge modelℳ\\mathcal\{M\}that evaluates a conversationccagainst a set of rubricsR=\{r1,…,rn\}R=\\\{r\_\{1\},\\ldots,r\_\{n\}\\\}\. For each rubric, the judge assigns a verdictviv\_\{i\}from a discrete value set𝒱\\mathcal\{V\}\. We study binary \(𝒱=\{0,1\}\\mathcal\{V\}=\\\{0,1\\\}\) and scored \(e\.g\.𝒱=\{1,…,5\}\\mathcal\{V\}=\\\{1,\\ldots,5\\\}\) evaluation; the framework extends to any ordinal scale\.

Inisolation evaluation, the judge processes each rubric in a separate inference call, producing verdictviisov\_\{i\}^\{\\text\{iso\}\}fromℳ​\(c,\{ri\}\)\\mathcal\{M\}\(c,\\\{r\_\{i\}\\\}\)\. Injoint evaluation, the judge assesses all rubrics in a single pass, producing verdictvijntv\_\{i\}^\{\\text\{jnt\}\}fromℳ​\(c,R\)​\[ri\]\\mathcal\{M\}\(c,R\)\[r\_\{i\}\]\. Joint evaluation is more efficient, but the verdictvijntv\_\{i\}^\{\\text\{jnt\}\}may depend on the other rubrics inRR, introducing rubric interference\.

An ideal judge produces the same verdict for rubricrrregardless of which other rubrics accompany it:

ℳ​\(c,R\)​\[r\]=ℳ​\(c,R′\)​\[r\]∀R,R′∋r\\mathcal\{M\}\(c,R\)\[r\]=\\mathcal\{M\}\(c,R^\{\\prime\}\)\[r\]\\quad\\forall\\;R,R^\{\\prime\}\\ni r\(1\)
The rest of this section describes how we test whether real judges satisfy it\.

### 3\.2Testing for Rubric Interference

We test consistency by changing the rubric set in controlled ways and checking whether verdicts on shared rubrics remain stable\. Table[1](https://arxiv.org/html/2608.14684#S3.T1)summarizes four operations, each targeting a different interference mechanism\.

OperationWhat changesWhat we compareExpansionSet size:1→n1\\to nviisov\_\{i\}^\{\\text\{iso\}\}vsvijntv\_\{i\}^\{\\text\{jnt\}\}SubsettingSet size:m→nm\\to nviv\_\{i\}in small vs large setReorderingRubric orderviv\_\{i\}across permutationsNoiseAdd unrelated rubricsviv\_\{i\}before vs after noiseTable 1:Four operations for testing rubric interference\. Each changes the rubric set while keeping the conversation and target rubric fixed\.Expansionis our primary test: we compare each rubric’s isolation verdict against its joint verdict\. We reportoverall consistency\(full set\) andconsistency\-at\-KK\(random subsets of sizeKK\) to trace how interference grows with rubric count\.Subsettinggeneralizes expansion by comparing two nested subsets of arbitrary size rather than anchoring on isolation; this tests whether interference accumulates monotonically but requires sampling valid superset\.Reorderingevaluates the same set under multiple permutations, isolating positional effects from content\-based interference\.Noise injectionappends rubrics from unrelated conversations, testing whether the model can ignore irrelevant context\.

For all operations, we report rubric\-level agreement and Cohen’sκ\\kappa, plus sample\-level exact match \(EM\)\. Full metric definitions are in Appendix[A](https://arxiv.org/html/2608.14684#A1)\.

## 4Self\-Anchored Rubric Alignment

![Refer to caption](https://arxiv.org/html/2608.14684v1/x2.png)Figure 2:The SARA training pipeline\. The student generates a joint evaluation \(top\), which is parsed into per\-rubric segments\. The teacher evaluates each segment under isolation context \(bottom\), producing interference\-free anchor logits\. The distillation loss aligns student and teacher logits on analysis segments; the preservation loss maintains structural tokens\.### 4\.1Overview

SARA trains a judge to resist rubric interference through on\-policy self\-distillation\. The core idea is simple: the same model that struggles with joint evaluation judges each rubric reliably in isolation\. These isolation judgments are stable and require no external annotation, making them natural training anchors\.

At each training step, the student model generates a joint evaluation and parses it into per\-rubric segments\. A slowly updated copy of the student \(the teacher\) then evaluates each segment in isolation context, producing interference\-free logits\. A divergence loss aligns the student’s joint\-mode logits with these anchors on analysis segments, while a preservation loss maintains the structural tokens \(verdicts, formatting\)\. Figure[2](https://arxiv.org/html/2608.14684#S4.F2)shows the pipeline\.

### 4\.2On\-Policy Generation and Segment Extraction

At each training step, the studentℳθ\\mathcal\{M\}\_\{\\theta\}generates a multi\-rubric evaluation for conversationccand rubric setRR:

y∼ℳθ\(⋅∣c,R\)=\[a1,s1,…,an,sn\]y\\sim\\mathcal\{M\}\_\{\\theta\}\(\\cdot\\mid c,R\)=\[a\_\{1\},s\_\{1\},\\;\\ldots,\\;a\_\{n\},s\_\{n\}\]\(2\)Hereaia\_\{i\}is the analysis segment for rubricrir\_\{i\}andsis\_\{i\}is the structural segment \(verdict, formatting, transition tokens\) \. Becauseyyis sampled from the current model at each step \(on\-policy\), the training distribution tracks the model’s evolving behavior\.

We parseyyinto per\-rubric segments using the structured output format \(Appendix[B](https://arxiv.org/html/2608.14684#A2)\)\. This produces ananalysis mask𝐦ana∈\{0,1\}\|y\|\\mathbf\{m\}^\{\\text\{ana\}\}\\in\\\{0,1\\\}^\{\|y\|\}that marks tokens in allaia\_\{i\}segments, and a complementarystructure mask𝐦str=𝟏−𝐦ana\\mathbf\{m\}^\{\\text\{str\}\}=\\mathbf\{1\}\-\\mathbf\{m\}^\{\\text\{ana\}\}that covers allsis\_\{i\}segments\.

Segment extraction assumes instruct\-mode outputs with a predictable rubric\-analysis\-verdict structure\. Thinking\-mode models require model\-specific parsing, discussed in Appendix[I](https://arxiv.org/html/2608.14684#A9)\.

### 4\.3Isolation\-Anchored Distillation

#### Teacher model\.

The teacherℳϕ\\mathcal\{M\}\_\{\\phi\}is a slowly updated copy of the student\. After each training step:

ϕ←βema​ϕ\+\(1−βema\)​θ\\phi\\leftarrow\\beta\_\{\\text\{ema\}\}\\,\\phi\+\(1\-\\beta\_\{\\text\{ema\}\}\)\\,\\theta\(3\)The high decay rate \(βema\\beta\_\{\\text\{ema\}\}close to 1\) provides a stable reference throughout training\. Using the teacher rather than the student itself as anchor avoids a degenerate loop where drifting judgments reinforce interference\.

Two considerations motivate this design over using the student’s own isolation logits directly\. First, EMA smooths the reference temporally and prevents the training signal from oscillating at each update\. Second, it breaks a potential degenerate loop: if the student anchored on its own isolation pass, any systematic bias would reinforce itself without correction\.

#### Distillation loss\.

For each analysis segmentaia\_\{i\}, we compare two forward passes over the same tokens: the studentℳθ\\mathcal\{M\}\_\{\\theta\}under the full rubric setRR, and the teacherℳϕ\\mathcal\{M\}\_\{\\phi\}under isolation context\{ri\}\\\{r\_\{i\}\\\}:

𝐩\\displaystyle\\mathbf\{p\}=ℳθ\(y\|c,R\)\\displaystyle=\\mathcal\{M\}\_\{\\theta\}\\\!\\left\(y\\;\\middle\|\\;c,\\;R\\right\)\(4\)𝐪^\(i\)\\displaystyle\\hat\{\\mathbf\{q\}\}^\{\(i\)\}=ℳϕ\(ai\|c,\{ri\}\)\\displaystyle=\\mathcal\{M\}\_\{\\phi\}\\\!\\left\(a\_\{i\}\\;\\middle\|\\;c,\\;\\\{r\_\{i\}\\\}\\right\)Here𝐩\\mathbf\{p\}denotes the student’s logits over its complete joint\-mode outputyy, conditioned on conversationccand all rubricsRR\. The teacher logits𝐪^\(i\)\\hat\{\\mathbf\{q\}\}^\{\(i\)\}cover only theii\-th analysis segmentaia\_\{i\}, conditioned on the single rubricrir\_\{i\}\. The student sees all rubrics and may exhibit interference; the teacher sees only the target rubric and produces interference\-free logits\. We align them across all analysis tokens:

ℒdistill=1\|𝒜\|​∑i=1n∑t∈𝒜iDJSD\(βd\)​\(𝐩t∥𝐪^t\(i\)\)\\mathcal\{L\}\_\{\\text\{distill\}\}=\\frac\{1\}\{\|\\mathcal\{A\}\|\}\\sum\_\{i=1\}^\{n\}\\sum\_\{t\\in\\mathcal\{A\}\_\{i\}\}D\_\{\\text\{JSD\}\}^\{\(\\beta\_\{d\}\)\}\\\!\\left\(\\mathbf\{p\}\_\{t\}\\;\\\|\\;\\hat\{\\mathbf\{q\}\}^\{\(i\)\}\_\{t\}\\right\)\(5\)where𝒜i\\mathcal\{A\}\_\{i\}indexes token positions in segmentaia\_\{i\},𝒜=⋃i𝒜i\\mathcal\{A\}=\\bigcup\_\{i\}\\mathcal\{A\}\_\{i\}is the full analysis token set marked by𝐦ana\\mathbf\{m\}^\{\\text\{ana\}\}\.

#### Preservation loss\.

Structural tokens \(verdicts, formatting, transitions\) need a separate signal\. We align the student’s logits with the teacher’s full\-context logits, both conditioned on the complete rubric setRR:

𝐪=ℳϕ\(y\|c,R\)\\mathbf\{q\}=\\mathcal\{M\}\_\{\\phi\}\\\!\\left\(y\\;\\middle\|\\;c,\\;R\\right\)\(6\)Here𝐪\\mathbf\{q\}denotes the teacher’s logits over the same outputyyunder the same full context\. The only difference from𝐩\\mathbf\{p\}is the model weights \(ϕ\\phivsθ\\theta\)\. We compute the preservation loss over structural token positions𝒮\\mathcal\{S\}marked by𝐦str\\mathbf\{m\}^\{\\text\{str\}\}:

ℒpreserve=1\|𝒮\|​∑t∈𝒮DJSD\(βp\)​\(𝐩t∥𝐪t\)\\mathcal\{L\}\_\{\\text\{preserve\}\}=\\frac\{1\}\{\|\\mathcal\{S\}\|\}\\sum\_\{t\\in\\mathcal\{S\}\}D\_\{\\text\{JSD\}\}^\{\(\\beta\_\{p\}\)\}\\\!\\left\(\\mathbf\{p\}\_\{t\}\\;\\\|\\;\\mathbf\{q\}\_\{t\}\\right\)\(7\)We use full\-context teacher logits here because structural tokens coordinate across all rubrics and have no meaningful isolation equivalent\.

#### Total loss\.

ℒ=ℒdistill\+α​ℒpreserve\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{distill\}\}\+\\alpha\\,\\mathcal\{L\}\_\{\\text\{preserve\}\}\(8\)
Since structural tokens are far fewer than analysis tokens, each structural token already receives higher per\-token weight at equalα\\alpha\. We setα=0\.4\\alpha\{=\}0\.4across all experiments\.

### 4\.4Rubric Context Augmentation

To expose the model to diverse interference patterns, the data collator applies two augmentations to each rubric setRRat every training step:

- •Rubric shuffling\.We randomly permute the rubric order at each step, preventing the model from memorizing position\-specific patterns\.
- •Noise injection\.With probabilitypnoisep\_\{\\text\{noise\}\}, we appendkkrubrics from unrelated conversations, training the model to ignore irrelevant criteria\.

These two augmentations target the interference sources in Table[1](https://arxiv.org/html/2608.14684#S3.T1)\. Shuffling addresses positional sensitivity and noise injection addresses content\-based distraction\.

## 5Experiments

### 5\.1Setup

We evaluate on three datasets spanning distinct domains and scoring formats \(Table[2](https://arxiv.org/html/2608.14684#S5.T2)\)\. HealthBench \(HB\) contains medical conversations evaluated against per\-sample rubric checklists with binary met/unmet verdicts\. Rubric counts range from 2 to 32 per sample, making it the most demanding test for interference\. FLASK evaluates general\-purpose instruction\-following across 3 fixed skill dimensions, each scored on a 1–5 scale\. ResearchQA \(RQA\) evaluates scientific question answering against per\-sample rubrics scored on a 0–4 scale\. We sampled 1000 instances from the full dataset\.

HBFLASKRQADomainMedicalGeneralScientificScoringBinary1–50–4Rubrics/sample2–323 \(fixed\)1–8Avg rubrics11\.53\.07\.5Samples50017401000Table 2:Dataset statistics\. HB = HealthBench, RQA = ResearchQA\.#### Models and baselines\.

We apply SARA to Qwen3\-8B, Qwen3\-14B, Qwen3\-32B, and Llama\-3\.1\-8B\-Instruct\. As a baseline, we compare againstSFTiso: standard supervised fine\-tuning on isolation outputs concatenated into joint format\. This represents the most direct approach to teaching joint\-mode behavior from isolation references\. Each dataset is split 8:1:1 for training, validation, and test\.

#### Training details\.

SARA uses full\-parameter fine\-tuning with EMA decay 0\.999, symmetric JSD \(βd=0\.5\\beta\_\{d\}\{=\}0\.5\), KL preservation \(βp=1\.0\\beta\_\{p\}\{=\}1\.0,α=0\.4\\alpha\{=\}0\.4\), and vLLM greedy decoding for on\-policy generation\. We select checkpoints by shuffle consistency on the validation set\.

#### Evaluation\.

All evaluations use greedy decoding with fixed random seeds\. We cap rubrics at 10 per sample\. For shuffle tests, we average over 5 random permutations\. We report rubric\-level agreement \(Agr\), Cohen’sκ\\kappa\(weighted for scored formats\), and sample\-level exact match \(EM\)\.

#### Infrastructure\.

All experiments run on 8×\\timesNVIDIA H20 GPUs with vLLM for batched inference\.

HealthBenchResearchQAFLASKAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowQwen3\-8B\.819\.630\.22\.774\.665\.22\.720\.663\.41\+SARA\.874\.736\.34\.914\.868\.59\.810\.801\.52Qwen3\-14B\.874\.747\.34\.768\.658\.23\.722\.665\.41\+SARA\.903\.806\.38\.900\.879\.52\.803\.802\.55Qwen3\-32B\.887\.775\.36\.720\.651\.25\.701\.652\.41\+SARA\.881\.762\.36\.880\.851\.42\.836\.822\.59Llama\-3\.1\-8B\.733\.467\.08\.711\.608\.12\.609\.637\.24\+SARA\.830\.652\.18\.853\.800\.34\.722\.760\.41Table 3:Overall consistency \(isolation vs\. joint mode\)\. Agr = rubric\-level agreement,κ\\kappa= Cohen’s kappa \(weighted for scored formats\), EM = sample\-level exact match\.

### 5\.2Main Results

Table[3](https://arxiv.org/html/2608.14684#S5.T3)presents overall consistency between isolation and joint mode\. SARA delivers consistent improvements across model scales, architectures, and evaluation domains\.

The largest gains appear on ResearchQA: Qwen3\-8B’s EM nearly triples \(\.22→\\to\.59\), and itsκ\\kappajumps from \.665 to \.868\. ResearchQA combines moderate rubric counts with per\-sample rubric definitions, creating ample opportunity for cross\-rubric contamination\. On HealthBench, binary verdicts set a higher baseline and lower ceiling, yet improvements remain steady across models up to 14B\. On FLASK, only 3 fixed rubrics limit the surface area for interference, yet gains persist \(e\.g\., Qwen3\-32B: EM \.41→\\to\.59\), confirming that even minimal rubric sets trigger measurable interference\.

SARA benefits weaker models more \(Llama\-3\.1\-8B: \+14\.2% Agr on ResearchQA\), consistent with our hypothesis that stronger models have implicitly learned partial invariance through scale\. Yet even Qwen3\-32B improves substantially on scored formats, suggesting scale alone does not fully resolve interference\. EM gains are disproportionately large relative to Agr across all settings \(e\.g\., Qwen3\-8B on ResearchQA: Agr \+\.140 but EM \+\.370\)\. This asymmetry reveals that baseline interference is not concentrated on a few hard rubrics but scattered across many—a single flipped rubric per sample suffices to break exact match, and SARA systematically corrects these distributed errors\. On scored datasets, SARA also narrows the gap between Agr andκ\\kappa\(Qwen3\-32B on ResearchQA: \.720/\.651→\\to\.880/\.851\), indicating that remaining disagreements concentrate at adjacent scores \(±\\pm1\) rather than large deviations—interference is not only less frequent but less severe\.

### 5\.3Consistency Without Quality Degradation

ConsistencyIso Agr w/Jnt Agr w/Agr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowBase∗↑\\uparrowBase∗↑\\uparrowQwen3\-14B\.768\.658\.23––\+SFTiso\.909\.872\.54\.892\.777\+SARA\.900\.879\.52\.925\.806Table 4:SFTisovs\. SARA on ResearchQA\. Left: consistency between evaluation modes\. Right: agreement with the untrained base model’s outputs\.#### Why not naive SFT?

A natural baseline is to directly fine\-tune on isolation outputs concatenated into joint format \(SFTiso: 5 random\-seed isolation samples per rubric as synthetic joint targets\)\. Table[4](https://arxiv.org/html/2608.14684#S5.T4)compares this approach against SARA on Qwen3\-14B \(ResearchQA\)\. Beyond consistency, we measure behavioral drift: “Iso\. Agr with Base∗” reports how much a trained model’s isolation outputs agree with the untrained base; “Joint Agr with Base∗” does the same for joint outputs\.

SFTisoachieves slightly higher iso–joint consistency \(Agr \.909 vs \.900, EM \.54 vs \.52\), but drifts further from the original model’s behavior in both modes \(Iso: \.892 vs \.925; Joint: \.777 vs \.806\)\. SFT forces the model to*reproduce*fixed target outputs, inadvertently shifting its judgment distribution; SARA teaches the model to*reason independently per rubric*, preserving its original evaluation behavior while reducing interference\.

Iso\. Agr w/Joint Agr w/Base∗GPT\-4\.1Base∗GPT\-4\.1Qwen3\-8B–\.729–\.773\+SARA\.921\.706\.874\.731Qwen3\-14B–\.801–\.795\+SARA\.912\.806\.896\.806Qwen3\-32B–\.830–\.815\+SARA\.896\.823\.894\.819Llama\-3\.1\-8B–\.702–\.726\+SARA\.839\.700\.755\.755Table 5:Evaluation quality on HealthBench\. Base∗: agreement with untrained base\. GPT\-4\.1: external reference judge\.
#### Does SARA preserve evaluation quality?

Table[5](https://arxiv.org/html/2608.14684#S5.T5)confirms across all models on HealthBench that SARA’s consistency gains do not come at the expense of judgment quality\. Agreement with the untrained base exceeds \.84 in all settings, confirming minimal behavioral drift\. GPT\-4\.1 agreement is preserved or improved \(Qwen3\-14B joint: \.795→\\to\.806; Llama\-3\.1\-8B joint: \.726→\\to\.755\): when isolation judgments are already reasonable, reducing interference also improves absolute accuracy\.

### 5\.4Interference Under Controlled Perturbations

We apply the perturbation framework from Table[1](https://arxiv.org/html/2608.14684#S3.T1)to trace how each interference source affects model behavior, and how SARA mitigates it\.

![Refer to caption](https://arxiv.org/html/2608.14684v1/x3.png)Figure 3:Mean Absolute Deviation \(MAD\) between isolation and joint verdicts on ResearchQA as co\-evaluated rubric countKKgrows\.#### Scaling with rubric count\.

Figure[3](https://arxiv.org/html/2608.14684#S5.F3)traces MAD on ResearchQA asKKgrows from 2 to 8\. For all base models, MAD rises steeply withKK: Qwen3\-32B climbs from \.203 atK=2K\{=\}2to \.362 atK=8K\{=\}8, nearly doubling\. SARA flattens these curves\. Qwen3\-8B’s MAD stays nearly constant \(\.080 atK=2K\{=\}2, \.080 atK=4K\{=\}4, \.081 atK=8K\{=\}8\), indicating that the model evaluates each rubric independently of set size\. Across all models, SARA compresses the MAD range from \.147–\.374 \(base\) to \.080–\.176 \(trained\) and eliminates the upward trend withKK\.

![Refer to caption](https://arxiv.org/html/2608.14684v1/x4.png)Figure 4:Shuffle invariance and noise robustness on ResearchQA \(EM\)\.
#### Shuffle and noise robustness\.

Figure[4](https://arxiv.org/html/2608.14684#S5.F4)reports EM under random rubric permutation \(shuffle\) and irrelevant rubric injection \(noise\) on ResearchQA\. Under shuffling, Qwen3\-14B achieves only \.262 EM; under noise, Qwen3\-32B drops to \.200\. SARA roughly doubles shuffle EM across all models \(e\.g\., Qwen3\-8B: \.298→\\to\.594\) and yields similar noise gains \(Qwen3\-8B: \.255→\\to\.620\)\. Full results across all datasets are in Appendix[D](https://arxiv.org/html/2608.14684#A4)\.

### 5\.5SARA Learns Transferable Consistency

To confirm that SARA teaches a general capability rather than fitting dataset\-specific patterns, we evaluate Qwen3\-14B trained on ResearchQA \(SARArqa\) directly on HealthBench and FLASK without further fine\-tuning \(Table[6](https://arxiv.org/html/2608.14684#S5.T6)\)\.

HealthBench\(Agr /κ\\kappa/ EM\)Qwen3\-14B\.874\.747\.34\+SARA\.903\.806\.38\+SARArqa\.890\.779\.34FLASK\(Agr /κ\\kappa/ EM\)Qwen3\-14B\.722\.665\.41\+SARA\.803\.802\.55\+SARArqa\.820\.800\.55Table 6:Cross\-dataset transfer\. SARArqa\(trained on ResearchQA\) evaluated zero\-shot on HealthBench and FLASK\.SARArqaimproves over the base model on both target datasets\. On FLASK it even outperforms the in\-domain SARA model in Agr \(\.820 vs \.803\)\. This confirms that SARA learns a domain\-agnostic ability to decouple per\-rubric reasoning from contextual interference\.

### 5\.6Analysis

#### Attention analysis\.

To understand how SARA reduces interference mechanistically, we compare attention patterns during joint evaluation \(Qwen3\-14B, ResearchQA, 10 test samples\)\. Figure[5](https://arxiv.org/html/2608.14684#S5.F5)shows the average attention matrix: for each output analysis segment \(row\), we compute what fraction of its attention flows to each input rubric span and each output analysis span\. Each row sums to one\.

![Refer to caption](https://arxiv.org/html/2608.14684v1/x5.png)Figure 5:Normalized attention from each output analysis segment to input rubric spans \(I1–I6\) and output analysis spans \(O1–O6\), averaged over 10 samples\.Two effects stand out\. First, SARA concentrates output self\-attention on the diagonal: the average self\-attention rises from \.623 to \.809 \(\+30% relative\), meaning each rubric’s analysis attends predominantly to its own prior reasoning rather than to other rubrics’ analyses\. Second, SARA suppresses cross\-rubric leakage\. Off\-diagonal attention in the output half drops from \.039 to \.014 \(–64% relative\)\. The base model’s lower triangle shows visible leakage—later rubrics attend heavily to earlier rubrics’ analyses \(e\.g\., Out R2 allocates \.168 to Out R1\)\. SARA nearly eliminates this pattern \(the same cell drops to \.048\)\.

The input half tells a consistent story: attention to other input rubrics decreases from \.018 to \.011 \(–38% relative\), confirming that SARA’s reasoning becomes less sensitive to the presence of co\-evaluated rubric definitions\. Together, these results show that SARA achieves interference reduction through a concrete mechanistic change: redirecting attention from cross\-rubric spans back to the model’s own analysis of the target rubric\.

#### Case study\.

Table[7](https://arxiv.org/html/2608.14684#S5.T7)shows a FLASK example where interference distorts verdicts bidirectionally\. The task is a fill\-in\-the\-blank question about business ethics; the assistant provides a plausible but imprecise answer without citations\. Both models agree in isolation: R1 \(comprehension\) and R2 \(factuality\) deserve 3/5, while R3 \(readability\) deserves 4–5/5\.

In joint mode, the base model’s R1 inflates from 3 to 4 \(softer phrasing: “somewhat relevant” replaces “lacks precision and completeness”\), while R2 deflates from 3 to 2 \(absolutist language: “does not provide*any*factual evidence” replaces “the concept*is*accurate … not supported by citation”\)\. SARA’s joint output matches isolation on all three rubrics, preserving both the critical tone of R1 and the balanced assessment of R2\. Full outputs are in Appendix[H](https://arxiv.org/html/2608.14684#A8)\.

Task: Fill in the blank about ethics management\.Response: “…business ethics management\.” \(no citation\)R2\(Factuality\): Is the response supported by reliable evidence or citation?Base — isolation \(3/5\):“While the concept of managing ethical issues through policies and programsis accurate, the specific term ‘business ethics management’ is not clearly defined or supported by a citation\.”Base — joint \(2/5✗\):“The responsedoes not provide anyfactual evidence or citation to support the claim … there isno reliable sourceexplicitly referenced to confirm its accuracy\.”SARA — joint \(3/5✓\):“The response lacks any citation or reference to a reliable source … While the conceptmay be accurate, the support is incomplete and not fully reliable\.”Table 7:Case study \(Qwen3\-14B, FLASK\)\. Base model’s joint R2 adopts absolutist language, deflating 3/5 to 2/5\. SARA maintains the balanced assessment\.
#### Additional observations\.

Qwen3\-32B shows limited gains on HealthBench because the base model already achieves \.887 Agr and binary verdicts provide a coarser distillation signal than scored verdicts\. On scored datasets, gains are clear \(ResearchQA: \.720→\\to\.880; FLASK: \.701→\\to\.836\)\. SARA\-trained models also generate longer outputs \(e\.g\., Qwen3\-8B on HealthBench: 405→\\to611 tokens\) due to more detailed per\-rubric analyses that match isolation depth\. Full token statistics are in Appendix[G](https://arxiv.org/html/2608.14684#A7)\.

We report hyperparameter sensitivity and preservation loss ablation of SARA in Appendix[F](https://arxiv.org/html/2608.14684#A6)\.

## 6Conclusion

We identify rubric interference as a systematic failure mode in multi\-rubric LLM evaluation and propose SARA, a self\-distillation method that aligns joint\-mode reasoning with the model’s own isolation judgments\. SARA improves consistency across datasets, model families, and scoring formats, and transfers across domains without retraining\. Attention analysis confirms the mechanism: SARA selectively suppresses cross\-rubric information flow while preserving within\-rubric coherence\.

Our results suggest a broader principle: when a model possesses a capability but fails to exercise it under complex conditions, self\-distillation from a simpler setting can close the gap without external supervision\. Looking ahead, SARA’s training signal complements standard SFT and RLHF objectives; combining interference resistance with reward model training could yield judges that are both accurate and consistent\. The measurement framework itself can also serve as a diagnostic protocol for any multi\-rubric judge before deployment\.

## Limitations

#### Thinking\-mode models\.

SARA’s segment extraction relies on a predictable rubric\-analysis\-verdict structure in instruct\-mode outputs\. Adapting SARA to thinking\-mode models requires paragraph\-level heuristics that are less precise, and preliminary results show smaller gains than in instruct mode \(Appendix[I](https://arxiv.org/html/2608.14684#A9)\)\. Achieving stronger results in thinking mode likely requires additional format supervision, such as training the model to use explicit rubric delimiters within its thinking block\.

#### Isolation as anchor\.

SARA treats isolation verdicts as interference\-free anchors\. This assumption is supported by the high self\-agreement of isolation judgments \(≥\\geq0\.98 across all models\), but isolation is not infallible\. In some cases, co\-evaluating related rubrics may surface useful context that improves judgment quality\. Our framework does not distinguish beneficial context from harmful interference\. Developing methods that preserve helpful inter\-rubric information while suppressing harmful interference remains an open direction\.

## Ethics Statement

Our work improves the consistency of LLM\-based evaluation\. We note that more consistent automated judges may encourage reduced human oversight; however, consistency does not guarantee correctness, and we recommend SARA\-trained models complement rather than replace human evaluation in high\-stakes domains\. All datasets and models used are publicly available under permissive licenses, and no personally identifiable information is collected or processed\.

## References

- On\-policy distillation of language models: learning from self\-generated mistakes\.External Links:2306\.13649,[Link](https://arxiv.org/abs/2306.13649)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px4.p1.1)\.
- R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel, J\. Heidecke, and K\. Singhal \(2025\)HealthBench: evaluating large language models towards improved human health\.External Links:2505\.08775,[Link](https://arxiv.org/abs/2505.08775)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p2.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu \(2024\)Self\-play fine\-tuning converts weak language models to strong language models\.External Links:2401\.01335,[Link](https://arxiv.org/abs/2401.01335)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px4.p1.1)\.
- C\. Chiang and H\. Lee \(2023\)Can large language models be an alternative to human evaluations?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 15607–15631\.External Links:[Link](https://aclanthology.org/2023.acl-long.870/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.870)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p1.1)\.
- Y\. Elazar, N\. Kassner, S\. Ravfogel, A\. Ravichander, E\. Hovy, H\. Schütze, and Y\. Goldberg \(2021\)Measuring and improving consistency in pretrained language models\.Transactions of the Association for Computational Linguistics9,pp\. 1012–1031\.External Links:[Link](https://aclanthology.org/2021.tacl-1.60/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00410)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Furlanello, Z\. Lipton, M\. Tschannen, L\. Itti, and A\. Anandkumar \(2018\)Born again neural networks\.InProceedings of the 35th International Conference on Machine LearningProceedings of the 41st International Conference on Machine LearningInternational Conference on Learning RepresentationsProceedings of the 2023 Conference on Empirical Methods in Natural Language ProcessingThe Thirteenth International Conference on Learning RepresentationsProceedings of the Fourth Workshop on Generation, Evaluation and Metrics \(GEM²\)The Thirteenth International Conference on Learning RepresentationsInternational Conference on Learning Representations,J\. Dy, A\. Krause, R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, F\. Berkenkamp, B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, Y\. Sun, H\. Bouamor, J\. Pino, K\. Bali, O\. Arviv, M\. Clinciu, K\. Dhole, R\. Dror, S\. Gehrmann, E\. Habba, I\. Itzhak, S\. Mille, Y\. Perlitz, E\. Santus, J\. Sedoc, M\. Shmueli Scheuer, G\. Stanovsky, O\. Tafjord, B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Proceedings of Machine Learning ResearchProceedings of Machine Learning Research, Vol\.8023520242024,pp\. 1607–1616\.External Links:[Link](https://proceedings.mlr.press/v80/furlanello18a.html)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Gu, L\. Dong, F\. Wei, and M\. Huang \(2024\)MiniLLM: knowledge distillation of large language models\.pp\. 32694–32717\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/8ac015d409635f196f9e3e9dcfb9a94e-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. M\. Hendryx \(2026\)Rubrics as rewards: reinforcement learning beyond verifiable domains\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=c1bTcrDmt4)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p2.1)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024\)Prometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4334–4353\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.248/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.248)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p1.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Lee, J\. Kim, J\. Kim, H\. Cho, J\. Kang, P\. Kang, and N\. Kim \(2025\)CheckEval: a reliable llm\-as\-a\-judge framework for evaluating text generation using checklists\.External Links:2403\.18771,[Link](https://arxiv.org/abs/2403.18771)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p1.1),[§1](https://arxiv.org/html/2608.14684#S1.p2.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Li, J\. Chen, Q\. Ai, Z\. Chu, Y\. Zhou, Q\. Dong, and Y\. Liu \(2025\)CalibraEval: calibrating prediction distribution to mitigate selection bias in LLMs\-as\-judges\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16537–16552\.External Links:[Link](https://aclanthology.org/2025.acl-long.808/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.808),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Li, Q\. Dong, J\. Chen, H\. Su, Y\. Zhou, Q\. Ai, Z\. Ye, and Y\. Liu \(2024\)LLMs\-as\-judges: a comprehensive survey on llm\-based evaluation methods\.External Links:2412\.05579,[Link](https://arxiv.org/abs/2412.05579)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. Li, S\. Dou, K\. Shao, C\. Chen, and H\. Hu \(2026\)Evaluating scoring bias in llm\-as\-a\-judge\.External Links:2506\.22316,[Link](https://arxiv.org/abs/2506.22316)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Y\. Lin, Y\. Deng, K\. Chandu, A\. Ravichander, V\. Pyatkin, N\. Dziri, R\. L\. Bras, and Y\. Choi \(2025\)WildBench: benchmarking LLMs with challenging tasks from real users in the wild\.External Links:[Link](https://openreview.net/forum?id=MKEHCx25xp)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: NLG evaluation using gpt\-4 with better human alignment\.Singapore,pp\. 2511–2522\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.153/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Mizrahi, G\. Kaplan, D\. Malkin, R\. Dror, D\. Shahaf, and G\. Stanovsky \(2024\)State of what art? a call for multi\-prompt LLM evaluation\.Transactions of the Association for Computational Linguistics12,pp\. 933–949\.External Links:[Link](https://aclanthology.org/2024.tacl-1.52/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00681)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Panickssery, S\. R\. Bowman, and S\. Feng \(2024\)LLM evaluators recognize and favor their own generations\.External Links:2404\.13076,[Link](https://arxiv.org/abs/2404.13076)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p4.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Pathak, R\. Gandhi, V\. Uttam, A\. Ramamoorthy, P\. Ghosh, A\. R\. Jindal, S\. Verma, A\. Mittal, A\. Ased, C\. Khatri, Y\. Nakka, Devansh, J\. S\. Challa, and D\. Kumar \(2025\)Rubric is all you need: improving llm\-based code evaluation with question\-specific rubrics\.InProceedings of the 2025 ACM Conference on International Computing Education Research V\.1,ICER ’25,pp\. 181–195\.External Links:[Link](http://dx.doi.org/10.1145/3702652.3744220),[Document](https://dx.doi.org/10.1145/3702652.3744220)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p1.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models'sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.pp\. 25055–25083\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/6c0e99d736da621403018ca7b32b1a4d-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Tan, S\. Zhuang, K\. Montgomery, W\. Y\. Tang, A\. Cuadron, C\. Wang, R\. Popa, and I\. Stoica \(2025\)JudgeBench: a benchmark for evaluating LLM\-based judges\.External Links:[Link](https://openreview.net/forum?id=G0dksFayVq)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- A\. S\. Thakur, K\. Choudhary, V\. S\. Ramayapally, S\. Vaidyanathan, and D\. Hupkes \(2025\)Judging the judges: evaluating alignment and vulnerabilities in LLMs\-as\-judges\.Vienna, Austria and virtual meeting,pp\. 404–430\.External Links:[Link](https://aclanthology.org/2025.gem-1.33/),ISBN 979\-8\-89176\-261\-9Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Link](https://aclanthology.org/2024.acl-long.511/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p4.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Wu and A\. F\. Aji \(2025\)Style over substance: evaluation biases for large language models\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 297–312\.External Links:[Link](https://aclanthology.org/2025.coling-main.21/)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p4.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.
- L\. S\. Yifei, A\. Chang, C\. Malaviya, and M\. Yatskar \(2025\)ResearchQA: evaluating scholarly question answering at scale across 75 fields with survey\-mined questions and rubrics\.External Links:2509\.00496,[Link](https://arxiv.org/abs/2509.00496)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p2.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. E\. Weston \(2024\)Self\-rewarding language models\.pp\. 57905–57923\.External Links:[Link](https://proceedings.mlr.press/v235/yuan24d.html)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.External Links:2601\.18734,[Link](https://arxiv.org/abs/2601.18734)Cited by:[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46595–46623\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf)Cited by:[§1](https://arxiv.org/html/2608.14684#S1.p1.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.14684#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ADetailed Metric Definitions

All experiments compare verdicts on the same conversation–rubric pair\(c,ri\)\(c,r\_\{i\}\)across two evaluation conditionsAAandBB\(e\.g\., isolation vs\. joint, or two different rubric permutations\)\. We use the same core metrics for both binary and scored formats\.

### A\.1Rubric\-Level Metrics

GivenNNverdict pairs\{\(vi\(A\),vi\(B\)\)\}i=1N\\\{\(v\_\{i\}^\{\(A\)\},v\_\{i\}^\{\(B\)\}\)\\\}\_\{i=1\}^\{N\}:

#### Agreement \(Agr\)\.

Fraction of rubrics with identical verdicts:

Agr=1N​∑i=1N𝟙​\[vi\(A\)=vi\(B\)\]\\text\{Agr\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{1\}\[v\_\{i\}^\{\(A\)\}=v\_\{i\}^\{\(B\)\}\]\(9\)For binary verdicts, this is equivalent to classification accuracy between two annotators\. For scored verdicts, it measures exact score match \(identical to what some prior work calls Exact Agreement Rate\)\.

#### Cohen’sκ\\kappa\.

Agreement corrected for chance:

κ=po−pe1−pe\\kappa=\\frac\{p\_\{o\}\-p\_\{e\}\}\{1\-p\_\{e\}\}\(10\)wherepo=Agrp\_\{o\}=\\text\{Agr\}andpep\_\{e\}is the expected agreement under independence\. For binary verdicts, we use standard \(unweighted\)κ\\kappa\. For scored verdicts, we use quadratic\-weightedκ\\kappa, which penalizes larger deviations more heavily\. Both variants are reported under the same symbolκ\\kappathroughout; the weighting scheme is determined by the scoring format\.

#### Mean Absolute Deviation \(MAD\)\.

Average per\-rubric deviation:

MAD=1N​∑i=1N\|vi\(A\)−vi\(B\)\|\\text\{MAD\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\|v\_\{i\}^\{\(A\)\}\-v\_\{i\}^\{\(B\)\}\|\(11\)For binary verdicts,MAD=1−Agr\\text\{MAD\}=1\-\\text\{Agr\}\. For scored verdicts, MAD captures the typical magnitude of disagreement\.

#### Root Mean Squared Error \(RMSE\)\.

RMSE=1N​∑i=1N\(vi\(A\)−vi\(B\)\)2\\text\{RMSE\}=\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(v\_\{i\}^\{\(A\)\}\-v\_\{i\}^\{\(B\)\}\)^\{2\}\}\(12\)More sensitive to large deviations than MAD\. Reported in appendix tables for scored formats\.

#### Pearson Correlation \(ρ\\rho\)\.

Linear correlation betweenv\(A\)v^\{\(A\)\}andv\(B\)v^\{\(B\)\}\. Measures whether the two conditions preserve relative ordering of scores\.

### A\.2Sample\-Level Metric

#### Exact Match \(EM\)\.

Fraction of samples whereallrubrics agree:

EM=1\|C\|∑c∈C𝟙\[∀ri∈Rc:vi\(A\)=vi\(B\)\]\\text\{EM\}=\\frac\{1\}\{\|C\|\}\\sum\_\{c\\in C\}\\mathbb\{1\}\\\!\\left\[\\forall\\,r\_\{i\}\\in R\_\{c\}:\\;v\_\{i\}^\{\(A\)\}=v\_\{i\}^\{\(B\)\}\\right\]\(13\)EM is the strictest metric: a single rubric disagreement within a sample counts as a failure\.

### A\.3Aggregate\-Level Metrics

For scored formats, we also compute the total score per sampleTc\(X\)=∑ri∈Rcvi\(X\)T\_\{c\}^\{\(X\)\}=\\sum\_\{r\_\{i\}\\in R\_\{c\}\}v\_\{i\}^\{\(X\)\}and report:

#### Total Exact Match \(TotalEM\)\.

Fraction of samples with identical total scores:

TotalEM=1\|C\|​∑c∈C𝟙​\[Tc\(A\)=Tc\(B\)\]\\text\{TotalEM\}=\\frac\{1\}\{\|C\|\}\\sum\_\{c\\in C\}\\mathbb\{1\}\[T\_\{c\}^\{\(A\)\}=T\_\{c\}^\{\(B\)\}\]\(14\)

#### Total MAE\.

Mean absolute error of total scores:

TotalMAE=1\|C\|​∑c∈C\|Tc\(A\)−Tc\(B\)\|\\text\{TotalMAE\}=\\frac\{1\}\{\|C\|\}\\sum\_\{c\\in C\}\|T\_\{c\}^\{\(A\)\}\-T\_\{c\}^\{\(B\)\}\|\(15\)

#### Total Pearson \(ρT\\rho\_\{T\}\)\.

Pearson correlation of total score sequences\. Measures whether sample\-level ranking is preserved across conditions\.

### A\.4Metric Selection

MetricHBFLASKRQAMain tables \(all formats\)Agr,κ\\kappa, EM✓✓✓Appendix \(scored only\)MAD, RMSE,ρ\\rho✓✓TotalEM, TotalMAE,ρT\\rho\_\{T\}✓✓OperationsOverall, Con\-KK✓✓✓Subsetting✓✓Shuffle✓✓✓Noise✓✓✓Table 8:Metric and operation applicability\. FLASK’s fixed 3\-rubric format does not support Con\-KKor Subsetting\.

## Appendix BEvaluation Prompts

We use the following task prompts for joint\-mode evaluation\. In isolation mode, the same prompt is used with a single\-rubric list\. The FLASK prompt follows the same format as ResearchQA with the scoring scale adapted to 1–5\.

### B\.1HealthBench \(Binary\)

Joint\-Mode Evaluation Prompt \(Binary\)\# Task You are a professional dialogue grader\. Your task is to evaluate the \*\*last response \(completion\)\*\* from the Assistant in the provided conversation against a list of scoring rubrics\. You must determine which criteria are met, and output only the indices of the met rubrics\. \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \# Input Data \*\*Conversation\*\*: \{conversation\} \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \*\*Rubrics\*\*: \{rubrics\} \(Each rubric contains a "criterion" and a "point\_value"\. Points can be positive or negative\.\) \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \# Scoring Logic 1\. \*\*Evaluation\*\*: For each item in the Rubrics, determine if the Assistant’s last response meets the specific criterion\. 2\. \*\*"Such as" Exception\*\*: If a criterion includes phrases like "such as", "for example", or "including", the response meets the criteria if it addresses the core requirement, even if it does not mention every specific example listed\. 3\. \*\*Negative Criteria\*\*: If a criterion describes an undesirable behavior \(e\.g\., "is overly verbose" or "contains inaccuracies"\), the criteria is "met" only if the Assistant actually exhibited that bad behavior\. \# Output Format For each rubric, output strictly in this format \(one entry per rubric, do NOT split a single rubric into multiple entries\): \*\*Rubric 1 Analysis\*\*: <analysis\>\. \*\*Verdict\*\*: Met/Not Met\. \*\*Rubric 2 Analysis\*\*: <analysis\>\. \*\*Verdict\*\*: Met/Not Met\. \.\.\. Met Rubric Indices: \[list of index numbers, separated by commas\]

### B\.2ResearchQA / FLASK \(Scored\)

Joint\-Mode Evaluation Prompt \(Scored\)\# Task You are a professional dialogue grader\. Your task is to evaluate the \*\*last response \(completion\)\*\* from the Assistant in the provided conversation against a list of scoring rubrics\. Each rubric describes a specific requirement\. You must judge the degree to which the response fulfills each requirement on a 1\-5 scale\. \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \# Input Data \*\*Conversation\*\*: \{conversation\} \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \*\*Rubrics\*\*: \{rubrics\} \(Each rubric describes a specific requirement that the response should fulfill\.\) \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- \# Scoring Logic 1\. \*\*Evaluation\*\*: For each rubric, determine how well the Assistant’s last response fulfills the described requirement\. 2\. \*\*Scoring Scale\*\*: \- \*\*5\*\*: Fully fulfilled \-\-\- the response completely and accurately addresses the requirement\. \- \*\*4\*\*: Mostly fulfilled \-\-\- the response addresses the requirement with minor omissions or imprecisions\. \- \*\*3\*\*: Partially fulfilled \-\-\- the response addresses some aspects but misses key elements\. \- \*\*2\*\*: Minimally fulfilled \-\-\- the response only tangentially or superficially relates to the requirement\. \- \*\*1\*\*: Not fulfilled \-\-\- the response fails to address the requirement or directly contradicts it\. 3\. \*\*Holistic Judgment\*\*: If the response falls between two levels, choose the lower one\. \# Output Format For each rubric, output strictly in this format \(one entry per rubric, do NOT split a single rubric into multiple entries\): \*\*Rubric 1 Analysis\*\*: <analysis\>\. \*\*Verdict\*\*: <score\>/5\. \*\*Rubric 2 Analysis\*\*: <analysis\>\. \*\*Verdict\*\*: <score\>/5\. \.\.\. Total Score: <s1\>\+<s2\>\+\.\.\.=<sum\>

The FLASK prompt uses the identical format with the scoring scale adjusted to match its 1–5 skill\-based rubrics\. For isolation\-mode evaluation, the same prompts are used with\|R\|=1\|R\|=1\(a single rubric in the rubric list\)\. The output format reduces to a single analysis–verdict pair\.

## Appendix CConsistency\-at\-K Full Results

Tables[9](https://arxiv.org/html/2608.14684#A3.T9)and[10](https://arxiv.org/html/2608.14684#A3.T10)report the full Consistency\-at\-KKresults underlying Figure[3](https://arxiv.org/html/2608.14684#S5.F3)\. FLASK is omitted because its fixed 3\-rubric format does not support variable\-KKevaluation\.

K=2K=2K=4K=4K=8K=8Agr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowHealthBenchQwen3\-8B\.871\.722\.754\.844\.672\.515\.843\.675\.266\+SARA\.914\.816\.832\.896\.779\.654\.885\.756\.403Qwen3\-14B\.901\.802\.806\.890\.779\.635\.874\.746\.355\+SARA\.953\.905\.912\.932\.864\.746\.912\.864\.468Qwen3\-32B\.895\.789\.796\.882\.763\.600\.884\.768\.350\+SARA\.913\.824\.836\.900\.798\.654\.898\.795\.415Llama\-3\.1\-8B\.797\.594\.627\.749\.498\.309\.733\.469\.094\+SARA\.861\.718\.741\.821\.636\.467\.824\.641\.240Table 9:Consistency\-at\-KKon HealthBench\. Metrics: Agr = rubric\-level agreement,κ\\kappa= Cohen’s kappa, EM = sample\-level exact match\.K=2K=2K=4K=4K=8K=8Agr↑\\uparrowMAD↓\\downarrowEM↑\\uparrowAgr↑\\uparrowMAD↓\\downarrowEM↑\\uparrowAgr↑\\uparrowMAD↓\\downarrowEM↑\\uparrowResearchQAQwen3\-8B\.867\.147\.761\.808\.229\.487\.787\.260\.216\+SARA\.928\.080\.866\.927\.080\.751\.922\.081\.568Qwen3\-14B\.867\.167\.751\.803\.278\.460\.787\.324\.243\+SARA\.926\.082\.862\.913\.096\.705\.909\.101\.460Qwen3\-32B\.837\.203\.718\.756\.335\.405\.760\.362\.216\+SARA\.892\.124\.797\.876\.144\.592\.910\.108\.378Llama\-3\.1\-8B\.788\.296\.628\.739\.374\.324\.740\.362\.027\+SARA\.889\.117\.809\.862\.171\.563\.872\.176\.324Table 10:Consistency\-at\-KKon ResearchQA\. Agr here is exact score agreement \(identical to EAR in prior work\)\. MAD captures deviation magnitude\.
## Appendix DShuffle Invariance and Noise Robustness Full Results

Figure[6](https://arxiv.org/html/2608.14684#A4.F6)summarizes shuffle invariance and noise robustness results\. Full numerical results are in Tables[11](https://arxiv.org/html/2608.14684#A4.T11)and[12](https://arxiv.org/html/2608.14684#A4.T12)\.

![Refer to caption](https://arxiv.org/html/2608.14684v1/x6.png)Figure 6:Sample\-level exact match \(EM\) under shuffle invariance \(top row\) and noise robustness at 40% noise ratio \(bottom row\)\. Gray bars denote base models; blue bars denote SARA\-trained models\.HealthBenchResearchQAFLASKAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowQwen3\-8B\.886\.770\.388\.783\.695\.298\.709\.667\.414\+SARA\.911\.819\.446\.921\.879\.594\.839\.832\.606Qwen3\-14B\.888\.773\.335\.791\.746\.262\.732\.657\.484\+SARA\.934\.867\.525\.912\.886\.538\.831\.816\.603Qwen3\-32B\.900\.800\.417\.740\.733\.220\.703\.658\.452\+SARA\.908\.816\.442\.888\.868\.456\.862\.868\.651Llama\-3\.1\-8B\.783\.566\.178\.741\.626\.210\.604\.642\.285\+SARA\.887\.769\.335\.852\.780\.338\.772\.799\.458Table 11:Shuffle invariance \(5 permutations, averaged\)\.HealthBenchResearchQAFLASKAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowAgr↑\\uparrowκ\\kappa↑\\uparrowEM↑\\uparrowQwen3\-8B\.876\.751\.340\.770\.663\.255\.740\.701\.454\+SARA\.908\.813\.430\.923\.877\.620\.826\.819\.572Qwen3\-14B\.896\.792\.460\.773\.712\.265\.737\.646\.477\+SARA\.924\.847\.480\.921\.880\.550\.819\.815\.592Qwen3\-32B\.892\.783\.420\.726\.714\.200\.694\.644\.431\+SARA\.905\.810\.430\.885\.861\.460\.847\.861\.615Llama\-3\.1\-8B\.755\.509\.200\.755\.621\.235\.589\.626\.250\+SARA\.873\.743\.370\.842\.772\.330\.713\.743\.345Table 12:Noise robustness \(40% noise ratio\)\.
## Appendix ESupplementary Metrics for Main Experiment

Table[13](https://arxiv.org/html/2608.14684#A5.T13)reports additional rubric\-level metrics \(MAD, RMSE, Pearsonρ\\rho\) for the scored datasets in the main experiment \(overall consistency: isolation vs\. joint mode\)\. Table[14](https://arxiv.org/html/2608.14684#A5.T14)reports aggregate\-level metrics computed from per\-sample total scores\.

ResearchQAFLASKMAD↓\\downarrowRMSE↓\\downarrowρ\\rho↑\\uparrowMAD↓\\downarrowRMSE↓\\downarrowρ\\rho↑\\uparrowQwen3\-8B\.273\.608\.816\.4731\.039\.728\+SARA\.101\.361\.918\.289\.762\.862Qwen3\-14B\.372\.844\.794\.441\.983\.743\+SARA\.113\.385\.942\.247\.605\.889Qwen3\-32B\.408\.832\.794\.471\.986\.737\+SARA\.148\.472\.911\.243\.662\.822Llama\-3\.1\-8B\.437\.919\.748\.542\.968\.763\+SARA\.186\.535\.886\.364\.766\.854Table 13:Supplementary rubric\-level metrics for overall consistency \(scored datasets\)\. MAD = mean absolute deviation, RMSE = root mean squared error,ρ\\rho= Pearson correlation\.ResearchQAFLASKTotalEM↑\\uparrowTotalMAE↓\\downarrowρT\\rho\_\{T\}↑\\uparrowTotalEM↑\\uparrowTotalMAE↓\\downarrowρT\\rho\_\{T\}↑\\uparrowQwen3\-8B\.2501\.620\.941\.4311\.316\.767\+SARA\.6400\.470\.989\.5570\.695\.920Qwen3\-14B\.2502\.100\.931\.4371\.206\.791\+SARA\.6100\.550\.988\.6090\.592\.913Qwen3\-32B\.3102\.340\.906\.4251\.264\.788\+SARA\.5300\.720\.983\.6150\.626\.910Llama\-3\.1\-8B\.2102\.100\.900\.3051\.316\.854\+SARA\.3900\.950\.973\.3860\.454\.915Table 14:Aggregate\-level metrics for overall consistency \(scored datasets\)\. TotalEM = fraction of samples with identical total scores, TotalMAE = mean absolute error of total scores,ρT\\rho\_\{T\}= Pearson correlation of total scores\.
## Appendix FHyperparameter Sensitivity

#### Hyperparameter sensitivity\.

Table[15](https://arxiv.org/html/2608.14684#A6.T15)ablates the two key hyperparameters on Qwen3\-14B \(ResearchQA\)\. Setting EMA decay=1\{=\}1\(teacher equals student, no momentum averaging\) drops EM from \.52 to \.48, confirming that the teacher must be decoupled from the student to provide a stable distillation target\. Performance is robust across decay∈\[\.99,\.999\]\\,\{\\in\}\\,\[\.99,\.999\]\. The JSD interpolation weightβd\\beta\_\{d\}shows similar robustness: EM varies by only \.02 acrossβd∈\[0\.1,0\.7\]\\beta\_\{d\}\\in\[0\.1,0\.7\]\. However,βd=0\.9\\beta\_\{d\}\{=\}0\.9causes EM to collapse to \.38—when the loss is dominated by the teacher distribution, the student loses the capacity to produce coherent independent judgments\.

ConfigurationAgr↑\\uparrowEM↑\\uparrowEMA decay\(βd=0\.5\\beta\_\{d\}\{=\}0\.5\)decay = 1\.0 \(no EMA\)\.890\.48decay = 0\.995\.911\.51decay = 0\.999 \(default\)\.900\.52decay = 0\.99\.918\.59JSD interpolationβd\\beta\_\{d\}\(decay=\{=\}\.999\)βd\\beta\_\{d\}= 0\.1\.905\.53βd\\beta\_\{d\}= 0\.3\.900\.54βd\\beta\_\{d\}= 0\.5 \(default\)\.900\.52βd\\beta\_\{d\}= 0\.7\.900\.52βd\\beta\_\{d\}= 0\.9\.867\.38Table 15:Hyperparameter ablation \(Qwen3\-14B, ResearchQA,ContractAllEM\)\. Default: decay=\\,\{=\}\\,\.999,βd=0\.5\\beta\_\{d\}\{=\}0\.5\.
#### The preservation loss is necessary for structural integrity\.

Settingα=0\\alpha=0\(no preservation loss\) causes the model to lose the ability to produce structural outputs within tens of training steps; evenα=0\.1\\alpha=0\.1makes a difference\. Unlike the EMA andβd\\beta\_\{d\}parameters \(Table[15](https://arxiv.org/html/2608.14684#A6.T15)\), the preservation coefficient is not a tunable dial but a binary structural requirement\.

## Appendix GOutput Token Statistics

Table[16](https://arxiv.org/html/2608.14684#A7.T16)reports the average number of generated tokens per sample in joint mode\. SARA\-trained models produce longer outputs due to more detailed per\-rubric analyses\. The increase is proportional to the average rubric count: HealthBench \(avg 11\.5 rubrics\) shows the largest absolute increase, while FLASK \(3 rubrics\) shows the smallest\.

HBRQAFLASKBase\+SBase\+SBase\+SQwen3\-8B405611453691230290Qwen3\-14B425659480764253319Qwen3\-32B466762501821262594Llama\-3\.1\-8B273465405520200209AvgΔ\\Delta\+220\+259\+117Table 16:Average output tokens per sample \(joint mode\)\. \+S = \+SARA\.Agr↑\\uparrowκ\\kappa↑\\uparrowMAD↓\\downarrowEM↑\\uparrowInstruct modeQwen3\-14B\.768\.658\.372\.23\+SARA\.900\.879\.113\.52Thinking modeQwen3\-14B\.765\.687\.323\.22\+SARA\.817\.771\.233\.29Table 17:Overall consistency on ResearchQA\. Thinking\-mode SARA improves over the base but underperforms instruct\-mode SARA\.
## Appendix HCase Study: Full Outputs

Table[18](https://arxiv.org/html/2608.14684#A8.T18)shows the complete joint\-mode outputs for the case study in Table[7](https://arxiv.org/html/2608.14684#S5.T7)\. The task asks to fill in the blank: “\_\_\_\_\_\_\_ is the direct attempt to formally or informally manage ethical issues or problems, through specific policies, practices and programmes\.” The assistant responds: “Let’s think step by step\. We refer to Wikipedia articles on business ethics for help\. The direct attempt to manage ethical issues through specific policies, practices, and programs is business ethics management\.”

Base Model \(joint\)\+SARA \(joint\)R1 Analysis: The response attempts to address the instruction by identifying the concept as “business ethics management\.” However, it does not fully answer the question, which is asking for the term that fills in the blank\. The response is somewhat relevant but misses the key requirement of directly providing the term that completes the sentence\.Verdict: 4/5\.\(iso: 3\)R1 Analysis: The response attempts to address the question by identifying “business ethics management” as the concept being described\. However, it does not fully or clearly define the term, nor does it provide a complete or precise answer to the question\. The response is somewhat relevant but lacks the clarity and completeness expected\.Verdict: 3/5\.\(iso: 3\)R2 Analysis: The responsedoes not provide anyfactual evidence or citation to support the claim that the term is “business ethics management\.” While the concept is plausible, there isno reliable sourceor background knowledge explicitly referenced to confirm its accuracy\.Verdict: 2/5\.\(iso: 3\)R2 Analysis: The assistant’s response states that the direct attempt to manage ethical issues through specific policies, practices, and programs is “business ethics management\.” However, the response lacks any citation or reference to a reliable source such as Wikipedia or any other authoritative text\. While the concept may be accurate, the support for the response is incomplete and not fully reliable\.Verdict: 3/5\.\(iso: 3\)R3 Analysis: The response is short and straightforward, with minimal ambiguity\. It is generally understandable, though it could be more clearly structured to directly answer the question by providing the term in the blank\.Verdict: 4/5\.\(iso: 4\)R3 Analysis: The response is clear and well\-structured, beginning with a statement that acknowledges the need to think step by step and referencing a source \(Wikipedia articles on business ethics\) for support\. The answer directly addresses the question and provides a concise explanation\. The organization is logical and easy to follow\.Verdict: 5/5\.\(iso: 5\)Total: 4\+2\+4=10Total: 3\+3\+5=11Table 18:Full joint\-mode outputs for the case study\. The highlighted region in the base model’s R2 shows absolutist phrasing \(“does not provide*any*”\) absent from its own isolation output, where it acknowledges partial accuracy\. Parenthetical annotations show each model’s isolation verdict for comparison\. SARA matches isolation on all three rubrics; the base model deviates on R1 \(\+1\) and R2 \(–1\)\.
## Appendix IAdapting SARA to Thinking Mode

Recent models support athinking modein which extended reasoning occurs inside a designated thinking block, and only the final answer appears in the visible output\. This section describes how we adapt SARA’s segment extraction and training to this setting, and reports preliminary results\.

### I\.1Differences from Instruct Mode

In instruct mode, SARA parses per\-rubric segments from the structured output using a simple regex that matches the “\*\*RubrickkAnalysis\*\*: \.\.\. \*\*Verdict\*\*: \.\.\.” pattern\. Both the reasoning and the verdict are visible in the output, making segment boundaries unambiguous\.

Thinking mode changes this in two ways:

1. 1\.Reasoning moves to the thinking block\.The output prompt instructs the model to perform all evaluation internally and produce only the final scores\. The visible output reduces to a single line of scores \(e\.g\., “4\+3\+5\+\.\.\.=<sum\>”\)\.
2. 2\.No structured segment boundaries\.The thinking block contains free\-form reasoning without enforced formatting\. Rubric analyses may be interleaved with reflection, planning, and self\-correction steps\.

### I\.2Segment Extraction

Because the thinking block lacks rigid formatting, we use a paragraph\-level heuristic\. We split the thinking content into paragraphs \(by double newlines\) and scan each paragraph for rubric mentions using the pattern “\[Rr\]ubric\\\\backslashs\+\(\\\\backslashd\+\)”\. For each rubrickk, we assign thefirstparagraph that mentions it as that rubric’s analysis segment\. Paragraphs that mention no specific rubric \(typically planning, reflection, or summary paragraphs\) are assigned to the structure mask\.

This heuristic is less precise than instruct\-mode parsing\. A paragraph may discuss multiple rubrics, and some rubric reasoning may be distributed across non\-contiguous paragraphs\. We accept this noise as a trade\-off for applicability\.

### I\.3Training Adjustments

We reduce the preservation loss weight fromα=0\.4\\alpha=0\.4\(instruct mode\) toα=0\.2\\alpha=0\.2\. The thinking block contains substantial reflection and summary content beyond pure structural tokens\. A high preservation weight on these paragraphs would overly constrain the model’s reasoning process, reducing the distillation signal’s ability to correct interference in the analysis segments\.

### I\.4Preliminary Results

We train SARA on Qwen3\-14B in thinking mode using ResearchQA and compare against the instruct\-mode results from the main experiments \(Table[17](https://arxiv.org/html/2608.14684#A7.T17)\)\.

Thinking\-mode SARA improves over the base model \(EM \.23→\\to\.29,κ\\kappa\.658→\\to\.771\) but substantially underperforms instruct\-mode SARA \(EM \.52,κ\\kappa\.879\)\. We attribute this gap to two factors:

#### Noisy segment boundaries\.

Paragraph\-level extraction is coarser than regex\-based parsing\. Misaligned segments mean the distillation loss sometimes aligns the wrong tokens with isolation anchors, diluting the training signal\.

#### Entangled reasoning\.

In thinking mode, the model often interleaves reasoning about multiple rubrics within a single paragraph, or revisits earlier rubrics during later reflection steps\. This entanglement makes it harder to isolate per\-rubric reasoning for targeted correction\.

### I\.5Discussion

These results suggest that SARA’s core principle \(using isolation judgments as anchors\) applies across output formats, but the effectiveness depends heavily on segment extraction quality\. Improving thinking\-mode parsing, for example through training the model to structure its thinking block with explicit rubric delimiters, or through attention\-based attribution methods, remains an open direction\.

## Appendix JDeclaration of AI Assistant Usage

We explicitly disclose the nature and scope of AI assistance in this work:

- •Writing and Refinement:We used Claude \(Anthropic\) strictly as a writing assistant to verify grammar, polish writing style, and refine the text of the manuscript\.
- •Coding Assistance:We employed Claude to draft routine boilerplate data processing scripts and infrastructure execution modules\. However, the human authors entirely conceptualized and developed all core SARA algorithms, distillation losses, and custom pipeline code\.
- •Human Ownership:The human authors maintain sole ownership of all core scientific ideas, mathematical formulations, experimental designs, and data interpretations\. We manually created all primary data visualizations, plots, and tables without any AI generation\.

The use of GPT\-4\.1, as detailed in Section[5](https://arxiv.org/html/2608.14684#S5), serves strictly as an automated reference baseline/judge within our evaluation framework, rather than an interactive assistant in our research or writing process\.