The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations

# The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Source: [https://arxiv.org/html/2608.18300](https://arxiv.org/html/2608.18300)
## The Lifecycle of LLM\-as\-a\-Judge for Large\-Scale Recommendation Explanations CCS:Computing methodologies Natural language generationCCS:Information systems Recommender systems

,JJ TanAffiliation:Netflix,Los Gatos,California,USAemail:[jjt@netflix\.com](mailto:[email protected]),Ishan GuptaNote:Work done while at Netflix\.Affiliation:email:[345ishaan@gmail\.com](mailto:[email protected]),Lars OldsAffiliation:Netflix,Los Gatos,California,USAemail:[golds@netflix\.com](mailto:[email protected]),Claire CampbellAffiliation:Netflix,Los Gatos,California,USAemail:[ccampbell@netflix\.com](mailto:[email protected]),David FagnanAffiliation:Netflix,Los Gatos,California,USAemail:[dfagnan@netflix\.com](mailto:[email protected]),Veli BalinAffiliation:Netflix,Los Gatos,California,USAemail:[vbalin@netflix\.com](mailto:[email protected]),Rohan GosainAffiliation:Netflix,Los Gatos,California,USAemail:[rgosain@netflix\.com](mailto:[email protected]),Louis GarciaAffiliation:Netflix,Los Gatos,California,USAemail:[louisg@netflix\.com](mailto:[email protected])andMinsu JangAffiliation:Netflix,Los Gatos,California,USAemail:[minsuj@netflix\.com](mailto:[email protected])

###### Abstract\.

LLM\-as\-a\-Judge, which leverages a large language model to evaluate natural language generated by another AI application or model, has become a standard, scalable approach for accelerating and extending costly human evaluation\. However, most work treats a judge as a static artifact, evaluating it once at construction or against a fixed benchmark\. In contrast, we argue that an LLM judge running in a production system is better understood as having a*lifecycle*: it must be built, trained, deployed, and continuously maintained as the surrounding data evolves, and each phase poses distinct technical and operational challenges\.

We present such a lifecycle for the LLM judges that evaluate user\-facing recommendation explanations at Netflix, where our pipeline generates and the judges assess hundreds of thousands of distinct show\-level explanations per week, served across the mobile experience to millions of members\. Our framework has four phases:\(I\) Birth, defining multiple evaluation criteria and building curated benchmark datasets with human labels and rationales;\(II\) Training, refining the judges’ rubrics via*Reasoning\-Aligned Rubric Tuning*\(RART\), a rubric\-tuning procedure that uses a meta\-judge over reasoning output as the learning signal;\(III\) Deployment, in which one judge serves two production roles: quality gating and reflective generation; and\(IV\) Monitoring, a continuous Human\-in\-the\-Loop alignment process that detects drift and triggers re\-tuning behind a human review gate\. We report*post\-launch*results from a five\-week A/B test over tens of millions of members, in which the judge\-aligned explanations shifted member viewing toward novel content \(previously unwatched\) and increased successful browse\-to\-play sessions relative to a no\-explanation control, with no quality\-related takedowns\.

###### Keywords:

LLM\-as\-a\-Judge, recommendation explanations, production machine learning, human\-in\-the\-loop evaluation, rubric tuning

## 1\.Introduction

A recommendation explanation, a short natural\-language message accompanying a recommended item, shapes how users perceive and trust a recommender system\. At Netflix, an explanation is member\-facing readable evidence for*why*a title was recommended based on what the member has previously watched\. We focus on*similarity\-based*explanations, which connect a recommended title to one reference title the member has interacted with via shared attributes such as genre and tone, e\.g\., “A funny, heartfelt holiday romance about love and new beginnings, much likeMy Secret Santa\.” These explanations aim to drive*content discovery*and build*trust*for our members, which is also why quality matters\. A misleading or poorly grounded explanation erodes the very trust the feature is meant to build\.

Evaluating quality at scale is hard\. Our pipeline produces hundreds of thousands of distinct item\-level explanations per week that can be presented to millions of members\. Explanations must be accurate, item\-specific, and free of sensitive or offensive content\. However human evaluation, the gold standard, cannot cover them at this volume\. LLM\-as\-a\-Judge\([20](https://arxiv.org/html/2608.18300#bib.bib1);[9](https://arxiv.org/html/2608.18300#bib.bib2);[6](https://arxiv.org/html/2608.18300#bib.bib4)\)offers a scalable proxy, but a judge is not a one\-shot artifact\. It must be initialized against trusted human labels, tuned to align with humans, deployed where it acts on production traffic, and maintained as the upstream data drifts\. To our knowledge, no prior work characterizes all four phases as a single, instrumented lifecycle\.

We therefore treat the judge not as a fixed evaluator but as a*lifelong agent*: a persistent component that must continually*learn*from accumulating human feedback, remain*aligned*with human judgment for the right reasons rather than by coincidence, and*evolve*through standing alignment audits and automated re\-tuning as the catalog, recommendation algorithms, and user population shift\.

In this paper, we present a case study of an LLM judge for user\-facing recommendation explanations at Netflix, framed as a lifecycle \(Figure[1](https://arxiv.org/html/2608.18300#S2.F1)\) with four phases:\(I\) Birth\(§[4](https://arxiv.org/html/2608.18300#S4)\),\(II\) Training\(§[5](https://arxiv.org/html/2608.18300#S5)\),\(III\) Deployment\(§[6](https://arxiv.org/html/2608.18300#S6)\), and\(IV\) Monitoring\(§[7](https://arxiv.org/html/2608.18300#S7)\)\. Our contributions are:

1. \(1\)A lifecycle view of LLM judges in a large\-scale recommender system, covering all four phases end to end\.
2. \(2\)Reasoning\-Aligned Rubric Tuning \(RART\): a rubric\-refinement method that iteratively updates the judge’s rubric using a meta\-judge that scores the judge’s reasoning, with an ablation study showing how reasoning alignment improves the overall performance of LLM judges\.
3. \(3\)An operational design for running the judge in production: a two\-role deployment \(a quality gate and the critic in a self\-reflection revision loop\) paired with continuous drift monitoring that flags judge\-degradation events and triggers re\-tuning behind a human review gate\.
4. \(4\)Practical takeaways for industry to ship LLM judges in production by operating this lifecycle at scale\.

## 2\.Related Work

#### Explainable recommendation\.

Explaining*why*an item was recommended is a long\-standing concern in recommender systems, from early work on surfacing collaborative\-filtering neighborhoods\([4](https://arxiv.org/html/2608.18300#bib.bib10)\)to surveys that organize explanation aims and their evaluation\([14](https://arxiv.org/html/2608.18300#bib.bib11);[19](https://arxiv.org/html/2608.18300#bib.bib12)\)around goals such as transparency, trust, persuasiveness, and effectiveness\. Our similarity\-based explanations sit in the transparency\-and\-trust branch of that taxonomy, since they justify a recommendation by grounding it in a title the member has already watched\. What differs in our setting is not the explanation style but the evaluation problem it creates\. The explanations are free\-form natural language written by an LLM at catalog scale, so the template\-level correctness that earlier systems could guarantee by construction no longer holds, and quality must itself be measured by a model\. That is the gap the rest of this section addresses\.

#### LLM\-as\-a\-Judge\.

Using an LLM to score another model’s output has become a standard substitute for costly human evaluation in open\-ended generation tasks\([20](https://arxiv.org/html/2608.18300#bib.bib1);[9](https://arxiv.org/html/2608.18300#bib.bib2);[21](https://arxiv.org/html/2608.18300#bib.bib3);[6](https://arxiv.org/html/2608.18300#bib.bib4)\)\. LLM judges also carry systematic biases, including sensitivity to answer position and verbosity\([15](https://arxiv.org/html/2608.18300#bib.bib8)\)and a preference for their own generations\([11](https://arxiv.org/html/2608.18300#bib.bib9)\); this is one reason we anchor ours to human labels and rationales at every phase of its lifecycle\. Public benchmarks such as JudgeBench\([13](https://arxiv.org/html/2608.18300#bib.bib5)\)and RewardBench\([7](https://arxiv.org/html/2608.18300#bib.bib6)\)evaluate judges on general\-purpose correctness, safety, and instruction\-following, and are the standard against which new judges are reported\. These benchmarks are necessarily static and domain\-general; they say little about how a judge behaves once deployed against a live, drifting item catalog, and none of them treat the judge itself as something with a lifecycle that must be re\-validated after launch\. Our Phase I benchmark is built exactly to address that gap\. It is a domain\-specific, rationale\-annotated dataset for similarity\-based recommendation explanations, continuously refreshed from production traffic rather than fixed at construction time\.

#### Self\-refinement and reflection\.

Using a model’s own critique to improve its output without gradient updates has been explored both for the generator \(Self\-Refine\([10](https://arxiv.org/html/2608.18300#bib.bib13)\), Reflexion\([12](https://arxiv.org/html/2608.18300#bib.bib14)\)\) and, closer to our setting, for the judge itself\. Meta\-Rewarding\([16](https://arxiv.org/html/2608.18300#bib.bib7)\)adds a meta\-judge that scores the judge’s own reasoning to improve reward\-model quality\. Our reasoning meta\-judgeℳ\\mathcal\{M\}\(§[5\.3](https://arxiv.org/html/2608.18300#S5.SS3)\) builds directly on this idea, with two differences that matter for a production deployment\. First, it is grounded in human rationales rather than unsupervised self\-consistency\. Second, it deliberately focuses on fail examples\. The reason is that in Phase III the same judge’s rejection reason is fed back to the generator as a revision instruction, so a right\-verdict but wrong\-reason case would propagate a misleading signal downstream\. In an offline reward\-modeling setting it would not\.

#### Prompt and rubric optimization\.

Textual analogues of gradient\-based optimization revise a prompt or rubric from feedback rather than backpropagating through weights\. They span evolutionary prompt search \(EvoPrompt\([3](https://arxiv.org/html/2608.18300#bib.bib15)\), Promptbreeder\([2](https://arxiv.org/html/2608.18300#bib.bib16)\)\), textual gradients \(TextGrad\([17](https://arxiv.org/html/2608.18300#bib.bib17)\), ACE\([18](https://arxiv.org/html/2608.18300#bib.bib18)\)\), reflective evolution over a Pareto pool of candidates \(GEPA\([1](https://arxiv.org/html/2608.18300#bib.bib21)\), built on DSPy\([5](https://arxiv.org/html/2608.18300#bib.bib20)\)\), and rubric synthesis directly from chosen/rejected pairs\([8](https://arxiv.org/html/2608.18300#bib.bib19)\)\. RART \(§[5\.1](https://arxiv.org/html/2608.18300#S5.SS1)\) is an instance of this family, specialized to rubric tuning\. It is closest to GEPA in spirit, but it is a greedy, single\-objective special case rather than a Pareto\-pool search \(§[9](https://arxiv.org/html/2608.18300#S9)\)\. Its textual gradient is also computed over*rationale*mismatches, which helps it to target reasoning quality and not just label accuracy\.

None of this prior work evaluates a judge continuously after deployment or ties re\-tuning to a monitored drift signal; each treats training\-time alignment as the end state\. Our contribution is orthogonal to and compatible with these methods\. We show what changes when a judge is embedded as one node in a closed\-loop production system rather than scored once against a fixed benchmark\.

![A four-phase cycle diagram. Phase I builds a human-labeled benchmark dataset from expert-crafted examples, LLM-synthesized boundary examples, and sampled production explanations. Phase II tunes each criterion's rubric with a reflector LLM and a reasoning meta-judge. Phase III deploys the tuned judge as a guardrail inside a generate-judge-revise loop that either serves or drops each explanation. Phase IV samples served and gated explanations weekly for human review, compares judge and human labels, and re-triggers Phase II when drift exceeds a threshold.](https://arxiv.org/html/2608.18300v1/lifecycle.png)Figure 1\.The four\-phase lifecycle of an LLM judge for recommendation explanations\.\(I\) Birth\(§[4](https://arxiv.org/html/2608.18300#S4)\): build a benchmark of human\-labeled, rationale\-annotated examples, split into train/validation/test\.\(II\) Training\(§[5](https://arxiv.org/html/2608.18300#S5)\): a reflector LLM tunes each criterion’s rubric from judge–human label and*reasoning*mismatches until validation metrics pass\.\(III\) Deployment\(§[6](https://arxiv.org/html/2608.18300#S6)\): the judge gates production explanations and drives self\-reflective revision with bounded retries\.\(IV\) Monitoring\(§[7](https://arxiv.org/html/2608.18300#S7)\): a weekly human\-rated sample detects judge–human drift and feeds benchmark augmentation, re\-triggering \(II\) when drift exceeds threshold\.A four\-phase cycle diagram\. Phase I builds a human\-labeled benchmark dataset from expert\-crafted examples, LLM\-synthesized boundary examples, and sampled production explanations\. Phase II tunes each criterion's rubric with a reflector LLM and a reasoning meta\-judge\. Phase III deploys the tuned judge as a guardrail inside a generate\-judge\-revise loop that either serves or drops each explanation\. Phase IV samples served and gated explanations weekly for human review, compares judge and human labels, and re\-triggers Phase II when drift exceeds a threshold\.

## 3\.System Overview

Figure[1](https://arxiv.org/html/2608.18300#S2.F1)summarizes the four phases and the artifacts flowing between them; the per\-phase mechanics are detailed in the caption\. LLM judges play three roles: the optimization target of Phase II, the feedback provider and guardrail decision\-maker of Phase III, and the monitored object of Phase IV\. Human labels and rationales are collected in Phase I and then continuously augmented at∼\\sim300/week in Phase IV\. Phase IV closes two loops: \(i\) A fast drift\-detection loop that re\-triggers Phase II re\-training when judge–human agreement decays, and \(ii\) a slower benchmark\-augmentation loop that keeps the Phase I dataset representative of the live catalog\.

Our system generates and evaluates hundreds of thousands of explanations per week, passing over 75% with a retry budget of 3, and was verified through an online A/B test in §[6\.2](https://arxiv.org/html/2608.18300#S6.SS2)\.

## 4\.Phase I: Birth, Establishing Ground Truth

The Birth phase is the most human\-intensive of the four\. Our internal writing experts define the must\-have criteria, author the labeling guidelines for human raters, hand\-craft adversarial examples, and anchor human rating\. The guidelines are also used as the seed rubrics for the LLM judges that are optimized in Phase II\.

Public LLM\-judge benchmarks\([13](https://arxiv.org/html/2608.18300#bib.bib5);[7](https://arxiv.org/html/2608.18300#bib.bib6)\)target general\-purpose evaluation and miss the constraints of recommendation explanations, which are item\-specific, context\-dependent, and short\. We therefore build a domain\-specific benchmark of similarity\-based explanations\. Each is conditioned on a target item and one or two reference items the user has interacted with in the past\. The criteria are split into*must\-have*pass/fail conditions that every explanation must satisfy before being shown, and softer stylistic and recommendation\-relevance criteria \(specific production criteria withheld for confidentiality\)\. For each criterion, the labeling guidelines, pass/fail conditions, and boundary examples are the sources of truth for our LLM judges and also guide human raters’ decisions\.

The benchmark dataset comprises three complementary sources: \(i\) expert\-crafted examples, each with a human rater’s pass/fail label, its known failure mode, and a rationale for failure, covering cases the other two sources cannot reliably produce; \(ii\) human\-rated, LLM\-synthesized examples near the criterion boundary, capturing difficult cases that production sampling rarely surfaces; and \(iii\) sampled explanations from our explanation\-generation system, collected before the production system was fully integrated\.

Before launch the benchmark contained roughly 900 human\-labeled explanations, split close to evenly between the two labels with a slight majority offail\(∼\\sim54%\)\. We deliberately hold the two classes near balance rather than matching production prevalence, where failures are far rarer\. Specificity \(Eq\.[1](https://arxiv.org/html/2608.18300#S5.E1)\),RAneg\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}\(Eq\.[3](https://arxiv.org/html/2608.18300#S5.E3)\) and the training dataset of Algorithm[1](https://arxiv.org/html/2608.18300#alg1)all depend on the human\-failed examples, where production sample would be too sparse\. The consequence for interpretation is that the alignment metrics of §[5\.2](https://arxiv.org/html/2608.18300#S5.SS2)measure agreement on a class\-balanced, difficulty\-enriched benchmark and are not defect rates on live traffic\. They are also reported per criterion\. Because the production gate requires an explanation to pass*every*must\-have criterion \(§[6\.1](https://arxiv.org/html/2608.18300#S6.SS1)\), end\-to\-end gate behavior does not follow directly from any single criterion’s value\.

The dataset is not fixed\. The Phase IV HITL pipeline \(§[7](https://arxiv.org/html/2608.18300#S7)\) appends∼\\sim300 freshly rated explanations each week, tracking distribution shift in the live catalog, so the counts above are the launch\-time starting point rather than the current size\.

Human raters are trained and calibrated by our Human Data Operations team, whose calibration protocol is internal and not detailed here\. Every fail label carries a free\-text rationale against the guideline\.

## 5\.Phase II: Training, Aligning the Judge to Humans

We train our judge using self\-reflection by iteratively refining each criterion’s rubric against human labels and their failure rationales\. We call this procedureReasoning\-Aligned Rubric Tuning\(RART\)\. One key component of RART is to ensure*reasoning alignment*between the LLM judge and human raters\.

### 5\.1\.Reasoning\-Aligned Rubric Tuning \(RART\)

Each LLM judge is instantiated from a small, task\-invariant prompt template comprising a fixed system message describing the recommendation explanation evaluation task plus a templated<criterion\>slot filled with the rubric being tuned, and a user message containing the explanation text, target\-item metadata, and reference\-item metadata\. The judge emits a JSON object \{label,reason\} for every explanation\. Tuning therefore reduces to refining the criterion rubrics; the surrounding prompt is held constant\.

Given a rubricRtR\_\{t\}at iterationtt, RART iterates the loop in Algorithm[1](https://arxiv.org/html/2608.18300#alg1): it scores the data using the current judge withRtR\_\{t\}, checks the validation metrics for early stopping, and otherwise asks a reflector LLMℛ\\mathcal\{R\}to propose a refined rubric from the judge’s errors\. Crucially, an “error” is not only a wrong label\. Before reflecting, a*rationale meta\-judge*ℳ\\mathcal\{M\}takes the examples that*both*the judge and the human labeled as*fail*and compares the judge’s free\-text reason against the human rater’s rationale \(§[5\.3](https://arxiv.org/html/2608.18300#S5.SS3)\)\. The reflector therefore targets two types of error: examples the judge mislabels; and agreed\-fail examples where the judge reaches the right verdict for the wrong reason\. We writeℓJ​\(x\),ℓH​\(x\)\\ell\_\{J\}\(x\),\\ell\_\{H\}\(x\)for the judge and human labels on examplexx, andrJ​\(x\),rH​\(x\)r\_\{J\}\(x\),r\_\{H\}\(x\)for their respective reasons\.

### 5\.2\.Alignment Metrics

We use three metrics aggregated over the benchmark dataset\. WriteℓJ​\(x\),ℓH​\(x\)∈\{pass,fail\}\\ell\_\{J\}\(x\),\\ell\_\{H\}\(x\)\\in\\\{\\textsc\{pass\},\\textsc\{fail\}\\\}for the judge and human labels on examplexx\.

Specificity\(*fail\-recall*\) is the judge’s rate of correctly rejecting bad explanations\. It is our most critical metric, since bad explanations that slip past the gate directly erode member trust:

\(1\)Spec=\|\{x:ℓJ\(x\)=fail,ℓH\(x\)=fail\}\|\|\{x:ℓH​\(x\)=fail\}\|\.\\mathrm\{Spec\}\\;=\\;\\frac\{\|\\\{x:\\ell\_\{J\}\(x\)=\\textsc\{fail\},\\,\\ell\_\{H\}\(x\)=\\textsc\{fail\}\\\}\|\}\{\|\\\{x:\\ell\_\{H\}\(x\)=\\textsc\{fail\}\\\}\|\}\.
Recall\(*pass\-recall*\) is the judge’s rate of correctly passing good explanations\. High recall is also needed to keep production explanation coverage high and revision cost low:

\(2\)Rec=\|\{x:ℓJ\(x\)=pass,ℓH\(x\)=pass\}\|\|\{x:ℓH​\(x\)=pass\}\|\.\\mathrm\{Rec\}\\;=\\;\\frac\{\|\\\{x:\\ell\_\{J\}\(x\)=\\textsc\{pass\},\\,\\ell\_\{H\}\(x\)=\\textsc\{pass\}\\\}\|\}\{\|\\\{x:\\ell\_\{H\}\(x\)=\\textsc\{pass\}\\\}\|\}\.
Algorithm 1Reasoning\-Aligned Rubric Tuning1:initial rubric

R0R\_\{0\}; train/val/test sets

𝒟tr,𝒟val,𝒟te\\mathcal\{D\}\_\{\\text\{tr\}\},\\mathcal\{D\}\_\{\\text\{val\}\},\\mathcal\{D\}\_\{\\text\{te\}\}; max iterations

NN
2:Judges:rubric\-conditioned judge

J⁡\(R\):x↦\(ℓJ,rJ\)J\(R\)\\\!:\\\!x\\mapsto\(\\ell\_\{J\},r\_\{J\}\)\(label, rationale\); rationale meta\-judge

ℳ⁡\(rJ,rH\)→\{agree,mismatch\}\\mathcal\{M\}\(r\_\{J\},r\_\{H\}\)\\\!\\to\\\!\\\{\\textsc\{agree\},\\textsc\{mismatch\}\\\}; reflector

ℛ⁡\(R,focus\)→R′\\mathcal\{R\}\(R,\\textit\{focus\}\)\\\!\\to\\\!R^\{\\prime\}\(revised rubric\)

3:tuned rubric

R⋆R^\{\\star\}
4:

R⋆←R0R^\{\\star\}\\leftarrow R\_\{0\};

s⋆←−∞s^\{\\star\}\\leftarrow\-\\infty
5:for

t=0t=0to

N−1N\-1do

6:Score

𝒟tr,𝒟val\\mathcal\{D\}\_\{\\text\{tr\}\},\\mathcal\{D\}\_\{\\text\{val\}\}with

J⁡\(Rt\)J\(R\_\{t\}\)⊳\\trianglerighteachxxgets\(ℓJ,rJ\)\(\\ell\_\{J\},r\_\{J\}\)

7:

s←s\\leftarrowweighted alignment metrics on

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}\(§[5\.2](https://arxiv.org/html/2608.18300#S5.SS2)\)

8:if

s\>s⋆s\>s^\{\\star\}then

9:

R⋆←RtR^\{\\star\}\\leftarrow R\_\{t\};

s⋆←ss^\{\\star\}\\leftarrow s
10:endif

11:ifall metrics clear the per\-criterion boundthen

12:break

13:endif

14:

𝒳←\{x:ℓJ​\(x\)≠ℓH​\(x\)\}\\mathcal\{X\}\\leftarrow\\\{x:\\ell\_\{J\}\(x\)\\neq\\ell\_\{H\}\(x\)\\\}⊳\\trianglerightlabel mismatches

15:

𝒩←\{x:ℓJ​\(x\)=ℓH​\(x\)=fail\}\\mathcal\{N\}\\leftarrow\\\{x:\\ell\_\{J\}\(x\)=\\ell\_\{H\}\(x\)=\\textsc\{fail\}\\\}⊳\\trianglerightagreed\-fail

16:

𝒩×←\{x∈𝒩:\\mathcal\{N\}\_\{\\\!\\times\}\\leftarrow\\\{x\\in\\mathcal\{N\}:ℳ\(rJ\(x\),rH\(x\)\)=mismatch\}\\mathcal\{M\}\(r\_\{J\}\(x\),r\_\{H\}\(x\)\)=\\textsc\{mismatch\}\\\}
17:⊳\\trianglerightagreed\-fail, wrong reason

18:

focus←𝒳∪𝒩×\\textit\{focus\}\\leftarrow\\mathcal\{X\}\\cup\\mathcal\{N\}\_\{\\\!\\times\}⊳\\trianglerightfocus set

19:

Rt\+1←ℛ⁡\(Rt,focus\)R\_\{t\+1\}\\leftarrow\\mathcal\{R\}\(R\_\{t\},\\ \\textit\{focus\}\)⊳\\trianglerightsample candidate rubric

20:endfor

21:return

R⋆R^\{\\star\}

Reasoning agreement rate\(RAneg\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}\), the reasoning\-agreement rate over the total failure examplesℱ=\{x:ℓH​\(x\)=fail\}\\mathcal\{F\}=\\\{x:\\ell\_\{H\}\(x\)=\\textsc\{fail\}\\\}: the fraction on which the judge and human not only both reject the explanation but also agree on*why*it fails\. The numerator is restricted to the*agreed\-fail*set𝒩=\{x:ℓJ​\(x\)=ℓH​\(x\)=fail\}⊆ℱ\\mathcal\{N\}=\\\{x:\\ell\_\{J\}\(x\)=\\ell\_\{H\}\(x\)=\\textsc\{fail\}\\\}\\subseteq\\mathcal\{F\}, since reasoning can only agree where both labels arefail; the denominator is the full set of human\-failed examples\. Reasoning agreement is judged by the rationale meta\-judgeℳ\\mathcal\{M\}:

\(3\)RAneg=\|\{x∈𝒩:ℳ⁡\(rJ​\(x\),rH​\(x\)\)=agree\}\|\|ℱ\|\.\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}=\\frac\{\|\\\{x\\in\\mathcal\{N\}:\\mathcal\{M\}\(r\_\{J\}\(x\),r\_\{H\}\(x\)\)=\\textsc\{agree\}\\\}\|\}\{\|\\mathcal\{F\}\|\}\.
For rubric*optimization*during RART we collapse these into a single weighted score

\(4\)s=ws​Spec\+wr​Rec\+wr​a​RAneg\.s\\;=\\;w\_\{s\}\\,\\mathrm\{Spec\}\+w\_\{r\}\\,\\mathrm\{Rec\}\+w\_\{ra\}\\,\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}\.We weight specificity above the other two terms,ws=3w\_\{s\}=3andwr=wr​a=1w\_\{r\}=w\_\{ra\}=1, because the false positive and false negative errors are not symmetric in production due to their risks\. A bad explanation that escapes from gating will be served to members, whereas a good one that fails will only be revised or dropped \(§[6\.1](https://arxiv.org/html/2608.18300#S6.SS1)\)\. TheRAneg\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}term additionally favors rubrics whose rejections are backed by the right reason rather than by coincidence\.

### 5\.3\.Reasoning\-Aligned Self\-Reflection

Figure 2\.Alignment\-metric lift over the default rubric on the test set for RART vs\. vanilla \(no reasoning alignment\) \(n=8n\{=\}8seeds, each reshuffling the train/validation/test split; bars show mean, error bars±\\pmstd\)\. RART = rationale\-aware reflection; vanilla = label\-only reflection\. All quantities areΔ=tuned−default\\Delta=\\text\{tuned\}\-\\text\{default\}; positive means improvement\. Significance onΔ\\Delta\(RART−\-vanilla\) by two\-sided sign test:†p<0\.10p\{<\}0\.10,∗p<0\.05p\{<\}0\.05,∗∗p<0\.01p\{<\}0\.01\.Three grouped bar panels, one per criterion, comparing RART against the label\-only vanilla ablation on lift over the default rubric for specificity, recall, and reasoning agreement\. RART shows the larger specificity and reasoning\-agreement lift on criteria 1 and 3; the two methods are close on criterion 2, where the default rubric is already near ceiling\.Treating every disagreement uniformly ignores that mismatches differ in information content\. The judge may agree on the label for the wrong reason, or disagree on the label while its rationale exposes a genuine rubric ambiguity\. Building on the*meta\-judge*idea\([16](https://arxiv.org/html/2608.18300#bib.bib7)\), we introduce a*reasoning meta\-judge*ℳ\\mathcal\{M\}that compares the judge’s reason against the human rationale and returnsrationale\_agreementorrationale\_mismatch\. We runℳ\\mathcal\{M\}only on the agreed\-fail examples, where a shared “fail” label can still have divergent reasons\. The rationale mismatches it finds become the reflector’s focus set, together with the label mismatches\. Agreements are left untouched, so the rubric preserves what the judge already gets right\.

Restricting the meta\-judge to agreed\-fail examples is also a deployment requirement\. In Phase III \(§[6](https://arxiv.org/html/2608.18300#S6)\) the same judge is the critic in a self\-reflection loop, where its rejection reason becomes the generator’s revision instruction\. A right\-verdict but wrong\-reason rejection therefore propagates a misleading signal\. Tuning on agreed\-fail examples drives the judge toward the*right reason*for each rejection\. Unlike Meta\-Rewarding\([16](https://arxiv.org/html/2608.18300#bib.bib7)\), whose meta\-judge is unsupervised, ours is grounded in human rationales\. The signal is a text “gradient” in the spirit of TextGrad\([17](https://arxiv.org/html/2608.18300#bib.bib17)\), ACE\([18](https://arxiv.org/html/2608.18300#bib.bib18)\), and GEPA\([1](https://arxiv.org/html/2608.18300#bib.bib21)\), but over rationales rather than generations, and complementary to rubric synthesis from chosen/rejected pairs\([8](https://arxiv.org/html/2608.18300#bib.bib19)\)\. Folding it into GEPA’s feedback function is a natural next step \(§[9](https://arxiv.org/html/2608.18300#S9)\)\.

#### Validating the meta\-judge\.

ℳ\\mathcal\{M\}is itself an LLM judge, and it provides both our reasoning\-agreement metric \(RAneg\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}\) and the training signal for RART\. We therefore validate it against humans directly\. Trained human raters independently labeled rationale agreement on a sample of 300 agreed\-fail rationale pairs, andℳ\\mathcal\{M\}’s verdicts matched their judgments98\.6%98\.6\\%of the time\. This supports usingℳ\\mathcal\{M\}as both an evaluation metric and a learning signal\.

To isolate the contribution of the reasoning signal, we compare RART \(rationale\-aware reflection\) against*vanilla*, an identical loop whose reflector sees only label mismatches \(Figure[2](https://arxiv.org/html/2608.18300#S5.F2);88seeds, held\-out test\)\. RART improves our most important metric,*specificity*, more than vanilla wherever the default rubric leaves headroom\. On criterion 1 this comes with a recall cost \(Figure[2](https://arxiv.org/html/2608.18300#S5.F2)\)\. This is acceptable because a falsely rejected explanation re\-enters the revision loop \(§[6\.1](https://arxiv.org/html/2608.18300#S6.SS1)\) and will be regenerated, whereas a bad explanation that reaches members cannot be recalled\. For criterion 3, label\-only vanilla caused specificity and reasoning agreement to collapse in every iteration\. The best\-checkpoint rule of Algorithm[1](https://arxiv.org/html/2608.18300#alg1)therefore returned essentially the default rubric, so its lift is≈0\\approx 0\. RART consistently lifts both specificity and reasoning agreement, with even a slight improvement on recall\. For criterion 2 the default rubric is already near ceiling, and RART and vanilla are indistinguishable\. Reasoning alignment therefore helps where there is room and does not hurt where there is none\. Across all three criteria, how much headroom the default rubric leaves is what governs how much RART can gain\.

## 6\.Phase III: Deployment, Putting the Judge to Work

The tuned judge from §[5](https://arxiv.org/html/2608.18300#S5)serves two production roles, both running on every generated explanation: \(a\) a*gate*that rejects explanations failing a configurable subset of criteria, and \(b\) the*critic*in a self\-reflective revision loop that returns flagged explanations to the generator for revision\. Explanations are generated per recommended item rather than per \(user, item\) pair; an online personalization model then selects the best\-fitting explanation for each \(user, item\) pair\. A single explanation can therefore be shared across many users to whom that item is surfaced, multiplying its impression footprint into the millions per day\.

### 6\.1\.Judge as Guardrail with Bounded\-Retry Revision

In production every explanation passes through a*generate→\\rightarrowjudge→\\rightarrowrevise*loop with a fixed retry budget\. The generator drafts an explanation conditioned on the user, target item, and reference item\(s\); the tuned judge \(§[5](https://arxiv.org/html/2608.18300#S5)\) scores it against the must\-have criteria of §[4](https://arxiv.org/html/2608.18300#S4), emitting a pass/fail label and reason per criterion\. If all criteria pass, the explanation is*served*; if any fails, the judge’s reason is appended to the generator prompt and the generator is re\-invoked, up toKKretries; an explanation still failing afterKKattempts is*dropped*\. Throughout, we writekkfor the retry index andKKfor the retry budget deployed in production\.

Dropping rather than serving a flagged explanation is a deliberate asymmetric choice\. A bad explanation is a trust hazard \(§[5\.2](https://arxiv.org/html/2608.18300#S5.SS2)\); the absence of an explanation merely forgoes an opportunity\. The retry\-then\-drop policy lets the generator self\-correct on most failures while preserving the guardrail\.

#### Cumulative pass rate vs\. revision budget\.

Figure[3](https://arxiv.org/html/2608.18300#S6.F3)reports cumulative judge pass rate versus retry budgetkkon a production sample ofn=1000n\{=\}1000explanations across four generator models\. Pass rate rises monotonically inkksince the judge’s free\-textreason\(§[5\.1](https://arxiv.org/html/2608.18300#S5.SS1)\) steers the next draft, but returns diminish fast\. The three strong generators capture≥80%\\geq 80\\%of their achievable lift byk=3k\{=\}3–44, which motivates a small fixedKK\. Revision only partly rescues weak generators \(model 3 stays below 50% even atk=12k\{=\}12\), so a sustained drop in thek=0k\{=\}0pass rate signals a generator\-side regression rather than judge drift\. For production we deployK=3K\{=\}3with the strongest model, which already captures most of the achievable lift while bounding per\-explanation generation cost\. At this budget, the end\-to\-end generate\-judge\-revise pipeline runs at a few thousand US dollars per week in model inference cost at this weekly volume\.

![Line chart of cumulative judge pass rate against revision budget k from 0 to 12 for four generator models. Three curves rise steeply through k equals three or four and then flatten at high pass rates. The fourth model remains below fifty percent across the whole range.](https://arxiv.org/html/2608.18300v1/self-reflection-as-revision.jpg)Figure 3\.Cumulative judge pass rate vs\. revision budgetkkonn=1000n\{=\}1000production explanations across four generator models\. Gains are monotonic but flatten beyondk≈4k\{\\approx\}4\. The weakest generator \(model 3\) stays well below the others\. This implies judge\-guided revision amplifies a capable generator rather than substituting for one\.Line chart of cumulative judge pass rate against revision budget k from 0 to 12 for four generator models\. Three curves rise steeply through k equals three or four and then flatten at high pass rates\. The fourth model remains below fifty percent across the whole range\.

### 6\.2\.Online A/B Evaluation

High offline judge–human alignment and low judged defect rates are*necessary*for explanations to benefit users, but not*sufficient*\. They only certify that served explanations meet our quality bar, not that having explanations improves user experience\. To test the latter directly, we ran a large\-scale A/B test on the mobile surface, comparing the judge\-aligned explanation pipeline \(§[5](https://arxiv.org/html/2608.18300#S5)–[6](https://arxiv.org/html/2608.18300#S6)\) against a no\-explanation control over tens of millions of members for 5 weeks\. Treatment shifted member viewing toward*novel content*, i\.e\., titles the member had not previously watched, by\+0\.2%\+0\.2\\%relative, a statistically significant increase \(p<0\.05p<0\.05\)\. This is consistent with explanations helping members discover unfamiliar titles they are more likely to enjoy, rather than only reinforcing already\-familiar viewing\. We also observed a\+0\.3%\+0\.3\\%relative increase in*sessions with a successful play*\(p<0\.05p<0\.05\), which indicates that members more often found something to watch during a browse session\. We read this as reduced browse friction, a proxy for a better platform discovery experience\. These relative lifts are small in absolute value, but at our scale of millions of members they correspond to a substantial aggregate effect\. Movements of this magnitude are also considered meaningful for a feature\-level intervention on this surface\. Together, these results indicate that judge\-aligned explanations are not just*quality\-compliant*but also*useful*\. They measurably change what members watch and how often they succeed in finding something to watch\.

As further supporting evidence, we observed no user\-initiated takedowns or escalations related to explanation quality during the online test, consistent with the system’s robustness in production\.

## 7\.Phase IV: Monitoring, Keeping the Judge Aligned

A judge well\-aligned at deployment will not stay aligned\. The catalog shifts \(new categories, seasonal content\), and the meaning of “high\-quality” itself might evolve over time\. Phase IV closes the loop\. Each week we draw a∼\\sim300\-explanation sample from the production explanation pool for Human\-in\-the\-Loop evaluation, stratified across the judge’s decision outcomes \(*served without revision*,*served after revision*,*dropped*\) and biased toward newer catalog items where drift is most likely\. The stratification distribution is set fixed from week to week, so successive weeks are measured on the similar composition and are comparable\. Human raters label them under the guidelines \(§[4](https://arxiv.org/html/2608.18300#S4)\)\. The relabeled sample is used for drift detection and is also appended to the train/validation pool, keeping the benchmark live \(the*continuous benchmark augmentation*in Figure[1](https://arxiv.org/html/2608.18300#S2.F1)\)\. We subsample the augmentation to preserve the benchmark’s class balance \(§[4](https://arxiv.org/html/2608.18300#S4)\)\. Each sampled explanation is reviewed by at least three human raters, and we take the majority label as ground truth\. The panel gives us both a consensus reference and a direct measure of how much human raters disagree on that sample, and we use the disagreement to set the bar for the judge\.

For each alignment metricMMof §[5\.2](https://arxiv.org/html/2608.18300#S5.SS2), we score the judge and each individual rater against the same majority label on the same weekly sample\. This yields one valueM⁡\(J\)M\(J\)for the judge and a setM⁡\(H\)=\{M⁡\(h1\),…,M⁡\(hR\)\}M\(H\)=\\\{M\(h\_\{1\}\),\\dots,M\(h\_\{R\}\)\\\}of per\-rater values,R≥3R\\geq 3, whose mean and standard deviation we compute directly across raters\. We require

\(5\)M⁡\(J\)≥mean⁡\(M⁡\(H\)\)−2​sd​\(M⁡\(H\)\),M\(J\)\\;\\geq\\;\\mathrm\{mean\}\\big\(M\(H\)\\big\)\\;\-\\;2\\,\\mathrm\{sd\}\\big\(M\(H\)\\big\),that is, the judge must score no worse than two standard deviations below the average rater, where the standard deviation measures disagreement among the raters themselves\. Falling below the band on any metric raises an*alert*\.

Equation[5](https://arxiv.org/html/2608.18300#S7.E5)is deliberately not a fixed threshold\. A harder weekly sample raises human disagreement, which widenssd⁡\(M⁡\(H\)\)\\mathrm\{sd\}\(M\(H\)\)and with it the acceptance band, so the judge is not penalized for difficulty that humans also find hard\. We apply the criterion both to the full weekly sample and separately to newly added titles, where catalog shift appears first and where a threshold calibrated on established content would misfire\. Requiring no degradation on the new titles is what makes the loop a shift detector rather than a generic regression test\.

A drift event triggers Phase II re\-tuning on the augmented benchmark\. The new rubric is then staged behind a manual review gate before deployment, with the previous rubric retained for rollback\. Across every weekly sample collected since launch, the judge has stayed within the human band on every metric, including on newly added titles, so the pipeline has stayed in its healthy operating region without requiring a triggered re\-tuning\. The loop still matters going forward because the catalog, recommender algorithms, and user population keep changing\. Thus today’s agreement says little about tomorrow’s\. Standing monitoring lets us catch and fix drift from our metrics rather than from degraded user\-facing explanations\.

Beyond the aggregate metric, the weekly human review has repeatedly surfaced failure patterns that a per\-criterion agreement score alone would not have flagged\. Some are borderline cases whose pass/fail depends on how confidently the explanation is phrased rather than on the underlying recommendation quality\. Others are concentrated in specific genres\. Stand\-up comparisons are one example where two specials share surface\-level tags but sit in very different cultural contexts, and the mismatch is easy to overlook\. A third group passes every existing must\-have criterion yet still reads as confusing or oddly framed\. None of these would necessarily move the per\-criterion agreement metrics, yet each calls for a change to the evaluation rubrics themselves\. When a pattern recurs, our writing and review experts revise the labeling guidelines of §[4](https://arxiv.org/html/2608.18300#S4)\. Because those guidelines are the source of truth for both the human raters and the judges’ seed rubrics, one revision updates both sides of the comparison at once\. This is precisely the value of keeping a human in the loop on an ongoing basis rather than only at launch\. Continuous human review surfaces the qualitative, long\-tail issues that quantitative drift detection is blind to, and is what allows us to catch emerging gaps in the rubric itself, not just degradation relative to it\.

## 8\.Takeaways

Operating this lifecycle at scale surfaced three lessons we expect to transfer to other teams shipping LLM judges\. First,*invest in the benchmark before the judge*\. A modest, rationale\-annotated benchmark is worth more than a large label\-only one, since the rationales are what make reasoning\-aligned tuning possible\. Second,*one tuned judge can serve many roles*\. Reusing the same judge for gating and for generation critique can reduce the alignment cost and keep production behavior consistent\. Third,*plan monitoring from day one*\. A recommender system platform’s items and users are constantly evolving, so its LLM judges, like any lifelong agent, must be monitored and re\-tuned as needed\.

## 9\.Limitations and Future Work

### 9\.1\.Limitations

Our evaluation has several scope boundaries worth stating directly\. First, the A/B test \(§[6\.2](https://arxiv.org/html/2608.18300#S6.SS2)\) covers one surface \(mobile\) and one explanation family \(similarity\-based\)\. Thus we do not yet know whether the same lifecycle and the same magnitude of lift transfer to other explanation styles or other surfaces, where the notion of a good explanation might differ\. Second, we withhold several production details for confidentiality, such as the specific definitions of must\-have criteria, and the generator and judge model details\. Third, our meta\-judge validation \(§[5\.3](https://arxiv.org/html/2608.18300#S5.SS3)\) measures agreement on rationale\-agreement classification, a narrower and simpler task than full explanation judging, so it should not be read as evidence that the primary judge matches human judgment at a comparable rate\. Fourth, RART is validated against a label\-only ablation \(vanilla\) but not against other textual\-optimization methods \(§[2](https://arxiv.org/html/2608.18300#S2)\) such as GEPA or TextGrad\. We expect RART’s focus on agreed\-fail rationale mismatches to help most in our deployment setting \(§[6\.1](https://arxiv.org/html/2608.18300#S6.SS1)\), but we have not yet measured whether a more general\-purpose optimizer would match or exceed it here\. Fifth, the primary judge and the rationale meta\-judgeℳ\\mathcal\{M\}are built on the same base model family, so their errors are plausibly correlated; the human validation ofℳ\\mathcal\{M\}\(§[5\.3](https://arxiv.org/html/2608.18300#S5.SS3)\) bounds this concern but does not remove it\. Sixth, the drift\-triggered re\-tuning path has not yet fired in production \(§[7](https://arxiv.org/html/2608.18300#S7)\)\. The monitoring loop is live in production, but the automated response to a drift event is so far validated only offline\. Finally, our online results report the metrics we judged most relevant to the user discovery experience \(novel content share, successful\-play sessions, and takedowns\), and do not characterize the effect of explanations on member behavior beyond these\.

### 9\.2\.Future Work

We see three natural extensions of RART\. First, recasting rubric tuning as*memory management*\. The full benchmark might be too large to reflect over in a single pass, so the tuner would iterate over it in chunks\. As it goes, it would consolidate the evaluation reasoning from previously seen data into a persistent scratchpad, which then steers each prompt update on new data\. This scratchpad is a*long\-term memory*where new Phase IV human rationales \(§[7](https://arxiv.org/html/2608.18300#S7)\) would update it incrementally instead of triggering a full re\-tune\. A*short\-term memory*would complement it, holding few\-shot examples retrieved per evaluation instance\. Second, combining weight\-space and text\-space optimization\. The judge would be reasoning\-finetuned on accumulated human rationales, giving a stronger base model\. Text\-based rubric optimization would then handle online adaptation, where it is more sample\-efficient and can spend more inference compute learning from a single example\. Third, casting reasoning\-aligned tuning as a*reflective prompt evolution*problem in the style of GEPA\([1](https://arxiv.org/html/2608.18300#bib.bib21)\), the reflective prompt optimizer in the DSPy framework\([5](https://arxiv.org/html/2608.18300#bib.bib20)\)\. RART is effectively a greedy, single\-objective special case of GEPA\. GEPA keeps a pool of candidates and mutates from the Pareto frontier of their per\-instance scores\. The fit is natural\. Our reasoning meta\-judgeℳ\\mathcal\{M\}already supplies the kind of rich textual feedback GEPA reflects on\. A Pareto pool would also let us trade off specificity, recall, andRAneg\\mathrm\{RA\}\_\{\\mathrm\{neg\}\}directly, rather than collapsing them into the single weighted scoress\(Eq\.[4](https://arxiv.org/html/2608.18300#S5.E4)\)\.

## References

- Agrawalet al\.\(2026\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. KhattabGEPA: reflective prompt evolution can outperform reinforcement learning\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=RQm2KQTM5r)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2608.18300#S5.SS3.p2.1),[§9\.2](https://arxiv.org/html/2608.18300#S9.SS2.p1.1)\.
- Fernandoet al\.\(2024\)C\. Fernando, D\. Banarse, H\. Michalewski, S\. Osindero, and T\. RocktäschelPromptbreeder: self\-referential self\-improvement via prompt evolution\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.235,pp\. 13481–13544\.External Links:[Link](https://proceedings.mlr.press/v235/fernando24a.html)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1)\.
- Guoet al\.\(2024\)Q\. Guo, R\. Wang, J\. Guo, B\. Li, K\. Song, X\. Tan, G\. Liu, J\. Bian, and Y\. YangConnecting large language models with evolutionary algorithms yields powerful prompt optimizers\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=ZG3RaNIsO8)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1)\.
- Herlockeret al\.\(2000\)J\. L\. Herlocker, J\. A\. Konstan, and J\. RiedlExplaining collaborative filtering recommendations\.InProceedings of the 2000 ACM Conference on Computer Supported Cooperative Work \(CSCW\),New York, NY, USA,pp\. 241–250\.External Links:[Document](https://dx.doi.org/10.1145/358916.358995)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px1.p1.1)\.
- Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. PottsDSPy: compiling declarative language model calls into state\-of\-the\-art pipelines\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=sY5N0zY5Od)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1),[§9\.2](https://arxiv.org/html/2608.18300#S9.SS2.p1.1)\.
- Kimet al\.\(2024\)S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. SeoPrometheus: inducing fine\-grained evaluation capability in language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=8euJaTveKw)Cited by:[§1](https://arxiv.org/html/2608.18300#S1.p2.1),[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. HajishirziRewardBench: evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1755–1797\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96),[Link](https://aclanthology.org/2025.findings-naacl.96/)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.18300#S4.p2.1)\.
- Liuet al\.\(2025\)T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. WangOpenRubrics: towards scalable synthetic rubric generation for reward modeling and LLM alignment\.External Links:2510\.07743,[Link](https://arxiv.org/abs/2510.07743)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2608.18300#S5.SS3.p2.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[§1](https://arxiv.org/html/2608.18300#S1.p2.1),[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. ClarkSelf\-Refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46534–46594\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px3.p1.1)\.
- Panicksseryet al\.\(2024\)A\. Panickssery, S\. R\. Bowman, and S\. FengLLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 8634–8652\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/1b44b878bb782e6954cd888628510e90-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px3.p1.1)\.
- Tanet al\.\(2025\)S\. Tan, S\. Zhuang, K\. Montgomery, W\. Y\. Tang, A\. Cuadron, C\. Wang, R\. A\. Popa, and I\. StoicaJudgeBench: a benchmark for evaluating LLM\-based judges\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=G0dksFayVq)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.18300#S4.p2.1)\.
- Tintarev and Masthoff \(2007\)N\. Tintarev and J\. MasthoffA survey of explanations in recommender systems\.In2007 IEEE 23rd International Conference on Data Engineering Workshop \(ICDEW\),pp\. 801–810\.External Links:[Document](https://dx.doi.org/10.1109/ICDEW.2007.4401070)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511),[Link](https://aclanthology.org/2024.acl-long.511/)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025\)T\. Wu, W\. Yuan, O\. Golovneva, J\. Xu, Y\. Tian, J\. Jiao, J\. E\. Weston, and S\. SukhbaatarMeta\-rewarding language models: self\-improving alignment with LLM\-as\-a\-meta\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 11537–11554\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.583),[Link](https://aclanthology.org/2025.emnlp-main.583/)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px3.p1.1),[§5\.3](https://arxiv.org/html/2608.18300#S5.SS3.p1.1),[§5\.3](https://arxiv.org/html/2608.18300#S5.SS3.p2.1)\.
- Yuksekgonulet al\.\(2025\)M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, P\. Lu, Z\. Huang, C\. Guestrin, and J\. ZouOptimizing generative AI by backpropagating language model feedback\.Nature639\(8055\),pp\. 609–616\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-08661-4),[Link](https://www.nature.com/articles/s41586-025-08661-4)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2608.18300#S5.SS3.p2.1)\.
- Zhanget al\.\(2026\)Q\. Zhang, C\. Hu, S\. Upasani, B\. Ma, F\. Hong, V\. Kamanuru, J\. Rainton, C\. Wu, M\. Ji, H\. Li, U\. Thakker, J\. Zou, and K\. OlukotunAgentic context engineering: evolving contexts for self\-improving language models\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=eC4ygDs02R)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px4.p1.1),[§5\.3](https://arxiv.org/html/2608.18300#S5.SS3.p2.1)\.
- Zhang and Chen \(2020\)Y\. Zhang and X\. ChenExplainable recommendation: a survey and new perspectives\.Foundations and Trends in Information Retrieval14\(1\),pp\. 1–101\.External Links:[Document](https://dx.doi.org/10.1561/1500000066)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\) Datasets and Benchmarks Track,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)Cited by:[§1](https://arxiv.org/html/2608.18300#S1.p2.1),[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025\)L\. Zhu, X\. Wang, and X\. WangJudgeLM: fine\-tuned large language models are scalable judges\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=xsELpEPn4A)Cited by:[§2](https://arxiv.org/html/2608.18300#S2.SS0.SSS0.Px2.p1.1)\.