Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains

# Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains
Source: [https://arxiv.org/html/2610.26932](https://arxiv.org/html/2610.26932)
DOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:RL4LLM\-Agents; Nov 14–15, 2026; Milan, ItalyISBN:978\-1\-4503\-XXXX\-X/2018/06Ashish BansalNote:Corresponding author\.email:[ashish\_bansal@vanguard\.com](mailto:[email protected])Affiliation:The Vanguard Group,Malvern,Pennsylvania,USAHossam Ameremail:[hossam\_amer@vanguard\.com](mailto:[email protected])Affiliation:The Vanguard Group,Toronto,Ontario,Canada,Keerat Kaur Gulianiemail:[keerat\_guliani@vanguard\.com](mailto:[email protected])Affiliation:The Vanguard Group,Toronto,Ontario,Canada,Parinita Edkeemail:[parinita\_edke@vanguard\.com](mailto:[email protected])Affiliation:The Vanguard Group,Toronto,Ontario,Canada,Lesley Mathesonemail:[lesley\_matheson@vanguard\.com](mailto:[email protected])Affiliation:The Vanguard Group,Malvern,Pennsylvania,USAandLeizer Teranemail:[leizer\_teran@vanguard\.com](mailto:[email protected])Affiliation:The Vanguard Group,Malvern,Pennsylvania,USA

Received 01 October 2026

###### Abstract\.

Large language models \(LLMs\) are increasingly serving as autonomous agents in non\-verifiable domains such as financial services, where they must interpret user intent, rewrite queries, and retrieve information from extensive document collections while satisfying domain\-specific requirements\. Reinforcement learning \(RL\) offers a natural way to train such agents from the consequences of their own behavior, but most recent RL successes rely on verifiable domains, where reward can be computed automatically as a sparse, binary signal\. Retrieval in non\-verifiable domains affords no such oracle: judging whether a query rewrite is “good” requires judgment that only a subject\-matter expert \(SME\) can supply even if this judgment is scarce, and that judgment is graded, not binary\. When applied in Group Relative Policy Optimization \(GRPO\), a widely used policy\-gradient algorithm, such sparse rewards make gradients vanish whenever all sampled outputs in a group succeed or fail together, stalling optimization\. Motivated by this reward sparsity, we propose a method that turns a small, fixed investment of SME judgment into a dense reward, combined with retrieval\-based metrics into a single hybrid, GRPO\-compatible reward for the non\-verifiable domains\. Experimental results demonstrate improvements of up to 6% in Recall@3 and 6\.7% in MRR@3 relative to sparse\-reward baselines\.

###### Keywords:

Dense and sparse rewards, GRPO, Subject matter expert feedback, Reinforcement Learning for agents

## 1\.Introduction

LLMs are increasingly deployed as autonomous agents in non\-verifiable domains such as financial services, where they interpret user intent, rewrite queries, and retrieve information from large collections of documents under domain\-specific constraints\([Liu et al\., 2025a](https://arxiv.org/html/2610.26932#bib.bib31);[Matlin et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib32);[Amer et al\., 2026b](https://arxiv.org/html/2610.26932#bib.bib43)\)\. Reinforcement learning \(RL\) is a default procedure for post\-training such agents, allowing them to explore and learn directly from the consequences of their own behavior\([Christiano et al\., 2017](https://arxiv.org/html/2610.26932#bib.bib11);[Ouyang et al\., 2022](https://arxiv.org/html/2610.26932#bib.bib10);[Moradi et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib27);[Amer et al\., 2026a](https://arxiv.org/html/2610.26932#bib.bib26)\)\.

Recent advances in RL have been particularly pronounced in*verifiable*domains such as math proofs, unit tests, and executable code, where a reward function can check the answer automatically and return a sparse, binary signal\([Guo et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib2);[Sheng et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib25);[Gunjal et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib28)\)\. This leaves open a large class of tasks for which no such automatic oracle exists\. Retrieval in non\-verifiable domains is one such task: whether a query rewrite is “good” depends on regulatory nuance, institution\-specific terminology, and document semantics that only a subject\-matter expert \(SME\) can judge, and that judgment is inherently graded, not pass/fail\. In this non\-verifiable, high\-expertise setting, RL cannot rely on the sparse binary rewards that verifiable domains afford, and SME time is too scarce to supply feedback at the scale RL normally demands\. We argue this calls for*dense*rewards grounded by a limited set of SME\-derived rubrics: signals that capture degrees of correctness rather than binary pass/fail, allowing RL to learn effectively from a limited budget of SME feedback\.

We ground this argument formally in Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2610.26932#bib.bib1)\), the RL policy gradient algorithm used throughout this work\. For a group ofGGcompletions\{o1,…,oG\}\\\{o\_\{1\},\\dots,o\_\{G\}\\\}sampled per promptqqand scored with rewardsrir\_\{i\}, GRPO forms the group\-relative advantage

\(1\)Ai=ri−mean\(r1:G\)std\(r1:G\)\.A\_\{i\}\\;=\\;\\frac\{r\_\{i\}\-\\operatorname\{mean\}\(r\_\{1:G\}\)\}\{\\operatorname\{std\}\(r\_\{1:G\}\)\}\\,\.Reward*density*is decisive here: under a sparse binary reward, any prompt whoseGGsamples all succeed or all fail leaves the group constant,std\(r1:G\)→0\\operatorname\{std\}\(r\_\{1:G\}\)\\to 0, and Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\) collapses to an uninformative advantage\. The policy is updated only on the fraction of prompts that happen to yield a mixed group; exploration that moves*closer*to a correct answer earns no signal, and training dies out \(see Figure[1](https://arxiv.org/html/2610.26932#S3.F1)\)\.

To overcome these challenges, this paper proposes a method that turns a small, fixed investment of SME judgment into a dense, hybrid, GRPO\-compatible reward for non\-verifiable domains\. We*elicit*SME judgment on a small sample of graded rewrites, articulating why one rewrite retrieves better than another; we*distil*those explanations into an LLM\-executable rubric; and*validate*the rubric against held\-out SME judgments before it is trusted for training\. The validated rubric score is then blended with an objective, automatically computed retrieval signal into a single dense reward, which we use to*fine\-tune*the rewriter with GRPO\. This turns a fixed, small expert budget into a continuous, scalable reward signal, and we demonstrate its effectiveness on a retrieval agent\. Overall, this paper contributes the following:

- •Scalable SME\-aligned reward pipeline:A four\-step framework that converts a small, fixed investment of expert judgment into a continuous, dense reward without relying on learned reward models or per\-query ground\-truth answers\.
- •Hybrid reward design:A formulation that bridges reinforcement learning with verifiable rewards \(RLVR\) and rubric\-based RL by combining objective, automated retrieval signals with SME semantic criteria\.
- •Empirical validation:Our experiments reveal that reward resolution, not reward correctness alone, is a key determinant of GRPO trainability for query rewriting\. The multi\-component reward produces a non\-zero reward variance in 44% of groups, compared with only 6\.5% for a binary reward \(6\.8×\\times\), providing a learning signal for substantially more groups and yielding improvements of up to 6% in Recall@3 and 6\.7% in MRR@3 over sparse\-reward baselines\.

## 2\.Related Work

RL for LLM alignmentRLHF aligns models to human preference via a learned reward model optimized with PPO\([Ouyang et al\., 2022](https://arxiv.org/html/2610.26932#bib.bib10);[Christiano et al\., 2017](https://arxiv.org/html/2610.26932#bib.bib11);[Schulman et al\., 2017](https://arxiv.org/html/2610.26932#bib.bib9)\); RLAIF/Constitutional AI replace human labels with model feedback\([Bai et al\., 2022](https://arxiv.org/html/2610.26932#bib.bib12);[Lee et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib13)\), and DPO works with chosen and rejected pairs\([Rafailov et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib14)\)\. GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.26932#bib.bib1);[Guo et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib2)\)discards the critic and estimates advantages from a sampled group\. DAPO\([Yu et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib29)\)resamples zero\-variance groups to fix training degeneracy; Dr\. GRPO\([Liu et al\., 2025b](https://arxiv.org/html/2610.26932#bib.bib30)\)dropsstd\(r1:G\)\\operatorname\{std\}\(r\_\{1:G\}\)from Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\) to fix a related difficulty bias\. Unlike GRPO, DAPO, and Dr\. GRPO, which modify the optimization process, our work targets reward design for a non\-verifiable domain, using GRPO as the underlying policy\-gradient algorithm\.

Reward density and shapingSparse terminal rewards are notoriously hard to learn from; reward shaping provides denser signal while preserving the optimal policy\([Ng et al\., 1999](https://arxiv.org/html/2610.26932#bib.bib15)\)\. In LLMs, process\-supervised and step\-level rewards outperform outcome\-only rewards on reasoning\([Lightman et al\., 2024](https://arxiv.org/html/2610.26932#bib.bib16);[Uesato et al\., 2022](https://arxiv.org/html/2610.26932#bib.bib17);[Setlur et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib7)\), and controlled studies show reward granularity interacts with base\-model coverage and prompt distribution to determine success\([Chen et al\., 2026a](https://arxiv.org/html/2610.26932#bib.bib4);[Clay et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib3);[Shao et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib5);[Chen et al\., 2026b](https://arxiv.org/html/2610.26932#bib.bib6)\)\. We apply this lens to GRPO, showing density is required not only for credit assignment but for a non\-degenerate advantage in Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\)\.

Reward modeling and its failure modesLearned rewards can be gamed \(reward\-model overoptimization, reward hacking, and hackable process reward models\([Gao et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib46);[Skalse et al\., 2022](https://arxiv.org/html/2610.26932#bib.bib47);[Tiwari et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib8)\)\), motivating auditable rewards grounded in explicit criteria\. LLM\-as\-a\-judge\([Zheng et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib18)\)makes it practical to execute a human\-authored rubric at scale, which we exploit to distil SME judgment into a transparent, weighted reward rather than an opaque scalar model\.

Financial LLMs and verifiable rewards\.Recent work post\-trains LLMs for finance rather than relying on pretraining alone\. Fin\-R1\([Liu et al\., 2025a](https://arxiv.org/html/2610.26932#bib.bib31)\)applies the same SFT\-then\-GRPO recipe we use, on a 7B model, to strengthen financial reasoning, showing GRPO is already used for financial LLM post\-training—the exact setting our reward is designed for\. IFF\([Matlin et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib32)\)introduces verifiable, chainable constraints for financial instruction\-following and motivates RLVR for finance as future work, but stops at evaluation, without training a policy against its verifiers\. Our SME rubric plays an analogous role \- grounding a reward in explicit, checkable criteria rather than an opaque judge score \- but closes the loop IFF leaves open by optimizing a policy against it with GRPO\.

Retrieval and query rewriting\.Retrieval\-augmented generation\([Lewis et al\., 2020](https://arxiv.org/html/2610.26932#bib.bib19)\)grounds LLM answers in an external corpus, and learned query rewriting helps when user vocabulary diverges from corpus terminology\([Ma et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib20)\)\. RL\-trained rewriters fall into two strands\. The first ties reward purely to retrieval outcomes, with no expert\-quality signal: DeepRetrieval\([Jiang et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib35)\)and RL\-QR\([Cha et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib36)\)optimize query generation directly against retrieval metrics via GRPO with no reference queries; ReaLiC\([Xu et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib41)\)adds latent reasoning structure but still no rubric; MaFeRw\([Wang et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib37)\)densifies the signal with similarity, ranking, and ROUGE terms but requires a per\-query gold answer\. The second strand \- Rubrics\-as\-Rewards\([Gunjal et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib28)\),\([Li et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib42)\), OpenRubrics\([Liu et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib38)\), EvoRubric\([Guan et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib39)\), RubricRL\([Feng et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib40)\)\- shows that structured, weighted, LLM\-executable rubrics outperform scalar LLM\-judge rewards in non\-verifiable domains, but only as a reward on the terminal answer and not for query rewrites derived from a small as well as fixed investment of SME feedback merged with retrieval based metrics\.

We bridge both strands with a dense reward that blends an objective retrieval signal with an SME\-distilled, validated rubric, applied to the intermediate query\-rewrite action rather than the terminal answer, with no learned reward model and no per\-query gold answer\. A small, fixed investment of SME judgment is distilled once into a weighted rubric and validated for fidelity, separating and correctly ranking gold vs\. baseline retrievals before GRPO training, which is needed in retrieval agents operating in non\-verifiable domains\.

## 3\.Methodology

Our work focuses on non\-verifiable tasks, where models are post\-trained using policy\-gradient methods\. Throughout our experiments, we use GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.26932#bib.bib1)\), a widely adopted policy optimization method for LLM post\-training\.

### 3\.1\.Problem Formulation

The GRPO Objective\.For each promptqq, GRPO drawsGGcompletions from the current policy, scores them, and updatesπθ\\pi\_\{\\theta\}with a clipped surrogate objective

\(2\)𝒥⁡\(θ\)=𝔼⁡\[1G​∑i=1G1\|oi\|​∑tmin⁡\(ρi,t​Ai,clip⁡\(ρi,t,1−ε,1\+ε\)​Ai\)\]−β𝔻KL\(πθ∥πref\),\\begin\{split\}\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\tfrac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\tfrac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{t\}\\min\\\!\\big\(\\rho\_\{i,t\}A\_\{i\},\\;\\mathrm\{clip\}\(\\rho\_\{i,t\},1\{\-\}\\varepsilon,1\{\+\}\\varepsilon\)A\_\{i\}\\big\)\\right\]\\\\ \-\\beta\\,\\mathbb\{D\}\_\{\\mathrm\{KL\}\}\\\!\\big\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\\big\),\\end\{split\}whereρi,t=πθ​\(oi,t∣q,oi,<t\)/πθold​\(oi,t∣q,oi,<t\)\\rho\_\{i,t\}=\\pi\_\{\\theta\}\(o\_\{i,t\}\\\!\\mid\\\!q,o\_\{i,<t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(o\_\{i,t\}\\\!\\mid\\\!q,o\_\{i,<t\}\)is the token\-level importance ratio andAiA\_\{i\}is the group\-relative advantage of Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\)\. Two properties of this objective drive the rest of our argument\. First, GRPO carries*no value network*: the group mean is the only baseline, so the reward’s spread*within a group*is the sole source of learning direction; if that spread vanishes, so does the gradient \- a failure mode we formalize and repair with our dense reward\. Second,AiA\_\{i\}is shared by every token of completionoio\_\{i\}; the whole completion is pushed up or down according to its standing relative to its group\. This is precisely why the granularity of the reward matters and not merely its presence: a reward that ranks completions correctly relative to one another rather than only classifying each in isolation\-yields a more informativeAiA\_\{i\}, which motivates the continuously weighted, multi\-component reward we construct\.

![Refer to caption](https://arxiv.org/html/2610.26932v1/figs/fig_hero_abc_v2.png)Figure 1\.The mechanism at the heart of the paper\.\(a\)The query\-rewriting agent: only the rewriterπθ\\pi\_\{\\theta\}is trainable: the downstream retriever and answer synthesizer are frozen, so the reward measures the rewriter’s causal effect on retrieval\.\(b\)One prompt sampled into a group ofG=4G\{=\}4rewrites \(value ofGGis illustrative here\), scored two ways\. A binary top\-3 hit reward collapses every rewrite that misses the threshold to00includingo3o\_\{3\}, which retrieves the answer document at rank 7 and is invisible to a top\-3 cutoff while the dense SME rubric of Eq\. \([4](https://arxiv.org/html/2610.26932#S3.E4)\) grades the same rewrites continuously and rankso3o\_\{3\}correctly as the best of the four\.\(c\)The consequence for GRPO: the binary reward leavesstd\(r1:G\)=0\\mathrm\{std\}\(r\_\{1:G\}\)=0for this group, so every advantageAiA\_\{i\}in Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\) is zero and no gradient reachesπθ\\pi\_\{\\theta\}\(the degenerate case of Eq\. \([3](https://arxiv.org/html/2610.26932#S3.E3)\)\); the dense SME reward keepsstd\(r1:G\)\>0\\mathrm\{std\}\(r\_\{1:G\}\)\>0because near\-misses are still ranked, so the group always spreads and every prompt yields a usable gradient\. Dense SME reward is curated using SME judgements \[Figure[4](https://arxiv.org/html/2610.26932#A4.F4), Appendix\]\.Why sparse rewards break GRPO\.Consider a binary rewardri∈\{0,1\}r\_\{i\}\\in\\\{0,1\\\}with per\-sample success probabilityppon a given prompt\. Because each of theGGcompletions is sampled independently from the same policyπθ\\pi\_\{\\theta\}conditioned onqq, the binary outcomes form an i\.i\.d\. Bernoulli sample\. The group is degenerate \(all successes or all failures\) with probability

\(3\)ρdeg=pG\+\(1−p\)G\.\\rho\_\{\\text\{deg\}\}=p^\{G\}\+\(1\-p\)^\{G\}\.
For a degenerate group,std\(r1:G\)=0\\operatorname\{std\}\(r\_\{1:G\}\)=0, so the normalized advantages in Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\) are undefined \(and clamped to00in practice\), causing the prompt to contribute no gradient\. Hereppis the policy’s per\-sample success probability on a given prompt the chance that a single sampled rewrite earns reward11so a lowppmarks a prompt the policy rarely gets right\. Substituting a hard prompt withp=0\.1p=0\.1andG=8G=8givesρdeg=0\.18\+0\.98≈0\+0\.43=0\.43\\rho\_\{\\text\{deg\}\}=0\.1^\{8\}\+0\.9^\{8\}\\approx 0\+0\.43=0\.43, dominated by the all\-fail term\(1−p\)G\(1\-p\)^\{G\}:43%43\\%of such groups produce no learning signal, leaving only57%57\\%with a usable gradient\. More clearly, the prompts for which the policy has the lowest success probability are precisely those most likely to yield no gradient signal, making it difficult for training to moveppaway from the floor\. IncreasingGGreduces this degeneracy, but only by incurring a proportional increase in rollout compute\. To gain further insight into Equation[3](https://arxiv.org/html/2610.26932#S3.E3), we empirically measured the fraction of rollout groups atG=8G=8in which all completions receive zero reward under the binary reward scheme\. We found that 18\.34% of the rollout groups are zero\-reward groups, resulting in no reward variance and consequently impeding training\.

A*dense*rewardri∈\[−1,1\]r\_\{i\}\\in\[\-1,1\]that scores*how good*each completion is, rather than whether it crossed a threshold, almost always assigns distinct values to the members of a group\. Even when allGGrewrites fail to retrieve the answer, one may recover the key entity while another does not; the dense reward ranks them,std\\operatorname\{std\}stays positive, and Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\) produces a gradient that moves the policy toward the marginally better rewrite\. Density thus converts the near\-miss structure of the task invisible to a binary reward into usable signal, which is exactly the regime where credit assignment under sparse rewards is hardest\([Ng et al\., 1999](https://arxiv.org/html/2610.26932#bib.bib15)\)\.

Composite, dense rewardWe turn a small, fixed investment of subject\-matter\-expert \(SME\) judgment into a continuous reward through a four\-step loop \(Figure[4](https://arxiv.org/html/2610.26932#A4.F4)\):*\(1\) Elicit\.*SMEs grade a small sample of rewrite behavior: roughly100100\(question, rewrite\) pairs sampled \(stratified\) so the sample spans rewrites that alter vs\. copy the question and that hit vs\. miss the answer\. For each pair, the expert assigns a headline outcome label \(0/1/20/1/2\) indicating whether the documents*retrieved*by the rewrite actually answer the question\. The expert also explains*why*, along four recurring axes: whether the rewrite preserved the information need, kept the salient entities, sharpened specificity, and stayed well\-formed\. These four axes, rather than the individual failure modes, become the rubric criteria\.*\(2\) Distil\.*Those explanations are encoded as an LLM\-executable rubric \- a judge prompt with the four named, individually weighted criteria of Eq\. \([11](https://arxiv.org/html/2610.26932#S3.E11)\) so the expensive human judgment is captured once and replayed at scale\.*\(3\) Validate\.*Before training, we check that the rubric is faithful to the elicited judgment\. Using the same stratified set, we compare the judge’sRSMER\_\{\\text\{SME\}\}against the held\-out human outcome labels and verify that it separates strong rewrites from weak baseline rewrites \(verbatim copies and answer\-missing queries\) and ranks them in the SME\-preferred order, guarding against a misspecified reward before it can mislead the policy\.*\(4\) GRPO fine\-tune\.*We optimize the rewriter against the validated reward with Eq\. \([2](https://arxiv.org/html/2610.26932#S3.E2)\), withRSMER\_\{\\text\{SME\}\}computed at each step by the frozen judge via Eq\. \([11](https://arxiv.org/html/2610.26932#S3.E11)\)\.

Reward composition\.For a questionqqand a candidate rewritexx, the dense reward,RR, is a weighted blend of an objective retrieval signal and an SME rubric signal:

\(4\)R=a​Rretrieval\+\(1−a\)​RSME,R=a\\,R\_\{\\text\{retrieval\}\}\+\(1\-a\)\\,R\_\{\\text\{SME\}\},The two terms are computed differently:RretrievalR\_\{\\text\{retrieval\}\}is a deterministic function ofxxand its retrieved passages, whileRSMER\_\{\\text\{SME\}\}is produced by a frozen judge LLM prompted with a rubric\. We describe each in turn\.

*\(1\) Retrieval signal*:RretrievalR\_\{\\text\{retrieval\}\}is computed by passingxxthrough the frozen retriever, taking the top\-kkpassages, and scoring them with a graded retrieval termggmodulated by two multiplicative gates that guard against faithless or malformed rewrites:

\(5\)Rretrieval=g⋅γfaith⋅γform\.R\_\{\\text\{retrieval\}\}=g\\cdot\\gamma\_\{\\text\{faith\}\}\\cdot\\gamma\_\{\\text\{form\}\}\.
All three terms are computed deterministically from the query, the rewrite, and the retrieved passages; no model is queried\.

- •g∈\[0,1\]g\\in\[0,1\]: a capped DCG@33over the answer\-bearing passages in the top\-kk\(a passage is answer\-bearing if it contains a gold answer string\), \(6\)g=min⁡\(1,∑r∈H1/log2⁡\(r\+1\)∑i=131/log2⁡\(i\+1\)\),g=\\min\\\!\\Big\(1,\\ \\tfrac\{\\sum\_\{r\\in H\}1/\\log\_\{2\}\(r\+1\)\}\{\\sum\_\{i=1\}^\{3\}1/\\log\_\{2\}\(i\+1\)\}\\Big\),whereHHindexes those passages by rankrr\. It rewards both placing an answer earlier and covering more answer\-bearing passages\.
- •γfaith∈\[b,1\]\\gamma\_\{\\text\{faith\}\}\\in\[b,1\]: increases with two lexical quantities—the*intent match*\(the fraction of the question’s content words retained in the rewrite\) and the*entity match*\(the fraction of the question’s salient entities, such as names, numbers, and dates, preserved\): \(7\)γfaith=b\+\(1−b\)​min⁡\(1,0\.5​Intent\+0\.5​Entity0\.8\)\.\\gamma\_\{\\text\{faith\}\}=b\+\(1\-b\)\\,\\min\\\!\\big\(1,\\ \\tfrac\{0\.5\\,\\mathrm\{Intent\}\+0\.5\\,\\mathrm\{Entity\}\}\{0\.8\}\\big\)\. The intent and entity terms are computed using simple lexical string matching overlaps, where we count how many words overlap and obtain a percentage given the input string length\.
- •γform∈\[c,1\]\\gamma\_\{\\text\{form\}\}\\in\[c,1\]: penalizes glued or camel\-cased tokens, chatter, markup, and multi\-line output\. \(8\)γform=c\+\(1−c\)​min⁡\(Wellformed,Conclusive\)\\gamma\_\{\\text\{form\}\}=c\+\(1\-c\)\\,\\min\(\\mathrm\{Wellformed\},\\mathrm\{Conclusive\}\)

We setb=0\.4b=0\.4andc=0\.5c=0\.5in all experiments \(Table[1](https://arxiv.org/html/2610.26932#S4.T1)\)\.γform\\gamma\_\{\\text\{form\}\}combines two deterministic surface\-form scores, each in\[0,1\]\[0,1\], computed directly from the rewrite string with no model call\. The first,Wellformed\\mathrm\{Wellformed\}, ensures that there are no unnecessary whitespace\-delimited or hyphen tokens\.Wellformed\\mathrm\{Wellformed\}is defined as follows:

\(9\)Wellformed=\{1,m=0\(no glued tokens\)max⁡\(0,1−0\.5​m\),m≥1,\\mathrm\{Wellformed\}=\\begin\{cases\}1,&m=0\\quad\\text\{\(no glued tokens\)\}\\\\ \\max\(0,\\ 1\-0\.5\\,m\),&m\\geq 1,\\end\{cases\}wheremmis the number of glued tokens\.

The second,Conclusive\\mathrm\{Conclusive\}, targets string\-level artifacts of a query that is not a single clean line\.Conclusive\\mathrm\{Conclusive\}is defined as follows:

\(10\)Conclusive=\{1,p=0\(clean single\-line query\)max⁡\(0,1−p\),p\>0,\\mathrm\{Conclusive\}=\\begin\{cases\}1,&p=0\\quad\\text\{\(clean single\-line query\)\}\\\\ \\max\(0,\\ 1\-p\),&p\>0,\\end\{cases\}
whereppis the cumulative penalty obtained by summing the penalties of all artifacts present in the query:0\.50\.5for a multi\-line query,0\.50\.5for a conversational preamble,0\.250\.25for a leading quote or bullet, and0\.250\.25for exceeding 240 characters\. The score decreases linearly with the accumulated penalty and is clipped at zero throughmax⁡\(0,1−p\)\\max\(0,1\-p\)\.

*\(2\) SME rubric signal*:RSMER\_\{\\text\{SME\}\}is SME judgment distilled into a weighted rubric and executed by a frozen judge LLM\. The judge is prompted withqq, the rewritexx, and, for each criterioncc, a short description with0/1/20/1/2anchors \(the full prompt is shown verbatim in Figure[3](https://arxiv.org/html/2610.26932#A4.F3), Appendix[D](https://arxiv.org/html/2610.26932#A4)\); it returns an integersc∈\{0,1,2\}s\_\{c\}\\in\\\{0,1,2\\\}per criterion \(as JSON\)\. We normalize by the top anchor and combine with fixed, SME\-elicited weightswcw\_\{c\}\(summing to11\):

\(11\)RSME=∑c∈𝒞wc​sc2∈\[0,1\]\.R\_\{\\text\{SME\}\}=\\sum\_\{c\\in\\mathcal\{C\}\}w\_\{c\}\\,\\frac\{s\_\{c\}\}\{2\}\\in\[0,1\]\.The four criteria \(weights in Table[1](https://arxiv.org/html/2610.26932#S4.T1)\) are intent fidelity, entity grounding, specificity/disambiguation, and well\-formedness\. These are the*semantic*counterparts of the lexicalIntent\\mathrm\{Intent\}andEntity\\mathrm\{Entity\}used in the gates above: the gates are cheap deterministic floors, while the rubric judges the same properties for meaning\. Keeping the score as a transparent per\-criterion sum \(rather than one opaque scalar\) makes a low reward traceable to a specific criterion and, following the Rubrics\-as\-Rewards results\([Gunjal et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib28)\), better separates degrees of quality in non\-verifiable domains\.

### 3\.2\.Training\-stability discipline

A dense reward removes the gradient\-starvation failure but does not by itself guarantee stable optimization\. We add three guardrails\.KL anchoring\.A penaltyβ=0\.1\\beta=0\.1to a frozen reference policyπref\\pi\_\{\\mathrm\{ref\}\}\(Eq\. \([2](https://arxiv.org/html/2610.26932#S3.E2)\)\) keeps the rewriter from drifting into degenerate text that games the rubric while abandoning fluency\.Bounded updates\.Asymmetric PPO\-style clipping \(ε=0\.2\\varepsilon=0\.2,εhigh=0\.28\\varepsilon\_\{\\mathrm\{high\}\}=0\.28\- a slightly looser ceiling that permits more upward exploration\) together with gradient\-norm clipping caps the size of any single step\.Variance monitoring\.Because Eq\. \([1](https://arxiv.org/html/2610.26932#S1.E1)\) is undefined when group variance vanishes, we track reward variance directly during training; its collapse is the earliest diagnostic that learning has stalled \(see Section[5](https://arxiv.org/html/2610.26932#S5)\)\.

## 4\.Experiment Setup

We fine\-tune the Qwen3 query\-rewriter with GRPO usingms\-swift\([Zhao et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib21)\), using parameter\-efficient LoRA adapters\([Hu et al\., 2021](https://arxiv.org/html/2610.26932#bib.bib23)\)under a fixed recipe: the same policy, optimizer, rollout size, and training budget are held constant across all reward conditions, so any difference in outcome is attributable to the reward\. All reported experiments use a Qwen3\-1\.7B rewriter trained with GRPO over a frozen retriever and a frozen answer synthesizer, on the Natural Questions split ofFlashRAG\([Jin et al\., 2025b](https://arxiv.org/html/2610.26932#bib.bib44)\)\. We split this data set into a train set \(79K examples\) and an evaluation set \(9K examples\) for GRPO finetuning\. During training, we haveG=8G\{=\}8completions per prompt with vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib22)\)colocated rollout generation\. This trains for11epoch over an effective batch of6464rollouts/step \(88prompts×\\times88completions; per\-device batch22, gradient accumulation44, across88GPUs\) under DeepSpeed ZeRO\-2\([Rajbhandari et al\., 2020](https://arxiv.org/html/2610.26932#bib.bib24)\)memory sharding on88×\\timesA100 80 GB GPUs, and uses the GRPO clipping and KL settings of Eq\. \([2](https://arxiv.org/html/2610.26932#S3.E2)\) \(ε=0\.2\\varepsilon\{=\}0\.2/εhigh=0\.28\\varepsilon\_\{\\text\{high\}\}\{=\}0\.28,β=0\.1\\beta\{=\}0\.1\)\. The reward is registered as an external plugin behind a common interface, so the dense reward of Eq\. \([4](https://arxiv.org/html/2610.26932#S3.E4)\) and the sparse binary top\-1010baseline are swapped in without any other change to the pipeline\. Table[4](https://arxiv.org/html/2610.26932#A1.T4)in Appendix[A](https://arxiv.org/html/2610.26932#A1)lists the complete configuration held constant across all runs\.

Reward\-granularity ladder\.Rather than a single binary\-vs\-dense contrast, we compare reward variants that expose progressively more retrieval signal to the group\. This isolates which form of density carries the gain, following the treatment of reward granularity as a first\-class axis in recent controlled studies\([Clay et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib3)\)\. We consider four conditions, reported in Table[2](https://arxiv.org/html/2610.26932#S5.T2): \(1\)Binary hit@kk, the sparse top\-kkbaseline; \(2\)DeepRetrieval\([Jiang et al\., 2025](https://arxiv.org/html/2610.26932#bib.bib35)\), a*quantized*retrieval reward: a step function over recall bands that is finer than a hit/miss reward\. We use their Literature Search reward scheme; \(3\)retrieval\-only dense, the retrieval reward in Equation[5](https://arxiv.org/html/2610.26932#S3.E5); and \(4\) the full blend of Eq\. \([4](https://arxiv.org/html/2610.26932#S3.E4)\), which adds the SME rubric signalRSMER\_\{\\text\{SME\}\}on top of the retrieval term at retrieval weightsa∈0\.85,0\.70,0\.55a\\in\{0\.85,0\.70,0\.55\}\(equivalently SME weights\(1−a\)∈0\.15,0\.30,0\.45\(1\-a\)\\in\{0\.15,0\.30,0\.45\}\)\. We expect the fraction of non\-degenerate groups, and downstream retrieval quality, to increase with reward density, with the gap between conditions \(1\) and \(4\) as the headline binary\-vs\-dense result\. The reward weights used throughout our experiments are given in Table[1](https://arxiv.org/html/2610.26932#S4.T1)\. The six sub\-score weights withinRretrievalR\_\{\\text\{retrieval\}\}follow the SME\-elicited ordering of importance described in Section[3](https://arxiv.org/html/2610.26932#S3)and are validated \(step \(3\) of our rubric\-distillation loop\) against the SME\-preferred ranking of gold vs\. baseline rewrites before training\.

Table 1\.Reward composition used in all experiments\.RretrievalR\_\{\\text\{retrieval\}\}is a retrieval driver modulated by two multiplicative guardrail gates with floorsbbandcc\(Eq\. \([5](https://arxiv.org/html/2610.26932#S3.E5)\)\);RSMER\_\{\\text\{SME\}\}is a weighted rubric over four SME\-elicited criteria \(Eq\. \([11](https://arxiv.org/html/2610.26932#S3.E11)\)\)\. The top\-level blend \(Eq\. \([4](https://arxiv.org/html/2610.26932#S3.E4)\)\) trades the two off withaa\(SME weight1−a1\-a\)\.Reward\-degeneracy screen\.We quantify how much learnable signal each reward exposes*before*committing training compute\. For each input prompts from our dataset we sampleG=8G\{=\}8candidate rewrites from the untrained rewriter, issue one retrieval pass per candidate against the frozen index, and apply every reward function to those same fixed top\-1010passages\. We then recordρdeg\\rho\_\{\\text\{deg\}\}, the fraction of prompt groups whose reward vector hasstd\(r1:G\)≈0\\operatorname\{std\}\(r\_\{1:G\}\)\\\!\\approx\\\!0\( Eq\. \([3](https://arxiv.org/html/2610.26932#S3.E3)\)\), and report its complement1−ρdeg1\-\\rho\_\{\\text\{deg\}\}as the % grad column of Table[2](https://arxiv.org/html/2610.26932#S5.T2)\. Because retrieval is executed once and every reward scores the same passages, the column isolates the effect of reward*form*from any difference in retrieval or policy\. During training, we additionally logρdeg\\rho\_\{\\text\{deg\}\}for every training step\.

Evaluation and statistical protocol\.The primary metric is retrieval success \(answer document in top\-33and top\-1010\) on on the evaluation set \(9000 examples\), with the untrained rewriter \(as\-is query\) as the baseline\. A held\-out, judge\-independent retrieval metric additionally guards against reward hacking: reward gains that do not transfer to it would indicate the policy is gaming the rubric rather than improving retrieval\.

## 5\.Results

We report retrieval atk=3k\\\!=\\\!3andk=10k\\\!=\\\!10\(Recall, MRR\) \(§[5\.1](https://arxiv.org/html/2610.26932#S5.SS1)\), and separately assess whether retrieval gains survive scoring by judges that never took part in training \(§[5\.2](https://arxiv.org/html/2610.26932#S5.SS2)\)\. We also provide an analysis on whether the rubric matters under retrieval\-depth mismatch \(typical in production settings\) in Appendix[B](https://arxiv.org/html/2610.26932#A2)\.

### 5\.1\.Dense rubric reward vs\. sparse RL and prompting

Table 2\.Retrieval quality onFlashRAG\-NQ\.*% grad*is the fraction of GRPO groups with non\-zero intra\-group reward variance, i\.e\.1−ρdeg1\-\\rho\_\{\\text\{deg\}\}of Eq\. \([3](https://arxiv.org/html/2610.26932#S3.E3)\): the share of prompts that actually produce a gradient this measures the existence and quantity of learnable distinctions in the reward, not whether the resulting gradient points somewhere useful\. Best per column in bold\.Sparse rewards starve GRPO of gradient\.The % grad column of Table[2](https://arxiv.org/html/2610.26932#S5.T2)is the empirical face of Eq\. \([3](https://arxiv.org/html/2610.26932#S3.E3)\)\. A binary hit@kkreward leaves only6\.5%6\.5\\%of groups with non\-zero reward variance, and DeepRetrieval’s reward a single scalar retrieval metric, denser than a hit/miss bit but still one number per rollout reaches only17\.5%17\.5\\%\. The retrieval only dense composite further raises this to41\.4%41\.4\\%, a6\.4×6\.4\\timesincrease \(over binary hit@kk\) in the fraction of prompts that contribute a usable update\. The mechanism argued in §[3](https://arxiv.org/html/2610.26932#S3)is therefore not hypothetical: under sparse reward the overwhelming majority of rollout compute is spent on groups whose advantage is identically zero\.

Density alone is not sufficient; rubric grounding is what pays\.In the primary generation setting the retrieval\-only dense reward keeps GRPO’s gradient alive but does*not*translate that into retrieval quality: it matches the no\-rewrite baseline atk=3k\\\!=\\\!3\(71\.94%71\.94\\%vs\.71\.91%71\.91\\%R@3, identical MRR@3\) and is slightly*worse*atk=10k\\\!=\\\!10\(81\.96%81\.96\\%vs\.83\.32%83\.32\\%R@10,0\.62750\.6275vs\.0\.65050\.6505MRR@10\)\. Blending in the SME\-distilled rubric is what moves the metrics: the best configuration reaches74%74\\%R@3 \(\+2\.09\+2\.09over no\-rewrite,\+2\.06\+2\.06over retrieval\-only dense\) and0\.65550\.6555MRR@3 \(\+0\.026\+0\.026\), with MRR@10 rising from0\.65050\.6505to0\.66960\.6696\. Read together, these two rows separate the paper’s two claims: reward*density*is what keeps the advantage non\-degenerate, while reward*content*\- the expert criteria the rubric makes gradable is what determines where the surviving gradient points\. A dense reward built from retrieval signals alone satisfies the first condition and fails the second\.

The gain is a plateau, not a tuned point\.Across the three rubric weights, the results are nearly flat: R@3 spans73\.6273\.62–74\.0074\.00, MRR@3 spans0\.64990\.6499–0\.65550\.6555, and MRR@10 spans0\.66860\.6686–0\.66960\.6696\. Every setting ina∈\{0\.85,0\.70,0\.55\}a\\in\\\{0\.85,\\ 0\.70,\\ 0\.55\\\}beats both the no\-rewrite baseline and the retrieval\-only dense reward on all four metrics, and the spread acrossaa\(0\.380\.38R@3\) is roughly five times smaller than the gap to the no\-rewrite baseline \(2\.092\.09R@3\)\. The effect is therefore a property of adding rubric supervision, not of selecting its weight\. Furthermore, the smaller absolute gain atk=10k\\\!=\\\!10reflects recall saturation, not a weaker effect: relative to few\-shot prompting the method reduces the miss rate by19\.0%19\.0\\%atk=3k\\\!=\\\!3and17\.2%17\.2\\%atk=10k\\\!=\\\!10\(gain normalized by the headroom100−R@​k100\-\\text\{R@\}k\), and MRR@10 still rises9\.6%9\.6\\%relative even where recall@10 is near its ceiling\.

### 5\.2\.Do the gains transfer, or did we game our own referee?

Table 3\.Judge\-independent answer quality\. AQ is the mean of OpenCompass and Mistral\-7B scores are rubric\-free LLM judges that were never part of the training reward; disagr\. is their disagreement rate\. Adding the SME rubric improves every column at bothkk\.The rubric is the referee during training, so its own score is guaranteed to rise and is by itself uninformative about whether the policy improved or merely learned the referee’s quirks\([Gao et al\., 2023](https://arxiv.org/html/2610.26932#bib.bib46);[Skalse et al\., 2022](https://arxiv.org/html/2610.26932#bib.bib47)\)\. We therefore re\-score the output the synthesized answer with two rubric\-free judges from different model families, independent of the reward\. The first judge is OpenCompass\([Cao et al\., 2026](https://arxiv.org/html/2610.26932#bib.bib45)\), while the second judge is Mistral\-7B model\([Jiang and others, 2023](https://arxiv.org/html/2610.26932#bib.bib48)\), drawn from a different model family than the SME training judge so that shared biases cannot mask a hack\.

For each evaluation question we run the pipeline end to end once\. The trained rewriter’s query retrieves the top\-kkpassages, and a*frozen*synthesizer LLM answers the question from those passages under a fixed instruction answer using*only*the provided context, say “I don’t know” if the answer is absent, and be concise with the topkkpassages supplied as numbered context and greedy decoding\. Each judge then scores that answer independently under a deliberately*rubric\-free*prompt \(shown verbatim in Figure[2](https://arxiv.org/html/2610.26932#A3.F2), Appendix[C](https://arxiv.org/html/2610.26932#A3)\): given the question, the gold answer as a reference, and the synthesized answer, it rates overall quality from11\(poor\) to1010\(excellent\) weighing correctness, completeness, and relevance and returns only the integer, with no access to our training rubric or checklist\. We normalize each score to\[0,1\]\[0,1\]by dividing by1010\. For a single question, AQ is the mean of the panel’s normalized scores; the AQ column of Table[3](https://arxiv.org/html/2610.26932#S5.T3)is that mean over all dev queries, and disagreement\. is the mean per\-question standard deviation across the two judges\. Both judges see the identical prompt and decode greedily, so a rise in AQ reflects answer quality a reward\-hacking policy could not manufacture by replaying the training signal\.

Retrieval gains reach the answer\.Table[3](https://arxiv.org/html/2610.26932#S5.T3)shows the same ordering as Table[2](https://arxiv.org/html/2610.26932#S5.T2)\. Every rubric\-trained run beats its retrieval\-only dense counterpart on Answer Quality \(AQ\) and on both judges: atk=3k\\\!=\\\!3, AQ rises→0\.75540\.7432\\\!\\rightarrow\\\!0\.7554and OpenCompass→0\.64750\.6369\\\!\\rightarrow\\\!0\.6475; atk=10k\\\!=\\\!10, AQ rises→0\.76160\.7528\\\!\\rightarrow\\\!0\.7616and OpenCompass→0\.65380\.6444\\\!\\rightarrow\\\!0\.6538\. The improvement is consistent in sign across all three independent measures, all three rubric weights, and both retrieval depths\.

Two judges, not one signal\.The two judges disagree on11\.911\.9–12\.1%12\.1\\%of items, and that rate is stable across all eight runs \(spread0\.00170\.0017\), so they are scoring independently rather than collapsing onto a single signal\. Their absolute calibration differs substantially \(≈0\.64\\approx\\\!0\.64vs\.≈0\.86\\approx\\\!0\.86\), yet they agree on the coarse ordering of the runs: both rank retrieval\-only dense lowest at each depth and every rubric\-trained run above it\. They do not agree on the fine ordering*within*the three rubric weights \(a=<<0\.85a\\\!=\\\!0\.70\\\!<\\\!0\.55\\\!<\\\!0\.85for OpenCompass,<<0\.700\.55\\\!<\\\!0\.85\\\!<\\\!0\.70for Mistral\-7B\), which is consistent with the weight sweep being a plateau rather than a ranking\.

Takeaway\.Algorithm, backbone, data, and compute are held fixed across conditions; only the reward changes\. Sparse rewards leave only∼6\.5%\\sim\\\!6\.5\\%of GRPO groups with a meaningful gradient; a dense reward built from retrieval signals alone fixes that but not the task; adding the SME\-distilled rubric produces the only configuration that improves retrieval at both depths, and those gains survive three judge\-independent measures\. The reward must be both dense enough to keep the advantage alive and grounded enough to point it somewhere useful\.

## 6\.Conclusion and Future Work

We presented a framework for training LLM agents in non\-verifiable, high\-expertise domains, where correctness is graded and expert feedback is scarce\. Our approach elicits SME judgment on a small sample of graded rewrites, distils it into an LLM\-executable rubric, validates the rubric against held\-out SME judgments, and blends its score with an objective retrieval signal into a single dense, GRPO\-compatible reward\. On a retrieval agent, this reward produced non\-zero reward variance in 44\.6% of groups versus 6\.5% under a binary reward \(6\.8×\\times\), and improved Recall@3 by up to 6% and MRR@3 by up to 6\.7% over sparse\-reward baselines\. These results suggest that reward resolution, not reward correctness alone, largely determines whether GRPO can learn from a limited expert budget\. Although we focused on query rewriting for one dataset, the pipeline makes no domain\-specific assumptions beyond the availability of a small set of SME\-graded examples, and we expect it to extend to other agentic components and to other domains, such as legal and medical, where judgment is graded and expert time is the bottleneck\. In an extension of the proposed method, adaptive group size or difficulty\-balanced prompt sampling could allocate rollouts to groups where the dense reward is most informative\. This proposed method along with its extensions can be further verified in a wider set of domains and LLM\-agents with different SME feedback requirements not only in single turn responses but also in multi\-turn responses\([Jin et al\., 2025a](https://arxiv.org/html/2610.26932#bib.bib33);[Kalyan and Andrews, 2025](https://arxiv.org/html/2610.26932#bib.bib34)\)\.

## Ethics and Privacy Statement

Our method reduces the expert effort needed to train LLM agents in regulated domains, but it also encodes a small group of SMEs’ judgments into a rubric that then acts as the reward at scale, so any bias or blind spot in those judgments can propagate into the trained rewriter and affect which documents users see\. In case of non\-verifiable tasks such as financial retrieval, they operate over proprietary and potentially sensitive documents and informs consequential decisions, a rewriter tuned to this reward should not be deployed without domain\-specific validation, access controls, and human oversight, and rubric agreement with held\-out SME judgments should not be taken as evidence of safety or fairness across user groups or institutions\. By lowering the cost of training such agents, this work could also make premature deployment easier, and we encourage practitioners to treat the validated rubric as an auditable artifact and to re\-validate it when the domain, corpus, or SME panel changes\.

## References

- Ameret al\.\(2026a\)H\. Amer, M\. Dialameh, H\. Rajabzadeh, W\. Ahmed, W\. Zhang, and Y\. LiuFLOP\-efficient training: early stopping based on test\-time compute awareness\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 35431–35449\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1)\.
- Ameret al\.\(2026b\)H\. Amer, R\. Karim, A\. Pourranjbar, W\. Zhang, W\. Ahmed, and B\. ChenDistributed hybrid parallelism for large language models: comparative study and system design guide\.arXiv preprint arXiv:2602\.09109\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1)\.
- Baiet al\.\(2022\)Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Caoet al\.\(2026\)M\. Cao, K\. Chen, H\. Duan, Y\. Fang, Z\. Fei, T\. Gao, G\. Jiaye, M\. Li, H\. Liu, J\. Liu,et al\.OpenCompass: a universal evaluation platform for large language models\.arXiv preprint arXiv:2605\.19276\.Cited by:[§5\.2](https://arxiv.org/html/2610.26932#S5.SS2.p1.1)\.
- Chaet al\.\(2025\)S\. Cha, D\. Kim, T\. Hahn, M\. Kim, Y\. Han, and B\. JeonAnnotation\-free reinforcement learning query rewriting via verifiable search reward\.arXiv preprint arXiv:2507\.23242\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Chenet al\.\(2026a\)F\. Chen, A\. Huang, N\. Golowich, S\. Malladi, A\. Block, J\. Ash, A\. Krishnamurthy, and D\. FosterThe coverage principle: how pre\-training enables post\-training\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 112225–112293\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1)\.
- Chenet al\.\(2026b\)Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. HuangDoes reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\.Advances in Neural Information Processing Systems38,pp\. 57654–57689\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1),[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Clayet al\.\(2026\)D\. Clay, S\. Gollapudi, S\. Harilal, M\. Jang, J\. Morrison, S\. Oh, and N\. JaquesDemystifying reinforcement learning post\-training of language models\.arXiv preprint arXiv:2608\.24949\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1),[§4](https://arxiv.org/html/2610.26932#S4.p2.1)\.
- Fenget al\.\(2026\)X\. Feng, Y\. Li, Z\. Wan, Z\. Gao, J\. Yuan, D\. Chen, and C\. QiaoRubricrl: simple generalizable rewards for text\-to\-image generation\.InEuropean Conference on Computer Vision,pp\. 131–149\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InInternational conference on machine learning,pp\. 10835–10866\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p3.1),[§5\.2](https://arxiv.org/html/2610.26932#S5.SS2.p1.1)\.
- Guanet al\.\(2026\)X\. Guan, X\. Hu, S\. Huang, Z\. Wang, B\. Zhang, Z\. Li, P\. Xie, B\. Liu, and J\. CaoEvoRubric: self\-evolving rubric\-driven rl for open\-ended generation\.arXiv preprint arXiv:2605\.29847\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Gunjalet al\.\(2026\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 127924–127945\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p2.1),[§2](https://arxiv.org/html/2610.26932#S2.p5.1),[§3\.1](https://arxiv.org/html/2610.26932#S3.SS1.p15.2)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p2.1),[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLora: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§4](https://arxiv.org/html/2610.26932#S4.p1.1)\.
- Jianget al\.\(2023\)A\. Q\. Jianget al\.Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§5\.2](https://arxiv.org/html/2610.26932#S5.SS2.p1.1)\.
- Jianget al\.\(2025\)P\. Jiang, J\. Lin, L\. Cao, R\. Tian, S\. Kang, Z\. Wang, J\. Sun, and J\. HanDeepretrieval: hacking real search engines and retrievers with large language models via reinforcement learning\.arXiv preprint arXiv:2503\.00223\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1),[§4](https://arxiv.org/html/2610.26932#S4.p2.1)\.
- Jinet al\.\(2025a\)B\. Jin, H\. Zeng, Z\. Yue, J\. Yoon, S\. Arik, D\. Wang, H\. Zamani, and J\. HanSearch\-r1: training llms to reason and leverage search engines with reinforcement learning\.arXiv preprint arXiv:2503\.09516\.Cited by:[§6](https://arxiv.org/html/2610.26932#S6.p1.1)\.
- Jinet al\.\(2025b\)J\. Jin, Y\. Zhu, Z\. Dou, G\. Dong, X\. Yang, C\. Zhang, T\. Zhao, Z\. Yang, and J\. WenFlashrag: a modular toolkit for efficient retrieval\-augmented generation research\.InCompanion Proceedings of the ACM on Web Conference 2025,pp\. 737–740\.Cited by:[§4](https://arxiv.org/html/2610.26932#S4.p1.1)\.
- Kalyan and Andrews \(2025\)V\. Kalyan and M\. AndrewsReinforcement learning for long\-horizon multi\-turn search agents\.arXiv preprint arXiv:2510\.24126\.Cited by:[§6](https://arxiv.org/html/2610.26932#S6.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§4](https://arxiv.org/html/2610.26932#S4.p1.1)\.
- Leeet al\.\(2023\)H\. Lee, S\. Phatale, H\. Mansoor, K\. R\. Lu, T\. Mesnard, J\. Ferret, C\. Bishop, E\. Hall, V\. Carbune, and A\. RastogiRlaif: scaling reinforcement learning from human feedback with ai feedback\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Liet al\.\(2026\)X\. Li, K\. Bao, M\. Li, Y\. Ma, Y\. Zhang, W\. Wang, F\. Feng, and D\. LiuARES: automated rubric synthesis for scalable llm reinforcement learning\.arXiv preprint arXiv:2605\.23454\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1)\.
- Liuet al\.\(2026\)T\. Liu, R\. Xu, T\. Yu, I\. Hong, C\. Yang, T\. Zhao, and H\. WangOpenrubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 17417–17437\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Liuet al\.\(2025a\)Z\. Liu, X\. Guo, Z\. Yang, F\. Lou, L\. Zeng, J\. Niu, M\. Li, Q\. Qi, Z\. Liu, Y\. Han,et al\.Fin\-r1: a large language model for financial reasoning through reinforcement learning\.arXiv preprint arXiv:2503\.16252\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1),[§2](https://arxiv.org/html/2610.26932#S2.p4.1)\.
- Liuet al\.\(2025b\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Maet al\.\(2023\)X\. Ma, Y\. Gong, P\. He, H\. Zhao, and N\. DuanQuery rewriting in retrieval\-augmented large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 5303–5315\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Matlinet al\.\(2025\)G\. Matlin, J\. Anirudh, A\. Shukla, Y\. Hassan, S\. Chava,et al\.Instruction following for finance: verifying language models’ ability to follow complex financial instructions\.InNeurIPS 2025 Workshop: Generative AI in Finance,Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1),[§2](https://arxiv.org/html/2610.26932#S2.p4.1)\.
- Moradiet al\.\(2025\)M\. M\. Moradi, H\. Amer, S\. Mudur, W\. Zhang, Y\. Liu, and W\. AhmedContinuous self\-improvement of large language models by test\-time training with verifier\-driven sample selection\.arXiv preprint arXiv:2505\.19475\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1)\.
- Nget al\.\(1999\)A\. Y\. Ng, D\. Harada, and S\. RussellPolicy invariance under reward transformations: theory and application to reward shaping\.InIcml,Vol\.99,pp\. 278–287\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1),[§3\.1](https://arxiv.org/html/2610.26932#S3.SS1.p5.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p1.1),[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Rajbhandariet al\.\(2020\)S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. HeZero: memory optimizations toward training trillion parameter models\.InSC20: international conference for high performance computing, networking, storage and analysis,pp\. 1–16\.Cited by:[§4](https://arxiv.org/html/2610.26932#S4.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Setluret al\.\(2025\)A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. KumarRewarding progress: scaling automated process verifiers for llm reasoning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 60808–60838\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1)\.
- Shaoet al\.\(2025\)R\. Shao, S\. S\. Li, R\. Xin, S\. Geng, Y\. Wang, S\. Oh, S\. S\. Du, N\. Lambert, S\. Min, R\. Krishna,et al\.Spurious rewards: rethinking training signals in rlvr\.arXiv preprint arXiv:2506\.10947\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p3.1),[§2](https://arxiv.org/html/2610.26932#S2.p1.1),[§3](https://arxiv.org/html/2610.26932#S3.p1.1)\.
- Shenget al\.\(2026\)L\. Sheng, W\. Ma, R\. Hong, X\. Wang, A\. Zhang, and T\. ChuaReinforcing chain\-of\-thought reasoning with self\-evolving rubrics\.arXiv preprint arXiv:2602\.10885\.Cited by:[§1](https://arxiv.org/html/2610.26932#S1.p2.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward gaming\.Advances in neural information processing systems35,pp\. 9460–9471\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p3.1),[§5\.2](https://arxiv.org/html/2610.26932#S5.SS2.p1.1)\.
- Tiwariet al\.\(2026\)R\. Tiwari, A\. Tomar, U\. Bamba, M\. Maheswaran, H\. Yang, M\. W\. Mahoney, K\. Keutzer, and A\. GholamiReward under attack: analyzing the robustness and hackability of process reward models\.arXiv preprint arXiv:2603\.06621\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p3.1)\.
- Uesatoet al\.\(2022\)J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. HigginsSolving math word problems with process\-and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p2.1)\.
- Wanget al\.\(2025\)Y\. Wang, H\. Zhang, L\. Pang, B\. Guo, H\. Zheng, and Z\. ZhengMaferw: query rewriting with multi\-aspect feedbacks for retrieval\-augmented large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 25434–25442\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Xuet al\.\(2026\)L\. Xu, G\. Qin, Z\. Yi, Y\. Cao, Z\. Luo, L\. Shi, and F\. KouReaLiC: reward\-aligned latent reasoning for conversational query rewriting\.Information Fusion,pp\. 104409\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p5.1)\.
- Yuet al\.\(2026\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p1.1)\.
- Zhaoet al\.\(2025\)Y\. Zhao, J\. Huang, J\. Hu, X\. Wang, Y\. Mao, D\. Zhang, Z\. Jiang, Z\. Wu, B\. Ai, A\. Wang,et al\.Swift: a scalable lightweight infrastructure for fine\-tuning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 29733–29735\.Cited by:[§4](https://arxiv.org/html/2610.26932#S4.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2610.26932#S2.p3.1)\.

## Appendix ATraining Configuration

Table[4](https://arxiv.org/html/2610.26932#A1.T4)lists the complete GRPO training configuration used for all experiments\.

Table 4\.Complete GRPO training configuration, held constant across all reward conditions\. Only the reward function and dynamic resampling differ between the dense runs and the binary control: resampling of zero\-variance groups is enabled for the dense reward but disabled for the binary baseline, since resampling would mask the very gradient stall the control is meant to expose\.
## Appendix BCross\-Depth Transfer of the Reward

Both the retrieval term of Eq\. \([4](https://arxiv.org/html/2610.26932#S3.E4)\) and the binary baseline are computed at a specific retrieval depthkk: the reward asks whether the answer document reached the topkk, and the training signal therefore inherits that cutoff\. The SME rubric terms do not intent match, entity grounding, and specificity are properties of the rewritten query, evaluated without reference to any retrieval depth\. This suggests a test of what each reward actually taught the policy\. If a rewriter has learned to satisfy one particular cutoff, moving the evaluation to a different depth should cost it; if it has learned to write better queries, it should not\.

We therefore train each reward variant atk∈\{3,10\}k\\\!\\in\\\!\\\{3,10\\\}and evaluate every resulting policy at both depths\. We call a cell*matched*when the training and evaluation depths agree \(these are the numbers reported in Table[5](https://arxiv.org/html/2610.26932#A2.T5)\) and*cross*when the policy was trained at the other depth\. Everything else backbone, retriever, synthesizer, recipe, and held\-out query set is held fixed\.

Table 5\.Cross\-depth transfer\.*matched*is train\-kk==eval\-kk;*cross*is the same reward trained at the other depth\.Δ\\Deltais cross−\-matched, so negative means the depth mismatch cost performance\. The no\-rewrite baseline involves no training and is depth\-agnostic\. Single run per cell\.Evaluated@3Evaluated@10Rewardmatched \(train@3\)cross \(train@10\)Δ\\Deltamatched \(train@10\)cross \(train@3\)Δ\\Delta*Recall \(%\)*No rewrite71\.91–83\.32–Retrieval only71\.9468\.98−2\.96\-2\.9681\.9680\.90−1\.06\-1\.06\+\+SMEa=0\.85a\\\!=\\\!0\.8574\.0073\.58−0\.42\-0\.4283\.9482\.92−1\.02\-1\.02\+\+SMEa=0\.70a\\\!=\\\!0\.7073\.6273\.56−0\.06\\mathbf\{\-0\.06\}83\.9183\.10−0\.81\\mathbf\{\-0\.81\}\+\+SMEa=0\.55a\\\!=\\\!0\.5573\.9273\.63−0\.29\-0\.2983\.9382\.86−1\.07\-1\.07*MRR*No rewrite0\.6296–0\.6505–Retrieval only0\.62960\.5955−0\.0341\-0\.03410\.62750\.6146−0\.0129\-0\.0129\+\+SMEa=0\.85a\\\!=\\\!0\.850\.65510\.6498−0\.0053\-0\.00530\.66900\.6705\+0\.0015\+0\.0015\+\+SMEa=0\.70a\\\!=\\\!0\.700\.64990\.6495−0\.0004\\mathbf\{\-0\.0004\}0\.66860\.6668−0\.0018\-0\.0018\+\+SMEa=0\.55a\\\!=\\\!0\.550\.65550\.6504−0\.0051\-0\.00510\.66960\.6713\+0\.0017\+0\.0017The SME quality term makes the reward robust to a depth mismatch\.Adding the rubric almost removes the cost of evaluating at a depth other than the one trained on\. A policy trained atk=10k\\\!=\\\!10and evaluated atk=3k\\\!=\\\!3loses only0\.060\.06–0\.420\.42recall points once the rubric is present, against2\.962\.96points for the retrieval\-only reward\. Ranking quality is preserved as well: for the SME rewards, cross\-depth MRR@10 is if anything marginally*higher*than the matched value \(e\.g\.0\.67050\.6705cross vs\.0\.66900\.6690matched ata=0\.85a\\\!=\\\!0\.85\)\. Because the rubric grades the rewritten query itself, intent, entity grounding, specificity, wellformedness, rather than whether a passage cleared a particular cutoff, the rewrite quality it rewards transfers across the retrieval depth it was trained on\.

The pure\-retrieval reward overfits to its training depth\.With no SME term, the reward is tied directly to the top\-kkretrieval signal it saw in training, so it is by far the largest loser under a depth swap:−2\.96\-2\.96recall@3 and−0\.0341\-0\.0341MRR when trained atk=10k\\\!=\\\!10and evaluated atk=3k\\\!=\\\!3\. When evaluated at a different depth, it even drops*below*the untrained baseline \(68\.9868\.98vs\.71\.9171\.91recall@3,0\.59550\.5955vs\.0\.62960\.6296MRR\), so training against the retrieval reward alone hurts, whereas every rubric\-blended policy stays above baseline under the same mismatch\.

At the deeper cutoff, the mismatch cost is small and even\.Evaluated atk=10k\\\!=\\\!10, all four rewards lose a comparable∼0\.8\\sim\\\!0\.8–1\.11\.1recall points, so the depth mismatch is both smaller and more uniform than atk=3k\\\!=\\\!3\. Ranking quality is essentially unaffected for the rubric rewards, two of the three are marginally higher cross than matched while the retrieval\-only reward still slips on MRR\. The deeper cut is more robust depth mismatch across every reward\.

## Appendix CAnswer\-Quality Judge Prompt

Figure[2](https://arxiv.org/html/2610.26932#A3.F2)gives the verbatim, rubric\-free prompt used to obtain the judge\-independent Answer Quality \(AQ\) reported in Table[3](https://arxiv.org/html/2610.26932#S5.T3)\. Both cross\-family judges receive this identical prompt; the answer being graded is synthesized by a frozen synthesizer from the top\-55retrieved passages\.

System:You are grading the quality of an ANSWER to a question\. Rate overall quality from 1 \(poor\) to 10 \(excellent\), weighing correctness, completeness, and relevance\. Do NOT use any external rubric or checklist give your own holistic judgment\. Respond with ONLY the integer score\. User:Question: \{question\} Reference answer\(s\): \{gold\} Answer to grade: \{answer\} Score \(1\-\-10\):

Figure 2\.Rubric\-free judge prompt used to obtain Answer Quality \(AQ\)\. Both cross\-family judges \(OpenCompass, Mistral\-7B\) receive this same prompt with greedy decoding and return a single integer, which we normalize to\[0,1\]\[0,1\]; per\-question AQ is the panel mean\. The synthesized answer being graded is produced by a frozen synthesizer from the top\-55retrieved passages\.
## Appendix DSME Feedback Elicitation and Annotation Protocol

SME annotation protocol\.To elicit and validate expert judgment we drew a random sample of100100\(question, rewrite\) pairs from Natural Questions, stratified across rewritten\-vs\-verbatim and hit\-vs\-miss so the sample spans the quality spectrum\. For each pair the annotator saw the question, the rewrite, the gold answer\(s\), and the top\-kkpassages the rewrite retrieved, and assigned \(i\) a headline retrieval\-quality label∈\{0,1,2\}\\in\\\{0,1,2\\\}grading the rewrite by whether its retrieved passages answer the question, and \(ii\) four rubric sub\-scores∈\{0,1,2\}\\in\\\{0,1,2\\\}\(intent fidelity, entity grounding, specificity/disambiguation, well\-formedness\) explaining*why*\. These100100labels form the gold set against which we validate the automated rubric judge \(step \(3\)\) before it is trusted as theRSMER\_\{\\text\{SME\}\}reward, and provide the few\-shot anchors in the judge prompt\.

Figure[3](https://arxiv.org/html/2610.26932#A4.F3)gives the verbatim prompt the frozen judge receives to produceRSMER\_\{\\text\{SME\}\}\.

System:You are a strict retrieval\-query reviewer\. You grade a REWRITTEN search query against a fixed rubric\. Score each criterion as an integer 0, 1, or 2 using the anchors given\. Judge only the rewrite’s quality as a search query for the user’s question \-\-do NOT try to answer the question yourself\. Respond with ONLY a minified JSON object mapping each criterion key to its integer score\. No prose, no code fences\. User:RUBRIC \(score each 0\-\-2\): \- intent\_fidelity \(weight 0\.35\): Preserves the user’s true information need without drifting to a different question or over\-narrowing it\. \[0=changes/loses the intent, 1=mostly preserved with some drift, 2=fully preserves the intent\] \- entity\_grounding \(weight 0\.25\): Surfaces the salient entities the answer depends on \(names, dates, places, domain terms\) precisely, without introducing wrong ones\. \[0=drops or corrupts key entities, 1=keeps some, 2=keeps/makes explicit all key entities correctly\] \- specificity\_disambiguation \(weight 0\.25\): Adds discriminating terms that target answer\-bearing evidence and resolves ambiguity \-\- WITHOUT keyword\-stuffing or padding\. \[0=vague/generic or stuffed with noise, 1=somewhat specific, 2=precisely specific and discriminating\] \- well\_formedness \(weight 0\.15\): A single, clean, searchable query: no chatter, meta\-text, quotes, markup, or multi\-line output\. \[0=malformed/has chatter, 1=minor issues, 2=clean single\-line query\] USER QUESTION: \{question\} REWRITTEN QUERY: \{rewrite\} Return JSON with exactly these keys: \[’intent\_fidelity’, ’entity\_grounding’, ’specificity\_disambiguation’, ’well\_formedness’\]

Figure 3\.Verbatim SME rubric judge prompt used to computeRSMER\_\{\\text\{SME\}\}\(Eq\. \([11](https://arxiv.org/html/2610.26932#S3.E11)\)\)\. The frozen judge scores each of the four criteria on a0/1/20/1/2scale from the anchors shown and returns a minified JSON object; we normalize by the top anchor \(22\) and combine with the weights of Table[1](https://arxiv.org/html/2610.26932#S4.T1)\. The criteria block is the domain\-general distillation used for the pilot; for an expert domain it is swapped for the SME\-elicited rubric while the prompt scaffold and JSON aggregation are unchanged\.![Four-stage loop turning SME judgment into a dense reward](https://arxiv.org/html/2610.26932v1/figs/fig_hero_loop.png)Figure 4\.From SME judgment to a dense reward\. A fixed budget of subject\-matter\-expert grading is elicited once, distilled into an LLM\-executable rubric of named, individually weighted criteria, and validated against the SME\-preferred ranking before it is trusted as a reward\. Only after the rubric separates gold from baseline rewrites do we optimize the rewriter against it\. Colour marks provenance, separating the stages that consume expert time from those that replay that judgment at scale\.Four\-stage loop turning SME judgment into a dense rewardA left\-to\-right flow of four boxes: Elicit, where SMEs grade a sample and articulate why; Distil, where the rubric becomes an LLM judge prompt; Validate, which checks that the rubric ranks as the SME does; and GRPO, which fine\-tunes against the validated reward\. Labels beneath the arrows name the artifact that passes between stages\. A dashed arrow returns from Validate to Distil, marked reweight or respecify before training\. Box colour encodes provenance: SME\-authored, LLM\-executed, or computed from retrieval\.
## Appendix EInterpreting the Uplift atk=3k\{=\}3versusk=10k\{=\}10

We note first that the absolute uplift atk=10k\{=\}10over few\-shot prompting is not small: our method improves R@10 from 80\.60 to 83\.94 \(\+3\.34 points, \+4\.1% relative\) and MRR@10 from 0\.6107 to 0\.6696 \(\+9\.6% relative\)\. The apparent difference from thek=3k\{=\}3gain is explained by headroom: R@10 under few\-shot prompting is already 80\.60, leaving only 19\.4 points of possible improvement, against 32\.1 atk=3k\{=\}3\. When we normalize each gain by the headroom available at that depth, the improvements are nearly identical—the method reduces the top\-3 miss rate by 19\.0% and the top\-10 miss rate by 17\.2%\. In other words, the smaller absolute gain atk=10k\{=\}10reflects the saturation of recall at depth 10, not a diminishing effect of the method\. This is consistent with the mechanism of query rewriting, which surfaces answer\-bearing passages toward the top of the ranking: the effect is largest at lowkk, which is also the operationally decisive regime for RAG, where the synthesizer is conditioned on only a handful of top passages \(five in our setup\)\. The rank\-sensitive metrics confirm the gain persists at depth—MRR@10 rises 9\.6% relative even where recall@10 is near its ceiling\.

## Appendix FRubric Quality and Robustness to Mis\-Specification

The quality of an SME\-distilled rubric is established by validation rather than assumed\. A rubric is admitted to training only after it separates gold from baseline rewrites in the SME\-preferred order and its score tracks the outcome\-grounded human retrieval\-quality label on held\-out annotated pairs \(§[3](https://arxiv.org/html/2610.26932#S3)\)\. Two properties make the reward robust to mis\-specification\. First, because the rubric is a transparent, weighted sum of named criteria rather than an opaque scalar model, a low reward is traceable to a specific criterion and can be inspected and corrected\. Second, the judge\-independent, cross\-family answer\-quality panel \(§[5\.2](https://arxiv.org/html/2610.26932#S5.SS2)\) provides an end\-to\-end check: a rubric that rewarded spurious surface features would raise the training reward while leaving these external judges unmoved, whereas we observe the two move together—the empirical signature of a reward that captures genuine quality\. A residual risk is that a rubric well\-specified on the annotated sample drifts on the broader population, which periodic SME re\-grading is intended to detect\.