On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers

# On-Policy Distillation Meets Off-Policy GRPO:Training Compact Instruction-Following Rerankers
Source: [https://arxiv.org/html/2609.01947](https://arxiv.org/html/2609.01947)
###### Abstract

Compact instruction\-following rerankers are attractive for deployment, but conventional distillation pipelines typically train students by offline imitation of teacher outputs on a fixed set of examples, constraining supervision to the teacher’s observed ranking space\. We revisit reranker distillation through the lens of reinforcement learning\.

We propose a two\-stage framework combining off\-policy teacher optimization with on\-policy student distillation\. In Stage 1, a 4B teacher reranker is strengthened with off\-policy GRPO using LLM\-judge feedback on 88K instruction\-following examples\. In Stage 2, a compact 1B student samples rankings from its own policy and receives soft teacher\-derived rewards on those rankings, coupling student exploration with knowledge transfer\.

Our strongest gains appear under distribution shift\. On MAIR\-11, the original 11\-subset, 869\-query evaluation, the proposed student reaches 0\.7670 nDCG@6, outperforming offline listwise KD by \+4\.6 points\. Controlled comparisons against offline pairwise RankNet KD and on\-policy GKD show that neither changing the offline distillation objective nor moving teacher\-distribution matching on\-policy reproduces the performance of reward\-based on\-policy distillation over student\-sampled rankings\. The advantage persists on MAIR\-Full: across all 126 tasks and 9,356 queries, the proposed method obtains the highest task\-macro point estimates among the evaluated distillation variants, reaching 0\.6808 nDCG@6 and 0\.7865 MRR@6\. It also exceeds two released 7B RL\-trained rerankers on the comparable MAIR\-11 evaluation, while the same Stage 2 training procedure consistently improves three architecturally distinct alternative student backbones\. On the 9,861\-query validation benchmark, the resulting 1B reranker achieves 0\.7624 nDCG@6 while providing a favorable quality–efficiency tradeoff relative to larger alternatives\.

## 1Introduction

Neural rerankers are now a core component of modern retrieval systems, providing fine\-grained relevance estimation beyond first\-stage retrieval pipelines\([Nogueira and Cho, 2019](https://arxiv.org/html/2609.01947#bib.bib1);[Khattab and Zaharia, 2020](https://arxiv.org/html/2609.01947#bib.bib2);[Yates et al\., 2021](https://arxiv.org/html/2609.01947#bib.bib3)\)\. Recent work extends reranking to*instruction\-following*settings, where users specify ranking criteria in natural language rather than through implicit query intent alone\([Asai et al\., 2023](https://arxiv.org/html/2609.01947#bib.bib4);[Weller et al\., 2025a](https://arxiv.org/html/2609.01947#bib.bib5);[Oh et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib6)\)\. This capability is especially important in enterprise and assistant\-facing retrieval, where ranking decisions may depend on temporal constraints, source preferences, audience requirements, or other multi\-factor instructions\.

Despite this progress, training*compact*instruction\-following rerankers that generalize robustly remains difficult\. Large proprietary rerankers offer strong performance but are costly and latency\-sensitive in production\. More fundamentally, existing student training approaches typically reduce distillation to offline imitation: the student is trained to match teacher\-provided labels or score distributions on a fixed set of examples, constraining the student to the teacher’s observed ranking space and potentially transferring teacher\-specific biases under distribution shift\.

We revisit reranker distillation from a reinforcement learning perspective\. Our central hypothesis is that effective capability transfer requires*student\-driven exploration*: the student should generate rankings from its own policy and learn from teacher\-derived rewards on those rankings\. We propose a two\-stage pipeline:Stage 1trains a 4B teacher using off\-policy GRPO with LLM\-judge feedback;Stage 2distills into a compact 1B reranker using on\-policy GRPO with soft teacher rewards\.

On\-policy distillation has recently emerged as a strong post\-training paradigm for reasoning and code generation, where the student receives teacher supervision on its own generated token trajectories\([Agarwal et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib25);[Lu and Thinking Machines Lab, 2025](https://arxiv.org/html/2609.01947#bib.bib26);[Qwen Team, 2025](https://arxiv.org/html/2609.01947#bib.bib27)\)\. We carry this paradigm into a different action space:*rankings*rather than token sequences\. A ranking is a permutation over candidate documents, and teacher supervision is a soft reward over sampled rankings rather than a logit distribution over next tokens\.

Our contributions:

- •Reward\-based on\-policy distillation for ranking\.We formulate reranker distillation as policy learning over ranking permutations: the student samples rankings from its own Plackett–Luce policy and receives soft teacher rewards on those student\-generated rankings\.
- •Controlled mechanism evidence\.On MAIR\-11, the proposed method improves over offline listwise KD by \+4\.6 nDCG@6 points\. Additional controls using offline pairwise RankNet KD \(A10\) and on\-policy GKD \(A9\) show that neither pairwise offline transfer nor on\-policy teacher\-distribution matching reproduces the performance of reward\-based on\-policy distillation\.
- •Broad OOD and architectural generalization\.The proposed method obtains the highest task\-macro point estimates among evaluated distillation variants on all 126 MAIR tasks, while the same Stage 2 procedure consistently improves three architecturally distinct alternative student backbones\. The 1B student also exceeds two released 7B RL\-trained rerankers on the comparable MAIR\-11 evaluation\.
- •Compact deployment\.The resulting 1B reranker reaches 0\.7624 nDCG@6 on the 9,861\-query validation benchmark while substantially reducing inference latency relative to 4B alternatives\.

## 2Related Work

#### Instruction Following Reranking\.

Neural rerankers improve retrieval quality by rescoring candidate documents, often using cross\-encoder architectures\([Nogueira and Cho, 2019](https://arxiv.org/html/2609.01947#bib.bib1);[Yates et al\., 2021](https://arxiv.org/html/2609.01947#bib.bib3)\)\. Recent work extends this to instruction\-following settings where models condition on natural\-language ranking criteria\([Asai et al\., 2023](https://arxiv.org/html/2609.01947#bib.bib4);[Weller et al\., 2025a](https://arxiv.org/html/2609.01947#bib.bib5);[Oh et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib6)\), supported by benchmarks such asFollowIRandInstructIR\. Systems such as ZeRank\-2\([ZeroEntropy, 2025](https://arxiv.org/html/2609.01947#bib.bib15)\)demonstrate that instruction\-following reranking benefits from RL\-based alignment, while compact open rerankers such as Llama\-Nemotron\-Rerank\-1B\-v2\([NVIDIA, 2025](https://arxiv.org/html/2609.01947#bib.bib16)\)provide strong efficiency\-oriented starting points\. We build on this setting but ask how a compact reranker should acquire instruction\-following behavior from a stronger teacher without inheriting the teacher’s fixed prediction distribution\.

#### Prompt\-based and LLM\-based Reranking\.

Promptriever\([Weller et al\., 2025b](https://arxiv.org/html/2609.01947#bib.bib21)\)makes dense retrieval models instruction\-sensitive through prompting\. RankGPT and related methods treat ranking as an inference\-time prompting problem\([Qin et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib22);[Ma et al\., 2023](https://arxiv.org/html/2609.01947#bib.bib19)\), while RankVicuna and RankZephyr show that open\-source LLMs can be effective zero\-shot listwise rerankers\([Pradeep et al\., 2023a](https://arxiv.org/html/2609.01947#bib.bib23);[Pradeep et al\., 2023b](https://arxiv.org/html/2609.01947#bib.bib20)\)\. We differ in goal: rather than large prompted LLMs at inference time, we train a compact 1B reranker through a two\-stage RL pipeline\.

#### Reinforcement Learning for Ranking\.

RL has become standard for optimizing non\-differentiable objectives in language generation and alignment\([Ouyang et al\., 2022](https://arxiv.org/html/2609.01947#bib.bib7);[Rafailov et al\., 2023](https://arxiv.org/html/2609.01947#bib.bib8)\)\. GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib9)\)provides efficient policy\-gradient optimization via group\-normalized advantages\. Prior work in offline RL emphasizes that policies trained on fixed data can be limited by distributional coverage, whereas on\-policy interaction can correct errors on states induced by the current policy\([Levine et al\., 2020](https://arxiv.org/html/2609.01947#bib.bib24)\)\. In reranking, RL has mainly been used as an alignment stage for large rerankers or in off\-policy settings\. Our method uses GRPO in two distinct roles: off\-policy for the teacher and on\-policy for student distillation\.

#### Knowledge Distillation for Retrieval\.

Knowledge distillation transfers behavior from large teachers to smaller students\([Hinton et al\., 2015](https://arxiv.org/html/2609.01947#bib.bib10)\)\. In retrieval, prior work commonly performs offline distillation from fixed teacher scores\([Hofstätter et al\., 2021](https://arxiv.org/html/2609.01947#bib.bib11)\)\. Born\-again distillation has shown that students can sometimes outperform teachers via a regularization effect from soft supervision\([Furlanello et al\., 2018](https://arxiv.org/html/2609.01947#bib.bib12)\)\. LLM\-as\-judge methods\([Zheng et al\., 2023](https://arxiv.org/html/2609.01947#bib.bib13);[Dubois et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib14)\)provide scalable soft reward signals richer than binary labels\. We depart from standard offline KD by treating distillation as policy learning: the student samples rankings from its own policy and receives teacher\-derived rewards, enabling exploration that offline KD does not\.

#### Positioning\.

Our method lies at the intersection of two previously separate directions\. Generalized on\-policy knowledge distillation trains students on self\-generated trajectories using teacher\-distribution matching\([Agarwal et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib25)\), while recent RL rerankers such as Rank\-R1, REARANK, and ERank optimize ranking behavior directly using reward signals\([Zhuang et al\., 2026](https://arxiv.org/html/2609.01947#bib.bib28);[Zhang et al\., 2025](https://arxiv.org/html/2609.01947#bib.bib29);[Cai et al\., 2026](https://arxiv.org/html/2609.01947#bib.bib34)\)\. Prior reranker\-distillation approaches instead predominantly transfer teacher information offline or address training–inference alignment through alternative objectives\([Choi et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib31);[Huang et al\., 2025](https://arxiv.org/html/2609.01947#bib.bib32);[Xu et al\., 2025](https://arxiv.org/html/2609.01947#bib.bib33)\)\. Our method occupies the intersection left open by these directions:*reward\-based on\-policy distillation over ranking permutations*\. The student samples complete rankings from its own Plackett–Luce policy, and a fixed teacher assigns a scalar soft reward to each student\-sampled permutation\. A9 makes this distinction empirical by retaining student\-generated rankings while replacing permutation\-level reward supervision with GKD\-style teacher\-distribution matching\.

## 3Method

#### Model naming conventions\.

ZeRank\-2denotes the 4B teacher backbone\([ZeroEntropy, 2025](https://arxiv.org/html/2609.01947#bib.bib15)\)\.Teacher GRPOrefers to ZeRank\-2 after Stage 1 off\-policy RL\.Base\-1Bdenotes the Llama\-Nemotron\-Rerank\-1B\-v2 student backbone\([NVIDIA, 2025](https://arxiv.org/html/2609.01947#bib.bib16)\), andDistilled\-1Bdenotes the final student after Stage 2 on\-policy distillation\.

### 3\.1Problem Setup

Given a queryqq, instructionii, and candidate document setD=\{d1,…,dn\}D=\\\{d\_\{1\},\\dots,d\_\{n\}\\\}, the reranker defines a scoring function over documents and induces a distribution over rankings\. Our training pipeline has two stages \(Figure[1](https://arxiv.org/html/2609.01947#S3.F1)\)\. Stage 1 trains a stronger teacher using off\-policy GRPO with LLM\-judge feedback\. Stage 2 distills the teacher into a compact student using on\-policy GRPO, where the student samples rankings from its own policy and receives teacher\-derived soft rewards\. The central difference from offline distillation is that supervision is generated on the student’s sampled rankings rather than on a fixed teacher\-generated distribution\.

Figure 1:Two\-stage training pipeline\. Both stages use the same GRPO machinery over Plackett–Luce–sampled rankings; they differ only in whose policy generates the rankings and what provides the reward\. Stage 1 aligns a 4B teacher with LLM\-judge feedback \(off\-policy with respect to the eventual student\)\. Stage 2 distills into a 1B student on\-policy: the student samples rankings from its own policy and receives soft teacher\-derived rewards, which drives the \+4\.6 nDCG@6 OOD improvement over offline KD \(§5\.2\)\.
### 3\.2Stage 1: Off\-Policy GRPO Teacher Training

For each training example\(q,i,D\)\(q,i,D\), the teacher induces a distribution over rankings and we sample candidate rankingsπ\(1\),…,π\(k\)\\pi^\{\(1\)\},\\dots,\\pi^\{\(k\)\}using Plackett–Luce sampling\. An LLM judge assigns a rewardr⁡\(π\(j\)\)r\(\\pi^\{\(j\)\}\)to each sampled ranking\. Following GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib9)\), we compute group\-normalized advantages

A⁡\(π\(j\)\)=r⁡\(π\(j\)\)−μσ,A\(\\pi^\{\(j\)\}\)=\\frac\{r\(\\pi^\{\(j\)\}\)\-\\mu\}\{\\sigma\},\(1\)whereμ,σ\\mu,\\sigmaare the mean and standard deviation of the sampled rewards\. The teacher is updated with

ℒteacher=\\displaystyle\\mathcal\{L\}\_\{\\text\{teacher\}\}=\{\}−𝔼π∼pθ​\[A⁡\(π\)​log⁡pθ​\(π\)\]\\displaystyle\-\\mathbb\{E\}\_\{\\pi\\sim p\_\{\\theta\}\}\\\!\\left\[A\(\\pi\)\\log p\_\{\\theta\}\(\\pi\)\\right\]\+βKLKL\(pθ∥pref\)−αentH\(pθ\)\.\\displaystyle\+\\beta\_\{\\text\{KL\}\}\\mathrm\{KL\}\(p\_\{\\theta\}\\\|p\_\{\\text\{ref\}\}\)\-\\alpha\_\{\\text\{ent\}\}H\(p\_\{\\theta\}\)\.\(2\)

### 3\.3Stage 2: On\-Policy GRPO Distillation

In Stage 2, the student samples rankings from its*own*policy:π\(1\),…,π\(k\)∼pθS\\pi^\{\(1\)\},\\dots,\\pi^\{\(k\)\}\\sim p\_\{\\theta\_\{S\}\}\. Letsjs\_\{j\}denote the student score for candidatedjd\_\{j\}\. These scores directly parameterize a Plackett–Luce distribution,

pθS​\(π\)=∏m=1nexp⁡\(sπm\)∑j∉π<mexp⁡\(sj\)\.p\_\{\\theta\_\{S\}\}\(\\pi\)=\\prod\_\{m=1\}^\{n\}\\frac\{\\exp\(s\_\{\\pi\_\{m\}\}\)\}\{\\sum\_\{j\\notin\\pi\_\{<m\}\}\\exp\(s\_\{j\}\)\}\.\(3\)
For each candidatedjd\_\{j\}, the fixed Stage 1 teacher produces a scalar utility

tj=zYesT​\(q,i,dj\)/5\.0,t\_\{j\}=z^\{T\}\_\{\\mathrm\{Yes\}\}\(q,i,d\_\{j\}\)/5\.0,\(4\)wherezYesTz^\{T\}\_\{\\mathrm\{Yes\}\}is the teacher’s final\-positionYes\-token logit\. We rank\-normalize the teacher utilities to obtain soft relevance values and score each student\-sampled permutation using nDCG@6 under those teacher\-derived relevances:

rT​\(π\)=nDCG​@​6​\(π,relT\)\.r\_\{T\}\(\\pi\)=\\mathrm\{nDCG@6\}\(\\pi;\\mathrm\{rel\}\_\{T\}\)\.\(5\)
Thus, the teacher does not generate the student’s training rankings; it*evaluates*rankings sampled by the student\. We compute the group\-normalized teacher advantage

AT​\(π\(j\)\)=rT​\(π\(j\)\)−μTσT,A\_\{T\}\(\\pi^\{\(j\)\}\)=\\frac\{r\_\{T\}\(\\pi^\{\(j\)\}\)\-\\mu\_\{T\}\}\{\\sigma\_\{T\}\},\(6\)
whereμT\\mu\_\{T\}andσT\\sigma\_\{T\}are the mean and standard deviation of teacher rewards within the sampled group\.

For the auxiliary regularizers, we use the first\-selection Plackett–Luce marginals

qS=softmax⁡\(s\),qT=softmax⁡\(t\),q\_\{S\}=\\mathrm\{softmax\}\(s\),\\qquad q\_\{T\}=\\mathrm\{softmax\}\(t\),\(7\)
and optimize

ℒstudent=\\displaystyle\\mathcal\{L\}\_\{\\text\{student\}\}=\{\}−𝔼π∼pθS​\[AT​\(π\)​log⁡pθS​\(π\)\]\\displaystyle\-\\mathbb\{E\}\_\{\\pi\\sim p\_\{\\theta\_\{S\}\}\}\\left\[A\_\{T\}\(\\pi\)\\log p\_\{\\theta\_\{S\}\}\(\\pi\)\\right\]\+λKLKL\(qS∥qT\)−αentH\(qS\)\.\\displaystyle\+\\lambda\_\{\\text\{KL\}\}\\mathrm\{KL\}\(q\_\{S\}\\\|q\_\{T\}\)\-\\alpha\_\{\\text\{ent\}\}H\(q\_\{S\}\)\.\(8\)
The KL term is therefore computed over the first\-selection Plackett–Luce marginal rather than over the combinatorial distribution of complete permutations\. The entropy bonus is analogously computed from the student’s first\-selection marginal\. The key property of the method is that supervision is evaluated on the student’s own sampled ranking space, making distillation an on\-policy learning process\.

### 3\.4Hypothesis: Why On\-Policy Distillation Generalizes

Offline distillation supervises the student only on a fixed teacher distribution, encouraging imitation of rankings the teacher already prefers\. On\-policy distillation instead evaluates teacher\-derived supervision on rankings sampled from the student’s own policy, expanding the effective coverage of the training signal\. We therefore expect the gap between offline KD and on\-policy distillation to be smallest in\-distribution and larger under distribution shift—exactly the pattern observed in our experiments\.

### 3\.5Roles of Objective Components

Our ablations show that the dominant learning signal comes from policy\-gradient optimization with teacher rewards\. The KL and entropy terms are optional stabilizers with limited effect on in\-distribution performance but can improve OOD robustness\. We retain them in the final model and analyze their effect in Section[6](https://arxiv.org/html/2609.01947#S6)\.

## 4Experimental Setup

### 4\.1Datasets

Our instruction\-following training and validation benchmark combines eight datasets spanning web search, code, mathematics, news, and multi\-hop retrieval\. The training split contains 88,755 examples and the held\-out validation benchmark contains 9,861 queries \(Table[1](https://arxiv.org/html/2609.01947#S4.T1)\)\.

Table 1:Instruction\-following reranking data used for training and validation\.For out\-of\-distribution evaluation, we use MAIR\([Sun et al\., 2024](https://arxiv.org/html/2609.01947#bib.bib30)\), a heterogeneous instructed\-retrieval benchmark containing 126 tasks across six domains\. We report two complementary evaluation settings\.

MAIR\-11denotes the original 11\-subset, 869\-query evaluation used for our controlled ablations and external\-model comparisons\. It spans ad hoc retrieval \(Core\_2017, DD\_2016\), FAQ matching \(Quora\), scientific and biomedical evidence retrieval \(SciFact, SciDocs, Trec\-Covid, NFCorpus, LitSearch\), financial retrieval \(FiQA\), and argumentative retrieval \(ArguAna, Touche\)\. These subsets were selected to reflect heterogeneous retrieval settings relevant to our target deployment scenario rather than to optimize performance on any single domain\.

MAIR\-Fullextends the evaluation to all 126 MAIR tasks, comprising 9,356 queries in total\. Because MAIR tasks vary substantially in size, we report task\-macro averages for this full\-benchmark analysis\.

#### Candidate pools and truncation\.

We use benchmark\-provided candidate pools where available and construct BM25 candidate pools otherwise\. Candidate pools are created once during preprocessing, frozen, and reused identically across all systems and ablations, preventing first\-stage retrieval variation from confounding reranker comparisons\. Model inputs are truncated to a maximum length of 512 tokens\. In the training and validation splits, typical candidate pools contain 3–18 documents per query \(10th–90th percentile\), with mean pool size 7\.5 and median 3\. The full observed range is 2–21 documents in training and 2–20 in validation\.

### 4\.2Models and Baselines

Our teacher model isZeRank\-2\([ZeroEntropy, 2025](https://arxiv.org/html/2609.01947#bib.bib15)\)\(4B\)\. Our primary student backbone isLlama\-Nemotron\-Rerank\-1B\-v2\([NVIDIA, 2025](https://arxiv.org/html/2609.01947#bib.bib16)\)\.

We compare against: \(i\) internal references \(Base\-1B, ZeRank\-2 base, Teacher GRPO\); \(ii\) controlled training ablations including supervised BCE, offline listwise KD \(A1\), off\-policy GRPO distillation \(A2\), on\-policy GRPO distillation \(A3\), distillation from the unaligned base teacher \(A8\), on\-policy GKD \(A9\), RankNet pairwise KD \(A10\), objective\-component removals, and hyperparameter sweeps; and \(iii\) external baselines including BGE Reranker v2 M3, Jina Reranker v2, Qwen3\-Reranker\-4B, Cohere Rerank v3\.5, Cohere Rerank v4\.0\-fast, Rank\-R1\-7B\([Zhuang et al\., 2026](https://arxiv.org/html/2609.01947#bib.bib28)\), and REARANK\-7B\([Zhang et al\., 2025](https://arxiv.org/html/2609.01947#bib.bib29)\)\. We exclude Jina, BGE, and Cohere from MAIR evaluation due to potential train–test contamination\. Rank\-R1\-7B and REARANK\-7B are evaluated zero\-shot, without any fine\-tuning or adaptation on MAIR, using the same frozen MAIR\-11 candidate pools as our controlled comparisons\. This isolates reranking quality while holding the first\-stage candidate set fixed across methods\.

A9 performs on\-policy GKD\-style teacher\-distribution matching on student\-sampled rankings, whereas A10 performs offline pairwise RankNet distillation from teacher preferences\. Together with A1 and A3, these controls disentangle the sampling policy from the form of teacher supervision\.

### 4\.3Training Configuration

#### Stage 1\.

Off\-policy GRPO with LLM\-judge rewards: group sizek=8k=8, learning rate5×10−65\\times 10^\{\-6\}with 3% warmup, batch size 64, 10 epochs\.

#### Stage 2\.

On\-policy GRPO with teacher\-derived soft rewards: group sizek=8k=8, temperature 1\.0, KL weight 1\.0, entropy coefficient 0\.01, learning rate2×10−62\\times 10^\{\-6\}with 3% warmup, batch size 64, 3 epochs\. Set B varies these hyperparameters\. All training on NVIDIA H200 GPUs\.

#### LLM Judge Validation\.

Because Stage 1 relies on judge\-derived rewards, we validate the signal in two ways\. Five strong LLM judges show high inter\-judge agreement on relevance ordering and on answer containment, and the Stage 1 judge reaches 92\.6% agreement with human relevance annotations \(174/188 examples, Cohen’sκ=0\.84\\kappa=0\.84\), with most disagreements being false negatives\. Full figures and per\-dataset breakdowns are provided in Appendix[H](https://arxiv.org/html/2609.01947#A8)\.

### 4\.4Evaluation Metrics

We reportnDCG@6andMRR@6as primary ranking metrics\([Järvelin and Kekäläinen, 2002](https://arxiv.org/html/2609.01947#bib.bib17);[Chapelle et al\., 2009](https://arxiv.org/html/2609.01947#bib.bib18)\)\.

For the validation benchmark, aggregate metrics are query\-micro averages\. We compute 95% confidence intervals using 10,000 percentile\-bootstrap resamples of the pooled per\-query metric values, sampling queries with replacement without stratification\.

For MAIR\-11, aggregate metrics are query\-micro averages over all 869 queries, computed identically to the validation benchmark; confidence intervals likewise use 10,000 percentile\-bootstrap resamples of the pooled per\-query metric values\. For MAIR\-Full, whose 126 tasks vary substantially in size, we instead report task\-macro averages: we first compute the mean metric within each task and then average equally across tasks, with confidence intervals obtained from 10,000 percentile\-bootstrap resamples of the resulting per\-task means, sampling tasks with replacement without stratification\.

## 5Main Results

### 5\.1Instruction\-Following Reranking Performance

Table[2](https://arxiv.org/html/2609.01947#S5.T2)reports aggregate performance on the 9,861\-query validation benchmark\.Distilled\-1Bachieves the best overall nDCG@6 among all evaluated models while remaining compact\.

Table 2:Main validation results on 9,861 instruction\-following queries\. Confidence intervals are bootstrap 95% CIs with 10,000 resamples\.Three patterns stand out\. First, RL\-based training substantially improves over the untrained 1B backbone, raising nDCG@6 from 0\.6972 to 0\.7624\. Second, the distilled student exceeds both the untrained 4B teacher backbone and the Stage 1 teacher\. We treat the \+4\.6\-point gain over offline KD on MAIR\-11, rather than the raw teacher–student gap, as the cleaner measure of the distillation contribution because the Stage 1 teacher exhibits concentrated OOD failures discussed in Section[5\.2](https://arxiv.org/html/2609.01947#S5.SS2)\. Third, Distilled\-1B remains competitive with strong external rerankers despite its compact size: it exceeds Cohere v3\.5, Cohere v4\.0\-fast, Rank\-R1\-7B, and REARANK\-7B in validation nDCG@6, while Jina v2 obtains a slightly higher MRR@6 point estimate\.

### 5\.2Out\-of\-Distribution Generalization on MAIR

Table 3:OOD results on MAIR\-11, the original 11\-subset, 869\-query evaluation\. Detailed ablation results are provided in Appendix[D](https://arxiv.org/html/2609.01947#A4)\.One feature of Table[3](https://arxiv.org/html/2609.01947#S5.T3)deserves direct comment\. Teacher GRPO reaches 0\.7422 nDCG@6 on the validation benchmark but only 0\.6880 on MAIR\-11, falling below the untrained Base\-1B \(0\.7119\)\. This degradation is concentrated rather than uniform: Teacher GRPO outperforms Base\-1B on 7 of the 11 MAIR\-11 subsets, but large failures on LitSearch and Touche dominate its aggregate decline\. The distilled student \(A3\) therefore does not uniformly outperform its teacher; a substantial part of its aggregate improvement comes from repairing these concentrated OOD failures\.

On MAIR\-11, A3 substantially outperforms off\-policy sampling \(A2\), offline listwise KD \(A1\), on\-policy GKD \(A9\), RankNet pairwise KD \(A10\), and distillation from the unaligned base teacher \(A8\)\. In particular, neither on\-policy teacher\-distribution matching \(A9\) nor offline pairwise RankNet distillation \(A10\) reproduces the OOD performance of reward\-based on\-policy distillation\. The A3–A1 gap is especially notable because the two variants are nearly tied on the validation benchmark, whereas A3 improves over offline KD by \+4\.6 nDCG@6 points on MAIR\-11\.

A3 also exceeds both released 7B RL\-trained rerankers on this comparable evaluation, reaching 0\.7670 nDCG@6 and 0\.8289 MRR@6 versus 0\.7342/0\.8092 for Rank\-R1\-7B and 0\.7315/0\.7987 for REARANK\-7B\.

We additionally evaluate the applicable distillation variants on MAIR\-Full, comprising all 126 tasks and 9,356 queries\. A3 achieves the highest task\-macro point estimates for both metrics, reaching 0\.6808 nDCG@6 and 0\.7865 MRR@6, compared with 0\.6657/0\.7565 for A9, 0\.6564/0\.7512 for A1, and 0\.6521/0\.7502 for A10\. Complete results are reported in Appendix[C](https://arxiv.org/html/2609.01947#A3)\.

## 6Ablation Study

Our ablations are designed to answer four primary mechanistic questions rather than to crown a single best configuration: \(i\) does reward\-based on\-policy distillation outperform alternative combinations of sampling policy and distillation objective? \(ii\) does*where*supervision is evaluated matter when the reward source is held fixed? \(iii\) is Stage 1 teacher strengthening necessary for effective on\-policy distillation? and \(iv\) does soft teacher supervision improve OOD robustness relative to hard labels? We then separately analyze the auxiliary KL and entropy terms\. Figure[2](https://arxiv.org/html/2609.01947#S6.F2)summarizes the validation comparison and validation–MAIR\-11 tradeoff across the core variants\. The analyses below address these questions in turn\.

![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/fig_main_comparison.png)\(a\)Validation comparison across major baselines and ablations\.
![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/fig_val_vs_mair_scatter_hugefont.png)\(b\)Validation–MAIR\-11 tradeoff across core training variants\. Top\-right is better\.

Figure 2:Core ablation evidence for the proposed training pipeline\. On\-policy student\-driven distillation provides the strongest overall tradeoff between in\-distribution fit and OOD generalization, while hard\-label supervision improves validation performance but harms transfer under distribution shift\.#### Sampling policy and supervision form\.

A9 and A10 complete a four\-way comparison of distillation strategies: A1 performs offline listwise KD, A10 performs offline pairwise RankNet KD, A9 performs GKD\-style teacher\-distribution matching on student\-sampled rankings, and A3 performs reward\-based on\-policy distillation over student\-sampled ranking permutations\. On validation, A1, A10, A9, and A3 obtain 0\.7608, 0\.7514, 0\.7476, and 0\.7624 nDCG@6, respectively\. On MAIR\-11, they obtain 0\.7212, 0\.7416, 0\.7386, and 0\.7670\. Thus, neither changing the offline distillation objective from listwise to pairwise nor performing teacher\-distribution matching on student\-sampled rankings reproduces the OOD performance of reward\-based on\-policy distillation\.

#### Where supervision is evaluated is the key mechanism\.

Replacing on\-policy student sampling with teacher\-driven off\-policy sampling \(A2\) consistently degrades performance, both on the validation benchmark and under distribution shift\. Since A2 and A3 both use teacher\-derived supervision but differ in whose policy generates the rankings, this gap isolates the role of*student exploration*rather than reward source alone\. These results support our central claim that effective distillation depends on evaluating supervision on the student’s own sampled ranking space\.

#### Teacher strengthening is a necessary precondition\.

We additionally test whether Stage 1 teacher optimization is necessary by running on\-policy GRPO from the unaligned base ZeRank\-2 teacher \(A8\)\. This variant improves over offline KD and the off\-policy student, but remains well below A3 on MAIR\-11\. On the validation benchmark, A8 remains competitive on several subsets; its main value here is as an OOD ablation\. The result indicates that the gains of the two\-stage pipeline do not come from student\-side on\-policy learning alone: they also depend on first improving the teacher with instruction\-aware RL before distillation\.

#### Soft rewards trade in\-distribution fit for OOD robustness\.

Replacing soft teacher rewards with hard labels \(A6\) yields the best validation performance among the core variants, but substantially reduces OOD performance on MAIR\-11\. This contrast exposes a clear accuracy–generalization tradeoff: hard labels fit the in\-distribution benchmark more aggressively, whereas soft rewards preserve relative preference information that regularizes the student under distribution shift\. For deployment\-oriented instruction\-following reranking, this tradeoff favors soft supervision over maximal validation fit\.

#### Policy\-gradient training is the dominant learning signal\.

Removing the KL term \(A4\) or entropy bonus \(A5\) leaves performance close to the full objective, indicating that policy\-gradient optimization with teacher rewards is the primary source of improvement\. A5 is numerically very close to A3 on both validation and MAIR\-11, while A4 is slightly stronger than A3 both in aggregate and in paired query\-level tests\. We therefore do not claim that the full objective is uniquely best\-performing in every setting\. Instead, we view A3 as a robust reference configuration that preserves the complete exploration\-aware formulation of the method, while A4 and A5 show that the auxiliary KL and entropy terms are secondary to the main gain from student\-driven on\-policy distillation with soft teacher rewards\.

To complement the aggregate tables, we also compare key variants with paired Wilcoxon signed\-rank tests over query\-level scores\. The proposed on\-policy student \(A3\) significantly outperforms offline KD \(A1\) on MAIR\-11 \(two\-sidedp=1\.95×10−9p=1\.95\\times 10^\{\-9\}; one\-sided A3\>\>A1p=9\.76×10−10p=9\.76\\times 10^\{\-10\}\), strengthening the claim that on\-policy distillation yields substantially more robust ranking behavior under distribution shift\. In contrast, the no\-KL variant \(A4\) achieves slightly stronger query\-level performance than A3 on MAIR\-11 \(two\-sidedp=0\.0269p=0\.0269; one\-sided A3<<A4p=0\.0134p=0\.0134\)\. However, this aggregate result does not imply uniform dominance: the subset\-level picture remains mixed, with A3 outperforming A4 on some MAIR\-11 subsets and A4 stronger on others\. We therefore interpret the core mechanism of the paper as student\-driven on\-policy distillation with soft teacher rewards, while treating the KL term as an optional regularizer rather than a uniformly beneficial component\.

To further illustrate out\-of\-distribution behavior, Figure[3](https://arxiv.org/html/2609.01947#S6.F3)shows MAIR\-11 performance across the core training variants and representative reference models\. The same pattern observed in Table[3](https://arxiv.org/html/2609.01947#S5.T3)holds visually at both nDCG@6 and MRR@6: on\-policy student\-driven distillation \(A3\) substantially improves over the teacher and the off\-policy variant \(A2\), while hard\-label training \(A6\) underperforms A3 under distribution shift despite its stronger in\-distribution fit\. The figure also makes clear that the gains are not limited to a single metric, but are consistent across both ranking quality and first\-hit retrieval quality\.

![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/fig_mair_bar.png)Figure 3:MAIR\-11 out\-of\-distribution performance across representative reference models and core training variants\. Error bars indicate bootstrap 95% confidence intervals\.
#### The training mechanism transfers across student architectures\.

To test whether the Stage 2 result depends on the Llama\-Nemotron\-1B student architecture, we keep the strengthened Stage 1 ZeRank\-2 teacher fixed and repeat Stage 2 training with three architecturally distinct students: BGE\-Reranker\-v2\-Gemma, MXBAI\-Rerank\-Large\-v1, and RankZephyr\-7B\. All three improve after training\. Validation nDCG@6 increases from 0\.7015 to 0\.7335 for BGE\-Gemma, 0\.7016 to 0\.7247 for MXBAI, and 0\.7132 to 0\.7416 for RankZephyr\. On MAIR\-11, the corresponding improvements are 0\.4866 to 0\.8173, 0\.6632 to 0\.7017, and 0\.7024 to 0\.7382\. The consistent gains across substantially different student architectures and reranking interfaces suggest that the Stage 2 training mechanism is not specific to the primary Llama\-Nemotron student\. Complete results are reported in Appendix[E](https://arxiv.org/html/2609.01947#A5)\.

### 6\.1Hyperparameter Sensitivity

We next test the robustness of the default training configuration by sweeping group size, Plackett–Luce temperature, KL weight, and entropy coefficient\. Across these sweeps, validation performance remains nearly flat, indicating that the method is not highly sensitive to hyperparameter tuning\.

The no\-entropy setting appears both as the Set A core ablation A5 and as the Set B sweep point B4 \(ent=0\.0\\mathrm\{ent\}=0\.0\); the two are numerically aligned as expected\.

This stability is important for two reasons\. First, it shows that the observed gains are not the result of narrow tuning around a single configuration\. Second, it clarifies the role of the auxiliary terms: while validation performance changes little across the sweeps, MAIR\-11 shows modest sensitivity to the KL coefficient, with stronger KL regularization improving robustness under distribution shift\. This is consistent with the view that KL regularization is not the main driver of learning, but can still improve generalization stability\. The full Set B plot and numeric tables are provided in Appendix[F](https://arxiv.org/html/2609.01947#A6)\.

Overall, the primary benefit comes from*on\-policy policy\-gradient training with soft teacher rewards*; KL and entropy serve as secondary regularizers rather than essential components\.

## 7Efficiency Analysis

A central motivation for compact rerankers is deployment efficiency\. Table[4](https://arxiv.org/html/2609.01947#S7.T4)compares single\-query inference latency under a fixed hardware setup\.

Table 4:Single\-query latency under a fixed hardware setup\.Distilled\-1B offers a favorable efficiency–quality tradeoff\. It is substantially faster than 4B rerankers while remaining competitive with or stronger than all evaluated baselines on ranking quality\. Compared to the teacher, the student reduces mean latency from 27\.0ms to 9\.2ms while also improving validation performance\.

## 8Discussion and Conclusion

Our results support four conclusions\. First, the strongest distillation behavior arises from combining student\-driven sampling with permutation\-level teacher rewards\. A9 and A10 show that neither on\-policy teacher\-distribution matching nor offline pairwise distillation reproduces A3’s OOD performance, while the A2–A3 comparison isolates the importance of evaluating supervision on the student’s own ranking distribution\.

Second, the effect extends beyond the original MAIR\-11 evaluation\. Across all 126 MAIR tasks, A3 obtains the highest task\-macro point estimates among the evaluated distillation variants\. A3 also exceeds the evaluated 7B Rank\-R1 and REARANK models on the comparable MAIR\-11 setting despite its substantially smaller size\.

Third, Stage 1 teacher strengthening and soft reward supervision remain complementary: distillation from the unaligned teacher \(A8\) stays below the full method under distribution shift, while hard\-label training \(A6\) fits the validation benchmark more aggressively but generalizes less robustly\.

Finally, the Stage 2 mechanism is not tied to the primary Llama\-Nemotron student: the same strengthened teacher and on\-policy procedure improves BGE\-Reranker\-v2\-Gemma, MXBAI\-Rerank\-Large\-v1, and RankZephyr\-7B, three architecturally distinct students, supporting reward\-based on\-policy distillation as a general training strategy across reranker architectures\.

## 9Limitations

Our study focuses on English\-language, text\-only reranking\. Stage 1 relies on LLM\-judge reward signals and therefore inherits potential judge bias and calibration error\. Although we validate the judge against human relevance annotations, this validation contains only 188 examples\. Most controlled ablations also use a single random seed, so small differences among closely clustered objective variants should be interpreted cautiously; Appendix[G\.3](https://arxiv.org/html/2609.01947#A7.SS3)provides additional multi\-seed evidence for A3, A4, and A5\.

We evaluate OOD behavior across all 126 MAIR tasks, but the Stage 1 teacher is not uniformly robust: on MAIR\-11 its aggregate degradation is concentrated in a small number of subsets, particularly LitSearch and Touche\. Cross\-architecture experiments show that Stage 2 transfers to three substantially different student families, but all experiments use the same strengthened ZeRank\-2 teacher\. Repeating the computationally expensive Stage 1 optimization across multiple teacher families would require additional multi\-GPU training runs and was outside our available compute budget\. Our experiments therefore establish student\-side architectural generality, while teacher\-side architectural and capability generality remain open\.

## 10Ethical Considerations

This work studies instruction\-following reranking for text retrieval and does not introduce new user\-facing generation capabilities\. The main ethical risks arise from misranking rather than content generation: instruction\-following rerankers may amplify bias present in training data, judge models, or candidate corpora, and may under\-rank relevant documents for particular domains, perspectives, or user groups\. Because our Stage 1 teacher is trained with judge\-derived reward signals, the pipeline can also inherit calibration errors or systematic preferences from the judge model\. We partially mitigate this risk by validating the judge against both other strong LLM evaluators and human relevance annotations, but these checks do not eliminate all bias\.

Our experiments are limited to English text\-only reranking and to the datasets and MAIR benchmark evaluated in the paper, so the findings should not be interpreted as guaranteeing fair or robust behavior across all domains or deployment settings\. In enterprise use, instruction\-following rerankers should be monitored with domain\-appropriate evaluation, auditing, and human oversight, especially in high\-stakes settings such as legal, medical, financial, or employment\-related retrieval\.

## Acknowledgments

This work was supported by SAP, and all training and evaluation were conducted on SAP\-provided NVIDIA H200 GPU infrastructure\. We thank our colleagues on the SAP Business AI team for helpful discussions and feedback as well as the anonymous reviewers and meta\-reviewer for suggestions that substantially improved the paper\.

Generative AI assistants were used for language editing, manuscript consistency checks and limited coding/debugging assistance\. All scientific decisions, experiments, analyses, citations, interpretations and final manuscript content were reviewed and verified by the authors\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/5be69a584901a26c521c2b51e40a4c20-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p4.1),[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1)\.
- Asaiet al\.\(2023\)A\. Asai, T\. Schick, P\. Lewis, X\. Chen, G\. Izacard, S\. Riedel, H\. Hajishirzi, and W\. YihTask\-aware retrieval with instructions\.InFindings of the Association for Computational Linguistics: ACL 2023,Toronto, Canada,pp\. 3650–3675\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.225),[Link](https://aclanthology.org/2023.findings-acl.225/)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p1.1),[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1)\.
- Caiet al\.\(2026\)Y\. Cai, Y\. Zhang, D\. Long, M\. Li, P\. Xie, and W\. ZhengERank: fusing supervised fine\-tuning and reinforcement learning for effective and efficient text reranking\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30121–30129\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i36.40261),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40261)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1)\.
- Chapelleet al\.\(2009\)O\. Chapelle, D\. Metzler, Y\. Zhang, and P\. GrinspanExpected reciprocal rank for graded relevance\.InProceedings of the 18th ACM Conference on Information and Knowledge Management,pp\. 621–630\.External Links:[Document](https://dx.doi.org/10.1145/1645953.1646033),[Link](https://doi.org/10.1145/1645953.1646033)Cited by:[§4\.4](https://arxiv.org/html/2609.01947#S4.SS4.p1.1)\.
- Choiet al\.\(2024\)N\. Choi, Y\. Lee, G\. Cho, H\. Jeong, J\. Kong, S\. Kim, K\. Park, S\. Cho, I\. Jeong, G\. Nam, S\. Han, W\. Yang, and J\. ChoiRRADistill: distilling llms’ passage ranking ability for long\-tail queries document re\-ranking on a search engine\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,Miami, Florida, US,pp\. 627–641\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.46),[Link](https://aclanthology.org/2024.emnlp-industry.46/)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.External Links:[Link](https://arxiv.org/abs/2404.04475)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px4.p1.1)\.
- Furlanelloet al\.\(2018\)T\. Furlanello, Z\. C\. Lipton, M\. Tschannen, L\. Itti, and A\. AnandkumarBorn again neural networks\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 1607–1616\.External Links:[Link](https://proceedings.mlr.press/v80/furlanello18a.html)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px4.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.External Links:[Link](https://arxiv.org/abs/1503.02531)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px4.p1.1)\.
- Hofstätteret al\.\(2021\)S\. Hofstätter, S\. Lin, J\. Yang, J\. Lin, and A\. HanburyEfficiently teaching an effective dense retriever with balanced topic aware sampling\.InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 113–122\.External Links:[Document](https://dx.doi.org/10.1145/3404835.3462891),[Link](https://doi.org/10.1145/3404835.3462891)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px4.p1.1)\.
- Huanget al\.\(2025\)S\. Huang, Z\. Ma, J\. Du, C\. Meng, W\. Wang, J\. Leng, M\. Guo, and Z\. LinGumbel reranking: differentiable end\-to\-end reranker optimization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 7142–7161\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.354),[Link](https://aclanthology.org/2025.acl-long.354/)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1)\.
- Järvelin and Kekäläinen \(2002\)K\. Järvelin and J\. KekäläinenCumulated gain\-based evaluation of ir techniques\.ACM Transactions on Information Systems20\(4\),pp\. 422–446\.External Links:[Document](https://dx.doi.org/10.1145/582415.582418),[Link](https://doi.org/10.1145/582415.582418)Cited by:[§4\.4](https://arxiv.org/html/2609.01947#S4.SS4.p1.1)\.
- Khattab and Zaharia \(2020\)O\. Khattab and M\. ZahariaColBERT: efficient and effective passage search via contextualized late interaction over BERT\.arXiv preprint arXiv:2004\.12832\.External Links:[Link](https://arxiv.org/abs/2004.12832)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p1.1)\.
- Levineet al\.\(2020\)S\. Levine, A\. Kumar, G\. Tucker, and J\. FuOffline reinforcement learning: tutorial, review, and perspectives on open problems\.arXiv preprint arXiv:2005\.01643\.External Links:[Link](https://arxiv.org/abs/2005.01643)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px3.p1.1)\.
- Lu and Thinking Machines Lab \(2025\)K\. Lu and Thinking Machines LabOn\-policy distillation\.Thinking Machines Lab: Connectionism\.External Links:[Document](https://dx.doi.org/10.64434/tml.20251026),[Link](https://thinkingmachines.ai/blog/on-policy-distillation/)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p4.1)\.
- Maet al\.\(2023\)X\. Ma, X\. Zhang, R\. Pradeep, and J\. LinZero\-shot listwise document reranking with a large language model\.arXiv preprint arXiv:2305\.02156\.External Links:[Link](https://arxiv.org/abs/2305.02156)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px2.p1.1)\.
- Nogueira and Cho \(2019\)R\. Nogueira and K\. ChoPassage re\-ranking with BERT\.arXiv preprint arXiv:1901\.04085\.External Links:[Link](https://arxiv.org/abs/1901.04085)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p1.1),[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1)\.
- NVIDIA \(2025\)NVIDIANvidia/llama\-nemotron\-rerank\-1b\-v2\.Note:Hugging Face model cardAccessed 2026\-04\-10External Links:[Link](https://huggingface.co/nvidia/llama-nemotron-rerank-1b-v2)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2609.01947#S3.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.01947#S4.SS2.p1.1)\.
- Ohet al\.\(2024\)H\. Oh, H\. Lee, S\. Ye, H\. Shin, H\. Jang, C\. Jun, and M\. SeoINSTRUCTIR: a benchmark for instruction following of information retrieval models\.arXiv preprint arXiv:2402\.14334\.External Links:[Link](https://arxiv.org/abs/2402.14334)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p1.1),[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px3.p1.1)\.
- Pradeepet al\.\(2023a\)R\. Pradeep, S\. Sharifymoghaddam, and J\. LinRankVicuna: zero\-shot listwise document reranking with open\-source large language models\.arXiv preprint arXiv:2309\.15088\.External Links:[Link](https://arxiv.org/abs/2309.15088)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px2.p1.1)\.
- Pradeepet al\.\(2023b\)R\. Pradeep, S\. Sharifymoghaddam, and J\. LinRankZephyr: effective and robust zero\-shot listwise reranking is a breeze\!\.arXiv preprint arXiv:2312\.02724\.External Links:[Link](https://arxiv.org/abs/2312.02724)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px2.p1.1)\.
- Qinet al\.\(2024\)Z\. Qin, R\. Jagerman, K\. Hui, H\. Zhuang, J\. Wu, L\. Yan, J\. Shen, T\. Liu, J\. Liu, D\. Metzler, X\. Wang, and M\. BenderskyLarge language models are effective text rankers with pairwise ranking prompting\.InFindings of the Association for Computational Linguistics: NAACL 2024,Mexico City, Mexico,pp\. 1504–1518\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.97),[Link](https://aclanthology.org/2024.findings-naacl.97/)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px2.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p4.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 53728–53741\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px3.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.External Links:[Link](https://arxiv.org/abs/2402.03300)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2609.01947#S3.SS2.p1.1)\.
- Sunet al\.\(2024\)W\. Sun, Z\. Shi, W\. J\. Long, L\. Yan, X\. Ma, Y\. Liu, M\. Cao, D\. Yin, and Z\. RenMAIR: a massive benchmark for evaluating instructed retrieval\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 14044–14067\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.778),[Link](https://aclanthology.org/2024.emnlp-main.778/)Cited by:[§4\.1](https://arxiv.org/html/2609.01947#S4.SS1.p2.1)\.
- Welleret al\.\(2025a\)O\. Weller, B\. Chang, S\. MacAvaney, K\. Lo, A\. Cohan, B\. Van Durme, D\. Lawrie, and L\. SoldainiFollowIR: evaluating and teaching information retrieval models to follow instructions\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 11926–11942\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.597),[Link](https://aclanthology.org/2025.naacl-long.597/)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p1.1),[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1)\.
- Welleret al\.\(2025b\)O\. Weller, B\. Van Durme, D\. Lawrie, A\. Paranjape, Y\. Zhang, and J\. HesselPromptriever: instruction\-trained retrievers can be prompted like language models\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/2cefdb2c4c3274b78cd450bac35228df-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2025\)Z\. Xu, Z\. Huang, S\. Zhuang, and V\. SrikumarDistillation versus contrastive learning: how to train your rerankers\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,Mumbai, India,pp\. 564–578\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-ijcnlp.33),[Link](https://aclanthology.org/2025.findings-ijcnlp.33/)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1)\.
- Yateset al\.\(2021\)A\. Yates, R\. Nogueira, and J\. LinPretrained transformers for text ranking: BERT and beyond\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials,Online,pp\. 1–4\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.naacl-tutorials.1),[Link](https://aclanthology.org/2021.naacl-tutorials.1/)Cited by:[§1](https://arxiv.org/html/2609.01947#S1.p1.1),[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1)\.
- ZeroEntropy \(2025\)ZeroEntropyZeroentropy/zerank\-2\-reranker\.Note:[https://huggingface\.co/zeroentropy/zerank\-2\-reranker](https://huggingface.co/zeroentropy/zerank-2-reranker)Hugging Face model card, 4B instruction\-following rerankerExternal Links:[Link](https://huggingface.co/zeroentropy/zerank-2-reranker)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2609.01947#S3.SS0.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2609.01947#S4.SS2.p1.1)\.
- Zhanget al\.\(2025\)L\. Zhang, B\. Wang, X\. Qiu, S\. Reddy, and A\. AgrawalREARANK: reasoning re\-ranking agent via reinforcement learning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 2458–2471\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.125),[Link](https://aclanthology.org/2025.emnlp-main.125/)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.01947#S4.SS2.p2.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px4.p1.1)\.
- Zhuanget al\.\(2026\)S\. Zhuang, X\. Ma, Z\. Yao, S\. Wang, B\. Koopman, J\. Lin, and G\. ZucconRank\-r1: enhancing reasoning in llm\-based document rerankers via reinforcement learning\.InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 4419–4425\.External Links:[Document](https://dx.doi.org/10.1145/3805712.3809961),[Link](https://doi.org/10.1145/3805712.3809961)Cited by:[§2](https://arxiv.org/html/2609.01947#S2.SS0.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.01947#S4.SS2.p2.1)\.

## Appendix AReproducibility and Compute

To support reproducibility, we release the final Distilled\-1B checkpoint, all datasets used in the paper, and the training and evaluation code for the main model\. All repositories listed below are publicly accessible and require no authentication to download\.

- •
- •
- •
- •
- •

The code repository contains the training scripts, the evaluation scripts, their corresponding launchers, and the exact training and evaluation configuration files used for the reported A3 runs, including the metric and bootstrap confidence\-interval implementations\. The HuggingFace repositories provide the final A3 checkpoint together with the three preprocessed datasets used in the paper \(the instruction\-following train/validation set, MAIR\-11, and MAIR\-Full\)\. Together these reproduce the reported validation, MAIR\-11, and MAIR\-Full results for the final model using the same candidate sets, relevance labels, metrics, and bootstrap confidence\-interval procedure used in the paper\.

All reported training and evaluation runs were conducted on NVIDIA H200 GPUs\. The Stage 1 teacher run required approximately 3 days and 23 hours on 2 H200 GPUs, the default Stage 2 on\-policy distillation run required approximately 4 hours and 50 minutes on a single H200, and the full validation\+MAIR evaluation suite required approximately 20 minutes on a single GPU\. The core reported pipeline therefore required approximately 195 H200 GPU\-hours in total, excluding non\-reported debugging and exploratory runs\.

## Appendix BAdditional Experimental Details

This appendix provides supplementary evidence for the main empirical claims of the paper\. In addition to the compact results presented in the main text, we report: \(i\) the complete MAIR\-Full evaluation across all 126 tasks and 9,356 queries, together with results on the 115 tasks not included in MAIR\-11; \(ii\) detailed MAIR\-11 and validation ablation tables, including the A9 on\-policy GKD and A10 RankNet pairwise KD controls; \(iii\) cross\-architecture transfer experiments using BGE\-Reranker\-v2\-Gemma, MXBAI\-Rerank\-Large\-v1, and RankZephyr\-7B; \(iv\) the full Set B hyperparameter sweeps; and \(v\) subset\-level, statistical, multi\-seed, and qualitative analyses that further characterize the behavior of the proposed method\.

Across these analyses, the central pattern is consistent: the strongest OOD behavior comes from combining student\-driven sampling with permutation\-level teacher rewards\. Neither offline pairwise distillation nor on\-policy teacher\-distribution matching reproduces the MAIR\-11 performance of A3, and A3 remains the strongest evaluated distillation variant when evaluation is expanded to MAIR\-Full\. At the same time, the appendix qualifies the aggregate story\. Closely related objective variants such as A4 and A5 remain tightly clustered with A3 and can outperform it on individual subsets or metrics, while the Stage 1 teacher itself exhibits concentrated rather than uniform OOD failures\. The cross\-architecture results further show that the Stage 2 training mechanism transfers across substantially different student backbones, while teacher\-side architectural generality remains outside the scope of the present experiments\.

## Appendix CMAIR\-Full Evaluation

Table[5](https://arxiv.org/html/2609.01947#A3.T5)reports task\-macro performance across the complete MAIR benchmark of 126 tasks and 9,356 queries\. A3 obtains the highest point estimate for both nDCG@6 and MRR@6 among the evaluated distillation variants, followed by A9\. This extends the main MAIR\-11 observation to a substantially broader collection of retrieval tasks\.

Table 5:MAIR\-Full evaluation across 126 tasks and 9,356 queries\. Values are task\-macro means with 95% bootstrap confidence intervals from 10,000 resamples\.### C\.1Performance on the Additional 115 MAIR Tasks

To isolate the newly added evaluation coverage, Table[6](https://arxiv.org/html/2609.01947#A3.T6)reports task\-macro performance on the 115 MAIR tasks not included in MAIR\-11, comprising 8,487 queries\. A3 again obtains the highest point estimate for both ranking metrics\.

Table 6:Task\-macro results on the 115 MAIR tasks not included in MAIR\-11, comprising 8,487 queries\. Values are means with 95% bootstrap confidence intervals\.

## Appendix DDetailed MAIR\-11 Ablation Results

Table[7](https://arxiv.org/html/2609.01947#A4.T7)reports the complete controlled ablation results on MAIR\-11\. This view complements the broader 126\-task evaluation by retaining the original 869\-query setting used for direct comparison among all core Set A variants\.

Table 7:Detailed MAIR\-11 ablation results on 869 queries\. Confidence intervals are bootstrap 95% CIs with 10,000 resamples\.### D\.1Comparable Distillation Ablations on Validation

Table[8](https://arxiv.org/html/2609.01947#A4.T8)compares the five distillation strategies for which directly comparable validation statistics are available\. A1 and A3 remain close in\-distribution, whereas the separation among the methods becomes substantially larger on MAIR\-11\.

Table 8:Comparable validation results for the primary distillation strategies on 9,861 queries\.
### D\.2Why does Teacher GRPO underperform Base\-1B on MAIR\-11?

The aggregate MAIR\-11 results might suggest that the Stage 1 teacher \(Teacher GRPO\) simply transfers worse than the untrainedBase\-1Bmodel under distribution shift\. A subset\-level analysis \(Table[9](https://arxiv.org/html/2609.01947#A4.T9); Figure[4](https://arxiv.org/html/2609.01947#A4.F4)\) shows a more specific pattern\. Teacher GRPO does*not*degrade uniformly across MAIR\-11: relative to Base\-1B, it is stronger on 7 of 11 subsets, including ArguAna, DD\_2016, FiQA, SciDocs, NFCorpus, Quora, and Trec\-Covid\. However, this improvement is offset by a small number of large OOD failures, most notably on LitSearch and Touche, which dominate the aggregate average\.

This pattern suggests that Stage 1 off\-policy RL does not merely make the teacher weaker OOD; rather, it produces a more*specialized*ranking policy\. The teacher remains strong on several subsets, but becomes brittle on a few domains that are farther from the effective support of the instruction\-following training distribution\. In particular, the extremely large drop on LitSearch indicates that the teacher’s OOD weakness is concentrated rather than uniform\.

The final A3 student does not uniformly dominate the teacher either\. Instead, its main gain comes from*repairing the teacher’s largest OOD failures*\. A3 recovers dramatically on LitSearch, Touche, and SciFact, while remaining slightly below the teacher on a number of subsets such as ArguAna, DD\_2016, SciDocs, and Trec\-Covid\. This is consistent with the paper’s main interpretation: on\-policy student distillation does not simply inherit the Stage 1 teacher, but acts as a corrective mechanism by evaluating supervision on rankings sampled from the student’s own policy\.

Table 9:Subset\-level MAIR\-11 diagnosis of Base\-1B, Teacher GRPO, and A3 using nDCG@6\. Teacher GRPO is stronger than Base\-1B on most subsets, but suffers a few large OOD failures, especially on LitSearch and Touche, which dominate its aggregate MAIR\-11 drop\. A3 does not uniformly beat the teacher; instead, its overall gain comes primarily from correcting those largest OOD failures\. Entries are within\-subset means; aggregate MAIR\-11 values quoted in the text are query\-micro averages over all 869 queries\.![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/teacher_base_a3_mair_deltas.png)Figure 4:Subset\-level MAIR\-11 diagnosis using nDCG@6 deltas\. The left bar for each subset shows the change from Base\-1B to Teacher GRPO; the right bar shows the change from Teacher GRPO to A3\. Teacher GRPO improves over Base\-1B on most subsets, but suffers a few concentrated OOD failures, especially on LitSearch and Touche\. A3’s aggregate gain comes mainly from repairing those largest teacher failures rather than uniformly dominating the teacher on every subset\.

## Appendix ECross\-Architecture Generalization

To test whether the Stage 2 training mechanism depends on the primary Llama\-Nemotron student backbone, we keep the strengthened ZeRank\-2 teacher fixed and apply the same Stage 2 procedure to three alternative student families: BGE\-Reranker\-v2\-Gemma, MXBAI\-Rerank\-Large\-v1, and RankZephyr\-7B\. These models differ substantially in backbone scale and reranking architecture, providing a direct test of student\-side architectural generality\.

#### Architecture\-specific student utilities\.

The teacher side is unchanged across all cross\-architecture experiments: the fixed Stage 1 ZeRank\-2 teacher assigns each candidate the scaled final\-positionYes\-token logit defined in Stage 2 method, which is then used to construct the rank\-normalized relevances for the nDCG@6 reward and the teacher distribution for distillation\. Only the student\-side mapping from the model’s native output to a scalar utility changes with architecture\.

For BGE\-Reranker\-v2\-Gemma, we use the rawYes\-token logit at the final prompt position as the candidate utility, without applying a sigmoid\. For MXBAI\-Rerank\-Large\-v1, we use the sequence\-classification relevance logit produced by the cross\-encoder directly, again without a sigmoid\. RankZephyr\-7B requires a different adaptation because it is a listwise generative reranker rather than a pointwise scoring model\. We first greedily generate its ranking permutation, then teacher\-force the generated permutation and assign each document a utility equal to the summed log\-probability of its document\-identifier token\(s\) at its generated position\. These document\-level utilities provide a scalar interface to the same training objective\. In every architecture, the resulting student utilities parameterize the Plackett–Luce policy used for on\-policy ranking samples and the student distribution used in the KL distillation term\. Thus, the Stage 2 objective and teacher reward construction remain fixed; only the architecture\-specific extraction of student utilities changes\.

### E\.1Validation Results

Table[10](https://arxiv.org/html/2609.01947#A5.T10)reports validation performance before and after applying Stage 2 training to each alternative student architecture\. All three backbones improve in both nDCG@6 and MRR@6, showing that the gains are not specific to the primary Llama\-Nemotron student\.

Table 10:Validation performance before and after Stage 2 training for three alternative student architectures\. The validation set contains 9,861 queries\. Values are means with 95% bootstrap confidence intervals\.
### E\.2MAIR\-11 Results

Table[11](https://arxiv.org/html/2609.01947#A5.T11)reports the corresponding MAIR\-11 results\. The improvements persist under distribution shift for all three student architectures, with particularly large gains for BGE\-Reranker\-v2\-Gemma\.

Table 11:MAIR\-11 performance before and after Stage 2 training for three alternative student architectures\. MAIR\-11 contains 869 queries across 11 subsets\.All cross\-architecture experiments use the same strengthened ZeRank\-2 teacher produced by Stage 1\. Repeating Stage 1 for multiple teacher families would require additional costly multi\-GPU teacher\-training runs and was outside our available compute budget\. We use ZeRank\-2 because it provides a strong open\-weight instruction\-following reranker teacher and keeps these experiments aligned with the central methodological question: whether reward\-based on\-policy distillation can transfer a strengthened reranking policy across heterogeneous student architectures\. These experiments therefore establish student\-side architectural generality; teacher\-side architectural generality remains open\.

## Appendix FFull Set B Numeric Ablation Results

The main paper summarizes Set B qualitatively through the robustness discussion\. Here we provide the full plot \(Figure[5](https://arxiv.org/html/2609.01947#A6.F5)\) and numeric tables for validation and MAIR\-11 so that the stability claims can be inspected directly\. These results confirm that the method is not highly sensitive to the exact default configuration: validation performance remains nearly flat across all sweeps, while MAIR\-11 shows only modest variation, most visibly in the KL sweep\.

![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/fig_set_b_sensitivity.png)Figure 5:Hyperparameter sensitivity of the default training setup\. Validation performance is nearly flat across all sweeps, while MAIR\-11 shows modest sensitivity primarily to the KL weight\. Red stars mark the default configuration used in the main results\.### F\.1Validation Results

Table[12](https://arxiv.org/html/2609.01947#A6.T12)confirms that the in\-distribution benchmark is highly stable with respect to group size, sampling temperature, KL weight, and entropy coefficient\. The narrow spread of scores suggests that the main gains are not coming from brittle tuning around a single configuration\.

Table 12:Full Set B validation results\. In\-distribution performance remains highly stable across all sweeps\.
### F\.2MAIR\-11 Results

Table[13](https://arxiv.org/html/2609.01947#A6.T13)shows the same sweeps on MAIR\-11\. While the overall method remains stable, MAIR\-11 displays somewhat stronger sensitivity to the KL coefficient than the validation set\. In particular, larger KL values improve OOD performance, consistent with our interpretation in the main text that KL is not the main source of learning but can still improve generalization stability\.

Table 13:Full Set B MAIR\-11 results\. OOD performance is broadly stable, with the most visible sensitivity appearing in the KL sweep\.

## Appendix GPer\-Dataset Validation Breakdown Across All Baselines

Aggregate validation results in the main paper hide meaningful differences across datasets\. Table[14](https://arxiv.org/html/2609.01947#A7.T14)provides the per\-dataset validation breakdown across the major baselines and the final distilled model\. For readability, the models are divided into two panels, with A3 repeated in both panels to provide a common reference\.

This view makes two points clear\. First, the distilled student is not winning only because of a single dataset; it remains competitive across web search, code, mathematics, instruction\-following, and multi\-hop retrieval settings\. Second, different baselines exhibit complementary strengths\. For example, Jina Reranker v2 is particularly strong on MS MARCO and Robust04, Cohere Rerank v4\.0\-fast is strongest on MetaMath and LeetCode, and the 7B RL\-trained rerankers reach saturation on FollowIR and InstructIR\.

Rank\-R1\-7B and REARANK\-7B are evaluated using their native sliding\-window listwise inference procedure \(window size 20, step size 10\), while the remaining models use their corresponding primary evaluation pipelines\. All systems operate on the same frozen validation candidate pools\. The overall row reports the query\-micro average across all 9,861 validation queries\.

Panel A: Internal and open\-weight reference models

Panel B: External baselines and 7B RL\-trained rerankers

Table 14:Per\-dataset validation nDCG@6 across major baselines and the final distilled model\. Models are divided into two panels for readability, with A3 repeated in both panels as a common reference\. Rank\-R1\-7B and REARANK\-7B use their native sliding\-window listwise inference procedure \(window size 20, step size 10\), whereas the remaining models use their corresponding primary evaluation pipelines\. All systems operate on the same frozen validation candidate pools\.Boldindicates the highest point estimate in each row across both panels, with ties highlighted\. Overall \(micro\) denotes the query\-micro average across all 9,861 validation queries\.The per\-dataset results show that the distilled student does not derive its aggregate performance from a single validation source\. A3 remains competitive across all eight datasets and obtains the highest or tied highest point estimate on WebQA, while remaining particularly strong on MS MARCO, MetaMath, and LeetCode\. At the same time, other models retain clear dataset\-specific strengths: Jina Reranker v2 is strongest on MS MARCO and Robust04, Cohere Rerank v4\.0\-fast is strongest on MetaMath and LeetCode, and several models reach saturation on FollowIR and InstructIR\.

The comparison with the released 7B RL\-trained rerankers shows a similarly heterogeneous pattern\. A3 exceeds both Rank\-R1\-7B and REARANK\-7B on InfoSearch, MS MARCO, MetaMath, LeetCode, and WebQA, whereas both 7B rerankers are stronger on Robust04 and reach 1\.000 on FollowIR and InstructIR\. Under query\-micro aggregation across the full 9,861\-query validation benchmark, A3 reaches 0\.762 nDCG@6, compared with 0\.750 for Rank\-R1\-7B and 0\.748 for REARANK\-7B, while using a substantially smaller model\. These results support the interpretation that the distilled model provides a strong overall quality–robustness tradeoff rather than uniformly dominating every individual validation subset\.

### G\.1Per\-Subset Comparison of A3, A4, and A5

To better understand the role of the KL and entropy terms, we compare A3 \(full objective\), A4 \(without KL\), and A5 \(without entropy\) at the per\-subset level on MAIR\-11 \(Table[15](https://arxiv.org/html/2609.01947#A7.T15)\)\. The three variants are very close in aggregate, and the subset\-level picture remains mixed: wins are split across domains, and none of the three variants consistently dominates the others\. This supports the interpretation in the main paper that policy\-gradient training with teacher\-derived soft rewards is the primary source of improvement, while KL and entropy act as auxiliary stabilizers rather than indispensable components\.

A4 is slightly strongest overall on MAIR\-11, while A3 and A5 remain very close to one another\. A3 remains strongest on several OOD subsets, including Core\_2017, DD\_2016, Trec\-Covid, and LitSearch, while A4 is strongest on others, including ArguAna, FiQA, Quora, and Touche\. A5 is also competitive across most subsets\. Overall, the absence of a consistent winner suggests that the differences among A3, A4, and A5 are modest relative to the larger gap between on\-policy student\-driven distillation and the offline or off\-policy alternatives\.

Table 15:Per\-subset MAIR\-11 comparison for A3 \(full objective\), A4 \(no KL\), and A5 \(no entropy\), reported for both nDCG@6 and MRR@6\. All three variants are very close in aggregate, and wins remain split across subsets rather than dominated by a single configuration\. Per\-subset entries are within\-subset means; the Overall MAIR\-11 row is the query\-micro average over all 869 queries and therefore does not equal the unweighted mean of the subset entries\.
### G\.2Paired Statistical Tests for Key Variant Comparisons

To complement the aggregate bootstrap confidence intervals reported in the main paper, we additionally compare key variants using paired Wilcoxon signed\-rank tests over query\-level nDCG@6 scores\. This analysis focuses on the two most important comparisons for the paper’s claims: A3 versus A1, which tests whether on\-policy distillation improves over offline KD, and A3 versus A4, which tests whether the KL term materially changes the behavior of the full objective\.

Table[16](https://arxiv.org/html/2609.01947#A7.T16)shows two qualitatively different outcomes\. First, A3 shows a small positive paired difference relative to A1 on validation and a substantially larger, statistically significant advantage on MAIR\-11\. On validation, the Wilcoxonpp\-value is extreme \(<10−320<10^\{\-320\}, below float64 resolution\) even though the bootstrap CI on the mean difference includes zero; this is not a contradiction, as the signed\-rank test measures the consistency of the sign of per\-query differences across 9,861 paired queries, whereas the CI reflects the magnitude of the mean effect\. A highly consistent but very small per\-query advantage therefore yields an extremeppalongside a near\-zero mean difference, and we accordingly read the validation comparison as directionally reliable but practically negligible, with the substantive A3–A1 gap emerging under distribution shift\. This pattern is consistent with the main result that the benefit of on\-policy student\-driven distillation becomes most pronounced under distribution shift\. Second, A4 is slightly but significantly stronger than A3 in paired tests, especially on validation\. At the same time, the per\-subset MAIR\-11 results do not show uniform dominance: A3 is strongest on several subsets, while A4 is strongest on others\. We therefore interpret A3 as a useful reference formulation of the full objective rather than as the uniquely best\-performing member of this closely related family\.

Table 16:Paired Wilcoxon signed\-rank tests over query\-level nDCG@6 scores for the two most important variant comparisons\. Positive mean differences favor A3; negative mean differences favor the comparison variant\.Figure[6](https://arxiv.org/html/2609.01947#A7.F6)provides a compact visual summary of these paired effect estimates\. The A3–A1 difference is small on validation but substantially larger on MAIR\-11, whereas the A3–A4 comparison favors A4 by a relatively small margin on both splits\.

Figure 6:Forest plot of paired query\-level nDCG@6 effect estimates for the two key variant comparisons\. Positive values favor A3; negative values favor the comparison variant\. The A3–A1 effect is small on validation and substantially larger on MAIR\-11, while A4 is slightly stronger than A3 in the paired comparisons\. Horizontal bars show bootstrap 95% confidence intervals\.
### G\.3Multi\-seed stability of A3, A4, and A5

A possible concern in Section[6](https://arxiv.org/html/2609.01947#S6)is that the close ordering among A3, A4, and A5 could reflect single\-seed noise rather than a stable relationship between these objective variants\. We therefore reran the tightly clustered A3/A4/A5 family with 3 random seeds and evaluated each run on both the validation benchmark and MAIR\-11\.

We focus on A3, A4, and A5 because these are the variants whose single\-seed results are closest in both validation and OOD evaluation, and hence are the only ones for which training\-seed variance could materially affect the interpretation\. In contrast, the larger gaps for A1, A2, A6, and A8 are already well separated from A3 in the main results and are less likely to be explained by small seed\-level fluctuations\.

Absolute values in Table[17](https://arxiv.org/html/2609.01947#A7.T17)are not directly comparable to Tables[2](https://arxiv.org/html/2609.01947#S5.T2)and[3](https://arxiv.org/html/2609.01947#S5.T3): the multi\-seed reruns select the best\-validation checkpoint per run, rather than the fixed final checkpoint reported in the main tables\. This table is therefore intended for comparing A3, A4, and A5*to one another*under a shared protocol across seeds, not for cross\-table comparison of absolute scores\.

Table[17](https://arxiv.org/html/2609.01947#A7.T17)reports the aggregate 3\-seed results, and Figure[7](https://arxiv.org/html/2609.01947#A7.F7)visualizes the same comparison for validation and MAIR\-11\. The multi\-seed results confirm that A3, A4, and A5 remain tightly clustered across seeds\. A4 is marginally strongest on the validation benchmark, while A3 is marginally strongest on MAIR\-11, with A5 remaining close to both\. This reinforces our interpretation that the main gain comes from student\-driven on\-policy learning with soft teacher rewards, while the KL and entropy terms act as secondary stabilizers rather than determining the core improvement\. We therefore treat A3/A4/A5 as a closely related family of strong variants rather than claiming a single uniformly dominant configuration\.

Table 17:Aggregate multi\-seed results for the tightly clustered A3/A4/A5 variants\. Values are mean±\\pmstandard deviation across 3 random seeds, computed under the multi\-seed rerun protocol \(best\-validation checkpoint selection\) and therefore not directly comparable to Tables[2](https://arxiv.org/html/2609.01947#S5.T2)and[3](https://arxiv.org/html/2609.01947#S5.T3); see text\.![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/multiseed_a3_a4_a5_val.png)\(a\)Validation benchmark over 3 random seeds\.
![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/multiseed_a3_a4_a5_mair.png)\(b\)MAIR\-11 OOD benchmark over 3 random seeds\.

Figure 7:Multi\-seed stability of the tightly clustered A3/A4/A5 family over 3 random seeds\. Dots show individual seeds; circles and error bars show mean±\\pmstandard deviation across seeds\. A4 is marginally strongest on the validation benchmark, while A3 is marginally strongest on MAIR\-11\. The small spread across seeds supports our interpretation that these three variants are closely related, and that the main gain comes from student\-driven on\-policy learning with soft teacher rewards, while KL and entropy act as secondary stabilizers\.
### G\.4Qualitative Comparisons

Table[18](https://arxiv.org/html/2609.01947#A7.T18)shows four representative qualitative comparisons centered on the final A3 student\. We include one example each for offline KD \(A1\), the Stage 1 teacher, hard\-label training \(A6\), and off\-policy student distillation \(A2\)\. In all four cases, A3 flips the rank\-1 decision: the relevant document is promoted to the top position, whereas the comparison model assigns rank 1 to a near\-miss or distractor\. These examples are intended as illustrations of the aggregate trends in the main paper rather than as standalone evidence\.

Two patterns recur across the examples\. First, many failures are not gross topical errors but*ranking*errors: the competing model retrieves a plausible candidate set but places the best instruction\- or claim\-aligned passage below less relevant distractors\. Second, the A3 student often improves by making finer distinctions among highly similar candidates rather than by discovering an entirely different document\. This is consistent with the broader empirical picture in the paper: student\-driven on\-policy distillation improves the ordering of candidate documents even when the relevant passage is already present in the pool\.

Table 18:Representative qualitative comparisons drawn from the automatically mined example pool\. Each case is a rank\-1 flip where A3 promotes the relevant document above the competing model\. Together, the examples illustrate four distinct effects discussed in the paper: improvement over offline KD, correction of teacher ranking errors, greater robustness than hard\-label training, and better ranking than the off\-policy student variant\.
### G\.5Offline KD vs On\-Policy Distillation

A1 and A3 are intentionally close on the in\-distribution validation benchmark: A1 reaches 0\.7608 nDCG@6 and A3 reaches 0\.7624, with heavily overlapping confidence intervals\. We therefore do not interpret the benefit of on\-policy distillation as a stronger in\-distribution fitting mechanism than offline KD\. Instead, the distinction becomes visible under distribution shift\. On MAIR\-11, A3 improves over A1 from 0\.7212 to 0\.7670 nDCG@6 and from 0\.7860 to 0\.8289 MRR@6, corresponding to gains of 4\.6 and 4\.3 points, respectively\. This result indicates that the student benefits from being trained on rankings sampled from its own policy rather than only matching a fixed teacher supervision distribution\.

This pattern is consistent with the central interpretation of the paper\. Offline KD transfers the teacher’s preferences efficiently, but only on the teacher\-supported ranking distribution\. In contrast, on\-policy distillation exposes the student to its*own*ranking errors and asks the teacher to evaluate those rankings\. The result is not a large in\-distribution gain over offline KD, but a more robust student under distribution shift\. For this reason, we treat A3 not as a universally better optimizer than A1, but as a more generalization\-oriented distillation strategy\.

## Appendix HLLM Judge Validation

Because Stage 1 relies on judge\-derived reward signals, it is important to verify that the supervision is directionally consistent across strong LLM evaluators\. We therefore compare five judges \(Gemini 2\.5, Claude Sonnet 4, GPT\-4o, Mistral, and OSS\-20B\) on criteria directly relevant to instruction\-following reranking\.

Figure[8](https://arxiv.org/html/2609.01947#A8.F8)shows that inter\-judge consistency is strong on two core dimensions\. For relevance, judges exhibit high Spearman rank correlation, indicating that they largely preserve the same ordering of candidate passages\. For answer containment, pairwise agreement is also high, suggesting that judges are broadly consistent in identifying whether a passage contains the required answer signal\. These results support our use of judge\-derived outputs as*soft*reward supervision for teacher training, while still treating them as an approximation rather than a substitute for human annotation\.

Beyond inter\-judge consistency, we also validate the Stage 1 judge against human annotations on a manually labeled relevance set \(Figure[9](https://arxiv.org/html/2609.01947#A8.F9)\)\. The LLM judge achieves 92\.6% agreement with human labels \(174/188 examples\), with Cohen’sκ=0\.84\\kappa=0\.84, indicating near\-perfect agreement\. This result suggests that the judge is not merely self\-consistent across strong LLM evaluators, but also closely aligned with human relevance judgments\. The disagreement pattern is asymmetric: most errors are false negatives, indicating that the judge is somewhat stricter than human annotators rather than overly permissive\.

![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/overall_spearman_relevance.png)\(a\)Spearman correlation: relevance
![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/overall_agreement_contains_answer.png)\(b\)Pairwise agreement: contains answer

Figure 8:Inter\-judge consistency across five LLM judges used to validate the Stage 1 reward signal\. Judges show strong agreement on relevance ranking and on whether a candidate passage contains the answer, which supports the use of judge\-derived outputs as soft supervision for teacher training\.![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/fig_human_judge_agreement.png)Figure 9:Agreement between the Stage 1 LLM judge and human relevance annotations\. The judge achieves 92\.6% agreement and Cohen’sκ=0\.84\\kappa=0\.84, with most disagreements coming from false negatives, indicating a conservative bias relative to human labeling\.### H\.1Per\-Dataset Human–Judge Agreement

To better understand whether judge–human agreement is concentrated in a small subset of the evaluation data, we report agreement broken down by dataset in Figure[10](https://arxiv.org/html/2609.01947#A8.F10)\. Agreement remains high across most datasets, with especially strong alignment on InstructIR, InfIR LeetCode, InfoSearch, and FollowIR\. The lowest agreement appears on InfIR Robust04, but this bucket is also small and does not change the overall conclusion that the judge is broadly aligned with human relevance judgments\.

![Refer to caption](https://arxiv.org/html/2609.01947v1/figs/fig_human_judge_per_dataset.png)Figure 10:Per\-dataset agreement between the LLM judge and human annotations\. The dashed line shows the overall agreement rate of 92\.6%\. Agreement is high across most datasets, with some variation in smaller buckets\.

## Appendix IExpanded Validation Visualizations Across All Baselines

This section provides subset\-level comparisons across all major baselines: Qwen3\-4B, Base\-1B, BGE v2 M3, Teacher GRPO, Cohere v3\.5, Cohere v4\-fast, Jina v2, Rank\-R1\-7B, REARANK\-7B, and A3 Distilled\-1B\. These plots complement the aggregate tables by showing where models differ across datasets\.

Figure[11](https://arxiv.org/html/2609.01947#A9.F11)summarizes per\-subset performance across all models\. Figure[12](https://arxiv.org/html/2609.01947#A9.F12)reports win/tie/loss counts against A3 Distilled\-1B across the 8 validation subsets\. Figure[13](https://arxiv.org/html/2609.01947#A9.F13)shows the overall shape of each model’s subset\-level profile, and Figure[14](https://arxiv.org/html/2609.01947#A9.F14)shows correlations among those profiles\.

![Refer to caption](https://arxiv.org/html/2609.01947v1/appendix_fullmodel_heatmap_blackfont_large.png)Figure 11:Validation nDCG@6 by dataset and model across all major baselines\. The full heatmap makes clear that the distilled student is competitive across a diverse range of subset types, while different baselines exhibit distinct strengths on specific domains\.Figure 12:Per\-dataset win/tie/loss counts relative to A3 Distilled\-1B on the validation benchmark\. The distilled student has one of the strongest overall profiles, although different baselines still win on specific subsets\.Figure 13:Radar view of validation performance profiles across all major baselines\. This visualization emphasizes the overall shape of each model’s strengths and weaknesses across the 8 validation subsets\.![Refer to caption](https://arxiv.org/html/2609.01947v1/appendix_fullmodel_correlation_heatmap_blackfont_large.png)Figure 14:Correlation of per\-dataset validation profiles across all major baselines\. Higher correlations indicate models that behave similarly across subsets, even when their aggregate scores differ\.