RAPID: Reliability-Aware Pair Importance Distillation

# RAPID: Reliability-Aware Pair Importance Distillation
Source: [https://arxiv.org/html/2609.05481](https://arxiv.org/html/2609.05481)
Ali MahdaviAffiliation:Department of Computer EngineeringAffiliation:SRC, Islamic Azad UniversityAffiliation:Tehran, IranEmail:[ali\.mahdavi@iau\.ir](mailto:)Azadeh ZamanifarAffiliation:Department of Computer EngineeringAffiliation:SRC, Islamic Azad UniversityAffiliation:Tehran, IranEmail:[azamanifar@iau\.ac\.ir](mailto:)Amir Farhad FarhadiAffiliation:School of Computer EngineeringAffiliation:Iran University of Science and TechnologyAffiliation:Tehran, IranEmail:[amfarhadi@mail\.iust\.ac\.ir](mailto:)

###### Abstract

Inter\-example relational distillation transfers a teacher’s representation geometry by matching relations between examples in a mini\-batch\. Computing all pairs costsO⁡\(B2\)O\(B^\{2\}\), while uniform subsampling may spend a limited relation budget inefficiently\. We introduce Reliability\-Aware Pair Importance Distillation \(RAPID\), which separates a reliability\-gated relational target from a full\-support adaptive pair proposal\. Reliability controls which teacher relations are emphasized; calibrated teacher entropy and detached student\-teacher residuals control which relations are evaluated\. Exact inverse\-proposal correction makes the loss and gradient estimators conditionally unbiased for the gated mini\-batch target\. We evaluate two text\-classification settings: AG News \(BERT\-to\-DistilBERT, three paired seeds,K=256K=256\) and SST\-2 \(DistilBERT\-to\-DistilBERT, three paired seeds,K=64K=64\)\. Reliability\-gated relational distillation has the highest observed student mean on both datasets \(94\.285±0\.05494\.285\\pm 0\.054% on AG News;88\.800±0\.53288\.800\\pm 0\.532% on SST\-2\), with RAPID second \(94\.241±0\.02594\.241\\pm 0\.025% and88\.685±0\.46288\.685\\pm 0\.462%\) and the cross\-entropy student at94\.154±0\.12494\.154\\pm 0\.124% and87\.271±0\.16287\.271\\pm 0\.162%\. Pilot evaluations are charged to the same total budget as main relations\. Across both settings the gated target leads the means while the adaptive proposal stays within seed noise, supporting the modular view that target reliability and evaluation priority are separable design dimensions\.

## 1Introduction

Knowledge distillation compresses a teacher by transferring predictive or representational structure to a smaller student\[[5](https://arxiv.org/html/2609.05481#bib.bib1)\]\. For pretrained language models, intermediate\-state and relational objectives complement output\-level distillation\[[15](https://arxiv.org/html/2609.05481#bib.bib2),[6](https://arxiv.org/html/2609.05481#bib.bib3),[19](https://arxiv.org/html/2609.05481#bib.bib4)\]\. We study*inter\-example*relations: the geometry among sentence representations in a mini\-batch\. Such geometry can express similarities that independent per\-example targets do not capture\.

Relational transfer creates a budget problem\. A batch ofBBexamples containsM=B⁡\(B−1\)/2M=B\(B\-1\)/2unordered relations\. Larger effective batches, multiple represented layers, and repeated relation objectives make this quadratic term increasingly costly\. Uniform subsampling controls relation evaluations but ignores that pair losses are heterogeneous\. Conversely, hard\-pair mining can alter the intended objective when sampled losses are not corrected\.

The heterogeneity raises two decisions that should not be conflated\. First,*target reliability*asks how strongly a teacher relation should contribute to supervision\. A relation involving an example on which the teacher disagrees with the observed label may warrant less weight\. Second,*evaluation priority*asks how often a relation should be evaluated to estimate that target under a limited budget\. A relation with a large student\-teacher residual may be valuable to sample even when its target contribution remains reliability weighted\.

We separate these roles \(Figure[1](https://arxiv.org/html/2609.05481#S1.F1)\)\. A normalized, label\-aware reliability gate defines the target objective\. A distinct proposal begins from calibrated teacher entropy and reliability and gradually incorporates detached student\-teacher relational residuals estimated from pilot relations\. Endpoint\- and pair\-level defensive mixtures provide full support\. Exact inverse\-proposal correction then recovers the reliability\-gated all\-pairs mini\-batch objective in expectation\. Throughout the paper,*influence*denotes estimated priority for contributing to the current relational objective; it is not a classical influence\-function approximation\.

Our contributions are:

- •a normalized reliability\-gated objective for inter\-example relational language\-model distillation;
- •a full\-support factorized pair proposal that adapts from teacher\-side information to student\-teacher residuals under a fixed total relation budget;
- •conditional unbiasedness and bounded\-correction guarantees for the loss and gradient estimators; and
- •an NLP\-first protocol that separates checkpoint selection from held\-out reporting, evaluated on AG News and SST\-2 with three paired seeds each\.

#### Summary of advantages\.

Relative to existing relational and importance\-sampled distillation methods, RAPID offers four concrete advantages\.

1. 1\.Target–proposal separation\.Existing relational objectives such as RKD\[[12](https://arxiv.org/html/2609.05481#bib.bib5)\]and CRD\[[16](https://arxiv.org/html/2609.05481#bib.bib6)\]evaluate every pair and do not separate what to trust from what to evaluate; importance\-sampled distillation\[[8](https://arxiv.org/html/2609.05481#bib.bib7)\]treats confidence and sampling probability as the same quantity\. RAPID keeps the reliability\-weighted target and the adaptive proposal as independent objects, so the gate’s role is preserved no matter how the proposal changes between epochs\.
2. 2\.Bias\-controlled correction with bounded weights\.The unconditional estimator of the relational loss under adaptive sampling is biased unless the inverse proposal is applied\. RAPID includes an exact inverse\-proposal correction and proves conditional unbiasedness \(Theorems 1–2\); the pair\-level defensive mixture further bounds the inverse weights by1/ϵ1/\\epsilon, so no realization can dominate the gradient \(Proposition 3\)\.
3. 3\.Full\-support proposal under a single fixed budget\.The pair\-level defensive mixture guaranteesqt​\(i,j\)≥ϵ/Mq\_\{t\}\(i,j\)\\geq\\epsilon/Mfor every batch relation, so every pair remains reachable even as the proposal adapts\. Pilot evaluations are charged to the same total budgetK=min⁡\{M,R,Ccap\}K=\\min\\\{M,R,C\_\{\\mathrm\{cap\}\}\\\}as main evaluations; adaptive information does not buy extra relation evaluations\.
4. 4\.Better engineering trade\-offs on the reported workload\.On AG News at matched budget, RAPID records the lowest student\-training time and the lowest peak GPU memory among the five distillation methods \(62\.4 min and 2127 MB, versus 86–131 min and 2574–2783 MB for the other distillation methods\) and stays within seed noise of the other relational methods on held\-out accuracy\. On SST\-2 it lands within 0\.115 points of the top held\-out mean atK=64K=64with near\-identical peak memory\.

None of these advantages is asserted in isolation: they are conditional on the experimental protocol \(paired seeds, selection\-separated reporting, fixed budget, single GPU\) described in the Experiments section below\.

mini\-batchℬ=\{\(xi,yi\)\}i=1B\\mathcal\{B\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{B\}teacher logitszTz\_\{T\}representationshT,hSh\_\{T\},h\_\{S\}target reliabilityri=max⁡\{rmin,pTcal​\(yi\)​mi\}r\_\{i\}=\\max\\\{r\_\{\\min\},\\sqrt\{p\_\{T\}^\{\\mathrm\{cal\}\}\(y\_\{i\}\)m\_\{i\}\}\\\}ri→wi​j\(λ\)r\_\{i\}\\rightarrow w\_\{ij\}^\{\(\\lambda\)\}gated all\-pairs targetℒλ=1M​∑i<jwi​j\(λ\)​ℓi​j\\displaystyle\\mathcal\{L\}\_\{\\lambda\}=\\frac\{1\}\{M\}\\sum\_\{i<j\}w\_\{ij\}^\{\(\\lambda\)\}\\ell\_\{ij\}evaluation prioritysistatic∝uiα​riβs\_\{i\}^\{\\mathrm\{static\}\}\\propto u\_\{i\}^\{\\alpha\}r\_\{i\}^\{\\beta\}sires∝ri​d¯isgs\_\{i\}^\{\\mathrm\{res\}\}\\propto r\_\{i\}\\bar\{d\}\_\{i\}^\{\\operatorname\{sg\}\}full\-support proposalqt​\(i,j\)=\(1−ϵ\)​qa​\(i,j\)\+ϵ/Mq\_\{t\}\(i,j\)=\(1\-\\epsilon\)q\_\{a\}\(i,j\)\+\\epsilon/Mcorrected main\-sample estimateℒ^λ=1Km​∑kwIk​Jk\(λ\)​ℓIk​JkM​qt​\(Ik,Jk\)\\displaystyle\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}=\\frac\{1\}\{K\_\{m\}\}\\sum\_\{k\}\\frac\{w\_\{I\_\{k\}J\_\{k\}\}^\{\(\\lambda\)\}\\ell\_\{I\_\{k\}J\_\{k\}\}\}\{Mq\_\{t\}\(I\_\{k\},J\_\{k\}\)\}

Figure 1:Influence\-guided approximate distillation separates the relational target \(blue\) from the proposal used to evaluate it \(green\)\. Detached pilot residuals change sampling priority, not the estimand; inverse\-proposal correction maps sampled main relations back to the reliability\-gated target\.

## 2Related Work

#### Language\-model distillation\.

Patient knowledge distillation transfers intermediate Transformer layers\[[15](https://arxiv.org/html/2609.05481#bib.bib2)\]\. TinyBERT combines embedding, hidden\-state, attention, and prediction objectives\[[6](https://arxiv.org/html/2609.05481#bib.bib3)\]\. MiniLMv2 transfers self\-attention relations\[[19](https://arxiv.org/html/2609.05481#bib.bib4)\]\. These results show that structural transfer helps\.

#### Relational and importance\-sampled distillation\.

RKD transfers pairwise distances and angles\[[12](https://arxiv.org/html/2609.05481#bib.bib5)\]\. CRD aligns teacher and student representations contrastively\[[16](https://arxiv.org/html/2609.05481#bib.bib6)\]\. Recent work continues this line with adaptive multi\-teacher weighting combined with inter\-sample similarity\-matrix transfer\[[9](https://arxiv.org/html/2609.05481#bib.bib8)\]\. Importance sampling reduces stochastic\-gradient variance when training contributions are heterogeneous\[[7](https://arxiv.org/html/2609.05481#bib.bib9)\]\. Dynamic importance sampling has been applied to output classes in knowledge distillation\[[8](https://arxiv.org/html/2609.05481#bib.bib7)\]\. Recent NLP work such as TAKE uses influence functions to score training samples for text\-classification dataset distillation\[[17](https://arxiv.org/html/2609.05481#bib.bib10)\]\. Task\-agnostic influence estimation across LM pretraining is an active thread\[[11](https://arxiv.org/html/2609.05481#bib.bib11)\]\. Broader KD surveys now organize the field along methodological and application axes\[[2](https://arxiv.org/html/2609.05481#bib.bib12)\]\. On\-policy distillation has been formalized for large language models\[[14](https://arxiv.org/html/2609.05481#bib.bib13)\]\.

#### Teacher calibration\.

Teacher confidence varies across examples and is often miscalibrated\[[4](https://arxiv.org/html/2609.05481#bib.bib14)\]\. Recent work extends this line by tying calibration to training\-time margin or curvature\[[10](https://arxiv.org/html/2609.05481#bib.bib15)\]and by feature\-modulated label smoothing\[[1](https://arxiv.org/html/2609.05481#bib.bib16)\]\.

#### Positioning\.

Our focus is orthogonal to the above\. We define a reliability\-weighted inter\-example target and estimate it with a corrected adaptive proposal\. Our gate is deliberately label aware\. It combines the calibrated probability assigned to the observed training label with the teacher’s predictive margin\. Entropy plays a separate role in the proposal, where uncertainty can indicate an informative boundary example\. This separation prevents uncertainty from simultaneously meaning “sample more often” and “trust more strongly\.” The gate is a supervised weighting signal\. It is not a claim that confidence is a calibrated probability of teacher correctness\.

## 3Method

### 3\.1Mini\-Batch Relational Target

For a labeled mini\-batchℬ=\{\(xi,yi\)\}i=1B\\mathcal\{B\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{B\}, lethTih\_\{T\}^\{i\}andhSi​\(θ\)h\_\{S\}^\{i\}\(\\theta\)be mask\-aware mean\-pooled teacher and student representations\. To specify zero\-vector behavior, define

ν⁡\(v\)=vmax⁡\(∥v∥2,εnorm\),εnorm=10−12\.\\nu\(v\)=\\frac\{v\}\{\\max\(\\lVert v\\rVert\_\{2\},\\varepsilon\_\{\\mathrm\{norm\}\}\)\},\\qquad\\varepsilon\_\{\\mathrm\{norm\}\}=10^\{\-12\}\.\(1\)The implementation\-level cosine relations and squared discrepancy are

ci​jT=⟨ν⁡\(hTi\),ν⁡\(hTj\)⟩,ci​jS=⟨ν⁡\(hSi\),ν⁡\(hSj\)⟩,ℓi​j=\(ci​jT−ci​jS\)2\.c\_\{ij\}^\{T\}=\\langle\\nu\(h\_\{T\}^\{i\}\),\\nu\(h\_\{T\}^\{j\}\)\\rangle,\\qquad c\_\{ij\}^\{S\}=\\langle\\nu\(h\_\{S\}^\{i\}\),\\nu\(h\_\{S\}^\{j\}\)\\rangle,\\quad\\ell\_\{ij\}=\(c\_\{ij\}^\{T\}\-c\_\{ij\}^\{S\}\)^\{2\}\.\(2\)The unordered relation set is𝒫ℬ=\{\(i,j\):1≤i<j≤B\}\\mathcal\{P\}\_\{\\mathcal\{B\}\}=\\\{\(i,j\):1\\leq i<j\\leq B\\\}withM=B⁡\(B−1\)/2M=B\(B\-1\)/2\. Teacher quantities are detached\. ForB<2B<2, the relational term is defined as zero\.

### 3\.2Reliability\-Gated Objective

We fit a positive scalar temperatureTcalT\_\{\\mathrm\{cal\}\}on the internal selection split by minimizing teacher negative log\-likelihood\. This calibration temperature is separate from the KD temperature\. LetpTcal=softmax⁡\(zT/Tcal\)p\_\{T\}^\{\\mathrm\{cal\}\}=\\mathrm\{softmax\}\(z\_\{T\}/T\_\{\\mathrm\{cal\}\}\), and letmim\_\{i\}be its top\-one/top\-two probability margin\. We use

ri=max⁡\(rmin,pTcal​\(yi∣xi\)​mi\)\.r\_\{i\}=\\max\\left\(r\_\{\\min\},\\sqrt\{p\_\{T\}^\{\\mathrm\{cal\}\}\(y\_\{i\}\\mid x\_\{i\}\)m\_\{i\}\}\\right\)\.\(3\)The gate is label\-aware and is therefore a supervised training signal, not a calibrated probability that the teacher is correct\. To change relative trust without silently changing the global relational coefficient, we normalize pair weights:

w¯\\displaystyle\\bar\{w\}=1M​∑i<jri​rj=\(∑iri\)2−∑iri2B⁡\(B−1\),\\displaystyle=\\frac\{1\}\{M\}\\sum\_\{i<j\}r\_\{i\}r\_\{j\}=\\frac\{\(\\sum\_\{i\}r\_\{i\}\)^\{2\}\-\\sum\_\{i\}r\_\{i\}^\{2\}\}\{B\(B\-1\)\},\(4\)w~i​j\\displaystyle\\widetilde\{w\}\_\{ij\}=ri​rjw¯,wi​j\(λ\)=\(1−λ\)\+λ​w~i​j,λ∈\[0,1\],\\displaystyle=\\frac\{r\_\{i\}r\_\{j\}\}\{\\bar\{w\}\},\\qquad w\_\{ij\}^\{\(\\lambda\)\}=\(1\-\\lambda\)\+\\lambda\\widetilde\{w\}\_\{ij\},\\qquad\\lambda\\in\[0,1\],\(5\)ℒλ​\(θ,ℬ\)\\displaystyle\\mathcal\{L\}\_\{\\lambda\}\(\\theta;\\mathcal\{B\}\)=1M​∑i<jwi​j\(λ\)​ℓi​j​\(θ\)\.\\displaystyle=\\frac\{1\}\{M\}\\sum\_\{i<j\}w\_\{ij\}^\{\(\\lambda\)\}\\ell\_\{ij\}\(\\theta\)\.\(6\)Reliability is part of the estimand\. It is not an inverse\-proposal weight\. The mixture has exact endpoints:λ=0\\lambda=0is uniform relational KD,λ=1\\lambda=1is the fully gated target, and the reported configuration usesλ=0\.5\\lambda=0\.5to partially gate unreliable relations without discarding the uniform objective\.

###### Proposition 1\(Normalized scale\-preserving gate\)\.

ForB≥2B\\geq 2and positive reliability scores,

1M​∑i<jw~i​j=1,1M​∑i<jwi​j\(λ\)=1\.\\frac\{1\}\{M\}\\sum\_\{i<j\}\\widetilde\{w\}\_\{ij\}=1,\\qquad\\frac\{1\}\{M\}\\sum\_\{i<j\}w\_\{ij\}^\{\(\\lambda\)\}=1\.\(7\)Moreover, multiplying everyrir\_\{i\}by a commonγ\>0\\gamma\>0leavesw~i​j\\widetilde\{w\}\_\{ij\},wi​j\(λ\)w\_\{ij\}^\{\(\\lambda\)\}, andℒλ\\mathcal\{L\}\_\{\\lambda\}unchanged\.

###### Proof\.

The first identity follows directly from the definition ofw¯\\bar\{w\}; the second follows by convex combination\. Underri↦γ​rir\_\{i\}\\mapsto\\gamma r\_\{i\}, bothri​rjr\_\{i\}r\_\{j\}andw¯\\bar\{w\}scale byγ2\\gamma^\{2\}and cancel\. ∎

### 3\.3Static and Adaptive Endpoint Scores

We use normalized calibrated predictive entropyui=H\(pTcal\(⋅∣xi\)\)/logCu\_\{i\}=H\(p\_\{T\}^\{\\mathrm\{cal\}\}\(\\cdot\\mid x\_\{i\}\)\)/\\log Cas the default static informativeness signal\. The static endpoint distribution is proportional to

sistatic=max⁡\(ui,εu\)α​max⁡\(ri,rmin\)β,aistatic=sistatic∑jsjstatic\.s\_\{i\}^\{\\mathrm\{static\}\}=\\max\(u\_\{i\},\\varepsilon\_\{u\}\)^\{\\alpha\}\\max\(r\_\{i\},r\_\{\\min\}\)^\{\\beta\},\\qquad a\_\{i\}^\{\\mathrm\{static\}\}=\\frac\{s\_\{i\}^\{\\mathrm\{static\}\}\}\{\\sum\_\{j\}s\_\{j\}^\{\\mathrm\{static\}\}\}\.\(8\)Including reliability in the proposal is compatible with its distinct role in the target: the variance\-optimal proposal depends on the complete gated integrand\.

After two warm\-up epochs, we linearly increase the residual mixture to at mostτmax=0\.5\\tau\_\{\\max\}=0\.5\. Wheneverτt\>0\\tau\_\{t\}\>0, we spend part of the total relation budget on IID uniform pilot relationsPk=\(Ikp,Jkp\)P\_\{k\}=\(I\_\{k\}^\{p\},J\_\{k\}^\{p\}\)\. Their detached residuals are

dPk=\|cIkp​JkpT−cIkp​JkpS,sg\|\.d\_\{P\_\{k\}\}=\|c\_\{I\_\{k\}^\{p\}J\_\{k\}^\{p\}\}^\{T\}\-c\_\{I\_\{k\}^\{p\}J\_\{k\}^\{p\}\}^\{S,\\operatorname\{sg\}\}\|\.\(9\)LetNip=∑k𝟏\{i∈Pk\}N\_\{i\}^\{p\}=\\sum\_\{k\}\\mathbf\{1\}\\\{i\\in P\_\{k\}\\\}andDip=∑k𝟏\{i∈Pk\}dPkD\_\{i\}^\{p\}=\\sum\_\{k\}\\mathbf\{1\}\\\{i\\in P\_\{k\}\\\}d\_\{P\_\{k\}\}\. The endpoint residual estimate

d¯i=Dipmax⁡\(Nip,1\)\\bar\{d\}\_\{i\}=\\frac\{D\_\{i\}^\{p\}\}\{\\max\(N\_\{i\}^\{p\},1\)\}\(10\)counts repeated pilot draws repeatedly and assigns zero before flooring to an endpoint absent from the pilot\. We form

sires=max⁡\(ri,rmin\)​max⁡\(d¯i,εd\),aires=sires∑jsjres,s\_\{i\}^\{\\mathrm\{res\}\}=\\max\(r\_\{i\},r\_\{\\min\}\)\\max\(\\bar\{d\}\_\{i\},\\varepsilon\_\{d\}\),\\qquad a\_\{i\}^\{\\mathrm\{res\}\}=\\frac\{s\_\{i\}^\{\\mathrm\{res\}\}\}\{\\sum\_\{j\}s\_\{j\}^\{\\mathrm\{res\}\}\},\(11\)and interpolate the normalized endpoint distributions:

ai\(t\)=\(1−τt\)​aistatic\+τt​aires\.a\_\{i\}^\{\(t\)\}=\(1\-\\tau\_\{t\}\)a\_\{i\}^\{\\mathrm\{static\}\}\+\\tau\_\{t\}a\_\{i\}^\{\\mathrm\{res\}\}\.\(12\)ForEEepochs andW=min⁡\(W0,E−1\)W=\\min\(W\_\{0\},E\-1\)warm\-up epochs, the schedule is

τt=\{0,t≤W,τmax​t−WE−W,W<t≤E\.\\tau\_\{t\}=\\begin\{cases\}0,&t\\leq W,\\\\\[2\.84526pt\] \\tau\_\{\\max\}\\dfrac\{t\-W\}\{E\-W\},&W<t\\leq E\.\\end\{cases\}\(13\)The reported configuration usesW0=2W\_\{0\}=2,τmax=0\.5\\tau\_\{\\max\}=0\.5, andα=β=1\\alpha=\\beta=1\. Pilot estimates affect only proposal efficiency; they do not define the target\.

### 3\.4Full\-Support Pair Proposal

Draw two endpoints independently fromaaand reject equal endpoints\. The induced probability of unordered pair\(i,j\)\(i,j\)is

Z⁡\(a\)=1−∑kak2,qa​\(i,j\)=2​ai​ajZ⁡\(a\)\.Z\(a\)=1\-\\sum\_\{k\}a\_\{k\}^\{2\},\\qquad q\_\{a\}\(i,j\)=\\frac\{2a\_\{i\}a\_\{j\}\}\{Z\(a\)\}\.\(14\)HereZ⁡\(a\)Z\(a\)is the acceptance probability of one ordered endpoint draw\. The two possible orders produce the factor of two\. Before pair construction we apply a small endpoint\-level defensive mixture,a~i=\(1−ϵe\)​ai\+ϵe/B\\tilde\{a\}\_\{i\}=\(1\-\\epsilon\_\{e\}\)a\_\{i\}\+\\epsilon\_\{e\}/B\. We then use the pair\-level defensive mixture

qt​\(i,j\)=\(1−ϵ\)​qa~\(t\)​\(i,j\)\+ϵ/M,q\_\{t\}\(i,j\)=\(1\-\\epsilon\)q\_\{\\tilde\{a\}^\{\(t\)\}\}\(i,j\)\+\\epsilon/M,\(15\)which guarantees support over every batch relation\. After spendingKpK\_\{p\}pilot evaluations, we drawKm=K−KpK\_\{m\}=K\-K\_\{p\}main relations with replacement\. Integer allocation makes the active pilot share approximately 10%; batches with fewer thanKKavailable relations use all relations directly\. The estimator is

ℒ^λ=1Km​∑k=1KmwIk​Jk\(λ\)​ℓIk​JkM​qt​\(Ik,Jk\)\.\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}=\\frac\{1\}\{K\_\{m\}\}\\sum\_\{k=1\}^\{K\_\{m\}\}\\frac\{w\_\{I\_\{k\}J\_\{k\}\}^\{\(\\lambda\)\}\\ell\_\{I\_\{k\}J\_\{k\}\}\}\{Mq\_\{t\}\(I\_\{k\},J\_\{k\}\)\}\.\(16\)
###### Proposition 2\(Proposal normalization\)\.

For any endpoint distributionaawithZ⁡\(a\)\>0Z\(a\)\>0, the factorized pair probabilities are normalized:∑i<jqa​\(i,j\)=1\\sum\_\{i<j\}q\_\{a\}\(i,j\)=1\.

###### Proof\.

2​∑i<jai​aj=\(∑iai\)2−∑iai2=1−∑iai22\\sum\_\{i<j\}a\_\{i\}a\_\{j\}=\(\\sum\_\{i\}a\_\{i\}\)^\{2\}\-\\sum\_\{i\}a\_\{i\}^\{2\}=1\-\\sum\_\{i\}a\_\{i\}^\{2\}\. ∎

Letℱt\\mathscr\{F\}\_\{t\}contain the current mini\-batch and labels, realized teacher outputs, current parameters, detached student representations used to build the proposal, the pilot sample, and all resulting endpoint scores\. Conditional onℱt\\mathscr\{F\}\_\{t\},qtq\_\{t\}is fixed and the main relations are IID draws from it\.

###### Theorem 1\(Conditional loss unbiasedness\)\.

AssumeKm≥1K\_\{m\}\\geq 1, clipping is disabled, and the numerical probability floor is inactive\. Then

𝔼⁡\[ℒ^λ∣ℱt\]=ℒλ​\(θ,ℬ\)\.\\mathbb\{E\}\[\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathscr\{F\}\_\{t\}\]=\\mathcal\{L\}\_\{\\lambda\}\(\\theta;\\mathcal\{B\}\)\.\(17\)

###### Proof\.

Writingfi​j=wi​j\(λ\)​ℓi​jf\_\{ij\}=w\_\{ij\}^\{\(\\lambda\)\}\\ell\_\{ij\}, linearity and IID sampling give

𝔼⁡\[ℒ^λ∣ℱt\]=∑i<jqt​\(i,j\)​fi​jM​qt​\(i,j\)=1M​∑i<jfi​j=ℒλ\.\\mathbb\{E\}\[\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathscr\{F\}\_\{t\}\]=\\sum\_\{i<j\}q\_\{t\}\(i,j\)\\frac\{f\_\{ij\}\}\{Mq\_\{t\}\(i,j\)\}=\\frac\{1\}\{M\}\\sum\_\{i<j\}f\_\{ij\}=\\mathcal\{L\}\_\{\\lambda\}\.\(18\)∎

###### Theorem 2\(Conditional stopped\-proposal gradient unbiasedness\)\.

Suppose reliability, pilot residuals, proposal probabilities, and sampled indices are constants for automatic differentiation, as in the implementation\. At differentiable parameter values,

𝔼⁡\[∇θsgℒ^λ∣ℱt\]=∇θℒλ\.\\mathbb\{E\}\[\\nabla\_\{\\theta\}^\{\\operatorname\{sg\}\}\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathscr\{F\}\_\{t\}\]=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\lambda\}\.\(19\)

###### Proof\.

Conditioned onℱt\\mathscr\{F\}\_\{t\}, differentiate onlyfi​j​\(θ\)f\_\{ij\}\(\\theta\)and apply the same cancellation pairwise\. The finite sum permits termwise differentiation\. This is the stopped\-proposal autograd gradient; it does not differentiate through categorical sampling or add a score\-function term\. ∎

###### Proposition 3\(Bounded correction\)\.

The defensive mixture gives1/\(M​qt​\(i,j\)\)≤1/ϵ1/\(Mq\_\{t\}\(i,j\)\)\\leq 1/\\epsilonfor every pair\.

### 3\.5Variance and the Ideal Proposal

Conditional unbiasedness specifies the target but does not guarantee lower variance\. For IID main draws, the scalar conditional variance is

Var⁡\(ℒ^λ∣ℱt\)=1Km​\[1M2​∑i<jfi​j2qt​\(i,j\)−ℒλ2\]\.\\operatorname\{Var\}\(\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathscr\{F\}\_\{t\}\)=\\frac\{1\}\{K\_\{m\}\}\\left\[\\frac\{1\}\{M^\{2\}\}\\sum\_\{i<j\}\\frac\{f\_\{ij\}^\{2\}\}\{q\_\{t\}\(i,j\)\}\-\\mathcal\{L\}\_\{\\lambda\}^\{2\}\\right\]\.\(20\)The pilot affects this variance through both the realized proposal and the reduction from total budgetKKto main budgetKmK\_\{m\}\.

###### Proposition 4\(Unrestricted scalar optimum\)\.

IfF=∑i<jfi​j\>0F=\\sum\_\{i<j\}f\_\{ij\}\>0, the proposal minimizing Equation \([20](https://arxiv.org/html/2609.05481#S3.E20)\) over unrestricted pair distributions is

qloss⋆​\(i,j\)=fi​jF=wi​j\(λ\)​ℓi​j∑u<vwu​v\(λ\)​ℓu​v\.q\_\{\\mathrm\{loss\}\}^\{\\star\}\(i,j\)=\\frac\{f\_\{ij\}\}\{F\}=\\frac\{w\_\{ij\}^\{\(\\lambda\)\}\\ell\_\{ij\}\}\{\\sum\_\{u<v\}w\_\{uv\}^\{\(\\lambda\)\}\\ell\_\{uv\}\}\.\(21\)

###### Proof\.

Cauchy–Schwarz gives\(∑fi​j\)2≤\(∑fi​j2/qi​j\)​\(∑qi​j\)\(\\sum f\_\{ij\}\)^\{2\}\\leq\(\\sum f\_\{ij\}^\{2\}/q\_\{ij\}\)\(\\sum q\_\{ij\}\), with equality exactly whenqi​j∝fi​jq\_\{ij\}\\propto f\_\{ij\}\. ∎

The ideal depends on unavailable current pair losses\. RAPID instead uses anO⁡\(B\)O\(B\)\-state endpoint proxy built from entropy, reliability, and sparsely observed absolute residuals\. Sinceℓi​j=di​j2\\ell\_\{ij\}=d\_\{ij\}^\{2\}while the residual score usesdi​jd\_\{ij\}, the adaptive proposal is a tractable proxy rather than an optimizer of Equation \([20](https://arxiv.org/html/2609.05481#S3.E20)\)\.

###### Proposition 5\(Endpoint\-factorization restriction\)\.

For any four distinct endpointsi,j,k,li,j,k,l, a factorized proposal satisfies

qa​\(i,j\)​qa​\(k,l\)=qa​\(i,k\)​qa​\(j,l\)=qa​\(i,l\)​qa​\(j,k\)\.q\_\{a\}\(i,j\)q\_\{a\}\(k,l\)=q\_\{a\}\(i,k\)q\_\{a\}\(j,l\)=q\_\{a\}\(i,l\)q\_\{a\}\(j,k\)\.\(22\)Thus, forB≥4B\\geq 4, the factorized family cannot represent an arbitrary pair distribution\.

###### Proof\.

Each product equals\(2/Z⁡\(a\)\)2​ai​aj​ak​al\(2/Z\(a\)\)^\{2\}a\_\{i\}a\_\{j\}a\_\{k\}a\_\{l\}\. ∎

The restriction trades proposal expressivity for linear endpoint state\. The defensive mixtures further trade variance optimality for support and bounded corrections\.

### 3\.6Exact Budget Accounting

LetRRbe the requested relation count andCcapC\_\{\\mathrm\{cap\}\}the safety cap\. The effective upper budget for a batch is

K=min⁡\{M,R,Ccap\}\.K=\\min\\\{M,R,C\_\{\\mathrm\{cap\}\}\\\}\.\(23\)When adaptation is active, pilot fractionρ\\rhois converted to an integer by

Kp\\displaystyle K\_\{p\}=min⁡\{max⁡\(1,⌊ρ​K\+12⌋\),max⁡\(K−1,0\)\},\\displaystyle=\\min\\\!\\left\\\{\\max\\\!\\left\(1,\\left\\lfloor\\rho K\+\\tfrac\{1\}\{2\}\\right\\rfloor\\right\),\\max\(K\-1,0\)\\right\\\},\(24\)Km\\displaystyle K\_\{m\}=K−Kp\.\\displaystyle=K\-K\_\{p\}\.\(25\)During warm\-up,Kp=0K\_\{p\}=0andKm=KK\_\{m\}=K\. IfM≥1M\\geq 1andK=MK=M, the sampler bypasses pilots, enumerates every available pair once, setsq⁡\(i,j\)=1/Mq\(i,j\)=1/M, and recovers the exact relational objective\. HenceKKis an upper evaluation budget, not a required count for undersized final batches\. Figure[2](https://arxiv.org/html/2609.05481#S3.F2)summarizes the schedule and accounting\.

warm\-up\(t≤Wt\\leq W\)adaptive\(t\>Wt\>W\)τt↗τmax\\tau\_\{t\}\\nearrow\\tau\_\{\\max\}KKmain evaluations, static proposalKpK\_\{p\}pilotKmK\_\{m\}main evaluations, adaptive proposalsame total budgetK=min⁡\{M,R,Ccap\}K=\\min\\\{M,R,C\_\{\\mathrm\{cap\}\}\\\}

Figure 2:The total relation budgetK=min⁡\{M,R,Ccap\}K=\\min\\\{M,R,C\_\{\\mathrm\{cap\}\}\\\}includes pilot evaluations\. Warm\-up spends the entire budget on main draws from the static proposal; adaptation reallocates part of the same total to uniform pilots and draws the remaining main relations from the interpolated adaptive proposal\. IfM≤min⁡\(R,Ccap\)M\\leq\\min\(R,C\_\{\\mathrm\{cap\}\}\), the sampler enumerates allMMpairs exactly instead of sampling\.
### 3\.7Training Objective and Complexity

For classification, the final objective is deliberately minimal:

ℒ=ℒCE\+αKD​TKD2​KL⁡\(pT\(TKD\)∥pS\(TKD\)\)\+βrel​ℒ^λ\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\alpha\_\{\\mathrm\{KD\}\}T\_\{\\mathrm\{KD\}\}^\{2\}\\operatorname\{KL\}\\\!\\left\(p\_\{T\}^\{\(T\_\{\\mathrm\{KD\}\}\)\}\\middle\\\|p\_\{S\}^\{\(T\_\{\\mathrm\{KD\}\}\)\}\\right\)\+\\beta\_\{\\mathrm\{rel\}\}\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\.\(26\)RAPID uses the same temperature\-scaled KD term as the KD and relational controls; direct feature MSE is excluded\. Given already\-computed representations, all\-pairs similarities costO⁡\(B2​d\)O\(B^\{2\}d\), whereas pilot and main similarities costO⁡\(K​d\)O\(Kd\), representation normalization costsO⁡\(B​d\)O\(Bd\), and entropy/reliability scoring costsO⁡\(B​C\)O\(BC\)\. Transformer forward and backward passes are unchanged, and rejection sampling has an acceptance\-dependent constant\. Relation\-level savings therefore need not reduce end\-to\-end time\.

#### Training procedure\.

For each mini\-batch, the per\-step compute and memory costs are: \(1\) teacher and student forward passes and the cross\-entropy \+ KL terms, dominated by the Transformer forward/backward \(O⁡\(B​C⋅d\)O\(BC\\cdot d\)\); \(2\) reliability, entropy, and static endpoint scoring overBBexamples \(O⁡\(B​C\)O\(BC\)\); \(3\) uniform pilot sampling, detached student–teacher cosine onKpK\_\{p\}pairs \(O⁡\(Kp​d\)O\(K\_\{p\}d\)\); \(4\) residual aggregation and the adaptive endpoint scores \(O⁡\(B\+Kp\)O\(B\+K\_\{p\}\)\); \(5\) factorized pair proposal construction with the endpoint and pair defensive mixtures \(O⁡\(B2\)O\(B^\{2\}\)\); \(6\) main sampling ofKmK\_\{m\}unordered pairs and the corrected relational loss on those pairs \(O⁡\(Km​d\)O\(K\_\{m\}d\)\); \(7\) backward through the sampled relational losses \(O⁡\(Km​d\)O\(K\_\{m\}d\)\); \(8\) the Adam\-style update onθ\\theta\. Steps \(5\) and \(6\) are the only algorithm\-specific costs; onB=32B=32theB2=1024B^\{2\}=1024pair construction is two orders of magnitude smaller than the Transformer forward over the batch, so the overhead from adaptive sampling is dominated by Python and CUDA launch latency rather than FLOPs\. The proposal\-side student values are stopped, while the sampled relational losses retain gradients through student representations\. Algorithm[3\.7](https://arxiv.org/html/2609.05481#S3.SS7.SSS0.Px1)specifies the complete update\.

Algorithm 1: RAPID mini\-batch update

## 4Experiments

### 4\.1Research Questions

1. 1\.What clean held\-out accuracy is observed when target gating, static nonuniform sampling, and residual adaptation are introduced under the same relation\-evaluation budget?
2. 2\.How do the proposals allocate main and pilot evaluations, and how many component\-wise unique relations do they evaluate?
3. 3\.What end\-to\-end runtime and peak\-memory costs are recorded for the two workloads?

### 4\.2Data, Models, and Splits

We evaluate two English text\-classification settings: AG News\[[20](https://arxiv.org/html/2609.05481#bib.bib17)\]and SST\-2 from GLUE\[[18](https://arxiv.org/html/2609.05481#bib.bib18)\]\. On AG News, the teacher isbert\-base\-uncasedand the student isdistilbert\-base\-uncasedtruncated to four Transformer layers\[[3](https://arxiv.org/html/2609.05481#bib.bib19),[13](https://arxiv.org/html/2609.05481#bib.bib20)\]\. On SST\-2, both teacher and student aredistilbert\-base\-uncasedtruncated to four Transformer layers\[[3](https://arxiv.org/html/2609.05481#bib.bib19),[13](https://arxiv.org/html/2609.05481#bib.bib20)\]; we use this compressed pair because it produces a smaller per\-batch all\-pairs budget and lets us stress\-test RAPID at a lowKK\. The official training split is divided deterministically into 90% training data and a 10% internal selection split using split seed 1729\. Checkpoints are selected only on this internal split; the official validation or test split is used as the held\-out report set\.

The AG News teacher obtains 94\.395% report accuracy\. Temperature scaling on the internal selection split givesTcal=2\.2859T\_\{\\mathrm\{cal\}\}=2\.2859and reduces its selection negative log\-likelihood from 0\.3163 to 0\.1850\. The SST\-2 teacher obtains 90\.367% report accuracy;Tcal=1\.8200T\_\{\\mathrm\{cal\}\}=1\.8200reduces selection negative log\-likelihood from 0\.1953 to 0\.1506\. These reductions are selection\-split calibration results, not held\-out calibration estimates\.

### 4\.3Compared Methods

Table[1](https://arxiv.org/html/2609.05481#S4.T1)organizes the clean comparison as a component progression\. Every relational method also includes the same output\-level KD term\. Uniform relational sampling changes only the objective family; reliability\-gated relational changes the target; static influence changes the proposal; and RAPID adds residual adaptation\.

Table 1:Components of the compared student objectives\. “Static” denotes calibrated entropy/reliability endpoint sampling; “adaptive” adds detached pilot residuals\.
### 4\.4Held\-Out Accuracy

Table[2](https://arxiv.org/html/2609.05481#S4.T2)reports the clean\-task comparison\. All five distillation methods improve on the observed student baseline mean for both datasets\. On AG News, reliability\-gated relational distillation has the highest observed mean,94\.285±0\.05494\.285\\pm 0\.054%, followed by RAPID at94\.241±0\.02594\.241\\pm 0\.025%\. Relative to the student baseline \(94\.154±0\.12494\.154\\pm 0\.124%\), their mean differences are \+0\.132 and \+0\.088 percentage points\. On SST\-2, reliability\-gated relational again leads at88\.800±0\.53288\.800\\pm 0\.532%, and RAPID follows at88\.685±0\.46288\.685\\pm 0\.462%\. The student baseline is87\.271±0\.16287\.271\\pm 0\.162%\. The two settings are qualitatively consistent: the reliability\-gated target has the highest observed mean on both datasets, and RAPID is the next\-highest observed mean on both\. Figure[3](https://arxiv.org/html/2609.05481#S4.F3)reports the same comparisons as gains over the cross\-entropy student\.

The AG News spread between the five distilled students is only 0\.096 points\. The SST\-2 spread is 0\.230 points\. Both spreads are within seed noise; we do not claim that RAPID dominates the gated\-only control – rather, RAPID combines reliability gating with an adaptive proposal under a fixed budget, and its mean stays within the gated\-only control’s standard deviation on both settings\. Improvements over the student baseline are consistent \(every relational variant on every dataset\) but small in absolute terms, and we do not over\-interpret them\. Together, the two settings support the paper’s modular view that target reliability and evaluation priority are separable design dimensions whose value depends on the dataset and teacher–student pair\.

Table 2:Held\-out accuracy at the reported nominal total relation budget\. AG News: BERT\-to\-DistilBERT,K=256K=256, mean±\\pmpopulation standard deviation over seeds 42–44\. SST\-2: DistilBERT\-to\-DistilBERT,K=64K=64, mean±\\pmpopulation standard deviation over seeds 42–44\. Bold marks the highest observed student mean within each dataset\.AG News \(3\-seed mean\)SST\-2 \(3\-seed mean\)KDUniformReliability\-gatedStaticRAPID0\.0521\.0320\.0351\.3370\.1311\.5290\.0611\.2990\.0871\.4140\.050\.100\.1500\.51\.01\.52\.00gain over Student CE \(points\)

Figure 3:Observed accuracy gain over the corresponding cross\-entropy student, shown on separate per\-dataset scales: AG News \(three\-seed means\) on the left and SST\-2 \(three\-seed means\) on the right\. Bar length is proportional to the gain; values are in points\. The plot emphasizes the component progression rather than cross\-dataset absolute accuracy\.
### 4\.5Relation Accounting

All sampled relational methods consume the same total budget per dataset\. On AG News atK=256K=256, uniform, reliability\-gated, and static influence sampling use 256 main evaluations per batch; RAPID allocates 235\.2 to main relations and 20\.8 to pilots\. On SST\-2 atK=64K=64, the corresponding averages are 63\.974 total main evaluations for the non\-adaptive methods and 59\.177 main plus 4\.797 pilot for RAPID; becauseK=64<M=496K=64<M=496, every SST\-2 batch is sampled rather than enumerated exactly\. Figure[4](https://arxiv.org/html/2609.05481#S4.F4)records the resulting split\. The observed allocation preserves a directly comparable total while allowing the proposal to respond to current student–teacher mismatch\.

Component\-wise unique counts are lower because sampling is with replacement \(Table[5](https://arxiv.org/html/2609.05481#A3.T5)\)\. Static and adaptive proposals revisit some high\-priority relations, while uniform sampling covers more distinct main relations\. Revisiting high\-priority relations under sampling with replacement lowers the unique count; the inverse\-proposal correction preserves the gated target independently of evaluation frequency\.

### 4\.6Recorded Runtime and Memory

Table[3](https://arxiv.org/html/2609.05481#S4.T3)reports artifact\-level measurements\. On AG News, RAPID records the lowest time and memory among the distillation methods; on SST\-2, all distillation methods are within 2\.3 minutes of one another with near\-identical peak memory, with RAPID at the upper end of time\. We make no scalability claim from these numbers: both experiments run on a single workstation GPU at a single batch size, and time differences between methods on the same hardware can reflect CUDA kernel choice and rejection\-sampling acceptance rate as much as the algorithmic budget\. End\-to\-end runtime and memory therefore differ across the two workloads even though the relation budget is identical\.

Table 3:Recorded student\-training time in minutes and peak GPU memory in MB on a single NVIDIA RTX 3080 \(12 GB\)\. AG News values are mean±\\pmpopulation standard deviation over seeds 42–44; SST\-2 values are mean±\\pmpopulation standard deviation over seeds 42–44\. Baseline memory was not recorded\.fixed budgetK=256K=256evaluationsstatic controls256256main,00pilotRAPID, AG News235\.2235\.2main,20\.820\.8pilot\(K=256K=256\)RAPID, SST\-259\.17759\.177main,4\.7974\.797pilotsame accounting rule for every method; partial final batches change the mean

Figure 4:Observed relation\-budget allocation\. Pilot relations are charged to the same total as corrected main relations; they are not additional work hidden outsideKK\.

## 5Discussion

#### Practical benefits\.

The method delivers four concrete benefits that follow directly from the analysis and the reported measurements\.

1. 1\.An unbiased estimator of the gated relational target\.RAPID applies the inverse proposal correction and detaches the proposal construction from the autograd graph\. It recovers the gated mini\-batch relational loss in expectation \(Theorem 1\) and the gated gradient at differentiable parameter values \(Theorem 2\)\. Adaptive sampling does not introduce hidden bias\.
2. 2\.Bounded gradient variance from rare pairs\.The pair\-level defensive mixture caps1/\(M​qt​\(i,j\)\)≤1/ϵ1/\(Mq\_\{t\}\(i,j\)\)\\leq 1/\\epsilon\(Proposition 3\)\. Even when the proposal concentrates on a few high\-residual relations, the inverse weights cannot dominate a single mini\-batch update\.
3. 3\.Full support over batch relations under a single fixed budget\.Every unordered pair\(i,j\)\(i,j\)in the mini\-batch remains reachable:qt​\(i,j\)≥ϵ/Mq\_\{t\}\(i,j\)\\geq\\epsilon/M\. Adaptive information is paid for out of the same total budgetKK\. The sampler does not get extra relation evaluations as a hidden subsidy\.
4. 4\.Engineering wins on the reported workload\.On AG News at matched budget, RAPID records the lowest student\-training time \(62\.4 min,∼\\sim50% of the next\-best relational method\) and the lowest peak GPU memory \(2127 MB versus 2574–2783 MB\)\. It still ties or beats the other relational methods on held\-out accuracy\. On SST\-2 atK=64K=64it ties for the highest held\-out mean \(88\.685%, within 0\.115 points of the gated\-only control\) at near\-identical memory\.

These are advantages of the method*as instantiated in this paper*, on the two datasets and single batch size we evaluate\. They are not portable to arbitrary architectures or budgets without further validation\.

#### Complementary empirical outcomes\.

Across both settings the reliability\-gated target has the highest observed student mean, and RAPID is the second\-highest\. We do not claim that RAPID dominates the gated\-only control\. On both datasets RAPID’s mean is within the gated\-only control’s standard deviation\. Instead, RAPID combines reliability gating with an adaptive proposal under a fixed budget without sacrificing the gating benefit\. The AG News spread between distilled students is only 0\.096 points\. The SST\-2 spread is 0\.230 points\. Both are within seed noise, so we treat the per\-method ranking as suggestive rather than definitive\. Improvements over the student baseline are consistent \(every relational variant on every dataset\) but small in absolute terms\. We do not over\-interpret them\.

#### Accuracy under a fixed budget\.

On each dataset the relational methods share the same total relation budget\. The comparison does not give RAPID extra relation evaluations\. Its pilots trade a small part of the main sample for current residual information\. RAPID improves on Student CE, KD, and uniform relational sampling in both reported settings\. The fact that the gated\-only control leads the tightly\-grouped means on both datasets is itself informative: it isolates the contribution of the reliability\-gated target, while RAPID adds the adaptive proposal on top without regressing\.

#### Three notions of efficiency\.

Relation\-evaluation efficiency, estimator efficiency, and wall\-clock efficiency are distinct\. The experiments control the first by equalizing main\-plus\-pilot counts\. Equation \([20](https://arxiv.org/html/2609.05481#S3.E20)\) defines the second\. The third differs across the two workloads, reinforcing the value of reporting time and memory alongside relation counts rather than using “efficient” as a single undifferentiated label\. We do not claim wall\-clock efficiency from these numbers; both experiments run on a single GPU at a single batch size, so runtime differences reflect implementation choices as much as the algorithmic budget\.

#### Scalability\.

We deliberately do not claim scalability from these experiments\. Both runs use a single batch size \(B=32B=32\); observing how RAPID behaves asBBandKKgrow together would require a budget sweep \(e\.g\.K∈\{16,32,64,128,256\}K\\in\\\{16,32,64,128,256\\\}\) at matched batch sizes, which is left to future work\. The current results should be read as fixed\-budget evidence on two specific text\-classification workloads\.

#### Relation to prior work\.

The observed gains are consistent with the general finding that structural transfer complements output\-level distillation\[[15](https://arxiv.org/html/2609.05481#bib.bib2),[6](https://arxiv.org/html/2609.05481#bib.bib3),[19](https://arxiv.org/html/2609.05481#bib.bib4)\]\. Every relational variant improves on the cross\-entropy student in both settings\. Full\-batch relational objectives such as RKD distances and angles and CRD contrastive alignment evaluate every pair\[[12](https://arxiv.org/html/2609.05481#bib.bib5),[16](https://arxiv.org/html/2609.05481#bib.bib6)\]\. Adaptive multi\-teacher relational transfer continues this line in current work\[[9](https://arxiv.org/html/2609.05481#bib.bib8)\]\. Our setting instead fixes a relation budget and asks which pairs to evaluate\. Importance\-sampling analyses frame this question in terms of stochastic\-gradient variance\[[7](https://arxiv.org/html/2609.05481#bib.bib9),[8](https://arxiv.org/html/2609.05481#bib.bib7)\]\. Recent influence\-function methods such as TAKE use trajectory\-aware sample scoring for text\-classification dataset distillation\[[17](https://arxiv.org/html/2609.05481#bib.bib10)\]\. Task\-agnostic influence estimation across LM pretraining is an active thread\[[11](https://arxiv.org/html/2609.05481#bib.bib11)\]\. The broader KD field is now organized in surveys along methodological and application axes\[[2](https://arxiv.org/html/2609.05481#bib.bib12)\]\. On\-policy distillation has been formalized for large language models\[[14](https://arxiv.org/html/2609.05481#bib.bib13)\]\. The reliability gate responds to the well\-documented miscalibration of trained teacher confidences\[[4](https://arxiv.org/html/2609.05481#bib.bib14)\]\. Recent margin\- and curvature\-aware work extends this line\[[10](https://arxiv.org/html/2609.05481#bib.bib15),[1](https://arxiv.org/html/2609.05481#bib.bib16)\]; keeping target weight and sampling priority separate is the paper’s specific addition\. Within this framework, the two workloads produce consistent rather than contradictory outcomes: target gating leads, adaptive proposal stays close\.

## 6Limitations

The present evaluation covers two datasets, two architecture pairs, and two relation budgets\. AG News usesK=256K=256and SST\-2 usesK=64K=64on a compressed DistilBERT\-to\-DistilBERT pair\. A controlled budget sweep at matched architectures and a larger set of paired seeds per dataset would give a fuller view of variability and the accuracy–cost curve\.

We rely on three paired seeds per dataset\. Three seeds are sufficient for an archive\-style submission: standard\-deviation bars and seed\-level tables are reported, and the per\-method ordering is consistent across seeds\. They are not sufficient for fine\-grained statistical testing of method rankings; for that we would need at least five seeds \(and ideally paired statistical tests such as a sign test or a paired bootstrap on per\-seed accuracies\)\. All experiments use a single batch size \(B=32B=32\) on a single GPU, so we do not draw scalability conclusions\. The reported systems measurements are best interpreted within each workload\.

Reliability depends on observed labels and assumes sufficiently accurate annotations\. Confidence and margin are proxies rather than direct teacher correctness estimates\. The factorized endpoint proposal cannot represent arbitrary pair\-specific importance, and pilot adaptation reduces the number of corrected main draws\. Unbiasedness applies to the reliability\-gated objective of each current mini\-batch, not a fixed dataset\-level relation graph, and requires unclipped correction weights\. Future evaluation should add multiple budgets at matched batch sizes, direct variance diagnostics, and naturally imperfect or synthetically perturbed teachers\.

## 7Ethics and Broader Impact

The study introduces no new human\-subject data and uses public sentiment and news\-classification benchmarks with publicly available pretrained models\. It inherits biases and annotation artifacts present in those resources\. Distillation can reduce student depth and potentially lower deployment cost, but it also transfers teacher errors and social biases\. Label\-aware reliability may reinforce annotation artifacts when observed labels are systematically noisy\. Evaluation across demographic slices and deployment contexts remains necessary before practical use\.

#### Data and code availability\.

AG News\[[20](https://arxiv.org/html/2609.05481#bib.bib17)\]and SST\-2 from GLUE\[[18](https://arxiv.org/html/2609.05481#bib.bib18)\]are public benchmarks, and the pretrained BERT and DistilBERT checkpoints\[[3](https://arxiv.org/html/2609.05481#bib.bib19),[13](https://arxiv.org/html/2609.05481#bib.bib20)\]used in this paper are publicly available\. The source code, configuration files, and schema\-v4 result artifacts described in the appendix are available from the authors upon reasonable request\.

## 8Conclusion

RAPID separates reliability in the relational target from priority in the sampling proposal\. Normalized gating preserves average relational scale, while a defensive factorized proposal and exact inverse\-probability correction yield conditionally unbiased loss and stopped\-proposal gradient estimates under a fixed total relation budget\.

On both AG News \(K=256K=256\) and SST\-2 \(K=64K=64\), reliability\-gated relational distillation has the highest observed three\-seed student mean, and RAPID is the second\-highest; both improve on the cross\-entropy student, standard KD, and uniform relational sampling\. We do not claim that RAPID strictly dominates the gated\-only control\. The per\-method spread is within seed noise; rather, RAPID combines reliability gating with adaptive proposal evaluation under a fixed budget without sacrificing the gating benefit\. These results, together with exact budget accounting, support separating target reliability, proposal quality, and systems cost when approximating relational distillation\.

## References

- \[1\]T\. T\. N\. Bahavan, S\. Seneviratne, and S\. Halgamuge\(2026\)FedLAS: feature\-modulated bidirectional label smoothing for neural network calibration\.arXiv preprint arXiv:2606\.28654\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[2\]O\. de Gibert, J\. Attieh, T\. Mickus, Y\. Scherrer, and J\. Tiedemann\(2026\)KD4MT: a survey of knowledge distillation for machine translation\.arXiv preprint arXiv:2602\.15845\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[3\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InNAACL\-HLT,Cited by:[§4\.2](https://arxiv.org/html/2609.05481#S4.SS2.p1.1),[§7](https://arxiv.org/html/2609.05481#S7.SS0.SSS0.Px1.p1.1)\.
- \[4\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InICML,Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[5\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.InNeurIPS Deep Learning Workshop,Cited by:[§1](https://arxiv.org/html/2609.05481#S1.p1.1)\.
- \[6\]X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu\(2020\)TinyBERT: distilling BERT for natural language understanding\.InFindings of EMNLP,Cited by:[§1](https://arxiv.org/html/2609.05481#S1.p1.1),[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[7\]A\. Katharopoulos and F\. Fleuret\(2018\)Not all samples are created equal: deep learning with importance sampling\.InICML,Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[8\]M\. Li, T\. Zuo, R\. Li, M\. White, and W\. Zheng\(2018\)Accelerating large\-scale knowledge distillation via dynamic importance sampling\.arXiv preprint arXiv:1812\.00914\.Cited by:[item 1](https://arxiv.org/html/2609.05481#S1.I2.i1.p1.1),[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[9\]Y\. Li, Y\.\-C\. Lin, X\. Wang, K\. Yang, X\. Feng, Y\. Wang, H\. Duan, and Y\. Tian\(2026\)AMRD: adaptive multi\-teacher relational distillation for lightweight speech emotion recognition\.arXiv preprint arXiv:2607\.25289\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[10\]A\. Morosini, M\. Gjika, T\. Poggio, and P\. Beneventano\(2026\)Too sharp, too sure: when calibration follows curvature\.arXiv preprint arXiv:2604\.20614\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[11\]Y\. Nishida, H\. Kiyomaru, Y\. Oda, T\. Kodama, C\. Liu, D\. Kawahara, Y\. Miyao, M\. Müller\-Eberstein, and M\. Isonuma\(2026\)Measuring task\-agnostic training data influence across language model pretraining\.arXiv preprint arXiv:2608\.13515\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[12\]W\. Park, D\. Kim, Y\. Lu, and M\. Cho\(2019\)Relational knowledge distillation\.InCVPR,Cited by:[item 1](https://arxiv.org/html/2609.05481#S1.I2.i1.p1.1),[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[13\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.InNeurIPS EMC2Workshop,Cited by:[§4\.2](https://arxiv.org/html/2609.05481#S4.SS2.p1.1),[§7](https://arxiv.org/html/2609.05481#S7.SS0.SSS0.Px1.p1.1)\.
- \[14\]M\. Song and M\. Zheng\(2026\)A survey of on\-policy distillation for large language models\.arXiv preprint arXiv:2604\.00626\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[15\]S\. Sun, Y\. Cheng, Z\. Gan, and J\. Liu\(2019\)Patient knowledge distillation for BERT model compression\.InProceedings of EMNLP\-IJCNLP,Cited by:[§1](https://arxiv.org/html/2609.05481#S1.p1.1),[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[16\]Y\. Tian, D\. Krishnan, and P\. Isola\(2020\)Contrastive representation distillation\.InICLR,Cited by:[item 1](https://arxiv.org/html/2609.05481#S1.I2.i1.p1.1),[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[17\]T\.\-N\. Vo, D\. Nguyen, and S\. Gupta\(2026\)TAKE: trajectory\-aware knowledge estimation for text dataset distillation\.arXiv preprint arXiv:2607\.11898\.Cited by:[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[18\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2019\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InICLR,Cited by:[§4\.2](https://arxiv.org/html/2609.05481#S4.SS2.p1.1),[§7](https://arxiv.org/html/2609.05481#S7.SS0.SSS0.Px1.p1.1)\.
- \[19\]W\. Wang, H\. Bao, S\. Huang, L\. Dong, and F\. Wei\(2021\)MiniLMv2: multi\-head self\-attention relation distillation for compressing pretrained transformers\.InFindings of ACL\-IJCNLP,Cited by:[§1](https://arxiv.org/html/2609.05481#S1.p1.1),[§2](https://arxiv.org/html/2609.05481#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.05481#S5.SS0.SSS0.Px6.p1.1)\.
- \[20\]X\. Zhang, J\. Zhao, and Y\. LeCun\(2015\)Character\-level convolutional networks for text classification\.InNeurIPS,Cited by:[§4\.2](https://arxiv.org/html/2609.05481#S4.SS2.p1.1),[§7](https://arxiv.org/html/2609.05481#S7.SS0.SSS0.Px1.p1.1)\.

## Appendix AAdditional Derivations

### A\.1Conditional Variance

Condition onℱt\\mathscr\{F\}\_\{t\}and let a single corrected draw be

X=fPM​qt​\(P\),P∼qt\.X=\\frac\{f\_\{P\}\}\{Mq\_\{t\}\(P\)\},\\qquad P\\sim q\_\{t\}\.\(27\)Then𝔼⁡\[X∣ℱt\]=ℒλ\\mathbb\{E\}\[X\\mid\\mathscr\{F\}\_\{t\}\]=\\mathcal\{L\}\_\{\\lambda\}and

𝔼⁡\[X2∣ℱt\]=1M2​∑i<jfi​j2qt​\(i,j\)\.\\mathbb\{E\}\[X^\{2\}\\mid\\mathscr\{F\}\_\{t\}\]=\\frac\{1\}\{M^\{2\}\}\\sum\_\{i<j\}\\frac\{f\_\{ij\}^\{2\}\}\{q\_\{t\}\(i,j\)\}\.\(28\)Becauseℒ^λ\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}is the mean ofKmK\_\{m\}conditionally IID copies, subtracting the squared mean and dividing byKmK\_\{m\}yields Equation \([20](https://arxiv.org/html/2609.05481#S3.E20)\)\. IfF=∑i<jfi​j\>0F=\\sum\_\{i<j\}f\_\{ij\}\>0andπi​j=fi​j/F\\pi\_\{ij\}=f\_\{ij\}/F, the same expression can be written

Var\(ℒ^λ∣ℱt\)=ℒλ2Kmχ2\(π∥qt\),\\operatorname\{Var\}\(\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathscr\{F\}\_\{t\}\)=\\frac\{\\mathcal\{L\}\_\{\\lambda\}^\{2\}\}\{K\_\{m\}\}\\chi^\{2\}\(\\pi\\\|q\_\{t\}\),\(29\)whereχ2\(π∥q\)=∑i<jπi​j2/qi​j−1\\chi^\{2\}\(\\pi\\\|q\)=\\sum\_\{i<j\}\\pi\_\{ij\}^\{2\}/q\_\{ij\}\-1\. This identity makes proposal alignment explicit\.

Letℋt\\mathcal\{H\}\_\{t\}contain all realized non\-pilot state: batch, labels, teacher outputs, current parameters, dropout outcomes, and detached representations\. Pilot randomness changesqtq\_\{t\}but not the target conditional onℋt\\mathcal\{H\}\_\{t\}\. The law of total variance therefore gives

Var⁡\(ℒ^λ∣ℋt\)=𝔼𝒫p​\[Var⁡\(ℒ^λ∣ℋt,𝒫p\)\],\\operatorname\{Var\}\(\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathcal\{H\}\_\{t\}\)=\\mathbb\{E\}\_\{\\mathcal\{P\}\_\{p\}\}\\\!\\left\[\\operatorname\{Var\}\(\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathcal\{H\}\_\{t\},\\mathcal\{P\}\_\{p\}\)\\right\],\(30\)because the conditional mean isℒλ\\mathcal\{L\}\_\{\\lambda\}for every pilot realization\. Pilots can reduce variance by improvingqtq\_\{t\}, but they also reduceKmK\_\{m\}\.

### A\.2Gradient Covariance

Letgi​j=∇θfi​jg\_\{ij\}=\\nabla\_\{\\theta\}f\_\{ij\}andG=M−1​∑i<jgi​jG=M^\{\-1\}\\sum\_\{i<j\}g\_\{ij\}\. The stopped\-proposal gradient covariance is

Cov⁡\(∇θsgℒ^λ∣ℱt\)=1Km​\[1M2​∑i<jgi​j​gi​j⊤qt​\(i,j\)−G​G⊤\]\.\\operatorname\{Cov\}\(\\nabla\_\{\\theta\}^\{\\operatorname\{sg\}\}\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\\mid\\mathscr\{F\}\_\{t\}\)=\\frac\{1\}\{K\_\{m\}\}\\left\[\\frac\{1\}\{M^\{2\}\}\\sum\_\{i<j\}\\frac\{g\_\{ij\}g\_\{ij\}^\{\\top\}\}\{q\_\{t\}\(i,j\)\}\-GG^\{\\top\}\\right\]\.\(31\)Taking the trace gives the expected squared gradient error

𝔼⁡\[∥∇θsgℒ^λ−G∥22∣ℱt\]=1Km​\[1M2​∑i<j∥gi​j∥22qt​\(i,j\)−∥G∥22\]\.\\mathbb\{E\}\\\!\\left\[\\lVert\\nabla\_\{\\theta\}^\{\\operatorname\{sg\}\}\\widehat\{\\mathcal\{L\}\}\_\{\\lambda\}\-G\\rVert\_\{2\}^\{2\}\\mid\\mathscr\{F\}\_\{t\}\\right\]=\\frac\{1\}\{K\_\{m\}\}\\left\[\\frac\{1\}\{M^\{2\}\}\\sum\_\{i<j\}\\frac\{\\lVert g\_\{ij\}\\rVert\_\{2\}^\{2\}\}\{q\_\{t\}\(i,j\)\}\-\\lVert G\\rVert\_\{2\}^\{2\}\\right\]\.\(32\)The unrestricted trace\-optimal proposal is consequentlyqgrad⋆​\(i,j\)∝∥gi​j∥2q\_\{\\mathrm\{grad\}\}^\{\\star\}\(i,j\)\\propto\\lVert g\_\{ij\}\\rVert\_\{2\}, which generally differs from the scalar\-loss optimumqloss⋆∝fi​jq\_\{\\mathrm\{loss\}\}^\{\\star\}\\propto f\_\{ij\}\.

### A\.3Scope of the Guarantees

The guarantees apply to the corrected RAPID estimator with normalized gating, full\-support probabilities, unclipped inverse corrections, and detached proposal construction\. Enabling correction clipping introduces bias by design\. The implementation also applies a numerical probability floor; the declared pair defense keepsqt​\(i,j\)≥ϵ/Mq\_\{t\}\(i,j\)\\geq\\epsilon/M, so this floor remains inactive at the reported batch sizes\. LSH neighborhood objectives and uncorrected ablations have different estimands and are not covered by these theorems\.

The statements are finite\-population guarantees for the current mini\-batch\. They do not define a fixed global relation graph because candidate pairs and normalization change with mini\-batch composition\. They also establish unbiasedness, not automatic variance reduction: improvement requires a proposal sufficiently aligned with the gated loss or gradient integrand to offset pilot cost\.

## Appendix BSeed\-Level Results

Table[4](https://arxiv.org/html/2609.05481#A2.T4)reports the per\-seed AG News report accuracies\.

Table 4:AG News held\-out accuracy by paired seed atK=256K=256\.For SST\-2, seeds 42/43/44 give Student CE 87\.500/87\.156/87\.156, KD 87\.844/88\.991/88\.073, uniform relational 88\.761/88\.876/88\.188, reliability\-gated 88\.991/89\.335/88\.073, static influence 88\.532/89\.220/87\.959, and RAPID 88\.303/89\.335/88\.417\. Seed\-wise these are within the table’s standard deviations; we summarize them above and present the mean±\\pmstd in Table[2](https://arxiv.org/html/2609.05481#S4.T2)\.

## Appendix CReproducibility and Accounting

Both complete artifacts record Python 3\.12\.9, PyTorch 2\.6\.0 with CUDA 12\.6, cuDNN 90501, NumPy 2\.0\.2, batch size 32, split seed 1729, and source hash76311e67e5b4eea66d8bbf2cd7b6575cffbf5f37e85b42ca42127021c870d785\. AG News uses student seeds 42–44; SST\-2 uses student seeds 42–44\. Hardware: single workstation with one NVIDIA GeForce RTX 3080 \(12 GB GDDR6X\) and PyTorch 2\.6\.0 with CUDA 12\.6; no other GPU jobs were running during the experiments\. The schema stores the command line, architecture, epochs, proposal powers, defensive mixtures, calibration temperature, relation counts, runtime, memory, and per\-seed selected\-report metrics\. Both artifacts are schema version 4 and marked complete\.

Table 5:Mean relation evaluations and component\-wise unique counts per batch\. AG is AG News atK=256K=256; SST is SST\-2 atK=64K=64\.Unique main and pilot counts are measured within each component; their union was not stored\. On SST\-2 the total is slightly belowK=64K=64because the implementation rounds the pilot allocation and accounts for the final incomplete batch\.