TAD：仅从教师轨迹中蒸馏视觉-语言-动作策略

# Distilling Vision-Language-Action Policiesfrom Teacher Trajectories Alone
Source: [https://arxiv.org/html/2610.33650](https://arxiv.org/html/2610.33650)
Da Wang††thanks:Corresponding author\.Affiliation:School of Computer and Information Technology, Shanxi University, Taiyuan, China

###### Abstract

Online distillation of vision\-language\-action \(VLA\) policies commonly relies on teacher action scores that an inference\-only interface may not expose\. Trajectory\-only Adversarial Distillation \(TAD\) instead turns a fixed bank of teacher joint\-position features into adaptive feedback: a discriminator compares feature marginals from teacher and student rollouts and supplies rewards for on\-policy learning\. The student interacts with the environment without online teacher queries or environment task rewards\. On ManiSkillPutOnPlateInScene25Main, TAD improves OpenVLA\-OFT \(7B\) from26\.2%26\.2\\%to76\.6%76\.6\\%success\-once \(\+50\.4\+50\.4percentage points\) and from20\.3%20\.3\\%to58\.6%58\.6\\%success\-at\-end\. It trails white\-box on\-policy distillation \(OPD\) at81\.6%81\.6\\%/59\.4%59\.4\\%by5\.05\.0/0\.80\.8percentage points \(n=256n\{=\}256evaluation trajectories per checkpoint\)\. The tested four\-epoch continuation achieves the highest TAD result without additional teacher data\. This best run uses5×5\\timesOPD’s student interaction budget and12\.5×12\.5\\timesits complete\-rollout batch\-passes\. The comparison demonstrates online VLA improvement without teacher scoring while quantifying unequal interaction and optimization budgets\.

## 1Introduction

Vision\-language\-action \(VLA\) models translate visual observations and language instructions into robot actions, allowing pretrained representations to support manipulation across tasks\. OpenVLA and OpenVLA\-OFT illustrate the potential of this approach and of supervised fine\-tuning \(SFT\) for adapting a large policy to robot demonstrations\[[11](https://arxiv.org/html/2610.33650#bib.bib11),[12](https://arxiv.org/html/2610.33650#bib.bib12)\]\. Demonstration\-based training nevertheless exposes the policy primarily to states visited by the data\-collection policy\. At deployment, small action errors can change subsequent observations and move execution away from that distribution\. Improving behavior on the student’s own visited states is therefore an important complement to fitting demonstration actions\.

On\-policy distillation \(OPD\) addresses this mismatch by obtaining teacher supervision on student\-generated behavior\[[9](https://arxiv.org/html/2610.33650#bib.bib9),[13](https://arxiv.org/html/2610.33650#bib.bib13)\]\. In the score\-based setting considered here, a fixed teacher evaluates the likelihood of action components selected by the student, supplying a direction for online optimization\. This access is stronger than the ability to request a demonstration or execute a teacher policy\. An inference\-only API or proprietary robot stack may return actions without exposing their likelihoods, model weights, or internal representations\. Consequently, a capable teacher may be available for collecting successful behavior but unavailable for scoring arbitrary student actions during training\.

We separate two resources that are often coupled in online distillation: student environment interaction and continuing teacher access\. Our setting permits the student to generate new rollouts throughout optimization, but permits teacher supervision only through an initially collected trajectory bank\. The bank remains fixed even as the student changes\. This restriction is not offline learning: the student still encounters the consequences of its own actions and can improve through closed\-loop interaction\. Nor does it provide corrective teacher labels at the student’s newly visited states\.

*Trajectory\-only Adversarial Distillation*\(TAD\) implements this setting with a state\-feature discriminator\. Teacher rollouts provide a fixed bank of normalized eight\-dimensional joint\-position features; recent student rollouts provide the comparison distribution\. The discriminator produces dense logit rewards for group\-relative policy optimization in RLinf\[[25](https://arxiv.org/html/2610.33650#bib.bib25),[23](https://arxiv.org/html/2610.33650#bib.bib23)\]\. Its feedback adapts to the student although the teacher data do not change\. The VLA policy itself receives visual\-language input, while proprioception supplies the discriminator features\. The method matches feature marginals rather than complete temporal trajectories, and the main and best runs use no environment task reward\.

A central empirical diagnostic is the distinction between discriminating behavior and improving it\. Early configurations separate teacher and student features accurately without raising deterministic task success\. Subsequent diagnostics examine discriminator inputs, reward construction, and actor optimization; continuations from a shared checkpoint examine regularization, update epochs, and reward mixing\. The tested four\-epoch continuation achieves the highest TAD evaluation with the unchanged teacher bank\. These observations motivate examining optimization alongside reward design, rather than interpreting either discriminator accuracy or a sampled training plateau as a direct measure of the supervision’s usefulness\.

On OpenVLA\-OFT \(7B\) and ManiSkill, TAD raises success\-once from26\.2%26\.2\\%to76\.6%76\.6\\%and success\-at\-end from20\.3%20\.3\\%to58\.6%58\.6\\%, compared with OPD’s81\.6%81\.6\\%/59\.4%59\.4\\%\. The comparison shares student initialization, teacher, task, and actor learning rate, but not the training objective or budget\. The best TAD run uses5×5\\timesas many student rollout trajectories and12\.5×12\.5\\timesas many complete\-rollout optimization batch\-passes\. These results quantify the trade\-off between teacher access and student\-side training effort\.

The paper develops three contributions:

1. 1\.A trajectory\-only distillation method coupling a fixed teacher feature bank, adaptive state rewards, and online optimization of a large VLA\.
2. 2\.Training diagnostics and continuation ablations that distinguish classification quality, sampled rollout behavior, and deterministic policy evaluation\.
3. 3\.A quantified comparison with white\-box OPD, reporting task performance together with the unequal interaction and optimization budgets required by the tested runs\.

## 2Related Work

### 2\.1VLA learning and online distillation

OpenVLA\[[11](https://arxiv.org/html/2610.33650#bib.bib11)\]and OpenVLA\-OFT\[[12](https://arxiv.org/html/2610.33650#bib.bib12)\]provide the model family underlying our experiments\. Supervised adaptation transfers demonstration behavior into a visual\-language policy, but its training distribution need not cover the states induced by the deployed student\. DAgger\[[17](https://arxiv.org/html/2610.33650#bib.bib17)\]addresses this issue by gathering expert action labels on learner\-visited states and aggregating them into the training set\. Its corrective\-label access differs from ours: TAD collects teacher trajectories before student optimization and makes no subsequent queries at student states\. Student interaction alone does not imply access to an expert who can correct each rollout\.

Distillation methods also differ in the supervision they request\. Knowledge distillation\[[8](https://arxiv.org/html/2610.33650#bib.bib8)\]transfers teacher output distributions, whereas sequence\-level distillation \(SeqKD\)\[[10](https://arxiv.org/html/2610.33650#bib.bib10)\]trains on teacher\-generated sequences\. Generalized knowledge distillation \(GKD\)\[[9](https://arxiv.org/html/2610.33650#bib.bib9)\]evaluates teacher distributions on student\-generated samples, bringing supervision closer to the student’s current behavior\. VLA\-OPD\[[13](https://arxiv.org/html/2610.33650#bib.bib13)\]applies on\-policy teacher scoring to embodied policies and supplies our white\-box comparison\. We retain online student rollouts but replace likelihood\-based teacher feedback with a discriminator learned from a fixed feature bank\. This is a change in supervision access and actor objective, not simply a different estimator of the same OPD loss\.

### 2\.2Adversarial imitation and observation\-based supervision

GAIL\[[1](https://arxiv.org/html/2610.33650#bib.bib1)\]formulates imitation through adversarial discrimination between expert and learner behavior\. AIRL\[[2](https://arxiv.org/html/2610.33650#bib.bib2)\]develops an adversarial reward\-learning formulation, while DAC\[[3](https://arxiv.org/html/2610.33650#bib.bib3)\]studies sample efficiency and reward bias with off\-policy learning\. These works motivate learning a reward from demonstrations rather than specifying a task reward by hand\. TAD uses a logit transformation of discriminator probabilities, but not AIRL’s full reward\-recovery architecture\. Its discriminator replay should likewise not be confused with off\-policy actor learning: the actor updates from the current rollout batch with fixed rewards\.

Observation\-based imitation removes the need for demonstrated action labels, but does not necessarily remove temporal structure\. GAIfO\[[18](https://arxiv.org/html/2610.33650#bib.bib18)\]discriminates state transitions, and AMP\[[4](https://arxiv.org/html/2610.33650#bib.bib4)\]uses transition pairs to learn motion priors\. Our main experiments instead discriminate single\-frame joint\-position features, so they compare a feature marginal rather than a transition distribution\. This narrower representation cannot directly encode object relations or temporal progress\. EF\-GAIfO\[[19](https://arxiv.org/html/2610.33650#bib.bib19)\]further studies observation\-based imitation under embodiment mismatch, including real\-robot validation of policies trained in simulation\. Our experiments instead retain the same robot embodiment for teacher and student\. Design studies of adversarial imitation\[[5](https://arxiv.org/html/2610.33650#bib.bib5)\]motivate examining inputs, reward transformations, and optimization together; gradient\-penalty regularization\[[6](https://arxiv.org/html/2610.33650#bib.bib6)\]motivates one continuation ablation\. The distinction between a learned proxy and the intended task objective\[[7](https://arxiv.org/html/2610.33650#bib.bib7)\]is especially relevant when discriminator accuracy and task success diverge\.

### 2\.3Black\-box generative distillation

Black\-box language\-model distillation offers related ways to replace unavailable teacher likelihoods\. GAD\[[16](https://arxiv.org/html/2610.33650#bib.bib16)\]trains a text discriminator on teacher and student outputs, using student SeqKD warmup and separate discriminator warmup before adversarial online learning\. Our student instead starts from SFT on a different demonstration dataset, and our recipe has no separate discriminator warmup\. This difference matters when interpreting early discriminator separation: the initial student need not already approximate the teacher bank\. ROPD\[[20](https://arxiv.org/html/2610.33650#bib.bib20)\]derives prompt\-specific rubrics from teacher–student contrasts and uses those rubrics to score student outputs\. TAD does not induce semantic evaluation criteria; it learns a reward directly from observed robot features\.

Adversarial objectives also appear in generative\-model acceleration\. ADD\[[15](https://arxiv.org/html/2610.33650#bib.bib15)\]combines adversarial learning with teacher distillation to accelerate diffusion sampling; an adversarial loss alone therefore does not establish trajectory\-only or black\-box teacher access\. AFD\[[21](https://arxiv.org/html/2610.33650#bib.bib21)\]studies black\-box adversarial flow distillation for video generation, but its teacher\-access and update protocol should not be identified with our fixed\-bank embodied setting\. These methods share the use of learned comparisons while differing in available supervision and optimization\. Our analysis concerns the interaction between feature\-based rewards and online control, where student actions change the future observations used for learning\.

### 2\.4Concurrent embodied adversarial distillation

FA\-OPD\[[14](https://arxiv.org/html/2610.33650#bib.bib14)\]is closely related embodied work that learns a flow\-matching teacher from demonstrations and co\-trains it with lightweight MLP students\. Its distillation combines adversarial and mean\-squared\-error objectives, rather than relying on adversarial rewards alone\. In contrast, our teacher is pre\-existing and supplies a fixed bank before optimization of an OpenVLA\-OFT \(7B\) student\. No teacher model is co\-trained, and no teacher action scores or additional demonstrations are requested during student training\.

The distinction concerns teacher construction, student scale, supervision availability, and the actor loss\. Our evaluation focuses on transferring behavior from a fixed external teacher into a large VLA\. Score\-based OPD provides a reference for measuring the performance and student\-side training budgets associated with replacing online teacher scores by a fixed trajectory bank\.

## 3Method

### 3\.1Problem formulation

We consider an episodic manipulation processℳ=\(𝒪,𝒜,P,renv,ρ0,H\)\\mathcal\{M\}=\(\\mathcal\{O\},\\mathcal\{A\},P,r^\{\\mathrm\{env\}\},\\rho\_\{0\},H\)\. The environment observationoto\_\{t\}includes camera images, a task instruction, and proprioceptionsts\_\{t\}\. The studentπθ​\(at∣xt\)\\pi\_\{\\theta\}\(a\_\{t\}\\mid x\_\{t\}\)receives only the visual\-language inputxt=x⁡\(ot\)x\_\{t\}=x\(o\_\{t\}\), not joint positions, and produces an action chunkata\_\{t\}\. Herettindexes chunk decisions\. A trajectoryτ=\(o0,a0,…,oH\)\\tau=\(o\_\{0\},a\_\{0\},\\ldots,o\_\{H\}\)induces a marginal distribution of featuresϕ⁡\(ot\)\\phi\(o\_\{t\}\)at those decisions, whereϕ:𝒪→ℝ8\\phi:\\mathcal\{O\}\\to\\mathbb\{R\}^\{8\}extracts eight joint positions \(qpos8\) and normalizes them using statistics fitted once on the teacher bank\. The same fixed map is applied to teacher and student observations\. It excludes object state; we compare feature marginals, not complete temporal trajectories\.

The teacherπT\\pi\_\{T\}supplies trajectories𝒟T=\{τiT\}\\mathcal\{D\}\_\{T\}=\\\{\\tau^\{T\}\_\{i\}\\\}once, with no logits, densities, gradients, weights, or hidden states\. We seek to improve an SFT\-initialized student through its own environment interaction\. White\-box OPD instead evaluates teacher log\-likelihoods for the student’s action components throughout training\. The methods share student initialization, teacher, benchmark, and actor learning rate, but not identical objectives or training settings\.

### 3\.2Discriminator and reward

A discriminatorDψ​\(z\)=sigmoid⁡\(fψ​\(z\)\)D\_\{\\psi\}\(z\)=\\operatorname\{sigmoid\}\(f\_\{\\psi\}\(z\)\), withz=ϕ⁡\(o\)z=\\phi\(o\), learns to assign high probabilities to teacher features and low probabilities to student features \(Figure[1](https://arxiv.org/html/2610.33650#S3.F1)\)\. The baseline binary cross\-entropy objective is

ℒD​\(ψ\)=−𝔼z∼ρT​\[log⁡Dψ​\(z\)\]−𝔼z∼ρℬ​\[log⁡\(1−Dψ​\(z\)\)\],\\mathcal\{L\}\_\{D\}\(\\psi\)=\-\\mathbb\{E\}\_\{z\\sim\\rho\_\{T\}\}\\big\[\\log D\_\{\\psi\}\(z\)\\big\]\-\\mathbb\{E\}\_\{z\\sim\\rho\_\{\\mathcal\{B\}\}\}\\big\[\\log\\big\(1\-D\_\{\\psi\}\(z\)\\big\)\\big\],\(1\)whereρT\\rho\_\{T\}is the fixed teacher feature distribution andρℬ\\rho\_\{\\mathcal\{B\}\}is the empirical student feature distribution in replayℬ\\mathcal\{B\}\. Replay retains the latest 4 student batches to reduce overfitting to one batch\. Equation \([1](https://arxiv.org/html/2610.33650#S3.E1)\) shows hard\-label BCE; training uses label smoothing withη=0\.1\\eta=0\.1, replacing the teacher target by1−η1\-\\etaand the student target byη\\eta\.

We turn each score into a dense scalar AIRL\-style logit reward\[[2](https://arxiv.org/html/2610.33650#bib.bib2)\]:

rt=log⁡Dψ​\(ϕ⁡\(ot\)\)−log⁡\(1−Dψ​\(ϕ⁡\(ot\)\)\),r\_\{t\}\\;=\\;\\log D\_\{\\psi\}\\\!\\big\(\\phi\(o\_\{t\}\)\\big\)\\;\-\\;\\log\\big\(1\-D\_\{\\psi\}\\\!\\big\(\\phi\(o\_\{t\}\)\\big\)\\big\),\(2\)using one feature sample per action chunk\. The reward is positive whenDψ\>1/2D\_\{\\psi\}\>1/2and negative whenDψ<1/2D\_\{\\psi\}<1/2: it measures learned teacher\-likeness, not ground\-truth task success\. We use this AIRL\-style transformation, not AIRL’s full reward\-recovery architecture\. Optional task\-reward mixing addsα​rtenv\\alpha\\,r^\{\\mathrm\{env\}\}\_\{t\}; the main and best runs use none\.

Main results use single\-frame qpos; transition features as in AMP/GAIfO\[[4](https://arxiv.org/html/2610.33650#bib.bib4),[18](https://arxiv.org/html/2610.33650#bib.bib18)\]are an extension, not evaluated here\.

### 3\.3Policy update

For each group ofGGrollouts sharing a task, TAD sums rewards into episode returnsRi=∑tri,tR\_\{i\}=\\sum\_\{t\}r\_\{i,t\}and applies group\-relative normalization\[[25](https://arxiv.org/html/2610.33650#bib.bib25)\]:

A^i=Ri−R¯stdsample⁡\(R1,…,RG\)\+δ,\\hat\{A\}\_\{i\}=\\frac\{R\_\{i\}\-\\bar\{R\}\}\{\\operatorname\{std\}\_\{\\mathrm\{sample\}\}\(R\_\{1\},\\ldots,R\_\{G\}\)\+\\delta\},\(3\)whereR¯=G−1​∑iRi\\bar\{R\}=G^\{\-1\}\\sum\_\{i\}R\_\{i\}, the sample standard deviation uses denominatorG−1G\-1, andδ=10−6\\delta=10^\{\-6\}\. A rollout’s advantage is shared by all its valid action tokens\. Here a token denotes a likelihood component of an action chunk, not necessarily an autoregressively generated symbol\. Withjjindexing these components, defineℓθ,i​t​j=log⁡πθ,j​\(ai​t​j∣xi​t\)\\ell\_\{\\theta,itj\}=\\log\\pi\_\{\\theta,j\}\(a\_\{itj\}\\mid x\_\{it\}\)andρi​t​j=exp⁡\(ℓθ,i​t​j−ℓold,i​t​j\)\\rho\_\{itj\}=\\exp\(\\ell\_\{\\theta,itj\}\-\\ell\_\{\\mathrm\{old\},itj\}\)\. The actor minimizes a PPO\-style clipped objective\[[24](https://arxiv.org/html/2610.33650#bib.bib24),[25](https://arxiv.org/html/2610.33650#bib.bib25)\]:

ℒpol​\(θ\)=−𝔼i,t,j​\[min⁡\(ρi​t​j​A^i,clip⁡\(ρi​t​j,1−εlow,1\+εhigh\)​A^i\)\],\\mathcal\{L\}\_\{\\mathrm\{pol\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{i,t,j\}\\Big\[\\min\\big\(\\rho\_\{itj\}\\hat\{A\}\_\{i\},\\;\\operatorname\{clip\}\(\\rho\_\{itj\},1\-\\varepsilon\_\{\\mathrm\{low\}\},1\+\\varepsilon\_\{\\mathrm\{high\}\}\)\\hat\{A\}\_\{i\}\\big\)\\Big\],\(4\)where the expectation masks invalid tokens andℓold,i​t​j\\ell\_\{\\mathrm\{old\},itj\}is evaluated under the rollout policy\. The potentially asymmetric clipping bounds limit incentives for large likelihood\-ratio changes\.

OPD uses a different actor objective, not Eq\. \([4](https://arxiv.org/html/2610.33650#S3.E4)\)\. Its token\-level signal is the teacher log\-likelihood minus the old student’s log\-likelihood\. WritingℓT,i​t​j=log⁡πT,j​\(ai​t​j∣xi​t\)\\ell\_\{T,itj\}=\\log\\pi\_\{T,j\}\(a\_\{itj\}\\mid x\_\{it\}\), the loss is

ℒOPD​\(θ\)=−𝔼i,t,j​\[ℓθ,i​t​j​stopgrad⁡\(ℓT,i​t​j−ℓold,i​t​j\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{i,t,j\}\\\!\\left\[\\ell\_\{\\theta,itj\}\\,\\operatorname\{stopgrad\}\\\!\\left\(\\ell\_\{T,itj\}\-\\ell\_\{\\mathrm\{old\},itj\}\\right\)\\right\],with invalid tokens masked\. This teacher\-score signal is neither a chunk\-level scalar nor a group\-normalized trajectory return\.

Algorithm[1](https://arxiv.org/html/2610.33650#alg1)describes the distributed reference implementation\. The current discriminator scores each rollout before its update\. Rewards remain fixed while the actor and discriminator are updated in parallel; the updated discriminator is used for the next rollout batch\.

Algorithm 1TAD: trajectory\-only adversarial distillation1:Collect teacher trajectories

𝒟T\\mathcal\{D\}\_\{T\}once; fit

ϕ\\phiand cache bank

ℱT=\{ϕ⁡\(otT\)\}\\mathcal\{F\}\_\{T\}=\\\{\\phi\(o\_\{t\}^\{T\}\)\\\}
2:Initialize SFT student

πθ\\pi\_\{\\theta\}, discriminator

DψD\_\{\\psi\}, and replay

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset
3:forstep

=1,…,N=1,\\ldots,Ndo

4:Set

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta; roll out the student and extract

\{ϕ⁡\(oi​t\)\}\\\{\\phi\(o\_\{it\}\)\\\}
5:Score the current batch with

DψD\_\{\\psi\}using Eq\. \([2](https://arxiv.org/html/2610.33650#S3.E2)\); fix these rewards

6:Append current features to

ℬ\\mathcal\{B\}; retain the latest 4 batches

7:Compute returns and advantages with Eq\. \([3](https://arxiv.org/html/2610.33650#S3.E3)\)

8:In parallel: update

θ\\thetawith Eq\. \([4](https://arxiv.org/html/2610.33650#S3.E4)\) on the current batch;

9:update

DψD\_\{\\psi\}with Eq\. \([1](https://arxiv.org/html/2610.33650#S3.E1)\) on

\(ℱT,ℬ\)\(\\mathcal\{F\}\_\{T\},\\mathcal\{B\}\)
10:endfor

TeacherπT\\pi\_\{T\}frozenFixed teacherfeature bankStudentπθ\\pi\_\{\\theta\}VLA \+ LoRAEnvironmentStudent featuresqpos8DiscriminatorDψD\_\{\\psi\}Rewardrtr\_\{t\}Eq\.[2](https://arxiv.org/html/2610.33650#S3.E2)once, via environmentactionobservationGRPO policy update \(Eq\.[4](https://arxiv.org/html/2610.33650#S3.E4)\)Figure 1:Fixed teacher data, online student learning\. The top path collects teacher features once\. Below, the student acts in the environment, receives observations, and learns from discriminator rewards\. The teacher never scores student actions during training\.

## 4Experimental Setup

Task\.We use ManiSkill3\[[26](https://arxiv.org/html/2610.33650#bib.bib26)\]PutOnPlateInScene25Main\-v3, a Bridge\-style pick\-and\-place task with a WidowX arm and randomized scene composition\.

Teacher and student\.The student is OpenVLA\-OFT \(7B\), fine\-tuned with LoRA\[[27](https://arxiv.org/html/2610.33650#bib.bib27)\]via SFT, with baseline success\-once26\.2%26\.2\\%and success\-at\-end20\.3%20\.3\\%\. The teacher uses the same architecture after GRPO RL and reaches93\.8%93\.8\\%/77\.3%77\.3\\%in our local evaluation\. The model card separately reports a success rate of94\.1%94\.1\\%;111Teacher model card:[https://huggingface\.co/RLinf/RLinf\-OpenVLAOFT\-GRPO\-ManiSkill3\-25ood/raw/main/README\.md](https://huggingface.co/RLinf/RLinf-OpenVLAOFT-GRPO-ManiSkill3-25ood/raw/main/README.md)\.its metric definition and evaluation protocol have not been established as equivalent to ours\. Policy inputs contain images and the instruction, not qpos; the latter is used separately for discriminator features\.

Metrics and scope\.Standalone evaluation uses greedy decoding on a fixed suite\. success\-once means success at any point; success\-at\-end requires success at termination\. Main results and continuation ablations usen=256n\{=\}256trajectories per checkpoint, not training seeds\. Differences are point estimates, not evidence of statistical equivalence; training\-seed variation is untested\. Sampled training rollouts are separate process diagnostics, not interchangeable with deterministic evaluations\.

OPD \(white\-box\)\.The fixed teacher scores individual likelihood components of student action chunks\. The actor uses the masked token\-level signalℓT,i​t​j−ℓold,i​t​j\\ell\_\{T,itj\}\-\\ell\_\{\\mathrm\{old\},itj\}in the unclipped OPD objective of §[3\.3](https://arxiv.org/html/2610.33650#S3.SS3), with learning rate5​e\-55\\text\{e\-5\}\. It does not use TAD’s group\-normalized returns or clipped policy loss\.

TAD \(black\-box\)\.A ReLU MLP\[256,256\]\[256,256\]discriminates qpos8 features normalized using the fixed teacher\-bank statistics\. AdamW trains it with the smoothed BCE described after Eq\. \([1](https://arxiv.org/html/2610.33650#S3.E1)\), label smoothing0\.10\.1, weight decay0\.010\.01, gradient clip1010, and the last 4 student batches\. The actor uses learning rate5​e\-55\\text\{e\-5\}and one update epoch in the baseline\. Both baseline and four\-epoch continuation use only Eq\. \([2](https://arxiv.org/html/2610.33650#S3.E2)\), without environment task reward\. Ablations add gradient penalty \(gp\_coef\) or task reward withα=0\.05\\alpha\{=\}0\.05\.

Collection and training units\.One outer iteration \(a training “step”\) uses 64 parallel environments and 4 rollout rounds:64×4=25664\\times 4=256student trajectories, each with an 80\-environment\-step horizon and action chunks of length 8\. Teacher collection uses the same environment configuration for 5 outer iterations, yielding12,800=5×4×64×\(80/8\)12\{,\}800=5\\times 4\\times 64\\times\(80/8\)feature samples at chunk boundaries\. This bank is never expanded\. Student interaction and optimizer batch\-passes are counted separately in §[6\.2](https://arxiv.org/html/2610.33650#S6.SS2)\.

System\.RLinf\[[23](https://arxiv.org/html/2610.33650#bib.bib23)\]provides Ray scheduling, an FSDP actor, and Hydra configuration\.

## 5Training Diagnostics and Ablations

Accurate discrimination does not guarantee a better policy\. The first three diagnostics probe inputs, reward source, and actor updates \(Tables[1](https://arxiv.org/html/2610.33650#S5.T1)–[3](https://arxiv.org/html/2610.33650#S5.T3)\); variant IDs identify configurations, not controlled single\-factor comparisons\. Several settings, training budgets, and evaluation checkpoints differ\. All except v8 use actor lr≤1​e\-5\\leq 1\\text\{e\-5\}\. Section[5\.4](https://arxiv.org/html/2610.33650#S5.SS4)then compares continuations from a shared checkpoint\.

### 5\.1Discriminator design: inputs and training strength

With state–action features, discriminator accuracy approached 0\.998 but the student did not improve \(v1\)\. Weaker discriminator training reduced saturation without raising success \(v2; Table[1](https://arxiv.org/html/2610.33650#S5.T1)\)\. Easy separation, potentially through action features, is not enough; neither is avoiding saturation\.

Table 1:Discriminator\-design diagnostics\. Success is standalone deterministic success\-once; these variants change several discriminator settings together\.State\-only qpos8 inputs were used next: v5 stopped at a host out\-of\-memory failure before evaluation, while v6 also changed reward and regularization\. Unlike GAD’s teacher\-output warmup\[[16](https://arxiv.org/html/2610.33650#bib.bib16),[10](https://arxiv.org/html/2610.33650#bib.bib10)\], our student starts from SFT on a different demonstration dataset; we use no separate discriminator warmup\.

### 5\.2Reward source and controls

Task\-reward mixing and additional discriminator updates left evaluation near SFT \(v3\)\. Even random\-noise rewards produced a sampled rollout peak of42%42\\%\(v4\), showing why a transient peak is not sufficient evidence of imitation\. This control does not show that learned rewards are irrelevant\. Combining qpos8, logit rewards, weight decay, and replay reduced discriminator saturation but still did not improve evaluation \(v6; Table[2](https://arxiv.org/html/2610.33650#S5.T2)\), leaving reward quality and actor optimization as possible bottlenecks\.

Table 2:Reward diagnostics\. Eval denotes standalone deterministic success\-once; v4 has only a sampled rollout observation\. v6 changes input, reward, weight decay, and replay together\.
### 5\.3Learning rate and update count

At actor lr2​e\-62\\text\{e\-6\}, both TAD \(v6\) and a separate OPD run improved little; OPD reached only28\.1%28\.1\\%evaluation success\. At1​e\-51\\text\{e\-5\}, one or four update epochs gave similar sampled rollout levels \(v7/v9\)\. At5​e\-55\\text\{e\-5\}, v8 reached68\.8%68\.8\\%deterministic success at step\-150 with the same discriminator recipe \(Table[3](https://arxiv.org/html/2610.33650#S5.T3)\)\. These unmatched runs motivate the chosen learning rate, not a causal threshold\. The low\-rate four\-epoch result does not preclude gains at the higher rate tested next\.

Table 3:Learning\-rate and update\-count diagnostics using the v6 discriminator recipe\. Reported budgets and checkpoints are not matched;\*denotes sampled rollout success rather than standalone evaluation\.
### 5\.4Ablations

Three branches resume the same baseline step\-150 checkpoint, change one setting each, and run to step\-300\. The baseline also continues to step\-300, but is evaluated only at steps 150 and 280: there is no matched baseline step\-300 evaluation, so these endpoints do not isolate the effect of update epochs at equal interaction budget\. Table[4](https://arxiv.org/html/2610.33650#S5.T4)preserves these endpoints and the raw rollout ranges\. Figure[2](https://arxiv.org/html/2610.33650#S5.F2)shows nine\-point smoothed curves; the baseline range is59\.4%59\.4\\%–69\.5%69\.5\\%over steps 150–300\.

Discriminator regularization\.The gradient\-penalty branch with coefficient 10 does not exceed the step\-150 baseline on both evaluation metrics: success\-once is close while success\-at\-end is lower\.222Archived summaries specify coefficient 10 but do not resolve whether discriminator probability or logit was penalized; the exact variant remains unverified\.Its smoothed rollout range is60\.9%60\.9\\%–70\.1%70\.1\\%over the available steps 196–300\.

Policy updates\.The tested four\-epoch continuation achieves the highest TAD evaluations, with success\-once / success\-at\-end7\.87\.8/12\.112\.1pp above step\-150 without new teacher data\. Yet the smoothed rollout range,59\.8%59\.8\\%–70\.0%70\.0\\%over steps 151–300, largely overlaps the baseline: deterministic gains need not produce equally large changes in sampled rollouts\.

Reward mixing\.The environment\-reward branch with coefficient 0\.05 has slightly higher success\-once than step\-150 and unchanged success\-at\-end, remaining below the four\-epoch continuation\. Its smoothed rollout range is58\.3%58\.3\\%–65\.5%65\.5\\%over steps 151–300\.

Figure 2:Continuations from step\-150\. Thin lines are raw sampled success; thick lines are centered nine\-point moving averages with available\-point boundary windows\. The gray band is the baseline v8 raw range, not a confidence interval\. The red dotted line is the baseline step\-150 deterministic success\-once \(68\.8%68\.8\\%\), not a branch evaluation; annotations give each branch’s independent evaluation\.Table 4:Continuation results\. Raw ranges cover the full available log intervals listed below\. Evaluations use 256 trajectories each; checkpoint labels distinguish the baseline’s step\-150/280 evaluations from the branches’ step\-300 evaluations\.

## 6Results and Analysis

### 6\.1Black\-box versus white\-box distillation

Teacher trajectories alone support a50\.450\.4pp gain in success\-once over SFT\. The best TAD checkpoint reaches76\.6%76\.6\\%/58\.6%58\.6\\%, versus OPD’s81\.6%81\.6\\%/59\.4%59\.4\\%\(Table[5](https://arxiv.org/html/2610.33650#S6.T5)\)\. Baseline TAD shows no further gain at its later evaluated checkpoint; the four\-epoch continuation reaches higher success with the same teacher bank\. Figure[3](https://arxiv.org/html/2610.33650#S6.F3)places these evaluations alongside sampled training curves\.

The best TAD run reaches approximately94%94\\%of OPD’s success\-once and99%99\\%of its success\-at\-end, compared with approximately84%84\\%of OPD’s success\-once for baseline TAD at step\-150\. These are ratios of raw success rates, not fractions of the improvement over SFT, and the training budgets differ\.

Table 5:Standalone deterministic evaluation on ManiSkillPutOnPlateInScene25Main, with 256 trajectories per checkpoint\. The four\-update\-epoch branch resumes from TAD step\-150 and continues to step\-300\.Methodckptsuccess\-oncesuccess\-at\-endnnStudent \(SFT\)–26\.2%20\.3%256TAD \(black\-box\)step\-15068\.8%46\.5%256step\-28066\.4%44\.1%256TAD \+ 4×\\timesupdate epochsstep\-30076\.6%58\.6%256OPD \(white\-box\)step\-2073\.8%53\.1%256step\-6081\.6%59\.4%256Teacher–93\.8%77\.3%256Figure 3:Sampled training success versus outer iteration\. Thin lines are raw values; centered moving averages use 11 points for baseline TAD, 5 for OPD, and 9 for the dashed four\-epoch continuation \(ue4\) from step\-150, with available\-point boundary windows\. Stars mark deterministic evaluations under a different decoding protocol\. The ue4 branch reaches76\.6%76\.6\\%evaluation success despite overlapping the baseline’s sampled rollout range\.
### 6\.2Training budgets

The best TAD run uses 300 iterations: one update epoch for the first 150 and four for the remaining 150\. OPD uses 60 iterations with one epoch per batch\. Thus TAD uses5×5\\timesthe student rollout trajectories \(300×256300\\times 256versus60×25660\\times 256\) and12\.5×12\.5\\timesthe optimizer batch\-passes \(\(150\+150×4\)/60\(150\+150\\times 4\)/60\), with no added teacher data\.

A batch\-pass is one optimization pass through a*complete rollout batch*, not one minibatch update\. These ratios measure neither GPU time, wall\-clock time, nor total computation\. Likewise, baseline TAD at step\-150 and OPD at step\-20 describe early progress at different success levels, not equal\-quality endpoints or final convergence\.

### 6\.3Reaching and retaining success

Baseline TAD at step\-150 trails OPD at step\-60 by12\.812\.8pp on success\-once and12\.912\.9pp on success\-at\-end; the higher\-update branch reduces these gaps to5\.05\.0/0\.80\.8pp\. Both methods can reach success and then lose it \(Figure[4](https://arxiv.org/html/2610.33650#S6.F4)\)\. This aggregate pattern does not establish a common cause\.

One actual TAD evaluation episode makes the distinction concrete \(Figure[5](https://arxiv.org/html/2610.33650#S6.F5)\): the student puts the object on the plate, then grasps it again and removes it\. The episode counts as success\-once, but not success\-at\-end\. Reaching the goal is not the same as leaving the scene in the desired state\.

Figure 4:Deterministic success\-once and success\-at\-end \(n=256n\{=\}256trajectories each\)\. Once\-to\-end gaps are 22\.3pp for baseline TAD at step\-150, 22\.2pp for OPD at step\-60, and 18\.0pp for TAD\+ue4 at step\-300\.![Refer to caption](https://arxiv.org/html/2610.33650v1/fig_failure.png)Figure 5:One episode from the TAD step\-150 deterministic evaluation, shown from left to right\. The student places the object on the plate \(green frame, success\-once true\), then re\-grasps and removes it \(red frame, success\-at\-end false\)\.
### 6\.4Interpretation and evidence scope

A fixed teacher bank can support online improvement while the student interacts with the environment and the discriminator adapts\. The four\-epoch continuation reaches higher evaluated success than the baseline’s evaluated checkpoints with the same bank, so the baseline plateau is not the highest performance demonstrated under this supervision\. Language\-model OPD work also examines optimizer effort\[[22](https://arxiv.org/html/2610.33650#bib.bib22)\]; the changing reward model and continuing environment interaction in TAD do not establish the same mechanism\.

The comparison concerns the source, granularity, and use of feedback, not scalar versus non\-scalar supervision\. OPD supplies a teacher\-minus\-old\-student log\-likelihood signal at each valid action token and optimizes an unclipped, stop\-gradient\-weighted loss\. TAD instead sums one discriminator reward per action chunk into a trajectory return, normalizes returns within a group, and shares the resulting advantage across valid action tokens in a clipped objective\. Its reward model also changes during training\. These differences may affect optimization, but the budget counts do not measure signal information content or identify a single cause of the performance gap\.

The chosen feature representation also limits what the reward can express\. qpos8 omits object pose, gripper–object relations, and temporal progress\. Similar feature marginals can therefore accompany different task outcomes\. Richer spatial or temporal features are candidates for future study, not tested explanations for the remaining gap\. State\-only discrimination is a design choice, not a requirement of trajectory\-only teacher access\.

Evidence covers one manipulation task and one student architecture\. Without behavioral cloning on the same teacher trajectories, we establish improvement over SFT and a quantified comparison with OPD, not superiority over BC\. Neither method’s best attainable performance is determined\. Archived summaries and curves do not substitute for complete per\-trajectory evaluation logs and frozen executed\-configuration snapshots, which are not currently available; numerical results are retained as reported\.

## 7Conclusion

TAD enables online VLA improvement using a fixed bank of teacher joint\-position features rather than teacher action scores\. An adaptive discriminator and on\-policy student updates produce substantial gains without environment task rewards\. The tested four\-epoch continuation achieves the highest TAD result with unchanged teacher data, while using more student interaction and optimization than the reported OPD endpoint\. The results demonstrate the feasibility of this restricted teacher\-access setting and quantify its performance–budget trade\-off on the evaluated manipulation task\.

## References

- \[1\]J\. Ho and S\. Ermon\.Generative adversarial imitation learning\.*NeurIPS*, 2016\.
- \[2\]J\. Fu, K\. Luo, and S\. Levine\.Learning robust rewards with adversarial inverse reinforcement learning\.*ICLR*, 2018\.
- \[3\]I\. Kostrikov et al\.Discriminator\-actor\-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning\.*ICLR*, 2019\.
- \[4\]X\. B\. Peng et al\.AMP: Adversarial motion priors for stylized physics\-based character control\.*SIGGRAPH*, 2021\.
- \[5\]M\. Orsini et al\.What matters for adversarial imitation learning?*NeurIPS*, 2021\.
- \[6\]I\. Gulrajani et al\.Improved training of Wasserstein GANs\.*NeurIPS*, 2017\.
- \[7\]J\. Skalse et al\.Defining and characterizing reward gaming\.*NeurIPS*, 2022\.
- \[8\]G\. Hinton, O\. Vinyals, and J\. Dean\.Distilling the knowledge in a neural network\.arXiv:1503\.02531, 2015\.
- \[9\]R\. Agarwal et al\.On\-policy distillation of language models: Learning from self\-generated mistakes\.*ICLR*, 2024\.
- \[10\]Y\. Kim and A\. M\. Rush\.Sequence\-level knowledge distillation\.*EMNLP*, 2016\.
- \[11\]M\. J\. Kim et al\.OpenVLA: An open\-source vision\-language\-action model\.*CoRL*, 2024\.
- \[12\]M\. J\. Kim, C\. Finn, and P\. Liang\.Fine\-tuning vision\-language\-action models: Optimizing speed and success\.*RSS*, 2025\.
- \[13\]Z\. Zhong et al\.VLA\-OPD: Bridging offline SFT and online RL for vision\-language\-action models via on\-policy distillation\.arXiv:2603\.26666, 2026\.
- \[14\]Z\. Wan et al\.Adversarial dual on\-policy distillation from expressive teacher\.arXiv:2605\.27095, 2026\.
- \[15\]A\. Sauer et al\.Adversarial diffusion distillation\.*ECCV*, 2024\.
- \[16\]T\. Ye et al\.Black\-box on\-policy distillation of large language models\.arXiv:2511\.10643, 2025\.
- \[17\]S\. Ross, G\. Gordon, and D\. Bagnell\.A reduction of imitation learning and structured prediction to no\-regret online learning\.*AISTATS*, 2011\.
- \[18\]F\. Torabi, G\. Warnell, and P\. Stone\.Generative adversarial imitation from observation\.arXiv:1807\.06158, 2018\.
- \[19\]Y\. Takebayashi et al\.Experience\-based feasibility\-aware generative adversarial imitation from observation under embodiment mismatch\.arXiv:2610\.01171, 2026\.
- \[20\]J\. Fang et al\.Rubric\-based on\-policy distillation\.arXiv:2605\.07396, 2026\.
- \[21\]Y\. Luo et al\.On\-policy adversarial flow distillation for autoregressive video generation\.arXiv:2605\.26105, 2026\.
- \[22\]Z\. Fu et al\.Rethinking on\-policy distillation of large language models II: One training example\.arXiv:2609\.04172, 2026\.
- \[23\]C\. Yu et al\.RLinf: Flexible and efficient large\-scale reinforcement learning via macro\-to\-micro flow transformation\.*OSDI*, 2026\.
- \[24\]J\. Schulman et al\.Proximal policy optimization algorithms\.arXiv:1707\.06347, 2017\.
- \[25\]Z\. Shao et al\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.arXiv:2402\.03300, 2024\.
- \[26\]S\. Tao et al\.ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI\.arXiv:2410\.00425, 2024\.
- \[27\]E\. J\. Hu et al\.LoRA: Low\-rank adaptation of large language models\.arXiv:2106\.09685, 2021\.