Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptation

# Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptation
Source: [https://arxiv.org/html/2606.31043](https://arxiv.org/html/2606.31043)
Ethan Hirschowitz University of Sydney, Australia ehir9923@uni\.sydney\.edu\.au &Fabio Ramos University of Sydney, Australia NVIDIA, US fabio\.ramos@sydney\.edu\.au

###### Abstract

Residual reinforcement learning adapts a pretrained robot policy by learning an additive correction to its actions\. While effective when adaptation amounts to shifting the base policy’s action distribution, additive corrections cannot change the distribution’s shape, scale, or state\-dependent geometry — limitations we formalize as wrong variance, miscalibrated confidence, and non\-uniform correction\. We show that these matter under dynamics shift: when the base distribution is geometrically mismatched to the shifted system, residual correction can underperform even the unadapted policy\. We proposeWarp RL, a policy adaptation method that replaces additive residuals with an invertible, state\-conditioned transformation of the base policy’s action distribution\. Instantiated with monotonic rational\-quadratic spline flows\[[9](https://arxiv.org/html/2606.31043#bib.bib33)\], Warp RL preserves identity initialization, strictly generalizes additive residual correction, and exposes a structured adaptation space suitable for both policy\-gradient and gradient\-free optimization\. Across a variety of ManiSkill3 manipulationtasks with controlled dynamics shifts, Warp RL matches residual correction when translation is sufficient and substantially outperforms it when adaptation requires distributional reshaping\. We further demonstrate that warping can replace additive correction in an off\-policy sim\-to\-real pipeline, achieving comparable success rate with 30% faster task completion on a real\-robot peg\-insertion task\.

> Keywords:Reinforcement Learning, Policy Adaptation, Normalizing Flows, Sim\-to\-Real Transfer

## 1Introduction

Residual RLa=z\+δθ​\(s\)a=z\+\\delta\_\{\\theta\}\(s\)action spacebaseπ\\pitranslatedπ\\piSame shape, shifted meanWarp RLa=Tθ​\(z,s\)a=T\_\{\\theta\}\(z,\\,s\)action spacebaseπ\\piwarpedπ\\piReshaped: scale, skew, and shiftResidual RL⊂\\subsetWarp RL

Figure 1:Residual RL is a special case of Warp RL: additive corrections shift the base policy distribution, while general identity\-initialized warps can also reshape its location, scale, skew, and geometry\.Robot policies trained in one setting often degrade when deployed under different conditions\. Changes in object mass, contact properties, actuator response, or unmodeled real\-world effects can render a well\-trained policy unreliable, even when the task itself is unchanged\. Retraining from scratch discards useful learned behavior and may be impractical on real hardware\. A common alternative is*residual reinforcement learning*: freeze the base policy and learn a lightweight correction on top of it\[[15](https://arxiv.org/html/2606.31043#bib.bib32),[25](https://arxiv.org/html/2606.31043#bib.bib31),[4](https://arxiv.org/html/2606.31043#bib.bib30),[3](https://arxiv.org/html/2606.31043#bib.bib29),[12](https://arxiv.org/html/2606.31043#bib.bib35)\]\. Given a base policy actionz∼πbase\(⋅∣s\)z\\sim\\pi\_\{\\text\{base\}\}\(\\cdot\\mid s\), a residual policy learns a state\-dependent offset and executesa=z\+δθ​\(s\)a=z\+\\delta\_\{\\theta\}\(s\)\. This is simple, stable, and naturally identity\-initialized by settingδ\\deltanear zero\.

However, the additive formulation imposes a structural constraint: it can translate the base action distribution but cannot change its geometry\. If the base policy is Gaussian, the residual shifts the mean while preserving the variance and shape\. Some dynamics shifts are well\-served by translation — a policy may need to push harder or bias its commands in a consistent direction\. But when the base policy is too uncertain, overconfident in the wrong region, or miscalibrated in a way that depends on the sampled action itself, applying the same offset to every sample is insufficient\. We show empirically that residual correction can stall or even degrade below the unadapted base policy under such shifts\.

We proposeWarp RL, a policy correction framework that replaces additive residual correction with an invertible, state\-conditioned transformation of base\-policy samples\. Instead of executingz\+δθ​\(s\)z\+\\delta\_\{\\theta\}\(s\), we execute

a=Tθ\(z,s\),z∼πbase\(⋅∣s\),a=T\_\{\\theta\}\(z,s\),\\qquad z\\sim\\pi\_\{\\text\{base\}\}\(\\cdot\\mid s\),\(1\)whereTθ​\(⋅,s\)T\_\{\\theta\}\(\\cdot,s\)is initialized to the identity\. This preserves the appeal of residual learning — adaptation begins from the pretrained policy and modifies only a compact correction module — but unlike an additive residual, an invertible warp can reshape the scale and nonlinear geometry of the action distribution\. Additive residual RL is recovered as the special case whereTθT\_\{\\theta\}is a pure translation \(Figure[1](https://arxiv.org/html/2606.31043#S1.F1)\)\.

We instantiate the warp as a state\-conditioned rational\-quadratic spline flow\[[9](https://arxiv.org/html/2606.31043#bib.bib33)\], which provides monotonicity, analytic invertibility, and exact identity initialization, and can be trained with policy\-gradient methods or evolution strategies\. Our contributions are:

1. 1\.We formalize additive residual RL as a translation operator and identify three structural failure modes under dynamics shift\. We thus introduce Warp RL, an invertible, state\-conditioned correction that strictly generalizes additive residuals, instantiated with identity\-initialized RQ\-spline flows compatible with both policy\-gradient and gradient\-free training\.
2. 2\.We demonstrate on four manipulation tasks that Warp RL recovers performance where residual correction stalls or degrades, with Warp\-ES outperforming all baselines on every task and an ablation confirming that the gains depend on the structured spline parameterization rather than the optimizer alone\.
3. 3\.We show that the Warp RL integrates into an off\-policy sim\-to\-real pipeline achieving comparable success rate to residual correction while reducing median cycle time by 30%\.

## 2Related Work

#### Residual reinforcement learning\.

Early residual methods\[[15](https://arxiv.org/html/2606.31043#bib.bib32),[25](https://arxiv.org/html/2606.31043#bib.bib31)\]augment classical controllers with learned additive corrections for contact\-rich manipulation\. The same principle has since been applied to behavior\-cloned policies\[[4](https://arxiv.org/html/2606.31043#bib.bib30),[3](https://arxiv.org/html/2606.31043#bib.bib29)\], sim\-to\-real pipelines with RLPD\[[12](https://arxiv.org/html/2606.31043#bib.bib35)\], and stochastic base policies with uncertainty\-guided exploration\[[8](https://arxiv.org/html/2606.31043#bib.bib7)\]\. Despite differences in base policy type and training algorithm, these methods share an additive correction formulation\. The present work shows that this shared formulation is a structural limitation and proposes an alternative that subsumes translation as a special case\.

#### Normalizing flows in reinforcement learning\.

Normalizing flows transform simple distributions through invertible maps with tractable densities\[[19](https://arxiv.org/html/2606.31043#bib.bib28),[16](https://arxiv.org/html/2606.31043#bib.bib14)\]\. In RL, flows have been used as the policy class for improved exploration\[[29](https://arxiv.org/html/2606.31043#bib.bib23)\], constrained action generation\[[22](https://arxiv.org/html/2606.31043#bib.bib13)\], and competitive performance across imitation and offline settings\[[11](https://arxiv.org/html/2606.31043#bib.bib22)\]\. Rational\-quadratic spline flows\[[9](https://arxiv.org/html/2606.31043#bib.bib33)\]offer element\-wise monotonic bijections with analytic inverses\. Our work differs in composing a flow*on top of*a frozen base policy as a correction mechanism, rather than training a flow as the policy itself\.

#### Sim\-to\-real transfer and online adaptation\.

Domain randomization anticipates deployment variation during source\-domain training\[[28](https://arxiv.org/html/2606.31043#bib.bib20),[21](https://arxiv.org/html/2606.31043#bib.bib19)\]\. Other approaches build adaptability into the policy itself: meta\-learning optimizes for fast gradient\-based adaptation\[[10](https://arxiv.org/html/2606.31043#bib.bib1)\], while system identification conditions the policy on estimated or latent dynamics parameters\[[30](https://arxiv.org/html/2606.31043#bib.bib3),[17](https://arxiv.org/html/2606.31043#bib.bib4)\]\. These require the source\-domain training to anticipate adaptation\. In contrast, post\-hoc correction methods keep an arbitrary frozen base and train a compact module on top\. RLPD\[[5](https://arxiv.org/html/2606.31043#bib.bib34)\]provides an efficient substrate for real\-world fine\-tuning, and SPARR\[[12](https://arxiv.org/html/2606.31043#bib.bib35)\]demonstrates this with additive residuals in a sim\-to\-real assembly pipeline\. Warp RL falls in this category — analogous to parameter\-efficient adapters for frozen foundation models\[[14](https://arxiv.org/html/2606.31043#bib.bib2)\]— and we show it serves as a drop\-in replacement for additive correction in such pipelines\.

## 3Preliminaries

### 3\.1Residual RL as a Translation Operator

Residual reinforcement learning improves a frozen base policyπbase\\pi\_\{\\text\{base\}\}by training a correctionδθ​\(s\)\\delta\_\{\\theta\}\(s\)such that the executed action isa=z\+δθ​\(s\)a=z\+\\delta\_\{\\theta\}\(s\), wherez∼πbase\(⋅∣s\)z\\sim\\pi\_\{\\text\{base\}\}\(\\cdot\\mid s\)\. For a Gaussian base policy withz∼𝒩​\(μbase​\(s\),σbase​\(s\)\)z\\sim\\mathcal\{N\}\(\\mu\_\{\\text\{base\}\}\(s\),\\sigma\_\{\\text\{base\}\}\(s\)\), the resulting distribution is:

πresidual​\(a∣s\)=𝒩​\(μbase​\(s\)\+δθ​\(s\),σbase​\(s\)\)\.\\pi\_\{\\text\{residual\}\}\(a\\mid s\)=\\mathcal\{N\}\\big\(\\mu\_\{\\text\{base\}\}\(s\)\+\\delta\_\{\\theta\}\(s\),\\;\\sigma\_\{\\text\{base\}\}\(s\)\\big\)\.\(2\)The mean shifts by a state\-dependent offset, butVar\[πresidual\(⋅∣s\)\]=σbase2\(s\)\\text\{Var\}\[\\pi\_\{\\text\{residual\}\}\(\\cdot\\mid s\)\]=\\sigma\_\{\\text\{base\}\}^\{2\}\(s\)— the variance, shape, and number of modes are unchanged\. Addition is a*translation operator*on the action distribution: it repositions without altering geometry\. Translation can correct systematic biases and calibration offsets, but it cannot widen an overconfident distribution, narrow an uncertain one, or reshape the distribution non\-uniformly across its support\. These are structural limitations of the additive formulation, not artifacts of insufficient training\.

### 3\.2Failure Modes Under Dynamics Shift

Translation suffices when the base policy’s distribution has the right geometry and merely needs repositioning\. When this assumption breaks down, no additive correction can resolve the mismatch\. We identify three such failure modes:

1. F1\.Wrong variance\.Shifted dynamics require precise, committed actions, but the base policy’s broad distribution cannot be narrowed by translation alone\.
2. F2\.Miscalibrated confidence\.The base concentrates mass around a region that was optimal under original dynamics but is suboptimal under the shift; translation preserves this tight concentration\.
3. F3\.Non\-uniform correction\.The required correction varies across the action distribution — e\.g\., high\-magnitude actions are too aggressive while moderate actions are appropriate\. An additive offset treats all samples identically\.

In each case, an effective correction must be a function of the base action itself, not only of the state\. Section[4](https://arxiv.org/html/2606.31043#S4)introduces a framework with exactly this property\.

### 3\.3Normalizing Flows

A normalizing flow is a learnable bijectionTθ:ℝd→ℝdT\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\rightarrow\\mathbb\{R\}^\{d\}that transforms samples from a base distribution into a more expressive target\[[19](https://arxiv.org/html/2606.31043#bib.bib28),[16](https://arxiv.org/html/2606.31043#bib.bib14)\]\. Ifz∼pzz\\sim p\_\{z\}anda=Tθ​\(z\)a=T\_\{\\theta\}\(z\), thenaafollows the distribution induced by passingpzp\_\{z\}throughTθT\_\{\\theta\}\. Sampling from this warped distribution is exact: drawingzzand applyingTθT\_\{\\theta\}is the definition of sampling from it\. WhenTθT\_\{\\theta\}is differentiable and invertible, log\-probabilities are available in closed form via the change\-of\-variables formula:

log⁡pa​\(a\)=log⁡pz​\(Tθ−1​\(a\)\)\+log⁡\|det∂Tθ−1∂a\|,\\log p\_\{a\}\(a\)=\\log p\_\{z\}\\\!\\left\(T\_\{\\theta\}^\{\-1\}\(a\)\\right\)\+\\log\\left\|\\det\\frac\{\\partial T\_\{\\theta\}^\{\-1\}\}\{\\partial a\}\\right\|,\(3\)which makes flows compatible with policy\-gradient methods without Monte Carlo approximation of the policy ratio\.

### 3\.4Identity Initialization

A principled correction should begin as the identity so that early training cannot degrade a known\-good starting point\. In residual RL this is achieved by initializingδθ≈0\\delta\_\{\\theta\}\\approx 0\[[15](https://arxiv.org/html/2606.31043#bib.bib32),[25](https://arxiv.org/html/2606.31043#bib.bib31)\]\. The same principle appears in warped probabilistic models, where output\-space transformations are initialized at \(or given a prior peaked on\) the identity\[[26](https://arxiv.org/html/2606.31043#bib.bib27),[18](https://arxiv.org/html/2606.31043#bib.bib26)\]\. We adopt this philosophy: the learned warp should start as the identity and deviate only when supported by target\-domain returns\.

## 4Method: Warp RL

### 4\.1Problem Setting

We consider a Markov decision processℳ=\(𝒮,𝒜,R,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},R,\\gamma\)with state space𝒮\\mathcal\{S\}, action space𝒜⊆ℝd\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{d\}, reward functionRR, and discount factorγ\\gamma\. A base policyπbase\(⋅∣s\)=𝒩\(μbase\(s\),σbase\(s\)\)\\pi\_\{\\text\{base\}\}\(\\cdot\\mid s\)=\\mathcal\{N\}\(\\mu\_\{\\text\{base\}\}\(s\),\\sigma\_\{\\text\{base\}\}\(s\)\)has been trained under a particular dynamics regime and performs well in that setting\. However, when deployment conditions differ — due to changed physical parameters, modified task geometry, or transfer from simulation to a physical robot — the base policy’s performance degrades\.

Our goal is to learn a correction that recovers performance under the shifted dynamics while satisfying two constraints: the base policy remains frozen and the corrected policy starts as the base policy\. We seek parametersθ\\thetafor a transformationTθT\_\{\\theta\}such thatπθ=Tθ∘πbase\\pi\_\{\\theta\}=T\_\{\\theta\}\\circ\\pi\_\{\\text\{base\}\}maximizes expected return under the shifted dynamics:

θ∗=arg⁡maxθ⁡𝔼πθ​\[∑t=0Hγt​R​\(st,at\)\],subject toTθ0=Id\.\\theta^\{\*\}=\\arg\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\\\!\\left\[\\sum\_\{t=0\}^\{H\}\\gamma^\{t\}\\,R\(s\_\{t\},a\_\{t\}\)\\right\],\\quad\\text\{subject to\}\\quad T\_\{\\theta\_\{0\}\}=\\text\{Id\}\.\(4\)In residual RL,TθT\_\{\\theta\}takes the forma=z\+δθ​\(s\)a=z\+\\delta\_\{\\theta\}\(s\)wherez∼πbase\(⋅∣s\)z\\sim\\pi\_\{\\text\{base\}\}\(\\cdot\\mid s\)\. As established in Section[3](https://arxiv.org/html/2606.31043#S3), this restricts the correction to a translation of the base distribution\. We propose a strictly more expressive alternative through Warp RL\.

### 4\.2Warping Framework

Rather than adding a state\-dependent offset to the base policy’s actions, we learn an invertible, state\-conditioned transformationTθ:ℝd×𝒮→ℝdT\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\times\\mathcal\{S\}\\rightarrow\\mathbb\{R\}^\{d\}that operates directly on samples from the base policy\. At each timestep, the warped policy\(1\)observes statess,\(2\)samplesz∼πbase\(⋅∣s\)z\\sim\\pi\_\{\\text\{base\}\}\(\\cdot\\mid s\)from the frozen base policy,\(3\)computesa=Tθ​\(z,s\)a=T\_\{\\theta\}\(z,s\), and\(4\)executesaain the environment\. The resulting action is a sample from the warped distribution induced by passingπbase\\pi\_\{\\text\{base\}\}throughTθT\_\{\\theta\}— this is exact, not an approximation\. By the change\-of\-variables formula \(Eq\.[3](https://arxiv.org/html/2606.31043#S3.E3)\), the warped policy has density

πθ​\(a∣s\)=πbase​\(Tθ−1​\(a,s\)∣s\)⋅\|det∂Tθ−1∂a\|,\\pi\_\{\\theta\}\(a\\mid s\)=\\pi\_\{\\text\{base\}\}\\\!\\big\(T\_\{\\theta\}^\{\-1\}\(a,s\)\\mid s\\big\)\\cdot\\left\|\\det\\frac\{\\partial T\_\{\\theta\}^\{\-1\}\}\{\\partial a\}\\right\|,\(5\)which is tractable wheneverTθT\_\{\\theta\}has an analytic inverse and Jacobian\.

ForTθT\_\{\\theta\}to serve as a principled correction, it must be*invertible*\(ensuring valid densities via Eq\.[3](https://arxiv.org/html/2606.31043#S3.E3)\),*state\-conditioned*\(so corrections vary across contexts\), and*identity\-initialized*\(Section[3\.4](https://arxiv.org/html/2606.31043#S3.SS4)\)\. The expressiveness of this framework depends on the family from whichTθT\_\{\\theta\}is drawn:

T​\(z,s\)=z\+δ​\(s\)⏟additive \(residual RL\)⊂T​\(z,s\)=α​\(s\)⊙z\+δ​\(s\),α​\(s\)\>0⏟affine bijection⊂T​\(z,s\)=f​\(z;ϕ​\(s\)\)⏟nonlinear bijection\\underbrace\{T\(z,s\)=z\+\\delta\(s\)\}\_\{\\text\{additive \(residual RL\)\}\}\\;\\subset\\;\\underbrace\{T\(z,s\)=\\alpha\(s\)\\odot z\+\\delta\(s\),\\;\\alpha\(s\)\>0\}\_\{\\text\{affine bijection\}\}\\;\\subset\\;\\underbrace\{T\(z,s\)=f\(z;\\,\\phi\(s\)\)\}\_\{\\text\{nonlinear bijection\}\}\(6\)Additive residual RL is a degenerate case: a pure translation with unit scale and no nonlinear reshaping\. To see that this inclusion is strict, observe that any translationT​\(z,s\)=z\+δ​\(s\)T\(z,s\)=z\+\\delta\(s\)is a monotonic bijection with unit Jacobian; the RQ\-spline family contains this map as a single linear segment with unit slope and offsetδ​\(s\)\\delta\(s\), but also admits nonlinear reshaping that translation cannot express\. An affine transformation introduces learnable per\-dimension scaling, recovering the ability to narrow or widen the distribution \([F1](https://arxiv.org/html/2606.31043#S3.I1.i1),[F2](https://arxiv.org/html/2606.31043#S3.I1.i2)\)\. A nonlinear, monotonic transformation subsumes both and can additionally apply non\-uniform reshaping across the support \([F3](https://arxiv.org/html/2606.31043#S3.I1.i3)\)\. The specific nonlinear instantiation we adopt is described next\.

### 4\.3Instantiation: State\-Conditioned RQ\-Spline Flow

We instantiateTθT\_\{\\theta\}as a rational\-quadratic \(RQ\) spline flow\[[9](https://arxiv.org/html/2606.31043#bib.bib33)\]\. Unlike coupling\-layer\[[7](https://arxiv.org/html/2606.31043#bib.bib10)\]or autoregressive\[[20](https://arxiv.org/html/2606.31043#bib.bib9)\]architectures, the RQ\-spline applies an independent monotonic bijection per action dimension in a single forward pass, yielding a diagonal Jacobian andO​\(d\)O\(d\)log\-probability evaluation via Eq\.[3](https://arxiv.org/html/2606.31043#S3.E3)\. The spline is defined on a bounded action domain\[ℓ,h\]d\[\\ell,h\]^\{d\}\. Gaussian base policies produce unbounded samplesuu; these are mapped in via an element\-wisetanh\\tanhsquash,z~=c\+r⊙tanh⁡\(u\)\\tilde\{z\}=c\+r\\odot\\tanh\(u\), whereccandrrare the domain center and half\-range\. This bounding map is itself an invertible differentiable function with a diagonal Jacobian, so the composed pipelineu↦z~↦a=Tθ​\(z~,s\)u\\mapsto\\tilde\{z\}\\mapsto a=T\_\{\\theta\}\(\\tilde\{z\},s\)remains fully invertible and admits tractable log\-probabilities\.

A conditioning MLP maps the observationsstoO​\(K\)O\(K\)raw parameters per action dimension forKKspline bins:KKbin widths \(softmax\),KKbin heights \(softmax\), andK\+1K\{\+\}1knot derivatives \(exponentiation\)\. WithK=4K\{=\}4andd=6d\{=\}6, the spline adds3​K\+1=133K\{\+\}1=13parameters per dimension; total correction\-module sizes are comparable across methods\. Identity initialization follows from zeroing the MLP output layer: uniform bins and unit knot derivatives recover the identity map \(Section[3\.4](https://arxiv.org/html/2606.31043#S3.SS4)\)\.

### 4\.4Training

The warping framework is agnostic to the choice of optimizer\. BecauseTθT\_\{\\theta\}is a self\-contained module with a compact parameter space, any method that can maximize expected return with respect toθ\\thetais applicable\.

#### Warp RL with PPO\.

The analytic invertibility of the RQ\-spline makes Warp RL directly compatible with policy\-gradient algorithms\. The composed forward pass samplesu∼𝒩​\(μbase​\(s\),σbase​\(s\)\)u\\sim\\mathcal\{N\}\(\\mu\_\{\\text\{base\}\}\(s\),\\sigma\_\{\\text\{base\}\}\(s\)\), squashes toz~\\tilde\{z\}, and applies the spline to producea=Tθ​\(z~,s\)a=T\_\{\\theta\}\(\\tilde\{z\},s\)\(Section[4\.3](https://arxiv.org/html/2606.31043#S4.SS3)\)\. Because every stage is invertible, the log\-probability of the warped policy is available in closed form:

log⁡πθ​\(a∣s\)=log⁡𝒩​\(u;μbase​\(s\),σbase​\(s\)\)−∑i=1dlog⁡\|∂z~i∂ui\|−∑i=1dlog⁡\|∂ai∂z~i\|,\\log\\pi\_\{\\theta\}\(a\\mid s\)=\\log\\mathcal\{N\}\\\!\\left\(u;\\,\\mu\_\{\\text\{base\}\}\(s\),\\,\\sigma\_\{\\text\{base\}\}\(s\)\\right\)\-\\sum\_\{i=1\}^\{d\}\\log\\left\|\\frac\{\\partial\\tilde\{z\}\_\{i\}\}\{\\partial u\_\{i\}\}\\right\|\-\\sum\_\{i=1\}^\{d\}\\log\\left\|\\frac\{\\partial a\_\{i\}\}\{\\partial\\tilde\{z\}\_\{i\}\}\\right\|,\(7\)where the first correction term is the squash Jacobian and the second is the spline Jacobian; both are diagonal, so the log\-determinants reduce to sums over action dimensions\. This enables direct application of the PPO clipped surrogate objective\[[24](https://arxiv.org/html/2606.31043#bib.bib12)\]:

LPPO​\(θ\)=𝔼t​\[min⁡\(rt​\(θ\)​A^t,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\],rt​\(θ\)=πθ​\(at∣st\)πθold​\(at∣st\),L^\{\\text\{PPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\min\\\!\\left\(r\_\{t\}\(\\theta\)\\,\\hat\{A\}\_\{t\},\\;\\text\{clip\}\(r\_\{t\}\(\\theta\),1\{\-\}\\epsilon,1\{\+\}\\epsilon\)\\,\\hat\{A\}\_\{t\}\\right\)\\right\],\\quad r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\},\(8\)where both the numerator and denominator are evaluated via Eq\.[7](https://arxiv.org/html/2606.31043#S4.E7)\. Entropy is estimated via Monte Carlo samples from the base distribution\.

#### Warp RL with SAC\.

The same log\-probability computation \(Eq\.[7](https://arxiv.org/html/2606.31043#S4.E7)\) makes Warp RL compatible with off\-policy algorithms such as SAC\[[13](https://arxiv.org/html/2606.31043#bib.bib11)\]\. In the RLPD setting \(Section[5\.2](https://arxiv.org/html/2606.31043#S5.SS2)\), we use the spline output as a bounded correction added to the clipped base\-policy mean,a=clip​\(μbase\+correction,−1,1\)a=\\text\{clip\}\(\\mu\_\{\\text\{base\}\}\+\\text\{correction\},\\;\{\-1\},\\;1\), rather than as the direct action\. This makes the comparison with additive residual correction structurally symmetric: both methods produce bounded adjustments to the same deterministic base action, isolating the correction architecture as the sole variable\.

#### Warp RL with Evolution Strategies\.

Evolution strategies\[[23](https://arxiv.org/html/2606.31043#bib.bib25)\]treat the spline as a black box, perturbingθ\\thetawith isotropic Gaussian noise and evaluating each candidate via rollouts:

∇θJ​\(θ\)≈1n​σ​∑i=1nFi​ϵi,ϵi∼𝒩​\(0,I\),\\nabla\_\{\\theta\}J\(\\theta\)\\approx\\frac\{1\}\{n\\sigma\}\\sum\_\{i=1\}^\{n\}F\_\{i\}\\,\\epsilon\_\{i\},\\qquad\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,I\),\(9\)whereFiF\_\{i\}is the return with parametersθ\+σ​ϵi\\theta\+\\sigma\\epsilon\_\{i\}\. Only the forward pass is needed — no inverse, Jacobian, or log\-probability computation\. We use antithetic sampling and centered\-rank fitness shaping; details are in Section[5](https://arxiv.org/html/2606.31043#S5)\.

## 5Experiments

We evaluate Warp RL on four manipulation tasks under controlled dynamics shifts in simulation and on a real\-robot peg\-insertion task\.

![Refer to caption](https://arxiv.org/html/2606.31043v1/figures/task_pushcube.png)

\(a\)PushCube
![Refer to caption](https://arxiv.org/html/2606.31043v1/figures/task_pusht.png)

\(b\)PushT
![Refer to caption](https://arxiv.org/html/2606.31043v1/figures/task_liftpeg.png)

\(c\)LiftPegUpright
![Refer to caption](https://arxiv.org/html/2606.31043v1/figures/task_peginsertion.png)

\(d\)PegInsertion
![Refer to caption](https://arxiv.org/html/2606.31043v1/figures/task_real_robot.jpg)

\(e\)Real robot

Figure 2:Evaluation tasks\. \(a–d\) ManiSkill3 simulation tasks\[[27](https://arxiv.org/html/2606.31043#bib.bib24)\]used for controlled dynamics\-shift experiments \(Section[5\.1](https://arxiv.org/html/2606.31043#S5.SS1)\)\. \(e\) Physical SO101 robot arm performing peg insertion \(Section[5\.2](https://arxiv.org/html/2606.31043#S5.SS2)\)\.### 5\.1Controlled Simulation Adaptation

#### Setup\.

We evaluate on four ManiSkill3 manipulation tasks\[[27](https://arxiv.org/html/2606.31043#bib.bib24)\]: PushCube, PushT, LiftPegUpright, and PegInsertionSide \(task descriptions and success criteria in Appendix[C](https://arxiv.org/html/2606.31043#A3)\)\. For each task, a PPO base policy is trained under source dynamics and then frozen\. We introduce a task\-specific dynamics shift, reported in Table[1](https://arxiv.org/html/2606.31043#S5.T1), and train only a lightweight correction module on top of the frozen base policy\. Observations are proprioceptive state vectors and actions are bounded joint delta\-position commands\. We compare three identity\-initialized corrections using the same frozen base checkpoint: \(i\)*Residual\-PPO*, an additive correctiona=z\+δθ​\(s\)a=z\+\\delta\_\{\\theta\}\(s\); \(ii\)*Warp\-PPO*, an RQ\-spline warpa=Tθ​\(z,s\)a=T\_\{\\theta\}\(z,s\)trained with PPO; and \(iii\)*Warp\-ES*, the same spline trained with evolution strategies\. Residual\-PPO and Warp\-PPO isolate the correction family under a shared optimizer; Warp\-PPO and Warp\-ES isolate the optimizer under a shared warp parameterization\. All correction modules use a two\-layer MLP conditioning network; spline and training details are given in Appendix[B](https://arxiv.org/html/2606.31043#A2)\. Additional baselines are compared in Appendix[F](https://arxiv.org/html/2606.31043#A6)\.

#### Results\.

Table 1:Success rate \(%\) under source and shifted dynamics\. Correction methods report IQM \[95% CI\] over 5 training×\\times5 evaluation seeds\[[1](https://arxiv.org/html/2606.31043#bib.bib6)\]\.Table[1](https://arxiv.org/html/2606.31043#S5.T1)summarizes adaptation under shifted dynamics\. On PushCube, all correction methods recover strong performance, consistent with the prediction that when the required adaptation is primarily translational, warping subsumes residual correction without penalty \(Eq\.[6](https://arxiv.org/html/2606.31043#S4.E6)\)\. On PegInsertionSide, Residual\-PPO \(57\.4%\) falls below the unadapted shifted base \(59\.6%\): the residual’s state\-only offset \(Eq\.[2](https://arxiv.org/html/2606.31043#S3.E2)\) applies the same correction to every sample, and cannot provide the action\-dependent adjustment this task requires \([F3](https://arxiv.org/html/2606.31043#S3.I1.i3)\)\. Warp\-PPO improves over Residual\-PPO on LiftPegUpright and is comparable elsewhere, indicating that changing the correction family alone is not sufficient in all settings\. The strongest results come from Warp\-ES, which outperforms both PPO\-based corrections on every task, suggesting that warping is not only a more expressive correction family but also a compact, identity\-initialized adapter well\-suited to gradient\-free optimization\. These patterns map onto the framework’s predictions: LiftPegUpright requires variance adjustment impossible under pure translation \([F1](https://arxiv.org/html/2606.31043#S3.I1.i1)\), while PegInsertionSide benefits from the non\-uniform probability redistribution enabled by the Jacobian term in the warped density \(Eq\.[5](https://arxiv.org/html/2606.31043#S4.E5),[F3](https://arxiv.org/html/2606.31043#S3.I1.i3)\)\.

![Refer to caption](https://arxiv.org/html/2606.31043v1/x1.png)Figure 3:For thresholdτ\\tau, the curve shows the fraction of task\-seed combinations achieving recovery≥τ\\geq\\tau\. Shaded: 95% bootstrap CIs\.To aggregate across tasks, we define the recovery ratioρ=\(Jmethod−Jshifted\)/\(Jsource−Jshifted\)\\rho=\(J\_\{\\text\{method\}\}\-J\_\{\\text\{shifted\}\}\)/\(J\_\{\\text\{source\}\}\-J\_\{\\text\{shifted\}\}\), whereρ≥1\\rho\\geq 1indicates full recovery of source\-dynamics performance\. Figure[3](https://arxiv.org/html/2606.31043#S5.F3)plots recovery performance profiles as suggested by\[[1](https://arxiv.org/html/2606.31043#bib.bib6)\]\. Warp\-ES clearly dominates, achieving full recovery \(τ≥1\.0\\tau\\geq 1\.0\) on approximately 50% of task\-seed combinations\. Near the full\-recovery threshold, Warp\-PPO retains slightly more mass than Residual\-PPO, consistent with the per\-task gains on LiftPegUpright and PegInsertionSide \(Table[1](https://arxiv.org/html/2606.31043#S5.T1)\)\.

#### Ablation: correction architecture\.

To disentangle the correction architecture from the training method, we train four correction families under the same ES optimizer, directly testing each level of the correction hierarchy \(Eq\.[6](https://arxiv.org/html/2606.31043#S4.E6), Table[2](https://arxiv.org/html/2606.31043#S5.T2)\)\. Residual\-ES \(translation\) tests whether the optimizer alone explains the gains\. Affine\-ES \(affine bijection\) tests whether learnable per\-dimension scaling suffices without nonlinear reshaping\. MLP\-ES replaces the structured spline with an unconstrained network, providing nonlinear capacity without monotonicity or invertibility\.

Table 2:Ablation: correction architecture under ES\. IQM \(%\) over 5 training seeds\.The results reveal that neither the ES optimizer nor unconstrained nonlinearity alone accounts for Warp\-ES’s gains\. Residual\-ES recovers well on PushCube but fails on PushT and PegInsertionSide, and is unstable on LiftPegUpright\. Affine\-ES and MLP\-ES perform comparably to each other, improving substantially on LiftPegUpright but degrading below Residual\-ES on PushCube, PushT, and PegInsertionSide\. Without monotonicity, unconstrained networks can reorder base actions, disrupting the base policy’s learned structure rather than refining it\. MLP\-ES uses a comparable number of parameters to Warp\-ES but performs substantially worse on three of four tasks, indicating that the gains are attributable to the structured spline parameterization rather than model capacity\. Warp\-ES is the only method that consistently improves across all four tasks\.

### 5\.2Real\-World Adaptation

#### Setup\.

Following SPARR\[[12](https://arxiv.org/html/2606.31043#bib.bib35)\], a PPO base policy trained on PegInsertionSide in ManiSkill3 is deployed on a physical SO101 6\-DOF robot arm\. Prior demonstrations \(20 successful rollouts with noise\) seed an RLPD\[[5](https://arxiv.org/html/2606.31043#bib.bib34)\]/SAC\[[13](https://arxiv.org/html/2606.31043#bib.bib11)\]correction trained for 50 online episodes\. Both corrections produce bounded adjustments added to the clipped base mean,a=clip​\(μbase\+correction,−1,1\)a=\\text\{clip\}\(\\mu\_\{\\text\{base\}\}\+\\text\{correction\},\\;\{\-1\},\\;1\), isolating the correction architecture as the sole variable\. Details are given in Appendix[E](https://arxiv.org/html/2606.31043#A5)\.

#### Results\.

Following SPARR, we evaluate each method over 20 deterministic episodes with operator\-labeled success\. Table[3](https://arxiv.org/html/2606.31043#S5.T3)reports success rate and cycle time \(episode duration in seconds, computed over successful episodes only\)\.

Table 3:Real\-robot peg insertion: 20 deterministic evaluation episodes per method\. Cycle time is steps / 20 Hz, reported for successful episodes only \(±\\pmone standard deviation\)\.
#### Discussion\.

Both corrections improve substantially over the unadapted base \(45%→\\to65–70%\), confirming that the sim\-to\-real gap for this task is addressable by lightweight online correction without retraining the base policy\. Success rates are comparable atn=20n\{=\}20\. The SO101’s hobby\-grade servos exhibit significant backlash, introducing episode\-to\-episode variability that makes small success\-rate differences difficult to resolve at this sample size; cycle time, as a continuous metric measured over successful episodes, provides a more reliable signal\. The key differentiator is accordingly execution efficiency\. Residual correction barely changes the base policy’s cycle time \(4\.66 s vs\. 4\.73 s for the base\), consistent with the translation view of Eq\.[2](https://arxiv.org/html/2606.31043#S3.E2): the additive offset shifts which actions lead to success without altering how the policy explores the insertion region, so successful trajectories remain similarly cautious\. Warp correction reduces median cycle time to 3\.25 s — 30% faster than residual — while maintaining comparable success\. This suggests that the spline’s ability to redistribute probability mass \(Eq\.[5](https://arxiv.org/html/2606.31043#S4.E5)\) produces more direct insertion trajectories: rather than merely biasing the mean toward the hole, the warp concentrates the action distribution around a narrow, effective approach corridor\. The higher variance in warp cycle times \(σ=0\.93\\sigma\{=\}0\.93s vs\.0\.190\.19s\) reflects this sharper specialization: most episodes are substantially faster, but occasional episodes take longer when the reshaped distribution misses the tight 2 mm insertion tolerance\. From a deployment perspective, faster cycle times translate directly to higher throughput, making the warp correction practically preferable even at similar success rates\.

## 6Conclusion

Additive residual RL is a translation operator on the base policy’s action distribution: it can shift the mean but cannot change the variance, shape, or action\-dependent geometry\. We introduced Warp RL, which replaces translation with an invertible, state\-conditioned spline flow that strictly generalizes additive correction\. In simulation, Warp\-ES outperforms all baselines on every task and is the only method that improves over the unadapted base on PegInsertionSide, where residual correction actively degrades\. On a real SO101 robot, warp correction achieves comparable success rate to residual correction while reducing median cycle time by 30%\. The framework is a drop\-in replacement for additive residuals in existing policy adaptation pipelines\.

## 7Limitations and Future Work

Warp RL increases the expressiveness of residual policy adaptation but introduces design choices absent in purely additive methods\. The RQ\-spline operates on a bounded domain, so Gaussian base\-policy samples must be mapped in before transformation; tasks with near\-saturated commands \(e\.g\., PegInsertionSide\) can be sensitive to this bounding\. Our experiments use PPO\-trained Gaussian bases; extending to behavior\-cloned, chunked\-action, or diffusion\-style base policies via using warping as a drop in with existing residual pipelines\[[4](https://arxiv.org/html/2606.31043#bib.bib30),[3](https://arxiv.org/html/2606.31043#bib.bib29),[8](https://arxiv.org/html/2606.31043#bib.bib7)\]is a natural next step\. The per\-dimension, per\-timestep warp does not model cross\-dimensional or temporal correlations beyond what state conditioning provides — autoregressive or sequence\-level warps could increase expressiveness for higher\-dimensional systems\. Finally, the real\-robot evaluation uses a single task withn=20n\{=\}20episodes per method, following established protocol\[[12](https://arxiv.org/html/2606.31043#bib.bib35)\]; additional tasks and repetitions would strengthen statistical conclusions\.

## References

- \[1\]\(2021\-08\)Deep reinforcement learning at the edge of the statistical precipice\.arXiv \[cs\.LG\]\.Cited by:[Table 5](https://arxiv.org/html/2606.31043#A6.T5),[§5\.1](https://arxiv.org/html/2606.31043#S5.SS1.SSS0.Px2.p2.3),[Table 1](https://arxiv.org/html/2606.31043#S5.T1)\.
- \[2\]A\. Almuzairee and H\. I\. Christensen\(2026\-02\)Squint: fast visual reinforcement learning for sim\-to\-real robotics\.arXiv \[cs\.RO\]\.Cited by:[Appendix E](https://arxiv.org/html/2606.31043#A5.SS0.SSS0.Px2.p1.1)\.
- \[3\]L\. Ankile, Z\. Jiang, R\. Duan, G\. Shi, P\. Abbeel, and A\. Nagabandi\(2025\-09\)Residual off\-policy RL for finetuning behavior cloning policies\.arXiv \[cs\.RO\]\.Cited by:[Appendix D](https://arxiv.org/html/2606.31043#A4.SS0.SSS0.Px2.p1.1),[Appendix F](https://arxiv.org/html/2606.31043#A6.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2606.31043#S1.p1.3),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2606.31043#S7.p1.1)\.
- \[4\]L\. Ankile, A\. Simeonov, I\. Shenfeld, M\. Torne, and P\. Agrawal\(2024\-07\)From imitation to refinement – residual RL for precise assembly\.arXiv \[cs\.RO\]\.Cited by:[Appendix D](https://arxiv.org/html/2606.31043#A4.SS0.SSS0.Px2.p1.1),[Appendix F](https://arxiv.org/html/2606.31043#A6.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2606.31043#S1.p1.3),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2606.31043#S7.p1.1)\.
- \[5\]P\. J\. Ball, L\. Smith, I\. Kostrikov, and S\. Levine\(2023\-02\)Efficient online reinforcement learning with offline data\.arXiv \[cs\.LG\]\.Cited by:[Appendix E](https://arxiv.org/html/2606.31043#A5.SS0.SSS0.Px3.p1.5),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1),[§5\.2](https://arxiv.org/html/2606.31043#S5.SS2.SSS0.Px1.p1.1)\.
- \[6\]C\. Chi, Z\. Xu, S\. Feng, E\. Cousineau, Y\. Du, B\. Burchfiel, R\. Tedrake, and S\. Song\(2023\-03\)Diffusion policy: visuomotor policy learning via action diffusion\.arXiv \[cs\.RO\]\.Cited by:[Table 4](https://arxiv.org/html/2606.31043#A3.T4.1.1.1.1.1)\.
- \[7\]L\. Dinh, J\. Sohl\-Dickstein, and S\. Bengio\(2016\-05\)Density estimation using real NVP\.arXiv \[cs\.LG\]\.Cited by:[§4\.3](https://arxiv.org/html/2606.31043#S4.SS3.p1.9)\.
- \[8\]L\. Dodeja, K\. Schmeckpeper, S\. Vats, T\. Weng, M\. Jia, G\. Konidaris, and S\. Tellex\(2025\-06\)Accelerating residual reinforcement learning with uncertainty estimation\.arXiv \[cs\.LG\]\.Cited by:[Appendix D](https://arxiv.org/html/2606.31043#A4.SS0.SSS0.Px2.p1.1),[Appendix D](https://arxiv.org/html/2606.31043#A4.SS0.SSS0.Px2.p2.1),[Appendix F](https://arxiv.org/html/2606.31043#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2606.31043#S7.p1.1)\.
- \[9\]C\. Durkan, A\. Bekasov, I\. Murray, and G\. Papamakarios\(2019\-06\)Neural spline flows\.arXiv \[stat\.ML\]\.Cited by:[§1](https://arxiv.org/html/2606.31043#S1.p4.1),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2606.31043#S4.SS3.p1.9)\.
- \[10\]C\. Finn, P\. Abbeel, and S\. Levine\(2017\-03\)Model\-agnostic meta\-learning for fast adaptation of deep networks\.arXiv \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1)\.
- \[11\]R\. Ghugare and B\. Eysenbach\(2025\-05\)Normalizing flows are capable models for RL\.arXiv \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px2.p1.1)\.
- \[12\]Y\. Guo, I\. Akinola, L\. Johannsmeier, H\. Hadfield, A\. Gupta, and Y\. Narang\(2026\-02\)SPARR: simulation\-based policies with asymmetric real\-world residuals for assembly\.arXiv \[cs\.RO\]\.Cited by:[Table 4](https://arxiv.org/html/2606.31043#A3.T4.6.6.2.2.2),[Appendix D](https://arxiv.org/html/2606.31043#A4.SS0.SSS0.Px2.p1.1),[Appendix E](https://arxiv.org/html/2606.31043#A5.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2606.31043#S1.p1.3),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1),[§5\.2](https://arxiv.org/html/2606.31043#S5.SS2.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2606.31043#S7.p1.1)\.
- \[13\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\-01\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.arXiv \[cs\.LG\]\.Cited by:[§4\.4](https://arxiv.org/html/2606.31043#S4.SS4.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2606.31043#S5.SS2.SSS0.Px1.p1.1)\.
- \[14\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2021\-06\)LoRA: low\-rank adaptation of large language models\.arXiv \[cs\.CL\]\.Cited by:[Appendix F](https://arxiv.org/html/2606.31043#A6.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1)\.
- \[15\]T\. Johannink, S\. Bahl, A\. Nair, J\. Luo, A\. Kumar, M\. Loskyll, J\. A\. Ojea, E\. Solowjow, and S\. Levine\(2018\-12\)Residual reinforcement learning for robot control\.arXiv \[cs\.RO\]\.Cited by:[§1](https://arxiv.org/html/2606.31043#S1.p1.3),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2606.31043#S3.SS4.p1.1)\.
- \[16\]I\. Kobyzev, S\. J\. D\. Prince, and M\. A\. Brubaker\(2019\-08\)Normalizing flows: an introduction and review of current methods\.arXiv \[stat\.ML\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2606.31043#S3.SS3.p1.9)\.
- \[17\]A\. Kumar, Z\. Fu, D\. Pathak, and J\. Malik\(2021\-07\)RMA: rapid motor adaptation for legged robots\.arXiv \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]M\. Lázaro\-Gredilla\(2012\)Bayesian warped gaussian processes\.InAdvances in Neural Information Processing Systems,F\. Pereira, C\. J\. Burges, L\. Bottou, and K\. Weinberger \(Eds\.\),Vol\.25\.Cited by:[§3\.4](https://arxiv.org/html/2606.31043#S3.SS4.p1.1)\.
- \[19\]G\. Papamakarios, E\. Nalisnick, D\. J\. Rezende, S\. Mohamed, and B\. Lakshminarayanan\(2019\-12\)Normalizing flows for probabilistic modeling and inference\.arXiv \[stat\.ML\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2606.31043#S3.SS3.p1.9)\.
- \[20\]G\. Papamakarios, T\. Pavlakou, and I\. Murray\(2017\-05\)Masked autoregressive flow for density estimation\.arXiv \[stat\.ML\]\.Cited by:[§4\.3](https://arxiv.org/html/2606.31043#S4.SS3.p1.9)\.
- \[21\]X\. B\. Peng, M\. Andrychowicz, W\. Zaremba, and P\. Abbeel\(2017\-10\)Sim\-to\-real transfer of robotic control with dynamics randomization\.arXiv \[cs\.RO\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1)\.
- \[22\]F\. Rietz, E\. Schaffernicht, S\. Heinrich, and J\. A\. Stork\(2024\-05\)Towards interpretable reinforcement learning with constrained normalizing flow policies\.arXiv \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px2.p1.1)\.
- \[23\]T\. Salimans, J\. Ho, X\. Chen, S\. Sidor, and I\. Sutskever\(2017\-03\)Evolution strategies as a scalable alternative to reinforcement learning\.arXiv \[stat\.ML\]\.Cited by:[§4\.4](https://arxiv.org/html/2606.31043#S4.SS4.SSS0.Px3.p1.1)\.
- \[24\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\-07\)Proximal policy optimization algorithms\.arXiv \[cs\.LG\]\.Cited by:[§4\.4](https://arxiv.org/html/2606.31043#S4.SS4.SSS0.Px1.p1.4)\.
- \[25\]T\. Silver, K\. Allen, J\. Tenenbaum, and L\. Kaelbling\(2018\-12\)Residual policy learning\.arXiv \[cs\.RO\]\.Cited by:[Appendix D](https://arxiv.org/html/2606.31043#A4.SS0.SSS0.Px2.p2.1),[§1](https://arxiv.org/html/2606.31043#S1.p1.3),[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2606.31043#S3.SS4.p1.1)\.
- \[26\]E\. Snelson, Z\. Ghahramani, and C\. Rasmussen\(2003\)Warped gaussian processes\.InAdvances in Neural Information Processing Systems,S\. Thrun, L\. Saul, and B\. Schölkopf \(Eds\.\),Vol\.16\.Cited by:[§3\.4](https://arxiv.org/html/2606.31043#S3.SS4.p1.1)\.
- \[27\]S\. Tao, F\. Xiang, A\. Shukla, Y\. Qin, X\. Hinrichsen, X\. Yuan, C\. Bao, X\. Lin, Y\. Liu, T\. Chan, Y\. Gao, X\. Li, T\. Mu, N\. Xiao, A\. Gurha, V\. N\. Rajesh, Y\. W\. Choi, Y\. Chen, Z\. Huang, R\. Calandra, R\. Chen, S\. Luo, and H\. Su\(2024\-10\)ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI\.arXiv \[cs\.RO\]\.Cited by:[Appendix C](https://arxiv.org/html/2606.31043#A3.p1.1),[Figure 2](https://arxiv.org/html/2606.31043#S5.F2),[§5\.1](https://arxiv.org/html/2606.31043#S5.SS1.SSS0.Px1.p1.2)\.
- \[28\]J\. Tobin, R\. Fong, A\. Ray, J\. Schneider, W\. Zaremba, and P\. Abbeel\(2017\-03\)Domain randomization for transferring deep neural networks from simulation to the real world\.arXiv \[cs\.RO\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1)\.
- \[29\]P\. N\. Ward, A\. Smofsky, and A\. J\. Bose\(2019\-06\)Improving exploration in soft\-actor\-critic with normalizing flows policies\.arXiv \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px2.p1.1)\.
- \[30\]W\. Yu, J\. Tan, C\. Karen Liu, and G\. Turk\(2017\-02\)Preparing for the unknown: learning a universal policy with online system identification\.arXiv \[cs\.LG\]\.Cited by:[§2](https://arxiv.org/html/2606.31043#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix Overview

- [A](https://arxiv.org/html/2606.31043#A1)Training Curves
- [B](https://arxiv.org/html/2606.31043#A2)Training and Hyperparameter Details
- [C](https://arxiv.org/html/2606.31043#A3)Task Descriptions
- [D](https://arxiv.org/html/2606.31043#A4)Residual Implementation
- [E](https://arxiv.org/html/2606.31043#A5)Real\-Robot Experiment Details

## Appendix ATraining Curves

Each curve reports the interquartile mean \(IQM\) of eval success rate across training seeds at each step, with shaded regions indicating a 95% bootstrap confidence interval \(10,000 resamples\) around the IQM\. This is the same aggregation used for Table[1](https://arxiv.org/html/2606.31043#S5.T1)\. Success rates here are measured by periodic held\-out evaluation during training \(with 32 evaluation environments\), and the horizontal line in each panel marks the unadapted base policy’s performance under the shifted dynamics, averaged over same seeds used for the periodic evaluation\. Figure[4](https://arxiv.org/html/2606.31043#A1.F4)shows the three main correction methods \(Residual\-PPO, Warp\-PPO, Warp\-ES\); Figure[5](https://arxiv.org/html/2606.31043#A1.F5)shows the ablation architectures \(Residual\-ES, Affine\-ES, MLP\-ES, Warp\-ES\) trained under the same ES optimizer\. Learning curves for the additional adaptation baselines \(FT\-KL, FT\-Last\) are reported alongside these methods in Appendix[F](https://arxiv.org/html/2606.31043#A6)\. Because policies are deployed at a single fixed checkpoint rather than as a training trajectory, all reported results \(including Table[1](https://arxiv.org/html/2606.31043#S5.T1)\) evaluate each method at its best held\-out checkpoint \(see Appendix[B](https://arxiv.org/html/2606.31043#A2)\); the curves below show training progress, so a method’s reported value corresponds to its best point along the curve rather than its value at the final training step\.

![Refer to caption](https://arxiv.org/html/2606.31043v1/x2.png)Figure 4:Learning curves for the main correction methods under shifted dynamics\. IQM of success rate across seeds; shaded==95% bootstrap CI\. Reported values \(Table[1](https://arxiv.org/html/2606.31043#S5.T1)\) use the best held\-out checkpoint per method, which may differ from the final\-step value shown here\.![Refer to caption](https://arxiv.org/html/2606.31043v1/x3.png)Figure 5:Learning curves for the ablation architectures \(all trained with ES\) under shifted dynamics\. IQM of success rate across seeds; shaded==95% bootstrap CI\. Reported values \(Table[2](https://arxiv.org/html/2606.31043#S5.T2)\) use the best held\-out checkpoint per method\.#### Main\-method dynamics\.

Warp\-ES is the only method that reaches and sustains a high success rate across all four tasks\. Its curves are less smooth than the policy\-gradient methods’, reflecting the discrete, population\-based nature of evolution strategies: each evaluation point is the current mean of a noisy gradient\-free search rather than a smooth gradient step\. This fluctuation is a property of the optimizer and does not indicate instability — Warp\-ES sustains the highest performance across tasks\. Among the PPO\-based corrections, Warp\-PPO and Residual\-PPO are broadly comparable; Warp\-PPO peaks early and then declines on LiftPegUpright and PegInsertionSide, whereas Residual\-PPO is flatter over training\. This non\-monotonicity is consistent with the bounded\-domain sensitivity of the spline under policy\-gradient optimization noted in Section[7](https://arxiv.org/html/2606.31043#S7), and motivates evaluating all methods at their best held\-out checkpoint rather than the final step\.

#### Ablation dynamics\.

The four architectures impose different inductive biases that interact with ES’s parameter\-space perturbation\. Warp\-ES applies a smooth, monotone transformation that preserves base\-action ordering, giving ES a stable search space in which perturbations change behavior in a controlled way\. Residual\-ES and MLP\-ES permit more flexible additive modifications but are more brittle: small parameter changes can induce large, poorly structured behavioral shifts\. Affine\-ES is more stable but too constrained for nonlinear corrections on the harder tasks\. The best correction architecture for ES is thus not the most expressive, but the one whose structure best preserves competent base behavior while exposing a smooth family of corrections\.

## Appendix BTraining and Hyperparameter Details

Tables[B](https://arxiv.org/html/2606.31043#A2)and[B](https://arxiv.org/html/2606.31043#A2)report hyperparameters for the simulation and real\-robot experiments respectively\. The remainder of this section describes the training budget and the checkpoint\-selection protocol used for all reported results\.

\\captionof

tableHyperparameters for simulation overlay training \(PPO and ES\)\.

†PegInsertionSide usesK=8K\{=\}8, rollout steps=16\{=\}16,γ=0\.97\\gamma\{=\}0\.97,λ=0\.95\\lambda\{=\}0\.95\.

‡One environment per population member\.

\\captionof

tableHyperparameters for real\-robot RLPD/SAC experiments\.

#### Training and checkpoint selection\.

Each method is trained for 2M environment steps on the shifted dynamics, with 5 training seeds\. Every 5 optimizer iterations we evaluate the current policy for 100 deterministic episodes on 32 parallel environments under a fixed monitoring seed \(distinct from both the training and the final evaluation seeds\), and retain the checkpoint with the highest mean return\. Return is used for selection because it is a denser, lower\-variance signal than thresholded success rate\. The reported values — including those in Table[1](https://arxiv.org/html/2606.31043#S5.T1)— are obtained by evaluating this best checkpoint on a separate held\-out grid of 5 training×\\times5 evaluation seeds\. Selecting on a monitoring seed and reporting on a disjoint held\-out grid keeps checkpoint selection independent of the reported metric\. This protocol reflects deployment, where a single checkpoint is selected and executed rather than a training trajectory; the learning curves in Appendix[A](https://arxiv.org/html/2606.31043#A1)therefore show training progress, and a method’s reported value corresponds to its best monitored checkpoint rather than its final\-step value\.

## Appendix CTask Descriptions

All simulation tasks use a Franka Emika Panda arm in ManiSkill3\[[27](https://arxiv.org/html/2606.31043#bib.bib24)\]\. Observations are proprioceptive state vectors; actions are bounded delta\-position commands\. Table[4](https://arxiv.org/html/2606.31043#A3.T4)summarizes each task and its success criterion\.

Table 4:Task descriptions and success criteria\. Simulation tasks use the Franka arm; the real\-robot task uses an SO101 arm\.
## Appendix DResidual Implementation

This section details the residual baseline used in the simulation \(PPO\) experiments and justifies why its exploration is inherited from the frozen base policy\. The real\-robot residual uses a different, off\-policy formulation, which we contrast briefly at the end of this section and describe in full in Appendix[E](https://arxiv.org/html/2606.31043#A5)\.

#### PPO setting: inherited, frozen exploration\.

In the simulation experiments the residual shifts the frozen base policy’s mean while inheriting its variance:

πresidual​\(a∣s\)=𝒩​\(μbase​\(s\)\+δθ​\(s\),σbase​\(s\)\)\.\\pi\_\{\\text\{residual\}\}\(a\\mid s\)=\\mathcal\{N\}\\\!\\big\(\\mu\_\{\\text\{base\}\}\(s\)\+\\delta\_\{\\theta\}\(s\),\\;\\sigma\_\{\\text\{base\}\}\(s\)\\big\)\.\(10\)The offsetδθ​\(s\)\\delta\_\{\\theta\}\(s\)is produced by a zero\-initialized MLP, so the correction begins at the identity\. Exploration is supplied entirely by the frozen base standard deviationσbase​\(s\)\\sigma\_\{\\text\{base\}\}\(s\): the residual introduces no exploration parameter of its own, andσbase​\(s\)\\sigma\_\{\\text\{base\}\}\(s\)is used as learned and held fixed throughout adaptation\. This is the simplest design that ties the correction’s exploration to the base policy’s own calibrated, state\-dependent uncertainty, and it keeps the adapted policy a Gaussian with a tractable likelihood\. Because the base standard deviation already encodes which dimensions and states the pretrained policy treats as uncertain, reusing it concentrates exploration where the base is itself unsure rather than injecting an exploration scale chosen independently of the base\.

#### Relation to prior residual methods\.

Residual methods are typically applied on top of behavior\-cloned, action\-chunked, or diffusion bases: ResiP\[[4](https://arxiv.org/html/2606.31043#bib.bib30)\]learns an on\-policy PPO residual over an action\-chunked BC base, while the off\-policy methods — SPARR with RLPD/SAC\[[12](https://arxiv.org/html/2606.31043#bib.bib35)\], ResFiT with an off\-policy DDPG\-style recipe\[[3](https://arxiv.org/html/2606.31043#bib.bib29)\], and the uncertainty\-guided SAC method ofDodejaet al\.\[[8](https://arxiv.org/html/2606.31043#bib.bib7)\]— adapt BC or diffusion bases and supply their own correction exploration\. None of these base classes exposes a calibrated, per\-state Gaussian exploration distribution that a residual could reuse\. Our simulation base is itself a PPO Gaussian policy with a learned state\-dependentσbase​\(s\)\\sigma\_\{\\text\{base\}\}\(s\), so inheriting exploration is a structural option here that is simply unavailable when the base is a BC, chunked, or diffusion policy\. Inherited frozen exploration is therefore the natural choice for this setting rather than a workaround\.

This setting also avoids a failure mode documented in the off\-policy methods, where the correction disrupts the base before learning to improve it\. The reported causes are specific to off\-policy actor–critic learning:Silveret al\.\[[25](https://arxiv.org/html/2606.31043#bib.bib31)\]attribute the early dip to the critic being poorly initialized relative to the actor and mitigate it with a critic “burn\-in” period, whileDodejaet al\.\[[8](https://arxiv.org/html/2606.31043#bib.bib7)\]frame it as an unconstrained\-exploration phase and suppress it by gating the residual on base\-policy uncertainty\. With an on\-policy PPO base and a PPO correction there is no separately initialized critic bootstrapping the residual, and inherited frozen exploration keeps the adapted policy within the base policy’s behavioral distribution from the first update; the dominant documented cause is therefore structurally absent here\. We note this concerns those specific off\-policy mechanisms, and is not a general claim that on\-policy residual learning never degrades early\.

#### Ablation: removing inheritance\.

The key property of this design is that exploration is*inherited*from the base policy’s calibrated, state\-dependent uncertainty\. To isolate the effect of inheritance, we compare it against the standard alternative used when no such base distribution is available: an independent, trainable exploration distribution, with a scale not tied toσbase​\(s\)\\sigma\_\{\\text\{base\}\}\(s\), initialized at the PPO default and free to drift\. This independent\-and\-trainable setup is the conventional one — it is what off\-policy SAC and the standard PPO default use, and what residual methods over non\-Gaussian bases must use — so the comparison is against the field\-standard exploration mechanism rather than a contrived alternative\. We compare the two on PushT in Figure[6](https://arxiv.org/html/2606.31043#A4.F6); both share the same zero\-initialized mean correction, so the only difference is whether exploration is inherited from the base or supplied independently\. The inherited residual rises to a high success rate and holds stably, whereas the independent variant improves briefly before destabilizing and converging well below its own starting performance\. We therefore inherit the base policy’s exploration and hold it fixed during adaptation: tying the correction’s exploration to the base policy’s calibrated uncertainty is what yields stable adaptation, even though both variants begin from the same identity\-initialized mean\. Inheritance is available precisely because our base is a Gaussian policy with a learnedσbase​\(s\)\\sigma\_\{\\text\{base\}\}\(s\); where this is unavailable — as in the off\-policy real\-robot setting \(Appendix[E](https://arxiv.org/html/2606.31043#A5)\) — the correction reverts to the standard independent, trainable exploration\.

![Refer to caption](https://arxiv.org/html/2606.31043v1/x4.png)Figure 6:Removing inheritance under PPO \(PushT\)\.The residual that inherits the base policy’s exploration \(σbase​\(s\)\\sigma\_\{\\text\{base\}\}\(s\), frozen\) adapts stably, whereas one whose exploration is supplied independently of the base destabilizes and converges below its starting performance\. Both share the same identity\-initialized mean correction\. IQM of success rate across seeds; shaded==95% bootstrap CI \(as in Appendix[A](https://arxiv.org/html/2606.31043#A1)\)\.
#### Real\-robot setting\.

Inheritance is specific to the on\-policy PPO setting\. The real\-robot experiments instead use an off\-policy SAC/RLPD correction, where the residual supplies its own learned exploration rather than inheriting the base variance\. The reason is the optimizer: SAC’s maximum\-entropy objective regulates the entropy of a*trainable*correction distribution through a learned temperature, which presupposes that the correction carries its own controllable exploration; a frozen inherited variance has no entropy for the temperature to act on\. The full real\-robot correction architecture is given in Appendix[E](https://arxiv.org/html/2606.31043#A5)\.

## Appendix EReal\-Robot Experiment Details

#### Hardware and task\.

Experiments use an SO101 6\-DOF robot arm \(~$300 hardware\) performing peg insertion\. The observation is a 35\-dimensional proprioceptive state vector; actions are 6\-dimensional bounded delta\-position commands in\[−1,1\]\[\-1,1\]\. The control loop runs at 20 Hz, matching the simulation training configuration\. The peg position between episodes is set manually with slight inter\-episode variability\.

#### Differences from SPARR\.

Our pipeline follows SPARR\[[12](https://arxiv.org/html/2606.31043#bib.bib35)\]in using RLPD with a frozen simulation\-trained base policy and online SAC correction, but differs in several deployment details\. Within SPARR’s fixed real\-world training budget of 30 minutes per task, we train for 50 online episodes\. We apply exponential moving average smoothing to the commanded actions \(αema=0\.3\\alpha\_\{\\text\{ema\}\}=0\.3\) for temporal consistency\. Inspired by the action down\-scaling used for safe sim\-to\-real transfer in Squint\[[2](https://arxiv.org/html/2606.31043#bib.bib5)\], we additionally scale the actions sent to the robot by a factor of 0\.5\. Success is labeled manually by the operator via a keypress during the episode \(early success\) or a prompt after the episode completes\.

#### Correction architecture\.

Both methods produce acorrectionthat is consumed by the bounded mapping and critic described below, and differ only in how it is produced\. The*residual*correction is produced by an MLP that receives\(s,μbase\)\(s,\\mu\_\{\\text\{base\}\}\)as input; a sample is drawn from a Gaussian with the MLP’s output as mean and a learnable standard deviation, squashed throughtanh\\tanh, and scaled by a residual scale of 1\.0\. This off\-policy residual supplies its own learned exploration, in contrast to the inherited frozen exploration of the simulation \(PPO\) residual \(Appendix[D](https://arxiv.org/html/2606.31043#A4), Eq\.[10](https://arxiv.org/html/2606.31043#A4.E10)\)\. The*warp*correction is produced by a state\-conditioned RQ spline in bounded \(in\-box\) mode with 4 bins, identity initialization \(calibrated\_init=0\\text\{calibrated\\\_init\}=0\), and no identity penalty\. The critic for both takesQ​\(s,μbase,correction\)Q\(s,\\mu\_\{\\text\{base\}\},\\text\{correction\}\)as input, with LayerNorm following RLPD\[[5](https://arxiv.org/html/2606.31043#bib.bib34)\]\.

#### Why the warp produces a correction rather than the final action\.

In the simulation experiments the warp outputs the action directly \(a=Tθ​\(z,s\)a=T\_\{\\theta\}\(z,s\)\); the additive residual it is compared against produces a correction added to the base sample, as throughout the paper\. In the real\-robot setting the warp output is instead used as a correction to the base mean,a=clip​\(μbase​\(s\)\+correction,−1,1\)a=\\text\{clip\}\(\\mu\_\{\\text\{base\}\}\(s\)\+\\text\{correction\},\\,\{\-1\},\\,1\)\. The real\-robot setting requires sample efficiency, which means RLPD/SAC with demonstrations, and RLPD’s machinery is organized around corrections to the base action\. The demonstration buffer is seeded with base\-policy rollouts paired with sampled corrections, transitions are stored as\(s,μbase,correction,r\)\(s,\\mu\_\{\\text\{base\}\},\\text\{correction\},r\), and the critic learnsQ​\(s,μbase,correction\)Q\(s,\\mu\_\{\\text\{base\}\},\\text\{correction\}\)\. Expressing the warp as a correction to the base action is therefore what lets it slot into this machinery unchanged; having it emit a standalone action would require redefining what the demonstrations contain and what the critic conditions on, i\.e\. a different algorithm\. The simulation setting has no such constraint, so the warp outputs the action directly there\.

#### The warp transform is unchanged by the correction interface\.

Conforming to SPARR’s interface changes only the*output\-to\-action mapping*, not the warp itself\. The distribution transform is identical to the simulation setting: at every environment step and every actor update we drawz∼𝒩​\(μbase​\(s\),σbase​\(s\)\)z\\sim\\mathcal\{N\}\(\\mu\_\{\\text\{base\}\}\(s\),\\sigma\_\{\\text\{base\}\}\(s\)\), squash viatanh\\tanh, push through the spline, and train the SAC actor with the exact change\-of\-variables log\-probability of this warped sample \(Eq\.[7](https://arxiv.org/html/2606.31043#S4.E7)\) — the base Gaussian log\-density minus the diagonal log\-Jacobians of the squash and the spline\. The warp is therefore not a deterministic nonlinearity applied toμbase\\mu\_\{\\text\{base\}\}; it is the same sample\-and\-warp transform, with its output relocated by a constant shift\. Becauseμbase\\mu\_\{\\text\{base\}\}is a constant additive offset, the reshaping of the action distribution is still performed entirely by the spline, exactly as in the simulation setting; theμbase\\mu\_\{\\text\{base\}\}shift only repositions the already\-warped distribution in action space\. The shift leaves the warped sample’s density — and hence the actor and entropy terms that consume it — unchanged except at the clip boundary\. The density’s role does shift between optimizers: under PPO it drives the importance ratio, whereas under SAC it enters only through the actor objective \(the policy and maximum\-entropy terms\), the critic being agnostic to the correction’s parameterization\. Either way the warp’s invertibility and analytic Jacobian are required to evaluate it, so the flow structure is load\-bearing in the off\-policy setting and not merely a bounded nonlinearity\.

#### Evaluation protocol\.

Each method is evaluated over 20 episodes using a deterministic policy, following SPARR\. For the*residual*, the deterministic correction istanh⁡\(μθ\)\\tanh\(\\mu\_\{\\theta\}\)scaled by the residual scale, whereμθ\\mu\_\{\\theta\}is the learned mean\. For the*warp*, we use a Monte Carlo mean overK=64K=64stochastic samples, averaged in a single batched forward pass\. This approximates𝔼​\[spline​\(tanh⁡\(z\)\)\]\\mathbb\{E\}\[\\text\{spline\}\(\\tanh\(z\)\)\]rather than evaluating the single\-point modespline​\(tanh⁡\(μbase\)\)\\text\{spline\}\(\\tanh\(\\mu\_\{\\text\{base\}\}\)\), addressing the mode\-versus\-expectation mismatch inherent in nonlinear flow transforms: the spline is trained on the distribution of stochastic inputs, and its output at the deterministic mean can differ systematically from the average output over the distribution\. AtK=64K=64the residual variance of the MC estimate is negligible relative to hardware noise\. The*base policy*uses its deterministic mean action with the same EMA smoothing and no correction\.

## Appendix FAdditional Baselines

The comparisons in the main paper isolate the*correction architecture*under a frozen base policy\. We complement them with two standard baselines that adapt the*policy network itself*, placing Warp RL on a three\-way axis: full\-capacity parameter\-space adaptation \(FT\-KL\), low\-capacity parameter\-space adaptation \(FT\-Last\), and output\-space adaptation \(Warp\)\. Both baselines start from the same frozen base checkpoint and use the same shifted\-dynamics configurations, rollout budget, and PPO hyperparameters as the correction methods in Section[5\.1](https://arxiv.org/html/2606.31043#S5.SS1), so the comparison varies*where*adaptation occurs rather than the optimizer or training budget\. All methods are evaluated at their best held\-out checkpoint \(Appendix[B](https://arxiv.org/html/2606.31043#A2)\), and each trains a fresh value function on the shifted domain\.

#### KL\-regularized fine\-tuning \(FT\-KL\)\.

We continue PPO on the shifted dynamics from the source checkpoint, updating the entire Gaussian policy \(mean network and global log\-standard\-deviation\), with a KL penalty between the current policy and a frozen copy of the source policy \(coefficientβ=0\.1\\beta=0\.1, fixed a priori\)\. This penalty anchors the policy to the source and is distinct from PPO’s trust\-region limit, which constrains successive iterates\. Because the full network is trainable, FT\-KL adapts its own exploration through the learned standard deviations, unlike the frozen\-base corrections, which inherit the base policy’s exploration \(Appendix[D](https://arxiv.org/html/2606.31043#A4)\)\.

#### Last\-layer fine\-tuning \(FT\-Last\)\.

As a parameter\-efficient counterpart, we freeze the policy’s hidden layers and adapt only its final affine layer and exploration parameters, reusing the source\-trained features\. This trains far fewer parameters than full fine\-tuning and isolates adaptation to the action mapping\.

#### Scope\.

We exclude recent residual methods built for behavior\-cloned or diffusion base policies\[[4](https://arxiv.org/html/2606.31043#bib.bib30),[3](https://arxiv.org/html/2606.31043#bib.bib29),[8](https://arxiv.org/html/2606.31043#bib.bib7)\]: these assume a different base policy class than our PPO Gaussian base, and integrating warping into such pipelines is the extension noted in Section[7](https://arxiv.org/html/2606.31043#S7)\. Off\-policy residual correction is compared in the real\-robot experiments \(Section[5\.2](https://arxiv.org/html/2606.31043#S5.SS2)\); the simulation comparisons are kept on\-policy to fix the optimizer across methods\. For the parameter\-efficient baseline we adapt the final layer rather than a low\-rank adapter \(e\.g\. LoRA\[[14](https://arxiv.org/html/2606.31043#bib.bib2)\]\), whose low\-rank weight updates target the large matrices of foundation models and offer little benefit on a small policy MLP\.

Table 5:Additional baselines: success rate \(%\) under shifted dynamics, alongside the frozen\-base corrections from Table[1](https://arxiv.org/html/2606.31043#S5.T1)\. All methods report IQM \[95% CI\] over 5 training×\\times5 evaluation seeds\[[1](https://arxiv.org/html/2606.31043#bib.bib6)\]and are evaluated at their best held\-out checkpoint\. Best per task in bold\.
#### Results\.

Table[5](https://arxiv.org/html/2606.31043#A6.T5)reports the two adaptation baselines alongside the frozen\-base corrections, with the corresponding learning curves in Figure[7](https://arxiv.org/html/2606.31043#A6.F7)\. Full fine\-tuning is competitive but inconsistent: FT\-KL exceeds Residual\-PPO on three of four tasks, yet underperforms every frozen\-base correction — including the additive residual — on PushCube \(74\.1% vs\. 95\.0% for Residual\-PPO\), where the required adaptation is largely translational and a full\-network rewrite is unnecessary and harder to optimize under the shared budget\. The parameter\-efficient FT\-Last is similarly mixed, strong on PushCube but weak on the harder PegInsertionSide and PushT tasks\. Neither parameter\-space baseline dominates the lightweight corrections\. Warp\-ES, by contrast, achieves the best result on every task, outperforming both fine\-tuning baselines throughout despite adapting only a small, identity\-initialized correction module on top of a frozen policy\. It is also among the most reliable: on the harder tasks the parameter\-space baselines show wide confidence intervals \(e\.g\. FT\-Last on LiftPegUpright, \[62\.7, 85\.2\]\), whereas Warp\-ES remains comparatively tight\. This indicates that, in this setting, output\-space adaptation of a competent frozen base is more effective than retraining the policy network itself, whether at full or reduced capacity\.

![Refer to caption](https://arxiv.org/html/2606.31043v1/x5.png)Figure 7:Learning curves under shifted dynamics for the additional adaptation baselines \(FT\-KL, FT\-Last\) alongside the 3 frozen\-base corrections\. IQM of success rate across seeds; shaded==95% bootstrap CI; the dashed line marks the unadapted base policy under the shifted dynamics\. Reported values \(Table[5](https://arxiv.org/html/2606.31043#A6.T5)\) use the best held\-out checkpoint per method, which may differ from the final\-step value shown here\.