REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement

# Learning Continuous Control Policies byIterative Action Refinement
Source: [https://arxiv.org/html/2609.21108](https://arxiv.org/html/2609.21108)
## RefinePPO: Learning Continuous Control Policies by Iterative Action Refinement

Sachini WeerasekaraAffiliation:Northeastern University, Boston, MA, USA\. \{weerasekara\.s, s\.kamarthi, j\.isaacs\}@northeastern\.eduSagar KamarthiAffiliation:Northeastern University, Boston, MA, USA\. \{weerasekara\.s, s\.kamarthi, j\.isaacs\}@northeastern\.eduJacqueline IsaacsAffiliation:Northeastern University, Boston, MA, USA\. \{weerasekara\.s, s\.kamarthi, j\.isaacs\}@northeastern\.edu

###### Abstract

Deep reinforcement learning \(DRL\) has achieved strong performance across a wide range of continuous\-control problems\. These continuous\-control policies, however, are often defined as direct mappings from an observed state to an action or action distribution, requiring a single feed\-forward network to construct an optimal control decision in one pass\. While effective, this formulation leaves little opportunity for the policy to reconsider or progressively improve an action once an initial prediction has been formed\. In this work, we explore an alternative approach: rather than learning only to directly predict an action, can a policy learn to iteratively improve one, and can this iterative process provide advantages during policy learning? We introduce*Iterative Action Refinement*\(IAR\), an iterative action\-construction method that constructs control actions through a sequence of learned residual corrections\. Starting from an initial proposal, a shared refinement network repeatedly conditions on the observed state and the current action proposal, allowing each refinement step to revise the action constructed by preceding steps\. The final refined proposal is then used to determine the action executed by the agent\. We integrate this iterative action\-construction mechanism with Proximal Policy Optimization \(PPO\), yieldingRefinePPO\. We evaluateRefinePPOacross 14 benchmark control tasks, complemented by controlled ablations of refinement depth and update schedules and analyses aimed at understanding why iterative refinement is effective\. Across these environments,RefinePPOmatches or exceeds the performance of standard PPO while demonstrating faster convergence on several tasks\.

## IIntroduction

Deep reinforcement learning has enabled solving increasingly complex continuous\-control problems in domains such as robotics\[[1](https://arxiv.org/html/2609.21108#bib.bib2),[2](https://arxiv.org/html/2609.21108#bib.bib3)\], and autonomous driving\[[3](https://arxiv.org/html/2609.21108#bib.bib4),[4](https://arxiv.org/html/2609.21108#bib.bib28)\]\. However, much of this progress has focused on improving how policies are optimized: developing more stable objectives\[[5](https://arxiv.org/html/2609.21108#bib.bib1)\], improving exploration\[[6](https://arxiv.org/html/2609.21108#bib.bib6)\], reducing variance\[[7](https://arxiv.org/html/2609.21108#bib.bib5)\], better generalization\[[8](https://arxiv.org/html/2609.21108#bib.bib29)\]and making better use of collected experience\[[9](https://arxiv.org/html/2609.21108#bib.bib7)\]\. Comparatively less attention has been given to a more basic question about the computation performed by the policy itself:*how should a neural policy construct an action from the current state?*

The dominant approach is straightforward\. Given an observation, a neural network predicts an optimal action or the parameters of an optimal action distribution in a single forward pass, which is then sent to the agent for execution\. Many successful continuous\-control methods share this design, which offers important practical advantages: it is simple, intuitive, and has already shown strong success across domains\. Proximal Policy Optimization \(PPO\)\[[5](https://arxiv.org/html/2609.21108#bib.bib1)\], for example, commonly represents a continuous policy as a Gaussian whose mean is predicted directly from the current state using a feedforward network, and the action is then sampled from the resulting distribution\[[10](https://arxiv.org/html/2609.21108#bib.bib21)\]\.

An alternative is to treat action generation not as a one\-shot prediction, but as an iterative process in which an initial action proposal is progressively improved before execution\. Rather than asking a network to produce the optimal action at once, the policy can learn a correction mechanism that repeatedly revises its current proposal based on both the observed state and what has already been constructed\.

This observation motivates the question we study in this work:*can continuous\-control policies benefit from learning how to improve an action proposal rather than only learning how to predict the optimal action directly?*\. Instead of requiring the policy to resolve the optimal control decision in one computation, we allow the policy to construct the optimal control policy progressively\. Each computation begins with the current proposal, determines how that proposal should be changed given the state, and passes the revised proposal to the next computation\. Such a process gives later computations access to what earlier computations have already constructed and turns action generation from a one\-shot prediction into a learned sequence of corrections\.

We introduce*Iterative Action Refinement*\(IAR\), an iterative action\-construction method based on this idea\. Starting from an initial action proposal,IARrepeatedly applies a shared residual network that conditions on both the observed state and the current proposal\. Given statessand proposalmkm\_\{k\}, the networkfθf\_\{\\theta\}predicts a correctionfθ​\(s,mk\)f\_\{\\theta\}\(s,m\_\{k\}\)and updates the proposal according to

mk\+1=mk\+ck\+1​fθ​\(s,mk\),m0=0m\_\{k\+1\}=m\_\{k\}\+c\_\{k\+1\}f\_\{\\theta\}\(s,m\_\{k\}\),\\qquad m\_\{0\}=0\(1\)forKKrefinement steps\. Each intermediatemkm\_\{k\}is therefore a latent action proposal rather than an action executed in the environment\. After the refinement process is complete, we construct the stochastic policy,

a∼𝒩⁡\(mK,diag⁡\(σ2\)\)a\\sim\\mathcal\{N\}\\left\(m\_\{K\},\\operatorname\{diag\}\(\\sigma^\{2\}\)\\right\)\(2\)
where𝒩\\mathcal\{N\}denotes a Gaussian distribution andσ\\sigmadenotes its vector of action\-wise standard deviations\. The final refined proposalmKm\_\{K\}serves as the Gaussian policy mean, while exploration retains the standard stochastic form used by the underlying continuous\-control policy\.

This alternative formulation provides several potential advantages for continuous\-control policies\. First, iterative refinement decomposes the state\-to\-action mapping into a sequence of conditional corrections, allowing the policy to progressively construct a control decision rather than requiring a single computation to produce the optimal action\. Second, because each refinement step conditions on the current action proposal, later computations can explicitly account for decisions made by earlier steps and adjust them in the context of the full action being constructed\. This self\-conditioning may be particularly useful in high\-dimensional control problems, where effective behavior requires coordination across multiple action dimensions\. Third, the refinement network is shared across steps, allowing the policy to perform additional computation without introducing a separate set of parameters for each refinement stage\. The refinement depthKKtherefore provides a direct mechanism for trading additional policy computation for progressively deeper action refinement\. Finally, weight sharing encourages the policy to learn a reusable correction rule that is applied across different intermediate action proposals, rather than associating each stage of computation with a separate transformation\. This may promote a more structured and generalizable action\-construction process, while potentially making useful control behaviors easier to learn and improving learning efficiency\.

We integrateIARwith PPO, yieldingRefinePPO\. The integration is deliberately minimal: refinement changes how the policy mean is computed, but does not alter the surrounding policy\-gradient algorithm\. The PPO clipped surrogate objective, probability ratio, critic, advantage estimator, and entropy formulation remain unchanged\. During policy evaluation, the actor performsKKdeterministic refinement steps before sampling the environment action; during optimization, gradients propagate through the complete refinement chain\. Consequently,RefinePPOintroduces iterative computation*within*each policy evaluation while PPO continues to perform policy optimization*across*collected experience\.

We evaluateRefinePPOon 14 continuous control tasks from classic control, Box2D and MuJoCo in Gymnasium complemented by controlled ablations of refinement depth, update schedules and analyses aimed at understanding why iterative refinement is effective\. We compare against standard PPO\[[5](https://arxiv.org/html/2609.21108#bib.bib1)\]\. Across the environments,RefinePPOachieves performance that is competitive with or exceeds the standard PPO baseline and exhibits faster convergence on several tasks\. These results indicate that iterative action construction can provide a useful alternative to conventional one\-pass action prediction, while also showing that the benefit depends on the choice of refinement depth and update dynamics\.

To summarize, our contributions are threefold\. First, we introduce Iterative Action Refinement \(IAR\), a simple iterative action\-construction mechanism in which a shared residual network progressively refines an action proposal\. Second, we characterize the refinement process within PPO asRefinePPOwithout modifying the underlying policy\-gradient objective\. Third, we provide an empirical evaluation across 14 benchmark continuous control tasks together with controlled ablations of refinement depth and update schedules and analyses aimed at understanding why iterative refinement is effective\. Taken together, these results motivate iterative action refinement as a simple but distinct computational bias for continuous\-control policies: rather than requiring a policy to construct its final action in a single pass, the actor can learn a reusable process for progressively improving its own action proposals\.

Remark: While we focus on PPO in this work,IARis not inherently tied to PPO\. ExtendingIARto other methods is a natural direction for future work\.

## IIRelated Work

In this section, we review prior work most closely related toIAR, including policy optimization for continuous control, residual reinforcement learning, iterative computation and learned optimization, and planning and optimization within policies\.

### II\-APolicy Optimization for Continuous Control

Policy\-gradient methods optimize expected return by directly differentiating a parameterized stochastic policy\[[11](https://arxiv.org/html/2609.21108#bib.bib13)\]\. Trust Region Policy Optimization \(TRPO\) stabilizes this process by constraining policy updates through a KL\-divergence trust region\[[12](https://arxiv.org/html/2609.21108#bib.bib12)\], while Proximal Policy Optimization \(PPO\) replaces the constrained optimization with a clipped surrogate objective that limits excessively large policy updates\[[5](https://arxiv.org/html/2609.21108#bib.bib1)\]\. PPO has consequently become a widely used baseline for continuous\-control and robotic locomotion tasks\. Our approach leaves the PPO objective and optimization procedure unchanged and instead modifies how the policy constructs the mean of its action distribution\.

Other continuous\-control methods, including DDPG\[lillicrap2015continuous\], TD3\[[13](https://arxiv.org/html/2609.21108#bib.bib14)\], and SAC\[[6](https://arxiv.org/html/2609.21108#bib.bib6)\], differ in their optimization objectives and exploration mechanisms but similarly rely on neural actors that directly construct actions or action\-distribution parameters from the current state\. In this work, we focus exclusively on integratingIARwith PPO in order to study the effect of iterative action refinement within a controlled policy\-optimization setting\. Investigating howIARcan be integrated with other policy\-gradient and actor–critic methods, and whether its benefits extend across different optimization frameworks, remains an important direction for future work\.

### II\-BResidual Reinforcement Learning

Residual learning has also been explored directly in reinforcement learning, particularly through residual policies that learn corrections to an existing controller or policy\. In residual reinforcement learning, the learned policy typically produces an additive correction to an action supplied by a fixed controller, allowing prior control knowledge to be combined with learned behavior\[[14](https://arxiv.org/html/2609.21108#bib.bib15),[15](https://arxiv.org/html/2609.21108#bib.bib16),[16](https://arxiv.org/html/2609.21108#bib.bib26),[17](https://arxiv.org/html/2609.21108#bib.bib27)\]and later work exploring more additions such as multi residual task learning\[[18](https://arxiv.org/html/2609.21108#bib.bib17)\]and mixture of experts in the context of residual reinforcement learning\[[19](https://arxiv.org/html/2609.21108#bib.bib18)\]\. AlthoughIARalso uses additive residual corrections, the role of the residual is fundamentally different\.IARdoes not correct the output of an external controller\. Instead, a single learned policy repeatedly corrects its own intermediate action proposal within one decision step\. The residual structure therefore operates*inside*the policy’s action\-construction process rather than between a learned policy and a pre\-existing controller\.

### II\-CIterative Computation and Learned Optimization

The iterative structure ofIARis related to methods that construct predictions through repeated updates\. Residual networks build representations through sequences of incremental transformations\[[20](https://arxiv.org/html/2609.21108#bib.bib8)\], while neural ODEs connect such residual updates to continuous\-time dynamics\[[21](https://arxiv.org/html/2609.21108#bib.bib19)\]\. Learned optimizers predict updates to candidate solutions rather than directly producing final solutions\[[22](https://arxiv.org/html/2609.21108#bib.bib9)\], and iterative amortized inference similarly improves an initial estimate through successive learned corrections\[[23](https://arxiv.org/html/2609.21108#bib.bib10)\]\. More recently, looped Transformers use weight\-tied iterations to progressively refine representations\[[24](https://arxiv.org/html/2609.21108#bib.bib22)\]\. These approaches motivate the broader principle underlyingIAR: a difficult prediction can be represented as a learned refinement process rather than a single direct mapping\.

Deep equilibrium models \(DEQs\) extend repeated weight\-tied computation by defining representations through fixed points of learned transformations\[[25](https://arxiv.org/html/2609.21108#bib.bib11),[26](https://arxiv.org/html/2609.21108#bib.bib25)\]\.IARinstead performs a finite, explicit number of refinement steps and backpropagates through the resulting computation normally\. Nevertheless, the fixed\-point perspective provides a useful interpretation of repeated action refinement and, under appropriate contraction conditions, of how successive proposals approach a solution\.

### II\-DPlanning and Optimization Within Policies

Iterative action construction is also related to approaches that perform planning or optimization as part of decision\-making\. Model\-predictive control and model\-based reinforcement learning methods can optimize candidate action sequences using a model of the system dynamics\[[21](https://arxiv.org/html/2609.21108#bib.bib19)\], while differentiable planning methods embed structured planning computations within trainable neural architectures\[[27](https://arxiv.org/html/2609.21108#bib.bib20),[28](https://arxiv.org/html/2609.21108#bib.bib23),[29](https://arxiv.org/html/2609.21108#bib.bib24)\]\.IARdiffers from these approaches in that it requires neither a dynamics model nor an explicit planning or action\-space optimization procedure at execution time\. Instead, the refinement rule is learned end\-to\-end through the policy objective, and inference consists of repeatedly applying the learned residual network to its current action proposal\. In this sense,IARcan be viewed as amortized iterative computation in action space: policy learning acquires a reusable update rule for constructing actions, rather than solving a new model\-based planning or optimization problem at every decision step\.

## IIIPreliminaries

We consider an infinite\-horizon discounted Markov decision process \(MDP\)ℳ=\(𝒮,𝒜,P,r,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma\), where𝒮\\mathcal\{S\}and𝒜⊆ℝda\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{d\_\{a\}\}denote the state and continuous action spaces, respectively,dad\_\{a\}is the action dimension,P⁡\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)is the transition kernel,r⁡\(s,a\)r\(s,a\)is the reward function, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. A stochastic policyπθ​\(a∣s\)\\pi\_\{\\theta\}\(a\\mid s\), parameterized byθ\\theta, induces a trajectoryτ=\(s0,a0,s1,a1,…\)\\tau=\(s\_\{0\},a\_\{0\},s\_\{1\},a\_\{1\},\\ldots\)and is optimized to maximize the expected discounted return

J⁡\(θ\)=𝔼τ∼πθ​\[∑t=0∞γt​r​\(st,at\)\]J\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\]\(3\)
For continuous actions, a standard PPO actor parameterizes the policy as a diagonal Gaussian,

πθ​\(a∣s\)=𝒩⁡\(a,μθ​\(s\),diag⁡\(σθ2\)\)\\pi\_\{\\theta\}\(a\\mid s\)=\\mathcal\{N\}\\left\(a;\\mu\_\{\\theta\}\(s\),\\operatorname\{diag\}\(\\sigma\_\{\\theta\}^\{2\}\)\\right\)\(4\)whereμθ​\(s\)∈ℝda\\mu\_\{\\theta\}\(s\)\\in\\mathbb\{R\}^\{d\_\{a\}\}is the state\-dependent mean produced by the actor network andσθ∈ℝ\>0da\\sigma\_\{\\theta\}\\in\\mathbb\{R\}\_\{\>0\}^\{d\_\{a\}\}is the vector of action\-wise standard deviations\. In the standard PPO parameterization considered here,μθ​\(s\)\\mu\_\{\\theta\}\(s\)is computed in a single forward pass, whilelog⁡σθ\\log\\sigma\_\{\\theta\}is a learned vector\.

For PPO, letθold\\theta\_\{\\mathrm\{old\}\}denote the behavior\-policy parameters andA^t\\hat\{A\}\_\{t\}a generalized advantage estimate\[[7](https://arxiv.org/html/2609.21108#bib.bib5)\]\. The likelihood ratio is

rt​\(θ\)=πθ​\(at∣st\)πθold​\(at∣st\)r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\)\}\(5\)and the clipped actor objective is

LCLIP​\(θ\)=𝔼t​\[min⁡\(rt​\(θ\)​A^t,clip⁡\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\]L^\{\\mathrm\{CLIP\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\\left\[\\min\\left\(r\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\},\\operatorname\{clip\}\(r\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{t\}\\right\)\\right\]\(6\)whereϵ\>0\\epsilon\>0denotes the PPO clipping parameter\.

## IVProblem Formulation

A standard continuous\-control actor constructs the policy mean through a direct mapping

μθ:𝒮→ℝda\\mu\_\{\\theta\}:\\mathcal\{S\}\\rightarrow\\mathbb\{R\}^\{d\_\{a\}\}\(7\)such that the complete action proposal is produced from the current state in a single computation\. We consider a more general formulation in which action construction is itself a sequential computation\.

For a fixed states∈𝒮s\\in\\mathcal\{S\}, letmk∈ℝdam\_\{k\}\\in\\mathbb\{R\}^\{d\_\{a\}\}denote an intermediate action proposal and consider a state\-conditioned transition operator

mk\+1=Gθ\(s,mk\),k=0,…,K−1m\_\{k\+1\}=G\_\{\\theta\}\(s,m\_\{k\}\),\\qquad k=0,\\ldots,K\-1\(8\)Starting from an initial proposalm0m\_\{0\}, repeated application ofGθG\_\{\\theta\}generates an internal trajectory in action space,

m0→m1→⋯→mKm\_\{0\}\\rightarrow m\_\{1\}\\rightarrow\\cdots\\rightarrow m\_\{K\}\(9\)The environment statessremains fixed throughout this internal computation, while the action proposal evolves\. Only the terminal proposal determines the policy mean,

μθ,K​\(s\)≜mK​\(s\)\\mu\_\{\\theta,K\}\(s\)\\triangleq m\_\{K\}\(s\)\(10\)whereas the intermediate proposals are latent computations and are never executed in the environment\.

This viewpoint casts action construction as a finite\-horizon dynamical system in action space\. The problem is then to learn dynamics that transform an initial proposal into a useful policy mean through successive state\-conditioned updates\. Equation \([8](https://arxiv.org/html/2609.21108#S4.E8)\) deliberately leaves the form of these dynamics unspecified\. We seek a simple realization that allows later computations to revise earlier action proposals, reuses the same learned transformation across steps, and can be incorporated into a stochastic continuous\-control policy without changing its underlying optimization objective\.

## VMethod: Iterative Action Refinement

![Refer to caption](https://arxiv.org/html/2609.21108v1/images/overview-v5.png)Fig\. 1:Overview ofRefinePPO\. Given the current environment statests\_\{t\}, the policy initializes an action proposalm0=𝟎m\_\{0\}=\\mathbf\{0\}and progressively refines it forKKsteps using the state\-conditioned residual networkfθf\_\{\\theta\}\. The statests\_\{t\}remains fixed throughout refinement, while each update conditions on the current action proposalmkm\_\{k\}\. The final proposalmKm\_\{K\}defines the mean of the Gaussian policy, from which a single actionata\_\{t\}is sampled and executed in the environment, producing the next statest\+1s\_\{t\+1\}\. Intermediate proposals are latent computations and are never executed in the environment\.### V\-AIterative Refinement Dynamics

We instantiate the action\-space dynamics in Eq\. \([8](https://arxiv.org/html/2609.21108#S4.E8)\) using*Iterative Action Refinement*\(IAR\)\. Given a statessand current proposalmkm\_\{k\}, a shared refinement networkfθf\_\{\\theta\}predicts a correction in action space\. Starting from the zero vectorm0=𝟎∈ℝdam\_\{0\}=\\mathbf\{0\}\\in\\mathbb\{R\}^\{d\_\{a\}\}, the proposal evolves according to

mk\+1=mk\+ck\+1fθ\(s,mk\),k=0,…,K−1m\_\{k\+1\}=m\_\{k\}\+c\_\{k\+1\}f\_\{\\theta\}\(s,m\_\{k\}\),\\qquad k=0,\\ldots,K\-1\(11\)where refinement schedulerck\+1\>0c\_\{k\+1\}\>0controls the magnitude of the corresponding update\. The same parametersθ\\thetaare shared across all refinement steps\.

AfterKKrefinements, the terminal proposal defines the mean of the stochastic policy,

πθ,K​\(a∣s\)=𝒩⁡\(a,mK​\(s\),diag⁡\(σθ2\)\)\\pi\_\{\\theta,K\}\(a\\mid s\)=\\mathcal\{N\}\\left\(a;m\_\{K\}\(s\),\\operatorname\{diag\}\(\\sigma\_\{\\theta\}^\{2\}\)\\right\)\(12\)Thus, the refinement trajectory is deterministic conditioned onssand the policy parameters, and stochasticity is introduced only after refinement through the final Gaussian policy\.

Under this parameterization, the abstract transition operator in Eq\. \([8](https://arxiv.org/html/2609.21108#S4.E8)\) takes the form

Gθ,k​\(s,m\)=m\+ck\+1​fθ​\(s,m\)G\_\{\\theta,k\}\(s,m\)=m\+c\_\{k\+1\}f\_\{\\theta\}\(s,m\)\(13\)Because each correction depends on the current proposal, later refinements explicitly condition on what earlier refinements have already constructed\.

###### Proposition 1\(One\-step refinement\)

ForK=1K=1,c1=1c\_\{1\}=1, andm0=𝟎∈ℝdam\_\{0\}=\\mathbf\{0\}\\in\\mathbb\{R\}^\{d\_\{a\}\}, theIARpolicy mean satisfiesμθ,1​\(s\)=fθ​\(s,𝟎\)\\mu\_\{\\theta,1\}\(s\)=f\_\{\\theta\}\(s,\\mathbf\{0\}\)\. Since the proposal input is fixed,μθ,1\\mu\_\{\\theta,1\}is functionally a direct mapping from state to action mean\.

Proof:Applying Eq\. \([11](https://arxiv.org/html/2609.21108#S5.E11)\) once givesm1=fθ​\(s,𝟎\)m\_\{1\}=f\_\{\\theta\}\(s,\\mathbf\{0\}\)Since the terminal proposal defines the policy mean,μθ,1​\(s\)=m1=fθ​\(s,𝟎\)\\mu\_\{\\theta,1\}\(s\)=m\_\{1\}=f\_\{\\theta\}\(s,\\mathbf\{0\}\), which depends only onss\.

Thus,K=1K=1contains no iterative refinement, whileK\>1K\>1allows each correction to condition on proposals generated by previous applications of the same network\.

### V\-BIntegration with PPO

IntegratingIARwith PPO requires only replacing the direct computation of the Gaussian mean with the refinement process\. During rollout, the actor computesmKm\_\{K\}and samples a single action from Eq\. \([12](https://arxiv.org/html/2609.21108#S5.E12)\)\. During a PPO update,mKm\_\{K\}is recomputed under the current parameters and the stored action is evaluated under the resulting policy\. The likelihood ratio is therefore

rt​\(θ\)=πθ,K​\(at∣st\)πθold,K​\(at∣st\)r\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta,K\}\(a\_\{t\}\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\},K\}\(a\_\{t\}\\mid s\_\{t\}\)\}\(14\)and is used directly in the standard PPO objective of Eq\. \([6](https://arxiv.org/html/2609.21108#S3.E6)\)\. The clipped objective, value loss, advantage estimator, and entropy term are otherwise unchanged\. The full algorithm is given in Algorithm[1](https://arxiv.org/html/2609.21108#alg1)\.

Algorithm 1RefinePPO0:State

ss, refinement depth

KK, coefficients

\{ck\}k=1K\\\{c\_\{k\}\\\}\_\{k=1\}^\{K\}, refinement network

fθf\_\{\\theta\}, log standard deviation

log⁡σθ\\log\\sigma\_\{\\theta\}
1:

m←𝟎m\\leftarrow\\mathbf\{0\}
2:for

k=1,…,Kk=1,\\ldots,Kdo

3:

m←m\+ck​fθ​\(s,m\)m\\leftarrow m\+c\_\{k\}f\_\{\\theta\}\(s,m\)
4:endfor

5:

πθ,K\(⋅∣s\)←𝒩\(m,diag\(σθ2\)\)\\pi\_\{\\theta,K\}\(\\cdot\\mid s\)\\leftarrow\\mathcal\{N\}\\\!\\left\(m,\\operatorname\{diag\}\(\\sigma\_\{\\theta\}^\{2\}\)\\right\)
6:iftrainingthen

7:Sample

a∼πθ,K\(⋅∣s\)a\\sim\\pi\_\{\\theta,K\}\(\\cdot\\mid s\)
8:Update

θ\\thetausing the standard PPO objective

9:else

10:Set

a←ma\\leftarrow m\{deterministic evaluation\}

11:endif

## VIWhy Iterative Action Refinement Can Work

In this section, we use a simple toy task to illustrate one potential utility ofIARand provide intuition for why it can be effective\. Specifically, to demonstrate the benefits of iterative refinement, consider a two\-dimensional actiona=\(a1,a2\)a=\(a\_\{1\},a\_\{2\}\)whose optimal components satisfy

a1⋆=ϕ⁡\(s\),a2⋆=ψ⁡\(s,a1⋆\)a\_\{1\}^\{\\star\}=\\phi\(s\),\\qquad a\_\{2\}^\{\\star\}=\\psi\(s,a\_\{1\}^\{\\star\}\)\(15\)Here, the appropriate value ofa2a\_\{2\}depends on the decision made fora1a\_\{1\}, representing a simple form of coordination between action dimensions\. Such dependencies arise naturally in real\-world tasks: in robotic manipulation, the appropriate gripper orientation may depend on the chosen end\-effector position; in autonomous driving, the appropriate steering angle may depend on the vehicle’s speed; and in locomotion, the placement of one foot may depend on the position and motion of the other limbs\.

A conventional PPO policy predicts both components directly from the state,

μθ​\(s\)=\[μθ,1​\(s\)μθ,2​\(s\)\]≈\[ϕ⁡\(s\)ψ⁡\(s,ϕ⁡\(s\)\)\]\\mu\_\{\\theta\}\(s\)=\\begin\{bmatrix\}\\mu\_\{\\theta,1\}\(s\)\\\\ \\mu\_\{\\theta,2\}\(s\)\\end\{bmatrix\}\\approx\\begin\{bmatrix\}\\phi\(s\)\\\\ \\psi\(s,\\phi\(s\)\)\\end\{bmatrix\}\(16\)Thus, although a sufficiently expressive network can represent the optimal policy, the dependency betweena1a\_\{1\}anda2a\_\{2\}must be learned implicitly within a single state\-to\-action computation, which may require more samples to reliably capture the coordination between action dimensions\.

In contrast,IARcan resolve this dependency progressively across refinement steps\. Recall that action generation always begins from the fixed initializationm0=𝟎m\_\{0\}=\\mathbf\{0\}\. ForK=2K=2, the first refinement receives\(s,𝟎\)\(s,\\mathbf\{0\}\)and can construct an intermediate action estimate such as

m1=m0\+fθ​\(s,m0\)=fθ​\(s,𝟎\)≈\[ϕ⁡\(s\)0\]m\_\{1\}=m\_\{0\}\+f\_\{\\theta\}\(s,m\_\{0\}\)=f\_\{\\theta\}\(s,\\mathbf\{0\}\)\\approx\\begin\{bmatrix\}\\phi\(s\)\\\\ 0\\end\{bmatrix\}\(17\)The second refinement then receives\(s,m1\)\(s,m\_\{1\}\)\. Unlike the first step, it therefore has explicit access to the intermediate decisionm1,1≈ϕ⁡\(s\)m\_\{1,1\}\\approx\\phi\(s\)and can use this information when refining the action:

m2=m1\+fθ​\(s,m1\)≈\[ϕ⁡\(s\)ψ⁡\(s,m1,1\)\]≈\[ϕ⁡\(s\)ψ⁡\(s,ϕ⁡\(s\)\)\]m\_\{2\}=m\_\{1\}\+f\_\{\\theta\}\(s,m\_\{1\}\)\\approx\\begin\{bmatrix\}\\phi\(s\)\\\\ \\psi\(s,m\_\{1,1\}\)\\end\{bmatrix\}\\approx\\begin\{bmatrix\}\\phi\(s\)\\\\ \\psi\(s,\\phi\(s\)\)\\end\{bmatrix\}\(18\)Thus, rather than requiring the compositionψ⁡\(s,ϕ⁡\(s\)\)\\psi\(s,\\phi\(s\)\)to be produced entirely within a single state\-to\-action computation, refinement provides a computational path in whichϕ⁡\(s\)\\phi\(s\)can first be represented in the intermediate action and then made directly available when constructing the dependent component\.

Importantly, this example does not imply thatIARis constrained to refine one action dimension at a time, nor that a conventional PPO policy cannot represent the same mapping\. Both policies may be sufficiently expressive to represent the optimal action\. The distinction is instead in the structure of the computation\. Later in experimental results, we will provide evidence that these behaviors are likely happening inIAR\.

## VIIExperimental Setup

### VII\-ABenchmarks and Baselines

We perform experiments on 14 continuous control tasks from classic control, Box2D and MuJoCo environments in Gymnasium\. The primary baseline is the standard PPO actor\[[5](https://arxiv.org/html/2609.21108#bib.bib1)\]\.

### VII\-BImplementation Details

We use CleanRL\[[10](https://arxiv.org/html/2609.21108#bib.bib21)\]implementation of continuous action PPO and further modify it to implementRefinePPO\. The baseline PPO implementation is same as the CleanRL implementation\. Unless otherwise noted, training uses 6 million environment steps for each envionrment\. We independently performed hyperparameter sweep for both baseline PPO andRefinePPO\. As fixed hyperparameters, the implementation uses orthogonal initialization withtanh\\tanhnonlinearities\. The critic contains two 64\-unit hidden layers\. The baseline actor contains two 128\-unit hidden layers, while the refinement field uses two 128\-unit hidden layers after concatenating the normalized observation and current action proposal\. All results are reported as average over 5 independent seeds\.

### VII\-CEvaluation Protocol

At fixed environment\-step intervals during training, we evaluate the deterministic policy usinga=mK​\(s\)a=m\_\{K\}\(s\)forRefinePPOanda=m⁡\(s\)a=m\(s\)for PPO, both without exploration noise\. For each training seed, we select the checkpoint achieving the highest evaluation return during training and use this checkpoint for final evaluation\. The selected checkpoint is then evaluated over a ten evaluation episodes, and the resulting returns are averaged to obtain the final performance for that seed\. We then average these per\-seed final returns across all training seeds and report the mean and standard deviation across seeds\.

### VII\-DRefinement Depth

The refinement depthKKdetermines the number of residual corrections applied before constructing the final action distribution inRefinePPO\. We considerK∈\{1,2,4,8\}K\\in\\\{1,2,4,8\\\}to evaluate the effect of increasing refinement depth on policy performance\. Here,K=1K=1represents the one\-step boundary case with no iterative refinement, while larger values ofKKprovide progressively more refinement steps\.

### VII\-ERefinement Schedules

The coefficients\{ck\}k=1K\\\{c\_\{k\}\\\}\_\{k=1\}^\{K\}in Eq\. \([11](https://arxiv.org/html/2609.21108#S5.E11)\) control the contribution of each residual update\. We consider three schedules:

Uniform:ck=1,Inverse:ck=1k,Average:ck=1K\\text\{Uniform: \}c\_\{k\}=1,\\quad\\text\{Inverse: \}c\_\{k\}=\\frac\{1\}\{k\},\\quad\\text\{Average: \}c\_\{k\}=\\frac\{1\}\{K\}\(19\)The uniform schedule applies every correction at full scale, whereas the inverse schedule progressively reduces later updates\. For the average schedule,∑k=1Kck=1\\sum\_\{k=1\}^\{K\}c\_\{k\}=1\.

## VIIIExperimental Results

TABLE I:Evaluation performance of PPO andRefinePPOacross 14 continuous\-control environments\. Results report the mean return±\\pmstandard deviation for the best\-performing hyperparameter configuration of each method\.\(a\)HalfCheetah\-v5\(b\)Pusher\-v5\(c\)Hopper\-v5\(d\)HumanoidStandup\-v5\(e\)Humanoid\-v5\(f\)Swimmer\-v5
Fig\. 2:Evaluation returns throughout training for PPO andRefinePPOacross six representative environments\. Solid lines show the mean evaluation return across five independent seeds, and shaded regions indicate variability across seeds\.We conduct experiments to evaluate the effectiveness ofRefinePPOand to better understand the design choices underlying iterative action refinement\. In particular, we aim to answer the following five research questions:

1. 1\.RQ1: Performance\.DoesRefinePPOachieve better performance than standard PPO?
2. 2\.RQ2: Sample Efficiency\.IsRefinePPOmore sample\-efficient than standard PPO?
3. 3\.RQ3: Mechanism\.What makesRefinePPOwork?
4. 4\.RQ4: Refinement Depth\.How does the refinement depthKKaffect performance?
5. 5\.RQ5: Refinement Schedule\.How does the iterative update schedule affect performance?

In the following sections, we take a deep dive on these\.

### VIII\-ARQ1: Performance

We first compare the final performance ofRefinePPOagainst standard PPO across the evaluation environments\. Table[I](https://arxiv.org/html/2609.21108#S8.T1)reports the mean evaluation return over five independent seeds\. Overall,RefinePPOoutperforms PPO in many of the environments while remaining competitive in the others, providing strong evidence for the utility ofRefinePPO\. The results also suggest that the benefits ofRefinePPOmay become more pronounced as the dimensionality of the action space increases\. One possible explanation is that higher\-dimensional action spaces make the policy to coordinate a larger number of action dimensions whichRefinePPOis better designed to handle\.

![Refer to caption](https://arxiv.org/html/2609.21108v1/action_component_settling.png)\(a\)Remaining refinement by action component\.Each heatmap entry\(j,k\)\(j,k\)shows the normalized remaining distanceek,je\_\{k,j\}between intermediate action componentmk,jm\_\{k,j\}and its terminal valuemK,jm\_\{K,j\}, as defined in Eq\.[20](https://arxiv.org/html/2609.21108#S8.E20)\. Smaller values indicate that the component is already closer to its terminal value\. Components are ordered such that those requiring the least subsequent refinement shown first\.
![Refer to caption](https://arxiv.org/html/2609.21108v1/early_to_late_refinement.png)\(b\)Earlier components inform later refinements\.Increase in five\-fold cross\-validatedR2R^\{2\}when earlier\-established components are added to a baseline that predicts the next correction using only the target component’s current value\. PositiveΔ​R2\\Delta R^\{2\}indicates additional predictive information from the earlier components\.

Fig\. 3:Empirical evidence for progressive action construction in a trainedIARpolicy on Humanoid Standup\.\(a\)Some action components require substantially less subsequent refinement than others\.\(b\)Intermediate values of earlier\-established components provide information about subsequent corrections to components that continue to be refined\. Together, the results are consistent with progressive, cross\-component action refinement\.
### VIII\-BRQ2: Sample Efficiency

We next examine how quickly the two methods learn as a function of environment interaction\. Figure[2](https://arxiv.org/html/2609.21108#S8.F2)shows the evaluation returns throughout training on six representative environments\.RefinePPOlearns more rapidly than PPO on all six representative environments, with the clearest improvement on Swimmer\-v5, where the performance gap emerges early and persists throughout training\. In contrast, Pusher\-v5 and Humanoid\-v5 exhibit similar learning dynamics forRefinePPOand PPO, with their learning curves largely tracking each other throughout training\. Overall, these results suggest thatRefinePPOcan improve sample efficiency in several environments while preserving learning efficiency comparable to PPO in others\.

### VIII\-CEmpirical Evidence for the Refinement Mechanism

We next examine whetherIARexhibits the progressive action\-construction behavior motivated with a example in Section[VI](https://arxiv.org/html/2609.21108#S6)\. In particular, we ask whether some action components become established earlier in the refinement trajectory, and whether these earlier components contain information useful for predicting subsequent refinements of other components\. We test this hypothesis with aRefinePPOtrained HumanoidStandup\-v5 rollout\.

#### When are action components established?

For each action dimensionjj, we measure its remaining distance from the terminal proposal after refinement stepkk:

ek,j=𝔼s​\[\|mK,j​\(s\)−mk,j​\(s\)\|\]𝔼s​\[\|mK,j​\(s\)\|\]\+ϵe\_\{k,j\}=\\frac\{\\mathbb\{E\}\_\{s\}\[\|m\_\{K,j\}\(s\)\-m\_\{k,j\}\(s\)\|\]\}\{\\mathbb\{E\}\_\{s\}\[\|m\_\{K,j\}\(s\)\|\]\+\\epsilon\}\(20\)We compute this quantity over 1,000 intermediate action trajectories from a trained Humanoid Standup policy\. A smallek,je\_\{k,j\}indicates that componentjjis already close to its terminal value at stepkk\. We order action dimensions bye1,je\_\{1,j\}, so that dimensions requiring the least subsequent refinement appear first\. Figure[3](https://arxiv.org/html/2609.21108#S8.F3)\(a\) shows that action components are not refined uniformly: some are already close to their terminal values after the first refinement, while others undergo substantially greater subsequent revision\.

#### Do earlier components inform later refinements?

We next test whether components that become established earlier contain information about how later\-refined components will subsequently change\. Using the ordering from previous analysis, we select the five earliest components and the five components with the most remaining refinement\. For each later refined componentjjand transitionk→k\+1k\\rightarrow k\+1, the prediction target is its next correction,

Δk\+1,j=mk\+1,j−mk,j\\Delta\_\{k\+1,j\}=m\_\{k\+1,j\}\-m\_\{k,j\}\(21\)
We compare two ridge regressions\. The baseline predictsΔk\+1,j\\Delta\_\{k\+1,j\}using only the component’s current valuemk,jm\_\{k,j\}, while the augmented model additionally receives the current values of the five earlier\-established components\. We evaluate both models using five\-fold cross\-validation and report

Δ​R2=Raugmented2−Rbaseline2\\Delta R^\{2\}=R^\{2\}\_\{\\mathrm\{augmented\}\}\-R^\{2\}\_\{\\mathrm\{baseline\}\}\(22\)
Thus, positiveΔ​R2\\Delta R^\{2\}indicates that earlier\-established action components provide predictive information about the subsequent refinement of componentjjbeyond its own current value\. Figure[3](https://arxiv.org/html/2609.21108#S8.F3)\(b\) reports this improvement for each later\-refined component and refinement transition\.

Together, the two analyses are consistent with progressive, coordinated action construction: some components require relatively little subsequent refinement, and their intermediate values contain information about how other components are refined at later steps\.

### VIII\-DRQ3: Refinement Depth

Fig\. 4:Refinement depthKKeffect on normalized return across environments\. Error bars indicate±1\\pm 1standard deviation\.We next study the effect of refinement depth usingK∈\{1,2,4,8\}K\\in\\\{1,2,4,8\\\}\. Figure[4](https://arxiv.org/html/2609.21108#S8.F4)reports the min–max normalized mean return across the evaluation environments, with error bars indicating one standard deviation across environments\. Performance generally improves as the refinement depth increases fromK=1K=1toK=4K=4, withK=4K=4achieving the highest average normalized return\. Importantly,K=1K=1corresponds to the non\-iterative boundary case: sincem0=𝟎m\_\{0\}=\\mathbf\{0\}, the policy directly maps the state and fixed initial proposal to an action\. The improvement forK\>1K\>1therefore suggests that repeatedly refining an action proposal can improve policy performance\.

The gains, however, do not increase monotonically with refinement depth, as performance slightly decreases fromK=4K=4toK=8K=8\. This suggests that a moderate number of refinement steps is sufficient to capture most of the benefit, while additional refinement provides diminishing returns\.

### VIII\-ERQ4: Refinement Schedule

Finally, we investigate how the iterative update schedule influences refinement\. Table[II](https://arxiv.org/html/2609.21108#S8.T2)compares the Uniform \(ck=1c\_\{k\}=1\), Inverse \(ck=1/kc\_\{k\}=1/k\), and Average \(ck=1/Kc\_\{k\}=1/K\) schedules\. We observe Average to be clearly providing the strongest overall performance across all the evaluated environments\.

The schedules control how strongly each refinement step can modify the current action proposal and therefore induce different refinement dynamics\. Uniform scaling allows every step to make a full residual correction, whereas the Inverse schedule progressively reduces the influence of later refinements\. The Average schedule distributes a fixed total update scale across allKKsteps\. The differences in performance indicate that simply performing repeated refinement is not sufficient, how corrections are accumulated also matters\.

TABLE II:Effect of refinement schedule onRefinePPOperformance across MuJoCo continuous\-control environments\. Results report the mean evaluation return for the Average, Uniform, and Inverse refinement schedules\.

## IXConclusion and Future Work

In this work, we introduced*Iterative Action Refinement*\(IAR\), an iterative action\-construction approach that allows a continuous\-control policy to progressively refine its action before execution\. Integrated with PPO,RefinePPOachieves improved or competitive performance across 14 continuous control tasks from classic control, Bos2D and MuJoCo environments, with faster learning in several environments\. Our analysis further suggests that later refinement steps use intermediate action proposals to coordinate and improve action components\. Overall, these results demonstrate that iterative action construction is a promising alternative to conventional one\-pass policy prediction\.

Future work can extendIARto other actor–critic algorithms such as SAC and TD3 and evaluate whether its benefits generalize beyond PPO\. Another promising direction is adaptive refinement, where the policy determines how many refinement steps are needed for each state\. Finally, evaluatingIARon more complex control problems, including real\-world robotic manipulations and other systems, could further establish the generality and practical value of iterative action refinement\. Such extensions would further clarify when iterative refinement is most beneficial and how it can be scaled to more challenging decision\-making settings\.

## References

- \[1\]\(2015\)Continuous control with deep reinforcement learning\.arXiv preprint arXiv:1509\.02971\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1)\.
- \[2\]S\. Levine, C\. Finn, T\. Darrell, and P\. Abbeel\(2016\)End\-to\-end training of deep visuomotor policies\.Journal of Machine Learning Research17\(39\),pp\. 1–40\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1)\.
- \[3\]B\. R\. Kiran, I\. Sobh, V\. Talpaert, P\. Mannion, A\. A\. Al Sallab, S\. Yogamani, and P\. Pérez\(2021\)Deep reinforcement learning for autonomous driving: a survey\.IEEE transactions on intelligent transportation systems23\(6\),pp\. 4909–4926\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1)\.
- \[4\]D\. Suo, V\. Jayawardana, and C\. Wu\(2024\)Model\-free learning of corridor clearance: a near\-term deployment perspective\.IEEE Transactions on Intelligent Transportation Systems\.External Links:[Document](https://dx.doi.org/10.1109/TITS.2023.3344473)Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1)\.
- \[5\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1),[§I](https://arxiv.org/html/2609.21108#S1.p2.1),[§I](https://arxiv.org/html/2609.21108#S1.p9.1),[§II\-A](https://arxiv.org/html/2609.21108#S2.SS1.p1.1),[§VII\-A](https://arxiv.org/html/2609.21108#S7.SS1.p1.1)\.
- \[6\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning,Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1),[§II\-A](https://arxiv.org/html/2609.21108#S2.SS1.p2.1)\.
- \[7\]J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. Abbeel\(2015\)High\-dimensional continuous control using generalized advantage estimation\.arXiv preprint arXiv:1506\.02438\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1),[§III](https://arxiv.org/html/2609.21108#S3.p3.1)\.
- \[8\]V\. Jayawardana, B\. Freydt, A\. Qu, C\. Hickert, Z\. Yan, and C\. Wu\(2025\)Intersectionzoo: eco\-driving for benchmarking multi\-agent contextual reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 75263–75289\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1)\.
- \[9\]T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver\(2015\)Prioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p1.1)\.
- \[10\]S\. Huang, R\. F\. J\. Dossa, C\. Ye, J\. Braga, D\. Chakraborty, K\. Mehta, and J\. G\. AraÚjo\(2022\)Cleanrl: high\-quality single\-file implementations of deep reinforcement learning algorithms\.Journal of Machine Learning Research23\(274\),pp\. 1–18\.Cited by:[§I](https://arxiv.org/html/2609.21108#S1.p2.1),[§VII\-B](https://arxiv.org/html/2609.21108#S7.SS2.p1.1)\.
- \[11\]R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour\(1999\)Policy gradient methods for reinforcement learning with function approximation\.Advances in neural information processing systems12\.Cited by:[§II\-A](https://arxiv.org/html/2609.21108#S2.SS1.p1.1)\.
- \[12\]J\. Schulman, S\. Levine, P\. Abbeel, M\. Jordan, and P\. Moritz\(2015\)Trust region policy optimization\.InInternational conference on machine learning,pp\. 1889–1897\.Cited by:[§II\-A](https://arxiv.org/html/2609.21108#S2.SS1.p1.1)\.
- \[13\]S\. Fujimoto, H\. Hoof, and D\. Meger\(2018\)Addressing function approximation error in actor\-critic methods\.InInternational conference on machine learning,pp\. 1587–1596\.Cited by:[§II\-A](https://arxiv.org/html/2609.21108#S2.SS1.p2.1)\.
- \[14\]T\. Johannink, S\. Bahl, A\. Nair, J\. Luo, A\. Kumar, M\. Loskyll, J\. A\. Ojea, E\. Solowjow, and S\. Levine\(2019\)Residual reinforcement learning for robot control\.In2019 international conference on robotics and automation \(ICRA\),pp\. 6023–6029\.Cited by:[§II\-B](https://arxiv.org/html/2609.21108#S2.SS2.p1.1)\.
- \[15\]T\. Silver, K\. Allen, J\. Tenenbaum, and L\. Kaelbling\(2018\)Residual policy learning\.arXiv preprint arXiv:1812\.06298\.Cited by:[§II\-B](https://arxiv.org/html/2609.21108#S2.SS2.p1.1)\.
- \[16\]S\. Weerasekara, N\. Darras, S\. Kamarthi, C\. Price, and J\. Isaacs\(2026\)Prototype guided post\-pretraining for single\-cell representation learning\.arXiv preprint arXiv:2605\.07938\.Cited by:[§II\-B](https://arxiv.org/html/2609.21108#S2.SS2.p1.1)\.
- \[17\]S\. Weerasekara, N\. Darras, N\. Fernandez, M\. Chen, A\. Ainbinder, and C\. Price\(2025\)CellClique: dissecting tumor microenvironments at the single cell level using generative ai and spatial transcriptomics\.Cancer Research85\(8\_Supplement\_1\),pp\. 2418–2418\.Cited by:[§II\-B](https://arxiv.org/html/2609.21108#S2.SS2.p1.1)\.
- \[18\]V\. Jayawardana, S\. Li, C\. Wu, Y\. Farid, and K\. Oguchi\(2024\)Generalizing cooperative eco\-driving via multi\-residual task learning\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),Cited by:[§II\-B](https://arxiv.org/html/2609.21108#S2.SS2.p1.1)\.
- \[19\]V\. Jayawardana, S\. Li, Y\. Farid, and C\. Wu\(2025\)Multi\-residual mixture of experts learning for cooperative control in multi\-vehicle systems\.arXiv preprint arXiv:2507\.09836\.Cited by:[§II\-B](https://arxiv.org/html/2609.21108#S2.SS2.p1.1)\.
- \[20\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p1.1)\.
- \[21\]R\. T\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. K\. Duvenaud\(2018\)Neural ordinary differential equations\.Advances in neural information processing systems31\.Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p1.1),[§II\-D](https://arxiv.org/html/2609.21108#S2.SS4.p1.1)\.
- \[22\]M\. Andrychowicz, M\. Denil, S\. Gomez, M\. W\. Hoffman, D\. Pfau, T\. Schaul, B\. Shillingford, and N\. De Freitas\(2016\)Learning to learn by gradient descent by gradient descent\.Advances in neural information processing systems29\.Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p1.1)\.
- \[23\]J\. Marino, Y\. Yue, and S\. Mandt\(2018\)Iterative amortized inference\.InInternational Conference on Machine Learning,Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p1.1)\.
- \[24\]A\. Giannou, S\. Rajput, J\. Sohn, K\. Lee, J\. D\. Lee, and D\. Papailiopoulos\(2023\)Looped transformers as programmable computers\.InInternational Conference on Machine Learning,pp\. 11398–11442\.Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p1.1)\.
- \[25\]S\. Bai, J\. Z\. Kolter, and V\. Koltun\(2019\)Deep equilibrium models\.Advances in neural information processing systems32\.Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p2.1)\.
- \[26\]S\. Weerasekara, W\. Li, J\. Isaacs, and S\. Kamarthi\(2025\)Improvements to disassembly lot sizing with task control through reinforcement learning\.Journal of Advanced Manufacturing and Processing7\(4\),pp\. e70032\.Cited by:[§II\-C](https://arxiv.org/html/2609.21108#S2.SS3.p2.1)\.
- \[27\]A\. Tamar, Y\. Wu, G\. Thomas, S\. Levine, and P\. Abbeel\(2016\)Value iteration networks\.Advances in neural information processing systems\.Cited by:[§II\-D](https://arxiv.org/html/2609.21108#S2.SS4.p1.1)\.
- \[28\]S\. Weerasekara, Z\. Lu, B\. Ozek, J\. Isaacs, and S\. Kamarthi\(2022\)Trends in adopting industry 4\.0 for asset life cycle management for sustainability: a keyword co\-occurrence network review and analysis\.Sustainability14\(19\),pp\. 12233\.Cited by:[§II\-D](https://arxiv.org/html/2609.21108#S2.SS4.p1.1)\.
- \[29\]S\. Weerasekara, W\. Li, J\. Isaacs, and S\. Kamarthi\(2024\)Reinforcement learning for disassembly task control\.Computers & Industrial Engineering190,pp\. 110044\.Cited by:[§II\-D](https://arxiv.org/html/2609.21108#S2.SS4.p1.1)\.