From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments

arXiv cs.LG Papers
reinforcement-learning continuous-time stochastic-control actor-critic neural-networks stochastic-differential-equations deep-learning

Summary

This paper presents a theoretical framework for deep reinforcement learning in continuous environments, modeling it as a continuous-time stochastic process using stochastic control theory. The authors characterize an actor-critic algorithm's dynamics in the infinite width limit of two-layer networks, deriving an equation for infinitesimal changes in state distribution under a vanishingly small learning rate.

arXiv:2606.04275v1 Announce Type: new Abstract: We present a novel theoretical framework for deep reinforcement learning (RL) in continuous environments by modeling the problem as a continuous-time stochastic process, drawing on insights from stochastic control. Building on previous work, we introduce a viable model of actor-critic algorithm that incorporates both exploration and stochastic transitions. For single-hidden-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time. Within this formulation, we characterize how the time-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two-layer networks. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor-critic algorithms. We empirically corroborate our theoretical result using a toy continuous control task.
Original Article
View Cached Full Text

Cached at: 06/05/26, 02:24 AM 

# From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
Source: [https://arxiv.org/abs/2606.04275](https://arxiv.org/abs/2606.04275)
[View PDF](https://arxiv.org/pdf/2606.04275)

> Abstract:We present a novel theoretical framework for deep reinforcement learning \(RL\) in continuous environments by modeling the problem as a continuous\-time stochastic process, drawing on insights from stochastic control\. Building on previous work, we introduce a viable model of actor\-critic algorithm that incorporates both exploration and stochastic transitions\. For single\-hidden\-layer neural networks, we show that the state of the environment can be formulated as a two time scale process: the environment time and the gradient time\. Within this formulation, we characterize how the time\-dependent random variables that represent the environment's state and estimate of the cumulative discounted return evolve over gradient steps in the infinite width limit of two\-layer networks\. Using the theory of stochastic differential equations, we derive, for the first time in continuous RL, an equation describing the infinitesimal change in the state distribution at each gradient step, under a vanishingly small learning rate\. Overall, our work provides a novel nonparametric formulation for studying overparametrized neural actor\-critic algorithms\. We empirically corroborate our theoretical result using a toy continuous control task\.

## Submission history

From: Saket Tiwari \[[view email](https://arxiv.org/show-email/badea599/2606.04275)\] **\[v1\]**Tue, 2 Jun 2026 23:02:54 UTC \(1,617 KB\)

Similar Articles

Position: Deployed Reinforcement Learning should be Continual

arXiv cs.LG

This position paper argues that deployed RL agents should never stop learning, as the train-then-fix paradigm inherently fails to address non-stationarity and distribution shift in real-world environments. The authors identify four sources of post-deployment non-stationarity and advocate for continual RL as the standard approach for deployed systems.

Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints

arXiv cs.LG

Proposes LILAC+, a framework for safe continual reinforcement learning under nonstationarity that uses three adaptive safety mechanisms: context-based safety constraints, adaptation-speed constraints, and budget-to-state safety enforcement. Evaluations in simulated driving environments show reduced safety violations under distribution shift while maintaining competitive performance.

Towards Continuous-time Causal Foundation Models

arXiv cs.LG

Proposes a continuity criterion for extending discrete-time causal prior-data fitted networks to continuous time using stochastic differential equations, introducing a taxonomy and fine-grid integration method that outperforms naive integration on irregular observation schedules.

Path-Coupled Bellman Flows for Distributional Reinforcement Learning

arXiv cs.LG

This paper introduces Path-Coupled Bellman Flows (PCBF), a continuous-time distributional reinforcement learning method that uses flow matching to model return distributions without heuristic projections. It addresses boundary mismatch and high-variance issues in previous flow-based approaches by coupling current and successor return flows through shared base noise.