Drift Q-Learning
Summary
Proposes DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement for offline RL, outperforming diffusion and flow methods on D4RL and OGBench while maintaining simplicity and efficiency.
View Cached Full Text
Cached at: 06/02/26, 03:41 PM
# Drift Q-Learning Source: [https://arxiv.org/abs/2606.00350](https://arxiv.org/abs/2606.00350) [View PDF](https://arxiv.org/pdf/2606.00350) > Abstract:Offline reinforcement learning requires improving a policy from fixed data while avoiding out\-of\-distribution actions with unreliable value estimates\. Diffusion and flow policies handle this trade\-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference\. We propose DriftQL, which combines a drift\-based behavioral regularizer with critic\-driven policy improvement\. The value signal biases the policy toward high\-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode\. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass\. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art\. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean\-data performance, positioning it as a promising alternative to diffusion and flow\-based methods while maintaining the simplicity and efficiency of deterministic approaches\. Project page:[this https URL](https://driftql.github.io/) ## Submission history From: Mohamad H Danesh \[[view email](https://arxiv.org/show-email/b71ab235/2606.00350)\] **\[v1\]**Fri, 29 May 2026 20:42:30 UTC \(1,995 KB\)
Similar Articles
Reversal Q-Learning
This paper proposes Reversal Q-Learning (RQL), an offline reinforcement learning algorithm that trains a flow policy using an expanded Markov decision process framework and techniques to enable off-policy RL without backpropagation through time. It achieves state-of-the-art performance on challenging simulated robotic tasks.
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
QPILOTS is a method that steers flow policies at inference time by using critic gradients projected from noisy intermediate states, achieving state-of-the-art performance on offline-to-online RL benchmarks and improving pretrained VLA models without modifying the base policy.
@svlevine: Diffusion (or flow) makes for excellent policies, but training them with RL is notoriously hard: BPTT is unstable, RL o…
New paper shows how to optimize flow matching actors for reinforcement learning by approximating the Jacobian of the flow denoising process with the identity matrix, making training feasible.
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
This paper proposes DRIFT, a framework that combines offline trajectories with importance-weighted supervised fine-tuning to efficiently achieve multi-turn interactive learning performance comparable to reinforcement learning.
@svlevine: A new way to do off-policy RL with diffusion: if we have off-policy data, we need to figure out what the diffusion late…
A new method for off-policy reinforcement learning with diffusion models, using flow reversal to handle off-policy data by reversing the diffusion process on it.