Trust Region Q Adjoint Matching
Summary
Trust Region Q-Adjoint Matching (TRQAM) addresses instability in off-policy reinforcement learning by adaptively controlling path-space KL divergence through projected dual descent, enabling stable fine-tuning of pretrained flow policies. The method consistently outperforms prior arts on 50 OGBench tasks, achieving a 68% success rate in offline RL compared to the strongest baseline's 46%.
View Cached Full Text
Cached at: 06/05/26, 06:09 PM
Paper page - Trust Region Q Adjoint Matching
Source: https://huggingface.co/papers/2605.27079
Abstract
Trust Region Q-Adjoint Matching (TRQAM) addresses instability in off-policy reinforcement learning by adaptively controlling path-space KL divergence through projected dual descent, enabling stable fine-tuning of pretrained flow policies.
Off-policy reinforcement learningofpretrained flow policiesremains challenging due to the instability of optimization arising from the multi-step sampling process. Recently,Q-learning with Adjoint Matching(QAM) addressed this issue by reformulating into a memorylessstochastic optimal control(SOC) problem with a learned critic. However, QAM inherits a fundamental fragility ofcritic-guided improvement: small critic errors are amplified when critics are ill-conditioned, often leading tomodel collapse. This paper introducesTrust Region Q-Adjoint Matching(TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL withpretrained flow policiesthroughprojected dual descent. Specifically, we optimize the trust-region parameter λ in SOC dynamics, and theoretically show that the path-space KL can be represented by a closed-form function of λ. As a result, our method can precisely control the exact deviation frompretrained flow policies, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior arts in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improves the strongest baseline at 46%.
View arXiv pageView PDFProject pageGitHub5Add to collection
Get this paper in your agent:
hf papers read 2605\.27079
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.27079 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.27079 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.27079 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
This paper introduces Trust Region Inverse Reinforcement Learning (TRIRL), a method that combines monotonic dual improvement with efficient local policy updates to outperform state-of-the-art imitation learning methods. It addresses the trade-off between stability and computational cost in IRL by using trust-region constraints.
Trust-Region Behavior Blending for On-Policy Distillation
Trust-Region behavior Blending (TRB) improves on-policy distillation by replacing poor early student rollouts with teacher-like behavior within a KL trust region during warmup, achieving stronger results on math-reasoning tasks.
Trust-Region Diffusion Policies for Massively Parallel On-Policy RL
Introduces TruDi, a method that enables training diffusion policies in massively parallel on-policy reinforcement learning by using a trust-region optimization rule to enforce KL constraints, achieving strong performance across 73 tasks.
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
This paper identifies a structural failure mode in sequential fine-tuning of shared-context multi-agent LLM teams, formalized as compounding occupancy shift, and proposes TeamTR, a trust-region framework that resamples trajectories and enforces per-agent divergence control, achieving 7.1% average improvement over baselines.
Reversal Q-Learning
This paper proposes Reversal Q-Learning (RQL), an offline reinforcement learning algorithm that trains a flow policy using an expanded Markov decision process framework and techniques to enable off-policy RL without backpropagation through time. It achieves state-of-the-art performance on challenging simulated robotic tasks.