UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

arXiv cs.LG Papers
reinforcement-learning large-language-models optimization exploration-stability-dilemma importance-sampling clipping reasoning

Summary

This paper introduces Unbounded Positive Asymmetric Optimization (UP), a universal plug-and-play objective that resolves the exploration-stability dilemma in RL-based LLM training by anchoring the policy with stop-gradient, enabling unclipped gradients for positive advantages while clipping negative ones.

arXiv:2607.06987v1 Announce Type: new Abstract: Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs). To achieve sample efficiency, modern RL frameworks rely on importance sampling (IS). However, these algorithms suffer from an exploration-stability dilemma. Pure IS often leads to catastrophic training instability, while standard clipping mechanisms used to mitigate this instability strictly constrain the policy update budget. By formalizing the concept of Probability Capacity (Cap), we reveal that conservative clipping structurally stifles exploration by prematurely truncating the update budget for correct but low-confidence reasoning paths. To break free from these constraints, we propose Unbounded Positive Asymmetric Optimization (UP), a universal and plug-and-play objective. UP theoretically restructures the optimization process by anchoring the policy to its current state via the stop-gradient operator. This asymmetric design unleashes unclipped, stable gradients for positive advantages to maximize exploration, while maintaining standard clipping safeguards for negative advantages to prevent training instability. Furthermore, our formulation readily extends across different optimization granularities, including token-level (GRPO, DAPO) and sequence-level (GSPO) frameworks. Extensive experiments demonstrate that UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms (DAPO, GSPO, and GRPO), model architectures (Dense, MoE, and vision-language), and training modalities (language and multimodal), validating UP as a truly universal plug-and-play enhancement for RL-based training.
Original Article
View Cached Full Text

Cached at: 07/09/26, 07:46 AM 

# UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma
Source: [https://arxiv.org/abs/2607.06987](https://arxiv.org/abs/2607.06987)
[View PDF](https://arxiv.org/pdf/2607.06987)

> Abstract:Reinforcement learning \(RL\) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models \(LLMs\)\. To achieve sample efficiency, modern RL frameworks rely on importance sampling \(IS\)\. However, these algorithms suffer from an exploration\-stability dilemma\. Pure IS often leads to catastrophic training instability, while standard clipping mechanisms used to mitigate this instability strictly constrain the policy update budget\. By formalizing the concept of Probability Capacity \(Cap\), we reveal that conservative clipping structurally stifles exploration by prematurely truncating the update budget for correct but low\-confidence reasoning paths\. To break free from these constraints, we propose Unbounded Positive Asymmetric Optimization \(UP\), a universal and plug\-and\-play objective\. UP theoretically restructures the optimization process by anchoring the policy to its current state via the stop\-gradient operator\. This asymmetric design unleashes unclipped, stable gradients for positive advantages to maximize exploration, while maintaining standard clipping safeguards for negative advantages to prevent training instability\. Furthermore, our formulation readily extends across different optimization granularities, including token\-level \(GRPO, DAPO\) and sequence\-level \(GSPO\) frameworks\. Extensive experiments demonstrate that UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms \(DAPO, GSPO, and GRPO\), model architectures \(Dense, MoE, and vision\-language\), and training modalities \(language and multimodal\), validating UP as a truly universal plug\-and\-play enhancement for RL\-based training\.

## Submission history

From: Chongyu Fan \[[view email](https://arxiv.org/show-email/0063ed31/2607.06987)\] **\[v1\]**Wed, 8 Jul 2026 04:21:42 UTC \(1,700 KB\)

Similar Articles

Utility-Constrained Policy Optimization

arXiv cs.LG

This paper introduces a simple yet powerful methodology for Utility-Constrained MDPs (UCMDPs) that enables risk-sensitive constraints without fixing constraint limits in advance, outperforming baselines on Safety Gymnasium benchmarks.

Gradient Extrapolation-Based Policy Optimization

arXiv cs.LG

The article introduces Gradient Extrapolation-Based Policy Optimization (GXPO), a method that approximates multi-step lookahead in RL training for LLMs using only three backward passes. It demonstrates improved reasoning performance on math benchmarks over standard GRPO while maintaining fixed active-phase costs.

The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

Hugging Face Daily Papers

We introduce MIPI (Monotonic Inference Policy Improvement) and its instantiation MIPU, a two-step RL framework for LLMs that addresses the training-inference mismatch by explicitly aligning optimization with inference-policy improvement. Under FP8-quantized rollout, MIPU achieves improved reasoning performance and training stability across Qwen3-1.7B and Qwen3-4B models.