Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
Summary
NudgeRL is a framework that enhances reinforcement learning with verifiable rewards (RLVR) by introducing structured exploration and strategy nudging, achieving better reasoning performance in large language models more efficiently than brute-force scaling methods.
View Cached Full Text
Cached at: 05/18/26, 06:24 AM
Paper page - Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
Source: https://huggingface.co/papers/2605.15726
Abstract
NudgeRL framework enhances reinforcement learning with verifiable rewards through structured exploration and strategy nudging to improve reasoning capabilities in large language models.
Reinforcement learning with verifiable rewards(RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models. However, its effectiveness is fundamentally limited byexploration: the policy can only improve on trajectories it has already sampled. While increasing the number ofrolloutsalleviates this issue, such brute-force scaling is computationally expensive, and existing approaches that modify the optimization objective provide limited control over what is explored. In this work, we propose NudgeRL, a framework for structured and diversity-drivenexplorationin RLVR. Our approach introduces Strategy Nudging, which conditions each rollout on lightweight,strategy-level contextsto induce diverse reasoning trajectories without relying on expensiveoracle supervision. To effectively learn from such structuredexploration, we further propose a unified objective, which decomposes the reward signal into inter- and intra-context components and incorporates adistillation objectiveto transfer discovered behaviors back to the base policy. Empirically, NudgeRL outperforms standardGRPOwith up to 8 times larger rollout budgets, while outperforming oracle-guided RL baseline on average across five challenging math benchmarks. These results demonstrate that structured, context-drivenexplorationcan serve as an efficient and scalable alternative to both brute-force rollout scaling and feasibility-oriented methods based on privileged information. Our code is available at https://github.com/tally0818/NudgeRL.
View arXiv pageView PDFGitHub2Add to collection
Get this paper in your agent:
hf papers read 2605\.15726
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.15726 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.15726 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.15726 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
This paper introduces LEEPS, a latent-guided explore-exploit prompt sampler for efficient reinforcement learning with verifiable rewards (RLVR) in LLMs. It adaptively balances reuse of informative prompts and exploration of uncertain ones, improving reasoning benchmark scores by 2.6-3.7% over baselines while adding only ~2 seconds of overhead per training step.
Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR
The paper introduces a teacher-guided curriculum learning approach for Reinforcement Learning with Verifiable Rewards (RLVR) to efficiently train language models on initially unsolvable mathematical problems, achieving significant data efficiency and expanding reasoning boundaries.
Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
This paper proposes an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high, improving performance on text-based and GUI-based benchmarks.
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
This paper investigates how reinforcement learning with verifiable rewards (RLVR) narrows the solution space in LLM reasoning by analyzing where diversity is lost, finding it concentrated at the 'entrance' of trajectories. It demonstrates interventions to recover breadth without compromising accuracy.
ExpRL: Exploratory RL for LLM Mid-Training
ExpRL is a new RL-based mid-training method that uses human-written reference solutions as dense reward scaffolds (never shown to the policy) to improve LLM reasoning, achieving significant gains on hard math benchmarks like AIME-2026.