Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Exploration
Summary
This paper introduces LoPE, a training framework that uses prompt-space perturbations to address the zero-advantage problem in reinforcement learning with verifiable rewards, thereby enhancing reasoning exploration in large language models.
View Cached Full Text
Cached at: 05/08/26, 07:15 AM
Paper page - Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Exploration
Source: https://huggingface.co/papers/2605.05566
Abstract
LoPE addresses the zero-advantage problem in reinforcement learning with verifiable rewards by usingLorem Ipsum perturbations to enhance exploration in large language model training.
Reinforcement learningwith verifiable rewards, particularlyGroup Relative Policy Optimization(GRPO), has significantly advanced the reasoning capabilities ofLarge Language Models(LLMs). However, in complex tasks, GRPO frequently suffers from the ``zero-advantage problem’’: when all sampled rollouts for a query fail, the relative advantage collapses to zero. Consequently, the model loses effective training signals for these questions, wasting the training data and computational budget. While simply increasing the sampling budget for these questions is a common remedy, the static sampling policy inherently constrains reasoning exploration, limiting the success rate. In this paper, we propose Lorem Perturbation for Exploration (LoPE), a simple yet effective training framework to break this exploration bottleneck. We posit that task-irrelevantprompt-space perturbationscan shift the model’s output distribution enough to unlock orthogonal reasoning pathways for hard questions. Specifically, LoPE prepends sequences stochastically assembled fromLorem Ipsumvocabulary (a pseudo-Latin placeholder text) to the prompts beforeresampling. Experiments across 1.7B, 4B, and 7B models demonstrate that LoPE significantly outperformsresamplingwith the original prompts. Further analysis reveals that other Latin-based random sequences with lowperplexityare also effective perturbations. Our results establish LoPE as a strong baseline for broadening exploration in LLMreinforcement learning.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.05566
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.05566 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.05566 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.05566 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
This paper introduces LEEPS, a latent-guided explore-exploit prompt sampler for efficient reinforcement learning with verifiable rewards (RLVR) in LLMs. It adaptively balances reuse of informative prompts and exploration of uncertain ones, improving reasoning benchmark scores by 2.6-3.7% over baselines while adding only ~2 seconds of overhead per training step.
Lexical Perturbations Disrupt LLM Reasoning: An Empirical Study of Attention Diversion
This paper empirically studies how lexical perturbations disrupt large language model reasoning through attention diversion, finding that character-level noise significantly degrades performance while filler insertions have little effect.
Prompt-Driven Exploration
The paper introduces Prompt-Driven Exploration (PDE), a method that uses a vision-language model to iteratively refine natural language prompts for reinforcement learning policies, enabling global exploration and successful policy learning even from zero-reward starts.
OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
OPERA proposes a reinforcement learning method for open-ended tasks using intrinsic rewards based on perplexity dynamics, replacing unreliable LLM-as-a-judge reward models. It achieves state-of-the-art results on Qwen3-8B, matching proprietary models in creative writing and other open-ended tasks.
SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
Researchers propose SPS (Steering Probability Squeezing), a training paradigm combining reinforcement learning with inverse reinforcement learning to address probability squeezing in LLM reasoning training, where probability mass concentrates too narrowly on high-reward trajectories, limiting exploration and multi-sample performance (Pass@k). Experiments on five reasoning benchmarks demonstrate improved exploration and Pass@k metrics.