LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

# LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
Source: [https://arxiv.org/html/2607.28077](https://arxiv.org/html/2607.28077)
###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals\. Pre\-rollout prompt selection can reduce this waste by screening prompts before rollout generation\. However, existing pre\-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts\. To address these limitations, we introduce LEEPS, a Latent\-Guided Explore–Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones\. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non\-trivial ratios\. It further uses representation\-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non\-zero reward variance, thereby making exploration more targeted without additional rollouts\. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2\.6% and 3\.7% over the strongest baseline for Qwen2\.5\-Math\-1\.5B and 7B, respectively, and generally improves faster during the training process\. It also achieves the highest average score across the three evaluated OOD general\-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step\. Code is available athttps://github\.com/ShuangLiangX/LEEPS\.

## Introduction

Reinforcement learning has become a key post\-training stage for improving the reasoning capabilities of Large Language Models \(LLMs\)\(Lambertet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib1); Guoet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib30); Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12)\)\. Recent systems increasingly adopt outcome\-level reinforcement learning with verifiable rewards \(RLVR\) for reasoning tasks\(Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12); Wenet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib2); Xieet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib3); Huet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib4)\), often optimized with Group Relative Policy Optimization \(GRPO\)\(Guoet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib30)\)\. This paradigm enables models to learn from automatically checkable final answers, reducing reliance on preference\-based reward modeling\(Ouyanget al\.[2022](https://arxiv.org/html/2607.28077#bib.bib26)\)and step\-level process supervision\(Uesatoet al\.[2022](https://arxiv.org/html/2607.28077#bib.bib5); Lightmanet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib6); Wanget al\.[2024a](https://arxiv.org/html/2607.28077#bib.bib7)\)\. However, the combination of sparse outcome rewards and group\-relative advantage estimation introduces a structural inefficiency: trivial \(either too easy or too hard\) prompts often produce rollout groups with identical rewards, where all responses are either correct or incorrect\(Zhenget al\.[2026](https://arxiv.org/html/2607.28077#bib.bib15)\)\. These zero\-variance prompt groups yield vanishing signals for the model updates, yet still require expensive generation, making uninformative rollouts a major source of computational waste in RLVR training\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x1.png)Figure 1:Training dynamics on Qwen2\.5\-Math\-7B\. Left: the non\-trivial prompt ratio over 180 training steps\. Right: the complementary cumulative distribution of prompt selection counts at step 180, showing the fraction of training prompts selected at leastkktimes\.Existing methods aim to increase the fraction of non\-trivial prompts by selecting prompts\. Online filtering methods adopt an observe\-then\-filter strategy: they first generate rollouts for sampled prompts and then discard zero\-variance groups\(Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12); Zhanget al\.[2025b](https://arxiv.org/html/2607.28077#bib.bib14); Xuet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib25)\)\. Despite their effectiveness, these methods still spend substantial computation on prompts that ultimately yield zero\-variance groups\. More recent methods move prompt selection before rollout by predicting prompt informativeness from rollout history\. Specifically, DPS\(Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\)models prompt\-solving dynamics to prioritize prompts likely to be informative, whereas MoPPS\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18)\)uses posterior sampling over prompt success rates to favor prompts of intermediate predicted difficulty\. Although pre\-rollout selection avoids the additional rollouts required by online filtering, it introduces new challenges\. As shown in Figure[1](https://arxiv.org/html/2607.28077#Sx1.F1), DPS maintains a high non\-trivial ratio by repeatedly selecting a narrow prompt subset, leaving potentially useful training data unused, whereas MoPPS achieves broader coverage but yields a lower non\-trivial ratio, particularly early in training\. This setting gives rise to two core challenges\. First, exploration itself can introduce zero\-variance groups and thereby reduce the non\-trivial ratio of the training batch\. Second, under a fixed exploration budget, the sampler must identify potentially useful prompts more effectively without additional rollout feedback, thereby reducing the cost of exploration\.

To address these challenges, we proposeLEEPS, a latent\-guided explore–exploit prompt sampler for efficient RLVR\. As shown in Figure[1](https://arxiv.org/html/2607.28077#Sx1.F1), LEEPS maintains a consistently high non\-trivial prompt ratio while covering substantially more prompts than DPS, thereby balancing training\-batch informativeness with the exploration of potentially useful prompts\. Specifically, LEEPS builds on two complementary designs:Adaptive Explore–Exploit Portfolio AllocationandLatent\-Guided Exploration\.

First, Adaptive Explore–Exploit Portfolio Allocation mitigates the reduction in the training\-batch non\-trivial ratio caused by exploration\. It partitions candidates into an exploit portfolio of prompts that have produced informative rollout signals and an explore portfolio of prompts whose utility remains uncertain\. Using recent rollout outcomes, LEEPS estimates the non\-trivial ratio produced by each portfolio and adjusts their batch quotas so that the expected non\-trivial ratio of the selected training batch remains close to a predefined target\. This adaptive allocation maintains a high non\-trivial ratio while preserving sufficient budget for exploration\.

Second, Latent\-Guided Exploration improves the efficiency of exploring uncertain prompts without additional rollouts\. It represents prompts using model hidden states and estimates the potential success rate of each exploration candidate from the observed outcomes of its latent nearest neighbors\. Figure[3](https://arxiv.org/html/2607.28077#Sx3.F3)shows that latent information provides a useful signal for screening uncertain prompts, supporting our latent\-guided exploration strategy\. This allows the fixed exploration budget to prioritize uncertain prompts that are more likely to yield non\-zero reward variance\.

Finally, we evaluate LEEPS on two Qwen2\.5\-Math model scales across six mathematical reasoning benchmarks\. Without requiring additional rollouts for prompt selection, LEEPS outperforms the strongest baseline in overall score by 2\.6% and 3\.7% for the 1\.5B and 7B models, respectively\. It also achieves the highest average score across three OOD general\-reasoning benchmarks at both model scales, indicating that its gains extend beyond mathematical reasoning tasks\. Further analyses show that LEEPS maintains a high non\-trivial ratio, generally improves faster than the baselines, and incurs only approximately 2 seconds of online sampling overhead per training step\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x2.png)Figure 2:Overview of LEEPS\. Candidate prompts are divided into exploit and explore portfolios, and the training budget is adaptively allocated according to their recent prompt non\-trivial ratio\. The exploit branch weights previously observed non\-trivial prompts by their success rates, whereas the explore branch uses latent\-neighbor estimates to identify and prioritize potentially informative prompts\. Rollout outcomes update the prompt statistics for subsequent prompt sampling\.
## Preliminaries

Zero\-Variance Prompt Groups in RLVR\.Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib31)\)is widely used for reinforcement learning with verifiable rewards \(RLVR\)\. Given a promptpp, the old policy samples a group ofGGresponses\{yp,j\}j=1G\\\{y\_\{p,j\}\\\}\_\{j=1\}^\{G\}\. We use a binary verifiable rewardR​\(p,yp,j\)∈\{0,1\}R\(p,y\_\{p,j\}\)\\in\\\{0,1\\\}, where a correct final answer receives reward11and an incorrect answer receives reward0\. GRPO computes the group\-relative advantage by normalizing each reward within its rollout group:

Ap,j=R​\(p,yp,j\)−R¯pstd⁡\(𝐑p\),A\_\{p,j\}=\\frac\{R\(p,y\_\{p,j\}\)\-\\bar\{R\}\_\{p\}\}\{\\operatorname\{std\}\(\\mathbf\{R\}\_\{p\}\)\},\(1\)
We summarize the rollout outcomes of promptppby its group success rate:

a​\(p\)=1G​∑j=1GR​\(p,yp,j\)\.a\(p\)=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\(p,y\_\{p,j\}\)\.\(2\)Whena​\(p\)=0a\(p\)=0ora​\(p\)=1a\(p\)=1, all responses in the group receive the same reward, soAp,j=0A\_\{p,j\}=0for every response\. We refer to the sampled rollout group as a*zero\-variance group*and regardppas a*trivial prompt*at that training step\. In contrast, a prompt group is*non\-trivial*if0<a​\(p\)<10<a\(p\)<1, meaning that it contains both correct and incorrect responses\. The non\-trivial ratio of a prompt batch is the fraction of its prompt groups that satisfy this condition\.

Problem Definition\.At each training step, the dataloader randomly samples a large candidate batchℬcand\\mathcal\{B\}\_\{\\mathrm\{cand\}\}with\|ℬcand\|\>B\|\\mathcal\{B\}\_\{\\mathrm\{cand\}\}\|\>Bfollowing prior work\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18); Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\), whereBBis the number of prompts used for each GRPO update\. Before generating rollouts, a pre\-rollout selector uses only information available at selection time to choose a subsetℬtrain⊂ℬcand\\mathcal\{B\}\_\{\\mathrm\{train\}\}\\subset\\mathcal\{B\}\_\{\\mathrm\{cand\}\}with\|ℬtrain\|=B\|\\mathcal\{B\}\_\{\\mathrm\{train\}\}\|=B\. Only the selected prompts are rolled out and used for further policy optimization\. The goal is to actively explore prompts whose utility remains uncertain while maintaining a high non\-trivial ratio in the selected training batch, without generating additional rollouts for selection\.

## Methods

This section presents LEEPS, a latent\-guided explore\-exploit prompt sampler for efficient RLVR\. As shown in Figure[2](https://arxiv.org/html/2607.28077#Sx1.F2), given a candidate batchℬcand\\mathcal\{B\}\_\{\\mathrm\{cand\}\}, LEEPS selects a training batchℬtrain\\mathcal\{B\}\_\{\\mathrm\{train\}\}through two components: Adaptive Explore–Exploit Portfolio Allocation, which distributes the rollout budget between the exploit and explore portfolios, and Latent\-Guided Exploration, which uses latent\-neighbor information to guide sampling within the explore portfolio\. After the selected prompts are rolled out, LEEPS updates their success rates and selection counts using the observed outcomes, providing historical signals for the next selection step\.

### Adaptive Explore–Exploit Portfolio Allocation

To maintain a high non\-trivial ratio without sacrificing broad exploration, we first introduce Adaptive Explore–Exploit Portfolio Allocation\. As shown in Figure[1](https://arxiv.org/html/2607.28077#Sx1.F1), MoPPS\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18)\)explores a broad portion of the training set but yields a lower non\-trivial prompt\-group ratio early in training\. In contrast, DPS\(Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\)maintains a high non\-trivial ratio by concentrating on prompts estimated to be useful, but consequently reuses a much smaller subset of prompts, leaving potentially non\-trivial prompts unexplored\. To balance the non\-trivial ratio and prompt coverage, LEEPS formulates pre\-rollout prompt selection as the adaptive allocation of a fixed rollout budget between two complementary prompt portfolios\.

Given an enlarged candidate batchℬcand\\mathcal\{B\}\_\{\\mathrm\{cand\}\}, LEEPS partitions the candidate prompts based on their latest observed group success ratesa^p\\hat\{a\}\_\{p\}and the counts they have been selected for rollout,npn\_\{p\}\. The exploit portfolio contains previously selected non\-trivial prompts, whereas the explore portfolio contains unseen prompts and prompts whose latest rollout groups are zero\-variance\. LEEPS then constructsℬtrain\\mathcal\{B\}\_\{\\mathrm\{train\}\}by adaptively allocating prompts from the two portfolios according to their recent non\-trivial ratios\.

Exploit portfolio\.The exploit portfolio contains prompts that have already produced non\-zero reward variance:

𝒫exploit=\{p∣np\>0,0<a^p<1\}\.\\mathcal\{P\}\_\{\\mathrm\{exploit\}\}=\\\{p\\mid n\_\{p\}\>0,\\;0<\\hat\{a\}\_\{p\}<1\\\}\.\(3\)These prompts are likely to provide useful learning signals because the current policy sometimes solves them and sometimes fails\. LEEPS performs weighted sampling from the exploit portfolio, assigning each prompt a sampling probability proportional to its Bernoulli\-variance weight:

wexploit​\(p\)=a^p​\(1−a^p\),w\_\{\\mathrm\{exploit\}\}\(p\)=\\hat\{a\}\_\{p\}\(1\-\\hat\{a\}\_\{p\}\),\(4\)which is maximized ata^p=0\.5\\hat\{a\}\_\{p\}=0\.5and decreases as the prompt becomes either easier or harder\. This weighting therefore favors prompts of intermediate difficulty, consistent with prior findings that these prompts are most informative for model learning\(Baeet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib11); Chenet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib20); Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18)\)\.

Explore portfolio\.The explore portfolio contains prompts whose utility remains uncertain under the current policy\. It includes both unseen prompts and prompts whose latest rollout outcomes are zero\-variance:

𝒫explore=\{p∣np=0\}∪\{p∣np\>0,a^p∈\{0,1\}\}\.\\mathcal\{P\}\_\{\\mathrm\{explore\}\}=\\\{p\\mid n\_\{p\}=0\\\}\\cup\\\{p\\mid n\_\{p\}\>0,\\;\\hat\{a\}\_\{p\}\\in\\\{0,1\\\}\\\}\.\(5\)Keeping these prompts in the explore portfolio prevents the sampler from prematurely discarding prompts after limited or uninformative observations\. This is important because a prompt that is zero\-variance at one stage of training may later become useful as the policy evolves\. LEEPS does not sample this portfolio randomly; instead, it prioritizes exploring candidates using a latent\-neighbor estimate of their potential success rate, which is detailed in the next subsection\. Leta^plat\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}denote this latent\-neighbor estimated success rate\. LEEPS assigns each explore prompt a count\-decayed latent uncertainty score and selects the highest\-scoring prompts:

wexplore​\(p\)=1−2​\|a^plat−0\.5\|np\+1,w\_\{\\mathrm\{explore\}\}\(p\)=\\frac\{1\-2\|\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}\-0\.5\|\}\{\\sqrt\{n\_\{p\}\+1\}\},\(6\)which favors prompts whose latent\-neighbor estimate suggests intermediate difficulty, and discourages repeatedly exploring the same prompts\.

Adaptive allocation\.LEEPS aims to keep the non\-trivial ratio of the selected training batch close to a predefined targetτ\\tau\(e\.g\.,0\.90\.9\)\. To achieve this target, LEEPS controls the explore–exploit allocation throughρ\\rho, the fraction of the batch assigned to the explore portfolio\. Specifically, approximatelyρ​B\\rho Bprompts are selected from the explore portfolio and\(1−ρ\)​B\(1\-\\rho\)Bfrom the exploit portfolio to formℬtrain\\mathcal\{B\}\_\{\\mathrm\{train\}\}\. To determineρ\\rho, LEEPS uses the recent non\-trivial ratios of prompts selected from the two portfolios, denoted byqexploreq\_\{\\mathrm\{explore\}\}andqexploitq\_\{\\mathrm\{exploit\}\}\. The estimated non\-trivial ratio of the resulting training batch is

ρ​qexplore\+\(1−ρ\)​qexploit,\\rho q\_\{\\mathrm\{explore\}\}\+\(1\-\\rho\)q\_\{\\mathrm\{exploit\}\},\(7\)and LEEPS choosesρ\\rhosuch that this estimate is as close as possible to the targetτ\\tau\. After the selected prompts are rolled out, their outcomes are used to updateqexploreq\_\{\\mathrm\{explore\}\}andqexploitq\_\{\\mathrm\{exploit\}\}, which determine the allocation ratio for the next training step\. For stability,ρ\\rhois clipped to a predefined range\.

### Latent\-Guided Exploration

To improve the efficiency of exploring prompts with uncertain utility, LEEPS introduces Latent\-Guided Exploration\. Unseen and zero\-variance prompts provide limited direct evidence about whether they will yield useful reward variation in subsequent rollouts, while random exploration treats all such prompts equally\. Prior work has primarily examined whether question\-only model representations can predict single\-response correctness under fixed inference settings\(Cencerradoet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib44); Zhanget al\.[2025a](https://arxiv.org/html/2607.28077#bib.bib48)\)\. LEEPS instead focuses on identifying prompts likely to yield non\-trivial rollout groups as the policy evolves\. Specifically, LEEPS extends prompt\-level prediction to local representation\-space neighborhoods, using the observed rollout behavior of nearby prompts to guide the exploration of uncertain candidates without additional rollouts\.

Figure[3](https://arxiv.org/html/2607.28077#Sx3.F3)supports this design: representing prompts with model hidden states and aggregating the success rates of their nearest neighbors provides a useful signal for identifying prompts likely to yield non\-trivial rollout groups\. Specifically, the figure evaluates the latent uncertainty term1−2​\|a^plat−0\.5\|1\-2\|\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}\-0\.5\|in Equation[6](https://arxiv.org/html/2607.28077#Sx3.E6)without selection\-count decay\. This signal generally becomes stronger at deeper layers and reaches its best performance near the upper layers, suggesting that latent neighborhoods capture information relevant to group\-level rollout outcomes\. Based on this observation, LEEPS retrieves neighbors with observed rollout outcomes and aggregates their success rates to obtain the latent\-neighbor estimatea^plat\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}, which is then used to prioritize candidates within the explore portfolio\.

Table 1:Main results on mathematical reasoning benchmarks\. The Rollouts column reports the cumulative number of responses generated during RL training, including those from zero\-variance groups that are subsequently discarded by online filtering\. Best and second\-best results within each model group are shown inboldandunderlined, respectively\.![Refer to caption](https://arxiv.org/html/2607.28077v1/x3.png)Figure 3:Layer\-wise AUROC of the latent\-neighbor score across contextual hidden layers for predicting whether prompts are non\-trivial on DAPO\-Math\-17K\. The dashed line marks random prediction \(AUROC=0\.5=0\.5\)\.Latent prompt representation\.For each promptpp, LEEPS extracts the hidden state of its final prompt token from a selected model layer and applies L2 normalization to obtain the representationhph\_\{p\}\. Before RL training, LEEPS precomputes a staticKK\-nearest\-neighbor cache over the full training set using cosine similarity:

c​\(p,p′\)=hp⊤​hp′\.c\(p,p^\{\\prime\}\)=h\_\{p\}^\{\\top\}h\_\{p^\{\\prime\}\}\.\(8\)
Neighbor\-based success\-rate estimation\.For each explore candidatepp, LEEPS estimates its potential success rate from neighbors with observed rollout outcomes\. Let𝒩Kobs​\(p\)\\mathcal\{N\}\_\{K\}^\{\\mathrm\{obs\}\}\(p\)denote the subset of itsKKnearest neighbors with available success rates\. LEEPS computes

a^plat=∑p′∈𝒩Kobs​\(p\)max⁡\(c​\(p,p′\),0\)​a^p′∑p′∈𝒩Kobs​\(p\)max⁡\(c​\(p,p′\),0\),\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}=\\frac\{\\sum\_\{p^\{\\prime\}\\in\\mathcal\{N\}\_\{K\}^\{\\mathrm\{obs\}\}\(p\)\}\\max\(c\(p,p^\{\\prime\}\),0\)\\hat\{a\}\_\{p^\{\\prime\}\}\}\{\\sum\_\{p^\{\\prime\}\\in\\mathcal\{N\}\_\{K\}^\{\\mathrm\{obs\}\}\(p\)\}\\max\(c\(p,p^\{\\prime\}\),0\)\},\(9\)wheremax⁡\(c​\(p,p′\),0\)\\max\(c\(p,p^\{\\prime\}\),0\)ensures non\-negative similarity weights\. The resulting estimate is used inwexplore​\(p\)w\_\{\\mathrm\{explore\}\}\(p\)to guide candidate selection\.

## Experiments

In this section, we evaluate LEEPS to demonstrate its effectiveness, the necessity of its key components, and its computational efficiency\. Additional implementation details and experiments are provided in the Appendix\.

### Experimental Setup

Models and Training Datasets\.We conduct experiments on two widely used backbone models, Qwen2\.5\-Math\-1\.5B and Qwen2\.5\-Math\-7B\(Yanget al\.[2024](https://arxiv.org/html/2607.28077#bib.bib36)\)\. All methods are trained on the same DAPO\-Math\-17K\(Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12)\)training set to ensure a fair comparison\. We focus on mathematical reasoning and use the rule\-based verifier for both RLVR training and in\-domain evaluation\. Our experiments are implemented using VERL\(Shenget al\.[2025](https://arxiv.org/html/2607.28077#bib.bib38)\)and vLLM\(Kwonet al\.[2023](https://arxiv.org/html/2607.28077#bib.bib39)\)\. The 1\.5B models are trained on 8 NVIDIA H20 GPUs, whereas the 7B models are trained on 8 NVIDIA A100 GPUs\. Following prior work\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18); Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\), all methods use a prompt batch size of 256 and sample 8 rollouts per prompt\. We set the maximum prompt and response lengths to 1024 and 3072 tokens\.

Baselines\.To evaluate LEEPS, we compare it against four representative baselines: \(1\) vanilla GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib31)\), which uniformly learns from the training data without prompt selection; \(2\) DS\(Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12)\), an online filtering method that filters prompt groups after rollout; \(3\) MoPPS\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18)\), a pre\-rollout method that uses posterior sampling over prompt success rates to favor prompts of intermediate predicted difficulty; and \(4\) DPS\(Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\), a pre\-rollout method that models prompt\-solving dynamics to prioritize informative prompts\.

Evaluation Benchmark\.We evaluate models on MATH\-500\(Hendryckset al\.[2021](https://arxiv.org/html/2607.28077#bib.bib40)\), Minerva\-Math\(Lewkowyczet al\.[2022](https://arxiv.org/html/2607.28077#bib.bib41)\), OlympiadBench\(Heet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib42)\), AMC23, AIME24, and AIME25\(Liet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib43)\)\. Following our evaluation protocol, we report pass@1 on MATH\-500, Minerva\-Math, and OlympiadBench\. Because AMC23, AIME24, and AIME25 contain relatively few and challenging problems, we report avg@16 by sampling 16 responses per problem with temperature 0\.6 and top\-p 1\.0\. To examine out\-of\-distribution \(OOD\) generalization, we additionally evaluate the MATH\-trained models on GPQA\-Diamond\(Reinet al\.[2023](https://arxiv.org/html/2607.28077#bib.bib47)\), ARC\-C\(Clarket al\.[2018](https://arxiv.org/html/2607.28077#bib.bib46)\), and MMLU\-Pro\(Wanget al\.[2024b](https://arxiv.org/html/2607.28077#bib.bib45)\), reporting pass@1 accuracy\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x4.png)Figure 4:Training\-sample characteristics for Qwen2\.5\-Math\-1\.5B and Qwen2\.5\-Math\-7B\. Panels \(a\)–\(b\) report the non\-trivial ratio, and Panels \(c\)–\(d\) report the medium ratio, defined as the fraction of selected prompt groups with0\.3≤a​\(p\)≤0\.70\.3\\leq a\(p\)\\leq 0\.7\. Curves are smoothed using a 7\-step moving average\.![Refer to caption](https://arxiv.org/html/2607.28077v1/x5.png)Figure 5:Training progress for 1\.5B and 7B models\. Panels \(a\)–\(b\) show overall score versus training step, and Panels \(c\)–\(d\) versus cumulative rollouts, with DS included only in the latter\. Dashed lines mark the shared rollout budgets of GRPO and the pre\-rollout selectors; shaded regions indicate DS rollouts beyond those budgets\.
### Main Results

This subsection evaluates LEEPS using benchmark results and complementary training diagnostics\. We examine three aspects: overall performance, the characteristics of the selected training samples, and training progress\.

Overall Performance\.As summarized in Table[1](https://arxiv.org/html/2607.28077#Sx3.T1), LEEPS achieves the highest overall score at both model scales\. It outperforms DPS, the strongest baseline at both scales, by 0\.99 and 1\.70 points for the 1\.5B and 7B models, corresponding to relative improvements of 2\.6% and 3\.7%, respectively, without requiring additional rollouts\.

Training Sample Characteristics\.LEEPS maintains a high non\-trivial ratio while selecting a larger proportion of prompts with intermediate difficulty\. Figure[4](https://arxiv.org/html/2607.28077#Sx4.F4)compares the non\-trivial ratio and the medium ratio, defined as the fraction of selected prompt groups whose success rates satisfy0\.3≤a​\(p\)≤0\.70\.3\\leq a\(p\)\\leq 0\.7\. Uniform GRPO produces the lowest non\-trivial ratios at both model scales\. DS maintains the highest non\-trivial ratio through post\-rollout filtering but, as shown in Table[1](https://arxiv.org/html/2607.28077#Sx3.T1), requires a substantially larger rollout budget\. Moreover, despite its high non\-trivial ratio, DS retains a smaller proportion of medium\-difficulty prompt groups, which tend to provide more informative learning signals\. This smaller proportion may partly explain why DS achieves lower final overall scores than the pre\-rollout selectors despite its high non\-trivial ratio\. Among the pre\-rollout selection methods, DPS maintains a high non\-trivial ratio but selects a smaller proportion of medium\-difficulty prompts, whereas MoPPS eventually reaches a comparable medium ratio but exhibits a substantially lower non\-trivial ratio in the early training stage\. LEEPS combines a consistently high non\-trivial ratio with a large proportion of medium\-difficulty prompts throughout training\.

Training Progress\.LEEPS reaches the highest overall score at both model scales and generally improves faster than the baselines at the same training step, as shown in Figure[5](https://arxiv.org/html/2607.28077#Sx4.F5)\(a\)–\(b\)\. This advantage persists in the rollout\-normalized comparison in Figure[5](https://arxiv.org/html/2607.28077#Sx4.F5)\(c\)–\(d\)\. Within the common rollout budget, LEEPS outperforms GRPO and the other pre\-rollout selectors, whereas DS remains below these methods and requires substantially more rollouts to complete training\. Because DS may generate multiple candidate batches for each policy update, it is included only in the rollout\-normalized panels\. Figure[1](https://arxiv.org/html/2607.28077#Sx1.F1)further shows that LEEPS covers substantially more prompts than DPS while maintaining a higher non\-trivial ratio than MoPPS, particularly early in training\. The performance gains of LEEPS are therefore associated with a high non\-trivial ratio, informative sample selection, and broad prompt coverage\.

### Ablation Study

We conduct ablation studies to examine how the components contribute to LEEPS\. As shown in Table[2](https://arxiv.org/html/2607.28077#Sx4.T2), the complete method achieves the best overall score at both model scales\. Figure[6](https://arxiv.org/html/2607.28077#Sx4.F6)further provides analysis for the random\-exploration and no\-portfolio variants on the 1\.5B model\. Specifically, we consider the following controlled variants:

- •w/ Random Exploration: uniformly samples from the explore portfolio without Latent\-Guided Exploration\.
- •w/ Uniform Exploitation: uniformly samples from the exploit portfolio without success\-rate\-based weights\.
- •w/o E\-E Portfolios: removes Adaptive Explore–Exploit Portfolio Allocation and uses only Latent\-Guided Exploration for prompt selection\.

Table 2:Ablation results for LEEPS\. Values are the best overall scores within the model\-specific selection ranges, across six mathematical reasoning benchmarks\.Random Exploration\.Replacing latent\-guided exploration with random sampling results in relative overall score decreases of 4\.2% and 3\.5% for the 1\.5B and 7B models, respectively\. Because prompts in the explore portfolio are unseen or have produced zero\-variance groups, random sampling treats candidates with substantially different potential utility equally\. Figure[6](https://arxiv.org/html/2607.28077#Sx4.F6)\(a\) shows a lower cumulative non\-trivial ratio among explored prompt groups, while Figure[6](https://arxiv.org/html/2607.28077#Sx4.F6)\(b\) shows a lower selected\-batch non\-trivial ratio\. These results indicate that latent information transfers observations from related prompts to make exploration more targeted and maintain a more reliable supply of useful training signals\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x6.png)Figure 6:Ablation diagnostics on Qwen2\.5\-Math\-1\.5B\. Panel \(a\) reports the cumulative non\-trivial ratio among explored prompt groups\. Panel \(b\) compares the selected\-batch non\-trivial ratio of LEEPS and its ablations, with curves smoothed using a 7\-step moving average\.Uniform Exploitation\.Uniform exploit sampling results in relative overall score decreases of 1\.7% and 2\.6% for the 1\.5B and 7B models, respectively\. Because this variant retains the portfolio structure and latent\-guided exploration, the drop isolates the benefit of success\-rate\-aware weighting, which prioritizes informative prompts near the current learning boundary\. Despite these reductions, the variant still outperforms the strongest baseline at both model scales, while full LEEPS achieves a larger margin\.

Without Explore\-Exploit Portfolio\.Removing the explore–exploit portfolios results in relative overall score decreases of 3\.1% and 4\.8% for the 1\.5B and 7B models, respectively\. Without separate portfolios, the sampler cannot explicitly retain verified useful prompts while reserving budget for continued exploration\. Figure[6](https://arxiv.org/html/2607.28077#Sx4.F6)\(b\) shows that selected\-batch quality consequently deteriorates during training; the proportion of all\-one prompt groups rises from 6\.3% over the first 50 steps to 21\.9% over the final 50 steps\. This result indicates that latent\-guided exploration can prioritize uncertain candidates but cannot alone maintain a stable supply of informative prompts as the policy evolves\.

### Additional Analysis

OOD Generalization\.To assess whether improved in\-domain training efficiency compromises broader reasoning ability, we evaluate the trained models on three OOD general\-reasoning benchmarks\. LEEPS achieves the highest average OOD score at both model scales, reaching 28\.08 for the 1\.5B model and 40\.95 for the 7B model, compared with 27\.94 and 40\.85 for the strongest respective baselines \(Table[3](https://arxiv.org/html/2607.28077#Sx4.T3)\)\. Although the margins are modest, these consistent gains across both model scales indicate that LEEPS improves in\-domain training efficiency without sacrificing OOD general\-reasoning performance\.

Table 3:OOD general reasoning results\. Best and second\-best results within each model group are shown inboldandunderlined, respectively\.Computational Efficiency\.Figure[7](https://arxiv.org/html/2607.28077#Sx4.F7)shows that rollout generation and policy updates dominate runtime, while prompt selection and sampler updates add only 2\.15 seconds \(2\.6%\) and 2\.26 seconds \(1\.0%\) per step for the 1\.5B and 7B models, respectively\. Sampler initialization takes 124 and 292 seconds, respectively, but occurs only once and does not affect steady\-state efficiency\. The end\-to\-end runtimes reported in Table[1](https://arxiv.org/html/2607.28077#Sx3.T1)also show the efficiency of LEEPS\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x7.png)Figure 7:Average per\-step runtime over 20 profiled steps for both model scales on 8 NVIDIA H20 GPUs\. LEEPS includes prompt selection and sampler updates\.

## Related Work

RL for LLM Training\.Reinforcement learning has been widely used in the post\-training of large language models to align model outputs with human preferences and task\-specific objectives\. A representative line of work is RLHF, which learns reward models from human preference annotations and optimizes language models with PPO\(Schulmanet al\.[2017](https://arxiv.org/html/2607.28077#bib.bib32); Ouyanget al\.[2022](https://arxiv.org/html/2607.28077#bib.bib26); Baiet al\.[2022](https://arxiv.org/html/2607.28077#bib.bib33)\)\. Later preference\-optimization methods, such as DPO, simplify this pipeline by directly optimizing preference pairs without explicit reward\-model training or online RL updates\(Rafailovet al\.[2023](https://arxiv.org/html/2607.28077#bib.bib29)\)\. More recent work explores reinforcement learning with verifiable rewards, where automatically checkable signals in domains such as mathematics and code are used to supervise LLM reasoning\(Cobbeet al\.[2021](https://arxiv.org/html/2607.28077#bib.bib34); Guoet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib30)\)\. On the algorithmic side, GRPO further reduces the complexity of PPO\-style optimization by using group\-relative rewards instead of a separate value model\(Shaoet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib31)\)\. Building on this line of work, we address the inefficiency caused by zero\-variance prompt groups in RLVR through pre\-rollout prompt selection\.

Data Selection for RLVR\.Prompt selection is important for RLVR because prompt utility changes throughout training\. Traditional curriculum\-based methods organize prompts using predefined difficulty levels or adjust the proportions of tasks at different difficulty levels\(Teamet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib8); Songet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib9); Parasharet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib10)\), but such signals may not capture the model’s evolving competence\. To obtain model\-specific signals, online filtering methods use fresh rollouts to identify informative prompts\(Baeet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib11); Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12); Zhanget al\.[2025b](https://arxiv.org/html/2607.28077#bib.bib14); Xuet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib25)\), incurring generation cost before filtering\. Building on this paradigm, history\-aware methods reuse past rollout accuracy as prior information to guide subsequent sampling and reduce redundant rollouts\(Chenet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib20); Zhenget al\.[2026](https://arxiv.org/html/2607.28077#bib.bib15); Shenet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib49); Shiet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib22)\)\. Complementary directions improve rollout efficiency by adaptively allocating different numbers of responses to prompts according to their estimated learning value\(Yaoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib16); Zenget al\.[2025](https://arxiv.org/html/2607.28077#bib.bib17); Fanget al\.[2026](https://arxiv.org/html/2607.28077#bib.bib21)\), or by training auxiliary models to predict prompt difficulty\(Gaoet al\.[2025](https://arxiv.org/html/2607.28077#bib.bib24); Sunet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib13)\)\. More recent methods make prompt\-selection decisions without additional rollouts using historical training signals\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18); Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\)\. However, existing pre\-rollout selectors face an exploration–exploitation trade\-off between exploiting historically informative prompts and exploring prompts with uncertain utility\. LEEPS addresses this trade\-off through adaptive explore–exploit portfolio allocation and latent\-guided exploration\.

## Conclusion

We presented LEEPS, a pre\-rollout prompt sampler that combines adaptive explore–exploit portfolio allocation with latent\-guided exploration to select informative prompts and use fixed rollout budgets more effectively without requiring additional selection rollouts\. Empirical results across six mathematical reasoning benchmarks demonstrate that LEEPS achieves the highest overall scores for both Qwen2\.5\-Math\-1\.5B and 7B under matched rollout budgets, and ablation studies further support the contributions of both components\. LEEPS also achieves the highest average score across the three evaluated OOD benchmarks at both model scales, while incurring only minor online sampling overhead\.

## References

- S\. Bae, J\. Hong, M\. Y\. Lee, H\. Kim, J\. Nam, and D\. Kwak \(2026\)Online difficulty filtering for reasoning oriented reinforcement learning\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 700–719\.Cited by:[Adaptive Explore–Exploit Portfolio Allocation](https://arxiv.org/html/2607.28077#Sx3.SSx1.p3.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- I\. V\. M\. Cencerrado, A\. P\. Masdemont, A\. G\. Hawthorne, D\. D\. Africa, and L\. Pacchiardi \(2025\)No answer needed: predicting llm answer accuracy from question\-only linear probes\.arXiv preprint arXiv:2509\.10625\.Cited by:[Latent\-Guided Exploration](https://arxiv.org/html/2607.28077#Sx3.SSx2.p1.1)\.
- X\. Chen, J\. Lu, M\. Kim, D\. Zhang, J\. Tang, A\. Piché, N\. Gontier, Y\. Bengio, and E\. Kamalloo \(2025\)Self\-evolving curriculum for llm reasoning\.arXiv preprint arXiv:2505\.14970\.Cited by:[Adaptive Explore–Exploit Portfolio Allocation](https://arxiv.org/html/2607.28077#Sx3.SSx1.p3.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[2nd item](https://arxiv.org/html/2607.28077#A2.I2.i2.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- Y\. Fang, J\. Lin, X\. Fu, C\. Qin, H\. Shi, C\. Hu, L\. Pan, K\. Zeng, and X\. Cai \(2026\)How to allocate, how to learn? dynamic rollout allocation and advantage modulation for policy optimization\.arXiv preprint arXiv:2602\.19208\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Z\. Gao, J\. Kim, W\. Sun, T\. Joachims, S\. Wang, R\. Y\. Pang, and L\. Tan \(2025\)Prompt curriculum learning for efficient llm post\-training\.arXiv preprint arXiv:2510\.01135\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024\)Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.Cited by:[3rd item](https://arxiv.org/html/2607.28077#A2.I1.i3.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[1st item](https://arxiv.org/html/2607.28077#A2.I1.i1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- J\. Hu, Y\. Zhang, Q\. Han, D\. Jiang, X\. Zhang, and H\. Shum \(2026\)Open\-reasoner\-zero: an open source approach to scaling up reinforcement learning on the base model\.Advances in Neural Information Processing Systems38,pp\. 162239–162262\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p1.1)\.
- N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.\(2024\)Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo,et al\.\(2022\)Solving quantitative reasoning problems with language models\.Advances in neural information processing systems35,pp\. 3843–3857\.Cited by:[2nd item](https://arxiv.org/html/2607.28077#A2.I1.i2.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. Huang, K\. Rasul, L\. Yu, A\. Q\. Jiang, Z\. Shen,et al\.\(2024\)Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions\.Hugging Face repository13\(9\),pp\. 9\.Cited by:[4th item](https://arxiv.org/html/2607.28077#A2.I1.i4.p1.1),[5th item](https://arxiv.org/html/2607.28077#A2.I1.i5.p1.1),[6th item](https://arxiv.org/html/2607.28077#A2.I1.i6.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- Y\. Mao, Y\. Qu, Q\. Wang, H\. Zou, and X\. Ji \(2026\)Dynamics\-predictive sampling for active rl finetuning of large reasoning models\.arXiv preprint arXiv:2603\.10887\.Cited by:[Appendix B](https://arxiv.org/html/2607.28077#A2.SSx2.p1.1),[Introduction](https://arxiv.org/html/2607.28077#Sx1.p2.1),[Preliminaries](https://arxiv.org/html/2607.28077#Sx2.p3.5),[Adaptive Explore–Exploit Portfolio Allocation](https://arxiv.org/html/2607.28077#Sx3.SSx1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p2.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- S\. Parashar, S\. Gui, X\. Li, H\. Ling, S\. Vemuri, B\. Olson, E\. Li, Y\. Zhang, J\. Caverlee, D\. Kalathil,et al\.\(2025\)Curriculum reinforcement learning from easy to hard tasks improves llm reasoning\.arXiv preprint arXiv:2506\.06632\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Y\. Qu, Q\. Wang, Y\. Mao, V\. T\. Hu, B\. Ommer, and X\. Ji \(2026\)Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?\.InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 1,pp\. 1240–1250\.Cited by:[Appendix B](https://arxiv.org/html/2607.28077#A2.SSx2.p1.1),[Introduction](https://arxiv.org/html/2607.28077#Sx1.p2.1),[Preliminaries](https://arxiv.org/html/2607.28077#Sx2.p3.5),[Adaptive Explore–Exploit Portfolio Allocation](https://arxiv.org/html/2607.28077#Sx3.SSx1.p1.1),[Adaptive Explore–Exploit Portfolio Allocation](https://arxiv.org/html/2607.28077#Sx3.SSx1.p3.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p2.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)Gpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[1st item](https://arxiv.org/html/2607.28077#A2.I2.i1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Preliminaries](https://arxiv.org/html/2607.28077#Sx2.p1.6),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p2.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p1.1)\.
- Q\. Shen, D\. Chen, Y\. Huang, Z\. Ling, Y\. Li, B\. Ding, and J\. Zhou \(2025\)BOTS: a unified framework for bayesian online task selection in llm reinforcement finetuning\.arXiv preprint arXiv:2510\.26374\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)Hybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p1.1)\.
- T\. Shi, Y\. Wu, L\. Song, T\. Zhou, and J\. Zhao \(2025\)Efficient reinforcement finetuning via adaptive curriculum learning\.arXiv preprint arXiv:2504\.05520\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- M\. Song, M\. Zheng, Z\. Li, W\. Yang, X\. Luo, Y\. Pan, and F\. Zhang \(2025\)Fastcurl: curriculum reinforcement learning with stage\-wise context scaling for efficient training r1\-like reasoning models\.arXiv preprint arXiv:2503\.17287\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Y\. Sun, J\. Shen, Y\. Wang, T\. Chen, Z\. Wang, M\. Zhou, and H\. Zhang \(2026\)Improving data efficiency for llm reinforcement fine\-tuning through difficulty\-targeted online data selection and rollout replay\.Advances in Neural Information Processing Systems38,pp\. 173449–173476\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao,et al\.\(2025\)Kimi k1\. 5: scaling reinforcement learning with llms\.arXiv preprint arXiv:2501\.12599\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins \(2022\)Solving math word problems with process\-and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui \(2024a\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024b\)Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.arXiv preprint arXiv:2406\.01574\.Cited by:[3rd item](https://arxiv.org/html/2607.28077#A2.I2.i3.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p3.1)\.
- X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao,et al\.\(2025\)Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms\.arXiv preprint arXiv:2506\.14245\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- T\. Xie, Z\. Gao, Q\. Ren, H\. Luo, Y\. Hong, B\. Dai, J\. Zhou, K\. Qiu, Z\. Wu, and C\. Luo \(2025\)Logic\-rl: unleashing llm reasoning with rule\-based reinforcement learning\.arXiv preprint arXiv:2502\.14768\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1)\.
- Y\. E\. Xu, Y\. Savani, F\. Fang, and J\. Z\. Kolter \(2025\)Not all rollouts are useful: down\-sampling rollouts in llm reinforcement learning\.arXiv preprint arXiv:2504\.13818\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p2.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Cited by:[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p1.1)\.
- J\. Yao, Y\. Hao, H\. Zhang, H\. Dong, W\. Xiong, N\. Jiang, and T\. Zhang \(2026\)Optimizing chain\-of\-thought reasoners via gradient variance minimization in rejection sampling and rl\.Advances in Neural Information Processing Systems38,pp\. 163245–163284\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2026\)Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[Appendix B](https://arxiv.org/html/2607.28077#A2.SSx1.p1.1),[Appendix B](https://arxiv.org/html/2607.28077#A2.SSx2.p1.1),[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.28077#Sx1.p2.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p1.1),[Experimental Setup](https://arxiv.org/html/2607.28077#Sx4.SSx1.p2.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Y\. Zeng, Z\. Sun, B\. Ji, E\. Min, H\. Cai, S\. Wang, D\. Yin, H\. Zhang, X\. Chen, and J\. Wang \(2025\)CurES: from gradient analysis to efficient curriculum learning for reasoning llms\.arXiv preprint arXiv:2510\.01037\.Cited by:[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- Q\. Zhang, Y\. Fu, Y\. Wang, L\. Yan, T\. Wei, K\. Xu, M\. Huang, and H\. Qiu \(2025a\)On the self\-awareness of large reasoning models’ capability boundaries\.arXiv preprint arXiv:2509\.24711\.Cited by:[Latent\-Guided Exploration](https://arxiv.org/html/2607.28077#Sx3.SSx2.p1.1)\.
- R\. Zhang, D\. Arora, S\. Mei, and A\. Zanette \(2025b\)Speed\-rl: faster training of reasoning models via online curriculum learning\.arXiv preprint arXiv:2506\.09016\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p2.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.
- H\. Zheng, Y\. Zhou, B\. Bartoldson, B\. Kailkhura, F\. Lai, J\. Zhao, and B\. Chen \(2026\)Act only when it pays: efficient reinforcement learning for llm reasoning via selective rollouts\.Advances in Neural Information Processing Systems38,pp\. 124321–124346\.Cited by:[Introduction](https://arxiv.org/html/2607.28077#Sx1.p1.1),[Related Work](https://arxiv.org/html/2607.28077#Sx5.p2.1)\.

## Appendix AAdditional Method Details

### GRPO Objective and Zero\-Variance Handling

Given a promptpp, the old policy samples a group ofGGresponses:

𝒴\(p\)=\{yp,j\}j=1G,yp,j∼πθold\(⋅∣p\)\.\\mathcal\{Y\}\(p\)=\\\{y\_\{p,j\}\\\}\_\{j=1\}^\{G\},\\qquad y\_\{p,j\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid p\)\.\(10\)Let𝐑p=\[R​\(p,yp,1\),…,R​\(p,yp,G\)\]\\mathbf\{R\}\_\{p\}=\[R\(p,y\_\{p,1\}\),\\ldots,R\(p,y\_\{p,G\}\)\]and letR¯p\\bar\{R\}\_\{p\}denote its mean\. GRPO computes the group\-normalized advantage as

Ap,j=R​\(p,yp,j\)−R¯pstd⁡\(𝐑p\)\.A\_\{p,j\}=\\frac\{R\(p,y\_\{p,j\}\)\-\\bar\{R\}\_\{p\}\}\{\\operatorname\{std\}\(\\mathbf\{R\}\_\{p\}\)\}\.\(11\)The token\-level importance ratio between the current and old policies is

rp,j,t​\(θ\)=πθ​\(yp,j,t∣p,yp,j,<t\)πθold​\(yp,j,t∣p,yp,j,<t\)\.r\_\{p,j,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{p,j,t\}\\mid p,y\_\{p,j,<t\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{p,j,t\}\\mid p,y\_\{p,j,<t\}\)\}\.\(12\)The policy is optimized using the clipped objective

𝒥GRPO\(θ\)=𝔼\[1G∑j=1G1\|yp,j\|∑t=1\|yp,j\|\[\\displaystyle\\mathcal\{J\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\\frac\{1\}\{\|y\_\{p,j\}\|\}\\sum\_\{t=1\}^\{\|y\_\{p,j\}\|\}\\Big\[min\(rp,j,t\(θ\)Ap,j,\\displaystyle\\min\\big\(r\_\{p,j,t\}\(\\theta\)A\_\{p,j\},\(13\)clip\(rp,j,t\(θ\),1−δ,1\+δ\)Ap,j\)\\displaystyle\\operatorname\{clip\}\(r\_\{p,j,t\}\(\\theta\),1\-\\delta,1\+\\delta\)A\_\{p,j\}\\big\)−βDKLp,j,t\]\],\\displaystyle\-\\beta D\_\{\\mathrm\{KL\}\}^\{p,j,t\}\\Big\]\\Bigg\],Here,rp,j,t​\(θ\)r\_\{p,j,t\}\(\\theta\)measures the token\-level probability change from the old policy to the current policy\. Clipping restricts this ratio to\[1−δ,1\+δ\]\[1\-\\delta,1\+\\delta\], and the minimum forms a conservative PPO\-style surrogate that limits overly large policy updates\. The factor1/\|yp,j\|1/\|y\_\{p,j\}\|normalizes contributions across responses of different lengths\. The KL term regularizes the updated policy toward the reference policy\. In our experiments, we setβ=0\\beta=0and disable both the KL loss in the policy objective and the KL penalty in the reward\.

### Algorithmic Details of LEEPS

Algorithm[1](https://arxiv.org/html/2607.28077#alg1)makes the operational sequence of LEEPS explicit\. Before RL training, the sampler extracts a representation for every training prompt and precomputes its latent\-neighbor cache\. The firstTcoldT\_\{\\mathrm\{cold\}\}training steps use only the explore portfolio to collect initial rollout feedback\. After this cold\-start phase, LEEPS estimates the recent non\-trivial ratios of the explore and exploit portfolios and adjusts the exploration fractionρ\\rho\. When either estimate is unavailable, it uses the default exploration fractionρ0\\rho\_\{0\}; the resulting value is clipped to\[ρmin,ρmax\]\[\\rho\_\{\\min\},\\rho\_\{\\max\}\]\.

Within the exploit portfolio, prompts are sampled without replacement according towexploitw\_\{\\mathrm\{exploit\}\}\. Within the explore portfolio, candidates with valid latent\-neighbor estimates are ranked bywexplorew\_\{\\mathrm\{explore\}\}, while candidates without observed neighbors are randomly ordered and used as fallback options\. If either portfolio cannot satisfy its assigned quota, the sampler fills the remaining positions from the unselected candidates in the other portfolio\. After rollout,a^p\\hat\{a\}\_\{p\}is replaced by the latest group success rate,npn\_\{p\}is incremented, and the portfolio\-level histories are updated\. Thus, all online decisions depend only on previously observed outcomes and require no additional selection rollouts\.

Algorithm 1LEEPS training procedureInput: Training set𝒟\\mathcal\{D\}, policyπθ\\pi\_\{\\theta\}, batch sizeBB, group sizeGG Parameters: Default exploration fractionρ0\\rho\_\{0\}, target non\-trivial ratioτ\\tau, cold\-start lengthTcoldT\_\{\\mathrm\{cold\}\}, bounds\[ρmin,ρmax\]\[\\rho\_\{\\min\},\\rho\_\{\\max\}\]

1:Precompute the latent neighborhood

𝒩​\(p\)\\mathcal\{N\}\(p\)for each

p∈𝒟p\\in\\mathcal\{D\}\.

2:Initialize prompt statistics

\{a^p,np\}\\\{\\hat\{a\}\_\{p\},n\_\{p\}\\\}and portfolio\-level histories\.

3:fortraining step

t=1,2,…t=1,2,\\ldotsdo

4:Draw a candidate batch

ℬcand\\mathcal\{B\}\_\{\\mathrm\{cand\}\}with

\|ℬcand\|\>B\|\\mathcal\{B\}\_\{\\mathrm\{cand\}\}\|\>B\.

5:

𝒫exploit←\{p∈ℬcand:np\>0,0<a^p<1\}\\mathcal\{P\}\_\{\\mathrm\{exploit\}\}\\leftarrow\\\{p\\in\\mathcal\{B\}\_\{\\mathrm\{cand\}\}:n\_\{p\}\>0,\\ 0<\\hat\{a\}\_\{p\}<1\\\}\.

6:

𝒫explore←ℬcand∖𝒫exploit\\mathcal\{P\}\_\{\\mathrm\{explore\}\}\\leftarrow\\mathcal\{B\}\_\{\\mathrm\{cand\}\}\\setminus\\mathcal\{P\}\_\{\\mathrm\{exploit\}\}\.

7:if

t≤Tcoldt\\leq T\_\{\\mathrm\{cold\}\}then

8:

ρ←1\\rho\\leftarrow 1\.

9:else

10:Estimate

qexploreq\_\{\\mathrm\{explore\}\}and

qexploitq\_\{\\mathrm\{exploit\}\}from recent rollout outcomes\.

11:Choose

ρ\\rhoto target

τ=ρ​qexplore\+\(1−ρ\)​qexploit\\tau=\\rho q\_\{\\mathrm\{explore\}\}\+\(1\-\\rho\)q\_\{\\mathrm\{exploit\}\}; use

ρ0\\rho\_\{0\}if feedback is insufficient\.

12:

ρ←clip⁡\(ρ,ρmin,ρmax\)\\rho\\leftarrow\\operatorname\{clip\}\(\\rho,\\rho\_\{\\min\},\\rho\_\{\\max\}\)\.

13:endif

14:

bexploit←round⁡\(\(1−ρ\)​B\)b\_\{\\mathrm\{exploit\}\}\\leftarrow\\operatorname\{round\}\(\(1\-\\rho\)B\)and

bexplore←round⁡\(ρ​B\)b\_\{\\mathrm\{explore\}\}\\leftarrow\\operatorname\{round\}\(\\rho B\)\.

15:Select up to

bexploitb\_\{\\mathrm\{exploit\}\}prompts from

𝒫exploit\\mathcal\{P\}\_\{\\mathrm\{exploit\}\}using

wexploitw\_\{\\mathrm\{exploit\}\}\.

16:Estimate

a^plat\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}for

p∈𝒫explorep\\in\\mathcal\{P\}\_\{\\mathrm\{explore\}\}from observed neighbors\.

17:Rank scored explore prompts by

wexplorew\_\{\\mathrm\{explore\}\}and randomly order unscored prompts\.

18:Select up to

bexploreb\_\{\\mathrm\{explore\}\}prompts from

𝒫explore\\mathcal\{P\}\_\{\\mathrm\{explore\}\}and backfill any quota shortfall\.

19:Combine both selections into

ℬtrain\\mathcal\{B\}\_\{\\mathrm\{train\}\}with

\|ℬtrain\|=B\|\\mathcal\{B\}\_\{\\mathrm\{train\}\}\|=B\.

20:Generate

GGresponses per selected prompt and compute each success rate

a​\(p\)a\(p\)\.

21:Update

\{a^p,np\}\\\{\\hat\{a\}\_\{p\},n\_\{p\}\\\}and the portfolio\-level histories using the observed outcomes\.

22:Update

πθ\\pi\_\{\\theta\}with GRPO using the generated responses\.

23:endfor

24:return

πθ\\pi\_\{\\theta\}

#### Complexity and Scalability\.

LEEPS introduces two sources of additional cost beyond standard RLVR training: one\-time initialization and per\-step prompt selection\. LetNNdenote the number of training prompts,ddthe dimension of each prompt representation,M=\|ℬcand\|M=\|\\mathcal\{B\}\_\{\\mathrm\{cand\}\}\|the candidate\-batch size,KstoredK\_\{\\mathrm\{stored\}\}the number of nearest neighbors stored for each prompt, andK≤KstoredK\\leq K\_\{\\mathrm\{stored\}\}the maximum number of neighbors with observed rollout outcomes used for success\-rate estimation\. During initialization, extracting all prompt representations requiresO​\(N​Cenc\)O\(NC\_\{\\mathrm\{enc\}\}\)time, whereCencC\_\{\\mathrm\{enc\}\}is the cost of encoding one prompt with the backbone model, and constructing the exact nearest\-neighbor lists requiresO​\(N2​d\)O\(N^\{2\}d\)time\. The resulting representations, neighbor indices, and similarities requireO​\(N​d\+N​Kstored\)O\(Nd\+NK\_\{\\mathrm\{stored\}\}\)persistent storage\. During RL training, aggregating up toKKobserved neighbors for each of theMMcandidate prompts gives theO​\(M​K\)O\(MK\)scoring cost, while sorting at mostMMexploration candidates by their scores gives theO​\(M​log⁡M\)O\(M\\log M\)ranking cost\. If the stored lists must be fully scanned to find observed neighbors, the scoring cost is at mostO​\(M​Kstored\)O\(MK\_\{\\mathrm\{stored\}\}\)\. The online sampler also stores the two rollout\-dependent statisticsa^p\\hat\{a\}\_\{p\}andnpn\_\{p\}for each prompt, requiringO​\(N\)O\(N\)storage, and temporarily storesO​\(M\)O\(M\)candidate scores at each step\. Consequently, whenMM,KK, andKstoredK\_\{\\mathrm\{stored\}\}are fixed, increasing the training\-set sizeNNmainly increases initialization time and persistent storage, without directly increasing the per\-step selection time\. Similarly, increasing the backbone size mainly raisesCencC\_\{\\mathrm\{enc\}\}and typicallydd, again affecting initialization rather than online selection, which requires no additional model forward passes\.

## Appendix BExperimental Details

### Training Data

All methods are trained on DAPO\-Math\-17K\(Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12)\), which provides mathematical reasoning problems paired with reference answers for rule\-based verification\. We use the original problem text without additional rewriting\. For both RLVR training and mathematical reasoning evaluation, each problem is represented by the following messages and then serialized using the model’s native chat template:

System:Please reason step by step, and put your final answer within \\boxed\{\}\. User:\{question\}

During RLVR training, a rule\-based verifier extracts the final boxed answer and checks it against the reference answer to produce a binary reward\.

### Training Configuration

Table[4](https://arxiv.org/html/2607.28077#A2.T4)summarizes the main training hyperparameters\. All methods at both model scales share the common settings\. The LEEPS\-specific settings control candidate construction, adaptive explore–exploit allocation, and latent\-neighbor retrieval\. Following DAPO\(Yuet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib12)\), we adopt the Clip\-Higher strategy, setting the lower and upper policy\-ratio clipping coefficients to 0\.20 and 0\.28, respectively, to permit larger probability increases and mitigate premature entropy collapse\. We disable both the KL loss in the policy objective and the KL penalty in the reward, following the prior work\(Quet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib18); Maoet al\.[2026](https://arxiv.org/html/2607.28077#bib.bib19)\)\.

Table 4:Training hyperparameters for Qwen2\.5\-Math\-1\.5B and Qwen2\.5\-Math\-7B\. Common settings are shared by all methods, while the lower block contains parameters specific to LEEPS\.For latent\-neighbor construction, we use the final\-token hidden states from layer 22 of Qwen2\.5\-Math\-1\.5B and layer 26 of Qwen2\.5\-Math\-7B\. For each prompt, we cache its 1,024 nearest candidates and use up toK=64K=64neighbors with observed rollout outcomes to computea^plat\\hat\{a\}^\{\\mathrm\{lat\}\}\_\{p\}\. The prompt representations and the resulting neighbor cache are computed once before RL training and kept fixed thereafter\. During training, LEEPS updates only the rollout\-dependent prompt statistics used with this cache\. This implementation avoids repeated representation extraction and nearest\-neighbor construction while allowing the exploration scores to adapt to newly observed rollout outcomes\.

For all offline latent\-neighbor AUROC analyses, we use DAPO\-Math\-17K, the same prompt pool used for subsequent RL training, with a random 75%/25% reference–query split \(seed 42\)\. Neighbor estimates are constructed from the reference pool and evaluated on the query set; this split is used only for the diagnostic, while RL training uses the complete dataset\.

### Evaluation Benchmarks and Protocol

#### Mathematical Reasoning Evaluation\.

The six in\-domain benchmarks cover complementary forms and levels of mathematical reasoning:

- •MATH\-500\(Hendryckset al\.[2021](https://arxiv.org/html/2607.28077#bib.bib40)\)is a curated 500\-problem subset of MATH covering algebra, geometry, number theory, counting and probability, and other competition\-mathematics subjects\.
- •Minerva\-Math\(Lewkowyczet al\.[2022](https://arxiv.org/html/2607.28077#bib.bib41)\)contains 272 quantitative reasoning problems drawn from mathematical and scientific contexts\.
- •OlympiadBench\(Heet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib42)\)targets challenging olympiad\-level reasoning\. We use 581 single\-answer problems from its English text\-only competition\-mathematics subset\.
- •AMC23\(Liet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib43)\)contains 40 problems from the 2023 American Mathematics Competitions\.
- •AIME24\(Liet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib43)\)contains 30 problems from the 2024 American Invitational Mathematics Examination\.
- •AIME25\(Liet al\.[2024](https://arxiv.org/html/2607.28077#bib.bib43)\)contains 30 problems from the 2025 American Invitational Mathematics Examination\.

We use the same message format as in training, with the original problem placed in the user message and the complete messages serialized using the model’s native chat template\. We report greedy pass@1 with temperature 0 for MATH\-500, Minerva\-Math, and OlympiadBench\. For AMC23, AIME24, and AIME25, we sample 16 responses per problem with temperature 0\.6 and top\-pp1\.0 and report avg@16\. The maximum generation length is 3072 tokens in all cases\. Generated answers are scored using the same rule\-based verification procedure, and the Overall score is the unweighted mean of the six primary benchmark metrics\.

#### OOD General\-Reasoning Evaluation\.

We additionally evaluate cross\-domain generalization on three multiple\-choice benchmarks that are not used for training\. Each method is evaluated on the OOD benchmarks using the same checkpoint as for the main mathematical reasoning results\.

- •GPQA\-Diamond\(Reinet al\.[2023](https://arxiv.org/html/2607.28077#bib.bib47)\)contains 198 expert\-validated, graduate\-level questions in biology, physics, and chemistry\.
- •ARC\-Challenge\(Clarket al\.[2018](https://arxiv.org/html/2607.28077#bib.bib46)\)contains 1,172 challenging grade\-school science questions that require both scientific knowledge and reasoning\.
- •MMLU\-Pro\(Wanget al\.[2024b](https://arxiv.org/html/2607.28077#bib.bib45)\)contains 12,032 questions spanning a broad range of academic and professional disciplines\. Its more challenging questions and expanded candidate sets make it more difficult than the original MMLU benchmark\.

For OOD evaluation, the original question and its letter\-labeled choices are placed in the user message\. Because these benchmarks require a choice rather than a free\-form mathematical answer, we use the following adjusted system instruction:

System:Please reason step by step, and only put your final choice within \\boxed\{\}\. User:\{question and letter\-labeled choices\}

All OOD results are greedy pass@1 accuracies\. We use temperature 0, generate one response per question, and set the maximum generation length to 3072 tokens\. The predicted choice is extracted from the final boxed expression and compared with the reference option\.

## Appendix CAdditional Experimental Results

### Experiments with the Llama Model

#### Experimental Configuration\.

We repeat the main experimental protocol using Llama\-3\.2\-3B\-Instruct as the backbone\. All settings follow the main experiments unless specified otherwise\. The maximum prompt and response lengths are set to 2,048 and 8,192 tokens, respectively\. For LEEPS, the target non\-trivial ratio is fixed toτ=0\.8\\tau=0\.8before training, and embedding layer 21 is used for the latent\-neighbor representation\. We evaluate every 10 training steps across the six benchmarks\.

Table 5:Results on Llama\-3\.2\-3B\-Instruct\. For each method, we report the checkpoint with the highest average score within the first 160 training steps\. The Rollouts column gives the cumulative number of generated responses up to the common 160\-step cutoff; DS may generate multiple candidate batches per policy update\. Best and second\-best scores among the trained methods are shown inboldandunderlined, respectively\.![Refer to caption](https://arxiv.org/html/2607.28077v1/x8.png)Figure 8:Additional results on Llama\-3\.2\-3B\-Instruct\. Panels \(a\)–\(b\) show the overall score versus training step and cumulative rollouts, respectively\. DS is omitted from Panel \(a\) because it may generate multiple candidate batches per policy update, but is included in the rollout\-normalized comparison in Panel \(b\)\. The dashed line marks the shared 328K\-rollout budget of GRPO and the pre\-rollout selectors, and the shaded region indicates additional rollouts used by DS\. Panels \(c\)–\(d\) show the non\-trivial ratio and medium ratio of the selected prompt groups, using a 7\-step centered moving average\.
#### Results and Analysis\.

As shown in Table[5](https://arxiv.org/html/2607.28077#A3.T5), LEEPS achieves the highest average score of 24\.26\. Among the methods using the shared 328K\-rollout budget, the strongest baseline is MoPPS at 23\.07; LEEPS improves upon it by 1\.19 points, corresponding to a relative gain of approximately 5\.2%\. DS also performs strongly, reaching 24\.23, but uses 1,231K rollouts by the 160\-step cutoff, or3\.76×3\.76\\timesthe shared budget\. Thus, LEEPS achieves performance comparable to DS while requiring substantially fewer rollouts\. Figure[8](https://arxiv.org/html/2607.28077#A3.F8)\(a\)–\(b\) further shows that LEEPS achieves the highest peak among the pre\-rollout selectors within the common rollout budget\.

The sample characteristics in Figure[8](https://arxiv.org/html/2607.28077#A3.F8)\(c\)–\(d\) help explain this efficiency difference\. DS keeps its non\-trivial ratio close to 100% through post\-rollout filtering, but this requires generating multiple candidate batches and discarding uninformative responses\. LEEPS instead maintains relatively high non\-trivial and medium ratios through most of training without generating additional rollouts for selection\. GRPO remains lower on both measures; MoPPS has a lower non\-trivial ratio, especially early in training; and DPS preserves a relatively high non\-trivial ratio but includes fewer medium\-difficulty prompts\. This pattern suggests that LEEPS retains a favorable balance between sample quality and rollout cost on the Llama backbone, providing further evidence that its sampling strategy extends beyond the Qwen2\.5\-Math family\.

### Hyperparameter Sensitivity

We examine the robustness of LEEPS along three dimensions: the amount of rollout\-labeled data used for layer calibration, the number of latent neighborsKK, and the target non\-trivial ratioτ\\tau\. The first two concern the reliability and sample efficiency of the offline latent\-neighbor construction, whereasτ\\taucontrols the online explore–exploit allocation\. The representation layer is selected once before training, using a predefined offline AUROC criterion applied across all candidate layers, rather than being tuned to downstream performance\. Because this layer\-wise diagnostic directly evaluates the signal used by latent\-guided exploration, we focus on the calibration data needed to make a stable layer choice, rather than repeating the full RL procedure for each layer\.

#### Calibration Sample Size\.

The full\-data diagnostic shows that the non\-triviality signal strengthens with depth but forms a broad upper\-layer plateau, where adjacent layers differ little in AUROC\. Layer calibration therefore needs only to locate this near\-optimal region rather than recover a unique maximizer\. We recompute the diagnostic on nested subsets and measure the Pearson correlation between each subset’s hidden\-layer AUROC profile \(L1–L28\) and the full\-data profile\. A high correlation means that layers relatively stronger or weaker in the full\-data profile show similar relative deviations under the subset; it measures preservation of the cross\-layer structure rather than equality of absolute AUROCs or exact layer ranks\. As shown in Figure[9](https://arxiv.org/html/2607.28077#A3.F9), the agreement fluctuates for very small subsets, rises sharply by 2,048 prompts, and remains stable thereafter\. With 2,048 prompts \(11\.4% of the complete pool\), the correlations reach 0\.940 and 0\.961 for the 1\.5B and 7B models, while the selected layers remain within 0\.005 AUROC of the corresponding full\-data optima\. Thus, 2,048 rollout\-labeled prompts suffice in this diagnostic to make a near\-optimal layer choice\. The main experiments use the full\-data maxima \(L22 and L26\), fixed throughout RL training\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x9.png)Figure 9:Sample efficiency of layer calibration\. Pearsonrrcompares the layer\-wise AUROC values estimated from each subset with those obtained using all 17,917 prompts;r=1r=1indicates the same cross\-layer pattern\. Representative calibration sizes are shown, and the vertical line marks 2,048 prompts\.
#### Number of NeighborsKK\.

Using the same offline AUROC setup, we varyKKto assess whether the latent\-neighbor signal depends critically on neighborhood size and whether a single shared setting can be used across model scales\. As shown in Figure[10](https://arxiv.org/html/2607.28077#A3.F10), AUROC improves as the neighborhood expands beyond very small values ofKK, remains stable over a broad intermediate range, and declines mildly for overly large neighborhoods, particularly for the 7B model\. This trend is consistent with a locality–stability trade\-off: very small neighborhoods provide limited evidence for the estimate, whereas excessively large neighborhoods may dilute local information with less related prompts\. Within this exploratory diagnostic, the broad plateau indicates that the latent\-neighbor signal is not highly sensitive to the exact choice ofKKand supportsK=64K=64as a robust shared default without per\-model tuning\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x10.png)Figure 10:Sensitivity of non\-trivial\-prompt prediction AUROC to the number of nearest neighborsKKfor Qwen2\.5\-Math\-1\.5B and Qwen2\.5\-Math\-7B on DAPO\-Math\-17K\. For eachKK, the representation layer with the highest AUROC is selected independently, so the curves form exploratory best\-layer envelopes rather than fixed\-layer comparisons\. The dashed vertical line marks the default settingK=64K=64\.
#### Target Non\-Trivial Ratioτ\\tau\.

We varyτ∈\{0\.80,0\.85,0\.90,0\.95\}\\tau\\in\\\{0\.80,0\.85,0\.90,0\.95\\\}to assess sensitivity to the shared main\-experiment settingτ=0\.90\\tau=0\.90\. Across the tested range, all settings yield broadly competitive Overall Score trajectories at both model scales, indicating that LEEPS is not highly sensitive to the exact choice ofτ\\tau\. Increasingτ\\taufrom0\.800\.80to0\.900\.90raises the realized non\-trivial ratio, whereasτ=0\.95\\tau=0\.95provides little further increase and can reduce performance\. Althoughτ=0\.90\\tau=0\.90attains the highest peak scores, the other settings remain competitive, suggesting a broad effective range rather than a narrowly tuned optimum\. Overall,τ\\taucontrols the balance between exploiting known non\-trivial prompts and exploring uncertain prompts\.

![Refer to caption](https://arxiv.org/html/2607.28077v1/x11.png)Figure 11:Sensitivity of LEEPS to the target non\-trivial ratioτ\\tauon Qwen2\.5\-Math\-1\.5B and Qwen2\.5\-Math\-7B\. Panels \(a\)–\(b\) show the Overall Score, while Panels \(c\)–\(d\) show the selected\-batch non\-trivial ratio using a 7\-step centered moving average\. The defaultτ=0\.90\\tau=0\.90is highlighted, and all curves are shown from training step 20 for clarity\.![Refer to caption](https://arxiv.org/html/2607.28077v1/x12.png)Figure 12:Extended training\-step comparison between GRPO and LEEPS\. LEEPS is shown through the model\-specific ranges used in the main experiments \(280 steps for 1\.5B and 180 steps for 7B\), whereas GRPO is extended to 600 and 500 steps, respectively\. Dashed horizontal lines indicate a common performance target: the best LEEPS score for the 1\.5B model and the best GRPO score for the 7B model\. Dotted vertical lines mark the first step at which each method reaches the target\. LEEPS achieves1\.93×1\.93\\timesand1\.80×1\.80\\timeshigher training\-step efficiency on the 1\.5B and 7B models, respectively\.

### Extended Training Efficiency against GRPO

LEEPS achieves1\.93×1\.93\\timesand1\.80×1\.80\\timeshigher training\-step efficiency than GRPO on the 1\.5B and 7B models, respectively, as shown in Figure[12](https://arxiv.org/html/2607.28077#A3.F12)\. On the 1\.5B model, LEEPS reaches an overall score of 39\.16 at step 270, whereas GRPO requires 520 steps to reach the same level\. On the 7B model, LEEPS surpasses the best GRPO score of 46\.47 at step 150, while GRPO reaches this score at step 270\. The extended trajectories further show that GRPO improves only gradually at the later stage on the 1\.5B model, while its 7B performance plateaus around 45–46 and declines after reaching its peak\. Because GRPO and LEEPS generate the same number of rollouts per training step, the efficiency factors also apply to their rollout budgets\. These observations indicate that LEEPS reaches strong performance earlier through more effective prompt selection\.

## Appendix DData Examples

Below, we present one DAPO\-Math\-17K training example and one example from each OOD benchmark\. The OOD examples reproduce the normalized choice order used for evaluation\. These benchmarks already provide answer candidates: GPQA\-Diamond supplies one correct answer and three distractors, ARC\-Challenge supplies labeled choices, and MMLU\-Pro supplies an option list\. Thus, no open\-ended question is converted into multiple\-choice form\.

DAPO\-Math\-17K Data ExampleQuestion:What is the tens digit of520055^\{2005\}?Reference Answer:22\.
GPQA\-Diamond Data ExampleQuestion:Which of the following \(effective\) particles is not associated with a spontaneously\-broken symmetry?Choices:\(A\) Phonon\(B\) Skyrmion\(C\) Pion\(D\) MagnonAnswer:\(B\) Skyrmion\.
ARC\-Challenge Data ExampleQuestion:An astronomer observes that a planet rotates faster after a meteorite impact\. Which is the most likely effect of this increase in rotation?Choices:\(A\) Planetary density will decrease\.\(B\) Planetary years will become longer\.\(C\) Planetary days will become shorter\.\(D\) Planetary gravity will become stronger\.Answer:\(C\) Planetary days will become shorter\.
MMLU\-Pro Data ExampleQuestion:The symbol for antimony isChoices:\(A\) Am\(B\) As\(C\) Ab\(D\) Au\(E\) Fe\(F\) Ag\(G\) An\(H\) At\(I\) W\(J\) SbAnswer:\(J\) Sb\.