最难的题，反而最教不会模型？GRPO 的死区就在这里：一组 rollout 全错，组内优势归零，梯度也随之消失。

https://arxiv.org/abs/2607.27787

LSPO 给这些“悬崖题”临时装上 LoRA：用标准推导短暂 SFT，再采样成功轨迹，经重要性采样校正后并回 RL batch；最终只更新基础模型，脚手架即用即弃。单一 1.5B、DeepMath 配置下，16 项 15 胜 1 平，均值提升 3.8 点；代价是必须有完整标准推导，泛化仍待验证。

A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Source: https://arxiv.org/html/2607.27787

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Abstract

Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on “cliff” prompts—those on which every sampled rollout in a group fails—the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model’s capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated overn=5n{=}5paired seeds per arm at a matched10001000-step reporting horizon, LSPO’s55-seed mean matches or beats a DAPO baseline onall1616(benchmark,pass@kk) cells(1515strict wins and one exact tie), with gains of up to+10.7+10.7points on AIME24/pass@4,+6.7+6.7points on AIME24 and AIME26 atpass@16, and+2.4+2.4points on MATH500/pass@1; averaged over the1616cells the improvement is+3.8+3.8points.

1Introduction

Reinforcement learning from verifiable rewards (RLVR) has become the dominant recipe for eliciting mathematical reasoning from large language models(Shaoet al.,2024; DeepSeek-AIet al.,2025). Group Relative Policy Optimization (GRPO)(Shaoet al.,2024)and its scaled-up successor DAPO(Yuet al.,2025)dispense with a learned critic by sampling a group ofKKrollouts per prompt and normalizing each rollout’s reward against the group mean to form an advantage. This group-relative construction has a structural blind spot. On acliffprompt—one where allKKsampled rollouts are wrong, so the group’s total reward is zero—every rollout carries the same reward, the within-group advantage is identically zero, and the policy-gradient contribution of that prompt vanishes exactly. The model therefore receives no learning signal from precisely the prompts at the frontier of its capability: the problems it cannot yet solve are the ones GRPO is structurally unable to learn from. As training proceeds and the easy prompts are mastered, the unsolved tail accumulates at this frontier, and an increasing share of the batch produces no gradient.

Several lines of work attack this regime, and they cleave naturally into two groups. The first targets the cliff directly, recovering the missing signal from ground-truth solutions (Section2.4). The second group combines RL with a low-rank adapter, though not aimed at the cliff problem specifically. Tina(Wanget al.,2025)trains a LoRA adapter(Huet al.,2021)by RL and ships the adapted weights as the final model. BRIDGE(Chenet al.,2025)fuses supervised and RL gradients in the base-model update and updates a LoRA teacher using a cooperative-gain meta-objective derived from a bilevel formulation.

We introduceLoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that wraps an existing policy-gradient RL loop and recovers gradient on cliff prompts without modifying the loss. On each step, LSPO detects the all-zero-reward cliffs, attaches a small low-rank adapter and runs a brief supervised step on the cliff prompts’ ground-truth solutions (updating the adapter only, base frozen), re-samples the cliffs with the adapter active to restore within-group reward variance, then splices the now-successful rollouts back into the RL batch and backpropagates the policy-gradient loss into the base alone. What distinguishes LSPO from prior adapter-RL methods is three design choices acting together: asampling-time low-rank scaffold— the adapter is a transient proposal distribution (never the deliverable), kept low-rank and fine-tuned for only a handful of steps, the intent being to keep the importance-sampling correction well-conditioned rather than in the high-variance regime of further-off-policy proposals;strict gradient routingvia a two-optimizer split that sends the supervised gradient to the adapter only and the RL gradient to the base only (no bilevel or meta-objective coupling); andsplice-and-discard, in which the adapter is reused only as an importance-sampling proposal for the spliced cliff rows and is stripped at checkpoint. The supervised signal therefore reaches the base only indirectly, through adapter-elicited rollouts, and the deliverable is a base-only model that carries no adapter weights. Figure1summarises one training step.

Our contributions are as follows:

• •Method.We introduce LSPO, a sampling-time low-rank scaffold that recovers RL gradient on zero-reward cliff prompts via a two-optimizer split (adapter trained by supervision only, base trained by RL only) and a splice-and-discard pipeline with an importance-sampling correction, yielding a base-only deliverable (Section3).
• •Headline empirical result.On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B(DeepSeek-AIet al.,2025), overn=5n{=}5paired seeds per arm at a matched10001000-step reporting horizon, the55-seed mean of LSPO (per-iteration variant) matches or beats a DAPO baseline onall1616(benchmark, pass@kk) cells—1515strict wins and one exact tie—across MATH500 and AIME24/25/26. The largest gains are+10.7+10.7points on AIME24/pass@4 and+6.7+6.7points on both AIME24 and AIME26 at pass@16; averaged over the1616cells the improvement is+3.8+3.8points. Notably LSPO improves pass@1 on every benchmark (+1.3+1.3to+4.7+4.7points), so the gain is not merely a broadening of the sampled distribution (Section4).
• •Cliff-conversion efficacy.We measure the mechanism directly: across the55runs the scaffold converts∼43%{\sim}43\%of otherwise-zero-gradient cliff groups into groups carrying usable policy-gradient signal, with a tight per-seed spread (Section4.3).

2Related Work and Background

2.1RL with verifiable rewards: GRPO and the DAPO recipe

Reinforcement learning with verifiable rewards (RLVR) trains a policyπθ\pi_{\theta}to generate a solutionyyto a promptxxand assigns a binary outcome rewardr​(x,y)∈{0,1}r(x,y)\in\{0,1\}that is11iff a verifier accepts the final answer(DeepSeek-AIet al.,2025). Group Relative Policy Optimization (GRPO)(Shaoet al.,2024)removes the value network used by PPO(Schulmanet al.,2017)by drawing a group ofGGrollouts{y1,…,yG}\{y_{1},\dots,y_{G}\}per prompt and normalizing rewards within the group to form the advantageAi^=(ri−μ)/σ\widehat{A_{i}}=(r_{i}-\mu)/\sigma, whereμ\muandσ\sigmaare the group reward mean and standard deviation. The policy is then updated with the PPO-style clipped surrogate

LGRPO=−𝔼​[min⁡(ρt​A^,clip​(ρt,1−εlow,1+εhigh)​A^)],ρt=πθ​(at∣st)πold​(at∣st),L_{\mathrm{GRPO}}=-\,\mathbb{E}\!\left[\min\!\Big(\rho_{t}\widehat{A},\;\mathrm{clip}(\rho_{t},\,1-\varepsilon_{\mathrm{low}},\,1+\varepsilon_{\mathrm{high}})\,\widehat{A}\Big)\right],\qquad\rho_{t}=\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\mathrm{old}}(a_{t}\mid s_{t})},(1)whereρt\rho_{t}is the importance ratio between the current policy and the sampling (behavior) policy.

2.2The cliff problem

The group-normalized advantage in Eq. (1) creates a sharp failure mode. For a given promptxxthe outcome falls into one of three cases: (i) allGGrollouts succeed, so every reward equals the group mean and all advantages are zero; (ii) the rollouts are mixed, yielding positive advantages for the successes and negative for the failures — the standard learning regime; or (iii) allGGrollouts fail, so the group reward sum is zero,σ\sigmais zero, and every within-group advantage is identically zero. Case (iii) — acliffprompt, the “learning cliff” ofZhanget al.(2026)— contributesnogradient to the policy update. These are precisely the prompts at the frontier of the model’s capability: too hard for any of theGGsamples to solve, yet exactly where a learning signal is most needed. Under standard GRPO the cliff boundary can therefore only advance indirectly, through weight sharing as the model learns on nearby intermediate-difficulty prompts; there is no direct gradient on the cliffs themselves.

2.3Low-rank adapters

Low-Rank Adaptation (LoRA)(Huet al.,2021)freezes the pretrained weight matrixW0W_{0}and learns a low-rank updateΔ​W=B​A\Delta W=BAwithB∈ℝd×rB\in\mathbb{R}^{d\times r},A∈ℝr×kA\in\mathbb{R}^{r\times k}and rankr≪min⁡(d,k)r\ll\min(d,k), so that the adapted forward pass computes(W0+B​A)​x(W_{0}+BA)x. Because onlyAAandBBare trainable, adaptation is parameter-efficient and the adapter can be attached, detached, or reset cheaply. LSPO exploits exactly this property: it uses a small, short-lived LoRA adapter purely as a sampling-time mechanism, never shipping it.

2.4Related work

Rescuing cliffs with a scaffold.

A cliff group can be repaired in place, by synthesising a non-zero advantage through entropy-modulated shaping or a confidence-weighted penalty on wrong answers(Leet al.,2026; Fenget al.,2025); this is far cheaper than LSPO, but on an all-fail group it can only push probability massawayfrom the observed failures, never toward a success. Methods that instead rescue the cliff—LSPO among them—re-sample the failed prompt under privileged guidance and return the successes to the batch. The move predates RLVR: STaR(Zelikmanet al.,2022)recovered failed problems by conditioning on the ground-truth answer, within a generate–filter–finetune lineage that consumes the recovered trajectory as a cross-entropy target rather than an RL sample. The closest of these hints in the context and corrects the importance ratio back to the hint-free prompt(Nathet al.,2025). LSPO’s scaffold is a weight perturbation rather than a prompt, so its ratio compares two parameter settings; and it splices only the verifier-passing rows, leaving the rest of the group on-policy, where that method replaces the group wholesale. The others differ mainly in how the hint is constructed—tiered from abstract concepts to concrete steps(Zhanget al.,2026), taken as a prefix of the ground-truth trace(Liuet al.,2025; Zhanget al.,2025), generated by the policy itself(Chenet al.,2026; Liaoet al.,2026), or produced by a separate hinter policy(Xiaet al.,2026); all place the scaffold in the prompt, so answer leakage into the sampled trajectory is a live concern, which for LSPO it structurally is not. LatentRevise also fits its scaffold rather than writing it: a soft prefix in input-embedding space, optimised against the gold answer and then discarded(Guoet al.,2026). The prefix is still an input, and refitting it per prompt against that prompt’s own answer is a more direct privileged channel than LSPO’s single low-rank update shared across the whole cliff set.

Off-policy rows and privileged supervision.

POPO substitutes a variance-bearing group drawn from a prioritised replay buffer under a decoupled importance correction(Maoet al.,2026); unlike replay, LSPO can help on prompts the policy has never solved. ZPPO argues that injecting a teacher’s response into the policy gradient breaks the on-policy assumption, and so places that response in the prompt instead, as an anonymised candidate the student must pick out(Leeet al.,2026). LSPO reaches the same conclusion by another route: the ground-truth solution trains only the adapter, and what enters the base’s gradient is the model’s own rollout, importance-corrected. LUFFY imports off-policy teacher traces into RLVR directly(Yanet al.,2025)—the further-off-policy regime that Section3argues against on variance grounds.

Where the supervision lands.

Hybrid distillation / privileged self-distillation (HDPO) adds a student–teacher divergence term to the RL objective, so that cliff prompts contribute a supervised distillation signal optimised jointly with the policy gradient on the model’s own parameters(Ding,2026); ReLIFT alternates phases of full-model supervised fine-tuning on hard-prompt ground-truth solutions with phases of RL, supplying cliff signal by overwriting the base weights with expert traces(Maet al.,2025). In both, a supervised loss is applied to the base’s own parameters—jointly with the policy gradient in the first case, in alternating phases in the second. LSPO applies none: the RL objective is left exactly as it was, the intervention acts only on the proposal distribution, and the supervision terminates in an adapter that is discarded. In BRIDGE neither signal is confined to one weight set: the base takes a fused SFT-plus-RL gradient, and the adapter is updated under a bilevel cooperative-gain objective(Chenet al.,2025). LSPO routes the two to disjoint parameter sets with no meta-objective, maintains two weight sets rather than three, and activates only on cliff prompts. Where Tina’s adapter is trained by RL and shipped as the final model(Wanget al.,2025), LSPO’s receives only supervised gradient and is discarded.

3Method: LoRA Scaffolded Policy Optimization

Figure 1:The LSPO cliff-rescue mechanism.(1)On a cliff promptpp, allKKrollouts from the base policyπ\pifail (red); the group reward sum is zero and standard GRPO has no learning signal.(2)A trainable LoRA adapterϕLoRA\phi_{\text{LoRA}}is composed with the (frozen) baseπ\piand fitted via SFT on the dataset’s ground-truth trajectoryTT.(3)Under the composed policyπ+ϕLoRA\pi+\phi_{\text{LoRA}}, some rollouts on the same prompt now pass the verifier (green): the cliff is surpassed. Successful rollouts are spliced into the GRPO batch as positive examples.(4)The GRPO step runs on the same batch with the LoRA branch scale set to 0 (dashed grey), so the RL gradient (purple, dotted) lands onπ\pionly;ϕLoRA\phi_{\text{LoRA}}receives no RL gradient and is reset before the next step.LSPO is a sampling-time mechanism that wraps an existing policy-gradient RL loop (here GRPO(Shaoet al.,2024)with a DAPO-style recipe(Yuet al.,2025)) and recovers gradient oncliffprompts—prompts on which every sampled rollout in the group scores zero reward, so the group-relative advantage, and hence the policy gradient on that prompt, is identically zero. The core idea is to attach a small low-rank adapter(Huet al.,2021)as atransient scaffold: it is supervised-fine-tuned on the cliff prompts’ own ground-truth solutions, used only to re-sample those prompts, and then discarded. Successful adapter-elicited rollouts are spliced back into the RL batch so that the within-group reward variance—and the RL gradient—is restored, while the gradient that actually updates the deliverable flows into the base model alone.

3.1Algorithm

Figure1illustrates the mechanism and Algorithm1states the per-step loop. Each RL step:(A)sampleKKcompletions per prompt from the base and flag as cliffs those whose group scores zero total reward;(B)attach a LoRA adapter and run a brief supervised step on the cliff prompts’ ground-truth solutions, base frozen;(C)re-sample the cliff prompts with the adapter on;(D)swap each successful re-rollout into its failed counterpart’s slot, leaving every other row untouched;(E)apply the RL update to the base alone and discard the adapter.

The swap in (D) is deliberately minimal: a failed re-rollout carries no signal the group does not already have from its base counterpart, so admitting it would add off-policy mass without adding information. Advantages are recomputed on the spliced batch, and each spliced row’s base-plus-adapter probability is retained for the importance correction (Section3.3).

Algorithm 1LoRA Scaffolded Policy Optimization (per-step loop)0:Base policy

πθ\pi_{\theta}, prompt set

𝒳\mathcal{X}, ground truth

{y∗}\{y^{*}\}, reward

RR, rollouts per prompt

KK, clip bounds

(εlow,εhigh)(\varepsilon_{\mathrm{low}},\varepsilon_{\mathrm{high}}), RL optimizer

OptRL\mathrm{Opt}_{\mathrm{RL}}(updates

θ\theta), SFT optimizer

OptSFT\mathrm{Opt}_{\mathrm{SFT}}(updates adapter

ϕ\phi)

1:foreach RL training stepdo

2:Sample batch

B⊂𝒳B\subset\mathcal{X}; for each

x∈Bx\in Bdraw

KKrollouts

y(k)∼πθ(⋅∣x)y^{(k)}\sim\pi_{\theta}(\cdot\mid x)and record per-row sampler density

πsamp←πθ\pi_{\mathrm{samp}}\leftarrow\pi_{\theta} 3:Score every rollout with

RR;

𝒞←{x∈B:∑kR​(x,y(k))=0}\mathcal{C}\leftarrow\{x\in B:\textstyle\sum_{k}R(x,y^{(k)})=0\}// cliffs: zero RL gradient

4:if

𝒞≠∅\mathcal{C}\neq\emptysetthen

5:Attach low-rank adapter

ϕ\phito

πθ\pi_{\theta}(base frozen);

ϕ←OptSFT\phi\leftarrow\mathrm{Opt}_{\mathrm{SFT}}on

{(x,y∗):x∈𝒞}\{(x,y^{*}):x\in\mathcal{C}\}// supervised gradient toϕ\phionly

6:Re-sample

𝒞\mathcal{C}with adapter on:

y~∼πθ+ϕ(⋅∣x)\tilde{y}\sim\pi_{\theta+\phi}(\cdot\mid x);

𝒮←{(x,y~):R​(x,y~)=1}\mathcal{S}\leftarrow\{(x,\tilde{y}):R(x,\tilde{y})=1\} 7:Splice

𝒮\mathcal{S}into

BBin place of the failed cliff rollouts; for each

(x,y~)∈𝒮(x,\tilde{y})\in\mathcal{S}overwrite

πsamp←πθ+ϕ\pi_{\mathrm{samp}}\leftarrow\pi_{\theta+\phi}// IS denominator switches to base+adapter

8:endif

9:Compute group-normalized advantages

A^y\hat{A}_{y}on the (possibly spliced) batch

BB 10:Per-token IS ratio:

ρt​(θ)=πθ​(yt∣x,y<t)πsamp​(yt∣x,y<t)\rho_{t}(\theta)\;=\;\dfrac{\pi_{\theta}(y_{t}\mid x,y_{<t})}{\pi_{\mathrm{samp}}(y_{t}\mid x,y_{<t})}//πsamp\pi_{\mathrm{samp}}equalsπθ\pi_{\theta}for on-policy rows,πθ+ϕ\pi_{\theta+\phi}for spliced cliffs

11:

θ←OptRL\theta\leftarrow\mathrm{Opt}_{\mathrm{RL}}on the clipped surrogate (RL gradient to base only; adapter receives none):

ℒGRPO=−1T​∑(x,y)∈B∑tmin⁡(ρt​A^y,clip​(ρt,1−εlow,1+εhigh)​A^y)\mathcal{L}_{\mathrm{GRPO}}\,=\,-\frac{1}{T}\sum_{(x,y)\in B}\sum_{t}\min\!\big(\rho_{t}\,\hat{A}_{y},\,\mathrm{clip}(\rho_{t},\,1{-}\varepsilon_{\mathrm{low}},\,1{+}\varepsilon_{\mathrm{high}})\,\hat{A}_{y}\big)where

T=∑(x,y)∈B|y|T=\sum_{(x,y)\in B}|y|is the total tokens in the batch

12:Discard / reset adapter

ϕ\phi 13:endfor

14:returnbase model

πθ\pi_{\theta}// adapter never shipped

3.2Gradient routing

The defining architectural choice of LSPO is that gradient routing isstrictly disjoint. We maintain two optimizers. The supervised optimizerOptSFT\mathrm{Opt}_{\mathrm{SFT}}updates the low-rank adapter only, on the cross-entropy of the cliff ground-truth solutions (stage B). The RL optimizerOptRL\mathrm{Opt}_{\mathrm{RL}}updates the base only, on the GRPO surrogate over the spliced batch (stage E). No parameter is touched by both objectives: the base never receives supervised gradient, and the adapter never receives RL gradient. The supervised signal reaches the baseonly indirectly—through the adapter-elicited rollouts that are spliced into the RL batch and then learned by the ordinary policy gradient. Because a fresh adapter is fitted at every step, that signal always measures the gap from thecurrentbase to the cliff solutions, with no adapter state carried across steps.

3.3Importance-sampling correction

Spliced rows are off-policy: they were drawn fromπμ:=πθ+ϕ\pi_{\mu}:=\pi_{\theta+\phi}, not from the baseπθ\pi_{\theta}whose parameters the update modifies. The correction is a change of measure. The quantity we want is the base’s policy gradient, and

𝔼y∼πθ​[A​(y)​∇θlog⁡πθ​(y)]=𝔼y∼πμ​[πθ​(y)πμ​(y)​A​(y)​∇θlog⁡πθ​(y)],\mathbb{E}_{y\sim\pi_{\theta}}\!\left[A(y)\,\nabla_{\theta}\log\pi_{\theta}(y)\right]=\;\mathbb{E}_{y\sim\pi_{\mu}}\!\left[\frac{\pi_{\theta}(y)}{\pi_{\mu}(y)}\,A(y)\,\nabla_{\theta}\log\pi_{\theta}(y)\right],(2)so each spliced row must carry the weightπθ​(y)/πμ​(y)\pi_{\theta}(y)/\pi_{\mu}(y), which factorises per token into∏tπθ​(yt∣y<t)/πμ​(yt∣y<t)\prod_{t}\pi_{\theta}(y_{t}\mid y_{<t})/\pi_{\mu}(y_{t}\mid y_{<t}).

The implementation instead uses the corresponding per-token ratio in the clipped surrogate,ρt=πθ​(yt∣y<t)/πprev​(yt∣y<t)\rho_{t}=\pi_{\theta}(y_{t}\mid y_{<t})/\pi_{\mathrm{prev}}(y_{t}\mid y_{<t})(Schulmanet al.,2017; Shaoet al.,2024). For ordinary rowsπprev\pi_{\mathrm{prev}}is the base; for spliced rows we setπprev=πμ\pi_{\mathrm{prev}}=\pi_{\mu}, using the base-plus-adapter log-probabilities recorded at stage (D). No new estimator is introduced — the denominator is simply the distribution that generated the row.

Keeping the adapter small keeps that ratio well-conditioned. This is a variance motivation rather than an exactness guarantee. Because the implementation selects verifier-passing proposal rows and applies clipped per-token ratios rather than an unclipped sequence-product weight, we do not claim that its update is an unbiased sequence-level policy-gradient estimator or provide a convergence guarantee; formal characterization is deferred to future work.

4Experiments

4.1Setup

We evaluate LSPO on mathematical reasoning. The base model is DeepSeek-R1-Distill-Qwen-1.5B(DeepSeek-AIet al.,2025), a distilled reasoning model. Training uses DeepMath-103K(Heet al.,2025), a public collection of competition-style mathematics problems with verifiable final answers (HuggingFacezwhe99/DeepMath-103K). We validate on four held-out benchmarks: MATH500(Hendryckset al.,2021; Lightmanet al.,2024)and the AIME 2024, 2025, and 2026 competition sets (AIME24/25/26),111We use the publicly released problem sets from the Mathematical Association of America’s American Invitational Mathematics Examination for 2024, 2025, and 2026. Problems and official answer keys are available via the MAA archive and community wikis such ashttps://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions.reporting best pass@kkfork∈{1,4,8,16}k\in\{1,4,8,16\}. Validation runs every 10 training steps. pass@kkis computed by the first-kk-slice estimator (whether any of the firstkkof the1616validation rollouts passes the verifier); it is unbiased for pass@kkunder exchangeability of the rollouts, though higher-variance than the combinatorial estimator ofChenet al.(2021)which averages over all size-kksubsets. The estimator is applied identically to both arms.

Seeds and reporting horizon.

Both arms are trained with the same seed set{42,43,44,45,46}\{42,43,44,45,46\}, and all analysis is paired seed-by-seed. We fix a1000-step reporting horizon: runs are launched with a 1500-step budget, and every peak, comparison, and aggregate reported in this paper is computed over validation events at step≤1000{\leq}1000, identically for LSPO and DAPO. We do not characterize behavior beyond this horizon in this version.

All training is run on8×8\timesH100 GPUs. The base RL algorithm is GRPO(Shaoet al.,2024)configured with the DAPO recipe(Yuet al.,2025): Clip-Higher with asymmetric ratio clipping (εlow=0.20\varepsilon_{\text{low}}{=}0.20,εhigh=0.28\varepsilon_{\text{high}}{=}0.28), token-level policy-gradient loss, and overlong-reward shaping (a soft length penalty that engages past 7168 generated tokens up to the 8192-token generation limit). Dynamic sampling is disabled. Each step samples 32 prompts with 16 generations per prompt (512 rollouts/step). The base model is updated with AdamW(Loshchilov and Hutter,2019)at learning rate1​e−61\mathrm{e}{-}6; we apply no reference-policy KL penalty.

The LSPO scaffold uses a LoRA(Huet al.,2021)adapter of rank 16 (α=64\alpha{=}64) fine-tuned on cliff ground-truth solutions forsft_steps_per_new_cliff=4{=}4steps with a separate AdamW optimizer at learning rate2.5​e−52.5\mathrm{e}{-}5. Cliffs are groups whose total reward is zero; only successful adapter-elicited rollouts replace their failed counterparts (partial swap), and spliced rows reuse the base-plus-adapter log-probabilities as the importance-sampling denominator. The adapter receives supervised gradient only and the base receives RL gradient only, per Section3. Unless stated otherwise, LSPO denotes the per-iteration variant (a fresh adapter each step). Ourbaselineisidenticalin every respect—data, recipe, optimizer, hardware, and step budget—with the LSPO scaffold disabled, i.e. plain DAPO-style GRPO. This isolates the contribution of the scaffold.

4.2Main result: peak-vs-peak comparison

Table1reports the full 16-cell peak comparison at the matched reporting horizon (step≤1000{\leq}1000) withn=5n{=}5paired seeds for both arms. For each (run, benchmark,kk) we take the peak value over that run’s validation trace, then average across the five seeds. LSPO’s mean matches or beats DAPO’s onevery cell, with15 strict wins and one exact tie(AIME25/pass@8:46.0046.00vs.46.0046.00; at integer counts both arms recover6969of150150seed-problem pairs).

The advantage is largest on AIME24 (mean+4.7+4.7to+10.7+10.7points acrosskk, peaking at+10.66+10.66points at pass@4) and on AIME26/pass@16 (+6.67+6.67points). MATH500 gains are smaller in absolute terms (+0.4+0.4to+2.4+2.4points) because both arms approach the benchmark ceiling. Notably, LSPO improves pass@1 on every benchmark (+1.3+1.3to+4.7+4.7points): the improvement is not merely a broadening of the sampled distribution but a shift in single-sample accuracy. Averaged over all1616cells, LSPO scores59.9059.90against DAPO’s56.0856.08, a+3.82+3.82-point improvement.

Because peaks are taken per cell independently, the peak steps differ across cells, so Table1reports best-achievable-per-metric during training rather than the performance of any single checkpoint. Both arms are measured under identical protocol, so the paired comparison is unaffected.

Benchmarkpass@kkLSPO meanLSPO rangeDAPO meanDAPO rangeΔ\Delta(pts)(5 seeds)(5 seeds)[email protected]–87.2083.8883.20–[email protected]–95.2093.3693.00–[email protected]–96.6095.5695.40–[email protected]–97.6097.0896.80–[email protected]–40.0032.6630.00–33.33+4.67\mathbf{+4.67}[email protected]–63.3348.6746.67–53.33+10.66\mathbf{+10.66}[email protected]–70.0056.6756.67–56.67+8.66\mathbf{+8.66}[email protected]–73.3365.3363.33–70.00+6.67\mathbf{+6.67}[email protected]–30.0027.3426.67–[email protected]–43.3338.0036.67–43.33+4.00\mathbf{+4.00}[email protected]–50.0046.0043.33–[email protected]–56.6751.3350.00–[email protected]–33.3325.3323.33–26.67+4.67\mathbf{+4.67}[email protected]–43.3340.0036.67–[email protected]–53.3346.0043.33–[email protected]–60.0050.0046.67–53.33+6.67\mathbf{+6.67}Mean over 16 cells59.90—56.08—+3.82+3.82Table 1:Peak pass@kkat the matched10001000-step reporting horizon, averaged acrossn=5n{=}5paired seeds per arm, on DeepSeek-R1-Distill-Qwen-1.5B trained with DeepMath-103K (8×8\timesH100). LSPO is the per-iteration variant; DAPO is the identical recipe with the scaffold disabled. LSPO’s mean matches or beats DAPO’s on all1616cells (1515strict wins++11exact tie at AIME25/pass@8).Δ\Deltaentries≥+4\geq{+}4points are highlighted in the final column. Peak steps vary by cell. AIME sets haveN=30N{=}30problems (one problem≈3.33\approx 3.33points), so single-cell deltas are coarse; the pattern across cells is the reliable signal.

4.3Cliff-conversion efficacy

Table1measures the outcome; this section measures the mechanism. Across the55LSPO runs at step≤1000{\leq}1000,10,86410{,}864cliff groups were detected in total. The scaffold — four iterations of LoRA fitting on the cliff ground-truth solutions followed by re-sampling — surpassed approximately43%43\%of them, i.e. produced at least one verifier-passing rollout on4,6804{,}680groups that were originally0-of-KKfailures and therefore carried exactly zero policy gradient. The per-seed surpass rate is tight (41.341.3–45.5%45.5\%, standard deviation1.5%1.5\%).

Two details sharpen the interpretation. First, the per-rollout post-scaffold pass rate is only5.8%5.8\%(cliff-weighted mean across seeds), so the group-level43%43\%predominantly reflectsoneof theK=16K{=}16re-sampled rollouts succeeding rather than confident solving — which is all GRPO requires, since a single success restores within-group reward variance. Second, the per-step surpass rate is bimodal (31%31\%of cliff-bearing steps have rate0.00.0;21%21\%have rate1.01.0), so the aggregate43%43\%is a mean of extremes rather than a typical value. By construction the DAPO baseline converts0%0\%of its cliffs: the leave-one-out advantage zeroes those groups’ gradients outright.

5Limitations

Scope.

We evaluate one model (DeepSeek-R1-Distill-Qwen-1.5B), one dataset (DeepMath-103K), one recipe family (GRPO with the DAPO configuration), and one scaffold configuration (per-iteration variant, LoRA rank1616, four SFT steps). The scaffold’s behavior plausibly depends on adapter rank, SFT step count, splice variant, and the size of the dataset’s answer space — a small-integer answer space makes verifier-accepted guessing more likely than a large one — and we do not establish how the results transfer across these axes.

Ground-truth requirement.

LSPO requires ground-truth solutions (not merely final answers) for the cliff prompts, since the scaffold is fit by supervised learning on those derivations. This is available in DeepMath-103K but restricts applicability to datasets that ship worked solutions.

6Conclusion

We introduced LSPO (LoRA Scaffolded Policy Optimization), a sampling-time mechanism that recovers reinforcement-learning gradient on zero-reward “cliff” prompts—those frontier prompts where every sampled rollout fails, the within-group advantage is identically zero, and standard GRPO therefore provides no learning signal. LSPO attaches a small low-rank adapter, fine-tunes it only on the cliff prompts’ ground-truth solutions, re-samples the cliffs with the adapter active to restore within-group reward variance, and splices the resulting successes back into the RL batch. Two design choices keep the deliverable a clean base-only model: strictly disjoint gradient routing via a two-optimizer split (the supervised signal reaches the adapter only and the RL gradient reaches the base only, reusing the standard policy-ratio term as the importance-sampling correction for the spliced rows), and a splice-and-discard policy that strips the adapter at checkpoint so it is never shipped. Overn=5n{=}5paired seeds at a matched10001000-step reporting horizon, LSPO’s mean matches or beats a DAPO baseline on all1616peak pass@kkcells (1515strict wins and one exact tie), with gains up to+10.7+10.7points on AIME24/pass@4 and+3.8+3.8points averaged across the1616cells, and with improvements on pass@1 for every benchmark. The mechanism is directly measurable: roughly43%43\%of otherwise-zero-gradient cliff groups are converted into groups that carry usable policy-gradient signal. We view LSPO as evidence that the cliff problem can be addressed at sampling time, with the supervision localized in transient low-rank parameters rather than written irreversibly into the base.

LLM Usage Statement

The research idea underlying LSPO—using a transient, sampling-time low-rank scaffold to recover reinforcement-learning gradient on zero-reward cliff prompts—originated with the author. However, an AI language model (Claude, Anthropic) was used extensively throughout this project in ways that go beyond minor writing assistance. Specifically: (1) the paper text, including the related-work survey, the method exposition, and the mechanism-analysis and discussion sections, was substantially drafted and edited with LLM assistance; (2) the LLM was used as a research collaborator to stress-test explanations for the observed behavior, to assemble and check the related-work survey, and to position LSPO against prior methods; a second model (OpenAI Codex, GPT) was used as an independent verifier, recomputing every reported number from the raw training logs and auditing citations; and (3) the importance-sampling correction described in Section3.3— reusing the standard PPO/GRPO policy-ratio term with the base-plus-adapter sampling log-probabilities as the ratio’s denominator on spliced cliff rows — was proposed by the LLM during the method-design phase and adopted by the author. All experimental results (training runs, metric measurements, and rollout dumps) were produced by the author without LLM involvement.

References

• J. C. Chen, B. X. Peng, P. K. Choubey, K. Huang, J. Zhang, M. Bansal, and C. Wu (2026)Nudging the boundaries of LLM reasoning.InInternational Conference on Learning Representations (ICLR),Note:arXiv:2509.25666External Links:LinkCited by:§2.4.
• L. Chen, X. Han, L. Shen, J. Bai, and K. Wong (2025)Beyond two-stage training: cooperative SFT and RL for LLM reasoning.arXiv preprint arXiv:2509.06948.External Links:LinkCited by:§1,§2.4.
• M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman,et al.(2021)Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374.External Links:LinkCited by:§4.1.
• DeepSeek-AI, D. Guo, D. Yang,et al.(2025)DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948.External Links:LinkCited by:2nd item,§1,§2.1,§4.1.
• K. Ding (2026)HDPO: hybrid distillation policy optimization via privileged self-distillation.arXiv preprint arXiv:2603.23871.External Links:LinkCited by:§2.4.
• Y. Feng, P. Jain, A. Hartshorn, Y. Duan, and J. Kempe (2025)Don’t waste mistakes: leveraging negative RL-groups via confidence reweighting.arXiv preprint arXiv:2510.08696.External Links:LinkCited by:§2.4.
• Y. Guo, X. Han, Q. Jia, G. Zhai, and J. Bai (2026)LatentRevise: learning from zero-hit reasoning.arXiv preprint arXiv:2606.29938.External Links:LinkCited by:§2.4.
• Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, Z. Zhang, R. Wang, Z. Tu, H. Mi, and D. Yu (2025)DeepMath-103K: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning.arXiv preprint arXiv:2504.11456.External Links:LinkCited by:§4.1.
• D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021)Measuring mathematical problem solving with the MATH dataset.InNeural Information Processing Systems,External Links:LinkCited by:§4.1.
• E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021)LoRA: low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685.External Links:LinkCited by:§1,§2.3,§3,§4.1.
• T. V. Le, M. Jeon, K. Vu, V. Lai, and E. Yang (2026)No prompt left behind: exploiting zero-variance prompts in LLM reinforcement learning via entropy-guided advantage shaping.InInternational Conference on Learning Representations (ICLR),Note:arXiv:2509.21880External Links:LinkCited by:§2.4.
• B. Lee, X. Lu, S. Diao, M. Kang, S. Muralidharan, K. Sapra, A. Tao, P. Molchanov, Y. Choi, Y. F. Wang, and R. Hachiuma (2026)Zone of proximal policy optimization: teacher in prompts, not gradients.arXiv preprint arXiv:2606.18216.External Links:LinkCited by:§2.4.
• B. Liao, H. Dong, X. Xu, C. Monz, and J. Bian (2026)Self-hinting language models enhance reinforcement learning.arXiv preprint arXiv:2602.03143.External Links:LinkCited by:§2.4.
• H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2024)Let’s verify step by step.InInternational Conference on Learning Representations,External Links:LinkCited by:§4.1.
• Z. Liu, C. Gong, X. Fu, Y. Liu, R. Chen, S. Hu, S. Zhang, R. Liu, Q. Zhang, and D. Tu (2025)GHPO: adaptive guidance for stable and efficient LLM reinforcement learning.arXiv preprint arXiv:2507.10628.External Links:LinkCited by:§2.4.
• I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization.InInternational Conference on Learning Representations,External Links:LinkCited by:§4.1.
• L. Ma, H. Liang, M. Qiang, L. Tang, X. Ma, Z. H. Wong, J. Niu, C. Shen, R. He, Y. Li, B. Cui, and W. Zhang (2025)Learning what reinforcement learning can’t: interleaved online fine-tuning for hardest questions.arXiv preprint arXiv:2506.07527.External Links:LinkCited by:§2.4.
• Y. Mao, Y. Qu, Q. Wang, H. Zou, and X. Ji (2026)RLVR without ineffective samples: group prioritized off-policy optimization for LLM reasoning.arXiv preprint arXiv:2606.01281.External Links:LinkCited by:§2.4.
• V. Nath, E. Lau, A. Gunjal, M. Sharma, N. Baharte, and S. Hendryx (2025)Adaptive guidance accelerates reinforcement learning of reasoning models.arXiv preprint arXiv:2506.13923.External Links:LinkCited by:§2.4.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017)Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.External Links:LinkCited by:§2.1,§3.3.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y.K. Li, Y. Wu, and D. Guo (2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.External Links:LinkCited by:§1,§2.1,§3.3,§3,§4.1.
• S. Wang, J. Asilis, Ö. F. Akgül, E. B. Bilgin, O. Liu, and W. Neiswanger (2025)Tina: tiny reasoning models via LoRA.arXiv preprint arXiv:2504.15777.External Links:LinkCited by:§1,§2.4.
• Y. Xia, C. Xu, Z. Yao, J. McAuley, and Y. He (2026)Learning to hint for reinforcement learning.arXiv preprint arXiv:2604.00698.External Links:LinkCited by:§2.4.
• J. Yan, Y. Li, Z. Hu, Z. Wang, G. Cui, X. Qu, Y. Cheng, and Y. Zhang (2025)Learning to reason under off-policy guidance.InAdvances in Neural Information Processing Systems (NeurIPS),Note:arXiv:2504.14945External Links:LinkCited by:§2.4.
• Q. Yu, Z. Zhang, R. Zhu, Y. Yuan,et al.(2025)DAPO: an open-source LLM reinforcement learning system at scale.arXiv preprint arXiv:2503.14476.External Links:LinkCited by:§1,§3,§4.1.
• E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman (2022)STaR: bootstrapping reasoning with reasoning.InAdvances in Neural Information Processing Systems (NeurIPS),Note:arXiv:2203.14465External Links:LinkCited by:§2.4.
• X. Zhang, S. Wu, Y. Zhu, H. Tan, S. Yu, Z. He, and J. Jia (2026)Scaf-GRPO: scaffolded group relative policy optimization for enhancing LLM reasoning.InInternational Conference on Learning Representations (ICLR),Note:arXiv:2510.19807External Links:LinkCited by:§2.2,§2.4.
• X. Zhang, Z. Huang, Y. Li, C. Ni, J. Chen, and S. Oymak (2025)BREAD: branched rollouts from expert anchors bridge SFT & RL for reasoning.InAdvances in Neural Information Processing Systems (NeurIPS),Note:arXiv:2506.17211External Links:LinkCited by:§2.4.