The hardest problems are paradoxically the ones models learn the least from? GRPO’s dead zone lies right here: when a whole group of rollouts is wrong, the within-group advantage goes to zero, and the gradient vanishes with it.

https://arxiv.org/abs/2607.27787

LSPO temporarily attaches a LoRA scaffold to these “cliff problems”: it briefly fine-tunes with standard derivations, then samples successful trajectories, applies an importance-sampling correction, and splices them back into the RL batch; only the base model is ultimately updated, and the scaffold is use-and-discard. In a single 1.5B, DeepMath configuration, it wins 15 and ties 1 across 16 cells, with a mean gain of 3.8 points; the cost is that complete standard solutions must be available, and generalization remains to be verified.

A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Source: https://arxiv.org/html/2607.27787

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

Abstract

Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on “cliff” prompts—those on which every sampled rollout in a group fails—the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model’s capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated overn=5n{=}5paired seeds per arm at a matched10001000-step reporting horizon, LSPO’s55-seed mean matches or beats a DAPO baseline onall1616(benchmark,pass@kk) cells(1515strict wins and one exact tie), with gains of up to+10.7+10.7points on AIME24/pass@4,+6.7+6.7points on AIME24 and AIME26 atpass@16, and+2.4+2.4points on MATH500/pass@1; averaged over the1616cells the improvement is+3.8+3.8points.

1Introduction

Reinforcement learning from verifiable rewards (RLVR) has become the dominant recipe for eliciting mathematical reasoning from large language models(Shaoet al.,2024 (https://arxiv.org/html/2607.27787#bib.bib5); DeepSeek-AIet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib9)). Group Relative Policy Optimization (GRPO)(Shaoet al.,2024 (https://arxiv.org/html/2607.27787#bib.bib5))and its scaled-up successor DAPO(Yuet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib6))dispense with a learned critic by sampling a group ofKKrollouts per prompt and normalizing each rollout’s reward against the group mean to form an advantage. This group-relative construction has a structural blind spot. On acliffprompt—one where allKKsampled rollouts are wrong, so the group’s total reward is zero—every rollout carries the same reward, the within-group advantage is identically zero, and the policy-gradient contribution of that prompt vanishes exactly. The model therefore receives no learning signal from precisely the prompts at the frontier of its capability: the problems it cannot yet solve are the ones GRPO is structurally unable to learn from. As training proceeds and the easy prompts are mastered, the unsolved tail accumulates at this frontier, and an increasing share of the batch produces no gradient.

Several lines of work attack this regime, and they cleave naturally into two groups. The first targets the cliff directly, recovering the missing signal from ground-truth solutions (Section2.4 (https://arxiv.org/html/2607.27787#S2.SS4)). The second group combines RL with a low-rank adapter, though not aimed at the cliff problem specifically. Tina(Wanget al.,2025 (https://arxiv.org/html/2607.27787#bib.bib1))trains a LoRA adapter(Huet al.,2021 (https://arxiv.org/html/2607.27787#bib.bib8))by RL and ships the adapted weights as the final model. BRIDGE(Chenet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib3))fuses supervised and RL gradients in the base-model update and updates a LoRA teacher using a cooperative-gain meta-objective derived from a bilevel formulation.

We introduceLoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that wraps an existing policy-gradient RL loop and recovers gradient on cliff prompts without modifying the loss. On each step, LSPO detects the all-zero-reward cliffs, attaches a small low-rank adapter and runs a brief supervised step on the cliff prompts’ ground-truth solutions (updating the adapter only, base frozen), re-samples the cliffs with the adapter active to restore within-group reward variance, then splices the now-successful rollouts back into the RL batch and backpropagates the policy-gradient loss into the base alone. What distinguishes LSPO from prior adapter-RL methods is three design choices acting together: asampling-time low-rank scaffold— the adapter is a transient proposal distribution (never the deliverable), kept low-rank and fine-tuned for only a handful of steps, the intent being to keep the importance-sampling correction well-conditioned rather than in the high-variance regime of further-off-policy proposals;strict gradient routingvia a two-optimizer split that sends the supervised gradient to the adapter only and the RL gradient to the base only (no bilevel or meta-objective coupling); andsplice-and-discard, in which the adapter is reused only as an importance-sampling proposal for the spliced cliff rows and is stripped at checkpoint. The supervised signal therefore reaches the base only indirectly, through adapter-elicited rollouts, and the deliverable is a base-only model that carries no adapter weights. Figure1 (https://arxiv.org/html/2607.27787#S3.F1)summarises one training step.

Our contributions are as follows:

• •Method.We introduce LSPO, a sampling-time low-rank scaffold that recovers RL gradient on zero-reward cliff prompts via a two-optimizer split (adapter trained by supervision only, base trained by RL only) and a splice-and-discard pipeline with an importance-sampling correction, yielding a base-only deliverable (Section3 (https://arxiv.org/html/2607.27787#S3)).
• •Headline empirical result.On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B(DeepSeek-AIet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib9)), overn=5n{=}5paired seeds per arm at a matched10001000-step reporting horizon, the55-seed mean of LSPO (per-iteration variant) matches or beats a DAPO baseline onall1616(benchmark, pass@kk) cells—1515strict wins and one exact tie—across MATH500 and AIME24/25/26. The largest gains are+10.7+10.7points on AIME24/pass@4 and+6.7+6.7points on both AIME24 and AIME26 at pass@16; averaged over the1616cells the improvement is+3.8+3.8points. Notably LSPO improves pass@1 on every benchmark (+1.3+1.3to+4.7+4.7points), so the gain is not merely a broadening of the sampled distribution (Section4 (https://arxiv.org/html/2607.27787#S4)).
• •Cliff-conversion efficacy.We measure the mechanism directly: across the55runs the scaffold converts∼43%{\sim}43\%of otherwise-zero-gradient cliff groups into groups carrying usable policy-gradient signal, with a tight per-seed spread (Section4.3 (https://arxiv.org/html/2607.27787#S4.SS3)).

2Related Work and Background

2.1RL with verifiable rewards: GRPO and the DAPO recipe

Reinforcement learning with verifiable rewards (RLVR) trains a policyπθ\pi_{\theta}to generate a solutionyyto a promptxxand assigns a binary outcome rewardr(x,y)∈{0,1}r(x,y)\in\{0,1\}that is11iff a verifier accepts the final answer(DeepSeek-AIet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib9)). Group Relative Policy Optimization (GRPO)(Shaoet al.,2024 (https://arxiv.org/html/2607.27787#bib.bib5))removes the value network used by PPO(Schulmanet al.,2017 (https://arxiv.org/html/2607.27787#bib.bib7))by drawing a group ofGGrollouts{y1,…,yG}\{y_{1},\dots,y_{G}\}per prompt and normalizing rewards within the group to form the advantageAi^=(ri−μ)/σ\widehat{A_{i}}=(r_{i}-\mu)/\sigma, whereμ\muandσ\sigmaare the group reward mean and standard deviation. The policy is then updated with the PPO-style clipped surrogate

LGRPO=−E[min⁡(ρtA^,clip(ρt,1−εlow,1+εhigh)A^)],ρt=πθ(at∣st)πold(at∣st),L_{\mathrm{GRPO}}=-\,\mathbb{E}\!\left[\min\!\Big(\rho_{t}\widehat{A},\;\mathrm{clip}(\rho_{t},\,1-\varepsilon_{\mathrm{low}},\,1+\varepsilon_{\mathrm{high}})\,\widehat{A}\Big)\right],\qquad\rho_{t}=\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\mathrm{old}}(a_{t}\mid s_{t})},(1)whereρt\rho_{t}is the importance ratio between the current policy and the sampling (behavior) policy.

2.2The cliff problem

The group-normalized advantage in Eq. (1 (https://arxiv.org/html/2607.27787#S2.E1)) creates a sharp failure mode. For a given promptxxthe outcome falls into one of three cases: (i) allGGrollouts succeed, so every reward equals the group mean and all advantages are zero; (ii) the rollouts are mixed, yielding positive advantages for the successes and negative for the failures — the standard learning regime; or (iii) allGGrollouts fail, so the group reward sum is zero,σ\sigmais zero, and every within-group advantage is identically zero. Case (iii) — acliffprompt, the “learning cliff” ofZhanget al.(2026 (https://arxiv.org/html/2607.27787#bib.bib17))— contributesnogradient to the policy update. These are precisely the prompts at the frontier of the model’s capability: too hard for any of theGGsamples to solve, yet exactly where a learning signal is most needed. Under standard GRPO the cliff boundary can therefore only advance indirectly, through weight sharing as the model learns on nearby intermediate-difficulty prompts; there is no direct gradient on the cliffs themselves.

2.3Low-rank adapters

Low-Rank Adaptation (LoRA)(Huet al.,2021 (https://arxiv.org/html/2607.27787#bib.bib8))freezes the pretrained weight matrixW0W_{0}and learns a low-rank updateΔW=BA\Delta W=BAwithB∈Rd×rB\in\mathbb{R}^{d\times r},A∈Rr×kA\in\mathbb{R}^{r\times k}and rankr≪min⁡(d,k)r\ll\min(d,k), so that the adapted forward pass computes(W0+BA)x(W_{0}+BA)x. Because onlyAAandBBare trainable, adaptation is parameter-efficient and the adapter can be attached, detached, or reset cheaply. LSPO exploits exactly this property: it uses a small, short-lived LoRA adapter purely as a sampling-time mechanism, never shipping it.

2.4Related work

Rescuing cliffs with a scaffold.

A cliff group can be repaired in place, by synthesising a non-zero advantage through entropy-modulated shaping or a confidence-weighted penalty on wrong answers(Leet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib25); Fenget al.,2025 (https://arxiv.org/html/2607.27787#bib.bib26)); this is far cheaper than LSPO, but on an all-fail group it can only push probability massawayfrom the observed failures, never toward a success. Methods that instead rescue the cliff—LSPO among them—re-sample the failed prompt under privileged guidance and return the successes to the batch. The move predates RLVR: STaR(Zelikmanet al.,2022 (https://arxiv.org/html/2607.27787#bib.bib30))recovered failed problems by conditioning on the ground-truth answer, within a generate–filter–finetune lineage that consumes the recovered trajectory as a cross-entropy target rather than an RL sample. The closest of these hints in the context and corrects the importance ratio back to the hint-free prompt(Nathet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib16)). LSPO’s scaffold is a weight perturbation rather than a prompt, so its ratio compares two parameter settings; and it splices only the verifier-passing rows, leaving the rest of the group on-policy, where that method replaces the group wholesale. The others differ mainly in how the hint is constructed—tiered from abstract concepts to concrete steps(Zhanget al.,2026 (https://arxiv.org/html/2607.27787#bib.bib17)), taken as a prefix of the ground-truth trace(Liuet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib21); Zhanget al.,2025 (https://arxiv.org/html/2607.27787#bib.bib23)), generated by the policy itself(Chenet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib20); Liaoet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib18)), or produced by a separate hinter policy(Xiaet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib19)); all place the scaffold in the prompt, so answer leakage into the sampled trajectory is a live concern, which for LSPO it structurally is not. LatentRevise also fits its scaffold rather than writing it: a soft prefix in input-embedding space, optimised against the gold answer and then discarded(Guoet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib24)). The prefix is still an input, and refitting it per prompt against that prompt’s own answer is a more direct privileged channel than LSPO’s single low-rank update shared across the whole cliff set.

Off-policy rows and privileged supervision.

POPO substitutes a variance-bearing group drawn from a prioritised replay buffer under a decoupled importance correction(Maoet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib27)); unlike replay, LSPO can help on prompts the policy has never solved. ZPPO argues that injecting a teacher’s response into the policy gradient breaks the on-policy assumption, and so places that response in the prompt instead, as an anonymised candidate the student must pick out(Leeet al.,2026 (https://arxiv.org/html/2607.27787#bib.bib22)). LSPO reaches the same conclusion by another route: the ground-truth solution trains only the adapter, and what enters the base’s gradient is the model’s own rollout, importance-corrected. LUFFY imports off-policy teacher traces into RLVR directly(Yanet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib36))—the further-off-policy regime that Section3 (https://arxiv.org/html/2607.27787#S3)argues against on variance grounds.

Where the supervision lands.

Hybrid distillation / privileged self-distillation (HDPO) adds a student–teacher divergence term to the RL objective, so that cliff prompts contribute a supervised distillation signal optimised jointly with the policy gradient on the model’s own parameters(Ding,2026 (https://arxiv.org/html/2607.27787#bib.bib4)); ReLIFT alternates phases of full-model supervised fine-tuning on hard-prompt ground-truth solutions with phases of RL, supplying cliff signal by overwriting the base weights with expert traces(Maet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib2)). In both, a supervised loss is applied to the base’s own parameters—jointly with the policy gradient in the first case, in alternating phases in the second. LSPO applies none: the RL objective is left exactly as it was, the intervention acts only on the proposal distribution, and the supervision terminates in an adapter that is discarded. In BRIDGE neither signal is confined to one weight set: the base takes a fused SFT-plus-RL gradient, and the adapter is updated under a bilevel cooperative-gain objective(Chenet al.,2025 (https://arxiv.org/html/2607.27787#bib.bib3)). LSPO routes the two to disjoint parameter sets with no meta-objective, maintains two weight sets rather than three, and activates only on cliff prompts. Where Tina’s adapter is trained by RL and shipped as the final model(Wanget al.,2025 (https://arxiv.org/html/2607.27787#bib.bib1)), LSPO’s receives only supervised gradient and is discarded.

3Method: LoRA Scaffolded Policy Optimization

Refer to captionFigure 1:The LSPO cliff-rescue mechanism.(1)On a