GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity

# 1 Introduction
Source: [https://arxiv.org/html/2607.00152](https://arxiv.org/html/2607.00152)
GRPO, Dr\. GRPO, and DAPO Are Three Operations on One Number: The Group\-Standard\-Deviation Identity

Yong Yi Bay\*Kathleen A\. Yearick\*

PhD, University of Illinois at Urbana\-Champaign

††footnotetext:\*Equal contribution\. Correspondence:\{yongyibay, kallie\.a\.yearick\}@gmail\.com\.

Abstract

- Three of the most popular methods for training language models to reason look like three different tricks\. They are not\. All three adjust a single number:*standard deviation*, reflecting how much a prompt’s sampled answers disagree\. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong\. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree\. Group Relative Policy Optimization \(GRPO\) divides by this number, GRPO Done Right \(Dr\. GRPO\) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization \(DAPO\) discards the groups where it is zero\. Each is presented as its own fix, yet this paper proves they are three settings of one dial\. That dial is not cosmetic: for right\-or\-wrong rewards, the disagreement is exactly the size of the training update, the*group\-standard\-deviation identity*\. A split group teaches the most, while a unanimous group teaches nothing and falls silent\. The same result says which problems deserve the most weight and how many tries each one needs\. This paper confirms the intuition on a large real difficulty dataset \(Big\-Math\) and in a controlled training run\. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly\.

KeywordsGRPO⋅\\cdotreward normalization⋅\\cdotsilent groups⋅\\cdotdifficulty bias⋅\\cdotgroup size⋅\\cdotdynamic sampling⋅\\cdotRLVR⋅\\cdotLLM reasoning

Teaching a language model to reason starts with a step that looks wasteful: the same prompt is answered many times over, on purpose\. This happens during reinforcement learning, not when a user chats with the model\. The model produces a group of candidate answers, an external verifier marks each one as correct or incorrect, and the optimizer updates the model so that rewarded answer paths become more likely\. The repeated answers are not redundant outputs for a user; they are measurements of the model’s current uncertainty on that prompt\.

A simple example captures the mechanism\. Suppose a model attempts the same math problem eight times\. If all eight attempts are wrong, there is no successful attempt to imitate\. If all eight are right, there is no failed attempt to move away from\. The useful training case is mixed: some attempts are right and some are wrong\. Only then can the training rule compare the two sides\. In this sense, a prompt teaches through its*within\-group disagreement*\.

Group Relative Policy Optimization\[GRPO;[1](https://arxiv.org/html/2607.00152#bib.bib1),[2](https://arxiv.org/html/2607.00152#bib.bib2)\], the workhorse of current verifiable reasoning training, is built around this comparison\. GRPO operates in the*reinforcement learning with verifiable rewards*setting \(RLVR\), where an automatic checker returns a reward, usually11for a correct final answer and0for an incorrect one\. The model supplies the candidate answers; the verifier supplies the rewards; GRPO supplies the rule that converts those rewards into*advantages*; the optimizer changes the model parameters\.

Policyπθ\\pi\_\{\\theta\}samplesGGanswers to promptxxVerifier marks eachanswer right or wrong,Ri∈\{0,1\}R\_\{i\}\\in\\\{0,1\\\}Group disagreementσ=k​\(G−k\)/G\\sigma=\\sqrt\{k\(G\-k\)\}/GTraining rule updatesθ\\thetaGRPO / Dr\. GRPO / DAPOGGanswersy1,…,yGy\_\{1\},\\dots,y\_\{G\}rewardsR1,…,RGR\_\{1\},\\dots,R\_\{G\}advantagesA^i\\hat\{A\}\_\{i\}updated model,next step

Figure 1:The training\-time loop studied in this paper\. The trainer samples one prompt many times to compare its correct and incorrect attempts, and the group reward standard deviationσ\\sigmameasures whether they disagree\.The mean subtraction inside GRPO needs little controversy: subtracting any action\-independent baseline preserves the policy gradient while reducing variance\[[3](https://arxiv.org/html/2607.00152#bib.bib3)\]\. The contested step is the next one, division by the group standard deviation\. It is often dismissed as a normalization detail, yetLiu et al\. \[[4](https://arxiv.org/html/2607.00152#bib.bib4)\]identify it as the source of a*question\-level difficulty bias*and remove it\. Its large\-group effect is already understood: for binary rewards it makes GRPO ascend not the raw success ratepp, but the*arcsine transform*𝔼x​\[2​arcsin⁡px\]\\mathbb\{E\}\_\{x\}\[2\\arcsin\\sqrt\{p\_\{x\}\}\], the classical variance\-stabilizing transform of a binomial proportion\[[5](https://arxiv.org/html/2607.00152#bib.bib5),[6](https://arxiv.org/html/2607.00152#bib.bib6)\]\.Thrampoulidis et al\. \[[7](https://arxiv.org/html/2607.00152#bib.bib7)\]established this surrogate\-reward view\.

The lens\.The central object is the group reward standard deviationσ\\sigma\. It is the amount of disagreement in the verifier’s marks inside one prompt’s sampled group\. GRPO divides byσ\\sigma, Dr\. GRPO drops that division, and DAPO’s dynamic sampling discards groups withσ=0\\sigma=0\. The paper’s claim is that these are not separate tricks but three operations on the same number\.

### This paper\.

Real training does not take the large\-group limit; it chooses a finite group sizeGGand updates from the sampled group it receives\. The paper’s main claim is the exact finite\-group accounting behind that update\. For binary rewards, ifkkofGGsampled answers are correct, then a single GRPO step on that prompt is, exactly and in any dimension,

g=1G​∑iA^i​∇θlog⁡πθ​\(yi\)=σ​\(s¯\+−s¯−\),σ=k​\(G−k\)G,0<k<G,g\\;=\\;\\frac\{1\}\{G\}\\sum\_\{i\}\\hat\{A\}\_\{i\}\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\)\\;=\\;\\sigma\\,\\big\(\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}\\big\),\\qquad\\sigma=\\frac\{\\sqrt\{k\(G\-k\)\}\}\{G\},\\quad 0<k<G,\(1\)*independently of the baseline*, wheres¯\+\\bar\{s\}\_\{\+\}ands¯−\\bar\{s\}\_\{\-\}are the mean scores of the correct and incorrect rollouts\. The update has two plain pieces\. The directions¯\+−s¯−\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}says what to favor: correct rollouts over incorrect rollouts\. The multiplierσ\\sigmasays how strongly to favor it: it vanishes for a unanimous group and peaks for an evenly split one\. Equation \([1](https://arxiv.org/html/2607.00152#S1.E1)\) is the*group\-standard\-deviation identity*: what sits in the advantage’s denominator is the length of the gradient itself\. The scalar caseg​\(k\)=k​\(G−k\)/Gg\(k\)=\\sqrt\{k\(G\-k\)\}/G\(wheres¯\+−s¯−=1\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}=1\) is the form used in the remaining analysis\. Averaging \([1](https://arxiv.org/html/2607.00152#S1.E1)\) over groups recovers the arcsine gradient ofThrampoulidis et al\. \[[7](https://arxiv.org/html/2607.00152#bib.bib7)\]asG→∞G\\to\\infty; the identity is the exact finite\-GGobject underneath that limit\. Figure[1](https://arxiv.org/html/2607.00152#S1.F1)locates the quantity inside the training loop, and Figure[2](https://arxiv.org/html/2607.00152#S1.F2)gives the resulting method map: GRPO, Dr\. GRPO, and DAPO act on one scalar quantity in three different ways\.

σ=k​\(G−k\)G\\displaystyle\\sigma=\\frac\{\\sqrt\{k\(G\-k\)\}\}\{G\}per\-prompt updateg=σ​\(s¯\+−s¯−\)g=\\sigma\\,\(\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}\): the group’s reward std times a right\-minus\-wrong contrastGRPOdivide advantages byσ\\sigmaDr\. GRPOremove the division byσ\\sigmaDAPOdiscard groups withσ=0\\sigma=0ascends2​arcsin⁡p2\\arcsin\\sqrt\{p\};difficulty bias1/p​\(1−p\)1/\\sqrt\{p\(1\-p\)\}ascends the raw ratepp;flat difficulty weightdrops the silent masspG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}

Figure 2:One object, three interventions\. The group reward standard deviationσ=k​\(G−k\)/G\\sigma=\\sqrt\{k\(G\-k\)\}/Gthat GRPO divides by, Dr\. GRPO drops, and DAPO filters to zero\.
### Why the exact form matters\.

A practitioner does not choose an asymptotic limit; a practitioner choosesGGand decides which prompts to keep\. The finite\-group identity turns both choices into closed forms\. The resulting contributions are:

1. 1\.The group\-standard\-deviation identity\(§[3](https://arxiv.org/html/2607.00152#S3), Theorem[1](https://arxiv.org/html/2607.00152#Thmtheorem1)\)\. The per\-prompt GRPO update is the contrastive score direction scaled by the group’s reward standard deviationk​\(G−k\)/G\\sqrt\{k\(G\-k\)\}/G, exactly, baseline\-free, and in any dimension\. Its expectation overk∼Binomial​\(G,p\)k\\sim\\mathrm\{Binomial\}\(G,p\)is an exact binomial sum; the large\-group arcsine gradientp​\(1−p\)\\sqrt\{p\(1\-p\)\}and its finite\-GGattenuation1−1/\(8​G​p​\(1−p\)\)1\-1/\(8Gp\(1\-p\)\)are its limit and first correction\.
2. 2\.A closed\-form group\-size law\(§[4](https://arxiv.org/html/2607.00152#S4), Corollary[2](https://arxiv.org/html/2607.00152#Thmcorollary2)\)\. A group of sizeGGrealizes a fractionφ≈1−1/\(8​G​p​\(1−p\)\)\\varphi\\approx 1\-1/\(8Gp\(1\-p\)\)of the large\-group gradient, so reaching fidelity1−ε1\-\\varepsilonneedsG≳1/\(8​ε​p​\(1−p\)\)G\\gtrsim 1/\(8\\varepsilon p\(1\-p\)\), the square of the difficulty weight: a coin\-flip prompt is faithful byG≈10G\\approx 10, a prompt at5%5\\%success needsG≈70G\\approx 70\(Table[2](https://arxiv.org/html/2607.00152#S4.T2)\)\. The budget is read off difficulty, not swept\.
3. 3\.The silent\-group rate\(§[5](https://arxiv.org/html/2607.00152#S5)\)\. A group is silent \(zero advantage everywhere\) with probabilitypG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}; this is exactly theσ=0\\sigma=0mass that DAPO’s dynamic sampling\[[8](https://arxiv.org/html/2607.00152#bib.bib8)\]over\-samples and discards, and its logged all\-correct fraction is the same functional family𝔼p​\[pn\]\\mathbb\{E\}\_\{p\}\[p^\{n\}\], whose shape and sub\-one plateau the closed form reproduces\.
4. 4\.The difficulty bias, exactly\(§[6](https://arxiv.org/html/2607.00152#S6)\)\. The rate at which GRPO converts success probability into objective is∂p2​arcsin⁡p=1/p​\(1−p\)\\partial\_\{p\}\\,2\\arcsin\\sqrt\{p\}=1/\\sqrt\{p\(1\-p\)\}, the very1/σ1/\\sigmathe advantage divides by; this is the question\-level difficulty bias ofLiu et al\. \[[4](https://arxiv.org/html/2607.00152#bib.bib4)\], and deletingσ\\sigmareverts the objective fromarcsin⁡p\\arcsin\\sqrt\{p\}topp\. Group\-mean centering is the leave\-one\-out \(RLOO\) advantage up to the constantG/\(G−1\)G/\(G\-1\), so the division is the only objective\-changing step\.
5. 5\.Validation on real difficulty distributions\(§[7](https://arxiv.org/html/2607.00152#S7)\)\. On Big\-Math\[[9](https://arxiv.org/html/2607.00152#bib.bib9)\],N=215,608N=215\{,\}608problems with empirical solve rates, the standardization moves13\.9%→24\.7%13\.9\\%\\\!\\to\\\!24\.7\\%of the implicit objective’s gradient mass onto extreme prompts; the silent\-group rate is44%44\\%at the common group sizeG=8G=8and matches direct subsampling of the logged rollouts to within two points\.
6. 6\.The predictions in a controlled run\(§[8](https://arxiv.org/html/2607.00152#S8)\)\. A real GRPO loop over6,0006\{,\}000Bernoulli\-logit prompts confirms the closed forms as training dynamics: the silent\-group rate tracks the measured wasted\-group fraction step by step \(R2=0\.999R^\{2\}=0\.999\), the realized gradient mass matches the finite\-GGreweighting, and the difficulty bias is visible as GRPO lifting the hardest prompts where Dr\. GRPO stalls\.

These results do not require a new algorithm or a large training run\. They follow from exact accounting of one sampled group, with the data\-dependent quantities read off published rollout statistics\.

## 2 Three Methods, One Operation Apart

This section names the objects in the training loop and spells out the three method formulas\. The setting is a single promptxxduring training\. The policyπθ\\pi\_\{\\theta\}samples responses, the verifier assigns rewards, and the training rule converts the reward pattern into a parameter update\. All comparisons below differ only in how they handle the group reward standard deviationσ\\sigma\.

### Acronyms and scope\.

The three names are Group Relative Policy Optimization \(GRPO\), GRPO Done Right \(Dr\. GRPO\), and Decoupled Clip and Dynamic Sampling Policy Optimization \(DAPO\)\. Dr\. GRPO also discusses length normalization, and DAPO contains additional engineering choices\. The present analysis isolates the common axis relevant to all three: the standard\-deviation operation on a group of verifier rewards\.

### GRPO advantages\.

For a promptxx, GRPO draws a group ofGGresponsesy1,…,yG∼πθ\(⋅∣x\)y\_\{1\},\\dots,y\_\{G\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)with scalar rewardsR1,…,RGR\_\{1\},\\dots,R\_\{G\}, and forms the standardized advantage

A^i=Ri−μσ,μ=1G​∑jRj,σ=1G​∑j\(Rj−μ\)2,\\hat\{A\}\_\{i\}\\;=\\;\\frac\{R\_\{i\}\-\\mu\}\{\\sigma\},\\qquad\\mu=\\frac\{1\}\{G\}\\sum\_\{j\}R\_\{j\},\\quad\\sigma=\\sqrt\{\\frac\{1\}\{G\}\\sum\_\{j\}\(R\_\{j\}\-\\mu\)^\{2\}\},\(2\)which is broadcast to every token ofyiy\_\{i\}and plugged into the usual clipped surrogate with a KL penalty to a reference policy\. The analysis here concerns the advantage construction; the clipping and KL terms are standard regularizers that do not affect the advantage’s expectation at the first, on\-policy step \(ratio=1=1\), and are held aside throughout\.

### Two updates, one difference\.

Dropping the division byσ\\sigmafrom \([2](https://arxiv.org/html/2607.00152#S2.E2)\) leaves the mean\-centered advantageA^i=Ri−μ\\hat\{A\}\_\{i\}=R\_\{i\}\-\\mu\. This second update is neither new nor specific to one method: it is the update used by Dr\. GRPO\[[4](https://arxiv.org/html/2607.00152#bib.bib4)\], and it equals the leave\-one\-out \(RLOO\) advantage\[[10](https://arxiv.org/html/2607.00152#bib.bib10),[11](https://arxiv.org/html/2607.00152#bib.bib11)\]and baselined REINFORCE\[[3](https://arxiv.org/html/2607.00152#bib.bib3)\]up to a learning\-rate constant \(Proposition[1](https://arxiv.org/html/2607.00152#Thmproposition1)\)\. The equivalence is straightforward: each method subtracts a baseline from the reward and stops there\. GRPO alone adds one further operation, division byσ\\sigma\. Thus the objective\-changing distinction is not the baseline; it is the standard\-deviation operation\. Table[1](https://arxiv.org/html/2607.00152#S2.T1)makes this explicit after the binary\-reward notation is introduced\.

### Binary rewards\.

In RLVR the reward is the verifier’s verdict,Ri∈\{0,1\}R\_\{i\}\\in\\\{0,1\\\}\. The verifier is the checker: for math it may compare final answers or symbolic forms, for code it may run tests, and for structured tasks it may apply a parser or schema\. GRPO does not decide truth by itself; it receives the reward vectorR1,…,RGR\_\{1\},\\dots,R\_\{G\}from this checker\. Writep=px​\(θ\)=ℙy∼πθ\(⋅\|x\)​\[R=1\]p=p\_\{x\}\(\\theta\)=\\mathbb\{P\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\[\\,R=1\\,\]for the policy’s success probability onxx, and letk=∑iRik=\\sum\_\{i\}R\_\{i\}be the number of correct samples in a group\. Thenμ=k/G\\mu=k/Gand, because rewards are Bernoulli,σ=μ​\(1−μ\)=k​\(G−k\)/G\\sigma=\\sqrt\{\\mu\(1\-\\mu\)\}=\\sqrt\{k\(G\-k\)\}/G*exactly*\. Whenk=0k=0all sampled answers are wrong; whenk=Gk=Gall sampled answers are right\. In both casesσ=0\\sigma=0, so there is no right\-versus\-wrong contrast inside the group\.

Table 1:The three methods as operations on one number\. HereΔ​s=s¯\+−s¯−\\Delta s=\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}andσ=k​\(G−k\)/G\\sigma=\\sqrt\{k\(G\-k\)\}/G; the advantage isA^i=\(Ri−μ\)/σ\\hat\{A\}\_\{i\}=\(R\_\{i\}\-\\mu\)/\\sigmafor GRPO andA^i=Ri−μ\\hat\{A\}\_\{i\}=R\_\{i\}\-\\mufor Dr\. GRPO\. GRPO scales the update byσ\\sigma, Dr\. GRPO removes that scaling, and DAPO changes which groups contribute\.
### The score identity\.

The argument uses the elementary policy\-gradient fact that for any baselinebbindependent ofyy,

𝔼y∼πθ\(⋅\|x\)​\[\(R​\(y\)−b\)​∇θlog⁡πθ​\(y∣x\)\]=∇θ𝔼y​\[R​\(y\)\]=∇θpx​\(θ\)\.\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\|x\)\}\\\!\\big\[\(R\(y\)\-b\)\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\\mid x\)\\big\]=\\nabla\_\{\\theta\}\\,\\mathbb\{E\}\_\{y\}\[R\(y\)\]=\\nabla\_\{\\theta\}\\,p\_\{x\}\(\\theta\)\.\(3\)

### Mean\-centering is leave\-one\-out, rescaled\.

The mean subtraction has a familiar form\. Excluding the sample being scored gives the leave\-one\-out \(RLOO\) baselinebi=1G−1​∑j≠iRjb\_\{i\}=\\frac\{1\}\{G\-1\}\\sum\_\{j\\neq i\}R\_\{j\}\[[10](https://arxiv.org/html/2607.00152#bib.bib10),[11](https://arxiv.org/html/2607.00152#bib.bib11)\], whose advantageRi−biR\_\{i\}\-b\_\{i\}is unbiased by \([3](https://arxiv.org/html/2607.00152#S2.E3)\), and subtracting the group mean is the same thing up to a constant\.

###### Proposition 1\(Group\-mean centering is rescaled RLOO\)\.

For any groupR1,…,RGR\_\{1\},\\dots,R\_\{G\}withG≥2G\\geq 2and RLOO baselinebi=1G−1​∑j≠iRjb\_\{i\}=\\frac\{1\}\{G\-1\}\\sum\_\{j\\neq i\}R\_\{j\},

Ri−μ=G−1G​\(Ri−bi\)for every​i\.R\_\{i\}\-\\mu\\;=\\;\\frac\{G\-1\}\{G\}\\,\(R\_\{i\}\-b\_\{i\}\)\\qquad\\text\{for every \}i\.

###### Proof\.

∑jRj=Ri\+\(G−1\)​bi\\sum\_\{j\}R\_\{j\}=R\_\{i\}\+\(G\-1\)b\_\{i\}, soμ=1G​\(Ri\+\(G−1\)​bi\)\\mu=\\tfrac\{1\}\{G\}\\big\(R\_\{i\}\+\(G\-1\)b\_\{i\}\\big\)andRi−μ=G−1G​\(Ri−bi\)R\_\{i\}\-\\mu=\\tfrac\{G\-1\}\{G\}\(R\_\{i\}\-b\_\{i\}\)\. ∎

Mean\-centering is therefore the unbiased RLOO advantage up to the constantG/\(G−1\)G/\(G\-1\), which a learning rate absorbs; both updates share it, so it is not where they differ\. The division byσ\\sigma, the step that does change the objective, is analyzed next\.

## 3 The Update Is the Group’s Standard Deviation

The main result is the mathematical version of the training story above\. Take one prompt, sampleGGanswers, and let the verifier split them into correct and incorrect sets\. GRPO moves the model toward the correct side and away from the incorrect side\. The theorem below states that the length of this move is exactly the group reward standard deviation\. Thus the same scalar that appears in the denominator of the advantage is also the scalar that measures how much learning signal the prompt actually produced\. Everything downstream \(the group\-size budget, the silent fraction, and the difficulty bias\) is read from this one form\. The statement holds for any policy in any dimension, not only for a scalar model\.

###### Theorem 1\(The group\-standard\-deviation identity\)\.

Fix a prompt and a group ofGGresponses with binary rewards,kkof them correct,0<k<G0<k<G\. Letsi=∇θlog⁡πθ​\(yi∣x\)s\_\{i\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid x\)be the score of responseii, and lets¯\+=1k​∑i:Ri=1si\\bar\{s\}\_\{\+\}=\\frac\{1\}\{k\}\\sum\_\{i:R\_\{i\}=1\}s\_\{i\}ands¯−=1G−k​∑i:Ri=0si\\bar\{s\}\_\{\-\}=\\frac\{1\}\{G\-k\}\\sum\_\{i:R\_\{i\}=0\}s\_\{i\}be the mean scores of its correct and incorrect responses\. The per\-prompt GRPO update is, exactly,

g=1G​∑iA^i​si=σ​\(s¯\+−s¯−\),σ=k​\(G−k\)G,g\\;=\\;\\frac\{1\}\{G\}\\sum\_\{i\}\\hat\{A\}\_\{i\}\\,s\_\{i\}\\;=\\;\\sigma\\,\\big\(\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}\\big\),\\qquad\\sigma=\\frac\{\\sqrt\{k\(G\-k\)\}\}\{G\},\(4\)*independently of any baseline*, andg=0g=0whenk∈\{0,G\}k\\in\\\{0,G\\\}\. The scalar coefficient is the group’s empirical reward standard deviation; the directions¯\+−s¯−\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}contrasts the correct and incorrect responses\. For a one\-dimensional Bernoulli\-logit promptp=ς​\(θ\)p=\\varsigma\(\\theta\), wheresi=yi−ps\_\{i\}=y\_\{i\}\-pands¯\+−s¯−=1\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}=1, this is the scalar form

g​\(k\)=k​\(G−k\)G=σ\.g\(k\)\\;=\\;\\frac\{\\sqrt\{k\(G\-k\)\}\}\{G\}\\;=\\;\\sigma\.\(5\)

###### Proof\.

With binary rewards the standardized advantage takes two values,A^\+=\(1−μ\)/σ=\(G−k\)/\(G​σ\)\\hat\{A\}\_\{\+\}=\(1\-\\mu\)/\\sigma=\(G\-k\)/\(G\\sigma\)on thekkcorrect responses andA^−=−μ/σ=−k/\(G​σ\)\\hat\{A\}\_\{\-\}=\-\\mu/\\sigma=\-k/\(G\\sigma\)on theG−kG\-kincorrect ones, withμ=k/G\\mu=k/G\. Since∑i:Ri=1si=k​s¯\+\\sum\_\{i:R\_\{i\}=1\}s\_\{i\}=k\\,\\bar\{s\}\_\{\+\}and∑i:Ri=0si=\(G−k\)​s¯−\\sum\_\{i:R\_\{i\}=0\}s\_\{i\}=\(G\-k\)\\,\\bar\{s\}\_\{\-\},

g=1G​\(A^\+​k​s¯\+\+A^−​\(G−k\)​s¯−\)=k​\(G−k\)G2​σ​\(s¯\+−s¯−\)=σ​\(s¯\+−s¯−\),g=\\frac\{1\}\{G\}\\big\(\\hat\{A\}\_\{\+\}\\,k\\,\\bar\{s\}\_\{\+\}\+\\hat\{A\}\_\{\-\}\\,\(G\-k\)\\,\\bar\{s\}\_\{\-\}\\big\)=\\frac\{k\(G\-k\)\}\{G^\{2\}\\sigma\}\\,\(\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}\)=\\sigma\\,\(\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}\),the last step usingk​\(G−k\)/\(G2​σ\)=σk\(G\-k\)/\(G^\{2\}\\sigma\)=\\sigmafromσ=k​\(G−k\)/G\\sigma=\\sqrt\{k\(G\-k\)\}/G\. Adding any baselinebbto the rewards shifts everyA^i\\hat\{A\}\_\{i\}equally and cancels because∑iA^i=0\\sum\_\{i\}\\hat\{A\}\_\{i\}=0\. The scalar form follows froms¯\+−s¯−=\(1−p\)−\(−p\)=1\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}=\(1\-p\)\-\(\-p\)=1\. ∎

The group\-standard\-deviation identity\.The update factors cleanly\. Its directions¯\+−s¯−\\bar\{s\}\_\{\+\}\-\\bar\{s\}\_\{\-\}points from the incorrect rollouts toward the correct ones; its length is the group’s reward standard deviationσ=k​\(G−k\)/G\\sigma=\\sqrt\{k\(G\-k\)\}/G, zero for a unanimous group and largest for an even split\. This is the paper’s central lens: for binary rewards the standard deviation is not a denominator used for normalization but the prompt’s learning signal\.

Averaging the identity over random groups gives an*exact*expected gradient, whose limit is the established large\-group arcsine reading\.

###### Proposition 2\(Exact expected gradient\)\.

For the scalar prompt \([5](https://arxiv.org/html/2607.00152#S3.E5)\) withk∼Binomial​\(G,p\)k\\sim\\mathrm\{Binomial\}\(G,p\), the expected per\-prompt gradient is the exact finite sum

𝔼​\[g\]=1G​∑k=1G−1\(Gk\)​pk​\(1−p\)G−k​k​\(G−k\)\.\\mathbb\{E\}\[g\]=\\frac\{1\}\{G\}\\sum\_\{k=1\}^\{G\-1\}\\binom\{G\}\{k\}p^\{k\}\(1\-p\)^\{G\-k\}\\sqrt\{k\(G\-k\)\}\.\(6\)

###### Corollary 1\(Large\-group attenuation, asymptotic\)\.

AsG→∞G\\to\\infty,

𝔼​\[g\]=p​\(1−p\)​\(1−18​G​p​\(1−p\)\+O​\(G−2\)\),\\mathbb\{E\}\[g\]=\\sqrt\{p\(1\-p\)\}\\left\(1\-\\frac\{1\}\{8\\,G\\,p\(1\-p\)\}\+O\(G^\{\-2\}\)\\right\),\(7\)the arcsine gradientp​\(1−p\)\\sqrt\{p\(1\-p\)\}attenuated by the relative factor1/\(8​G​p​\(1−p\)\)1/\(8Gp\(1\-p\)\)\. This is the interior expansion of the exact sum \([6](https://arxiv.org/html/2607.00152#S3.E6)\); it loses accuracy nearp∈\{0,1\}p\\in\\\{0,1\\\}, where the unanimous masspG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}is not negligible\. AtG=8,p=0\.05G=8,\\,p=0\.05the exact realized fraction𝔼​\[g\]/p​\(1−p\)\\mathbb\{E\}\[g\]/\\sqrt\{p\(1\-p\)\}is0\.540\.54, against the expansion’s0\.670\.67\.

###### Proof\.

A second\-order delta\-method expansion off​\(μ\)=μ​\(1−μ\)f\(\\mu\)=\\sqrt\{\\mu\(1\-\\mu\)\}aboutμ=p\\mu=p, withf′′​\(p\)=−14​\(p​\(1−p\)\)−3/2f^\{\\prime\\prime\}\(p\)=\-\\tfrac\{1\}\{4\}\(p\(1\-p\)\)^\{\-3/2\}andVar⁡\(μ\)=p​\(1−p\)/G\\operatorname\{Var\}\(\\mu\)=p\(1\-p\)/G, gives𝔼​\[f​\(μ\)\]≈p​\(1−p\)−18​G​\(p​\(1−p\)\)−1/2\\mathbb\{E\}\[f\(\\mu\)\]\\approx\\sqrt\{p\(1\-p\)\}\-\\tfrac\{1\}\{8G\}\(p\(1\-p\)\)^\{\-1/2\}; the excluded unanimous terms areO​\(pG\+\(1−p\)G\)O\(p^\{G\}\+\(1\-p\)^\{G\}\)\. ∎

The numberσ\\sigmais computed from the sampled group alone; it does not require knowledge of the policy’s true success probabilitypp\. It is therefore an observable training diagnostic: before estimating any global difficulty, the sampled rollouts already reveal how much signal the prompt produced\. Figure[3](https://arxiv.org/html/2607.00152#S3.F3)\(left\) confirms the scalar form: the gradient lands onk​\(G−k\)/G\\sqrt\{k\(G\-k\)\}/G, and the Monte\-Carlo markers at three baselines coincide because the baseline cancels in \([4](https://arxiv.org/html/2607.00152#S3.E4)\)\. Averaging over groups gives the arcsine gradientp​\(1−p\)\\sqrt\{p\(1\-p\)\}, attenuated at finiteGGby Corollary[1](https://arxiv.org/html/2607.00152#Thmcorollary1)\. The identity itself is the finite\-GGstatement used by an actual training step\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x1.png)Figure 3:The group\-standard\-deviation identity\.*Left:*the per\-prompt gradient atG=8G=8, zero when all samples agree and largest at an even split, landing onk​\(G−k\)/G\\sqrt\{k\(G\-k\)\}/G; Monte\-Carlo markers at three baselinesb∈\{0\.2,0\.5,0\.8\}b\\in\\\{0\.2,0\.5,0\.8\\\}\.*Right:*the realized fraction𝔼​\[g\]/p​\(1−p\)\\mathbb\{E\}\[g\]/\\sqrt\{p\(1\-p\)\}against the asymptotic law1−1/\(8​G​p​\(1−p\)\)1\-1/\(8Gp\(1\-p\)\)of Corollary[1](https://arxiv.org/html/2607.00152#Thmcorollary1)\.
## 4 How Many Samples a Prompt Needs

Group size is usually fixed before training, often by convention \(G=8G=8or1616\) rather than by calculation\. Corollary[1](https://arxiv.org/html/2607.00152#Thmcorollary1)turns this choice into a difficulty\-dependent quantity\. The question becomes: how many samples are needed before a finite group behaves like the large\-group limit? Define the*gradient fidelity*

φ​\(G,p\)=𝔼​\[g\]p​\(1−p\)∈\[0,1\],\\varphi\(G,p\)\\;=\\;\\frac\{\\mathbb\{E\}\[g\]\}\{\\sqrt\{p\(1\-p\)\}\}\\;\\in\\;\[0,1\],\(8\)as the fraction of the large\-group \(arcsine\) gradient realized by a group of sizeGGat difficultypp\. It approaches11asG→∞G\\to\\infty, and the interior expansion \([7](https://arxiv.org/html/2607.00152#S3.E7)\) givesφ≈1−1/\(8​G​p​\(1−p\)\)\\varphi\\approx 1\-1/\(8Gp\(1\-p\)\)\. Solving this expression forGGgives the group\-size law\.

###### Corollary 2\(The group\-size law\)\.

A group of sizeGGrealizes interior fidelityφ≥1−ε\\varphi\\geq 1\-\\varepsilonat difficultypponce

G≥G⋆​\(ε,p\)=18​ε​p​\(1−p\)=w​\(p\)28​ε,w​\(p\)=1p​\(1−p\)\.G\\;\\geq\\;G^\{\\star\}\(\\varepsilon,p\)\\;=\\;\\frac\{1\}\{8\\,\\varepsilon\\,p\(1\-p\)\}\\;=\\;\\frac\{w\(p\)^\{2\}\}\{8\\,\\varepsilon\},\\qquad w\(p\)=\\frac\{1\}\{\\sqrt\{p\(1\-p\)\}\}\.\(9\)The budget is the mid\-difficulty cost1/\(2​ε\)1/\(2\\varepsilon\)times a difficulty penalty1/\[4​p​\(1−p\)\]≥11/\[4p\(1\-p\)\]\\geq 1: the requirement grows as the*square*of the difficulty weightw​\(p\)w\(p\)that returns as GRPO’s reweighting in §[6](https://arxiv.org/html/2607.00152#S6)\. The penalty is11atp=12p=\\tfrac\{1\}\{2\},2\.82\.8atp=0\.1p=0\.1, and5\.35\.3atp=0\.05p=0\.05\. Nearp∈\{0,1\}p\\in\\\{0,1\\\}the unanimous masspG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}\(§[5](https://arxiv.org/html/2607.00152#S5)\) costs more than the interior expansion accounts for, soG⋆G^\{\\star\}understates the requirement there; the exact group size sits besideG⋆G^\{\\star\}in Table[2](https://arxiv.org/html/2607.00152#S4.T2)\.

Table 2:The group\-size law, exact against closed form\. Each entry is the exact group size at which the gradient fidelityφ\\varphiof \([8](https://arxiv.org/html/2607.00152#S4.E8)\) reaches the column target, with the closed\-form budgetG⋆=1/\(8​ε​p​\(1−p\)\)G^\{\\star\}=1/\(8\\varepsilon p\(1\-p\)\)of \([9](https://arxiv.org/html/2607.00152#S4.E9)\) in parentheses\.The law can be read from a table rather than swept\. Three cases summarize the effect\. A coin\-flip prompt is sample\-efficient:G=11G=11already realizes95%95\\%of the large\-group gradient, and the conventionalG=8G=8realizes93%93\\%\. A very hard prompt is more expensive: a prompt solved5%5\\%of the time needsG=69G=69for the same95%95\\%fidelity, roughly six times the coin\-flip budget, and atG=8G=8realizes only about half of its large\-group gradient \(φ=0\.54\\varphi=0\.54\)\. This is the same lens again: if the group rarely contains both right and wrong answers, more samples are needed before the standard deviation reveals a stable learning signal\. The closed form is most accurate in the high\-fidelity regime that forces large groups: at99%99\\%fidelity andp=0\.05p=0\.05,G⋆G^\{\\star\}gives263263against the exact requirement273273\. The discrepancy is concentrated at small groups near the extremes, where unanimous groups remain common\. Figure[4](https://arxiv.org/html/2607.00152#S4.F4)shows both views: fidelity rising to one withGG\(left\), and the budget’s difficulty bathtub \(right\)\. The practical reading is simple: one uniform group size spends too much on mid\-difficulty prompts and too little on the hard tails, precisely where the silent\-group rate of §[5](https://arxiv.org/html/2607.00152#S5)is largest\. The same budgeting question returns one stage later at inference, governed by a different mechanism: when a fixed prompt is answered by drawing many responses and returning one, the limiting factor is not within\-group difficulty but how strongly the responses agree, so the independent\-draw accounting here gives way to a correlation ceiling that caps selection from extra samples even while coverage keeps climbing\[[12](https://arxiv.org/html/2607.00152#bib.bib12)\]\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x2.png)Figure 4:The group\-size law\.*Left:*the gradient fidelityφ​\(G,p\)=𝔼​\[g\]/p​\(1−p\)\\varphi\(G,p\)=\\mathbb\{E\}\[g\]/\\sqrt\{p\(1\-p\)\}against group size for four difficulties, with the closed form1−1/\(8​G​p​\(1−p\)\)1\-1/\(8Gp\(1\-p\)\)dashed\.*Right:*the group size required for95%95\\%and99%99\\%fidelity against difficulty, exact \(solid\) and the lawG⋆G^\{\\star\}\(dashed\)\.
## 5 Silent Groups, and What DAPO Discards

The identity \([4](https://arxiv.org/html/2607.00152#S3.E4)\) setsg=0g=0exactly when a group is unanimous\. A*silent group*does not mean that the prompt is unimportant; it means that this particular sampled group produced no contrast between right and wrong responses\. In the student analogy, all attempts are either failures or successes, so there is no within\-prompt comparison to learn from\. Thus the second practical decision, which groups to keep, also has a closed form\. If every sampled answer is correct or every sampled answer is wrong, the reward standard deviation is zero, the standardized advantage is undefined, and the prompt contributes no gradient\. The probability of this event, the*silent\-group rate*, is

ℙ​\[group silent\]=pG\+\(1−p\)G,\\mathbb\{P\}\[\\text\{group silent\}\]\\;=\\;p^\{G\}\+\(1\-p\)^\{G\},\(10\)which is large for easy or hard prompts\. At any interiorppit decays geometrically inGG, but atp∈\{0,1\}p\\in\\\{0,1\\\}it is pinned at11: no group size can produce signal when the policy is always wrong or always right \(Figure[5](https://arxiv.org/html/2607.00152#S5.F5)\)\. This is exactly the failure mode targeted by DAPO’s*dynamic sampling*\[[8](https://arxiv.org/html/2607.00152#bib.bib8)\]: DAPO over\-samples and discards prompts whose group accuracy is0or11, keeping only groups with0<k<G0<k<G\. In this notation, dynamic sampling is simply a keep rule on the same scalar: retain the group whenσ\>0\\sigma\>0and replace it whenσ=0\\sigma=0\. Discarding is not the only response to a silent group: a fixed\-reference sign advantage instead assigns a nonzero update to a unanimous group, scoring each response against a constant rather than the group mean\[[13](https://arxiv.org/html/2607.00152#bib.bib13)\]\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x3.png)Figure 5:The silent\-group ratepG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}of \([10](https://arxiv.org/html/2607.00152#S5.E10)\)\. Groups are most often silent near the easy and hard extremes, where all sampled answers tend to share the same reward\. LargerGGreduces the silent mass in the interior but cannot remove the endpointsp=0p=0andp=1p=1\.### The same accounting on a real run\.

The comparison with DAPO has two levels\. The first is structural and requires no fitting\. DAPO\[[8](https://arxiv.org/html/2607.00152#bib.bib8)\]keeps only groups with0<k<G0<k<G, so the discarded groups are exactly the silent mass in \([10](https://arxiv.org/html/2607.00152#S5.E10)\)\. The fraction logged in Fig\. 3b of DAPO \(the avg@32 all\-correct rate\) is the all\-correct component𝔼p​\[p32\]\\mathbb\{E\}\_\{p\}\[p^\{32\}\], and the corresponding training\-time discarded component atG=16G=16is𝔼p​\[p16\]\\mathbb\{E\}\_\{p\}\[p^\{16\}\]\. The second level concerns the curve over training time\. As training moves the Big\-Math difficulty distribution toward mastery and the mean solve rate increases,𝔼p​\[p32\]\\mathbb\{E\}\_\{p\}\[p^\{32\}\]traces the shape of the logged curve with one free timescale, atR2=0\.92R^\{2\}=0\.92\(bootstrap0\.910\.91–0\.960\.96; Figure[6](https://arxiv.org/html/2607.00152#S5.F6), left\)\. This is a consistency check rather than a unique fit: a generic two\-parameter saturating curve fits the same points atR2=0\.98R^\{2\}=0\.98\. The closed form nevertheless explains the qualitative shape without another mechanism\. The discarded fraction rises because more prompts become all\-correct, yet it saturates below one because a persistent contested mass remains \(Figure[6](https://arxiv.org/html/2607.00152#S5.F6), right\)\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x4.png)Figure 6:DAPO’s discarded\-prompt fraction and the closed form\.*Left:*DAPO’s logged accuracy\-11fraction \(avg@32, Fig\. 3b\[[8](https://arxiv.org/html/2607.00152#bib.bib8)\]\), the closed\-form𝔼p​\[p32\]\\mathbb\{E\}\_\{p\}\[p^\{32\}\]fit with bootstrap1010–90%90\\%band, and a generic saturating baseline\.*Right:*the closed\-form all\-correct mass𝔼p​\[p32\]\\mathbb\{E\}\_\{p\}\[p^\{32\}\]versus policy competence on Big\-Math, with the frozen anchor and DAPO’s plateau\.

## 6 What the Division Optimizes

Averaging the identity over groups recovers the known large\-group picture and makes the difference between GRPO and Dr\. GRPO exact\. The single operation “divide byσ\\sigma” does not merely rescale a step; it changes the implicit objective\. By \([6](https://arxiv.org/html/2607.00152#S3.E6)\), asG→∞G\\to\\inftythe per\-prompt GRPO gradient tends top​\(1−p\)\\sqrt\{p\(1\-p\)\}, the gradient of2​arcsin⁡p2\\arcsin\\sqrt\{p\}; Dr\. GRPO tends top​\(1−p\)p\(1\-p\), the gradient ofpp\. This is the surrogate\-reward reading\[[7](https://arxiv.org/html/2607.00152#bib.bib7)\]: GRPO ascends𝔼x​\[2​arcsin⁡px\]\\mathbb\{E\}\_\{x\}\[2\\arcsin\\sqrt\{p\_\{x\}\}\], while Dr\. GRPO ascends the raw success rate𝔼x​\[px\]\\mathbb\{E\}\_\{x\}\[p\_\{x\}\]\. Both objectives have the same endpoint \(all prompts solved\), but they allocate training pressure differently before that endpoint is reached\. Figure[7](https://arxiv.org/html/2607.00152#S6.F7)plots both: the left panel contrasts the two objectives, and the right overlays Monte\-Carlo gradients on the closed forms\. The remaining gap on the GRPO curve is precisely the finite\-GGattenuation of \([6](https://arxiv.org/html/2607.00152#S3.E6)\)\.

### The difficulty bias is the derivative\.

Liu et al\. \[[4](https://arxiv.org/html/2607.00152#bib.bib4)\]\(Dr\. GRPO\) observed empirically that standard\-deviation normalization induces a “question\-level difficulty bias”: very easy and very hard prompts are over\-weighted relative to medium ones\. Removing the normalization removes that bias\. The large\-group limit makes the mechanism exact\. The weight GRPO places on an incremental improvement at difficultyppis the derivative of the transform,

w​\(p\)=∂∂p​2​arcsin⁡p=1p​\(1−p\),w\(p\)\\;=\\;\\frac\{\\partial\}\{\\partial p\}\\,2\\arcsin\\sqrt\{p\}\\;=\\;\\frac\{1\}\{\\sqrt\{p\(1\-p\)\}\},\(11\)the same1/σ1/\\sigmathat appears in the advantage \([2](https://arxiv.org/html/2607.00152#S2.E2)\)\. This weight has a bathtub shape \(Figure[8](https://arxiv.org/html/2607.00152#S6.F8)\): it is smallest atp=12p=\\tfrac\{1\}\{2\}, wherew=2w=2, and diverges likep−1/2p^\{\-1/2\}and\(1−p\)−1/2\(1\-p\)^\{\-1/2\}nearp=0p=0andp=1p=1\. A prompt solved5%5\\%or95%95\\%of the time receivesw≈4\.6w\\approx 4\.6, more than twice the weight of a coin\-flip prompt\. Dr\. GRPO setsw≡1w\\equiv 1, which integrates back to the raw objectivepp\. The difficulty bias is therefore not an accidental side effect\. It is the price of using the variance\-stabilizing transform: estimator variance is equalized by assigning extra weight to the extremes, where the raw success rate changes slowly\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x5.png)Figure 7:The large\-group limit of the identity\.*Left:*raw success ratepp\(Dr\. GRPO\) against the arcsine transform2π​arcsin⁡p\\frac\{2\}\{\\pi\}\\arcsin\\sqrt\{p\}\(GRPO\)\.*Right:*expected per\-prompt gradient: closed\-form curves with Monte\-Carlo markers over groups of sizeG=64G=64on a Bernoulli\-logit prompt\.![Refer to caption](https://arxiv.org/html/2607.00152v1/x6.png)Figure 8:The difficulty weightw​\(p\)=∂p2​arcsin⁡p=1/p​\(1−p\)w\(p\)=\\partial\_\{p\}\\,2\\arcsin\\sqrt\{p\}=1/\\sqrt\{p\(1\-p\)\}of \([11](https://arxiv.org/html/2607.00152#S6.E11)\) \(log scale\), against the flat weightw≡1w\\equiv 1of Dr\. GRPO\. GRPO assigns extra marginal weight to the easiest and hardest prompts, while Dr\. GRPO weights each unit of raw success\-rate improvement equally\.

## 7 The Lens on Real Difficulty Data

The closed forms above are functions of difficultypp, so their practical effect depends on the*distribution*ofppin real data\. Big\-Math\[[9](https://arxiv.org/html/2607.00152#bib.bib9)\]provides such a distribution:N=215,608N=215\{,\}608competition and textbook problems, each annotated with the empirical solve ratep^=k/64\\hat\{p\}=k/64of Llama\-3\.1\-8B over6464rollouts\.111The ungatedopen\-r1/Big\-Math\-RL\-Verified\-Processedmirror is used\. Llama\-3\.1\-8B’s per\-prompt solve rate is treated as a stand\-in for a policy’s success probabilitypxp\_\{x\}; it is a realistic, fixed difficulty distribution, not a specific training run\. All numbers below are exact functions of the publishedp^\\hat\{p\}histogram\.The distribution \(Figure[9](https://arxiv.org/html/2607.00152#S7.F9), left\) is sharply bimodal:4\.0%4\.0\\%of problems are never solved \(p^=0\\hat\{p\}=0\) and7\.2%7\.2\\%are always solved \(p^=1\\hat\{p\}=1\), with broad mass in between\. Thus the extreme\-difficulty regime is not a theoretical corner case; it occupies a visible part of the corpus, precisely where reweighting and silent groups matter most\.

### The reweighting, quantified\.

In the large\-group limit \([7](https://arxiv.org/html/2607.00152#S3.E7)\), each prompt contributes per\-prompt gradient mass∝p​\(1−p\)\\propto\\sqrt\{p\(1\-p\)\}under GRPO and∝p​\(1−p\)\\propto p\(1\-p\)under Dr\. GRPO, the gradients of the two implicit objectives\. Normalizing each over the corpus gives the share of the total gradient budget spent at each difficulty \(Figure[9](https://arxiv.org/html/2607.00152#S7.F9), right, and Table[4](https://arxiv.org/html/2607.00152#S7.T4)\)\. Relative to Dr\. GRPO, GRPO’s standardization nearly doubles the share allocated to extreme\-difficulty prompts \(13\.9%→24\.7%13\.9\\%\\to 24\.7\\%, a factor1\.781\.78\) and reduces the share allocated to medium\-difficulty prompts \(22\.8%→17\.5%22\.8\\%\\to 17\.5\\%\)\. At finiteGGthe realized shift is milder, because the attenuation of Corollary[1](https://arxiv.org/html/2607.00152#Thmcorollary1)discounts the same extremes where silent groups are common; the controlled run of §[8](https://arxiv.org/html/2607.00152#S8)measures14\.3%→17\.0%14\.3\\%\\to 17\.0\\%atG=8G=8, approaching the large\-group24\.7%24\.7\\%asGGgrows\. In either view, the difficulty bias is not marginal on this corpus; it reallocates a visible share of the training signal\.

### Silent groups, quantified\.

Using the silent\-group rate \([10](https://arxiv.org/html/2607.00152#S5.E10)\) on the samep^\\hat\{p\}histogram, Table[4](https://arxiv.org/html/2607.00152#S7.T4)reports the fraction of prompts that yield no signal as a function ofGG\. At the common choiceG=8G=8,44%44\\%of prompts produce no GRPO gradient; even atG=64G=64,17%17\\%do not\. An irreducible11\.2%11\.2\\%\(p^∈\{0,1\}\\hat\{p\}\\in\\\{0,1\\\}\) are silent at everyGG: no amount of additional sampling creates a right\-versus\-wrong contrast for prompts that are always wrong or always right under the logged policy\. This is the mass that dynamic sampling\[[8](https://arxiv.org/html/2607.00152#bib.bib8)\]must replace by over\-sampling\. The closed form is not only an assumption: drawing size\-GGgroups directly from the6464logged rollouts per problem, with no Bernoulli model, gives a measured silent fraction within two points ofpG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}forGGwell below the rollout budget \(43%43\\%against44%44\\%atG=8G=8; Table[4](https://arxiv.org/html/2607.00152#S7.T4), lower row\), the small gap being the finite\-pool correction\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x7.png)Figure 9:Real difficulty distribution and the standardization reweighting \(Big\-Math,N=215,608N=215\{,\}608\)\.*Left:*histogram of empirical solve ratesp^\\hat\{p\}\(Llama\-3\.1\-8B, 64 rollouts\)\.*Right:*share of per\-prompt gradient budget by difficulty under Dr\. GRPO \(∝p​\(1−p\)\\propto p\(1\-p\)\) and GRPO \(∝p​\(1−p\)\\propto\\sqrt\{p\(1\-p\)\}\)\.Table 3:Share of total gradient mass by prompt difficulty on Big\-Math, GRPO versus Dr\. GRPO\.
Table 4:Fraction of Big\-Math prompts whose group is silent \(σ=0\\sigma=0\), by group sizeGG: the closed form \([10](https://arxiv.org/html/2607.00152#S5.E10)\) against direct subsampling of the6464logged rollouts\.

## 8 The Lens in a Live Training Run

The preceding sections use static accounting: for a fixed difficultypp, the formulas predict gradient mass, silent groups, and group\-size requirements\. This section checks whether the same accounting remains visible during training\. The setup is fully reproducible:M=6,000M=6\{,\}000prompts, each a one\-dimensional Bernoulli\-logit policypx=ς​\(θx\)p\_\{x\}=\\varsigma\(\\theta\_\{x\}\)with initial difficulty drawn from the real Big\-Math solve\-rate distribution, are trained for150150steps at group sizeG=8G=8under three advantage rules: GRPO \(divide byσ\\sigma\), Dr\. GRPO \(do not divide\), and DAPO \(resample degenerate groups\)\. At each step every prompt draws a fresh group, forms the advantage, and updatesθx\\theta\_\{x\}by the actual sampled gradient\. Three predictions are compared with the measured run in Figure[10](https://arxiv.org/html/2607.00152#S8.F10)\.

### The silent\-group rate predicts wasted groups\.

Across the whole run the measured fraction of unanimous groups tracks the closed form𝔼​\[pG\+\(1−p\)G\]\\mathbb\{E\}\[p^\{G\}\+\(1\-p\)^\{G\}\]evaluated at the current difficulty, withR2=0\.999R^\{2\}=0\.999\(Figure[10](https://arxiv.org/html/2607.00152#S8.F10)a\)\. The fraction*rises*toward one as training proceeds, because mastered prompts \(p→1p\\to 1\) increasingly become all\-correct\. The same all\-correct mass that climbs in DAPO’s logged run \(§[5](https://arxiv.org/html/2607.00152#S5)\) is reproduced here from the identity alone\.

### The reweighting predicts which difficulties dominate\.

Binning the realized gradient mass by difficulty, GRPO and Dr\. GRPO match their finite\-GGclosed forms exactly \(Figure[10](https://arxiv.org/html/2607.00152#S8.F10)b\): GRPO spends17\.0%17\.0\\%of its gradient mass on the extreme prompts \(p^<0\.1\\hat\{p\}<0\.1or\>0\.9\>0\.9\) against Dr\. GRPO’s14\.3%14\.3\\%\. This is the finite\-GGrealized version of the large\-group24\.7%24\.7\\%of §[7](https://arxiv.org/html/2607.00152#S7); the gap is the attenuation of Corollary[1](https://arxiv.org/html/2607.00152#Thmcorollary1), which discounts the very extremes where silent groups concentrate\.

### The difficulty bias is visible in the trajectories\.

Because the GRPO step scales likeσ\\sigmawhile the Dr\. GRPO step scales likeσ2\\sigma^\{2\}, GRPO moves relatively faster whereσ\\sigmais small, namely near the unanimous extremes\. The effect is dynamic \(Figure[10](https://arxiv.org/html/2607.00152#S8.F10)c\): GRPO lifts the initially\-hardest quartile of prompts to a mean solve rate of0\.990\.99over the run, whereas Dr\. GRPO reaches0\.880\.88; DAPO, which never spends an update on a silent group, is fastest, at the cost of3\.5×3\.5\\timesoversampling concentrated on those same extremes\. Thus the bias is not only a static reallocation of gradient budget\. It changes the learning trajectory, especially for prompts that begin near the hard end of the difficulty scale\.

![Refer to caption](https://arxiv.org/html/2607.00152v1/x8.png)Figure 10:The identity’s predictions in a controlled GRPO run \(M=6,000M=6\{,\}000Bernoulli\-logit prompts, Big\-Math initial difficulty,G=8G=8\)\.*\(a\)*measured silent\-group fraction against the closed form over training\.*\(b\)*realized gradient mass by difficulty, measured against the finite\-GGclosed form\.*\(c\)*mean solve rate of the initially\-hardest quartile under GRPO, Dr\. GRPO, and DAPO\.

## 9 Related Work

### GRPO and its critic\-free relatives\.

Group Relative Policy Optimization \(GRPO\) was introduced for mathematical reasoning\[[1](https://arxiv.org/html/2607.00152#bib.bib1)\]and popularized by R1\-style training\[[2](https://arxiv.org/html/2607.00152#bib.bib2)\]\. Its closest relatives drop the critic in favor of group or leave\-one\-out baselines: RLOO and “back\-to\-basics” REINFORCE\[[10](https://arxiv.org/html/2607.00152#bib.bib10),[11](https://arxiv.org/html/2607.00152#bib.bib11)\], which the identity \([4](https://arxiv.org/html/2607.00152#S3.E4)\) subsumes through the rescaling of Proposition[1](https://arxiv.org/html/2607.00152#Thmproposition1)\.Liu et al\. \[[4](https://arxiv.org/html/2607.00152#bib.bib4)\]propose Dr\. GRPO, identifying a length bias and a question\-level difficulty bias in GRPO and removing both the length and standard\-deviation normalizers; §[6](https://arxiv.org/html/2607.00152#S6)above expresses their difficulty bias exactly as∂p2​arcsin⁡p\\partial\_\{p\}\\,2\\arcsin\\sqrt\{p\}\. DAPO\[[8](https://arxiv.org/html/2607.00152#bib.bib8)\]adds dynamic sampling, which §[5](https://arxiv.org/html/2607.00152#S5)writes as removing the silent\-group mass \([10](https://arxiv.org/html/2607.00152#S5.E10)\)\. A parallel line keeps the group but replaces the group\-relative center with a constant reference, so unanimous groups still produce a signal\[[13](https://arxiv.org/html/2607.00152#bib.bib13)\]; a related reading recasts the same correct\-minus\-incorrect contrast as an implicit preference objective\[[14](https://arxiv.org/html/2607.00152#bib.bib14)\]\.

### The large\-group surrogate\-reward view\.

The asymptotics of the standard\-deviation division are well understood\.Thrampoulidis et al\. \[[7](https://arxiv.org/html/2607.00152#bib.bib7)\], in a study of Pass@K advantage shaping, establish that GRPO is \(up to clipping\) RLOO applied to the surrogate reward2​arcsin⁡p2\\arcsin\\sqrt\{p\}, and identify this with the binomial variance\-stabilizing transform\[[5](https://arxiv.org/html/2607.00152#bib.bib5),[6](https://arxiv.org/html/2607.00152#bib.bib6)\]; related analyses read GRPO as a process reward model\[[15](https://arxiv.org/html/2607.00152#bib.bib15)\], give a local\-curvature/adaptive\-gradient account of the normalizer\[[16](https://arxiv.org/html/2607.00152#bib.bib16)\], and derive REINFORCE, PPO, and GRPO from a common expected\-reward objective\[[17](https://arxiv.org/html/2607.00152#bib.bib17)\]\. The group\-standard\-deviation identity \([4](https://arxiv.org/html/2607.00152#S3.E4)\) supplies the exact finite\-group object whose group average \([6](https://arxiv.org/html/2607.00152#S3.E6)\) becomes this surrogate gradient\. It holds for a single group of any sizeGGin any dimension, depends on no baseline, and turns two open design choices into closed forms: the group\-size law \([9](https://arxiv.org/html/2607.00152#S4.E9)\) and the silent\-group rate \([10](https://arxiv.org/html/2607.00152#S5.E10)\)\. Those quantities are then measured on a real215,608215\{,\}608\-problem corpus in §[7](https://arxiv.org/html/2607.00152#S7)and confirmed as training dynamics in §[8](https://arxiv.org/html/2607.00152#S8)\.

### Variance\-stabilizing transforms\.

The arcsine transform2​arcsin⁡p2\\arcsin\\sqrt\{p\}that stabilizes the variance of a binomial proportion is classical\[[5](https://arxiv.org/html/2607.00152#bib.bib5),[6](https://arxiv.org/html/2607.00152#bib.bib6)\]and standard in the analysis of binomial and count data\. It appears here as the large\-group limit \([6](https://arxiv.org/html/2607.00152#S3.E6)\) of the per\-group standard deviation, the object the GRPO update is shown to equal\.

### Lineage\.

Methodologically this paper follows the tradition of explaining a widely used but under\-theorized method by exhibiting a closed\-form equivalence to a classical object, asLevy and Goldberg \[[18](https://arxiv.org/html/2607.00152#bib.bib18)\]did for word2vec\[[19](https://arxiv.org/html/2607.00152#bib.bib19)\], and as the closed\-form, search\-free treatment ofBay and Yearick \[[20](https://arxiv.org/html/2607.00152#bib.bib20)\]does for scaling laws\.

## 10 Discussion

The practical message is not that one existing method is universally best\. The identity separates three design choices that are often discussed together\. First, the objective choice: dividing byσ\\sigmagives GRPO the variance\-stabilized arcsine objective, while removing the division gives Dr\. GRPO the raw success\-rate objective\. Second, the compute choice: groups withσ=0\\sigma=0have no right\-versus\-wrong contrast, so DAPO\-style dynamic sampling avoids spending updates on them\. Third, the sampling choice: the group\-size law says how largeGGmust be at a given difficulty before the finite group faithfully realizes the large\-group signal\.

- •The signal is disagreement\.By \([4](https://arxiv.org/html/2607.00152#S3.E4)\), the prompt’s instantaneous learning signal is computable from the sampled rollouts alone\. A mixed group teaches because it contains both sides of the comparison\. A unanimous group is silent because there is no within\-prompt contrast\.
- •The methods are operations, not mysteries\.GRPO scales byσ\\sigma, Dr\. GRPO omits that scaling, and DAPO skips groups withσ=0\\sigma=0\. This makes the algorithmic landscape easier to reason about than a list of unrelated heuristics\.
- •Group size is a difficulty\-dependent budget\.The fidelity \([8](https://arxiv.org/html/2607.00152#S4.E8)\) and silent\-group rate \([10](https://arxiv.org/html/2607.00152#S5.E10)\) describe the two thingsGGbuys: how much of the large\-group gradient a retained group realizes, and how often a usable mixed group appears\. A single uniformGGcan be reasonable for simplicity, but the law explains why it underserves the easy and hard extremes\.
- •Standardization is a modeling choice\.Dividing byσ\\sigmatrades raw\-accuracy alignment for a variance\-stabilized objective that gives more marginal weight to extreme difficulties\. This can help rescue hard prompts, but it is also the source of the difficulty bias\. The right choice depends on whether the training goal prioritizes raw success\-rate alignment, hard\-prompt pressure, or compute efficiency\.

More broadly, reading GRPO through the group standard deviation suggests a template for analyzing other reward\-shaping choices: start with one prompt, one sampled group, and one exact update\. Rank\-based advantages, quantile advantages, reward clipping, and length normalization should each admit the same kind of single\-group accounting\.

### Limitations\.

The identity \([4](https://arxiv.org/html/2607.00152#S3.E4)\) concerns the advantage construction under binary rewards and an on\-policy first step\. It deliberately sets aside clipping, the KL penalty, off\-policy staleness, and non\-binary rewards, each of which merits the same single\-group treatment\. The controlled run of §[8](https://arxiv.org/html/2607.00152#S8)validates the closed forms as dynamics on a tractable Bernoulli\-logit policy, where the score is scalar\. A full language\-model training loop that logs silent groups and per\-difficulty gradient mass acrossGGis the natural next test; the theorem predicts what such a run should find because \([4](https://arxiv.org/html/2607.00152#S3.E4)\) holds in any dimension\. Whether the extra weight that standardization places on the hardest prompts improves generalization beyond the trained difficulty range is a further, downstream question, continuous with the broader study of when models extrapolate past their training distribution\[[21](https://arxiv.org/html/2607.00152#bib.bib21)\]\. The DAPO comparison \(§[5](https://arxiv.org/html/2607.00152#S5)\) rests on the exact structural identification of the discarded mass; its time\-evolution fit is a consistency check, not a unique prediction\. A logged per\-prompt solve\-rate history would allow the all\-correct curve to be predicted rather than fit\. None of these limitations affects the core identity \([4](https://arxiv.org/html/2607.00152#S3.E4)\), which is exact for any single group of any size and any policy dimension\.

## 11 Conclusion

During RLVR training, a prompt teaches through the pattern of rewards assigned to its sampled answers\. For binary rewards, this paper shows that one GRPO step on a prompt equals the reward standard deviation of its sampled group,k​\(G−k\)/G\\sqrt\{k\(G\-k\)\}/G, times the contrast between correct and incorrect response scores\. The result is finite\-group, baseline\-free, and valid in any policy dimension\. A prompt pushes hardest when its answers are split and gives no signal when all sampled answers agree\.

This identity makes the relationship among GRPO, Dr\. GRPO, and DAPO concrete\. GRPO divides by the group standard deviation and thereby follows the variance\-stabilized arcsine objective\. Dr\. GRPO drops the division and returns to the raw success\-rate objective\. DAPO’s dynamic sampling removes the groups where the same standard deviation is zero\. The methods are therefore not three disconnected tricks; they are three operations on one number\.

Because the identity holds at the finite group sizes used in training, two practical quantities become closed forms\. The group\-size law,G≳1/\(8​ε​p​\(1−p\)\)G\\gtrsim 1/\(8\\varepsilon p\(1\-p\)\), gives the number of samples needed to realize a target fraction of the large\-group gradient\. The silent\-group rate,pG\+\(1−p\)Gp^\{G\}\+\(1\-p\)^\{G\}, gives the fraction of groups that provide no right\-versus\-wrong contrast\. Both are borne out in a controlled run and on a215,608215\{,\}608\-problem difficulty corpus\. The group standard deviation, long read as a normalizer, is the size of the learning signal itself\.

### Reproducibility\.

All code and data are available at[https://github\.com/bay\-yearick\-lab/grpo\-standard\-deviation\-identity](https://github.com/bay-yearick-lab/grpo-standard-deviation-identity)\. All claims, including the general identity \([4](https://arxiv.org/html/2607.00152#S3.E4)\), are verified numerically \(scripts/checks\.py\), and the closed forms are exposed as a small diagnostic API \(scripts/grpo\_diagnostics\.py\)\. The controlled run of §[8](https://arxiv.org/html/2607.00152#S8)isscripts/experiment\_dynamics\.py; figures and tables are regenerated from the public Big\-Math solve\-rate annotations byscripts/make\_figures\.pyandscripts/analyze\_rollouts\.py\. The DAPO comparison \(§[5](https://arxiv.org/html/2607.00152#S5)\) is digitized from the published figure byscripts/digitize\_dapo\.pyand stored with provenance indata/dapo/\.

## References

- Shao et al\. \[2024\]Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\.DeepSeekMath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- DeepSeek\-AI \[2025\]DeepSeek\-AI\.DeepSeek\-R1: Incentivizing reasoning capability in LLMs via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- Williams \[1992\]Ronald J\. Williams\.Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.*Machine Learning*, 8\(3–4\):229–256, 1992\.
- Liu et al\. \[2025\]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding R1\-Zero\-like training: A critical perspective\.*arXiv preprint arXiv:2503\.20783*, 2025\.
- Bartlett \[1936\]M\. S\. Bartlett\.The square root transformation in analysis of variance\.*Supplement to the Journal of the Royal Statistical Society*, 3\(1\):68–78, 1936\.
- Anscombe \[1948\]F\. J\. Anscombe\.The transformation of Poisson, binomial and negative\-binomial data\.*Biometrika*, 35\(3–4\):246–254, 1948\.
- Thrampoulidis et al\. \[2026\]Christos Thrampoulidis, Sadegh Mahdavi, and Wenlong Deng\.Advantage shaping as surrogate reward maximization: Unifying Pass@K policy gradients\.*Transactions on Machine Learning Research*, 2026\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=R1RhBFUk8t](https://openreview.net/forum?id=R1RhBFUk8t)\.
- Yu et al\. \[2025\]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al\.DAPO: An open\-source LLM reinforcement learning system at scale\.*arXiv preprint arXiv:2503\.14476*, 2025\.
- Albalak et al\. \[2025\]Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, and Nick Haber\.Big\-Math: A large\-scale, high\-quality math dataset for reinforcement learning in language models\.*arXiv preprint arXiv:2502\.17387*, 2025\.
- Ahmadian et al\. \[2024\]Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker\.Back to basics: Revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 12248–12267, 2024\.
- Kool et al\. \[2019\]Wouter Kool, Herke van Hoof, and Max Welling\.Buy 4 REINFORCE samples, get a baseline for free\!*ICLR Deep RL Meets Structured Prediction Workshop*, 2019\.
- Bay and Yearick \[2026a\]Yong Yi Bay and Kathleen A\. Yearick\.When more sampling hurts: The modal ceiling and correlation ceiling of test\-time scaling\.*arXiv preprint arXiv:2606\.28661*, 2026a\.
- Nie et al\. \[2026\]Wenhua Nie, Jianan Wu, Junlin Liu, Ziwei Li, Zheng Lin, Zijian Zhang, Yilong Fan, Haoran Zheng, and Jyh\-Shing Roger Jang\.Gradient starvation in binary\-reward GRPO: Why group\-mean centering fails and why the simplest fix works\.*arXiv preprint arXiv:2605\.07689*, 2026\.
- Wu et al\. \[2025\]Yihong Wu, Liheng Ma, Lei Ding, Muzhi Li, Xinyu Wang, Kejia Chen, Zhan Su, Zhanguang Zhang, Chenyang Huang, Yingxue Zhang, Mark Coates, and Jian\-Yun Nie\.It takes two: Your GRPO is secretly DPO\.*arXiv preprint arXiv:2510\.00977*, 2025\.
- Sullivan and Koller \[2025\]Michael Sullivan and Alexander Koller\.GRPO is secretly a process reward model\.*arXiv preprint arXiv:2509\.21154*, 2025\.
- Ge et al\. \[2026\]Cheng Ge, Caitlyn Heqi Yin, Hao Liang, and Jiawei Zhang\.Why GRPO needs normalization: A local\-curvature perspective on adaptive gradients\.*arXiv preprint arXiv:2601\.23135*, 2026\.
- Shen et al\. \[2026\]Jianghan Shen, Siqi Luo, Yue Li, Jiyao Liu, Wanying Qu, Yi Zhang, Ziyan Huang, Tianbin Li, Ming Hu, Xiaohong Liu, Yirong Chen, and Junjun He\.A first\-principles derivation of LLM policy optimization: From expected reward to GRPO and its structural extensions\.*arXiv preprint arXiv:2606\.16733*, 2026\.
- Levy and Goldberg \[2014\]Omer Levy and Yoav Goldberg\.Neural word embedding as implicit matrix factorization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 27, pages 2177–2185, 2014\.
- Mikolov et al\. \[2013\]Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean\.Efficient estimation of word representations in vector space\.*arXiv preprint arXiv:1301\.3781*, 2013\.
- Bay and Yearick \[2026b\]Yong Yi Bay and Kathleen A\. Yearick\.Solve for the hyperparameter, skip the search: Kolmogorov\-optimal scaling laws for spline regression\.*arXiv preprint arXiv:2606\.23575*, 2026b\.
- Bay and Yearick \[2024\]Yong Yi Bay and Kathleen A\. Yearick\.Machine learning vs deep learning: The generalization problem\.*arXiv preprint arXiv:2403\.01621*, 2024\.