A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

# A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism
Source: [https://arxiv.org/html/2607.12640](https://arxiv.org/html/2607.12640)
Chengguang Gan1,Zhixi Cai2,Yunhao Liang3,Hanjun Wei3,Shiwen Ni4,Qinghao Zhang5 1Independent Researcher2Monash University3University of Chinese Academy of Sciences 4Shenzhen University of Advanced Technology5Pusan National University Correspondence:[chengguangg1024@gmail\.com](https://arxiv.org/html/2607.12640v1/mailto:[email protected])

###### Abstract

Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization \(GRPO\) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent\. We ask whether it adds skill to a small language and vision\-language model web agent at the 4B to 8B scale, or whether it mostly reshapes behavior the supervised model already has\. Across a control grid of 18 runs that varies learning rate, KL weight, seed, initialization, and clipping, no configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered\. On the text track, moderate to high learning rates make it credibly worse\. The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set\-of\-Marks screenshot observations, and scaling the backbone to 8B; the credible harm is a text\-track finding and is only nominal under Set\-of\-Marks\. To show that the null reflects the setting and not a broken pipeline, we run the identical harness, reward, and recipe on tasks whose reward is reachable by sampling, and there the success rate rises by 22 points with a paired interval that excludes zero\. GRPO therefore helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one\. We then explain the failure\. A middle learning rate degrades the agent and a high one collapses it, and the two regimes form a double dissociation: grafting localizes the degrade regime to the attention and MLP blocks, while the collapse regime cannot be traced to any single group, and the embedding change that dominates the weight movement is causally inert\. At 4B, effective rank in the late layers tracks capability in both directions; at 8B the two come apart\. This coupling is specific to the smaller model, so we report it as scale\-dependent\.

A Learning\-Rate\-Gated Failure of GRPO in a Small Language and Vision\-Language Model Web Agent: A Controlled Null and Its Mechanism

Chengguang Gan1, Zhixi Cai2, Yunhao Liang3, Hanjun Wei3, Shiwen Ni4, Qinghao Zhang51Independent Researcher2Monash University3University of Chinese Academy of Sciences4Shenzhen University of Advanced Technology5Pusan National UniversityCorrespondence:[chengguangg1024@gmail\.com](https://arxiv.org/html/2607.12640v1/mailto:[email protected])

## 1Introduction

![Refer to caption](https://arxiv.org/html/2607.12640v1/x1.png)Figure 1:The question and the answer at a glance\. GRPO adds no credible gain on tasks the agent has already mastered, but the same recipe gains 22 points where the reward is reachable by sampling\.Post\-training with reinforcement learning has become routine for language and multimodal agents\. A supervised model is first taught to follow the task format, and GRPO or a close relative\(Shaoet al\.,[2024](https://arxiv.org/html/2607.12640#bib.bib1); Yuet al\.,[2026](https://arxiv.org/html/2607.12640#bib.bib2); Liuet al\.,[2025](https://arxiv.org/html/2607.12640#bib.bib3)\)is then run against a verifiable reward in the hope of pushing capability past what supervision alone reached\. For web agents the appeal is direct, since rollouts are cheap to score against task success and the recipe that lifted mathematics and coding models is expected to lift an agent that clicks, types, and navigates\(Qiet al\.,[2025](https://arxiv.org/html/2607.12640#bib.bib6)\)\. What remains unclear is whether this second stage adds new skill to a small language and vision\-language model agent, or whether it mostly sharpens choices the supervised model could already make\(Chenet al\.,[2026](https://arxiv.org/html/2607.12640#bib.bib13)\)\. This matters in practice\. If reinforcement learning adds capability, more of it is the right investment; if it only sharpens what the supervised model already does, the effort belongs in better supervision or distillation\.

We study one such agent under tight control: a 4B language and vision\-language model\(Baiet al\.,[2025](https://arxiv.org/html/2607.12640#bib.bib10)\)driving a web agent on the MiniWoB benchmark\(Shiet al\.,[2017](https://arxiv.org/html/2607.12640#bib.bib7); Liuet al\.,[2018](https://arxiv.org/html/2607.12640#bib.bib8)\)\. Holding the harness, reward, and task set fixed, we sweep the reinforcement\-learning choices a practitioner would tune and score every run with clean greedy decoding and a paired test against the supervised baseline\. The result is a null\. No configuration in the grid of 18 runs credibly beats supervision on tasks the agent has already largely mastered, and once the learning rate leaves a narrow low band the agent gets worse\. The null is also robust\. It survives adding evaluation and training seeds, adding the schedule and group\-size choices the grid omitted, switching from a text observation to a Set\-of\-Marks screenshot, and scaling the backbone to 8B\. Figure[1](https://arxiv.org/html/2607.12640#S1.F1)previews the study and its outcome\.

One reading of a null is that the pipeline is broken or underpowered and would fail to climb anywhere\. A positive control rules this out\. Using the identical harness, reward, and recipe, we select tasks whose reward is reachable by sampling, where the model already succeeds under temperature more often than under greedy decoding, and retrain\. There the agent climbs by up to 22 points, with a paired interval that excludes zero\. GRPO improves the agent when the sampled policy can already beat the greedy one, and it stalls when greedy is the better of the two, which is the regime a competent supervised agent occupies\. A practitioner can act on this by measuring the gap between sampled and greedy success first and expecting little from reinforcement learning when that gap is not positive\. For a competent small web agent, extra skill comes from supervision or distillation, not from more reinforcement learning\.

We then open the model to ask why the harmful regimes fail\. A low learning rate leaves an update that is real but too small to change behavior; a middle rate degrades the agent, and a high rate collapses it\. The two failures are different lesions\. The degrade regime destroys the effective rank\(Roy and Vetterli,[2007](https://arxiv.org/html/2607.12640#bib.bib11)\)of the late layers while leaving earlier layers intact, whereas the collapse regime holds that rank at or above supervision and instead destroys the readout, an inversion we confirm at nearly two thousand fixed states\. To test whether this correlation is causal, we graft weights\. Restoring the attention or MLP blocks from initialization repairs the degraded agent, while restoring the embedding, whose drift dominates the raw weight change, does nothing, which points to the compute blocks as the site of the failure and clears the embedding despite its large movement\(Ilharcoet al\.,[2022](https://arxiv.org/html/2607.12640#bib.bib12)\)\. Read in the other direction, the same probe shows late\-layer rank rising with capability where the agent climbs and falling where it degrades, so at 4B rank tracks skill both ways\. At 8B this coupling breaks down, and we say so\.

Our choice of a single benchmark is deliberate and follows from the interpretability goal\. A mechanistic account requires holding everything except the intervention fixed, so that a change in effective rank or success rate can be attributed to the learning rate and not to uncontrolled variation in the environment\. MiniWoB provides that control\. Its tasks are deterministic under a seed, its success check is exact, and the breadth of interface skills it covers, from clicking and typing to selection and navigation, makes it a broad probe of web\-agent behavior, not a narrow one\. A benchmark such as WebArena\(Zhouet al\.,[2024](https://arxiv.org/html/2607.12640#bib.bib9)\)is closer to deployment but introduces run\-to\-run variation that is hard to hold constant, and that variation would confound the attributions this study rests on\. We treat MiniWoB as the controlled setting in which the mechanism can be read cleanly, and we are clear about what that costs in generality\.

Our contributions are the following\.

1. 1\.A controlled null: across a grid of 18 runs and several robustness checks, GRPO does not credibly add skill to a competent small language and vision\-language model web agent, and higher learning rates credibly remove it\.
2. 2\.A positive control that identifies the cause of the null as a headroom condition, together with a cheap test of sampled against greedy success that can be applied beforehand\.
3. 3\.A mechanistic account of the failure as a learning\-rate\-gated double dissociation, causally localized by weight grafting, with the dominant embedding drift shown to be a correlational red herring\.
4. 4\.Evidence that late\-layer effective rank tracks capability in both directions at 4B and dissociates at 8B, reported as a property that depends on scale\.
5. 5\.A full battery of paired statistics, equivalence tests, and interpretability measurements, released so others can check the null for themselves\.

## 2Related Work

Table 1:The three experiments of this study, the principle behind each, and what each is for\. Section[3](https://arxiv.org/html/2607.12640#S3)specifies the methods and statistics; the results follow in later sections\.Reinforcement learning with verifiable rewards has driven recent gains in reasoning models, and GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.12640#bib.bib1)\), with refinements such as decoupled clipping\(Yuet al\.,[2026](https://arxiv.org/html/2607.12640#bib.bib2)\)and less biased advantage estimation\(Liuet al\.,[2025](https://arxiv.org/html/2607.12640#bib.bib3)\)on top of the proximal policy optimization objective\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.12640#bib.bib4)\), is now a standard recipe\. Whether these gains reflect genuinely new capability or a sharpening of behavior the base model can already produce is contested, with evidence that reinforcement learning often fails to extend a model beyond what sampling already reaches\(Chenet al\.,[2026](https://arxiv.org/html/2607.12640#bib.bib13)\)\. That question has been argued mostly on reasoning benchmarks through pass@k\. We bring it to an interactive agent and ask it as a practitioner would, against the agent’s own supervised baseline\.

For web agents, both supervised and reinforcement learning pipelines have raised success rates across interactive benchmarks, from the MiniWoB environment\(Shiet al\.,[2017](https://arxiv.org/html/2607.12640#bib.bib7); Liuet al\.,[2018](https://arxiv.org/html/2607.12640#bib.bib8)\)to the larger WebArena\(Zhouet al\.,[2024](https://arxiv.org/html/2607.12640#bib.bib9)\), and online curriculum methods report further gains\(Qiet al\.,[2025](https://arxiv.org/html/2607.12640#bib.bib6)\)\. This line of work shows that reinforcement learning can help a web agent\. It does not settle whether it helps a small language and vision\-language model agent on tasks the agent has already learned to solve, which is the regime a competent deployed system usually occupies and the one we isolate here\.

Our mechanistic analysis builds on tools for reading a model’s internal state\. Effective rank\(Roy and Vetterli,[2007](https://arxiv.org/html/2607.12640#bib.bib11)\)measures how many directions a representation actually uses, and editing in weight space shows that behavior can be moved by recombining parameter groups\(Ilharcoet al\.,[2022](https://arxiv.org/html/2607.12640#bib.bib12)\)\. We turn these into a causal test rather than a correlational one: we restore a single component group from its initialization, keep the rest, and re\-measure success, which separates the parameters that carry a failure from those that merely move the most\.

Two gaps motivate this paper\. The sharpening question has not been settled for an agent by a controlled comparison against its own supervised start with a matched positive control, and where a reinforcement learning null is reported it is seldom explained at the level of the weights\. We address both\. We establish a controlled null for a small language and vision\-language model web agent, use a positive control to show the null is a property of the task rather than the training pipeline, and give a causal account of the failure at the level of individual weight groups\. To our knowledge this is the first study to pair a controlled reinforcement learning null for a web agent with a mechanism for why it occurs\.

## 3Experimental Setup

Our aim is to decide whether GRPO adds skill to a competent small language and vision\-language web agent and, when it does not, to explain why\. The design follows directly\. We fix a single agent, environment, reward, and task set, so that any change in success rate traces to the reinforcement\-learning choices and not to the setup, and on that fixed harness we run the three experiments summarized in Table[1](https://arxiv.org/html/2607.12640#S2.T1)\. The rest of this section specifies the agent and reward \(§[3\.1](https://arxiv.org/html/2607.12640#S3.SS1)\), the GRPO recipe and grid \(§[3\.2](https://arxiv.org/html/2607.12640#S3.SS2)\), the evaluation and statistics that make the null credible \(§[3\.3](https://arxiv.org/html/2607.12640#S3.SS3)\), and the interpretability probes \(§[3\.4](https://arxiv.org/html/2607.12640#S3.SS4)\)\.

### 3\.1Agent, Environment, and Reward

The agent is Qwen3\-VL at 4B and 8B\(Baiet al\.,[2025](https://arxiv.org/html/2607.12640#bib.bib10)\), run on MiniWoB\(Shiet al\.,[2017](https://arxiv.org/html/2607.12640#bib.bib7); Liuet al\.,[2018](https://arxiv.org/html/2607.12640#bib.bib8)\)\. At each step the policyπθ\\pi\_\{\\theta\}reads an observation and emits one action from a compact vocabulary of clicks, typing, selection, navigation, and an explicitfinish, up to a fixed step budget\. Two observation tracks share all other logic: a*text*track that serializes the page, and a*Set\-of\-Marks*track\(Yanget al\.,[2023](https://arxiv.org/html/2607.12640#bib.bib14)\)that presents the screenshot with numbered marks\. The control grid and the mechanism use the text track; the Set\-of\-Marks track is a modality check\. The reward is a sparse binary terminal signal, broadcast to every step of the episode,

r​\(τ\)=𝟏​\[success⁡\(τ\)\]∈\{0,1\}\.r\(\\tau\)=\\mathbf\{1\}\\\!\\left\[\\operatorname\{success\}\(\\tau\)\\right\]\\in\\\{0,1\\\}\.\(1\)

### 3\.2GRPO Recipe and Control Grid

For each training prompt the policy samples a group ofGGrollouts\. FollowingLiuet al\.\([2025](https://arxiv.org/html/2607.12640#bib.bib3)\), the advantage is the group mean\-centered reward, with no division by the group standard deviation,

Ai=ri−1G​∑j=1Grj\.A\_\{i\}=r\_\{i\}\-\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}r\_\{j\}\.\(2\)The per\-token importance ratio between the current policy and the policy that produced the rollout is

ρc​\(θ\)=πθ​\(yc∣x,y<c\)πθold​\(yc∣x,y<c\),\\rho\_\{c\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{c\}\\mid x,y\_\{<c\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{c\}\\mid x,y\_\{<c\}\)\},\(3\)and the clipped surrogate minimized over a segment ofCCcompletion tokens is

ℒGRPO=−1C​∑c=1Cmin⁡\(ρc​A,clip⁡\(ρc,1−ϵlo,1\+ϵhi\)​A\),\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}=\-\\frac\{1\}\{C\}\\sum\_\{c=1\}^\{C\}\\min\\\!\\Big\(\\rho\_\{c\}A,\\ \\operatorname\{clip\}\(\\rho\_\{c\},1\-\\epsilon\_\{\\mathrm\{lo\}\},1\+\\epsilon\_\{\\mathrm\{hi\}\}\)\\,A\\Big\),\(4\)with an asymmetric clip\-higher bound\(Yuet al\.,[2026](https://arxiv.org/html/2607.12640#bib.bib2)\),ϵlo=0\.20<ϵhi=0\.28\\epsilon\_\{\\mathrm\{lo\}\}=0\.20<\\epsilon\_\{\\mathrm\{hi\}\}=0\.28\. An optional KL anchor addsβ​D^KL\\beta\\widehat\{D\}\_\{\\mathrm\{KL\}\}to the loss, using the non\-negativek3k\_\{3\}estimator against the frozen initializationπref\\pi\_\{\\mathrm\{ref\}\}withuc=log⁡\(πref/πθ\)u\_\{c\}=\\log\(\\pi\_\{\\mathrm\{ref\}\}/\\pi\_\{\\theta\}\),

D^KL=1C​∑c\(euc−uc−1\)\.\\widehat\{D\}\_\{\\mathrm\{KL\}\}=\\frac\{1\}\{C\}\\sum\_\{c\}\\big\(e^\{u\_\{c\}\}\-u\_\{c\}\-1\\big\)\.\(5\)The learning rate follows a linear warmup over a fractionffof theRRrounds, then a cosine decay,

ηt=η0⋅\{\(t\+1\)/w,t<w,12​\(1\+cos⁡π​t−wR−w\),t≥w,w=⌈f​R⌉\.\\eta\_\{t\}=\\eta\_\{0\}\\cdot\\begin\{cases\}\(t\+1\)/w,&t<w,\\\\\[3\.0pt\] \\tfrac\{1\}\{2\}\\big\(1\+\\cos\\pi\\tfrac\{t\-w\}\{R\-w\}\\big\),&t\\geq w,\\end\{cases\}\\qquad w=\\lceil fR\\rceil\.\(6\)The controlled null \(experiment A\) is a grid of 18 runs varying learning rate, KL weightβ\\beta, seed, initialization, and the clip bound, with a recipe\-ablation battery that adds the warmup and cosine schedule and the group sizeG∈\{8,16,32\}G\\in\\\{8,16,32\\\}\.

### 3\.3Evaluation Protocol and Statistics

Every run is scored with clean greedy decoding on the setℰ\\mathcal\{E\}of 11 tasks at 5 seeds, giving 55 matched episodes\. The success rate overℰ\\mathcal\{E\}is

SR=1\|ℰ\|​∑e∈ℰ𝟏​\[success⁡\(e\)\],\\mathrm\{SR\}=\\frac\{1\}\{\|\\mathcal\{E\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\}\\mathbf\{1\}\[\\operatorname\{success\}\(e\)\],\(7\)reported at levelzzwith the Wilson score interval\(Wilson,[1927](https://arxiv.org/html/2607.12640#bib.bib17)\)

p^\+z22​n±z​p^​\(1−p^\)n\+z24​n21\+z2/n\.\\frac\{\\widehat\{p\}\+\\tfrac\{z^\{2\}\}\{2n\}\\pm z\\sqrt\{\\tfrac\{\\widehat\{p\}\(1\-\\widehat\{p\}\)\}\{n\}\+\\tfrac\{z^\{2\}\}\{4n^\{2\}\}\}\}\{1\+z^\{2\}/n\}\.\(8\)Because a MiniWoB reward is deterministic given the task seed, each\(task,seed\)\(\\text\{task\},\\text\{seed\}\)pair is matched across arms and every arm is compared to the supervised baseline with a paired test\. On the discordant pairs\(b,c\)\(b,c\)the exact two\-sided McNemarpp\-value is\(McNemar,[1947](https://arxiv.org/html/2607.12640#bib.bib15)\)

p=2​∑i=0min⁡\(b,c\)\(b\+ci\)​2−\(b\+c\),p=2\\\!\\\!\\sum\_\{i=0\}^\{\\min\(b,c\)\}\\\!\\binom\{b\+c\}\{i\}2^\{\-\(b\+c\)\},\(9\)and a task\-clustered bootstrap resamples the 11 task clusters to give a95%95\\%interval on the success\-rate differenceΔ=SRarm−SRSFT\\Delta=\\mathrm\{SR\}\_\{\\mathrm\{arm\}\}\-\\mathrm\{SR\}\_\{\\mathrm\{SFT\}\}\. We call an arm*credibly better*or*credibly worse*only when this interval excludes zero; a positive point estimate whose interval touches zero is no credible difference\. Equivalence within a marginδ\\deltais a two one\-sided test\(Schuirmann,[1987](https://arxiv.org/html/2607.12640#bib.bib16)\): the90%90\\%interval ofΔ\\Deltalies inside\[−δ,\+δ\]\[\-\\delta,\+\\delta\]\.

### 3\.4Interpretability Measurements

On a fixed cache ofNNhidden states we read a small set of quantities\. The effective rank of a layer is the exponential of the entropy of its normalized covariance eigenvalues\{λk\}\\\{\\lambda\_\{k\}\\\}\(Roy and Vetterli,[2007](https://arxiv.org/html/2607.12640#bib.bib11)\), and we also report the participation ratio and the stable rank of the same state matrixXX,

erank=exp⁡\(−∑kpk​log⁡pk\),pk=λk∑jλj,\\operatorname\{erank\}=\\exp\\\!\\Big\(\\\!\-\\\!\\sum\_\{k\}p\_\{k\}\\log p\_\{k\}\\Big\),\\quad p\_\{k\}=\\frac\{\\lambda\_\{k\}\}\{\\sum\_\{j\}\\lambda\_\{j\}\},\(10\)PR=\(∑kλk\)2∑kλk2,sr=∥X∥F2∥X∥22\.\\mathrm\{PR\}=\\frac\{\\big\(\\sum\_\{k\}\\lambda\_\{k\}\\big\)^\{2\}\}\{\\sum\_\{k\}\\lambda\_\{k\}^\{2\}\},\\qquad\\mathrm\{sr\}=\\frac\{\\lVert X\\rVert\_\{F\}^\{2\}\}\{\\lVert X\\rVert\_\{2\}^\{2\}\}\.\(11\)Three probes read the output distributionpn=softmax⁡\(ℓn\)p\_\{n\}=\\operatorname\{softmax\}\(\\ell\_\{n\}\)\. Argmax agreement with the initialization counts how often the top next\-token prediction is unchanged,

agr=1N​∑n𝟏​\[arg⁡max⁡ℓnθ=arg⁡max⁡ℓnref\],\\operatorname\{agr\}=\\frac\{1\}\{N\}\\sum\_\{n\}\\mathbf\{1\}\\\!\\left\[\\arg\\max\\ell^\{\\theta\}\_\{n\}=\\arg\\max\\ell^\{\\mathrm\{ref\}\}\_\{n\}\\right\],\(12\)while the mean entropy and the degeneracy, the fraction of states whose mass collapses onto one token above a thresholdτ\\tau, are

H\\displaystyle H=1N​∑n\(−∑vpn,v​log⁡pn,v\),\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{n\}\\Big\(\\\!\-\\\!\\sum\_\{v\}p\_\{n,v\}\\log p\_\{n,v\}\\Big\),\(13\)deg\\displaystyle\\operatorname\{deg\}=1N​∑n𝟏​\[maxv⁡pn,v\>τ\]\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{n\}\\mathbf\{1\}\\\!\\big\[\\textstyle\\max\_\{v\}p\_\{n,v\}\>\\tau\\big\]\.A causal graft restores one component group𝒢\\mathcal\{G\}, attention, MLP, or embedding, from initialization and re\-evaluates,

θ′=θ,θ𝒢′←θ𝒢init,recovery=SR​\(θ′\)\.\\theta^\{\\prime\}=\\theta,\\quad\\theta^\{\\prime\}\_\{\\mathcal\{G\}\}\\leftarrow\\theta^\{\\mathrm\{init\}\}\_\{\\mathcal\{G\}\},\\qquad\\text\{recovery\}=\\mathrm\{SR\}\(\\theta^\{\\prime\}\)\.\(14\)Finally, the headroom of a task is the gap between the sampled and greedy success of the supervised policy,

Δhead=SRsample−SRgreedy\.\\Delta\_\{\\mathrm\{head\}\}=\\mathrm\{SR\}\_\{\\mathrm\{sample\}\}\-\\mathrm\{SR\}\_\{\\mathrm\{greedy\}\}\.\(15\)GRPO can climb only whenΔhead\>0\\Delta\_\{\\mathrm\{head\}\}\>0; the positive control \(experiment B\) selects tasks on this criterion, and the grafts drive the mechanism analysis \(experiment C\)\.

## 4When GRPO Fails

We begin with the regime a deployed agent occupies, the tasks it has already learned to solve\. The supervised policy clears the 11\-task grid at a success rate of 49\.1% \(27 of 55 matched episodes\), an aggregate that mixes tasks it solves almost every time with a harder frontier\. The question is whether any reinforcement learning configuration moves that number upward, and none does\. The two strongest arms are nominally above supervision by 3\.6 points, yet their paired intervals stretch from zero to about ten points and the exact McNemar test returnsppbetween 0\.50 and 0\.69, so the nudge is indistinguishable from noise \(Table[2](https://arxiv.org/html/2607.12640#S4.T2)\)\. Nor does the result hinge on the learning rate alone\. Varying the KL weight, the initialization, and the clip bound each moves the point estimate by at most two points and none of them credibly, so no axis of the 18\-run grid turns into a gain\. Even taking the best arm as the headline, an upward\-biased choice, leaves an estimate that only matches supervision plus a statistically empty 3\.6 points\.

A flat result invites the reading that the tasks leave no room to improve, but they do\. Restricting the score to the frontier subset, defined as the tasks whose supervised success sits between 0\.2 and 0\.8, the baseline is 37\.1% and the best configuration reaches 42\.9%, a gain whose paired interval still includes zero, so the headroom that exists is not credibly converted into gain\. Stated the other way, two one\-sided tests place four of the five strongest configurations within five points of supervision, one of them episode for episode identical, which reads the outcome as an equivalence: on the mastered tasks GRPO matches the supervised policy but does not improve on it\.

The one variable that moves success is the learning rate, and it moves it only downward\. Holding initialization and the KL weight fixed on the text track, clean success falls as the step size grows, from 52\.7% at3×10−63\\times 10^\{\-6\}, through the supervised baseline near5×10−65\\times 10^\{\-6\}, down to 33\.3% at1×10−51\\times 10^\{\-5\}and zero at2×10−52\\times 10^\{\-5\}\(Figure[2](https://arxiv.org/html/2607.12640#S4.F2)\)\. Three regimes fall out\. At the low end the update is a functional no\-op that leaves behavior at supervision\. A middle rate degrades the agent by a credible 15 points, and the highest rate collapses it to zero, both with intervals that exclude the baseline and McNemarppbelow 0\.04 and 0\.001 \(Table[2](https://arxiv.org/html/2607.12640#S4.T2)\)\. The only rates that leave the agent unharmed are the ones too small to change behavior, and no rate above that band clears the baseline\. This monotone shape is specific to the 4B text track; on the Set\-of\-Marks track and at 8B the nominal peak sits elsewhere, so we keep the shape scoped to 4B text\.

The null does not rest on any single choice in the grid, and the full battery is collected in Appendices[C](https://arxiv.org/html/2607.12640#A3)and[D](https://arxiv.org/html/2607.12640#A4)\. Adding 25 evaluation seeds lowers the baseline to 44\.7% and still surfaces no credible winner; the one arm that reachesp=0\.001p=0\.001under an unclustered test loses that status once episodes are clustered by task, and a replication over 6 training seeds averages 49\.7%, back on the baseline, so its apparent edge was a lucky training seed\. The warmup and cosine schedule the grid omitted remove the collapse pathology, since moderate rates now dip and recover instead of dying, but a recovery is not a gain and the schedule still produces no credible improvement\. A larger sampling group of 16 or 32 does not help, and at a high rate it breaks the agent sooner, since a larger group is a larger effective step\. The Set\-of\-Marks screenshot raises the baseline to 63\.6% and again yields no credible gain, and the 8B backbone repeats the pattern, a nominal low\-rate nudge that never clears the interval and credibly worse success at the higher rates\. The null holds across every axis a practitioner would turn\.

A null could also be an artifact of reward\-hacking or of scoring a checkpoint after it broke, and neither is at work here\. If the agent were gaming the sparse reward by finishing early, the flat runs would fill with premature terminations, but the per\-episode taxonomy puts reward\-hacking near zero in every regime, and the degrade and collapse runs fail because they emit invalid output, not because they stop short\. If the null came from scoring the final checkpoint after it had already broken, then selecting the best checkpoint by held\-out success would surface a hidden win\. It does not\. Scoring every saved round and running a paired test on the best one, the strongest flat run still gains only 3\.6 points with an interval that touches zero, and the brief early rise of the degrade run falls apart before it is credible\. The null comes from the setting itself: it is not an artifact of when we stopped, and it is not the agent gaming the reward\. That still leaves the worry that the pipeline cannot climb anywhere, which the next section settles\.

![Refer to caption](https://arxiv.org/html/2607.12640v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.12640v1/x3.png)

![Refer to caption](https://arxiv.org/html/2607.12640v1/x4.png)

Figure 2:The null across the grid \(4B, text\)\. Left: clean greedy success against learning rate, flat at the low rates where the update is a functional no\-op, then degrading and collapsing as the step grows, with no setting clearing the supervised baseline \(dashed\)\. Middle: a forest plot of all 18 arms with Wilson intervals, none of which clears the baseline\. Right: the per\-task success matrix, where the moderate and high rates drive whole tasks to zero while the low\-rate arm tracks supervision\.Table 2:Learning rate is the only dial that moves success, and only downward \(4B, text track, supervised initialization, no KL\)\. The degrade and collapse rows pool the three and two seeds run at those rates, andΔ\\DeltaSR is the task\-clustered paired difference against the supervised baseline, so for a pooled cell it need not equal the difference of the marginal rates\. The low\-rate no\-op is indistinguishable from supervision; the moderate and high rates are credibly worse\.
## 5When GRPO Succeeds

The previous section ended on the obvious worry that the pipeline simply cannot climb anywhere\. We test it by keeping the harness, reward, and recipe exactly as before, still on the 4B text track, and changing only the tasks\. We profiled the supervised policy on all 119 MiniWoB tasks, recording both its greedy success and the success it reaches under temperature sampling, and selected ten tasks with reachable headroom, where sampling already succeeds more often than greedy and there is therefore a policy for GRPO to move toward that greedy does not already reach\. Two examples fix the idea: drawing a line, where greedy never succeeds but sampling does more than half the time, and filling an autocomplete field, where sampling roughly doubles the greedy rate\.

On this set the same recipe climbs\. Scored on the ten tasks at 5 seeds, 50 matched episodes, the supervised baseline solves 20\.0%, and the strongest configuration reaches 42\.0%, a gain of 22 points whose task\-clustered interval runs from 8 to 40 and excludes zero, with McNemarp=0\.007p=0\.007\(Figure[3](https://arxiv.org/html/2607.12640#S5.F3)\)\. The climb is not one lucky run\. Five of the six configurations rise between 12 and 22 points, four of them credibly, and only a single run at a middle rate degrades\. The low learning rate that was a pure no\-op on the mastered grid gains 12 to 18 points here, so the variable that decides the outcome is the task and not the step size\.

![Refer to caption](https://arxiv.org/html/2607.12640v1/x5.png)Figure 3:Positive control on the headroom set\. The supervised baseline \(dashed\) solves the ten tasks at 20%; five of six configurations climb, and the best reaches 42% with a paired interval that excludes zero\. The low rate that does nothing on the mastered grid climbs here\.Table[3](https://arxiv.org/html/2607.12640#S5.T3)puts the two settings side by side\. The recipe that cannot separate from supervision on the mastered grid produces a credible 22\-point gain on the headroom set, so the null is not a broken or underpowered pipeline; it is GRPO failing where a working GRPO would also fail, on tasks whose reward the greedy policy already reaches\. This explains the null in terms of the objective itself\. GRPO optimizes sampled rollouts, so it can only climb toward a policy that sampling reaches\. On the mastered grid the gap runs the wrong way: greedy is at 0\.49 against a sampled 0\.32 to 0\.38, so there is nothing above greedy to move toward\. On the headroom set the gap is positive by construction, and the agent climbs\. The gap of Eq\.[15](https://arxiv.org/html/2607.12640#S3.E15)is therefore a cheap screen a practitioner can run before committing compute, since it predicts both the null and the climb from the supervised policy alone\.

Table 3:The same harness, reward, and recipe on two task sets, the mastered 11\-task grid and the 10\-task headroom set, each at 5 seeds\. On the mastered grid the best arm gains only\+3\.6\+3\.6points, with a task\-clustered95%95\\%interval\[\+0\.0,\+10\.9\]\[\+0\.0,\+10\.9\]that includes zero\. On the headroom set, where sampling already beats greedy, it gains a credible\+22\.0\+22\.0points, interval\[\+8\.0,\+40\.0\]\[\+8\.0,\+40\.0\]\. The null is a property of the task, not of the pipeline\.
## 6Why GRPO Fails

The headroom condition says when GRPO fails to help, but not how the harmful learning rates damage the model, so we open it up\. Reading a small set of quantities on a fixed cache of hidden states shows that the two harmful regimes damage the model in different ways\. The middle rate that degrades the agent destroys the effective rank of the late layers, read at layer 35, which falls from about nine at supervision to near one while the earlier layers keep their rank\. The high rate that collapses the agent leaves that late\-layer rank at or above supervision yet drives the argmax agreement with initialization to zero \(Figure[4](https://arxiv.org/html/2607.12640#S6.F4), left\)\. Both regimes push the readout down, collapse completely and degrade in part, but only degrade also destroys the representation\. This rank inversion is stable across the full cache of 1999 states, so it is not an artifact of thin sampling\.

![Refer to caption](https://arxiv.org/html/2607.12640v1/x6.png)

![Refer to caption](https://arxiv.org/html/2607.12640v1/x7.png)

![Refer to caption](https://arxiv.org/html/2607.12640v1/x8.png)

Figure 4:The rank account \(layer 35\)\. Left: effective rank by regime at 4B, where degrade collapses the rank and collapse preserves or raises it\. Middle: on the positive\-control family, rank stays healthy where the agent climbs and collapses where it degrades\. Right: at 8B, rank and argmax agreement never fall together across the learning\-rate sweep, so the coupling does not transfer\. Full probes, including argmax agreement and degeneracy by regime, are in Appendix[E](https://arxiv.org/html/2607.12640#A5)\.Rank and agreement are correlations, and to make them causal we graft\. Restoring one component group from initialization into the trained model, keeping the rest, and re\-scoring isolates the weights that carry the failure\. For the degraded agent, restoring the attention or the MLP alone lifts frontier success from 11% to 37% and 40%, at or just above the supervised level, while restoring the embedding leaves it near 14% \(Figure[5](https://arxiv.org/html/2607.12640#S6.F5); the full recovery table for both regimes is in Appendix[E](https://arxiv.org/html/2607.12640#A5)\)\. Either compute pathway is sufficient, so the failure is localizable, and the direction holds across three degrade checkpoints\. This is where a correlational reading would have gone wrong\. The embedding drifts more than any other group in the raw weight difference, so an argument from movement alone would blame it, yet the graft shows the embedding is causally inert while the attention and MLP carry the damage\. To rule out a generic undoing of drift, we restore a random group of matched parameter count as a null; across two dozen draws it recovers to a mean of 19% and a 95th percentile of 23%, and the real attention and MLP grafts sit well above that band, so the localization is specific and not a side effect of moving weights back toward their start\.

![Refer to caption](https://arxiv.org/html/2607.12640v1/x9.png)Figure 5:Causal grafting on the degraded agent\. Restoring the attention or MLP from initialization recovers frontier success past the random\-null band \(its mean to 95th percentile\), while restoring the embedding does not, so the damage is localized to the compute pathways and not to the group that moved most\.The collapsed agent behaves differently under the same test\. No single group restores it, attention and MLP alone recover almost nothing, and only restoring every group together returns it to the supervised level, so collapse is a distributed corruption of the readout rather than a lesion in one place\. This completes a double dissociation: degrade destroys the late\-layer rank and is localizable to a single compute pathway, while collapse spares the rank and cannot be localized at all\.

For rank to be a meaningful measure, it should rise when the agent improves, not only fall when it breaks\. On the family used for the positive control it does\. Reading the same late\-layer rank on three checkpoints from that family, the supervised start sits at rank 6\.9 with 20% success, the configuration that climbs to 42% keeps its rank healthy at 7\.1, and the configuration that degrades to 12% has its rank cut to 3\.2 \(Figure[4](https://arxiv.org/html/2607.12640#S6.F4), middle\)\. The loss shows up as a sharp rank collapse and the gain as a rank that stays healthy rather than falling, so within this family rank health and capability move together, with the caveat that these are single checkpoints and the upward move is small\.

This coupling is a property of the 4B model and does not survive a change of scale\. Reading the same probe across the learning\-rate sweep on the 8B backbone, on the text track, we find no setting where rank and argmax agreement fall together \(Figure[4](https://arxiv.org/html/2607.12640#S6.F4), right\)\. At the rates that degrade the 4B model the 8B late\-layer rank collapses to about two while the argmax agreement stays between 0\.78 and 0\.95, so the readout is untouched even though the representation is not, and only at the highest rate does the agreement collapse, with the rank left high\. The larger model absorbs a late\-layer rank collapse without changing what it predicts\. We therefore scope the account: the link between rank and capability holds at 4B, shown in both directions by grafting and by the positive control, and it breaks at 8B\. We treat that as a limit of the 4B account\.

The mechanism, then, is a pair of learning\-rate\-gated lesions\. A middle rate erases the late\-layer rank that the attention and MLP carry, and a high rate erases the readout in a way spread across the whole model\. Neither adds anything the supervised agent lacked\.

## 7Discussion

The result carries a direct lesson for practice\. On a small web agent that already performs its tasks, the next increment of skill does not come from a second stage of reinforcement learning, because the agent’s greedy policy already sits above what its own sampling reaches and GRPO can pursue only the sampled policy\. The headroom gap of Eq\.[15](https://arxiv.org/html/2607.12640#S3.E15)is the screen to run first: a value at or below zero on a held\-out set is the signal to spend the budget on supervision instead\. In a distillation pilot, expert supervision lifted the base model by 30 points on a task set where reinforcement learning on top of any initialization added nothing \(Appendix[C](https://arxiv.org/html/2607.12640#A3)\)\.

Two boundaries on the claim should be read alongside it\. The study is one benchmark and one model family, chosen so the mechanism can be read without the confounds a noisier environment adds, and the robustness battery covers the recipe choices a reviewer would raise \(schedule, group size, seeds, observation track, and model scale\), none of which changes the verdict\. The credible harm from a large step is established on the 4B text track and is only nominal on the Set\-of\-Marks track, so we state it as a text\-track result and not a universal one\. The mechanism has its own scope\. The link between late\-layer rank and capability that grafting and the positive control establish at 4B does not hold at 8B, where a rank collapse and a change in behavior never coincide, so the rank account is a property of the smaller model and not a law across scale\.

The finding brings the sharpening debate, so far settled largely on reasoning tasks by pass@k\(Chenet al\.,[2026](https://arxiv.org/html/2607.12640#bib.bib13)\), to an interactive agent with a controlled answer\. Where that debate asks whether reinforcement learning extends a model beyond its base, we ask the same of an agent against its own supervised start and find that it does not, then show with the positive control that the boundary is not the method but the headroom of the task\. The contested general claim becomes a specific one we can test: reinforcement learning extends the agent only when its sampled policy already beats its greedy policy\.

## 8Conclusion

We asked whether GRPO adds skill to a small language and vision\-language web agent on tasks it has already learned\. Across the grid and the robustness checks it does not, and past a narrow band of learning rates it subtracts skill instead\. The same pipeline climbs 22 points once the task has headroom, so the null is about the task, not the method\. The failure has a clear mechanism: a middle rate erases the late\-layer rank that grafting traces to attention and MLP, a high rate erases the readout that no single group repairs, and the dominant embedding drift is inert\. For a competent small agent, more skill comes from better supervision, and GRPO earns its compute only where sampling beats greedy\.

## Limitations

This study is deliberately narrow, and several boundaries should be kept in mind\. It uses a single benchmark, MiniWoB, and a single model family, Qwen3\-VL at 4B and 8B; the null and the mechanism may look different on a more varied environment such as WebArena or on a different backbone\. The credible harm from a large learning rate is established on the 4B text track, and on the Set\-of\-Marks track it is only nominal, so the harm side of the result is text\-track evidence and not a universal claim\. The mechanism has a scale boundary of its own: the link between late\-layer rank and capability holds at 4B but breaks at 8B, so the rank account is specific to the smaller model\. The causal grafting and rank\-health readings rest on a small number of checkpoints per regime and their point estimates are noisy, so we report them qualitatively\. Finally, the null concerns the regime where the agent has already mastered its tasks, and the positive control shows that the same recipe does climb once the task has headroom; we therefore make no claim that reinforcement learning is useless for web agents in general, only that it adds no skill in the competent\-agent regime we isolate\.

## References

- Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.12640#S3.SS1.p1.1)\.
- Z\. Chen, R\. Lu, A\. Zhao, Z\. Wang, Y\. Yue, S\. Song, and G\. Huang \(2026\)Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\.Advances in Neural Information Processing Systems38,pp\. 57654–57689\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p1.1),[§2](https://arxiv.org/html/2607.12640#S2.p1.1),[§7](https://arxiv.org/html/2607.12640#S7.p3.1)\.
- G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi \(2022\)Editing models with task arithmetic\.arXiv preprint arXiv:2212\.04089\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p4.1),[§2](https://arxiv.org/html/2607.12640#S2.p3.1)\.
- E\. Z\. Liu, K\. Guu, P\. Pasupat, T\. Shi, and P\. Liang \(2018\)Reinforcement learning on web interfaces using workflow\-guided exploration\.arXiv preprint arXiv:1802\.08802\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p2.1),[§2](https://arxiv.org/html/2607.12640#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.12640#S3.SS1.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025\)Understanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p1.1),[§2](https://arxiv.org/html/2607.12640#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.12640#S3.SS2.p1.1)\.
- Q\. McNemar \(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[§3\.3](https://arxiv.org/html/2607.12640#S3.SS3.p1.6)\.
- Z\. Qi, X\. Liu, I\. L\. Iong, H\. Lai, X\. Sun, J\. Sun, X\. Yang, Y\. Yang, S\. Yao, W\. Xu,et al\.\(2025\)Webrl: training llm web agents via self\-evolving online curriculum reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 79791–79821\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p1.1),[§2](https://arxiv.org/html/2607.12640#S2.p2.1)\.
- O\. Roy and M\. Vetterli \(2007\)The effective rank: a measure of effective dimensionality\.In2007 15th European signal processing conference,pp\. 606–610\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p4.1),[§2](https://arxiv.org/html/2607.12640#S2.p3.1),[§3\.4](https://arxiv.org/html/2607.12640#S3.SS4.p1.3)\.
- D\. J\. Schuirmann \(1987\)A comparison of the two one\-sided tests procedure and the power approach for assessing the equivalence of average bioavailability\.Journal of pharmacokinetics and biopharmaceutics15\(6\),pp\. 657–680\.Cited by:[§3\.3](https://arxiv.org/html/2607.12640#S3.SS3.p1.12)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2](https://arxiv.org/html/2607.12640#S2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p1.1),[§2](https://arxiv.org/html/2607.12640#S2.p1.1)\.
- T\. Shi, A\. Karpathy, L\. Fan, J\. Hernandez, and P\. Liang \(2017\)World of bits: an open\-domain platform for web\-based agents\.InInternational Conference on Machine Learning,pp\. 3135–3144\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p2.1),[§2](https://arxiv.org/html/2607.12640#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.12640#S3.SS1.p1.1)\.
- E\. B\. Wilson \(1927\)Probable inference, the law of succession, and statistical inference\.Journal of the American Statistical Association22\(158\),pp\. 209–212\.Cited by:[§3\.3](https://arxiv.org/html/2607.12640#S3.SS3.p1.3)\.
- J\. Yang, H\. Zhang, F\. Li, X\. Zou, C\. Li, and J\. Gao \(2023\)Set\-of\-mark prompting unleashes extraordinary visual grounding in gpt\-4v\.arXiv preprint arXiv:2310\.11441\.Cited by:[§3\.1](https://arxiv.org/html/2607.12640#S3.SS1.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2026\)Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p1.1),[§2](https://arxiv.org/html/2607.12640#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.12640#S3.SS2.p1.7)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2024\)Webarena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 15585–15606\.Cited by:[§1](https://arxiv.org/html/2607.12640#S1.p5.1),[§2](https://arxiv.org/html/2607.12640#S2.p2.1)\.

The appendices below collect the full result tables, a gallery of the supporting figures, and the algorithms referenced in the main text\. All numbers are transcribed from the same frozen result tables used in the body\.

## Appendix AThe Control Grid in Full

The controlled null is an 18\-run grid on the 4B text track that varies the learning rate, the KL weight, the seed, the initialization \(supervised or base\), and the clip bound\. Every arm is scored by clean greedy decoding on the 11 tasks at 5 seeds, giving 55 matched episodes, and compared to the supervised baseline with the paired test of the main text\. Table[4](https://arxiv.org/html/2607.12640#A1.T4)lists the representative arms\. Classified by their training dynamics the 18 runs split into 9 functional no\-ops, 6 degrade runs, and 3 collapse runs, and no arm’s paired interval clears the baseline\. Reporting the best of eighteen is itself an upward\-biased statistic, and even that best matches supervision plus a statistically empty3\.63\.6points\. The “no headroom” reading is refuted on the frontier subset, the tasks whose supervised success lies in\[0\.2,0\.8\]\[0\.2,0\.8\]: the supervised frontier baseline is37\.1%37\.1\\%\[23\.2,53\.7\]\[23\.2,53\.7\]\(13 of 35\) and the best configuration reaches only42\.9%42\.9\\%\[28\.0,59\.1\]\[28\.0,59\.1\]\(15 of 35\), whose interval lower bound of28\.028\.0sits below the baseline point estimate\.

Table 4:Representative arms of the 18\-run control grid \(4B, text\)\. Clean SR is greedy success over 55 matched episodes with a Wilson interval;Δ\\DeltaSR is the task\-clustered paired difference against the supervised baseline\. The middle\- and high\-rate rows pool 3 and 2 seeds\. The KL rescue arm is episode\-identical to supervision \(discordant pairsb/c=0/0b/c=0/0\)\. No arm is credibly better than supervision\.
## Appendix BPaired Statistics, Equivalence, and Checkpoint Selection

The verdict uses the exact McNemar test on the discordant pairs and a task\-clustered bootstrap on the 11 task clusters, both over episode\-matched arms\. Beyond the failure to reject, a two one\-sided test states a positive equivalence: four of the five strongest configurations lie within±5\\pm 5points of supervision \(Table[5](https://arxiv.org/html/2607.12640#A2.T5)\), one of them episode\-identical\. Because the reported success is the final checkpoint, we also retrained the three regime exemplars saving every round and paired\-tested the single best\-by\-eval checkpoint of each; none is credibly superior \(Table[6](https://arxiv.org/html/2607.12640#A2.T6)\), so the null is not an artifact of scoring an already broken checkpoint\.

Table 5:Two one\-sided equivalence at the 25\-seed grain\. Four of five top configurations are statistically equivalent to supervision within±5\\pm 5points\. The lone exception is the lucky\-seed arm, whose six\-seed replication averages49\.7%49\.7\\%\(Table[10](https://arxiv.org/html/2607.12640#A3.T10)\)\.Table 6:Best\-by\-eval checkpoint selection\. Scoring every saved round and paired\-testing the maximum\-SR checkpoint of each regime surfaces nominal nudges \(flat drifts to52\.7%52\.7\\%, degrade peaks near51%51\\%before breaking\) but none survives the paired test\.
## Appendix CRobustness Battery

The null is unchanged by every recipe and evaluation choice a reviewer would raise\. Table[7](https://arxiv.org/html/2607.12640#A3.T7)collects the axes\. Widening evaluation to 25 seeds lowers the baseline to44\.7%44\.7\\%and leaves no credible winner\. The one borderline arm \(McNemarp=0\.001p=0\.001under an unclustered test\) loses credibility once episodes are clustered by task, and a six\-seed replication of it averages49\.7%49\.7\\%\(Table[10](https://arxiv.org/html/2607.12640#A3.T10)\), so the edge was a lucky training seed\. Adding the warmup and cosine schedule removes the collapse pathology without producing a gain, and a larger group breaks the agent sooner at high rates \(Table[8](https://arxiv.org/html/2607.12640#A3.T8)\)\. A per\-episode taxonomy shows the failures are output degeneration, not reward\-hacking: premature finishing is near zero in every regime \(Table[9](https://arxiv.org/html/2607.12640#A3.T9)\)\. On a separate distillation pilot \(5 tasks, 6 seeds,n=30n=30\), expert supervision lifts the base model from50%50\\%to80%80\\%while GRPO on top of any initialization adds nothing, which is the evidence that the constructive lever is supervision rather than reinforcement learning\.

Table 7:The null survives every robustness axis\. Base is supervised success and BestΔ\\Deltais the largest nominal point\-estimate gain of any arm, in percent, except the training\-seed row, which reports the six\-seed mean\. Every listed gain has a paired interval that includes zero\.Table 8:Recipe ablation\. A standard warmup and cosine schedule turns the constant\-rate collapse into a dip\-and\-recover at moderate rates but yields no credible gain, and a larger sampling group is a larger effective step that breaks high rates faster\.Table 9:Per\-episode failure\-mode taxonomy\. Reward\-hacking \(premature finish\) is near zero in every regime, so the null is not reward gaming; degrade and collapse fail by emitting invalid output while flat and KL preserve the supervised structure\.Table 10:Six\-seed replication of the nominal\-best configuration \(SFT init, constant lr3×10−63\\times 10^\{\-6\},G=8G=8\)\. The mean is49\.7%49\.7\\%\(±2\.5\\pm 2\.5sd,95%95\\%CI\[47\.7,51\.7\]\[47\.7,51\.7\]\), on the supervised baseline of49\.1%49\.1\\%, so the\+5\.1\+5\.1\-point single\-seed result was training\-seed noise\.
## Appendix DObservation Track and Model Scale

The no\-gain null generalizes across observation track and model scale, while the credible harm from a large rate is established on the 4B text track and is only nominal elsewhere\. On the Set\-of\-Marks track the supervised baseline is higher, at63\.6%63\.6\\%, and no configuration credibly beats it; the high\-rate degradation is directional but not credible \(Table[11](https://arxiv.org/html/2607.12640#A4.T11)\)\. At 8B on Set\-of\-Marks the baseline is also63\.6%63\.6\\%, the low\-rate nudge is again nominal, and the higher rates are credibly worse \(Table[12](https://arxiv.org/html/2607.12640#A4.T12)\)\. The 8B mechanism cache \(372 states, text\) reproduces the late\-layer rank inversion: supervised13\.613\.6, flat15\.115\.1, degrade1\.91\.9, collapse19\.619\.6\.

Table 11:Set\-of\-Marks track at 4B\. No configuration credibly beats the supervised baseline, and the high\-rate degradation is nominal \(p=0\.070p=0\.070, task\-clustered interval touches zero\), not credible as it is on the text track\.Table 12:Set\-of\-Marks track at 8B\. The low\-rate nudge is nominal and single\-seed \(p=0\.062p=0\.062\), the same lucky\-seed pattern seen at 4B, and the higher rates are credibly worse\.
## Appendix EMechanism in Detail

The interpretability probes read a fixed cache of hidden states\. Movement magnitude does not predict the failure mode: collapse moves less inL2L\_\{2\}than degrade yet is more destructive, and the single largest total movement in the grid is the KL rescue arm, which is behaviorally at the baseline \(Table[13](https://arxiv.org/html/2607.12640#A5.T13)\)\. The rank and output\-behavior probes give the double dissociation in full, including next\-token entropy \(Table[14](https://arxiv.org/html/2607.12640#A5.T14)\)\. Causal grafting localizes degrade to the attention or MLP and refutes the embedding, whose drift is largest; a magnitude\-matched random\-component null over 24 draws recovers only to a mean of19\.3%19\.3\\%and a 95th percentile of22\.9%22\.9\\%, below the real grafts, while for collapse no single group recovers \(Table[15](https://arxiv.org/html/2607.12640#A5.T15)\)\. The rank\-to\-capability coupling that holds at 4B does not transfer to 8B, where rank and argmax agreement never fall together across the sweep \(Table[16](https://arxiv.org/html/2607.12640#A5.T16)\)\. The KL anchor is a mode\-preserving stabilizer, not a teacher: every KL arm holds argmax agreement at1\.01\.0while next\-token entropy inflates to4\.94\.9–7\.77\.7, and KL suppresses embedding drift \(∼0\.006\{\\sim\}0\.006with KL against∼0\.011\{\\sim\}0\.011without, at1×10−51\\times 10^\{\-5\}\) rather than freezing the weights\.

Table 13:Weight movement by regime\. The outcome is not monotone in movement: collapse moves less than degrade yet destroys more, and the KL rescue arm moves the most of all while staying at the supervised baseline\.Table 14:Full fixed\-state probes atN=1999N=1999\(4B, text\)\. Degrade destroys the late\-layer rank while collapse preserves or raises it; both drive the readout down, collapse completely and degrade in part\. Entropy is not monotone across regimes, so entropy inflation alone is harmless under greedy decoding\.Table 15:Frontier success \(%\) after restoring a component group from initialization\. For degrade, attention or MLP alone recovers above the random\-null 95th percentile, and the direction holds across three degrade checkpoints \(recovery3333–111%111\\%of the none\-to\-all gap\)\. For collapse no single group recovers\. Values are over 35 episodes and noisy in the point estimate\.Table 16:The 8B rank\-to\-behavior map across the learning\-rate sweep\. No rate drops rank and argmax agreement together: at8×10−68\\times 10^\{\-6\}and1×10−51\\times 10^\{\-5\}the rank collapses while agreement stays high, and at2×10−52\\times 10^\{\-5\}agreement collapses while the rank is preserved, so the coupling seen at 4B does not transfer\.
## Appendix FThe Positive Control in Detail

The positive control keeps the harness, reward, and recipe of the 4B text grid and changes only the tasks\. We profiled the supervised policy on all 119 MiniWoB tasks and selected ten with a positive sampled\-minus\-greedy gap \(Table[17](https://arxiv.org/html/2607.12640#A6.T17)\)\. The same GRPO recipe, run over a small learning\-rate sweep at two seeds, climbs on this set: five of six configurations rise and the strongest gains a credible 22 points \(Table[18](https://arxiv.org/html/2607.12640#A6.T18)\)\. Reading the late\-layer rank on three checkpoints from this family shows rank health accompanying the gain and rank collapse accompanying the loss \(Table[19](https://arxiv.org/html/2607.12640#A6.T19)\), the constructive complement to the degrade lesion of the mastered grid\.

Table 17:Example headroom tasks\. The reward is reachable by sampling that greedy decoding misses, soΔhead=SRsample−SRgreedy\>0\\Delta\_\{\\mathrm\{head\}\}=\\mathrm\{SR\}\_\{\\mathrm\{sample\}\}\-\\mathrm\{SR\}\_\{\\mathrm\{greedy\}\}\>0, the criterion the ten selected tasks satisfy and the mastered grid does not\.Table 18:Per\-configuration positive control on the ten headroom tasks \(50 matched episodes\)\. Five of six configurations climb, four credibly, and the low rate that is a no\-op on the mastered grid gains1212to1818points here\.Table 19:Rank health on the positive\-control family\. The climbing checkpoint keeps its late\-layer rank at the supervised level while success rises, and the degrading checkpoint’s rank collapses as success falls, so rank tracks capability in both directions at 4B\. These are single checkpoints and the upward move is small\.
## Appendix GFigure Gallery

This section collects the supporting figures that do not appear in the main text, grouped by the role they play in the argument\. All success rates are clean greedy decoding on the 11\-task text track \(11 tasks by 5 seeds, 55 episodes per run\) unless a caption states otherwise, and all intervals are Wilson95%95\\%or task\-clustered paired bootstrap intervals as noted\.

### G\.1Controlled null

![Refer to caption](https://arxiv.org/html/2607.12640v1/x10.png)Figure 6:Paired re\-analysis of every grid arm against SFT on the 55 matched \(task, seed\) episodes, using a task\-clustered bootstrap95%95\\%interval and an exact McNemar test on the discordant pairs\. No arm is credibly better than SFT\. The nominal winners at lr3×10−63\\times 10^\{\-6\}\(\+3\.6\+3\.6pp\) have intervals\[\+0\.0,\+10\.9\]\[\+0\.0,\+10\.9\]and\[\+0\.0,\+9\.1\]\[\+0\.0,\+9\.1\]with McNemarp=0\.688p=0\.688and0\.5000\.500\. The moving regimes are credibly worse: pooled lr1×10−5=−15\.01\\times 10^\{\-5\}=\-15\.0pp\[−20\.9,−9\.5\]\[\-20\.9,\-9\.5\]and pooled lr2×10−5=−49\.12\\times 10^\{\-5\}=\-49\.1pp\[−62\.7,−36\.4\]\[\-62\.7,\-36\.4\]\. The KL\-rescued arms have discordant countsb/c=0/0b/c=0/0, i\.e\. per\-episode outcomes identical to SFT on all 55 episodes\.
### G\.2Robustness

![Refer to caption](https://arxiv.org/html/2607.12640v1/x11.png)Figure 7:The null under a five\-fold increase in evaluation seeds \(25 seeds, 275 episodes per config\)\. The 25\-seed SFT baseline is44\.7%44\.7\\%\[39\.0,50\.6\]\[39\.0,50\.6\]\(123/275\), slightly below the 5\-seed49\.1%49\.1\\%\. Under the task\-clustered paired bootstrap no config is credibly better: the largest nominal gap is\+5\.1\+5\.1pp with McNemarp=0\.001p=0\.001but a bootstrap interval whose lower bound touches0, so it is not credible and is a single training seed\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x12.png)Figure 8:Training\-seed replication of the one borderline arm \(\+5\.1\+5\.1pp at 25 eval seeds\)\. The exact configuration \(SFT\-init, constant lr3×10−63\\times 10^\{\-6\},G=8G=8, 15 rounds\) was retrained across six training seeds and clean\-evaluated at the final checkpoint:49\.1,52\.7,45\.5,49\.1,50\.9,50\.9%49\.1,52\.7,45\.5,49\.1,50\.9,50\.9\\%\. The mean is49\.7%±2\.549\.7\\%\\pm 2\.5\(sd\),95%95\\%CI\[47\.7,51\.7\]\[47\.7,51\.7\], sitting on the SFT baseline of49\.1%49\.1\\%, so the\+5\.1\+5\.1pp was training\-seed noise\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x13.png)Figure 9:Recipe ablation adding a warmup \(15%15\\%\) plus cosine schedule and sweeping group sizeG∈\{8,16,32\}G\\in\\\{8,16,32\\\}, paired against SFT \(49\.1%49\.1\\%\)\. No run credibly beats SFT\. The schedule removes the collapse pathology at moderate learning rate \(constant\-lr1×10−51\\times 10^\{\-5\}degraded and2×10−52\\times 10^\{\-5\}collapsed in the grid; with the schedule,8×10−68\\times 10^\{\-6\}and1\.5×10−51\.5\\times 10^\{\-5\}atG=8G=8become no\-diff\), but recovery is not learning\. LargerGGdoes not help and makes high\-lr collapse worse\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x14.png)Figure 10:Two one\-sided equivalence at the 25\-seed grain: a config is declared equivalent to SFT within±δ\\pm\\deltawhen its task\-clustered paired90%90\\%interval of \(config minus SFT\) lies inside\[−δ,\+δ\]\[\-\\delta,\+\\delta\]\. Four of five top configs are equivalent within±5\\pm 5pp: base\-init flat\[\+0\.0,\+4\.1\]\[\+0\.0,\+4\.1\]; KL0\.10\[\+0\.0,\+0\.0\]\[\+0\.0,\+0\.0\]\(episode\-identical\); recipeG​16G16\[−2\.7,\+4\.1\]\[\-2\.7,\+4\.1\]; base\-init sanity\[−2\.3,\+0\.9\]\[\-2\.3,\+0\.9\]\. The lone exception is the lucky\-seed arm,\[\+0\.0,\+15\.0\]\[\+0\.0,\+15\.0\], the same seed noise resolved by the six\-seed replication \(Fig\.[8](https://arxiv.org/html/2607.12640#A7.F8)\)\. This is a positive equivalence claim, not merely a failure to reject\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x15.png)Figure 11:Best\-by\-eval checkpoint trajectory for the three regime exemplars, clean\-evaluated at every saved round\. Best\-by\-eval selection surfaces nominal nudges \(the flat run drifts to a max of52\.7%52\.7\\%by rounds 7 to 11; the degrade run reads50\.9%50\.9\\%at round 1 before falling to3535–40%40\\%\), but none survives the episode\-matched paired test: flat max\+3\.6\+3\.6pp\[\+0\.0,\+9\.1\]\[\+0\.0,\+9\.1\]\(p=0\.500p=0\.500\), degrade r1\+1\.8\+1\.8pp\[−9\.1,\+12\.7\]\[\-9\.1,\+12\.7\]\(p=1\.000p=1\.000\), collapse−49\.1\-49\.1pp\. The null is not an artifact of scoring an already\-broken final checkpoint\.
### G\.3Mechanism

![Refer to caption](https://arxiv.org/html/2607.12640v1/x16.png)Figure 12:Fixed\-state effective rank by network depth \(layers 6, 12, 18, 24, 30, 35\) on identical inputs, localizing the degrade lesion to the late layers\. Baselines and flat runs hold a late\-layer \(L35\) effective rank of about6\.46\.4to6\.76\.7, whereas the no\-KL degrade runs collapse L35 to about1\.01\.0to1\.21\.2while earlier layers \(L6 to L18\) stay healthy\. Collapse runs preserve L35 rank, the opposite lesion\. The dissociation survives atN=1999N=1999states, so it is not a small\-NNartifact\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x17.png)Figure 13:Fixed\-state output behavior on identical decision states: baseline argmax agreement, degeneracy \(unparseable or no\-op fraction\), and next\-token entropy\. Baselines and flat runs preserve the policy \(agreement about0\.950\.95to1\.01\.0, degeneracy near0\) even when entropy is inflated, so entropy inflation alone is harmless under greedy decoding\. Degrade corrupts the argmax \(agreement0\.070\.07to0\.50\.5, degeneracy0\.50\.5to0\.930\.93\); collapse destroys it entirely \(agreement0, degeneracy0\.80\.8to1\.01\.0\) in two flavors, high\-entropy garble and low\-entropy mode collapse\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x18.png)Figure 14:Weight\-movement magnitude relative to initialization \(mean relativeL2L\_\{2\}and sparse fraction\) versus learning rate and regime, showing that magnitude does not predict the failure mode\. Low\-lr flat runs move least\. Collapse moves less inL2L\_\{2\}than degrade yet is far more destructive\. The single largest total movement in the grid belongs to the KL\-rescued run, which is behaviorally at the SFT baseline \(49\.1%49\.1\\%\)\. The outcome is not monotone in how far the weights move; direction and structure decide it\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x19.png)Figure 15:Per\-episode failure\-mode taxonomy by regime \(correct, valid\-but\-wrong target, premature\-finish, invalid or unparseable, truncated\)\. Reward\-hacking by premature finishing stays at roughly0to2%2\\%across every regime, so the null is not explained by reward\-gaming\. The moving regimes fail through output degeneration: degrade is63%63\\%invalid and collapse is98%98\\%invalid\. Flat and KL runs preserve the SFT structure \(about50%50\\%correct and40%40\\%valid\-but\-wrong\), the base capability ceiling rather than a new failure\.
### G\.4Generalization

![Refer to caption](https://arxiv.org/html/2607.12640v1/x20.png)Figure 16:Set\-of\-Marks \(vision\) learning\-rate sweep for the 4B model, paired against the SoM SFT baseline of63\.6%63\.6\\%\[50\.4,75\.1\]\[50\.4,75\.1\]\(35/55\), higher than the text baseline and itself a text\-versus\-SoM asymmetry\. No SoM config credibly beats the baseline: lr3×10−63\\times 10^\{\-6\}is\+3\.6\+3\.6pp \(p=0\.500p=0\.500\), lr1×10−51\\times 10^\{\-5\}is a nominal\+7\.3\+7\.3pp \(p=0\.219p=0\.219\), and lr2×10−52\\times 10^\{\-5\}is−10\.9\-10\.9pp \(p=0\.070p=0\.070, interval touching0\), a nominal\-but\-not\-credible degrade\. The no\-gain null generalizes to SoM; the credible lr\-collapse established on text is only nominal here\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x21.png)Figure 17:Text\-track learning\-rate sweep for the 8B model, paired against the 8B\-SFT baseline of47\.3%47\.3\\%\[34\.7,60\.2\]\[34\.7,60\.2\]\(26/55\)\. No 8B config credibly beats the baseline\. The low\-lr arms give the familiar nominal nudge \(lr3×10−63\\times 10^\{\-6\}up to52\.7%52\.7\\%, not credible\), while the moving regimes break: lr1×10−51\\times 10^\{\-5\}seed 2 is credibly worse \(36\.4%36\.4\\%,p=0\.031p=0\.031\) and both lr2×10−52\\times 10^\{\-5\}runs collapse to0\.0%0\.0\\%\(p<0\.001p<0\.001\)\. The null and the high\-lr breakage carry over from 4B to 8B\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x22.png)Figure 18:8B mechanism interpretability onN=372N=372fixed states\. The late\-layer \(L35\) effective rank matches the 4B pattern: SFT13\.613\.6, flat15\.115\.1, degrade collapsed to1\.91\.9, collapse preserved at19\.619\.6\. The honest scale nuance is that the 8B degrade keeps argmax agreement high \(0\.9540\.954\) with intact success despite the rank collapse, whereas the 4B degrade drops agreement to0\.070\.07to0\.50\.5\. So the rank leg of the dissociation reproduces at 8B while the rank\-to\-behavior coupling does not\.![Refer to caption](https://arxiv.org/html/2607.12640v1/x23.png)Figure 19:8B Set\-of\-Marks learning\-rate sweep, paired against the 8B\-SoM\-SFT baseline of63\.6%63\.6\\%\[50\.4,75\.1\]\[50\.4,75\.1\]\(35/55\)\. No 8B\-SoM config credibly beats the baseline: lr3×10−63\\times 10^\{\-6\}is the same low\-lr nominal nudge \(\+9\.1\+9\.1pp,p=0\.062p=0\.062, single seed\), while lr1×10−51\\times 10^\{\-5\}\(−18\.2\-18\.2pp,p=0\.002p=0\.002\) and lr2×10−52\\times 10^\{\-5\}\(−25\.5\-25\.5pp,p=0\.001p=0\.001\) are both credibly worse\. The lowest learning rate gives a nominal positive nudge that never clears the paired interval, here as at 4B\.

## Appendix HAlgorithms and Additional Definitions

This section states the three procedures the main text refers to by name, and recaps two estimators used in the evaluation\. Algorithm[1](https://arxiv.org/html/2607.12640#alg1)is the update we sweep; the only knobs the grid varies are the learning rateηt\\eta\_\{t\}, the KL weightβ\\beta, the clip pair\(ϵlo,ϵhi\)\(\\epsilon\_\{\\mathrm\{lo\}\},\\epsilon\_\{\\mathrm\{hi\}\}\), the seed, and the initialization ofθ\\theta\. Algorithm[2](https://arxiv.org/html/2607.12640#alg2)is the interval behind every “credible” verdict, and Algorithm[3](https://arxiv.org/html/2607.12640#alg3)is the causal test behind the localization claim\.

Algorithm 1GRPO update for the web agent1:policy

πθ\\pi\_\{\\theta\}, frozen initialization

πref\\pi\_\{\\mathrm\{ref\}\}, prompts, group size

GG, clip

\(ϵlo,ϵhi\)\(\\epsilon\_\{\\mathrm\{lo\}\},\\epsilon\_\{\\mathrm\{hi\}\}\), KL weight

β\\beta, schedule

ηt\\eta\_\{t\}
2:foreach round

ttand prompt

xxdo

3:sample

GGrollouts

\{τi\}\\\{\\tau\_\{i\}\\\}from

πθold\(⋅∣x\)\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid x\)
4:

ri←𝟏​\[success⁡\(τi\)\]r\_\{i\}\\leftarrow\\mathbf\{1\}\[\\operatorname\{success\}\(\\tau\_\{i\}\)\]⊳\\trianglerightsparse terminal reward

5:

Ai←ri−1G​∑jrjA\_\{i\}\\leftarrow r\_\{i\}\-\\tfrac\{1\}\{G\}\\sum\_\{j\}r\_\{j\}⊳\\trianglerightgroup mean\-centered, no std

6:foreach completion token

ccof each rolloutdo

7:

ρc←πθ​\(yc∣x,y<c\)/πθold​\(yc∣x,y<c\)\\rho\_\{c\}\\leftarrow\\pi\_\{\\theta\}\(y\_\{c\}\\mid x,y\_\{<c\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(y\_\{c\}\\mid x,y\_\{<c\}\)
8:

gc←min⁡\(ρc​A,clip⁡\(ρc,1−ϵlo,1\+ϵhi\)​A\)g\_\{c\}\\leftarrow\\min\\\!\\big\(\\rho\_\{c\}A,\\ \\operatorname\{clip\}\(\\rho\_\{c\},1\-\\epsilon\_\{\\mathrm\{lo\}\},1\+\\epsilon\_\{\\mathrm\{hi\}\}\)\\,A\\big\)
9:endfor

10:

ℒ←−1C​∑cgc\+β​D^KL​\(πθ∥πref\)\\mathcal\{L\}\\leftarrow\-\\tfrac\{1\}\{C\}\\sum\_\{c\}g\_\{c\}\+\\beta\\,\\widehat\{D\}\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\)
11:

θ←θ−ηt​∇θℒ\\theta\\leftarrow\\theta\-\\eta\_\{t\}\\,\\nabla\_\{\\theta\}\\mathcal\{L\}
12:endfor

Algorithm 2Task\-clustered paired bootstrap forΔ​SR\\Delta\\mathrm\{SR\}1:matched per\-episode outcomes for an arm and for SFT over tasks

𝒯\\mathcal\{T\}\(

\|𝒯\|=11\|\\mathcal\{T\}\|=11\) and seeds, resamples

BB
2:for

b=1b=1to

BBdo

3:draw

\|𝒯\|\|\\mathcal\{T\}\|task clusters from

𝒯\\mathcal\{T\}with replacement

4:

Δb←SRarm−SRSFT\\Delta\_\{b\}\\leftarrow\\mathrm\{SR\}\_\{\\mathrm\{arm\}\}\-\\mathrm\{SR\}\_\{\\mathrm\{SFT\}\}over the drawn clusters

5:endfor

6:returnthe

2\.52\.5and

97\.597\.5percentiles of

\{Δb\}\\\{\\Delta\_\{b\}\\\}
7:Verdict:*credibly better/worse*iff the interval excludes

0;*equivalent within*

δ\\deltaiff the

90%90\\%interval lies inside

\[−δ,\+δ\]\[\-\\delta,\+\\delta\]

Algorithm 3Causal component grafting1:trained weights

θ\\theta, initialization

θinit\\theta^\{\\mathrm\{init\}\}, component group

𝒢\\mathcal\{G\}\(attention, MLP, embedding, or a random parameter set of matched size\)

2:

θ′←θ\\theta^\{\\prime\}\\leftarrow\\theta
3:

θ𝒢′←θ𝒢init\\theta^\{\\prime\}\_\{\\mathcal\{G\}\}\\leftarrow\\theta^\{\\mathrm\{init\}\}\_\{\\mathcal\{G\}\}⊳\\trianglerightrestore one group, keep the rest

4:return

SR​\(πθ′\)\\mathrm\{SR\}\(\\pi\_\{\\theta^\{\\prime\}\}\)⊳\\trianglerightrecovery attributable to𝒢\\mathcal\{G\}

Two estimators complete the evaluation\. The headroom criterion that selects the positive\-control tasks compares the greedy success to the mean success under temperature sampling of the supervised policy,

Δhead=1K​∑k𝟏​\[success⁡\(τk\)\]−𝟏​\[success⁡\(τgreedy\)\],\\Delta\_\{\\mathrm\{head\}\}=\\tfrac\{1\}\{K\}\\textstyle\\sum\_\{k\}\\mathbf\{1\}\[\\operatorname\{success\}\(\\tau\_\{k\}\)\]\-\\mathbf\{1\}\[\\operatorname\{success\}\(\\tau\_\{\\mathrm\{greedy\}\}\)\],\(16\)averaged over evaluation seeds, and a task is climbable whenΔhead\>0\\Delta\_\{\\mathrm\{head\}\}\>0\. The random\-component null for grafting draws a setℛ\\mathcal\{R\}of non\-attention parameters with\|ℛ\|\|\\mathcal\{R\}\|equal to the attention parameter count, restoresℛ\\mathcal\{R\}from initialization, and repeats the draw2424times to form the null distribution whose mean and 95th percentile Table[15](https://arxiv.org/html/2607.12640#A5.T15)reports; a real graft is a specific cause only when its recovery exceeds that percentile\.