部分正确的工具缓存可能逆转组归一化策略更新

# Marginally Correct Tool Caches Can ReverseGroup-Normalized Policy Updates
Source: [https://arxiv.org/html/2609.26866](https://arxiv.org/html/2609.26866)
Shivam Gupta††thanks:LLM\-based tools assisted with research exploration, code, mathematical analysis, manuscript preparation, and internal review\. Internal automated reviews are not external peer review\.

20 September 2026

###### Abstract

Tool\-result caching reduces repeated execution in agent training, but also couples rollout randomness\. We study a two\-action model in which independent and shared execution preserve every rollout’s conditional reward distribution\. Despite this marginal agreement, sharing one stochastic result per group can reverse the expected group\-normalized policy update\. We derive an exact finite\-group expression: against a constant alternative, the shared update follows the probability of winning minus the probability of losing, rather than the difference in expected reward\. A Bernoulli specialization yields a wrong\-direction region and a non\-vanishing update\-variance floor as group size grows\. Centering without group standard\-deviation scaling preserves the expected\-return direction in this model, using an existing estimator control\. Exhaustive finite sums verify 540 configurations and 3,240 estimator evaluations, with a separate ordered\-sequence checker\. An implementation audit reproduces the sharing path in a pinned, unmodified TVCache stack using 256 scripted rollouts\. These results do not measure language\-model training performance or refute TVCache’s deterministic\-output contract\. They establish that marginal output validity alone cannot certify a stochastic cache as training\-equivalent\.

## 1Introduction

An execution optimization can change a learner without changing its model, prompt, reward function, or the marginal quality of any one tool response\. Consider a training group containing several identical calls to a stochastic tool\. Executing each call draws several outcomes\. Executing one call and broadcasting its result draws one outcome and repeats it\. Both procedures can return the correct distribution to each individual caller, while producing different joint distributions across the group\. A learner that compares rollouts within that group can distinguish the two procedures\.

This distinction matters for group\-relative training\. GRPO forms advantages from rewards centered and scaled within a group of completions\[[1](https://arxiv.org/html/2609.26866#bib.bib1)\]\. Stateful tool caches address a different correctness question: whether reuse respects the environment state that produced an output\[[6](https://arxiv.org/html/2609.26866#bib.bib6)\]\. State consistency is essential, but does not by itself preserve the sampling law of a stochastic tool\. This paper asks a narrower question:*if a cache preserves each rollout’s conditional reward distribution, must it preserve the expected group\-normalized update?*

The answer is no\. Our counterexample uses an action with a constant payoff and an action with a random payoff\. Under shared execution, all occurrences of the random action receive the same newly sampled result within a group\. The cache is refreshed for every group\. Consequently the effect requires neither stale state nor a permanently unlucky cached answer\. The normalizer removes the magnitude of the realized payoff gap, leaving only its sign\. For Bernoulli payoffs, the resulting preference threshold is1/21/2, even when the alternative has a substantially higher expected payoff\.

Group\-normalization bias and centering\-based alternatives are established\[[2](https://arxiv.org/html/2609.26866#bib.bib2),[3](https://arxiv.org/html/2609.26866#bib.bib3)\]\. We study how a controlled change in tool execution induces that bias, with exact statistical consequences and an auditable implementation path\. Our contributions are:

1. 1\.A finite\-group characterization of shared\-outcome updates for an arbitrary scalar reward distribution, with a Bernoulli sign\-reversal example\.
2. 2\.A separation between marginal reward agreement and update agreement, including asymptotic means, a persistent variance floor, and an existing centered\-estimator control\.
3. 3\.Exhaustive numerical checks and a pinned runtime audit that keep mathematical claims, scripted execution evidence, and untested deployment implications separate\.

The engineering motivation is practical: a lower physical tool\-call count is not, by itself, evidence of equivalent training\. The results identify a condition to test when optimizing expensive stochastic tools\. They do not measure production prevalence, model accuracy loss, or monetary savings\.

## 2Related work and scope

#### Group\-relative optimization\.

DeepSeekMath introduces GRPO as a critic\-free group\-relative policy optimization method\[[1](https://arxiv.org/html/2609.26866#bib.bib1)\]\. Liu et al\. identify biases associated with group\-level standard\-deviation scaling and response\-length normalization, and propose Dr\. GRPO\[[2](https://arxiv.org/html/2609.26866#bib.bib2)\]\. Our centered\-only control uses this existing principle\. We analyze a one\-step, on\-policy score estimator, not the complete clipped, multi\-epoch, token\-weighted training algorithm\.

#### Reward magnitude and stochastic feedback\.

Che et al\. show that in a sparse\-answer regime, group normalization can erase the magnitude of an error penalty and change a decision threshold\[[3](https://arxiv.org/html/2609.26866#bib.bib3)\]\. Our Bernoulli independent\-execution model is affinely equivalent to their three\-reward setting\. Sharing makes the relevant two\-level reward structure arise at every interior action probability, rather than only in a sparse\-action regime\. This close relationship limits any claim of conceptual novelty\. El Mansouri et al\. study correction for noisy rewards\[[4](https://arxiv.org/html/2609.26866#bib.bib4)\]; Xin measures within\-group verifier\-error dependence\[[5](https://arxiv.org/html/2609.26866#bib.bib5)\]\. Here the outcomes need not be erroneous: the intervention changes their dependence directly\.

#### Tool caching\.

TVCache reuses tool results across stateful trajectories\[[6](https://arxiv.org/html/2609.26866#bib.bib6)\]\. Its Appendix B assumes that tool outputs depend only on sandbox state and arguments\. Our stochastic setting is outside that deterministic\-output contract, so the counterexample does not refute its correctness argument\. CacheRL studies cached agent\-training environments, including cache\-aware reward design and masking of injected tool tokens\[[7](https://arxiv.org/html/2609.26866#bib.bib7)\]\. Correctly excluding environment tokens from the policy loss does not, by itself, establish equivalence of the joint reward law\. We neither reproduce those papers’ training results nor claim that their reported benchmarks exhibit our effect\.

## 3Model and estimators

LetG≥2G\\geq 2be the rollout\-group size\. Each rollout independently choosesai∼Bern⁡\(p\)a\_\{i\}\\sim\\operatorname\{Bern\}\(p\), where0<p<10<p<1andp=σ⁡\(θ\)p=\\sigma\(\\theta\)is parameterized by a logitθ\\theta\. Actionai=0a\_\{i\}=0, denoted A, returns the constantcc\. Actionai=1a\_\{i\}=1, denoted B, returns a scalar rewardYYwith finite first momentμ=𝔼⁡\[Y\]\\mu=\\mathbb\{E\}\[Y\]\. Actions are independent of tool randomness\. The expected return and its gradient are

J⁡\(θ\)=\(1−p\)​c\+p​μ,J′​\(θ\)=p⁡\(1−p\)​\(μ−c\)\.J\(\\theta\)=\(1\-p\)c\+p\\mu,\\qquad J^\{\\prime\}\(\\theta\)=p\(1\-p\)\(\\mu\-c\)\.\(1\)The policy score for a sampled action is∂θlog⁡πθ​\(ai\)=ai−p\\partial\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{i\}\)=a\_\{i\}\-p\.

#### Two execution laws\.

Under*independent execution*, each B invocation receives an independent copyYiY\_\{i\}\. Under*shared execution*, oneYYis drawn independently for the group and reused for every B invocation\. Drawing on the first B cache miss is equivalent to drawing in advance because actions are independent ofYY\. There is no persistence between groups\. In both modes,

ℒ⁡\(ri∣ai=0\)=δc,ℒ⁡\(ri∣ai=1\)=ℒ⁡\(Y\)\.\\mathcal\{L\}\(r\_\{i\}\\mid a\_\{i\}=0\)=\\delta\_\{c\},\\qquad\\mathcal\{L\}\(r\_\{i\}\\mid a\_\{i\}=1\)=\\mathcal\{L\}\(Y\)\.\(2\)Thus the intervention preserves conditional reward marginals, not only their means\. When two rollouts choose B, their outcomes are independent in the first mode and identical in the second\. This is the only distributional change\.

#### Updates\.

Writer¯=G−1​∑iri\\bar\{r\}=G^\{\-1\}\\sum\_\{i\}r\_\{i\}andsr2=G−1​∑i\(ri−r¯\)2s\_\{r\}^\{2\}=G^\{\-1\}\\sum\_\{i\}\(r\_\{i\}\-\\bar\{r\}\)^\{2\}\. We study the ascent estimators

Uϵ\\displaystyle U\_\{\\epsilon\}=1G​∑i=1G\(ai−p\)​ri−r¯sr\+ϵ,ϵ≥0,\\displaystyle=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\(a\_\{i\}\-p\)\\frac\{r\_\{i\}\-\\bar\{r\}\}\{s\_\{r\}\+\\epsilon\},\\qquad\\epsilon\\geq 0,\(3\)V\\displaystyle V=1G​∑i=1G\(ai−p\)​\(ri−r¯\)\.\\displaystyle=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\(a\_\{i\}\-p\)\(r\_\{i\}\-\\bar\{r\}\)\.\(4\)Atsr=0s\_\{r\}=0,U0U\_\{0\}is defined to be zero\. Positive updates increase the probability of B\. The denominator is the population standard deviation within the group\. Using the sample standard deviation changes the scale ofU0U\_\{0\}by a positive, group\-size\-dependent factor, and therefore does not change its sign\. We make no such scale\-equivalence claim for nonzeroϵ\\epsilonwithout adjusting that constant\.

These estimators describe the reward component of an on\-policy update at unit importance ratio\. We do not model KL penalties, clipping after parameter movement, adaptive optimizers, variable token lengths, or multi\-turn feedback\.

## 4Exact consequences of sharing

LetN=∑iai∼Bin⁡\(G,p\)N=\\sum\_\{i\}a\_\{i\}\\sim\\operatorname\{Bin\}\(G,p\)and define

wG​\(n\)=n⁡\(G−n\)G,hG​\(n\)=wG​\(n\)2,SG​\(p\)=𝔼⁡\[wG​\(N\)\]\.w\_\{G\}\(n\)=\\frac\{\\sqrt\{n\(G\-n\)\}\}\{G\},\\quad h\_\{G\}\(n\)=w\_\{G\}\(n\)^\{2\},\\quad S\_\{G\}\(p\)=\\mathbb\{E\}\[w\_\{G\}\(N\)\]\.\(5\)For interiorppandG≥2G\\geq 2,SG​\(p\)\>0S\_\{G\}\(p\)\>0\.

###### Theorem 1\(Shared outcomes replace payoff magnitude by ordering\)\.

Under shared execution, forϵ=0\\epsilon=0,

U0=wG​\(N\)​sgn⁡\(Y−c\),𝔼⁡\[U0\]=SG​\(p\)​\(Pr⁡\(Y\>c\)−Pr⁡\(Y<c\)\),U\_\{0\}=w\_\{G\}\(N\)\\operatorname\{sgn\}\(Y\-c\),\\qquad\\mathbb\{E\}\[U\_\{0\}\]=S\_\{G\}\(p\)\\bigl\(\\Pr\(Y\>c\)\-\\Pr\(Y<c\)\\bigr\),\(6\)wheresgn⁡\(0\)=0\\operatorname\{sgn\}\(0\)=0\. Consequently, conditional marginal agreement in \([2](https://arxiv.org/html/2609.26866#S3.E2)\) does not ensure agreement with the expected\-return direction\.

###### Proof\.

Centered rewards sum to zero, so the−p\-pscore terms cancel\. The centered numerator equalshG​\(N\)​\(Y−c\)h\_\{G\}\(N\)\(Y\-c\)\. The shared group contains only the two reward valuesccandYY, givingsr=wG​\(N\)​\|Y−c\|s\_\{r\}=w\_\{G\}\(N\)\|Y\-c\|\. Division yields \([6](https://arxiv.org/html/2609.26866#S4.E6)\) when0<N<G0<N<GandY≠cY\\neq c\. Both sides are zero otherwise\. FinallyNNandYYare independent, so their expectations factor\. ∎

The theorem concerns the*direction of the update*, not a claim that an arbitrary full learning algorithm maximizes a median objective\. It separates the frequency of beating an alternative from the value of doing so\. A small gain on many calls and a large loss on fewer calls can have positive ordering balance but negative mean advantage\.

###### Corollary 2\(Bernoulli threshold and wrong\-direction region\)\.

ForY∼Bern⁡\(q\)Y\\sim\\operatorname\{Bern\}\(q\)and0<c<10<c<1,

𝔼⁡\[U0,share\]=\(2​q−1\)​SG​\(p\)\.\\mathbb\{E\}\[U\_\{0,\\mathrm\{share\}\}\]=\(2q\-1\)S\_\{G\}\(p\)\.\(7\)For everyG≥2G\\geq 2and interiorpp, the shared normalized update has the opposite sign toJ′J^\{\\prime\}whenever1/2<q<c<11/2<q<c<1or0<c<q<1/20<c<q<1/2\.

The threshold isq=1/2q=1/2for the shared update andq=cq=cfor expected return\. This statement does not say that independent normalization is unbiased\. AtG=2G=2, both modes have the same expected update: a group containing both actions includes only one stochastic B result\.

###### Proposition 3\(Centering preserves the mean\-gradient direction\)\.

For either execution law and any integrableYY,

𝔼⁡\[V\]=\(1−1G\)​p​\(1−p\)​\(μ−c\)\.\\mathbb\{E\}\[V\]=\\left\(1\-\\frac\{1\}\{G\}\\right\)p\(1\-p\)\(\\mu\-c\)\.\(8\)Multiplication byG/\(G−1\)G/\(G\-1\)removes the self\-including\-baseline factor\.

###### Proof\.

Conditional onN=nN=n, the centered numerator is\(G−n\)\(∑i:ai=1ri−nc\)/G2\(G\-n\)\\bigl\(\\sum\_\{i:a\_\{i\}=1\}r\_\{i\}\-nc\\bigr\)/G^\{2\}\. Both laws give conditional sum expectationn​μn\\mu\. Apply𝔼⁡\[N⁡\(G−N\)\]=G⁡\(G−1\)​p​\(1−p\)\\mathbb\{E\}\[N\(G\-N\)\]=G\(G\-1\)p\(1\-p\)\. ∎

This is an existing baseline correction\. It does not restore the entire distribution of the update, nor establish equivalence after nonlinear clipping or adaptive optimizer transformations\.

###### Theorem 4\(A persistent mean shift and variance floor\)\.

SupposeY∼Bern⁡\(q\)Y\\sim\\operatorname\{Bern\}\(q\)with0<q<10<q<1,0<c<10<c<1, and fixed interiorpp\. AsG→∞G\\to\\infty,

𝔼⁡\[U0,ind\]\\displaystyle\\mathbb\{E\}\[U\_\{0,\\mathrm\{ind\}\}\]⟶p​\(1−p\)​\(q−c\)p​q​\(1−q\)\+p⁡\(1−p\)​\(q−c\)2,\\displaystyle\\longrightarrow\\frac\{p\(1\-p\)\(q\-c\)\}\{\\sqrt\{pq\(1\-q\)\+p\(1\-p\)\(q\-c\)^\{2\}\}\},\(9\)𝔼⁡\[U0,share\]\\displaystyle\\mathbb\{E\}\[U\_\{0,\\mathrm\{share\}\}\]⟶\(2​q−1\)​p⁡\(1−p\),\\displaystyle\\longrightarrow\(2q\-1\)\\sqrt\{p\(1\-p\)\},\(10\)Var⁡\(U0,ind\)\\displaystyle\\operatorname\{Var\}\(U\_\{0,\\mathrm\{ind\}\}\)⟶0,Var⁡\(U0,share\)⟶4​q​\(1−q\)​p​\(1−p\)\.\\displaystyle\\longrightarrow 0,\\qquad\\operatorname\{Var\}\(U\_\{0,\\mathrm\{share\}\}\)\\longrightarrow 4q\(1\-q\)p\(1\-p\)\.\(11\)At finiteGG, the shared variance is exactly

Var⁡\(U0,share\)=\(1−1/G\)​p​\(1−p\)−\(2​q−1\)2​SG​\(p\)2\.\\operatorname\{Var\}\(U\_\{0,\\mathrm\{share\}\}\)=\(1\-1/G\)p\(1\-p\)\-\(2q\-1\)^\{2\}S\_\{G\}\(p\)^\{2\}\.\(12\)

###### Proof\.

Under independent execution, laws of large numbers give the covariancep​\(1−p\)​\(q−c\)p\(1\-p\)\(q\-c\)and reward variancep​q​\(1−q\)\+p⁡\(1−p\)​\(q−c\)2pq\(1\-q\)\+p\(1\-p\)\(q\-c\)^\{2\}\. Under sharing, use \([6](https://arxiv.org/html/2609.26866#S4.E6)\) andN/G→pN/G\\to p\. In either mode, Cauchy–Schwarz gives\|U0\|≤\(N/G\)​\(1−N/G\)≤1/2\|U\_\{0\}\|\\leq\\sqrt\{\(N/G\)\(1\-N/G\)\}\\leq 1/2\. Bounded convergence therefore applies to first and second moments\. For the shared second moment,\(Y−c\)\(Y\-c\)is never zero, soU02=hG​\(N\)U\_\{0\}^\{2\}=h\_\{G\}\(N\)and𝔼⁡\[U02\]=\(1−1/G\)​p​\(1−p\)\\mathbb\{E\}\[U\_\{0\}^\{2\}\]=\(1\-1/G\)p\(1\-p\)\. Subtracting the squared mean proves \([12](https://arxiv.org/html/2609.26866#S4.E12)\) and its limit\. ∎

In the wrong\-direction regions, sufficiently large independent groups and shared groups point in opposite directions\. Increasing the number of policy samples does not increase the number of independent tool outcomes in the shared group\. Independent groups can still average down this noise; the result is a within\-group variance floor, not irreducible uncertainty across all data\.

### 4\.1Nonzero numerical stabilization

Forϵ\>0\\epsilon\>0in the Bernoulli setting, define

Aϵ=𝔼⁡\[hG​\(N\)​\(1−c\)wG​\(N\)​\(1−c\)\+ϵ\],Bϵ=𝔼⁡\[hG​\(N\)​cwG​\(N\)​c\+ϵ\]\.A\_\{\\epsilon\}=\\mathbb\{E\}\\\!\\left\[\\frac\{h\_\{G\}\(N\)\(1\-c\)\}\{w\_\{G\}\(N\)\(1\-c\)\+\\epsilon\}\\right\],\\qquad B\_\{\\epsilon\}=\\mathbb\{E\}\\\!\\left\[\\frac\{h\_\{G\}\(N\)c\}\{w\_\{G\}\(N\)c\+\\epsilon\}\\right\]\.\(13\)Then𝔼⁡\[Uϵ,share\]=q​Aϵ−\(1−q\)​Bϵ\\mathbb\{E\}\[U\_\{\\epsilon,\\mathrm\{share\}\}\]=qA\_\{\\epsilon\}\-\(1\-q\)B\_\{\\epsilon\}and the zero\-update threshold isqϵ=Bϵ/\(Aϵ\+Bϵ\)q\_\{\\epsilon\}=B\_\{\\epsilon\}/\(A\_\{\\epsilon\}\+B\_\{\\epsilon\}\)\. Forc\>1/2c\>1/2,1/2<qϵ<c1/2<q\_\{\\epsilon\}<c; forc<1/2c<1/2,c<qϵ<1/2c<q\_\{\\epsilon\}<1/2\. These inequalities follow termwise by comparingAϵA\_\{\\epsilon\}andBϵB\_\{\\epsilon\}, and\(1−c\)​Bϵ\(1\-c\)B\_\{\\epsilon\}withc​AϵcA\_\{\\epsilon\}\. Asϵ↓0\\epsilon\\downarrow 0the threshold tends to1/21/2; asϵ→∞\\epsilon\\to\\inftyit tends tocc\. A nonzero stabilizer therefore does not automatically restore the intended preference threshold\. Appendix[C](https://arxiv.org/html/2609.26866#A3)states the endpoint conventions\.

## 5Exhaustive numerical verification

### 5\.1Protocol and controls

Before executing the sweep, we fixedG∈\{2,4,8,16,32,64\}G\\in\\\{2,4,8,16,32,64\\\},p∈\{0\.1,0\.5,0\.9\}p\\in\\\{0\.1,0\.5,0\.9\\\},c∈\{0\.1,0\.3,0\.5,0\.7,0\.9\}c\\in\\\{0\.1,0\.3,0\.5,0\.7,0\.9\\\}, andq∈\{0,0\.2,0\.4,0\.6,0\.8,1\}q\\in\\\{0,0\.2,0\.4,0\.6,0\.8,1\\\}\. This gives 540 configurations\. For each configuration we evaluate both execution modes with centering alone and with normalization atϵ∈\{0,10−4\}\\epsilon\\in\\\{0,10^\{\-4\}\\\}, for 3,240 estimator evaluations\. These are finite probability\-weighted sums computed in floating arithmetic, not Monte Carlo observations or independent trials\.

WithN=nN=n, letKKcount successful B outcomes\. Independent execution usesK\|N=n∼Bin⁡\(n,q\)K\\mid N=n\\sim\\operatorname\{Bin\}\(n,q\); shared execution usesK∈\{0,n\}K\\in\\\{0,n\\\}with probabilities1−q,q1\-q,q\. The centered numerator is

V⁡\(n,k\)=\(G−n\)​\(k−n​c\)G2\.V\(n,k\)=\\frac\{\(G\-n\)\(k\-nc\)\}\{G^\{2\}\}\.\(14\)The implementation computes the reward variance from centered squared deviations, avoiding subtraction of nearly equal second moments\. It accumulates weighted sums with compensated summation\.

A separately implemented checker enumerates ordered rollout outcomes and evaluates the literal score expression, rather than the count reduction\. It checks 288 small\-group combinations\. Other controls verify probability mass, the reward mean, the centered identity, the shared formula, deterministicq∈\{0,1\}q\\in\\\{0,1\\\}equivalence, andG=2G=2equivalence\. All six numerical test methods pass\. Maximum probability\-mass error is below3\.34×10−153\.34\\times 10^\{\-15\}; maximum marginal\-mean error is below1\.78×10−151\.78\\times 10^\{\-15\}\. Numerical precision checks are not statistical confidence intervals\.

### 5\.2A preselected witness

The protocol selectedp=0\.5,c=0\.9,q=0\.8p=0\.5,c=0\.9,q=0\.8analytically before execution\. Both laws have mean reward0\.850\.85and true gradient−0\.025\-0\.025\. AtG=64G=64, independent normalization gives−0\.078867\-0\.078867, while sharing gives\+0\.297628\+0\.297628\. The centered estimator gives−0\.024609\-0\.024609under either law \(Table[1](https://arxiv.org/html/2609.26866#S5.T1)and Figure[1](https://arxiv.org/html/2609.26866#S5.F1)\)\. Thus the intervention reverses the expected update while leaving the conditional reward marginals unchanged\.

Table 1:Exact finite\-sum witness, rounded to six decimals\. Positive updates increase B, the lower\-mean action\. Normalized columns useϵ=0\\epsilon=0\.Figure 1:Increasing rollout count separates the two normalized updates in the preselected witness\. Centering alone preserves their expected direction\. Panels have different vertical scales\. Curves are exhaustive calculations, not learning curves or measured model performance\.Across the entire specified grid, the modes have opposite normalized\-update signs in 54 of 540 configurations at each testedϵ\\epsilon\. Independent normalization opposes the expected\-return gradient in 54 configurations, and shared normalization in 108\. Centering alone has no such sign discrepancy\. The deterministic controls agree exactly\. The grid was constructed to examine the mechanism; its fractions must not be interpreted as deployment prevalence\. Theϵ=10−4\\epsilon=10^\{\-4\}result verifies robustness to one practical numerical stabilizer, not all implementation conventions\.

## 6A pinned implementation audit

### 6\.1Execution boundary

To connect the intervention to executable infrastructure, we inspected and ran TVCache revision3a4f95a6582e\[[8](https://arxiv.org/html/2609.26866#bib.bib8)\]\. The adaptive follow\-up protocol was written after source inspection and after the numerical study, before executing the runtime probe\. The probe uses unmodified implementations of the semantic stateful executor, async client, Flask routes, and immutable prefix\-tree backend\. Only the HTTP transport is replaced with an in\-process bridge to Flask’s test client\. A scripted environment implements the tool interface\. Socket connections and DNS resolution are disabled during execution\.

Each case has 32 sequential one\-call rollouts, alternating A and B, with a new executor for each rollout and a shared backend within the case\. A returns the string0\.9\. B either always returns0\.8, or alternates1,0on physical execution, starting with1\. The latter is a deliberately varying stream, not a Bernoulli sample\. We compare direct execution, shared identity, a separate task namespace per rollout, and a separate draw identifier in tool arguments\.

Table 2:Runtime audit\. Counts are per case of 32 scripted rollouts\. Each mode is run once with constant outputs and once with a varying stream\.
### 6\.2Observed behavior and interpretation

All eight cases satisfy the protocol’s expectations \(Table[2](https://arxiv.org/html/2609.26866#S6.T2)\)\. The 256 scripted rollouts include 64 direct\-environment controls and 192 rollouts through the cache stack\. In total there are 196 physical tool calls and 648 in\-process HTTP requests\. The socket and DNS guards record zero attempts\. Three full runs in separate processes produce byte\-identical result JSON\. Before import, all 29 exported source files pass hash verification\.

The shared case executes only one A call and one B call\. Reuse preserves the constant\-output control and repeats the first B result in the varying fixture\. Identity\-separated controls retain all physical executions\. These controls trade away reuse; they are not a new or cost\-free fix\. The experiment establishes an implementation path for sharing, not a failure of the vendor’s stated deterministic contract or a measured training regression\.

The inspected video integration labels loading and preprocessing as state\-changing and includes a model\-backed caption call\. This motivates a separate variability measurement, which we have not performed\. Importantly, TVCache’s paper lists its EgoSchema training objective as importance sampling, while other workloads use GRPO\[[6](https://arxiv.org/html/2609.26866#bib.bib6)\]\. We therefore do not infer that its caption workload uses the group\-normalized estimator studied here\.

## 7Implications, limitations, and reproducibility

#### Cache equivalence is estimator\-dependent\.

For the mean\-centered estimator in this model, sharing preserves the expected gradient up to the usual finite\-group factor\. For the normalized estimator, it need not\. An appropriate training audit must specify both the execution law and the consuming estimator\. Matching tool\-response means or individual output distributions alone is weaker than matching the expected update\.

#### Cost must be reported separately\.

In the one\-step Bernoulli model, independent execution requiresG​pGpexpected physical B calls per group, whereas sharing requires1−\(1−p\)G1\-\(1\-p\)^\{G\}\. That saving comes with different gradient variance\. Equal\-rollout comparisons and equal\-physical\-call comparisons answer different questions\. Our work reports no time\-to\-accuracy or dollar\-efficiency result\. Appendix[B](https://arxiv.org/html/2609.26866#A2)gives exact centered\-estimator variances to support such comparisons without assuming that identical expected gradients imply identical learning behavior\.

#### Scope of the evidence\.

There are two actions, one decision per rollout, a fixed policy, and a constant alternative\. Theorems concern a specific on\-policy reward estimator\. We do not train or evaluate a language model, establish an end\-to\-end convergence result, or measure the frequency of stochastic tools in deployed agents\. Actual multi\-turn policies can respond to cached observations, changing their action distribution as well as their rewards\. Cache lifetime, concurrent misses, partial sharing, eviction, and stale state can introduce additional effects outside the model\. A production result would require representative tool distributions, measured reuse, repeated training runs, and equal\-budget comparisons with existing optimizers\. Automated internal review does not replace independent scientific review or the author’s responsibility for the manuscript\.

#### Reproducibility\.

The artifact contains pre\-execution protocols, the independent checker, all finite\-sum results, runtime traces, source and result hashes, exact dependency versions, and the figure\-generation code\. The numerical study requires only Python’s standard library; plotting uses Matplotlib\. The runtime probe uses Python 3\.12 with Flask 3\.1\.2, HTTPX 0\.28\.1, and Requests 2\.32\.5, with all transitive versions pinned\. Third\-party code is retrieved from the pinned Git revision, kept unmodified, and excluded from redistribution\. No credentials, paid API calls, participant data, or model outputs are required to reproduce the reported experiments\. The public artifact, including source and numerical results, is available at[https://github\.com/shi1720/tool\-cache\-coupling](https://github.com/shi1720/tool-cache-coupling)\.

#### Broader impact\.

The intended use is auditing training optimizations before relying on their efficiency claims\. A marginally valid cache can be useful and still change the learning problem\. Conversely, this constructed counterexample should not be used to assert that tool caching is generally unsafe or that a named system’s published results are invalid\.

## 8Conclusion

Sharing a stochastic tool result changes the joint sampling law seen by a group\-relative learner\. In a controlled model with identical conditional reward marginals, the shared normalized update follows an ordering comparison rather than the mean\-payoff advantage\. Sharing can reverse the expected normalized update and leave a within\-group variance floor as rollout count grows\. Exhaustive calculations and a pinned runtime audit make the mechanism reproducible\. The practical lesson is specific: evaluate stochastic reuse against the estimator it serves, and measure physical tool outcomes separately from rollout count\.

## References

- \[1\]Zhihong Shao et al\.DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.*arXiv preprint arXiv:2402\.03300*, 2024\.[https://arxiv\.org/abs/2402\.03300](https://arxiv.org/abs/2402.03300)\.
- \[2\]Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding R1\-Zero\-Like Training: A Critical Perspective\.*arXiv preprint arXiv:2503\.20783*, 2025\.[https://arxiv\.org/abs/2503\.20783](https://arxiv.org/abs/2503.20783)\.
- \[3\]Xujun Che, Yuchen Yuan, Weida Zhao, and Chenyang Yu\.Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error\-Penalized Reinforcement Learning\.*arXiv preprint arXiv:2608\.00301*, 2026\.[https://arxiv\.org/abs/2608\.00301](https://arxiv.org/abs/2608.00301)\.
- \[4\]Omar El Mansouri, Fathinah Asma Izzati, Mohamed El Amine Seddik, and Salem Lahlou\.Noise\-corrected GRPO: From Noisy Rewards to Unbiased Gradients\.*arXiv preprint arXiv:2510\.18924*, 2025\.[https://arxiv\.org/abs/2510\.18924](https://arxiv.org/abs/2510.18924)\.
- \[5\]Esther Xin\.Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2\.5 Rollouts\.*arXiv preprint arXiv:2609\.06386*, 2026\.[https://arxiv\.org/abs/2609\.06386](https://arxiv.org/abs/2609.06386)\.
- \[6\]Abhishek Vijaya Kumar, Bhaskar Kataria, Byungsoo Oh, Emaad Manzoor, and Rachee Singh\.TVCache: A Stateful Tool\-Value Cache for Post\-Training LLM Agents\.*arXiv preprint arXiv:2602\.10986*, 2026\.[https://arxiv\.org/abs/2602\.10986](https://arxiv.org/abs/2602.10986)\.
- \[7\]Md Amirul Islam, Sumiran Thakur, Huancheng Chen, Su Min Park, Jiayun Wang, and Gyuhak Kim\.CacheRL: Multi\-Turn Tool\-Calling Agents via Cached Rollouts and Hybrid Reward\.*arXiv preprint arXiv:2606\.14179*, 2026\.[https://arxiv\.org/abs/2606\.14179](https://arxiv.org/abs/2606.14179)\.
- \[8\]TVCache authors\.TVCache source repository, revision3a4f95a6582eea1e4b7e84a9f3a0c74eaa8fde02\.Accessed 20 September 2026\.[https://github\.com/TVCache/TVCache](https://github.com/TVCache/TVCache)\.

## Appendix AExact summation and equality controls

ForbG​\(n,p\)=\(Gn\)​pn​\(1−p\)G−nb\_\{G\}\(n;p\)=\\binom\{G\}\{n\}p^\{n\}\(1\-p\)^\{G\-n\}, the independent expected update is

𝔼⁡\[Uϵ,ind\]=∑n=0GbG​\(n,p\)​∑k=0n\(nk\)​qk​\(1−q\)n−k​\(G−n\)​\(k−n​c\)G2​\(s⁡\(n,k\)\+ϵ\),\\mathbb\{E\}\[U\_\{\\epsilon,\\mathrm\{ind\}\}\]=\\sum\_\{n=0\}^\{G\}b\_\{G\}\(n;p\)\\sum\_\{k=0\}^\{n\}\\binom\{n\}\{k\}q^\{k\}\(1\-q\)^\{n\-k\}\\frac\{\(G\-n\)\(k\-nc\)\}\{G^\{2\}\(s\(n,k\)\+\\epsilon\)\},\(15\)where the fraction is zero for a zero denominator and

m⁡\(n,k\)\\displaystyle m\(n,k\)=\(\(G−n\)​c\+k\)/G,\\displaystyle=\(\(G\-n\)c\+k\)/G,\(16\)s​\(n,k\)2\\displaystyle s\(n,k\)^\{2\}=\[\(G−n\)​\(c−m\)2\+k​\(1−m\)2\+\(n−k\)​m2\]/G\.\\displaystyle=\\bigl\[\(G\-n\)\(c\-m\)^\{2\}\+k\(1\-m\)^\{2\}\+\(n\-k\)m^\{2\}\\bigr\]/G\.\(17\)The shared sum replaces the inner binomial distribution with mass1−q1\-qatk=0k=0and massqqatk=nk=n\. Whenn=0n=0, only one outcome is counted, with total mass one\. The numerical implementation stores the update expectation and variance, probability mass, and expected reward for every estimator\.

The deterministic controlsq=0q=0andq=1q=1give the same reward vector under both laws, conditional on the action vector\. Their update distributions agree, not just their means\. ForG=2G=2, groups with equal actions have identical scores and a zero centered update even if their rewards differ\. Mixed\-action groups have one B sample, so the update distributions agree in that case too\. These facts explain the zero differences observed by the controls\.

## Appendix BVariance of the centered estimator

In the Bernoulli model, condition onNNand writeh=hG​\(N\)h=h\_\{G\}\(N\)\. The law of total variance gives

Var⁡\(Vshare\)\\displaystyle\\operatorname\{Var\}\(V\_\{\\mathrm\{share\}\}\)=q⁡\(1−q\)​𝔼​\[h2\]\+\(q−c\)2​Var⁡\(h\),\\displaystyle=q\(1\-q\)\\mathbb\{E\}\[h^\{2\}\]\+\(q\-c\)^\{2\}\\operatorname\{Var\}\(h\),\(18\)Var⁡\(Vind\)\\displaystyle\\operatorname\{Var\}\(V\_\{\\mathrm\{ind\}\}\)=q⁡\(1−q\)​𝔼​\[N​\(G−N\)2G4\]\+\(q−c\)2​Var⁡\(h\)\.\\displaystyle=q\(1\-q\)\\mathbb\{E\}\\\!\\left\[\\frac\{N\(G\-N\)^\{2\}\}\{G^\{4\}\}\\right\]\+\(q\-c\)^\{2\}\\operatorname\{Var\}\(h\)\.\(19\)The conditional means are the same,h⁡\(q−c\)h\(q\-c\)\. Sharing replaces the conditional variance of a sum ofNNindependent Bernoulli outcomes,N​q​\(1−q\)Nq\(1\-q\), byN2​q​\(1−q\)N^\{2\}q\(1\-q\)\. Substitution into the squared coefficient\(G−N\)2/G4\(G\-N\)^\{2\}/G^\{4\}proves the formulas\. Correcting the baseline factor multiplies either variance byG2/\(G−1\)2G^\{2\}/\(G\-1\)^\{2\}\.

These expressions make an important distinction explicit: a mean\-direction repair does not preserve gradient noise\. Any efficiency claim must account for the number of independently sampled groups and the physical call budget\. We do not turn these identities into a wall\-clock or convergence claim\.

## Appendix CStabilizer threshold and general reward ties

Forϵ\>0\\epsilon\>0,AϵA\_\{\\epsilon\}andBϵB\_\{\\epsilon\}are strictly positive at interiorppandcc\. Ifc\>1/2c\>1/2, then for every0<n<G0<n<G, the corresponding summands satisfyBn\>AnB\_\{n\}\>A\_\{n\}and\(1−c\)​Bn<c​An\(1\-c\)B\_\{n\}<cA\_\{n\}\. The first follows becausex/\(w​x\+ϵ\)x/\(wx\+\\epsilon\)is increasing in positivexx; the second follows by comparing the denominators after the common numeratorh​c​\(1−c\)hc\(1\-c\)is factored out\. Summing preserves both strict inequalities, yielding1/2<qϵ<c1/2<q\_\{\\epsilon\}<c\. The reversed inequalities apply whenc<1/2c<1/2; whenc=1/2c=1/2, the threshold is exactly1/21/2\. Boundary action counts contribute zero\. Finite sums permit both limits inϵ\\epsilonto be taken termwise\.

For general sharedYY, ties atY=cY=ccontribute zero\. The exact second moment is𝔼⁡\[U02\]=\(1−1/G\)​p​\(1−p\)​Pr⁡\(Y≠c\)\\mathbb\{E\}\[U\_\{0\}^\{2\}\]=\(1\-1/G\)p\(1\-p\)\\Pr\(Y\\neq c\)\. Thus the variance extension of Theorem[4](https://arxiv.org/html/2609.26866#Thmtheorem4)subtractsSG​\(p\)2​\[Pr⁡\(Y\>c\)−Pr⁡\(Y<c\)\]2S\_\{G\}\(p\)^\{2\}\[\\Pr\(Y\>c\)\-\\Pr\(Y<c\)\]^\{2\}from this expression\. Finite first moment is needed to compare with expected return; the bounded normalized update itself is well\-defined for any almost surely finite scalarYY\.

## Appendix DRuntime audit boundaries

The runtime audit tests a narrow execution path: sequential one\-call rollouts, no stored sandbox forks, no eviction, no expiry, and no production network\. The backend’s cleanup threads are stopped after each case\. Distinct identities are supplied by the experiment, not discovered by an agent\. The varying\-output fixture is not the probabilistic environment used in the theorem, and its returned means must not be used as evidence for marginal agreement\. That agreement is proved and numerically verified only in the analytical study\.