Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation

# Efficiently Estimating Uncertainty Dynamics in Text Generation
Source: [https://arxiv.org/html/2608.19611](https://arxiv.org/html/2608.19611)
Amir ZurSatchel GrantTal HaklayCan RagerOwen LewisAffiliation:Thomas McGrath, Jack Merullo, Ekdeep Singh Lubana, Atticus GeigerAffiliation:![[Uncaptioned image]](https://arxiv.org/html/2608.19611v1/gf_color.png)

###### Abstract

LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i\.e\., its uncertainty\. Resampling\-based analyses characterize this distribution, revealing which steps of a rollout determine how the model arrives at its answer\. However, a major limitation of these approaches is that resampling text sequences at every token or sentence in a reasoning chain is very costly\. Our work strives to make resampling analysis more computationally efficient, while also shedding light on an important scientific question: what is the right statistical model for explaining uncertainty dynamics in text generation? We show that when resampling many reasoning chains, uncertainty dynamics converge to stable patterns, and noise is largely an artifact of sampling rather than an LLM’s sensitivity to each individual token or reasoning step\. We develop a statistical model for smoothing noisy low\-sample rollout data to better approximate high\-sample data, allowing us to significantly cut sampling costs\.

## 1Introduction

††footnotetext:Code and interactive dashboard:[https://github\.com/ericb\-goodfire/forking\-fast](https://github.com/ericb-goodfire/forking-fast)When Large Language Models \(LLMs\) engage in long\-form text generation such as reasoning, there are many possible paths they might take at each step or token that they generate\. Prior work has shown that specific steps during reasoning can have a significant impact on the model’s overall behavior, while many other steps are not as important\([1](https://arxiv.org/html/2608.19611#bib.bib1);[3](https://arxiv.org/html/2608.19611#bib.bib8);[15](https://arxiv.org/html/2608.19611#bib.bib15);[11](https://arxiv.org/html/2608.19611#bib.bib9);[4](https://arxiv.org/html/2608.19611#bib.bib14)\)\. One method for understanding which steps of reasoning \(and text generation more generally\) are most important is resampling: given a single reasoning rollout, resample alternate rollouts at each step of reasoning, then collect the final answers of all rollouts and aggregate them into a distribution\.

Resampling methods such as Forking Paths Analysis \(FPA\)\([1](https://arxiv.org/html/2608.19611#bib.bib1)\)can provide critical insight into uncertainty dynamics in text generation, where uncertainty measures how likely an LLM is to pick one answer over another\. However, this approach is very costly and often requires millions of tokens to analyze a single reasoning chain\. In this work, we propose a more efficient way to approximate these uncertainty dynamics by developing a statistical model of their distribution, and using this model to smooth noisy low\-sample estimates\.

We evaluate our approach by studying Chain\-of\-Thought reasoning\([10](https://arxiv.org/html/2608.19611#bib.bib13)\)in Llama\-3\-8B\-Instruct\([7](https://arxiv.org/html/2608.19611#bib.bib6)\)and the native reasoning model DeepSeek\-R1\-Distill\-Llama\-8B\([6](https://arxiv.org/html/2608.19611#bib.bib7)\)as they solve problems in tinyMMLU\([12](https://arxiv.org/html/2608.19611#bib.bib4);[8](https://arxiv.org/html/2608.19611#bib.bib5)\)\. After collecting and analyzing reasoning data with nearly two billion tokens in total, we find three main results:

1. 1\.When resampling a small number of times, the uncertainty dynamics of reasoning rollouts are quite noisy[1](https://arxiv.org/html/2608.19611#bib.bib1)\. However, when resampling hundreds of times, the uncertainty dynamics become increasingly smooth, except at key forking points which have sharp changes\. We find that variation across reasoning rollouts is modeled well as multinomial sampling noise \(§[3\.3](https://arxiv.org/html/2608.19611#S3.SS3)\), and reconstruction error decays in proportion to the square root of the amount of samples collected\.
2. 2\.We develop a statistical model which allows us to approximate high\-sample uncertainty dynamics by smoothing lower\-sample data\. This model uses change point detection to identify forking points, and kernel\-pooling to smooth estimates between these points\. By smoothing low\-sample data, we can effectively multiply the effective sample size by3\.3×3\.3\\times\(S=30S=30\) to5×5\\times\(S=5S=5\) \(§[3\.1](https://arxiv.org/html/2608.19611#S3.SS1)\)\.
3. 3\.We find that resampling everyNNtokens or steps, instead of every step, can improve data efficiency without smoothing\. However, this approach loses precision in estimating forking points, and it benefits less from smoothing compared toN=1N=1\. By combining this approach with our smoothing model, we can cut the total budget by1/81/8with only a small increase in error \(§[3\.2](https://arxiv.org/html/2608.19611#S3.SS2)\)\.

Figure 1:Uncertainty dynamicsoto\_\{t\}for one question \(tinyMMLU question 39; Llama\-3\-8B\-Instruct\)\. Colors represent the fraction of rollouts atttthat end with different final answers\. \(a\) When collecting a massive number of resampled rollouts \(S=1000S\{=\}1000\) at every token \(N=1N\{=\}1\), the outcome distribution becomes increasingly smooth, except at forking points\. \(c\) Uncertainty dynamics with a lower\-sample analysis \(S=20S\{=\}20,N=2N\{=\}2\) using1%1\\%of the reference’s sampling cost\. \(b\) Smoothing the lower\-sample dataoto\_\{t\}closely approximates the high\-sample data and recovers the same forking points\. More examples in Appendix[B](https://arxiv.org/html/2608.19611#A2)and our interactive dashboard \([https://github\.com/ericb\-goodfire/forking\-fast](https://github.com/ericb-goodfire/forking-fast)\)\.
## 2Methods

#### Forking Paths Analysis

Following[1](https://arxiv.org/html/2608.19611#bib.bib1), we analyze uncertainty dynamics relative to fixed reasoning chainxx, abase path, generated by greedy decoding\. At each token or sentence positionttper intervalNN, for example every token or every sentence \(N=1N\{=\}1\), we resample continuations from the prefixx<tx\_\{<t\}\. Specifically, for every alternative token with probabilityp⁡\(xt∣x<t\)≥0\.05p\(x\_\{t\}\\mid x\_\{<t\}\)\\geq 0\.05, we sampleSScontinuationsx\>tx\_\{\>t\}with temperatureτ=1\.0\\tau=1\.0and extract each continuation’s outcome, in our case a multiple choice answerA/B/C/D/Other\. We then aggregate these final answers into a weighted distribution over outcomes, such that each timestepttindexes a distributionoto\_\{t\}over final answers weighted by the probability of the tokenp⁡\(xt∣x<t\)p\(x\_\{t\}\\mid x\_\{<t\}\)and continuationp⁡\(x\>t∣x≤t\)p\(x\_\{\>t\}\\mid x\_\{\\leq t\}\), which can be visualized as a timeseries \(Fig\.[1](https://arxiv.org/html/2608.19611#S1.F1)\)\.

In this work, we consider two methods for reducing the sampling cost of FPA: decreasing the number of samplesSSat each token position, and increasing the sampling interval to resample everyNNtokens or sentences, instead of at every step \(i\.e\.N=1N\{=\}1\)\. Our goal is to use this lower\-sample data to approximate uncertainty dynamics inoto\_\{t\}with highSSandN=1N\{=\}1\.

#### Estimating uncertainty dynamics

When do sharp changes in the outcome distributionoto\_\{t\}correspond to decision points in the reasoning path, and when do they correspond to sampling noise? Figure[1](https://arxiv.org/html/2608.19611#S1.F1)a shows a referenceoto\_\{t\}with a high resample rateS=1000S\{=\}1000andN=1N\{=\}1\. We find that with large amounts of resampling data, the outcome distributionoto\_\{t\}becomes increasingly smooth at almost all positionstt, except for keyforking pointswhereoto\_\{t\}suddenly and dramatically changes\([1](https://arxiv.org/html/2608.19611#bib.bib1)\)\. Compare this to Figure[1](https://arxiv.org/html/2608.19611#S1.F1)c, which shows a lower\-sample outcome distributionoto\_\{t\}withS=20S\{=\}20andN=2N\{=\}2\. It is difficult to visually discern from the lower\-sample data whether sharp changes inoto\_\{t\}are genuine uncertainty dynamics, or whether these fluctuations are an artifact of sampling noise\.

Since we drawSScontinuations independently for each possible next\-tokenxt=wx\_\{t\}=w, the answer counts form a multinomial distributionct\(w\)∼Multinomial⁡\(S,ot\(w\)\)c\_\{t\}^\{\(w\)\}\\sim\\mathrm\{Multinomial\}\(S,o\_\{t\}^\{\(w\)\}\), and the branch weightsp~w\\tilde\{p\}\_\{w\}are known exactly from the next\-token distributionp⁡\(xt=w∣x<t\)p\(x\_\{t\}=w\\mid x\_\{<t\}\), so the weighted estimate ofoto\_\{t\}has variance∑wp~w2​ot,k\(w\)​\(1−ot,k\(w\)\)/S\\sum\_\{w\}\\tilde\{p\}\_\{w\}^\{2\}\\,o^\{\(w\)\}\_\{t,k\}\(1\-o^\{\(w\)\}\_\{t,k\}\)/Sfor each answer categorykk\. Sampling a branch weighted byp~w\\tilde\{p\}\_\{w\}and one of its continuations yields i\.i\.d\. draws fromoto\_\{t\}itself, so counts constructed this way satisfyct∼Multinomial⁡\(S,ot\)c\_\{t\}\\sim\\mathrm\{Multinomial\}\(S,o\_\{t\}\)exactly\. Therefore, we expect sampling noise to decrease in proportion to1/S1/\\sqrt\{S\}\. That is, we expect a negative linear relation between the log distance between the true outcome distribution \(which we approximate with highSSandN=1N\{=\}1\) andS\\sqrt\{S\}as we increase the number of resampled outputs\. We empirically test this in Section[3\.3](https://arxiv.org/html/2608.19611#S3.SS3)\.

#### Statistical model

If we simply increase the sample countSS, we can expect to get closer to the model’s true uncertainty dynamicsoto\_\{t\}at a rate ofS\\sqrt\{S\}\. Next, we propose a method for more efficiently approximating the trueoto\_\{t\}using reduced sampling rates by varyingSSandNN\.

Our statistical model builds on a few qualitative observations about the shape ofoto\_\{t\}\. We observe, as do[1](https://arxiv.org/html/2608.19611#bib.bib1), that uncertainty dynamics inoto\_\{t\}follow a pattern where large segments ofoto\_\{t\}are relatively stable, remaining flat or very slowly drifting overtt\. These stable segments are separated by sharp changes atforking points, where the outcome distribution suddenly changes, sometimes after only a single token\.

Our model follows a three\-stage process: change point segmentation, kernel\-weighted Dirichlet pooling, and cross\-validation tuning\. First, we apply pruned exact linear time \(PELT\) change point detection\([9](https://arxiv.org/html/2608.19611#bib.bib2);[14](https://arxiv.org/html/2608.19611#bib.bib3)\)with an exact multinomial cost\. These change points divide the base path into segments in which we expect the outcome distribution to either remain flat or slowly drift\. Next, within these segments we pool answer counts from neighboring data points, weighting them with a Gaussian kernel, and use the result to parameterize a Dirichlet distribution\. Finally, we use cross\-validation to tune three hyperparameters of the model: the cost function \(multinomial likelihood vs\.L2L\_\{2\}loss\) and penalty value for PELT, and the kernel bandwidth used for Gaussian smoothing within segments\. This cross\-validation operates over the same data being smoothed, by splitting e\.g\.S=20S\{=\}20into 5 folds each withS=4S\{=\}4data points and finding the hyperparameters for eachS=16S\{=\}16that best explains the held out fold\.

We evaluate the overall success of our statistical model by measuring total variation distance \(TVD\) of smoothed low\-sampleoto\_\{t\}relative to ground truth high\-sampleoto\_\{t\}data withS=200,N=1S\{=\}200,N\{=\}1\. TVD measures the distance between two distributionsppandqq:TVD​\(p,q\)=12​∑k=1K∣pk−qk∣\\text\{TVD\}\(p,q\)=\\frac\{1\}\{2\}\\sum\_\{k=1\}^\{K\}\\mid p\_\{k\}\-q\_\{k\}\\mid\. In Appendix[E](https://arxiv.org/html/2608.19611#A5)we report ablations to our model, which confirm that PELT can accurately identify forking points and that kernel pooling produces accurate flat regions\.

Figure 2:Accuracy versus number of resampled rollouts\.We find that with raw data observationsoto\_\{t\}\(dotted lines\), greater resample spacingNNleads to slightly lower accuracy with equalSS, with considerably more difference at higherSS\. This effect is significantly more pronounced with the smoothedoto\_\{t\}\(solid lines\), which also shows that increasinglog⁡S\\log Shas a consistent linear effect on log\-TVD\. LargerNNhaving decreasing accuracy for a givenSShas a relatively simple explanation, since this means fewer samples collected overall\. Accuracy on the y\-axis is measured by total variation distance \(TVD\) relative to theS=200S=\{200\}referenceoto\_\{t\}\.

## 3Experimental Results

#### Data\.

We analyze forking paths on the tinyMMLU dataset\([12](https://arxiv.org/html/2608.19611#bib.bib4)\)\(n=100n\{=\}100\), collecting a high\-sample reference FPA dataset withS=200,N=1S\{=\}200,N\{=\}1\. For LLama\-3\-8B\-Instruct, we resample at every token, whereas for DeepSeekR1\-Distill\-Llama\-8B we resample at every sentence, since reasoning models generate much longer reasoning chains\. In total, we collected1\.771\.77B tokens for this FPA dataset\.

We evaluate forking path analyses with decreasing sampling ratesS∈\[5,100\]S\\in\[5,100\]and intervalsN∈\{1,2,4,8,16,32\}N\\in\\\{1,2,4,8,16,32\\\}, against the reference analysis withS=200S\{=\}200andN=1N=\{1\}\. We report total variation distance \(TVD\) between this reference FPA against each lower\-sample analysis to characterize the tradeoff between precision and sampling efficiency\. For each of these conditions, we then use our statistical model to produced smoothedoto\_\{t\}estimates, and compare these to theS=200S\{=\}200data\.

### 3\.1Estimatingoto\_\{t\}from fewer samples

Across all questions, the statistical model’s excess TVD above the reference noise floor falls monotonically from0\.02650\.0265\(S=5S\{=\}5\) to0\.00560\.0056\(S=100S\{=\}100\)\. For Llama, the sample efficiency multiplier at 4\-token spacing is5\.0×5\.0\\timesatS=5S\{=\}5,3\.3×3\.3\\timesatS=30S\{=\}30, and1\.0×1\.0\\timesatS=100S\{=\}100\(Figure[2](https://arxiv.org/html/2608.19611#S2.F2), dark blue lines\)\. At 1\-token spacing, the sample efficiency multiplier grows to22\.1×22\.1\\timesatS=5S\{=\}5,7\.3×7\.3\\timesatS=30S\{=\}30, and1\.6×1\.6\\timesatS=100S\{=\}100\(Figure[2](https://arxiv.org/html/2608.19611#S2.F2), red lines\)\. We find similar results with the reasoning model’s sentence resampling with multipliers4\.9×4\.9\\timesatS=5S\{=\}5,3\.1×3\.1\\timesatS=30S\{=\}30, and1\.1×1\.1\\timesatS=100S\{=\}100\(Appendix[C](https://arxiv.org/html/2608.19611#A3)\)\.

We find that with raw data, increasing the spacing between observation has relatively little effect on accuracy overall \(Fig\.[2](https://arxiv.org/html/2608.19611#S2.F2), dotted lines\), except with highSS\. This may be because sampling noise dominates fluctuations inoto\_\{t\}unlessSSis sufficiently high \(e\.g\., Fig\.[1](https://arxiv.org/html/2608.19611#S1.F1), Bottom\)\. An exception to this trend is regions around forking points, where using a wider stride gives less accurateoto\_\{t\}estimates \(Fig\.[7](https://arxiv.org/html/2608.19611#A4.F7)\)\. However, the smoothed data shows a very different pattern, where higher sampling granularity reliably leads to more accurateoto\_\{t\}estimates \(Fig\.[2](https://arxiv.org/html/2608.19611#S2.F2), solid lines\), which also holds for forking regions \(Fig\.[7](https://arxiv.org/html/2608.19611#A4.F7)\)\.

Figure 3:Accuracy versus token cost\.We find that overall, with rawoto\_\{t\}data, wider spacing between resampled pointsNNleads to better accuracy with lower overall token cost \(dotted lines\)\. Our statistical model significantly increases the accuracy for mostSSandNN, but also negates the improvement of largerNNincreasing accuracy with equal cost \(solid lines\)\. TVD between theS=200S\{=\}200reference and each low\-sample estimator is shown on the y\-axis, with colors representing differentNN\. The black star represents the sampling parameters used in\([1](https://arxiv.org/html/2608.19611#bib.bib1)\)\.
### 3\.2Tradeoff between accuracy and cost

Figure[3](https://arxiv.org/html/2608.19611#S3.F3)maps accuracy against token cost, comparing the raw data against model\-smoothed data\. Here, we see a clear pattern with the raw data: wider token strides lead to more accurate approximations of high\-sample data, given the same token budget\. Intuitively, averaging data over a fixed window leads to theoto\_\{t\}estimates having lower variance, since sampling noise is averaged out\. When data is smoothed, however, we instead see that changing sampling window has relatively little effect on token efficiency, but instead represents a different section of the same Pareto frontier\. In other words, with smoothed data, changingSSand the spacing between observation samplestthas roughly the same effect \(Fig\.[3](https://arxiv.org/html/2608.19611#S3.F3), solid lines\)\.

### 3\.3Sampling noise decreases with sample size

To measure how sampling noise changes with sample size, we divided theS=1000S\{=\}1000samples for each of two questions into disjoint, equally sized subsets\. At each position, we then computed the pairwise total variation distance \(TVD\) between the outcome distributions estimated from these subsets\. As Figure[11](https://arxiv.org/html/2608.19611#A6.F11)shows, the pooled TVD between estimates from these independent sample subsets decreases steadily asSSincreases\. On log\-log axes, the fitted slope is−0\.4903\-0\.4903throughS=200S\{=\}200, closely matching the−1/2\-1/2slope expected under independent multinomial sampling, suggesting that noise decreases in proportion to1/S1/\\sqrt\{S\}\.

The fullS=200S\{=\}200dataset shows similar scaling, with slopes of−0\.4757\-0\.4757for Llama when resampling every token and−0\.4838\-0\.4838for DeepSeek when resampling every sentence\. AtS=200S\{=\}200, the variation across runs \(TVD=0\.0451\\text\{TVD\}=0\.0451\) is nearly identical to the sampling variation within a run \(TVD=0\.0442\\text\{TVD\}=0\.0442\)\. These results support modeling the observed counts at each position as multinomial samples:ct∼Mult⁡\(S,ot\)c\_\{t\}\\sim\\mathrm\{Mult\}\(S,o\_\{t\}\)\.

## 4Discussion

Here we have shown that heavy cost of resampling steps in reasoning and text generation is in fact a modeling problem\. The noise ofoto\_\{t\}is exactly multinomial and the curves are smooth with sparse forking points, so a large resampling budget mostly re\-measures structure that neighboring positions already contain\. An estimator that pools within segments and breaks at change points converts that structure into∼15×\\sim 15\\timesefficiency gains within segments, and∼4×\\sim 4\\timesefficiency gains without losing accuracy in identifying forking points\.

This work opens a number of exciting avenues for future work\. It suggests that there could be a mechanistic theory of in\-context learning that can explain these kinds of learning dynamics\([13](https://arxiv.org/html/2608.19611#bib.bib10);[2](https://arxiv.org/html/2608.19611#bib.bib11)\)\. Finally, while our method enables us to post\-hoc smooth approximate outcome distributionsoto\_\{t\}, an alternate approach which integrates modeling would be to more efficiently choose particular token indices to sample from, similar to approaches in optimal experiment design\([5](https://arxiv.org/html/2608.19611#bib.bib12)\)\.

## References

- Bigelowet al\.\(2024a\)E\. Bigelow, A\. Holtzman, H\. Tanaka, and T\. UllmanForking paths in neural text generation\.External Links:2412\.07961,[Link](https://arxiv.org/abs/2412.07961)Cited by:[item 1](https://arxiv.org/html/2608.19611#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.19611#S1.p1.1),[§1](https://arxiv.org/html/2608.19611#S1.p2.1),[§2](https://arxiv.org/html/2608.19611#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.19611#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.19611#S2.SS0.SSS0.Px3.p2.1),[Figure 3](https://arxiv.org/html/2608.19611#S3.F3)\.
- Bigelowet al\.\(2024b\)E\. Bigelow, E\. S\. Lubana, R\. Dick, H\. Tanaka, and T\. UllmanIn\-context learning dynamics with random binary sequences\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 56330–56373\.Cited by:[§4](https://arxiv.org/html/2608.19611#S4.p2.1)\.
- Bogdanet al\.\(2025\)P\. C\. Bogdan, U\. Macar, N\. Nanda, and A\. ConmyThought anchors: which llm reasoning steps matter?\.arXiv preprint arXiv:2506\.19143\.Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p1.1)\.
- Boppanaet al\.\(2026\)S\. Boppana, A\. Ma, M\. Loeffler, R\. Sarfati, E\. Bigelow, A\. Geiger, O\. Lewis, and J\. MerulloReasoning theater: disentangling model beliefs from chain\-of\-thought\.arXiv preprint arXiv:2603\.05488\.Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p1.1)\.
- Chaloner and Verdinelli \(1995\)K\. Chaloner and I\. VerdinelliBayesian experimental design: a review\.Statistical science,pp\. 273–304\.Cited by:[§4](https://arxiv.org/html/2608.19611#S4.p2.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang,et al\.DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p3.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian,et al\.The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p3.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p3.1)\.
- Killicket al\.\(2012\)R\. Killick, P\. Fearnhead, and I\. A\. EckleyOptimal detection of changepoints with a linear computational cost\.Journal of the American Statistical Association107\(500\),pp\. 1590–1598\.Cited by:[item Segment\+Pool:](https://arxiv.org/html/2608.19611#A5.I1.ix3.p1.1),[§2](https://arxiv.org/html/2608.19611#S2.SS0.SSS0.Px3.p3.1)\.
- Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.Advances in neural information processing systems35,pp\. 22199–22213\.Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p3.1)\.
- Macaret al\.\(2026\)U\. Macar, P\. Bogdan, S\. Rajamanoharan, and N\. NandaThought branches: interpreting llm reasoning requires resampling\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 92567–92600\.Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p1.1)\.
- Maia Poloet al\.\(2024\)F\. Maia Polo, L\. Weber, L\. Choshen, Y\. Sun, G\. Xu, and M\. YurochkinTinyBenchmarks: evaluating LLMs with fewer examples\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p3.1),[§3](https://arxiv.org/html/2608.19611#S3.SS0.SSS0.Px1.p1.1)\.
- Nandaet al\.\(2023\)N\. Nanda, L\. Chan, T\. Lieberum, J\. Smith, and J\. SteinhardtProgress measures for grokking via mechanistic interpretability\.arXiv preprint arXiv:2301\.05217\.Cited by:[§4](https://arxiv.org/html/2608.19611#S4.p2.1)\.
- Truonget al\.\(2020\)C\. Truong, L\. Oudre, and N\. VayatisSelective review of offline change point detection methods\.Signal Processing167,pp\. 107299\.Cited by:[item Segment\+Pool:](https://arxiv.org/html/2608.19611#A5.I1.ix3.p1.1),[Appendix G](https://arxiv.org/html/2608.19611#A7.p1.1),[§2](https://arxiv.org/html/2608.19611#S2.SS0.SSS0.Px3.p3.1)\.
- Zhanget al\.\(2025\)A\. Zhang, Y\. Chen, J\. Pan, C\. Zhao, A\. Panda, J\. Li, and H\. HeReasoning models know when they’re right: probing hidden states for self\-verification\.arXiv preprint arXiv:2504\.05419\.Cited by:[§1](https://arxiv.org/html/2608.19611#S1.p1.1)\.

## Appendix ALimitations

Scope\.Our experiments are limited to tinyMMLU multiple\-choice questions and two 8B\-parameter models: one instruction\-tuned model and one reasoning\-distilled model\. We analyze five\-class outcome distributions over relatively short rollouts \(continuations of at most 400 tokens for Llama and 1536 tokens for DeepSeek\)\. Whether the results generalize to longer horizons, open\-ended outcomes, or larger models remains to be tested\.

Hyperparameter tuning\.The Full Model and its hyperparameter tuning procedure were developed using two questions from one model\. The tinyMMLUS=200S\{=\}200dataset and all DeepSeek data were used only for evaluation; however, they come from the same task family and use models at the same parameter scale as the development data\.

Estimator effectiveness depends on how big the forks are\.The estimator’s advantage in fork regions is established at a forking threshold of0\.100\.10, but it degrades at0\.150\.15\(n=35n\{=\}35questions\), and at the largest forks of0\.200\.20\(n=21n\{=\}21\), smoothing is a marginally worse estimator than raw data\.

Point estimates only\.The estimators’ 90% credible intervals do not achieve their nominal coverage on the observed data: the Full Model’s empirical coverage ranges from0\.480\.48to0\.640\.64across the development questions\. We therefore report and recommend the point estimates rather than the credible intervals\.

Reference curves are estimates\.The reference curves are themselves finite\-sample estimates \(S=200S\{=\}200orS=1000S\{=\}1000\) and thus have a nonzero noise floor\. Figure\-level TVDs computed against the full reference include this floor\. For hypothesis\-bearing comparisons, we use leave\-replicate\-out references to prevent overlap between an estimate and its reference from artificially reducing the measured error\.

## Appendix BAdditional Example Reconstructions

Figure[4](https://arxiv.org/html/2608.19611#A2.F4)extends the comparison in Figure[1](https://arxiv.org/html/2608.19611#S1.F1)to three questions from the tinyMMLUS=200S\{=\}200evaluation set\. For each question, the reference is theS=200S\{=\}200every\-token curve\. The reduced run usesS=15S\{=\}15at 4\-token spacing, with samples nested within the same set of draws\.

Figure 4:Reconstructions for three questions from the full tinyMMLU evaluation set using Llama\-3\-8B\-Instruct\. From top to bottom, the rows show theS=200S\{=\}200every\-token reference, the Full Model’s reconstruction from a nestedS=15S\{=\}15run at 4\-token spacing, and Raw Data from the same reduced run\.
## Appendix CDeepSeek\-R1\-Distill Results

Figures[5](https://arxiv.org/html/2608.19611#A3.F5)and[6](https://arxiv.org/html/2608.19611#A3.F6)reproduce the analyses in main\-text Figures[2](https://arxiv.org/html/2608.19611#S2.F2)and[3](https://arxiv.org/html/2608.19611#S3.F3)for DeepSeek\-R1\-Distill\-Llama\-8B\. They show accuracy and sampling cost on the reasoning model with sentence\-level resampling, with observation spacings fromN=1N\{=\}1toN=8N\{=\}8sentences\. The main trends carry over: TVD decreases asSSincreases at every spacing, the densest observation spacing performs best, and the Full Model yields its largest improvement over Raw Data at smallSS\. Section[3\.1](https://arxiv.org/html/2608.19611#S3.SS1)reports the corresponding effective\-sample multipliers\.

Figure 5:DeepSeek counterpart to Figure[2](https://arxiv.org/html/2608.19611#S2.F2)\. Pooled TVD to the fullS=200S\{=\}200reference is plotted against the number of samples per position,SS, with one line per observation spacing in sentences\. Dotted lines are results for raw data, solid lines are smoothed data\.Figure 6:DeepSeek counterpart to Figure[3](https://arxiv.org/html/2608.19611#S3.F3)\. Pooled TVD is plotted against the number of tokens generated per question; colors match Figure[5](https://arxiv.org/html/2608.19611#A3.F5)\. Dotted lines are results for raw data, solid lines are smoothed data\. The star marks the baseline setting ofS=30S\{=\}30at 1\-sentence spacing\.
## Appendix DForking Region Accuracy and Cost

Figures[7](https://arxiv.org/html/2608.19611#A4.F7)and[8](https://arxiv.org/html/2608.19611#A4.F8)reproduce the accuracy and budget analyses from the main text after restricting evaluation to forking regions, where a forking region is defined as positions within±10\\pm 10tokens of a forking point\. Forking points are defined as pointsttsuch thatT​V​D​\(ot,ot\+N\)\>ϵTVD\(o\_\{t\},o\_\{t\+N\}\)\>\\epsilon, i\.e\. points where there is a significant difference in the outcome distribution between two adjacent steps\. We report results at forking thresholdsϵ∈\{0\.10,0\.15,0\.20\}\\epsilon\\in\\\{0\.10,0\.15,0\.20\\\}\.

We observer that performance in forking regions deteriorates earlier and more rapidly than overall performance as the observation spacing becomes coarser andNNincreases\. AtS=30S\{=\}30, fork\-region TVD increases from0\.06690\.0669to0\.09360\.0936to0\.15870\.1587as spacing increases fromN=4N\{=\}4toN=8N\{=\}8andN=16N\{=\}16tokens\. It then saturates near0\.160\.16at spacings of 32–64 tokens, indicating that the fork has effectively been missed\. Flat\-region performance degrades more gradually, and the same ordering holds at scale for both models\.

Second, the Full Model’s advantage over Raw Data near forks is concentrated at moderate forks \(threshold0\.100\.10\), disappears for larger forks, and even reverses with the largest forks \(Section[3\.2](https://arxiv.org/html/2608.19611#S3.SS2)\)\. In the threshold\-0\.150\.15panels, the solid and dotted curves at dense spacings nearly coincide\. This agreement reflects the powered null result of the preregistered test rather than a plotting artifact\.

Figure 7:Forking region counterpart to Figure[2](https://arxiv.org/html/2608.19611#S2.F2)\. TVD to theS=200S\{=\}200reference is evaluated only at forking regions, using forking thresholds of0\.100\.10\(left\) and0\.150\.15\(right\)\. Solid lines show Smoothed Data, and dotted lines show Raw Data; colors match those in the main text\.Figure 8:Fork\-region counterpart to Figure[3](https://arxiv.org/html/2608.19611#S3.F3)\. TVD in forking regions is plotted against the number of tokens generated per question at forking thresholds of0\.100\.10and0\.150\.15\. Solid lines show Smoothed Data, and dotted lines show Raw Data\.
## Appendix EComponent Ablation of the Full Model

Ablating the Full Model’s components one at a time on theS=200S\{=\}200tinyMMLU dataset reveals which components contribute to accuracy in flat and forking regions\. We report paired per\-question TVD differences relative to the Full Model, computed against leave\-replicate\-out references, with 95%tt\-intervals\.

### E\.1Estimators

We compare four estimators of increasing complexity:

Raw Data:Empirical frequencies with linear interpolation between observed positions\.

Kernel Pooling:Gaussian\-kernel\-weighted Dirichlet pooling of neighboring counts\. This estimator is designed for regions between forks, where increments are small and nearby positions have similar values ofoto\_\{t\}\.

Segment\+Pool:PELT change point detection\([9](https://arxiv.org/html/2608.19611#bib.bib2);[14](https://arxiv.org/html/2608.19611#bib.bib3)\)with an exact multinomial cost, followed by per\-segment Dirichlet pooling\. The resulting piecewise\-constant fit preserves boundaries at detected forks\.

Full Model:PELT segmentation followed by Kernel Pooling truncated at change point boundaries\. This estimator accommodates gradual drift within segments while preserving sharp changes at forks\. The penalty and bandwidth hyperparameters are selected by cross\-validation on low\-sample runs\.

### E\.2Further Model Ablations

Segmentation preserves forks\.Removing segmentation and using Kernel Pooling alone retains 93–96% of the pooled improvement but degrades accuracy in forking regions\. The TVD difference relative to the Full Model is\+0\.0113\+0\.0113\[\+0\.0039,\+0\.0187\]\[\+0\.0039,\+0\.0187\]for Llama at forking threshold0\.150\.15and\+0\.0091\+0\.0091\[\+0\.0036,\+0\.0145\]\[\+0\.0036,\+0\.0145\]for DeepSeek at threshold0\.100\.10\. Figure[10](https://arxiv.org/html/2608.19611#A5.F10)shows two examples in which Kernel Pooling blurs forks that the Full Model preserves\. On the recorded DeepSeek example, the respective TVDs are0\.27200\.2720and0\.00880\.0088\.

Cross\-validated tuning matters at large forks\.Replacing cross\-validated hyperparameters with fixed values increases TVD by0\.01260\.0126\[\+0\.0051,\+0\.0202\]\[\+0\.0051,\+0\.0202\]for Llama at threshold0\.150\.15\.

Kernel pooling provides a small but consistent pooled gain\.Replacing kernel pooling with flat per\-segment pooling \(Segment\+Pool\) increases pooled TVD by0\.00210\.0021\[\+0\.0015,\+0\.0026\]\[\+0\.0015,\+0\.0026\]for Llama atS=30S\{=\}30\.

The detection cost has little end\-to\-end effect\.Fixing PELT to the L2 cost increases pooled TVD by≤0\.0010\\leq 0\.0010throughout the ablation\. Its performance is statistically indistinguishable from that of the Full Model in both headline fork conditions; for example, the difference for Llama atS=30S\{=\}30and threshold0\.100\.10is\+0\.0003\+0\.0003\[−0\.0004,\+0\.0009\]\[\-0\.0004,\+0\.0009\]\. This variant retains 97–100% of the Full Model’s improvement over Raw Data\.

Figure[9](https://arxiv.org/html/2608.19611#A5.F9)summarizes these paired differences\. For pooled accuracy, every smoothing variant is within a few thousandths of the Full Model, whereas Raw Data has substantially higher TVD\. In forking regions, the no\-segmentation variant differs reliably from zero, while the L2\-detection variant does not\.

Figure 9:Paired per\-question TVD differences between each ablation variant and the Full Model for DeepSeek\-R1\-Distill\-Llama\-8B at 1\-sentence spacing\. Error bars show 95%tt\-intervals over tinyMMLU, and the dashed zero line denotes the Full Model\. Results are shown for pooled accuracy \(left\) and forking region accuracy at threshold0\.100\.10\(right\)\. Removing segmentation harms accuracy near forks but not pooled accuracy; fixing the detection cost to L2 is indistinguishable from the Full Model in both panels\.Figure 10:Ablation overlays for two example questions\. Reconstructions useS=30S\{=\}30on the densest spacingNNand are evaluated against leave\-replicate\-outS=200S\{=\}200references; dashed lines mark reference forks\. The left column shows Llama\-3\-8B\-Instruct question 46 using the ablation variants, cross\-validation, and operating point from the recordedS=200S\{=\}200store\. The right column shows the recorded DeepSeek example, where the fork occurs when continuations stop resolving to an answer\. Removing segmentation \(Kernel Pooling\) blurs the forks, whereas removing kernel pooling \(Segment\+Pool\) reduces gradual drift to discrete steps\. The L2 detection\-cost variant is visually indistinguishable from the Full Model\.

## Appendix FThe1/S1/\\sqrt\{S\}Noise Law

Figure[11](https://arxiv.org/html/2608.19611#A6.F11)presents the replicate\-noise measurements underlying Section[3\.3](https://arxiv.org/html/2608.19611#S3.SS3)\. For theS=1000S\{=\}1000development set, pooled TVD between disjoint replicates is plotted againstSSon log–log axes\. The fitted slope is−0\.4903\-0\.4903overall and−0\.5047\-0\.5047in the tail\. The figure also shows the exact i\.i\.d\. multinomial null, computed by resampling the per\-branch histograms through the same statistic; the ratios of measured to null TVD range from0\.980\.98to1\.011\.01\. Finally, the per\-model means from theS=200S\{=\}200tinyMMLU dataset have slopes of−0\.4767\-0\.4767and−0\.4577\-0\.4577\. Their absolute TVDs are lower because those questions are less variable on average, but the error decreases at a similar rate\.

Figure 11:Replicate TVD versus the number of samples per position,SS, on log–log axes\. The measured curve remains within2%2\\%of the exact i\.i\.d\. multinomial null throughS=200S\{=\}200\. Triangles show the full dataset means for both models\.
## Appendix GSegmentation Cost in PELT Implementation

During this work, we uncovered a bug in the implementation of PELT in theruptureslibrary\([14](https://arxiv.org/html/2608.19611#bib.bib3)\)\.

The segment\-based estimators use PELT change point detection with an exact multinomial cost\. If a custom cost object does not inherit fromBaseCost, the library silently replaces it with the default least\-squares cost,CostL2\. The fit still completes and returns change points without warning, but results are significantly affected \(Figure[12](https://arxiv.org/html/2608.19611#A7.F12)\)\.

Figure 12:Number of detected segments as a function of the PELT penalty forS=30S\{=\}30dense\-grid counts on the two development questions\. Curves show the exact multinomial cost, a trend\-aware cost, and the silently substituted L2 cost\. The dashed line marks the ideal segment count\.
## Appendix HLLM Use Statement

The research reported in this paper and the initial manuscript draft were produced by the autonomous LLM agent Silico \([https://www\.goodfire\.com/silico](https://www.goodfire.com/silico)\) under human direction\.