一个 Token 值多少算力？基于 Mixture-of-Agents 测量大模型的每 Token 充分算力

# What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute
Source: [https://arxiv.org/html/2610.02491](https://arxiv.org/html/2610.02491)
Weijia Han2Hai Helen Li1Yiran Chen1 1Duke University 2University of Washington

###### Abstract

Large language models spend the same amount of computation on every token they generate, regardless of how difficult each token is to produce\. Methods such as speculative decoding and model routing are built on the premise that much of this computation is unnecessary, yet the computation an individual token actually requires has not been measured\. We measure it through a Mixture\-of\-Agents \(MoA\) lens: a panel of fifteen language models of increasing capacity, drawn from three families, in which every agent attempts to reproduce a reference sequence token by token, conditioned on the correct preceding tokens\. We define the inference cost of the smallest agent that succeeds as the token’s sufficient compute, which upper\-bounds what the token requires\. On three core benchmarks, a 0\.5B agent reproduces 92–95% of reference tokens\. Across Qwen, OLMo, and R1\-distilled panels, the most expensive 10% account for 64–80% of estimated FLOPs\. On all 500 MATH\-500 problems, the MoA\-derived map helps model routing reduce projected latency from 7\.59 to 5\.12 seconds while slightly improving accuracy, relative to the best confidence\-routing baseline\. The MoA\-map helps drafting use 32\.6% fewer draft tokens and approximately 20% lower projected latency than fixed\-window drafting at similar accuracy\. These comparisons reveal remaining allocation headroom, motivating controllers that exploit sufficient\-compute structure\.

## 1Introduction

Large language models \(LLMs\), such as the GPT series, have made a great impact for tasks ranging from writing to programming\[[7](https://arxiv.org/html/2610.02491#bib.bib53),[1](https://arxiv.org/html/2610.02491#bib.bib54)\]\. Their adoption produces recurring computational expense at a large scale: by July 2025, ChatGPT alone received more than 2\.5 billion user messages per day\[[11](https://arxiv.org/html/2610.02491#bib.bib56)\]\. Therefore, efficiently serving these requests is becoming critical\[[45](https://arxiv.org/html/2610.02491#bib.bib55),[30](https://arxiv.org/html/2610.02491#bib.bib27)\]\. One approach to reducing this computational cost is to use only as much computation as is necessary\.Speculative decodinglets a small model propose several tokens for a larger model to verify together\[[32](https://arxiv.org/html/2610.02491#bib.bib1),[12](https://arxiv.org/html/2610.02491#bib.bib2)\]\.Model cascades and routerschoose among models according to the input\[[13](https://arxiv.org/html/2610.02491#bib.bib8),[44](https://arxiv.org/html/2610.02491#bib.bib7)\], while collaborative decoding makes that choice within a response\[[52](https://arxiv.org/html/2610.02491#bib.bib33),[20](https://arxiv.org/html/2610.02491#bib.bib35)\]\.Adaptive\-depthmethods similarly vary the layers evaluated for each prediction\[[50](https://arxiv.org/html/2610.02491#bib.bib4),[49](https://arxiv.org/html/2610.02491#bib.bib5)\]\. These approaches exploit the possibility that a cheaper computation can supply a useful prediction\. Their gains depend on how often this is possible\. However, running the model alone does not reveal when less computation would have been sufficient\.

Figure 1:Measuring which model suffices at each token\. \(a\) Each column is a token from a GSM8K response\. Every model receives the same reference prefix; green cells reproduce the next token and outlines mark the smallest matching model\. \(b\) Within the Qwen2\.5 family, the 0\.5B model reproduces 91\.9–95\.2% of tokens\. The “unresolved” means that none matches\.This motivates our central question:*How much computation is sufficient for each token in a successful response?*A response\-level accuracy or cost average hides where that computation is needed\. Even a high agreement rate between a small and a large model leaves open how much computation the remaining disagreements require\. Existing measurements of oracle module allocation, small/large\-model disagreement, and learned layer use establish that computation can vary within a response\[[2](https://arxiv.org/html/2610.02491#bib.bib9),[61](https://arxiv.org/html/2610.02491#bib.bib10),[37](https://arxiv.org/html/2610.02491#bib.bib6)\]\. We measure this variation by identifying, for each token, the smallest model that can successfully reproduce it\. We then examine whether this token\-level information can improve existing efficient serving methods\.

Mixture\-of\-Experts \(MoE\) has been widely known, which selects subnetworks inside a model\[[51](https://arxiv.org/html/2610.02491#bib.bib45),[19](https://arxiv.org/html/2610.02491#bib.bib44)\]\. Mixture\-of\-Agents \(MoA\)combines different language models, either through response aggregation\[[56](https://arxiv.org/html/2610.02491#bib.bib36)\]or through agent selection during decoding\[[8](https://arxiv.org/html/2610.02491#bib.bib37),[62](https://arxiv.org/html/2610.02491#bib.bib34)\]\. Our measurement uses token\-level MoA\. Given a reference sequence, we query a panel of fifteen models from Qwen2\.5, OLMo\-2, and DeepSeek\-R1\-Distill\-Qwen at each token\. The FLOPs of the cheapest agent whose prediction matches the next token are that token’s*sufficient compute*\. A successful agent supplies an observed upper bound on the computation cost of that token\.

Figure[1](https://arxiv.org/html/2610.02491#S1.F1)provides an overview of our measurement\. Within the Qwen2\.5 panel, the 0\.5B agent reproduces 91\.9–95\.2% of reference tokens on GSM8K, MATH\-500, and HumanEval\. The remaining tokens are few but expensive\. In all three panels, the most expensive 10% of tokens account for 64–80% of estimated FLOPs\. Families also agree on which positions are expensive\. After controlling for surprisal and position, the reproduction orderings of Qwen2\.5, OLMo\-2, and Falcon3 families have pairwise partial rank correlations of 0\.32–0\.48\. This agreement makes the map informative beyond the behavior of a single model family\. Details will be discussed in Section[1](https://arxiv.org/html/2610.02491#S4.T1)\.

We then ask whether this map would help existing methods allocate computation\. On all 500 MATH\-500 problems, we give a routing controller and a speculative\-decoding controller oracle access to the map of 0\.5B agreement along a 72B reference response\. Routing to 0\.5B at positions the map marks as reproducible reaches 69\.6% accuracy at a projected latency of 5\.12 seconds, compared with 68\.4% and 7\.59 seconds for the best confidence\-threshold cascade\. Setting each draft length to the map’s run of reproducible positions proposes 32\.6% fewer draft tokens than a fixed 16\-token window and lowers projected latency by about 20% at similar accuracy\. Our contributions can be summarized as:

1. 1\.We formalize sufficient compute through token\-level MoA and introduce a panel\-based method for measuring it along verified responses\.
2. 2\.We show that a sparse tail of tokens carries most of the estimated compute and that three families share a reproduction ordering\.
3. 3\.We evaluate how routing and speculative decoding exploit the measurement\. Both improve projected efficiency at comparable accuracy over baselines\. The difference is allocation headroom that current controllers leave unrealized\.

In this paper, we first define sufficient compute \(Section[2](https://arxiv.org/html/2610.02491#S2)\) and derive a token\-level MoA procedure for measuring it \(Section[3](https://arxiv.org/html/2610.02491#S3)\)\. The experiments then examine where compute is concentrated, whether cheap reproduction supports continued generation, and how existing efficiency methods exploit this information \(Section[4](https://arxiv.org/html/2610.02491#S4)\)\. Section[5](https://arxiv.org/html/2610.02491#S5)places our study in the context of prior work\. Section[6](https://arxiv.org/html/2610.02491#S6)discusses the implications and limitations of our findings, and Section[7](https://arxiv.org/html/2610.02491#S7)concludes the paper\.

## 2Problem formulation and Background

Sufficient compute at a reference prefixLetxxbe a prompt andy⋆=\(y1⋆,…,yT⋆\)y^\{\\star\}=\(y\_\{1\}^\{\\star\},\\ldots,y\_\{T\}^\{\\star\}\)a reference accepted by a binary task checkerV⁡\(x,y\)∈\{0,1\}V\(x,y\)\\in\\\{0,1\\\}, soV⁡\(x,y⋆\)=1V\(x,y^\{\\star\}\)=1\. At positiontt, the reference state isst⋆=\(x,y<t⋆\)s\_\{t\}^\{\\star\}=\(x,y\_\{<t\}^\{\\star\}\): the prompt and all preceding reference tokens\. A model panelΠF=\{πF,1,…,πF,KF\}\\Pi\_\{F\}=\\\{\\pi\_\{F,1\},\\ldots,\\pi\_\{F,K\_\{F\}\}\\\}contains next\-token distributionsπF,k​\(v∣s\)\\pi\_\{F,k\}\(v\\mid s\)over a shared vocabulary𝒱\\mathcal\{V\}\. The indexFFidentifies the panel\. LetcF,k​\(s\)\>0c\_\{F,k\}\(s\)\>0be the FLOPs of modelkk’s next\-token prediction at prefixss\. A prediction\-success ruleM⁡\(π,s,v\)∈\{0,1\}M\(\\pi,s,v\)\\in\\\{0,1\\\}specifies when a model succeeds on target tokenvvat statess\. We require the model’s prediction to equal the reference token\.

###### Definition 2\.1\(Panel\-relative sufficient compute\)\.

For reference tokenyt⋆y\_\{t\}^\{\\star\}, define the successful agents and their minimum prediction cost by

𝒜FM​\(t\)=\{k:M⁡\(πF,k,st⋆,yt⋆\)=1\},dFM​\(t\)=mink∈𝒜FM​\(t\)⁡cF,k​\(st⋆\)\.\\mathcal\{A\}\_\{F\}^\{M\}\(t\)=\\\{k:M\(\\pi\_\{F,k\},s\_\{t\}^\{\\star\},y\_\{t\}^\{\\star\}\)=1\\\},\\qquad d\_\{F\}^\{M\}\(t\)=\\min\_\{k\\in\\mathcal\{A\}\_\{F\}^\{M\}\(t\)\}c\_\{F,k\}\(s\_\{t\}^\{\\star\}\)\.\(1\)We setmin⁡∅=\+∞\\min\\varnothing=\+\\inftyand call such a position*unresolved*\. Appendix[C\.1](https://arxiv.org/html/2610.02491#A3.SS1)reports how often this occurs and how such positions enter the cost summaries\. The sequencedFM​\(1:T\)d\_\{F\}^\{M\}\(1\{:\}T\)is the sequence’s sufficient\-compute map\. We writedF​\(t\)d\_\{F\}\(t\)when the success rule is fixed\.

It serves as an upper bound since adding agents can only tighten the bound\. The measured map depends on the reference trajectory, and another correct response may have a different cost profile\.

Speculative decoding and RoutingThe map makes precise a local opportunity shared by several efficiency methods: a prediction produced by a large model may also be available from a cheaper one\. Speculative decoding uses a draft model to propose multiple tokens and verifies them with a target model in parallel\[[32](https://arxiv.org/html/2610.02491#bib.bib1),[12](https://arxiv.org/html/2610.02491#bib.bib2)\]\. Its distribution\-preserving acceptance rule depends on draft/target probabilities, and its speed benefit depends on accepted tokens per verification round and the relative cost of drafting and verification\. Greedy reference agreement gives a concrete local comparison; speculative sampling uses the full distributions to decide acceptance\.

Routing changes which model supplies the output\. FrugalGPT and RouteLLM select models at the request level\[[13](https://arxiv.org/html/2610.02491#bib.bib8),[44](https://arxiv.org/html/2610.02491#bib.bib7)\]; collaborative decoding extends selection to individual steps\[[52](https://arxiv.org/html/2610.02491#bib.bib33),[62](https://arxiv.org/html/2610.02491#bib.bib34),[20](https://arxiv.org/html/2610.02491#bib.bib35)\]\. At that granularity, a cheaper prediction also changes the prefix on which later choices are made\. A cost map measured at reference states consequently supplies information about those states, while a deployed controller must act along the states it generates\. This distinction motivates both the exact\-reference selector and the later continuation experiments\.

From MoE to token\-level MoAMixtures provide a way to express the choice of computation\. An MoE layer maps a token representationhhtoh′h^\{\\prime\}by combiningJJexpert functionsEjE\_\{j\}through a gatewjw\_\{j\}:

h′=∑j=1Jwj​\(h\)​Ej​\(h\),wj​\(h\)≥0,∑j=1Jwj​\(h\)=1\.h^\{\\prime\}=\\sum\_\{j=1\}^\{J\}w\_\{j\}\(h\)E\_\{j\}\(h\),\\qquad w\_\{j\}\(h\)\\geq 0,\\quad\\sum\_\{j=1\}^\{J\}w\_\{j\}\(h\)=1\.\(2\)The experts are subnetworks within a model\[[27](https://arxiv.org/html/2610.02491#bib.bib52)\]; sparse gates activate only a subset at each token\[[51](https://arxiv.org/html/2610.02491#bib.bib45),[19](https://arxiv.org/html/2610.02491#bib.bib44)\]\. This separates parameter capacity from the computation performed for a particular prediction\.

MoA makes language models the available policies\. In response\-level MoA, agents produce candidate answers and an aggregator conditions on them, possibly across several rounds\[[56](https://arxiv.org/html/2610.02491#bib.bib36)\]\. Token\-level MoA selects an agent’s next\-token policy at a shared decoding state\[[52](https://arxiv.org/html/2610.02491#bib.bib33),[8](https://arxiv.org/html/2610.02491#bib.bib37),[62](https://arxiv.org/html/2610.02491#bib.bib34)\]\. For our measurement, this gives a direct interpretation of𝒜FM​\(t\)\\mathcal\{A\}\_\{F\}^\{M\}\(t\): it is the set of agents that can supply the required prediction at the current reference state\. Section[3](https://arxiv.org/html/2610.02491#S3)derives the controller that selects among those agents\.

## 3Measuring sufficient compute with token\-level MoA

Figure 2:Minimum\-cost policy selection\. At the illustrated prefix, the 3B, 7B, and 32B agents all reproduce the reference token “for” and reach the same next state\. Their remaining costJt\+1J\_\{t\+1\}is therefore identical, so the controller selects the agent with the lowest immediate prediction FLOPs: 3B in this example\. Repeating this selection gives the model sequence shown along the bottom\.To connect sufficient compute to sequence generation, we treat each model as a policy and model selection as a decision made at every token\. We first formulate this MoA process, then derive the minimum\-cost controller for reproducing a verified response\.

Token\-level MoA as a Markov Decision Process\.Decoding can be represented as a token\-level Markov decision process\[[48](https://arxiv.org/html/2610.02491#bib.bib46),[5](https://arxiv.org/html/2610.02491#bib.bib47),[41](https://arxiv.org/html/2610.02491#bib.bib48),[9](https://arxiv.org/html/2610.02491#bib.bib49)\]\. At steptt, the statest=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\)contains the prompt and the tokens generated so far\. The action is the next tokenvv\. Appending this token deterministically produces the next state,st\+1=\(x,y<t​v\)s\_\{t\+1\}=\(x,y\_\{<t\}v\)\. The state contains all information used by the next\-token policies, satisfying the Markov property\. Generation terminates upon emitting an end\-of\-sequence token or reaching a prescribed maximum length\. The resulting responseyyconstitutes a*rollout*, evaluated by the terminal rewardV⁡\(x,y\)V\(x,y\)\.

In token\-level MoA, a controller chooses which model supplies the next\-token policy\[[8](https://arxiv.org/html/2610.02491#bib.bib37)\]\. Each agentπF,k∈ΠF\\pi\_\{F,k\}\\in\\Pi\_\{F\}defines a distribution over the shared vocabulary\. Letσ⁡\(k∣s\)\\sigma\(k\\mid s\)denote the probability of selecting agentkkat statess, whereσ⁡\(k∣s\)≥0\\sigma\(k\\mid s\)\\geq 0and∑k=1KFσ⁡\(k∣s\)=1\\sum\_\{k=1\}^\{K\_\{F\}\}\\sigma\(k\\mid s\)=1\. The controller first selects an agent and then samples a token from that agent’s policy\. Marginalizing over the selected agent gives

πσ​\(v∣s\)=∑k=1KFσ⁡\(k∣s\)​πF,k​\(v∣s\)\.\\pi\_\{\\sigma\}\(v\\mid s\)=\\sum\_\{k=1\}^\{K\_\{F\}\}\\sigma\(k\\mid s\)\\pi\_\{F,k\}\(v\\mid s\)\.\(3\)Thus, the controller selects among model policies, while the selected policy determines the next token\. Different agents can produce different tokens and therefore lead to different subsequent states\.

We measure prediction cost using the FLOP functionscF,k​\(s\)c\_\{F,k\}\(s\)defined in Section[2](https://arxiv.org/html/2610.02491#S2)\. We consider an idealized setting in which each agent’s prefix cache is available and switching between agents incurs no additional cost\. LetKtK\_\{t\}denote the agent selected at steptt, and letT⁡\(Y\)T\(Y\)be the length of the generated responseYY\. For an expected compute budgetBB, the controller’s objective is

maxσ⁡𝔼σ​\[V⁡\(x,Y\)\]subject to𝔼σ​\[∑t=1T⁡\(Y\)cF,Kt​\(st\)\]≤B\.\\max\_\{\\sigma\}\\;\\mathbb\{E\}\_\{\\sigma\}\[V\(x,Y\)\]\\quad\\text\{subject to\}\\quad\\mathbb\{E\}\_\{\\sigma\}\\left\[\\sum\_\{t=1\}^\{T\(Y\)\}c\_\{F,K\_\{t\}\}\(s\_\{t\}\)\\right\]\\leq B\.\(4\)The expectation accounts for both agent selection and token generation\. An agent choice affects its immediate prediction cost and the continuation from the resulting prefix\.

Algorithm 1Measuring sufficient compute by token\-level policy selectionPrompt

xx; verified reference

y⋆1:Ty^\{\\star\}\_\{1:T\}; panel

ΠF\\Pi\_\{F\}with FLOP functions

cF,k​\(s\)c\_\{F,k\}\(s\)
Matches

AFA\_\{F\}; selected agents

k⋆1:Tk^\{\\star\}\_\{1:T\}; costs

dF​\(1:T\)d\_\{F\}\(1\{:\}T\); unresolved positions

UU

Evaluate reference states

1:

U←∅U\\leftarrow\\varnothing
2:for

t=1,…,Tt=1,\\ldots,Tdo

3:

st⋆←\(x,y<t⋆\)s\_\{t\}^\{\\star\}\\leftarrow\(x,y\_\{<t\}^\{\\star\}\)
4:for

k=1,…,KFk=1,\\ldots,K\_\{F\}do

5:

aF,k,t←𝟏\[gF,k\(st⋆\)=yt⋆\]a\_\{F,k,t\}\\leftarrow\\mathbf\{1\}\[g\_\{F,k\}\(s\_\{t\}^\{\\star\}\)=y\_\{t\}^\{\\star\}\]
6:

cF,k,t←cF,k​\(st⋆\)c\_\{F,k,t\}\\leftarrow c\_\{F,k\}\(s\_\{t\}^\{\\star\}\)
7:endfor

8:

𝒜F​\(t\)←\{k:aF,k,t=1\}\\mathcal\{A\}\_\{F\}\(t\)\\leftarrow\\\{k:a\_\{F,k,t\}=1\\\}

Select least\-cost agents

9:if

𝒜F​\(t\)≠∅\\mathcal\{A\}\_\{F\}\(t\)\\neq\\varnothingthen

10:

kt⋆←arg⁡mink∈𝒜F​\(t\)⁡cF,k,tk\_\{t\}^\{\\star\}\\leftarrow\\arg\\min\_\{k\\in\\mathcal\{A\}\_\{F\}\(t\)\}c\_\{F,k,t\}
11:

dF​\(t\)←cF,kt⋆,td\_\{F\}\(t\)\\leftarrow c\_\{F,k\_\{t\}^\{\\star\},t\}
12:else

13:

kt⋆←⊥k\_\{t\}^\{\\star\}\\leftarrow\\bot;

dF​\(t\)←\+∞d\_\{F\}\(t\)\\leftarrow\+\\infty
14:

U←U∪\{t\}U\\leftarrow U\\cup\\\{t\\\}
15:endif

16:endfor

17:return

AFA\_\{F\},

k⋆1:Tk^\{\\star\}\_\{1:T\},

dF​\(1:T\)d\_\{F\}\(1\{:\}T\),

UU

Exact\-reference selection and measurement\.To measure sufficient compute, we fix a verified responsey⋆y^\{\\star\}and ask which agents can reproduce it at minimum cost\. Define

gF,k\(s\)=argmaxv∈𝒱πF,k\(v∣s\),aF,k,t=𝟏\[gF,k\(st⋆\)=yt⋆\]\.g\_\{F,k\}\(s\)=\\arg\\max\_\{v\\in\\mathcal\{V\}\}\\pi\_\{F,k\}\(v\\mid s\),\\qquad a\_\{F,k,t\}=\\mathbf\{1\}\\left\[g\_\{F,k\}\(s\_\{t\}^\{\\star\}\)=y\_\{t\}^\{\\star\}\\right\]\.\(5\)The matrixAF=\(aF,k,t\)A\_\{F\}=\(a\_\{F,k,t\}\)records which agents reproduce which reference tokens\. At positiontt, the admissible agents are𝒜F​\(t\)=\{k:aF,k,t=1\}\\mathcal\{A\}\_\{F\}\(t\)=\\\{k:a\_\{F,k,t\}=1\\\}, and their prediction costs arecF,k,t=cF,k​\(st⋆\)c\_\{F,k,t\}=c\_\{F,k\}\(s\_\{t\}^\{\\star\}\)\. If every reference position is resolved, then𝒜F​\(t\)≠∅\\mathcal\{A\}\_\{F\}\(t\)\\neq\\varnothingfor alltt\. A controller reproduces the reference by choosing an admissible agent at each position\. The minimum\-cost selection problem is therefore

min⁡∑t=1T\(k1,…,kT\)∈∏t=1T𝒜F​\(t\)⁡cF,kt,t\.\\min\_\{\(k\_\{1\},\\ldots,k\_\{T\}\)\\in\\prod\_\{t=1\}^\{T\}\\mathcal\{A\}\_\{F\}\(t\)\}\\sum\_\{t=1\}^\{T\}c\_\{F,k\_\{t\},t\}\.\(6\)
The key simplification is that every admissible agent emits the same tokenyt⋆y\_\{t\}^\{\\star\}and reaches the same next statest\+1⋆s\_\{t\+1\}^\{\\star\}\. Consequently, choosing among these agents changes the immediate cost without changing the remaining reference or the available future choices\. LetJtJ\_\{t\}be the minimum cost of reproducing the remaining reference fromst⋆s\_\{t\}^\{\\star\}\. Under the additive\-cost assumptions above, its Bellman recursion becomes

Jt=mink∈𝒜F​\(t\)⁡\(cF,k,t\+Jt\+1\)=dF​\(t\)\+Jt\+1,JT\+1=0\.J\_\{t\}=\\min\_\{k\\in\\mathcal\{A\}\_\{F\}\(t\)\}\\left\(c\_\{F,k,t\}\+J\_\{t\+1\}\\right\)=d\_\{F\}\(t\)\+J\_\{t\+1\},\\qquad J\_\{T\+1\}=0\.\(7\)The continuation costJt\+1J\_\{t\+1\}is shared by all admissible agents\. Minimizing the total reference cost therefore reduces to selecting the least costly matching agent at each position:

kt⋆=argmink∈𝒜F​\(t\)cF,k,t,σy⋆⋆\(k∣st⋆\)=𝟏\[k=kt⋆\]\.k\_\{t\}^\{\\star\}=\\arg\\min\_\{k\\in\\mathcal\{A\}\_\{F\}\(t\)\}c\_\{F,k,t\},\\qquad\\sigma\_\{y^\{\\star\}\}^\{\\star\}\(k\\mid s\_\{t\}^\{\\star\}\)=\\mathbf\{1\}\[k=k\_\{t\}^\{\\star\}\]\.\(8\)This selector reproduces the reference from the initial prompt, with minimum cost:

CF,ref⋆​\(x,y⋆\)=J1=∑t=1TdF​\(t\)\.C\_\{F,\\mathrm\{ref\}\}^\{\\star\}\(x,y^\{\\star\}\)=J\_\{1\}=\\sum\_\{t=1\}^\{T\}d\_\{F\}\(t\)\.\(9\)Figure[2](https://arxiv.org/html/2610.02491#S3.F2)illustrates this tokenwise measurement\. BecauseV⁡\(x,y⋆\)=1V\(x,y^\{\\star\}\)=1, the selected agent sequence supplies a correct response at costCF,ref⋆​\(x,y⋆\)C\_\{F,\\mathrm\{ref\}\}^\{\\star\}\(x,y^\{\\star\}\)\. It upper\-bounds the minimum cost of generating any correct responseCF,correct⋆​\(x\)≤CF,ref⋆​\(x,y⋆\)\.C\_\{F,\\mathrm\{correct\}\}^\{\\star\}\(x\)\\leq C\_\{F,\\mathrm\{ref\}\}^\{\\star\}\(x,y^\{\\star\}\)\.Appendix[B\.2](https://arxiv.org/html/2610.02491#A2.SS2)specifies the candidate class, states the assumptions under which Eq\. \([9](https://arxiv.org/html/2610.02491#S3.E9)\) is the minimum over all selectors that reproducey⋆y^\{\\star\}, and gives the local witness bound for larger model classes\.

Algorithm[1](https://arxiv.org/html/2610.02491#alg1)computes the matching matrix and the resulting selector\. If no agent matches a token, the algorithm records that position as unresolved and assignsdF​\(t\)=\+∞d\_\{F\}\(t\)=\+\\infty\. We implement the reference\-state evaluations using teacher forcing: a causal forward pass supplies each model with only the preceding reference tokens at each prediction position\. We retain both the selected agent identitykt⋆k\_\{t\}^\{\\star\}and its prediction FLOPsdF​\(t\)d\_\{F\}\(t\)\. The reported sufficient compute measures the selected agent’s prediction cost\. The resulting measurements characterize compute along verified trajectories\.

## 4Experiments

Models\.Our main measurement panel comprises fifteen models across three families: Qwen2\.5 \(0\.5B, 1\.5B, 3B, 7B, 14B, 32B, 72B\)\[[46](https://arxiv.org/html/2610.02491#bib.bib19)\], OLMo\-2 \(1B, 7B, 13B, 32B\)\[[43](https://arxiv.org/html/2610.02491#bib.bib20)\], and DeepSeek\-R1\-Distill\-Qwen \(1\.5B, 7B, 14B, 32B\)\[[22](https://arxiv.org/html/2610.02491#bib.bib58)\]as reasoning\-distilled candidates\. Falcon3\[[55](https://arxiv.org/html/2610.02491#bib.bib57)\]\(1B, 3B, 7B, 10B\) is tested as extended family candidates\. The reference generator samples four responses per prompt at temperature 0\.7 and top\-pp0\.95\. Qwen2\.5\-72B\-Instruct generates the references used throughout; Llama\-3\.3\-70B\-Instruct generates a second reference set for the cross\-family analysis\. Table[A2](https://arxiv.org/html/2610.02491#A1.T2)lists every model role, including a Qwen3\-30B\-A3B MoE generator as a sensitivity check \(Appendix[C\.2](https://arxiv.org/html/2610.02491#A3.SS2)\)\. Llama\-3\.1\-8B\-Instruct\[[21](https://arxiv.org/html/2610.02491#bib.bib21)\]supplies token surprisal for statistical controls\. Appendix[A](https://arxiv.org/html/2610.02491#A1)provides additional setup details\.

Datasets\.The main experiments use three core datasets covering mathematical reasoning and code generation: GSM8K \([16](https://arxiv.org/html/2610.02491#bib.bib22)\), MATH\-500 \([33](https://arxiv.org/html/2610.02491#bib.bib23)\), and HumanEval \([14](https://arxiv.org/html/2610.02491#bib.bib24)\)\. Nine additional benchmarks extend the analysis to competition mathematics, programming, question answering, knowledge, and commonsense reasoning, including LiveCodeBench\[[28](https://arxiv.org/html/2610.02491#bib.bib25)\], BBH\[[54](https://arxiv.org/html/2610.02491#bib.bib41)\], and MMLU\-Pro\[[57](https://arxiv.org/html/2610.02491#bib.bib43)\]\. Together, the twelve benchmarks contain 50,019 evaluation prompts\. Table[A1](https://arxiv.org/html/2610.02491#A1.T1)lists all datasets\.

Hardware and software\.Generation and teacher\-forced scoring run on a server equipped with eight 80 GB NVIDIA A100\-SXM4 GPUs\. We use vLLM 0\.14\.1\[[30](https://arxiv.org/html/2610.02491#bib.bib27)\]with BF16 precision\.

Measurement\.Our primary criterion is exact reproduction of the reference token\. Appendix[B](https://arxiv.org/html/2610.02491#A2)defines the teacher\-forced scoring and the top\-five andp≥0\.5p\\geq 0\.5alternatives, whose evaluation results are in Table[A4](https://arxiv.org/html/2610.02491#A3.T4)\. We record the selected model and its architecture\-derived FLOPs\. The count includes attention, projection, and feed\-forward operations\. To compare families with different tokenizers, we align their predictions to common words and assign each word the maximum selected model index among its tokens; unresolved words receive an ordinal category above the panel\. We estimate uncertainty with 2,000 bootstrap resamples of whole items, retaining the dependence among tokens in each response\. Where available, we report±\\pmone bootstrap SD\.

### 4\.1Compute concentrates in a sparse tail

Figure 3:Token reproduction and estimated FLOP concentration across Qwen2\.5, OLMo\-2, and DeepSeek\-R1\-Distill\-Qwen\.Top: distributions of the least\-cost matching model for GSM8K, MATH\-500, and HumanEval\. Labels on the bar give the smallest\-model share\. Hatched gray segments denote unresolved tokens \(Unres\.\), which might require larger models to reproduce\.Bottom: cumulative estimated FLOPs over all reference tokens, ordered from most to least expensive, with a log horizontal axis\. Resolved tokens retain their measured architecture\-derived FLOP costs; each unresolved token is assigned twice the prediction FLOPs of the largest model in its panel\. Vertical dashed lines and markers identify the top 1% and 10% of all tokens\.Most tokens match the smallest model and cost concentrates in the tail\.Across all three measurement panels, most reference tokens are assigned to the smallest available model \(Shown in Figure[3](https://arxiv.org/html/2610.02491#S4.F3)\)\. On the same Qwen\-generated references, the 0\.5B Qwen model reproduces 91\.86–95\.24% of tokens across GSM8K, MATH\-500, and HumanEval\. The 1B OLMo and 1\.5B R1\-Distill\-Qwen models reproduce 89\.34–91\.74% and 89\.80–92\.27%, respectively\. The predominance of inexpensive predictions persists across all three model families\.

The most expensive 10% of tokens account for 63\.66–79\.54% of estimated FLOPs in Qwen, 63\.75–69\.37% in OLMo, and 67\.19–74\.76% in R1\-Distill\-Qwen\. The top 1% account for 52\.94–60\.14%, 16\.25–19\.87%, and 11\.47–15\.02%, respectively\. The estimated compute is therefore concentrated in a small fraction of positions across all three panels, with Qwen exhibiting a sharper concentration within the most expensive percentile\. We next examine whether the panels agree on the ordering of expensive positions, with each panel supplying its own cost scale\.

Different families share a reproduction ordering\.

Table 1:Model families share a reproduction ordering\. Partial Spearman correlations of aligned word\-level reproduction indices, pooled across GSM8K, MATH\-500, and HumanEval, controlling for generator surprisal, probe surprisal, and position\. Columns distinguish reference generators\. Values are estimates±\\pmone item\-bootstrap SD\. A 0 indicates no rank association\.Different model families exhibit a shared ordering of reproduction difficulty\. After controlling for generator surprisal, probe surprisal, and position, the partial rank correlation between each pair of families, pooled over the three core datasets, ranges from 0\.322 to 0\.398 on Qwen\-generated references and from 0\.417 to 0\.475 on Llama\-generated references \(Table[1](https://arxiv.org/html/2610.02491#S4.T1)\)\. Positions that require larger models in one family tend to require them in another, and the pattern extends to text generated by another family\. The association is positive on every benchmark and moves by less than 0\.05 under additional controls, alternative success criteria, a second verified response per problem, memorization screening, and GSM\-Symbolic \(Appendix[C](https://arxiv.org/html/2610.02491#A3), Tables[A3](https://arxiv.org/html/2610.02491#A3.T3)and[A4](https://arxiv.org/html/2610.02491#A3.T4)\)\. It is also stronger than greedy or soft speculative\-rejection signals on the same text \(0\.32/0\.42 versus 0\.15–0\.38; Appendix[C\.2](https://arxiv.org/html/2610.02491#A3.SS2)\)\.

### 4\.2Cheap reproduction does not ensure reliable continuation

The cost map measures token reproduction conditioned on the reference prefix\. To assess whether low reproduction costs translate into reliable continuation, we allow smaller models to continue generating the remaining response and report accuracy drops relative to Qwen\-72B\.

Effect of continuation duration\.We vary the duration of small\-model generation before Qwen\-72B resumes\. Both models condition on the generated continuation without receiving additional reference tokens\. The baseline is an uninterrupted Qwen\-72B continuation from the same prefix\. Longer handovers produce larger accuracy losses, with the magnitude depending on model capacity \(Figure[4](https://arxiv.org/html/2610.02491#S4.F4), left\)\. For the 0\.5B model, the accuracy difference relative to Qwen\-72B is−0\.09±0\.19\-0\.09\\pm 0\.19percentage points at one token and decreases to−9\.44±0\.50\-9\.44\\pm 0\.50at 64 tokens\. Larger models mitigate this degradation: at 64 tokens, the losses are4\.68±0\.394\.68\\pm 0\.39points for 1\.5B and1\.50±0\.301\.50\\pm 0\.30points for 7B\. These results motivate a joint choice of model capacity and delegation length: realizing the savings suggested by the sufficient\-compute map requires accounting for how delegated predictions affect the rest of the trajectory\. Appendix[D](https://arxiv.org/html/2610.02491#A4)specifies the detail\.

Figure 4:Accuracy drops with the smaller model continuation\. Differences from a 72B continuation of the same verified prefix\.\(a\): a small model generates up toLLtokens, then 72B resumes\. Shading shows±\\pmone item\-bootstrap SD\.\(b\): full\-suffix generation from reference character fractionτ\\tau\.Effect of model capacity and reference\-prefix length\.We evaluate full\-suffix continuation from reference fractionsτ∈\{0,0\.25,0\.5,0\.75,0\.9\}\\tau\\in\\\{0,0\.25,0\.5,0\.75,0\.9\\\}, allowing the selected model to generate the entire remaining response\. Relative to Qwen\-72B, the 0\.5B model loses 54\.08 accuracy points when generating from the prompt \(τ=0\\tau=0\) and 37\.64 points when starting from the reference midpoint\.

### 4\.3Sufficient\-compute maps reveal headroom for efficient inference

We evaluate whether the sufficient\-compute map can guide model routing and speculative decoding\. Figure[5](https://arxiv.org/html/2610.02491#S4.F5)compares measured answer accuracy with projected generation latency on MATH\-500\. The map is supplied as an oracle: a greedy 72B response provides the reference for each problem, the controller receives the positions at which 0\.5B reproduces it\. Appendix[E\.1](https://arxiv.org/html/2610.02491#A5.SS1)specifies the design\.

Figure 5:Accuracy versus latency on MATH\-500\. \(a\) Confidence routing selects 0\.5B when its maximum next\-token probability is at leastθ\\theta\. MoA\-map routing uses the sufficient\-compute map\. \(b\) Draft lengths follow a fixed window, model confidence, or consecutive sufficient\-compute matches\. Gray squares show the 72B\-only baseline\. Vertical bars show one item\-bootstrap standard deviation\.Model routing\.Our baseline, token\-level cascade\[[42](https://arxiv.org/html/2610.02491#bib.bib60)\], uses the 0\.5B model when its maximum next\-token probability reaches a threshold and defers to 72B otherwise\. MoA\-map routing replaces this confidence gate with measured agreement: it selects 0\.5B at positions where the map indicates that it reproduces the token, and 72B elsewhere\. MoA\-map routing achieves 69\.60% accuracy, compared with 68\.40% for the best\-performing confidence setting \(θ=0\.99\\theta=0\.99\), yielding the best accuracy–latency tradeoff \(shown in Figure[5](https://arxiv.org/html/2610.02491#S4.F5)a\)\. The comparison identifies an opportunity beyond confidence\-based routing: token agreement can guide delegation to a smaller model while retaining high answer accuracy\.

Speculative decoding\.The map can also determine how many tokens the small model should draft before verification in speculative decoding\[[32](https://arxiv.org/html/2610.02491#bib.bib1)\]\. MoA\-map drafting follows consecutive positions marked as reproducible by 0\.5B and sends the resulting block to 72B for verification\. It achieves 69\.40% accuracy, compared with 69\.00% for both a fixed 16\-token window and confidence\-adaptive drafting\[[39](https://arxiv.org/html/2610.02491#bib.bib61)\]\. Meanwhile, the projected latencies are 3\.99, 4\.98, and 4\.69 seconds per response, reducing latency by up to 19\.9% \(shown in Figure[5](https://arxiv.org/html/2610.02491#S4.F5)b\)\. MoA\-map drafting generates 32\.6% fewer draft tokens than the fixed\-window baseline: anticipating disagreement reduces proposals that the target would discard\. This reduction in draft work explains its lower latency\. Together, the two interventions use sufficient\-compute map to identify where small\-model predictions suffice and how long drafting should continue before verification\.

## 5Related work

Efficient serving\.Efficient LLM serving combines systems optimizations with adaptive computation\. Orca combines iteration\-level scheduling with selective batching\[[59](https://arxiv.org/html/2610.02491#bib.bib59)\], while vLLM reduces KV\-cache fragmentation and enables memory sharing through PagedAttention\[[30](https://arxiv.org/html/2610.02491#bib.bib27)\]\. Speculative decoding preserves a target distribution by verifying draft proposals\[[32](https://arxiv.org/html/2610.02491#bib.bib1),[12](https://arxiv.org/html/2610.02491#bib.bib2)\]\. Its acceptance rate depends on alignment between distributions; a rejected proposal can still be a valid continuation\[[3](https://arxiv.org/html/2610.02491#bib.bib3)\]\. FrugalGPT and RouteLLM allocate requests across models to balance cost and quality\[[13](https://arxiv.org/html/2610.02491#bib.bib8),[44](https://arxiv.org/html/2610.02491#bib.bib7)\], while token\-level cascades make small/large\-model decisions within a response\[[42](https://arxiv.org/html/2610.02491#bib.bib60)\]\. At the architectural level, MoE increases parameter capacity through sparse expert activation\[[51](https://arxiv.org/html/2610.02491#bib.bib45),[19](https://arxiv.org/html/2610.02491#bib.bib44)\]\. Expert placement and load balancing help translate this sparsity into serving efficiency\[[18](https://arxiv.org/html/2610.02491#bib.bib50),[38](https://arxiv.org/html/2610.02491#bib.bib51)\]\. Our work complements these approaches by measuring which complete model can reproduce each token given its reference prefix, then testing whether this information improves model selection and speculative draft length\.

Token\-level control\.CITER and R2R learn small/large\-model routing\[[62](https://arxiv.org/html/2610.02491#bib.bib34),[20](https://arxiv.org/html/2610.02491#bib.bib35)\], while COLLAB selects agents using long\-term utility\[[8](https://arxiv.org/html/2610.02491#bib.bib37)\]\. Our instrument measures local reference cost before testing its usefulness for these sequential decisions\. Duo\-LLM studies oracle module allocation, FS\-GEN studies small/large\-model collaboration, and FlexiDepth learns layer skipping\[[2](https://arxiv.org/html/2610.02491#bib.bib9),[61](https://arxiv.org/html/2610.02491#bib.bib10),[37](https://arxiv.org/html/2610.02491#bib.bib6)\]\. CALM and Mixture\-of\-Depths also allocate computation within a model\[[50](https://arxiv.org/html/2610.02491#bib.bib4),[49](https://arxiv.org/html/2610.02491#bib.bib5)\]\. Appendix[C\.3](https://arxiv.org/html/2610.02491#A3.SS3)quantifies agreement between our reproduction indices and FlexiDepth’s learned allocations; Appendix[F](https://arxiv.org/html/2610.02491#A6)relates our measurement design to these methods\.

Shared difficulty and trajectory interventions\.Earlier work finds shared learning order across classifiers\[[23](https://arxiv.org/html/2610.02491#bib.bib11)\]and studies token learning trajectories across model scales\[[58](https://arxiv.org/html/2610.02491#bib.bib13),[10](https://arxiv.org/html/2610.02491#bib.bib12)\]\. Our cross\-family test asks whether a related ordering persists within verified generated text\. Its interpretation depends on the operational criterion\[[26](https://arxiv.org/html/2610.02491#bib.bib15)\]\. Critical\-token studies locate influential positions through contrasting continuations\[[34](https://arxiv.org/html/2610.02491#bib.bib14)\]\. Scheduled sampling and work on exposure bias distinguish supplied\-prefix prediction from free generation\[[6](https://arxiv.org/html/2610.02491#bib.bib16),[24](https://arxiv.org/html/2610.02491#bib.bib18),[4](https://arxiv.org/html/2610.02491#bib.bib17)\]\. Model handovers also appear in Reasoning Relay, LLM Shepherding, and RelayGen\[[36](https://arxiv.org/html/2610.02491#bib.bib29),[17](https://arxiv.org/html/2610.02491#bib.bib30),[53](https://arxiv.org/html/2610.02491#bib.bib31)\]\. Our duration sweep relates these continuation interventions to a separately measured reproduction cost\.

## 6Discussion and limitations

Discussion\.Our study uses sufficient compute as an analytical reference for identifying efficiency headroom in existing serving methods\. We show that better allocation information can improve the accuracy–efficiency trade\-off\. However, constructing the map requires evaluating every candidate model along known reference sequences\. Developing practical controllers that recover these benefits without exhaustive evaluation or advance reference access remains an open problem\. The shared ordering of reproduction demands across model families raises a further question: are measured maps transferable? Cross\-family agreement motivates this possibility but does not establish transferability\.

In our experiments, longer words show a weak positive association with reproduction demand, whereas digit\-containing and operator/punctuation spans show weak negative associations\. Regressions on digit and operator indicators, word length, frequency, and position explain only 2\.5–3\.2% of the variance in ranked reproduction demand \(Appendix[C\.2](https://arxiv.org/html/2610.02491#A3.SS2)\)\. Thus, simple surface categories provide limited guidance about which tokens require larger models and identifying the contextual and semantic properties of the expensive tail could be an interesting direction\.

Limitations\.Exact reproduction excludes alternative valid wordings and reasoning paths, particularly in open\-ended natural language\. Meanwhile, exact agreement may be more informative for structured outputs whose tokens have fixed meanings, such as tokenized visual predictions\. The results are conditional on the evaluated model panels and benchmarks\. Most reproduction analyses use checker\-passing references, limiting coverage of unsuccessful trajectories and difficult problems\.

## 7Conclusion

We introduced a token\-level MoA framework for measuring sufficient compute\. Most reference tokens can be reproduced by the smallest model, while a sparse expensive tail accounts for disproportionate assigned computation\. Shared reproduction orderings extend across the tested families, but handover experiments show that local reproduction alone does not ensure reliable continuation\. On MATH\-500, oracle\-guided routing and drafting improve projected efficiency at comparable accuracy, revealing remaining allocation headroom\. Exploiting this opportunity without advance reference access or exhaustive model evaluation remains a central challenge for practical inference\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1)\.
- \[2\]K\. Alizadeh\-Vahid, S\. I\. Mirzadeh, H\. Shahrkokhi, D\. Belenko, F\. Sun, M\. Cho, M\. H\. Sekhavat, M\. Nabi, and M\. Farajtabar\(2024\)Duo\-llm: a framework for studying adaptive computation in large language models\.InNeurIPS Efficient Natural Language and Speech Processing Workshop,pp\. 443–455\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p2.1),[§1](https://arxiv.org/html/2610.02491#S1.p2.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[3\]G\. Bachmann, S\. Anagnostidis, A\. Pumarola, M\. Georgopoulos, A\. Sanakoyeu, Y\. Du, E\. Schönfeld, A\. Thabet, and J\. Kohler\(2025\)Judge decoding: faster speculative sampling requires going beyond model alignment\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 40894–40914\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p3.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[4\]G\. Bachmann and V\. Nagarajan\(2024\)The pitfalls of next\-token prediction\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=76zq8Wkl6Z)Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[5\]D\. Bahdanau, P\. Brakel, K\. Xu, A\. Goyal, R\. Lowe, J\. Pineau, A\. Courville, and Y\. Bengio\(2017\)An actor\-critic algorithm for sequence prediction\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SJDaqqveg)Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p1.1),[§3](https://arxiv.org/html/2610.02491#S3.p2.1)\.
- \[6\]S\. Bengio, O\. Vinyals, N\. Jaitly, and N\. Shazeer\(2015\)Scheduled sampling for sequence prediction with recurrent neural networks\.InAdvances in Neural Information Processing Systems,C\. Cortes, N\. Lawrence, D\. Lee, M\. Sugiyama, and R\. Garnett \(Eds\.\),Vol\.28,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2015/file/e995f98d56967d946471af29d7bf99f1-Paper.pdf)Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[7\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1)\.
- \[8\]S\. Chakraborty, S\. Bhatt, U\. M\. Sehwag, S\. S\. Ghosal, J\. Qiu, M\. Wang, D\. Manocha, F\. Huang, A\. Koppel, and S\. Ganesh\(2025\)Collab: controlled decoding using mixture of agents for LLM alignment\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=7ohlQUbTpp)Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p1.1),[§1](https://arxiv.org/html/2610.02491#S1.p3.1),[§2](https://arxiv.org/html/2610.02491#S2.p6.1),[§3](https://arxiv.org/html/2610.02491#S3.p3.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[9\]S\. Chakraborty, S\. S\. Ghosal, M\. Yin, D\. Manocha, M\. Wang, A\. S\. Bedi, and F\. Huang\(2024\)Transfer q\-star: principled decoding for llm alignment\.Advances in Neural Information Processing Systems37,pp\. 101725–101761\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p1.1),[§3](https://arxiv.org/html/2610.02491#S3.p2.1)\.
- \[10\]T\. A\. Chang, Z\. Tu, and B\. K\. Bergen\(2024\)Characterizing learning curves during language model pre\-training: learning, forgetting, and stability\.Transactions of the Association for Computational Linguistics12,pp\. 1346–1362\.Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[11\]A\. Chatterji, T\. Cunningham, D\. J\. Deming, Z\. Hitzig, C\. Ong, C\. Y\. Shan, and K\. Wadman\(2025\)How people use chatgpt\.Technical reportNational Bureau of Economic Research\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1)\.
- \[12\]C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper\(2023\)Accelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§2](https://arxiv.org/html/2610.02491#S2.p3.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[13\]L\. Chen, M\. Zaharia, and J\. Zou\(2023\)FrugalGPT: how to use large language models while reducing cost and improving performance\.External Links:2305\.05176Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§2](https://arxiv.org/html/2610.02491#S2.p4.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[14\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4](https://arxiv.org/html/2610.02491#S4.p2.1)\.
- \[15\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1)\.
- \[16\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2610.02491#S4.p2.1)\.
- \[17\]Z\. Dong, H\. Sharma, E\. O’Toole, J\. P\. Champati, and K\. Wu\(2026\)Pay for hints, not answers: llm shepherding for cost\-efficient inference\.arXiv preprint arXiv:2601\.22132\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p4.1),[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[18\]Z\. Du, S\. Li, Y\. Wu, X\. Jiang, J\. Sun, Q\. Zheng, Y\. Wu, A\. Li, H\. H\. Li, and Y\. Chen\(2024\)Sida: sparsity\-inspired data\-aware serving for efficient and scalable large mixture\-of\-experts models\.Proceedings of Machine Learning and Systems6,pp\. 224–238\.Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[19\]W\. Fedus, B\. Zoph, and N\. Shazeer\(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p3.1),[§2](https://arxiv.org/html/2610.02491#S2.p5.2),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[20\]T\. Fu, Y\. Ge, Y\. You, E\. Liu, Z\. Yuan, G\. Dai, S\. Yan, H\. Yang, and Y\. Wang\(2025\)R2R: efficiently navigating divergent reasoning paths with small\-large model token routing\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=DpeJYRFRQY)Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§2](https://arxiv.org/html/2610.02491#S2.p4.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[21\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4](https://arxiv.org/html/2610.02491#S4.p1.1)\.
- \[22\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§A\.1](https://arxiv.org/html/2610.02491#A1.SS1.p1.1),[§4](https://arxiv.org/html/2610.02491#S4.p1.1)\.
- \[23\]G\. Hacohen, L\. Choshen, and D\. Weinshall\(2020\)Let’s agree to agree: neural networks share classification order on real datasets\.InInternational Conference on Machine Learning,pp\. 3950–3960\.Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[24\]T\. He, J\. Zhang, Z\. Zhou, and J\. Glass\(2021\)Exposure bias versus self\-recovery: are distortions really incremental for autoregressive text generation?\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5087–5102\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.415/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.415)Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[25\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1)\.
- \[26\]A\. Z\. Jacobs and H\. Wallach\(2021\)Measurement and fairness\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 375–385\.Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[27\]R\. A\. Jacobs, M\. I\. Jordan, S\. J\. Nowlan, and G\. E\. Hinton\(1991\)Adaptive mixtures of local experts\.Neural computation3\(1\),pp\. 79–87\.Cited by:[§2](https://arxiv.org/html/2610.02491#S2.p5.2)\.
- \[28\]N\. Jain, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\(2025\)Livecodebench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 58791–58831\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p2.1),[§4](https://arxiv.org/html/2610.02491#S4.p2.1)\.
- \[29\]T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov\(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Link](https://aclanthology.org/Q19-1026/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p2.1)\.
- \[30\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p5.1),[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§4](https://arxiv.org/html/2610.02491#S4.p3.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[31\]H\. Kydlíček\(2025\)Math\-verify: mathematical expression verification\.External Links:[Link](https://github.com/huggingface/Math-Verify)Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p4.1)\.
- \[32\]Y\. Leviathan, M\. Kalman, and Y\. Matias\(2023\)Fast inference from transformers via speculative decoding\.InInternational conference on machine learning,pp\. 19274–19286\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§2](https://arxiv.org/html/2610.02491#S2.p3.1),[§4\.3](https://arxiv.org/html/2610.02491#S4.SS3.p3.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[33\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[§4](https://arxiv.org/html/2610.02491#S4.p2.1)\.
- \[34\]Z\. Lin, T\. Liang, J\. Xu, Q\. Liu, X\. Wang, R\. Luo, C\. Shi, S\. Li, Y\. Yang, and Z\. Tu\(2025\)Critical tokens matter: token\-level contrastive estimation enhances LLM’s reasoning capability\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=fnz1g18EdI)Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[35\]J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang\(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in neural information processing systems36,pp\. 21558–21572\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1)\.
- \[36\]L\. Lu, J\. Zhang, S\. E\. Chua, S\. Kim, K\. Zhu, S\. O’Brien, and V\. Sharma\(2025\)Evaluating stability and interchangeability of large language models in mathematical reasoning\.InSocially Responsible and Trustworthy Foundation Models at NeurIPS 2025,External Links:[Link](https://openreview.net/forum?id=QqXPAuKi76)Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p4.1),[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[37\]X\. Luo, W\. Wang, and X\. Yan\(2025\)Adaptive layer\-skipping in pre\-trained llms\.arXiv preprint arXiv:2503\.23798\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p2.1),[§1](https://arxiv.org/html/2610.02491#S1.p2.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[38\]H\. Ma, Z\. Du, and Y\. Chen\(2025\)MoE\-GPS: guidlines for prediction strategy with expert duplication in moe load balancing\.InMachine Learning for Systems 2025,External Links:[Link](https://openreview.net/forum?id=VCh5vBSxxb)Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[39\]J\. Mamou, O\. Pereg, J\. Gante, L\. Tunstall, D\. Korat, N\. Timor, and M\. Wasserblat\(2024\)Faster assisted generation with dynamic speculation\.Note:Hugging Face BlogExternal Links:[Link](https://huggingface.co/blog/dynamic_speculation_lookahead)Cited by:[§E\.1](https://arxiv.org/html/2610.02491#A5.SS1.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2610.02491#S4.SS3.p3.1)\.
- \[40\]I\. Mirzadeh, K\. Alizadeh\-Vahid, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. Farajtabar\(2025\)Gsm\-symbolic: understanding the limitations of mathematical reasoning in large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 94743–94765\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1)\.
- \[41\]S\. Mudgal, J\. Lee, H\. Ganapathy, Y\. Li, T\. Wang, Y\. Huang, Z\. Chen, H\. Cheng, M\. Collins, T\. Strohman, J\. Chen, A\. Beutel, and A\. Beirami\(2024\)Controlled decoding from language models\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=bVIcZb7Qa0)Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p1.1),[§3](https://arxiv.org/html/2610.02491#S3.p2.1)\.
- \[42\]H\. Narasimhan, W\. Jitkrittum, A\. S\. Rawat, S\. Kim, N\. Gupta, A\. K\. Menon, and S\. Kumar\(2025\)Faster cascades via speculative decoding\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 44949–44987\.Cited by:[§E\.1](https://arxiv.org/html/2610.02491#A5.SS1.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2610.02491#S4.SS3.p2.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[43\]T\. OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, A\. Ettinger, M\. Guerquin, D\. Heineman, H\. Ivison, P\. W\. Koh, J\. Liu, S\. Malik, W\. Merrill, L\. J\. V\. Miranda, J\. Morrison, T\. Murray, C\. Nam, J\. Poznanski, V\. Pyatkin, A\. Rangapur, M\. Schmitz, S\. Skjonsberg, D\. Wadden, C\. Wilhelm, M\. Wilson, L\. Zettlemoyer, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi\(2025\)2 olmo 2 furious\.External Links:2501\.00656,[Link](https://arxiv.org/abs/2501.00656)Cited by:[§A\.1](https://arxiv.org/html/2610.02491#A1.SS1.p1.1),[§4](https://arxiv.org/html/2610.02491#S4.p1.1)\.
- \[44\]I\. Ong, A\. Almahairi, V\. Wu,et al\.\(2025\)RouteLLM: learning to route LLMs with preference data\.InInternational Conference on Learning Representations \(ICLR\),External Links:2406\.18665Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§2](https://arxiv.org/html/2610.02491#S2.p4.1),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[45\]R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean\(2023\)Efficiently scaling transformer inference\.Proceedings of machine learning and systems5,pp\. 606–624\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1)\.
- \[46\]Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§A\.1](https://arxiv.org/html/2610.02491#A1.SS1.p1.1),[§4](https://arxiv.org/html/2610.02491#S4.p1.1)\.
- \[47\]Qwen Team\(n\.d\.\)Qwen2\.5 speed benchmark\.Note:[https://qwen\.readthedocs\.io/en/v2\.5/benchmark/speed\_benchmark\.html](https://qwen.readthedocs.io/en/v2.5/benchmark/speed_benchmark.html)Official BF16 vLLM 0\.6\.3 benchmarks on A100 80GB GPUs; accessed 2026\-09\-25Cited by:[§E\.2](https://arxiv.org/html/2610.02491#A5.SS2.p1.1)\.
- \[48\]M\. Ranzato, S\. Chopra, M\. Auli, and W\. Zaremba\(2015\)Sequence level training with recurrent neural networks\.arXiv preprint arXiv:1511\.06732\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p1.1),[§3](https://arxiv.org/html/2610.02491#S3.p2.1)\.
- \[49\]D\. Raposo, S\. Ritter, B\. Richards, T\. Lillicrap, P\. C\. Humphreys, and A\. Santoro\(2024\)Mixture\-of\-depths: dynamically allocating compute in transformer\-based language models\.arXiv preprint arXiv:2404\.02258\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[50\]T\. Schuster, A\. Fisch, J\. Gupta, M\. Dehghani, D\. Bahri, V\. Tran, Y\. Tay, and D\. Metzler\(2022\)Confident adaptive language modeling\.Advances in Neural Information Processing Systems35,pp\. 17456–17472\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[51\]N\. Shazeer, \*\. Mirhoseini, \*\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p3.1),[§2](https://arxiv.org/html/2610.02491#S2.p5.2),[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[52\]Z\. Shen, H\. Lang, B\. Wang, Y\. Kim, and D\. Sontag\(2024\)Learning to decode collaboratively with multiple language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12974–12990\.Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p1.1),[§2](https://arxiv.org/html/2610.02491#S2.p4.1),[§2](https://arxiv.org/html/2610.02491#S2.p6.1)\.
- \[53\]J\. Song, Y\. Kim, and J\. Kim\(2026\)RelayGen: intra\-generation model switching for efficient reasoning\.arXiv preprint arXiv:2602\.06454\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p4.1),[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[54\]M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. H\. Chi, D\. Zhou,et al\.\(2023\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1),[§4](https://arxiv.org/html/2610.02491#S4.p2.1)\.
- \[55\]F\. Team\(2024\)The falcon 3 family of open models\.External Links:[Link](https://huggingface.co/blog/falcon3)Cited by:[§A\.1](https://arxiv.org/html/2610.02491#A1.SS1.p1.1),[§4](https://arxiv.org/html/2610.02491#S4.p1.1)\.
- \[56\]J\. Wang, J\. WANG, B\. Athiwaratkun, C\. Zhang, and J\. Zou\(2025\)Mixture\-of\-agents enhances large language model capabilities\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=h0ZfDIrj7T)Cited by:[§1](https://arxiv.org/html/2610.02491#S1.p3.1),[§2](https://arxiv.org/html/2610.02491#S2.p6.1)\.
- \[57\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.\(2024\)Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1),[§4](https://arxiv.org/html/2610.02491#S4.p2.1)\.
- \[58\]M\. Xia, M\. Artetxe, C\. Zhou, X\. V\. Lin, R\. Pasunuru, D\. Chen, L\. Zettlemoyer, and V\. Stoyanov\(2023\)Training trajectories of language models across scales\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13711–13738\.Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p3.1)\.
- \[59\]G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun\(2022\)Orca: a distributed serving system for\{\\\{transformer\-based\}\\\}generative models\.In16th USENIX symposium on operating systems design and implementation \(OSDI 22\),pp\. 521–538\.Cited by:[§5](https://arxiv.org/html/2610.02491#S5.p1.1)\.
- \[60\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)Hellaswag: can a machine really finish your sentence?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4791–4800\.Cited by:[Appendix A](https://arxiv.org/html/2610.02491#A1.p3.1)\.
- \[61\]K\. Zhang, J\. Wang, N\. Ding, B\. Qi, E\. Hua, X\. Lv, and B\. Zhou\(2024\)Fast and slow generating: an empirical study on large and small language models collaborative decoding\.arXiv preprint arXiv:2406\.12295\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p2.1),[§1](https://arxiv.org/html/2610.02491#S1.p2.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.
- \[62\]W\. Zheng, Y\. Chen, W\. Zhang, S\. Kundu, Y\. Li, Z\. Liu, E\. P\. Xing, H\. Wang, and H\. Yao\(2025\)Citer: collaborative inference for efficient large language model decoding with token\-level routing\.arXiv preprint arXiv:2502\.01976\.Cited by:[Appendix F](https://arxiv.org/html/2610.02491#A6.p1.1),[§1](https://arxiv.org/html/2610.02491#S1.p3.1),[§2](https://arxiv.org/html/2610.02491#S2.p4.1),[§2](https://arxiv.org/html/2610.02491#S2.p6.1),[§5](https://arxiv.org/html/2610.02491#S5.p2.1)\.

## Appendix ADatasets, models, and selected populations

Benchmark coverage\.Table[A1](https://arxiv.org/html/2610.02491#A1.T1)separates attempted prompts, each generator’s verified pool, and their intersection\. The primary collection contains twelve datasets and 50,019 prompts\. GSM\-Symbolic adds 5,000 prompts as a separate robustness analysis\. Each core dataset has two\-generator intersection coverage of at least 60%\. The expanded collection is reported per dataset and is not added to the pooled core estimand\.

TableA1:Benchmark coverage before and after verification\. The first three rows define the core\. Counts refer to items\.GSM8K uses the test split, MATH\-500 the released 500\-problem evaluation set, and HumanEval the 164 official function\-completion problems\. LiveCodeBench uses 219 problems with usable stdin/stdout tests from the 400\-problem mirror available to the pipeline\. Both public and private stored tests are included\. This is a subset, without a post\-training\-cutoff guarantee\[[28](https://arxiv.org/html/2610.02491#bib.bib25)\]\. NQ\-Open uses 3,610 validation questions with normalized exact\-match answer aliases\[[29](https://arxiv.org/html/2610.02491#bib.bib26)\]\.

The extensions useevalplus/mbppplus\(378 test items;[35](https://arxiv.org/html/2610.02491#bib.bib40)\),HuggingFaceH4/aime\_2024\(30 problems\),allenai/ai2\_arc\(ARC\-Challenge test;[15](https://arxiv.org/html/2610.02491#bib.bib38)\),lukaemon/bbh\(the loaded task test splits;[54](https://arxiv.org/html/2610.02491#bib.bib41)\),TIGER\-Lab/MMLU\-Pro\(test;[57](https://arxiv.org/html/2610.02491#bib.bib43)\),cais/mmlu\(all subjects, test;[25](https://arxiv.org/html/2610.02491#bib.bib42)\), andRowan/hellaswag\(validation;[60](https://arxiv.org/html/2610.02491#bib.bib39)\)\. BBH contains varied answer formats, not exclusively multiple\-choice questions\. MBPP\+ prompts include a supplied assertion specifying the required function call\. GSM\-Symbolic uses the released main split\[[40](https://arxiv.org/html/2610.02491#bib.bib32)\]; its transformed problems provide a replication substrate, but their public release does not establish absence from model training\.

Prompts and checkers\.Mathematical prompts request step\-by\-step reasoning and a final answer; code prompts request executable code; knowledge and reasoning prompts request an answer in the format specified by the dataset adapter\. GSM8K extracts a final numeric answer and compares it with numeric tolerance10−610^\{\-6\}\. MATH\-500 uses symbolic answer equivalence through Math\-Verify\[[31](https://arxiv.org/html/2610.02491#bib.bib28)\]\. HumanEval executes extracted code with the stored official tests and a timeout\. LiveCodeBench runs the stored stdin/stdout cases; MBPP\+ runs the executable tests available in the loaded records\. The multiple\-choice adapters parse the final option, BBH uses task\-answer matching, and NQ\-Open normalizes case, articles, punctuation, and whitespace before exact alias matching\. These are outcome checkers: intermediate reasoning is retained even when a valid final answer follows an incorrect step\.

Generation settings\.Reference sampling usesK=4K=4, temperature 0\.7, top\-pp0\.95, and seed 20260729\. Output caps are 512 tokens for GSM8K, HumanEval, MBPP\+, ARC\-Challenge, and MMLU; 1,024 for MATH\-500 and LiveCodeBench; 64 for NQ\-Open; 2,048 for AIME; 768 for BBH and MMLU\-Pro; and 384 for HellaSwag\. GSM\-Symbolic uses 512\. The earliest passing sample is selected independently for each generator\. A selected reference can be non\-modal under its own generator because it was sampled\. Generation and scoring use bfloat16 with vLLM\[[30](https://arxiv.org/html/2610.02491#bib.bib27)\]; the project runtime records version 0\.14\.1\.

### A\.1Model panel and auxiliary models

Table[A2](https://arxiv.org/html/2610.02491#A1.T2)summarizes model roles and sizes\. Qwen2\.5, OLMo\-2, and R1\-Distill\-Qwen supply the fifteen candidate models displayed in Figure[3](https://arxiv.org/html/2610.02491#S4.F3)\[[46](https://arxiv.org/html/2610.02491#bib.bib19),[43](https://arxiv.org/html/2610.02491#bib.bib20),[22](https://arxiv.org/html/2610.02491#bib.bib58)\]\. The cross\-family agreement analysis in Table[1](https://arxiv.org/html/2610.02491#S4.T1)uses Qwen2\.5, OLMo\-2, and the additional Falcon3 panel\[[55](https://arxiv.org/html/2610.02491#bib.bib57)\]\. These are distinct panel comparisons\.

TableA2:Model roles and sizes\. Measurement panels supply candidate next\-token policies; the Falcon panel tests cross\-family agreement\. Reference generators supply candidate responses for task verification; the additional MoE generator tests sensitivity to the reference source\. The probability probe supplies surprisal controls\. Sizes are nominal billions of parameters; the final row distinguishes total and active parameters\. Qwen\-72B serves both as a panel member and a generator\.Qwen model identifiers have the formQwen/Qwen2\.5\-\{size\}B\-Instruct\. The OLMo identifiers areallenai/OLMo\-2\-0425\-1B\-Instruct,allenai/OLMo\-2\-1124\-7B\-Instruct,allenai/OLMo\-2\-1124\-13B\-Instruct, andallenai/OLMo\-2\-0325\-32B\-Instruct\. Falcon identifiers aretiiuae/Falcon3\-\{size\}B\-Instruct\. The second main generator ismeta\-llama/Llama\-3\.3\-70B\-Instruct; the probability probe ismeta\-llama/Llama\-3\.1\-8B\-Instruct\.

The reasoning\-distilled panel contains four DeepSeek\-R1\-Distill\-Qwen models, with 1\.5, 7, 14, and 32B parameters\. Their identifiers followdeepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-<size\>B\. These models test sensitivity to reasoning distillation\. A third generator,Qwen/Qwen3\-30B\-A3B, supplies mixture\-of\-experts reference text with thinking mode disabled\. Its nominal 30B total and 3B active parameter counts describe architecture, not a measured change in per\-token compute allocation\. These extensions use the three core datasets\.

## Appendix BMeasurement and statistical details

Teacher\-forced scoring\.For each model, the user prompt is rendered using its chat template and assistant\-generation header\. The reference answer is tokenized separately without special tokens and appended to the prompt tokens\. vLLM returns prompt log probabilities for the answer positions\. The cache stores target rank, target log probability, argmax identity, character offsets, and a truncated distribution\. Log probabilities are rounded to four decimal places\. Each model receives the reference prefix even when its own preceding prediction disagrees with the reference\. Every model is evaluated at every position; the algorithm does not stop after a smaller model matches\. The primary pass indicator is target rank one\. Alternative criteria use rank at most five or target log probability at leastlog⁡\(0\.5\)\\log\(0\.5\), compared at the stored float32 precision\.

The entropy control sums−p​log⁡p\-p\\log pover the returned top\-twenty entries and the target entry when additionally returned\. It is an unnormalized truncated entropy proxy\.

Word alignment\.The alignment routine identifies non\-whitespace runs and removes leading and trailing ASCII punctuation, retaining all\-punctuation spans\. A native token is assigned to the word with maximal character overlap; ties go to the first word, and zero\-overlap tokens are dropped\. Only common mapped words are joined across families\. At wordww, the ordinal statistic is

DF\(w\)=maxt:t↦wRF\(t\),RF\(t\)=min\{k:aF,k,t=1\}\.D\_\{F\}\(w\)=\\max\_\{t:\\,t\\mapsto w\}R\_\{F\}\(t\),\\qquad R\_\{F\}\(t\)=\\min\\\{k:a\_\{F,k,t\}=1\\\}\.\(10\)An unresolved token gives the word the unresolved index above the panel\. The maximum summarizes constituent token indices; because predictions need not be monotone in size, it does not certify that a single model reproduces all constituent tokens\. Surprisal uses sums over assigned tokens\.

Partial rank correlation\.Letr⁡\(⋅\)r\(\\cdot\)assign average ranks to ties and let

Z=\[𝟏,r⁡\(Sgen\),r⁡\(Sprobe\),r⁡\(log⁡\(1\+position\)\)\]\.Z=\[\\mathbf\{1\},r\(S\_\{\\rm gen\}\),r\(S\_\{\\rm probe\}\),r\(\\log\(1\+\\mathrm\{position\}\)\)\]\.WritingPZP\_\{Z\}for least\-squares projection onto the columns ofZZ, we compute

ρpartial=corr⁡\(\(I−PZ\)​r​\(DQ\),\(I−PZ\)​r​\(DO\)\)\.\\rho\_\{\\rm partial\}=\\operatorname\{corr\}\\big\(\(I\-P\_\{Z\}\)r\(D\_\{Q\}\),\(I\-P\_\{Z\}\)r\(D\_\{O\}\)\\big\)\.\(11\)Tables report the estimate±\\pmthe SD of 2,000 item\-bootstrap estimates\. This bootstrap SD estimates the standard error under item sampling; models and generation settings are held fixed\. Main\-text estimates and figure bars or bands use the same bootstrap\-SD convention where these SDs are available\. Point\-only results omit uncertainty when no SD is available\. The bootstrap samples item identifiers with replacement and retains all word rows belonging to each sampled item, including multiplicities\. Ranks and regressions are recomputed for each resample\. Core Qwen text contributes 319,336 words from 1,800 items; Llama text contributes 302,959 words from 1,797 items\. The intersection sensitivity retains 307,850 and 293,549 words, respectively, from 1,758 items, giving point estimates of 0\.319 and 0\.415\.

### B\.1Architecture\-derived FLOP accounting

We count matrix\-multiplication FLOPs from the released model configurations, with one multiply\-add counted as two operations\. The count includes attention projections, the two attention matrix products, gated feed\-forward layers, MoE router projections, and the vocabulary projection\. Embedding lookup, normalization, activations, bias additions, rotary embeddings, softmax, and expert dispatch/combine are outside this arithmetic convention\. Counts describe logical causal operations; they do not include kernel padding, communication, or memory movement\.

Lethhbe hidden width,qqthe total query\-head width,rrthe total key/value\-head width,LLthe number of layers, andVVvocabulary size\. For layerℓ\\ell, letIℓactI\_\{\\ell\}^\{\\rm act\}be the sum of the intermediate widths of the feed\-forward networks activated for one token\. A dense gated MLP hasIℓact=IℓI\_\{\\ell\}^\{\\rm act\}=I\_\{\\ell\}; a sparse layer withaℓa\_\{\\ell\}selected equal\-sized experts hasIℓact=aℓ​IℓI\_\{\\ell\}^\{\\rm act\}=a\_\{\\ell\}I\_\{\\ell\}\. LetEℓE\_\{\\ell\}denote the number of experts scored by a router, and setEℓ=0E\_\{\\ell\}=0in a dense layer\. Define

A\\displaystyle A=∑ℓ=1L\[4​h​\(q\+r\)\+6​h​Iℓact\+2​h​Eℓ\],\\displaystyle=\\sum\_\{\\ell=1\}^\{L\}\\left\[4h\(q\+r\)\+6hI\_\{\\ell\}^\{\\rm act\}\+2hE\_\{\\ell\}\\right\],B\\displaystyle B=4​L​q,\\displaystyle=4Lq,H\\displaystyle H=2​h​V\.\\displaystyle=2hV\.\(12\)The first term counts query, key, value, and output projections; the second counts the gate, up, and down feed\-forward matrices\. The router multiplies anhh\-dimensional token by anh×Eℓh\\times E\_\{\\ell\}matrix\. The two attention products together requireBBFLOPs per query/key position pair across layers\. For a prefix ofpptokens with the preceding keys and values cached,

cF,k​\(s\)=AF,k\+BF,k​p\+HF,k\.c\_\{F,k\}\(s\)=A\_\{F,k\}\+B\_\{F,k\}p\+H\_\{F,k\}\.\(13\)The saved prompt length and native\-token position giveppseparately for each model\. The model ordering is unchanged throughout the measured panels; FLOP concentration sorts the resulting per\-token values, including their prefix dependence\. Qwen3\-30B\-A3B uses eight active experts out of 128 per layer, so only those eight feed\-forward networks enterIℓactI\_\{\\ell\}^\{\\rm act\}\.

For a request withm≥1m\\geq 1input tokens andn≥1n\\geq 1generated tokens, causal prefill processesm⁡\(m\+1\)/2m\(m\+1\)/2attention pairs\. Only its final position requires vocabulary logits for generation\. The remainingn−1n\-1predictions use cached decoding at prefix lengthsm\+1,…,m\+n−1m\+1,\\ldots,m\+n\-1\. Thus

ℱk​\(m,n\)=m​Ak\+Bk​m​\(m\+1\)2\+Hk\+∑j=1n−1\[Ak\+Bk​\(m\+j\)\+Hk\]\.\\mathcal\{F\}\_\{k\}\(m,n\)=mA\_\{k\}\+\\frac\{B\_\{k\}m\(m\+1\)\}\{2\}\+H\_\{k\}\+\\sum\_\{j=1\}^\{n\-1\}\\left\[A\_\{k\}\+B\_\{k\}\(m\+j\)\+H\_\{k\}\\right\]\.\(14\)This function increases with requested output length\. Reserving its value at a request’s output cap therefore bounds the counted FLOPs of every completion admitted under that cap\. Model configurations, their source revisions, and file hashes accompany the FLOP\-count implementation\.

### B\.2What the bound means

Local witness bound\.Fix a prefix and a target token\. For a candidate model class𝒞\\mathcal\{C\}containing the tested family, letd𝒞​\(t\)d\_\{\\mathcal\{C\}\}\(t\)be the infimum FLOP count at that prefix of a member whose deterministic prediction equals that token\. Whenever a tested member succeeds,d𝒞​\(t\)≤dF​\(t\)d\_\{\\mathcal\{C\}\}\(t\)\\leq d\_\{F\}\(t\)\. The comparison fixes the model class, prediction rule, and supplied prefix\.

Exact\-reference selector\.Assume all panel models share a token alphabet, decode greedily with a fixed deterministic tie\-breaking rule, and have additive next\-token FLOP costsck​\(s\)\>0c\_\{k\}\(s\)\>0, available prefix caches, and no switching or state\-construction overhead\. Suppose every token of a finite referenceyyis resolved\. A selector may choose any model at each prefix, with advance knowledge of the full reference and all panel predictions\. Writerollout⁡\(σ,x\)\\operatorname\{rollout\}\(\\sigma,x\)for the response generated by a deterministic selectorσ\\sigmawith promptxx, andC⁡\(σ\)C\(\\sigma\)for its additive prediction cost\. Among selectors that reproduceyyexactly,

minσ:rollout⁡\(σ,x\)=yC\(σ\)=∑t=1TdF\(t\)\.\\min\_\{\\sigma:\\,\\operatorname\{rollout\}\(\\sigma,x\)=y\}C\(\\sigma\)=\\sum\_\{t=1\}^\{T\}d\_\{F\}\(t\)\.\(15\)To attain the right\-hand side, choose a cheapest matching model at position one\. Inductively, the generated prefix then equals the reference prefix, so the cheapest matching model at the next position emits the required token\. Conversely, every selector reproducingyymust choose a matching model at each position and therefore incurs at leastdF​\(t\)d\_\{F\}\(t\)there\. Summing proves Eq\.[15](https://arxiv.org/html/2610.02491#A2.E15)\. If stopping is to be reproduced, it must also be included in the reference alphabet and coverage assumption\.

Equation[15](https://arxiv.org/html/2610.02491#A2.E15)characterizes selectors constrained to one reference\. For a checker\-passingyy, the task\-correct objectiveminσ:V⁡\(x,rollout⁡\(σ,x\)\)=1C\(σ\)\\min\_\{\\sigma:V\(x,\\operatorname\{rollout\}\(\\sigma,x\)\)=1\}C\(\\sigma\)allows all such selectors and those producing other correct responses\. Its optimum is therefore at most∑tdF​\(t\)\\sum\_\{t\}d\_\{F\}\(t\)under the same assumptions\. This establishes the direction of the bound when alternative solutions are allowed\. The empirical corpus includes unresolved tokens and separately tokenized families; the cost proxy omits map acquisition and model\-state transfer\. This exact\-reference result interprets the measured quantity under its stated assumptions, while the allocation studies evaluate concrete policies\.

## Appendix CAdditional reproduction results

The results below supplement the concentration and shared\-ordering findings\. Figure[6](https://arxiv.org/html/2610.02491#A3.F6)restores the per\-dataset view summarized by Table[1](https://arxiv.org/html/2610.02491#S4.T1); Table[A3](https://arxiv.org/html/2610.02491#A3.T3)gives its numerical estimates\. Tables[A4](https://arxiv.org/html/2610.02491#A3.T4)provide additional controls\.

Figure 6:Cross\-family agreement across benchmark subsets\. Partial Spearman correlation of Qwen/OLMo word indices after controlling generator surprisal, probe surprisal, and position; bars show±\\pmone item\-bootstrap SD\. Zero denotes no remaining rank association, and larger positive values indicate stronger shared ordering\. Pooling uses the three core datasets\. The right column gives each generator’s verified item count \(Qwen/Llama\), including six/nine AIME items; Table[A1](https://arxiv.org/html/2610.02491#A1.T1)specifies the evaluated populations\.TableA3:Per\-dataset Qwen/OLMo partial correlations with bootstrap SDs \(standard errors\)\.TableA4:Sensitivity of cross\-family agreement, with bootstrap SDs \(standard errors\)\. The frequency/entropy and loss\-gap rows extend the primary controls successively\. Other rows specify separate controls or populations\.TableA5:Gini coefficients of architecture\-derived prediction FLOPs among resolved tokens; parentheses give unresolved\-token percentages among all tokens\. These values exclude the imputed costs used in Figure[3](https://arxiv.org/html/2610.02491#S4.F3)\.### C\.1Concentration with unresolved\-token costs

Figure[3](https://arxiv.org/html/2610.02491#S4.F3)uses the same 1,800 Qwen\-generated verified references across the Qwen, OLMo, and R1\-Distill\-Qwen panels: 1,281 GSM8K, 375 MATH\-500, and 144 HumanEval responses\. Each panel uses its own native tokenization\. The distribution bars divide by all reference tokens, including unresolved positions\.

An unresolved position has no observed successful agent and therefore no finite measured sufficient compute under Definition[2\.1](https://arxiv.org/html/2610.02491#S2.Thmtheorem1)\. For the displayed cumulative\-cost curves, we assign it twice the architecture\-derived prediction FLOPs of the panel’s largest model at that prefix\. Writingkmaxk\_\{\\max\}for this model, the plotted cost is

d~F​\(t\)=\{dF​\(t\),𝒜F​\(t\)≠∅,2​cF,kmax​\(st⋆\),𝒜F​\(t\)=∅\.\\widetilde\{d\}\_\{F\}\(t\)=\\begin\{cases\}d\_\{F\}\(t\),&\\mathcal\{A\}\_\{F\}\(t\)\\neq\\varnothing,\\\\ 2c\_\{F,k\_\{\\max\}\}\(s\_\{t\}^\{\\star\}\),&\\mathcal\{A\}\_\{F\}\(t\)=\\varnothing\.\\end\{cases\}\(16\)The multiplier is an assumed cost for the visualization\. It does not establish what an untested model would cost or that an unresolved token necessarily requires a larger model\. The matching outcomes themselves are unchanged by this convention\.

Figure[3](https://arxiv.org/html/2610.02491#S4.F3)assigns each unresolved position twice the cost of the largest panel model\. This section reports the same shares among resolved tokens only \(top 10%: 51–60% for Qwen2\.5, 40–44% for OLMo\-2, 26–32% for R1\-Distill\-Qwen\) and Gini coefficients of resolved costs \(Tables[A5](https://arxiv.org/html/2610.02491#A3.T5)and[A6](https://arxiv.org/html/2610.02491#A3.T6)\); allocation is uneven under both conventions\.

Within each panel and dataset, tokens are ordered by decreasing plotted cost\. The top\-ppshare divides their cumulative cost by the total cost of all reference tokens\. Fractional percentile boundaries are evaluated by linear interpolation of cumulative token counts and costs\. Table[A6](https://arxiv.org/html/2610.02491#A3.T6)gives the values displayed at 1% and 10%, alongside the corresponding resolved\-only calculation\. In the resolved\-only calculation, both the percentile and total cost use resolved tokens\. The two columns therefore summarize different populations\.

TableA6:Concentration on Qwen\-generated verified references under two cost conventions\. Unresolved is the percentage of all native tokens with no matching panel model\. Resolved\-only columns give FLOP shares for the top 1% / top 10% of resolved tokens\. All\-token columns give the corresponding shares among all tokens after assigning each unresolved position twice the largest model’s prediction FLOPs\. All entries are percentages\.The resolved\-only Gini coefficients in Table[A5](https://arxiv.org/html/2610.02491#A3.T5)use the measured architecture\-derived costs without imputation\. They also include Llama\-generated references, which provide a comparison of reference sources\. The pooled resolved\-only Qwen cost share of 3\.03% for half the assigned FLOPs is another such summary; it should not be substituted for the all\-token percentile shares in Figure[3](https://arxiv.org/html/2610.02491#S4.F3)\.

### C\.2Controls and reference variation

Control variants\.Table[A4](https://arxiv.org/html/2610.02491#A3.T4)reports each variant with its own bootstrap SD\. Frequency is estimated from the available text, and entropy is approximate as described above\. The loss\-gap controls use each family’s smallest and largest model on the same reference\. The high\-confidence subset requires every assigned generator token to have stored probability at least 0\.5; it retains 306,490 Qwen\-text words and 290,328 Llama\-text words\. This is stricter than merely asking whether a token is the generator’s argmax\. Surface\-feature regressions on ranks, using digit/operator indicators, word length, frequency, and position, haveR2=0\.025R^\{2\}=0\.025and0\.0320\.032\. These regressions quantify that feature set’s linear explanatory power\.

Reference variation and memorization screening\.The second\-target analysis uses the next available checker\-passing sample among the original four candidates\. Its pooled correlations are 0\.328 and 0\.409; mean item demand correlates 0\.574 and 0\.653 across two targets from the same generator\. A separate screening campaign tests answer recall without demonstrated solution ability and continuation of problem text\. Screening covers six model sizes: Qwen 0\.5, 1\.5, 7, and 72B, and OLMo 7 and 32B\. The rule flags 33 of the 1,758 intersection items\. Removing flagged identifiers from the analysis leaves correlations near the originals; the pooled 0\.5B handover difference atτ=0\.25\\tau=0\.25changes from−0\.504\-0\.504to−0\.500\-0\.500\. This checks sensitivity to these flags, without certifying that unflagged items were absent from training\.

Difficulty and generator extensions\.The MATH\-500 difficulty stratification gives Qwen/OLMo correlations of 0\.315 and 0\.460 on levels 1 to 3, versus 0\.337 and 0\.464 on levels 4 and 5, for Qwen and Llama text\. The harder groups contain 177 and 166 verified items, respectively\. On Qwen3\-30B\-A3B text, the pooled core correlation is0\.5200\.520across 293,082 words\. The R1\-Distill\-Qwen panel correlates 0\.255/0\.379 with Qwen and 0\.267/0\.369 with OLMo, on Qwen/Llama text\.

Acceptance proxies\.For each of Qwen and OLMo, we compare its smallest and largest model using greedy disagreement and a soft overlap computed from cached truncated probabilities\. Cross\-family partial correlations are 0\.150/0\.199 for greedy rejection and 0\.179/0\.384 for soft rejection, versus 0\.322/0\.417 for reproduction indices\. Truncated overlap omits probability mass and is a proxy for acceptance, not an exact full\-vocabulary speculative acceptance probability\. These correlations compare the operational signals under the same controls\.

Scope of cross\-family comparisons\.The cross\-family comparisons do not separately control architecture, tokenization, training data, or instruction tuning\. The R1\-Distill\-Qwen panel shares Qwen lineage, and independently developed families may also share training data\. Positive associations therefore establish shared ordering within the evaluated panels, without identifying its source or demonstrating that a map transfers between families\. The reference\-variation and memorization\-screening analyses test sensitivity to particular alternatives; they do not establish the absence of training overlap or memorization\.

### C\.3Comparison with FlexiDepth allocations

TableA7:Agreement with FlexiDepth layer allocations, with bootstrap SDs\. Each split has 100 responses; the first word is excluded\.The released FlexiPatterns records comprise 600 generations and 82,783 labeled native tokens\. We align layer allocations and panel indices to words and omit the first word of each response because it includes prefill effects\. The pooled analysis subtracts each task split’s mean from allocation and panel index, then computes partial rank correlation controlling log position and word length\. Within\-split estimates use the same controls\. This centering procedure removes split means before the rank transformation\. The pooled partial correlations are 0\.209 for Qwen and 0\.213 for OLMo\. On the 68 verified addition outputs, the Qwen correlation is0\.2730\.273\. Multiplication provides an allocation comparison on incorrect generations, with no verified subset\.

## Appendix DIntervention protocols and full results

Evaluation population and conditions\.The handover experiments use the 1,758 core problems with verified references from both generators \(1,275 GSM8K, 347 MATH\-500, and 136 HumanEval\)\. The supplied prefix is taken from the Qwen reference\. A smaller model and the 72B control start from the same prefix; subsequent tokens are generated without additional reference tokens\. Accuracy differences are paired by problem after averaging 16 completions per condition\. The length sweep uses 0\.5B, 1\.5B, and 7B atL∈\{1,2,4,8,16,32,64\}L\\in\\\{1,2,4,8,16,32,64\\\}from the reference midpoint\. Full\-suffix handovers additionally include 32B and varyτ∈\{0,0\.25,0\.5,0\.75,0\.9\}\\tau\\in\\\{0,0\.25,0\.5,0\.75,0\.9\\\}\.

Prefix construction and pairing\.For a reference withCCcharacters, a handover at fractionτ\\taustarts at⌊τ​C⌋\\lfloor\\tau C\\rfloor, snapped backward to whitespace\. The generated text is appended to this supplied prefix before verification\. All models receive the same problem and prefix within a comparison\. Each condition has 16 completions at temperature 0\.7 and top\-pp0\.95\. Seeds are deterministic functions of item and position for full\-suffix handovers, or item and seed index for the length sweep\. Reusing seeds organizes paired conditions; it does not couple model outputs token by token\.

The full\-suffix analysis contains 35,156 paired item/size/start cells\. Four of the nominal1,758×4×51,758\\times 4\\times 5cells collapse because snapped start positions coincide\. The pooled size ordering has zero inversions among 15 adjacent comparisons\. The starting\-position ordering has two inversions among 16 comparisons\. Table[A9](https://arxiv.org/html/2610.02491#A4.T9)supplies all dataset\-specific estimates\.

Length\-sweep control\.The smaller model emits at mostLLnative tokens, possibly stopping earlier\. Its sampled text is appended to the verified midpoint prefix; 72B then receives that combined text in a new request and continues\. The control is a single uninterrupted 72B request from the midpoint\. Both 72B requests have the usual dataset output cap \(512 for GSM8K and HumanEval, 1,024 for MATH\-500\)\. Thus the intervention can generate up toLLadditional tokens before the 72B continuation and resets the sampler at the handback\. There are1,758×7×3×16=590,6881,758\\times 7\\times 3\\times 16=590,688small\-model completion conditions, each compared with its item/length control\.

TableA8:Handover\-length accuracy differences \(percentage points\), with bootstrap SDs \(standard errors\)\.TableA9:Full\-suffix handover accuracy differences \(percentage points\)\. Prefix fractions use characters, snapped to a word boundary\.Reference cost and continuation loss\.Across the same 1,758 items, the fraction of suffix words assigned to the 14B rung or higher has Spearman correlation 0\.195 with the midpoint 0\.5B accuracy loss\. This aggregate reference\-cost statistic provides only a weak item\-level predictor of the outcome of the handover\. A handover changes the generating policy and hence subsequent prefix states, so its effect includes deviations, alternative solution paths, and recovery\. This comparison separates the local matching measurement from the downstream consequences of choosing a policy for a longer continuation\.

## Appendix EAllocation protocols and complete policy comparisons

This appendix specifies the full\-dataset routing and drafting experiments in Figure[5](https://arxiv.org/html/2610.02491#S4.F5), including the timing projection\.

### E\.1Model routing and speculative decoding on MATH\-500

The allocation experiment in Figure[5](https://arxiv.org/html/2610.02491#S4.F5)evaluates ten policies on all 500 MATH\-500 problems\. Table[A10](https://arxiv.org/html/2610.02491#A5.T10)reports every point in that figure and additionally logical FLOPs\. These policies use Qwen2\.5\-0\.5B\-Instruct as the small model and Qwen2\.5\-72B\-Instruct as the target, with greedy decoding and a common 1,024\-token output cap\. Each policy’s answer is checked independently using the same symbolic\-equivalence checker\. No problem is excluded because the target or another policy gives an incorrect answer\.

TableA10:All ten allocation policies on the full MATH\-500 evaluation set \(500 problems\)\. Accuracy is independently checked for each generated response\. Projected time includes prompt processing and cached model calls under Appendix[E\.2](https://arxiv.org/html/2610.02491#A5.SS2); it excludes oracle\-map acquisition and is not measured deployment latency\. Values are means±\\pmone item\-bootstrap SD\. Confidence thresholds control small\-model selection; adaptive drafting stops below probability 0\.4 with a 16\-token cap\. MoA\-map policies use advance reference\-token agreement\.#### Information supplied to the oracle\.

For each problem, a fresh greedy target response provides a referencez=\(z1,…,zT\)z=\(z\_\{1\},\\ldots,z\_\{T\}\), including its recorded termination token where present\. The map stores

bt=\[argmaxvπ0\.5​B\(v∣x,z<t\)=zt\]\.b\_\{t\}=\\mathbf\{1\}\\\!\\left\[\\arg\\max\_\{v\}\\pi\_\{0\.5\\mathrm\{B\}\}\(v\\mid x,z\_\{<t\}\)=z\_\{t\}\\right\]\.\(17\)This is the small\-model matching component of the sufficient\-compute measurement\. The target response is retained regardless of correctness, so the map describes agreement with that response rather than a guarantee of a correct answer\. The controller receives the matching indicators\. Reference generation and map acquisition are supplied information and are excluded from policy resource counts\. Thus the experiment isolates the allocation opportunity available with advance knowledge of agreement\.

#### Routing policies\.

The confidence baseline implements the TokenCascade Chow rule\[[42](https://arxiv.org/html/2610.02491#bib.bib60)\]\. At the current generated prefixss, it uses the small model whenmaxv⁡π0\.5​B​\(v∣s\)≥θ\\max\_\{v\}\\pi\_\{0\.5\\mathrm\{B\}\}\(v\\mid s\)\\geq\\thetaand otherwise requests a target prediction\. We evaluate all four thresholdsθ∈\{0\.5,0\.8,0\.95,0\.99\}\\theta\\in\\\{0\.5,0\.8,0\.95,0\.99\\\}, together with standalone small\- and target\-model baselines\. MoA\-map routing selects the small model at mapped positions withbt=1b\_\{t\}=1\. Before emitting that prediction, it checks that the actual small\-model token equals the stored reference token; otherwise it requests the target\. After the generated prefix departs from the reference, it uses the target\. All routing policies use the same batch scheduler, which collects deferred target requests after at most 16 small\-model steps\. The target processes any missing prefix positions when it is next invoked\.

#### Speculative policies\.

The fixed baseline proposes up to 16 small\-model tokens before target verification\. The adaptive baseline follows confidence\-based dynamic lookahead\[[39](https://arxiv.org/html/2610.02491#bib.bib61)\]: it appends each proposed token and stops the block if that token’s probability is below 0\.4, a termination token is reached, or 16 tokens have been proposed\. MoA\-map drafting uses the length of the upcoming consecutive run ofbt=1b\_\{t\}=1, capped at 16 and by the remaining output allowance\. A zero\-length run invokes the target directly\. After departure from the mapped prefix, it uses the adaptive baseline\. For every policy, the target verifies a proposed block through greedy token equality, accepts its matching prefix, and supplies the first correction token or a bonus token when the full block matches\. Both caches are truncated to the resulting accepted prefix\.

#### Execution and resource accounting\.

The live allocation runs use a common PyTorch 2\.9\.1 research backend with bfloat16 weights, math SDPA attention, and float32 vocabulary projection\. The target is distributed over four A100 80GB GPUs; the draft shares one of these GPUs\. Batches contain at most 32 problems, and a forward call processes at most 128 new positions\. This backend differs from the vLLM reference\-generation pipeline used for the reproduction study\.

The logical FLOP ledger applies the architecture terms in Appendix[B\.1](https://arxiv.org/html/2610.02491#A2.SS1)to actual forwarded positions and requested logits\. It includes prompt prefill, cache catch\-up, rejected drafts, and target verification\. The ledger does not measures memory traffic or communication\.

#### Complete comparisons and their interpretation\.

Against confidence routing atθ=0\.99\\theta=0\.99, MoA\-map routing increases accuracy from 68\.4% to 69\.6% and reduces projected time from 7\.59 to 5\.12 seconds\. MoA\-map drafting proposes 301,691 tokens across the dataset, compared with 447,731 for the fixed\-window policy, a 32\.6% reduction\. Accepted draft\-token counts are similar: 268,577 and 269,286\. The map therefore mainly reduces proposals that do not survive verification\. The mean number of target calls is also similar, 57\.62 versus 56\.87 per response\. Projected times are 3\.99 versus 4\.98 seconds\. Table[A11](https://arxiv.org/html/2610.02491#A5.T11)gives paired comparisons for every tested routing threshold and both speculative baselines\.

TableA11:Paired differences between each MoA\-map policy and the indicated baseline on the same 500 problems \(map minus baseline\)\. Positive accuracy differences favor the map; negative time differences favor the map\. Accuracy is in percentage points \(pp\), and time differences are in seconds\. Values show the mean difference±\\pmone item\-bootstrap SD; projected time follows Appendix[E\.2](https://arxiv.org/html/2610.02491#A5.SS2)\.
#### Output agreement and uncertainty\.

Finite\-precision block and incremental evaluations need not produce identical greedy predictions\. Map fallback occurs on 100 routing responses and 169 speculative responses\. All departures and length\-capped responses remain in the accuracy and resource totals\. Consequently, small accuracy differences should be read as empirical outcomes of this backend, rather than evidence that changing speculative draft length intrinsically improves an exactly preserved target policy\.

Means and paired differences use 2,000 bootstrap resamples of the 500 problem identifiers\. Tables report one bootstrap SD, which describes sampling variation across problems rather than repeated timing runs\. Projection uncertainty from the timing assumptions is examined separately in Appendix[E\.2](https://arxiv.org/html/2610.02491#A5.SS2)\.

### E\.2Projection from published inference benchmarks

We separate the measured answer accuracies from an illustrative latency projection\. Accuracies use all 500 MATH\-500 problems generated with our experimental backend\. The horizontal coordinates instead describe a warm, single\-request execution scenario anchored to the official Qwen2\.5 BF16 benchmarks for vLLM 0\.6\.3 on A100 80GB GPUs: four GPUs for the 72B target and one for the 0\.5B model\[[47](https://arxiv.org/html/2610.02491#bib.bib62)\]\. We assume both models remain resident and execute sequentially without interference\. The published one\-token\-input rates are 31\.37 and 311\.55 output tokens per second, respectively, for 2,048\-token outputs\. These rates measure whole requests, rather than isolated cached forwards\.

For each modelmm, letKmK\_\{m\}denote the number of recorded forward calls andFmF\_\{m\}the total number of input positions processed, including prompt tokens, cache catch\-up, and rejected speculative tokens\. We use

T^=∑m∈\{small,target\}\[am​Km\+bm​\(Fm−Km\)\]\.\\widehat\{T\}=\\sum\_\{m\\in\\\{\\mathrm\{small\},\\mathrm\{target\}\\\}\}\\left\[a\_\{m\}K\_\{m\}\+b\_\{m\}\(F\_\{m\}\-K\_\{m\}\)\\right\]\.\(18\)A single\-position call costsama\_\{m\}; processingnnpositions together costsam\+bm​\(n−1\)a\_\{m\}\+b\_\{m\}\(n\-1\)\. Thus, parallel verification need not cost as much asnnsequential target calls\. Prompt processing is included in the same ledger, so it is not added a second time\. The projection retains the recorded call counts, including chunked processing, and is computed separately for every response before averaging across items\.

We anchorama\_\{m\}at the inverse of the shortest\-input published generation rate\. To setbmb\_\{m\}, write the published total time at input lengthPPasTm​\(P\)=2048/vm​\(P\)T\_\{m\}\(P\)=2048/v\_\{m\}\(P\)and fit

bm=arg⁡minb≥0⁡∑P∈\{6144,14336,30720\}\[Tm​\(P\)−2048​am−b⁡\(P−1\)\]2\.b\_\{m\}=\\mathop\{\\arg\\min\}\_\{b\\geq 0\}\\sum\_\{P\\in\\\{6144,14336,30720\\\}\}\\left\[T\_\{m\}\(P\)\-2048a\_\{m\}\-b\(P\-1\)\\right\]^\{2\}\.\(19\)This gives\(am,bm\)=\(3\.210,0\.03683\)\(a\_\{m\},b\_\{m\}\)=\(3\.210,0\.03683\)milliseconds for 0\.5B and\(31\.878,0\.28557\)\(31\.878,0\.28557\)milliseconds for 72B\. The fit reproduces these published whole\-request times with a maximum relative error of 2\.13%\. The fit describes the published whole\-request measurements; cached block\-verification timings are not separately calibrated\.

The substantive assumption is transferring the marginal cost of a longer input to extra positions within a cached forward call\. Longer\-input timings also include increased attention cost during subsequent generation; therefore, the public benchmark does not separately identify prefill, cache catch\-up, or verification costs\. The projection does not model their kernel differences, context\-dependent nonlinearities, model\-switching costs, or additional controller overhead\. It also excludes queueing, model loading, network delivery and oracle\-map acquisition\. Consequently, these coordinates are conditional model\-execution estimates, not measured end\-to\-end deployment latencies or a demonstrated vLLM speedup\. The answer accuracies remain those of the original backend; identical outputs after migration to another engine have not been established\.

We assess sensitivity using bulk\-position coefficients scaled by\{0,0\.5,1,2\}\\\{0,0\.5,1,2\\\}, small\-model times scaled by\{0\.5,1,2\}\\\{0\.5,1,2\\\}, and an added cost of\{0,0\.5,1\}\\\{0,0\.5,1\\\}target\-call equivalents per speculative verification\. All 36 combinations are applied to the full dataset, without selecting parameters according to policy ranking\. Within these scenarios, MoA\-map drafting has lower projected latency than fixed\-window and adaptive drafting; its relative reductions range from 8\.3–24\.7% and 7\.3–18\.4%, respectively\. These ranges describe chosen assumptions, not statistical intervals or guaranteed prediction bounds\. In particular, they do not cover arbitrary policy\-specific controller overhead or unobserved block cost nonlinearities\.

## Appendix FRelation to earlier measurements

Sequence prediction has long been formulated through policies and future rewards\[[48](https://arxiv.org/html/2610.02491#bib.bib46),[5](https://arxiv.org/html/2610.02491#bib.bib47)\]\. Controlled Decoding and TransferQ∗Q^\{\*\}use this formulation for decoding\-time alignment\[[41](https://arxiv.org/html/2610.02491#bib.bib48),[9](https://arxiv.org/html/2610.02491#bib.bib49)\]\. COLLAB selects off\-the\-shelf model policies in a shared decoding MDP using long\-term utility\[[8](https://arxiv.org/html/2610.02491#bib.bib37)\]; CITER formulates small/large\-model token routing as an MDP\[[62](https://arxiv.org/html/2610.02491#bib.bib34)\]\. Our measurement assigns a cost to a specified next\-token action at a reference state\. The handover and allocation experiments then examine the consequences of letting selected agents generate the subsequent states\. This connects the measurement to the policy\-switching problem while keeping local reproduction and terminal task reward distinct\.

Several prior methods expose token\-dependent computational behavior through a particular inference system\. Duo\-LLM searches internal module allocations to minimize perplexity under a compute budget\[[2](https://arxiv.org/html/2610.02491#bib.bib9)\]\. FS\-GEN examines disagreement and collaboration frequencies for small and large models\[[61](https://arxiv.org/html/2610.02491#bib.bib10)\]\. FlexiDepth records a learned layer allocation policy within Llama\-3\-8B\[[37](https://arxiv.org/html/2610.02491#bib.bib6)\]\. Our contribution is the cross\-family measurement design: independently score a shared, checker\-passing trajectory across several model sizes, compare the resulting orderings, and test their relation to continuation outcomes\.

This design retains the exact\-reference criterion’s dependence on wording\. Judge Decoding demonstrates that valid continuations can be rejected by alignment\-based acceptance\[[3](https://arxiv.org/html/2610.02491#bib.bib3)\]\. Top\-five and probability thresholds change our matching rule but still evaluate a chosen reference\. The shared\-prefix experiment therefore measures reproducibility of a successful trajectory, while the handovers assess task correctness after the trajectory is allowed to change\.

Generation\-stage switching is an established design class\. Reasoning Relay studies the transfer of partial reasoning between models\[[36](https://arxiv.org/html/2610.02491#bib.bib29)\]; LLM Shepherding uses a larger model’s prefix to support a smaller model\[[17](https://arxiv.org/html/2610.02491#bib.bib30)\]; RelayGen selects segment\-level switch points for efficient inference\[[53](https://arxiv.org/html/2610.02491#bib.bib31)\]\. The handover experiments here vary capacity, starting position, and duration under a common task checker\. They characterize the relationship between conditional reproduction and sustained continuation, rather than introducing model switching itself\.