MACRO: Markov Chain Routing of Transformer Layers

# MACRO: Markov Chain Routing of Transformer Layers
Source: [https://arxiv.org/html/2608.05872](https://arxiv.org/html/2608.05872)
###### Abstract

Standard Large Language Models \(LLMs\) execute layers sequentially\. Dynamic layer routing, i\.e\. search for a different execution path through layers involving layer repetitions, skips and other moves, can improve performance\. Existing routing approaches often require updating model weights, running expensive search loops per test instance, or demand ground\-truth labels during inference\. In this work, we proposeMarkov Chain Routing of Transformer Layers \(MACRO\), a framework that learns task\-specific routes over LLM architectures without modifying underlying parameters\. MACRO models layer routing as a context\-dependent Markov policy conditioned on layer indices, computation budget phases, directional displacements, and operator context, supporting skip, repeat, and residual hidden\-state addition operations\. The Markov route distribution is updated via feedback on training data and decoded using a top\-kkViterbi algorithm to isolate high\-probability candidate programs\. We evaluate MACRO across diverse reasoning and knowledge benchmarks on multiple open\-weight LLMs\. MACRO achieves a\+5\.0%average accuracy improvement over the unrouted baselines, with largest gains on small models\. We outperform the best dynamic routing approach Dr\. LLM by\+7\.2%, while reducing route\-search time9\.4×\\times\(from 14\.8 to 1\.6 hours\)\. Our code is publicly available athttps://github\.com/Batorskq/MACRO\.

## Introduction

Standard Large Language Models \(LLMs\) operate on a rigid computational schedule, executing transformer blocks sequentially from the first layer to the last\. This approach ignores the potential for dynamic computation pathways\. Prior work has explored adaptive depth primarily to accelerate inference via early exits\(Teerapittayanon, McDanel, and Kung[2016](https://arxiv.org/html/2608.05872#bib.bib37); Xin et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib43); Zhou et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib46)\)or structured layer dropping\(Fan, Grave, and Joulin[2020](https://arxiv.org/html/2608.05872#bib.bib8); Elhoushi et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib7)\)\. However, actively restructuring the execution route of a frozen model toimprove reasoning accuracyon complex tasks remains a highly challenging and relatively unexplored frontier\. Figure[1](https://arxiv.org/html/2608.05872#Sx1.F1)illustrates what such a restructuring can buy: on GSM8K, rewinding Qwen3\-1\.7B from layer 7 back to layer 3 and replaying five blocks raises test accuracy from 43\.4% to 69\.5%, without changing a single weight\.

![Refer to caption](https://arxiv.org/html/2608.05872v1/x1.png)Figure 1:A singleMACROroute on Qwen3\-1\.7B for GSM8K\. The baseline executes the 28 transformer blocks once, front to back\. The searched route runsL0–L7, rewinds toL3, and then continues toL27, soL3–L7execute twice and 33 block calls are made in total\. The weights are frozen and only the execution order changes\.Recent efforts to achieve accuracy gains through layer routing highlight significant practical bottlenecks\. Dr\.LLM\(Heakl et al\.[2026](https://arxiv.org/html/2608.05872#bib.bib14)\)bypasses the need for test\-time labels by training lightweight routers, yet it necessitates an expensive offline Monte Carlo Tree Search \(MCTS\)\-driven data generation and training pipeline\. Furthermore, purely exhaustive approaches, such as Neuroanatomy\(Ng[2026](https://arxiv.org/html/2608.05872#bib.bib30)\), operate in a constrained search space and are computationally prohibitive for per\-task adaptation or search over more complex routes\.

To overcome these limitations, we introduceMarkov Chain Routing of Transformer Layers \(MACRO\)\. We formalize layer routing as a discrete, context\-dependent stochastic process\. We choose Markov chains\(Norris[1998](https://arxiv.org/html/2608.05872#bib.bib31)\)for the following reasons: First, unlike deep reinforcement learning or auxiliary neural routers, a Markov transition matrix is lightweight and requires no parameter updates or backpropagation through the LLM\. Second, it naturally mitigates overfitting on small validation datasets by structurally constraining the decision space\. Most importantly, modeling the routing as a Markov chain allows us to leverage exact, polynomial\-time sequence decoding \(the Viterbi algorithm\) instead of relying on stochastic sampling or computationally expensive tree searches at test time\.

MACRO operates through on a compact state and action space: The state space encompasses the current layer, the remaining computation budget, the incoming displacement and the operator context\. From any state, the policy can execute feasible actions, which include moving to a nearby layer, adding a previous hidden state \(drawing inspiration from recurrent\-depth behaviors\(Chen et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib3); Bae et al\.[2026](https://arxiv.org/html/2608.05872#bib.bib1)\)\), or rejoining the standard suffix\. Rather than employing per\-sample routing, the framework learns a single, task\-specific Markov transition distribution\. This global policy is iteratively updated using an estimation\-of\-distribution approach on a subset of training data, augmented by a replay buffer to ensure stable convergence\. Once the Markov chain is fully trained, exploration is completely halted\. At this final stage, we employ a top\-kkViterbi algorithm to deterministically extract the highest\-probability valid routing programs, selecting the absolute best\-performing route via a held\-out validation set for uniform test\-time deployment\.

In summary, the main contributions of this work are:

- •We propose MACRO, a parameter\-efficient framework for adaptive layer routing in frozen and quantized LLMs, modeled as a context\-dependent Markov policy\.
- •We introduce an exact top\-kkViterbi decoding scheme equipped with structural feasibility masks, which eliminates the need for per\-sample test\-time search, auxiliary neural network training, or ground\-truth inference oracles\.
- •We demonstrate empirically that MACRO consistently outperforms prior dynamic routing methods across diverse reasoning and knowledge benchmarks, yielding a\+5\.0%average accuracy improvement over the non\-routed baseline,\+7\.2%over Dr\. LLM, and\+1\.5%over Dr\. LLM with an equivalent action space, while reducing the route\-search computational cost by9\.4×\\times\.

## Related Work

### Adaptive Depth and Layer Routing

Much prior work exploits redundancy in transformer depth mainly to make inference faster through early exits\(Teerapittayanon, McDanel, and Kung[2016](https://arxiv.org/html/2608.05872#bib.bib37); Xin et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib43); Liu et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib23); Zhou et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib46); Elbayad et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib6); Liu et al\.[2021a](https://arxiv.org/html/2608.05872#bib.bib24); Hou et al\.[2020](https://arxiv.org/html/2608.05872#bib.bib15); Schuster et al\.[2022](https://arxiv.org/html/2608.05872#bib.bib34); Chen et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib2); Jazbec et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib16)\), structured layer dropping\(Fan, Grave, and Joulin[2020](https://arxiv.org/html/2608.05872#bib.bib8); Liu et al\.[2021b](https://arxiv.org/html/2608.05872#bib.bib25); Men et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib27); Zhao et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib45)\), and LayerSkip\(Elhoushi et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib7)\)\. Mixture\-of\-Depths\(Raposo et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib33)\), adaptive layer skipping\(Luo, Wang, and Yan[2025](https://arxiv.org/html/2608.05872#bib.bib26)\), and router tuning\(He et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib13)\)similarly adapt which layers or tokens receive computation, while earlier dynamic networks learn input\-dependent execution paths\(Wu et al\.[2018](https://arxiv.org/html/2608.05872#bib.bib42); Wang et al\.[2018](https://arxiv.org/html/2608.05872#bib.bib38)\)\. Our objective differs: we use layer routing to improve accuracy while keeping the pretrained model fixed\.

The closest works are Dr\.LLM\(Heakl et al\.[2026](https://arxiv.org/html/2608.05872#bib.bib14)\), CoLA\(Li, Li, and Zhou[2025](https://arxiv.org/html/2608.05872#bib.bib21)\), and Neuroanatomy\(Ng[2026](https://arxiv.org/html/2608.05872#bib.bib30)\)\. Dr\.LLM trains lightweight routers from MCTS\-derived supervision, requiring route\-label generation and router training\. CoLA performs per\-sample MCTS over skipped and looped layers with ground\-truth reward, and Neuroanatomy exhaustively searches a limited route space\.MACROlearns one Markov route distribution per task from train/validation feedback and decodes it with top\-kkViterbi, avoiding per\-test ground\-truth search and any model\-weight updates\.

### Recurrent and Expert Routing

Universal Transformers\(Dehghani et al\.[2018](https://arxiv.org/html/2608.05872#bib.bib5)\), looped transformers\(Giannou et al\.[2023](https://arxiv.org/html/2608.05872#bib.bib12); Yang et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib44); Zhu et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib47)\), and recurrent\-depth models\(Chen et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib3); Geiping et al\.[2026](https://arxiv.org/html/2608.05872#bib.bib11); Bae et al\.[2026](https://arxiv.org/html/2608.05872#bib.bib1)\)show that repeated computation over depth can support algorithmic or reasoning behavior\. Mixture\-of\-experts models\(Shazeer et al\.[2017](https://arxiv.org/html/2608.05872#bib.bib35); Lepikhin et al\.[2021](https://arxiv.org/html/2608.05872#bib.bib19); Fedus, Zoph, and Shazeer[2022](https://arxiv.org/html/2608.05872#bib.bib9)\)and routing experts\(Wu et al\.[2025](https://arxiv.org/html/2608.05872#bib.bib41)\)route tokens across parameter experts; in contrast, our experts are the existing blocks of a single pretrained transformer\. The lightweight Markov policy\(Norris[1998](https://arxiv.org/html/2608.05872#bib.bib31)\), feasibility masks, and top\-kkViterbi decoding remain compatible with frozen and quantized LLMs\.

SoftRule

Table 1:Qualitative comparison of layer\-adaptation methods\. “Acc\.” denotes an explicit accuracy\-improvement objective; “No Labels” means no test labels are used; “No Per\-Ex\.” means no per\-example test\-time search is required; “No Pre\-Gen” means no offline route\-label generation stage is required\.

## Method

![Refer to caption](https://arxiv.org/html/2608.05872v1/x2.png)Figure 2:The left panel summarizes the complete workflow\.\(1\)Starting from a forward\-biased priorπ0\\pi\_\{0\}, each iteration samplesNNrouting programs, scores them by executing the frozen model on training data, and refits the policy using the top\-performing programs, weighted exponentially by training score, until convergence or the iteration limit\.\(2a\)Top\-KKViterbi decoding ranks feasible programs under the learned policy asP1≻P2≻⋯≻PKP\_\{1\}\\succ P\_\{2\}\\succ\\cdots\\succ P\_\{K\}\.\(2b\)Held\-out validation selects among the execution\-distinct candidates independently of their Viterbi rank\. Panel\(A\)expands the sampling, scoring, and refitting steps\. Panel\(B\)shows the masked categorical policy conditioned on layer, budget phase, incoming displacement, and operation provenance, over local LAYER, parameterized ADD, and deterministic REJOIN actions\.### Problem Statement

Consider a pretrained transformer with frozen weights, whose forward pass appliesLLlayer blocksf1,…,fLf\_\{1\},\\dots,f\_\{L\}in a fixed order\. A routerrrelaxes this schedule into a program over the blocks\. At each step, it chooses which block to apply, so blocks may be skipped, repeated, or revisited by going back to earlier ones\. The route may also add an earlier hidden state to the running hidden state before applying the next block\. Letℛ\\mathcal\{R\}be the space of such routes and letMrM\_\{r\}denote the model run under routerr\. Given a dataset𝒟=\{\(xi,yi\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, we seek the route that maximizes accuracy:

r⋆\\displaystyle r^\{\\star\}=arg​maxr∈ℛ⁡Acc⁡\(r\),\\displaystyle=\\operatorname\*\{arg\\,max\}\_\{r\\in\\mathcal\{R\}\}\\operatorname\{Acc\}\(r\),\(1\)Acc⁡\(r\)\\displaystyle\\operatorname\{Acc\}\(r\)=1N​∑i=1N𝟏​\{Mr​\(xi\)=yi\}\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{M\_\{r\}\(x\_\{i\}\)=y\_\{i\}\\\}\.

### Dataset Split

Given a benchmark dataset𝒟\\mathcal\{D\}, we partition it into three disjoint subsets: a training set𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\}, a validation set𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}, and a test set𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}:

𝒟=𝒟train∪𝒟val∪𝒟test\.\\mathcal\{D\}=\\mathcal\{D\}\_\{\\mathrm\{train\}\}\\cup\\mathcal\{D\}\_\{\\mathrm\{val\}\}\\cup\\mathcal\{D\}\_\{\\mathrm\{test\}\}\.The training and validation sets are used during routing search, whereas the test set is held out and used only for final evaluation\. Throughout this procedure, the model parameters remain fixed, and optimization is performed over the space of routings\. To reduce overfitting during search, we further partition the training set intoIIdisjoint subsets,

𝒟train=⋃i=1I𝒟i,\\mathcal\{D\}\_\{\\mathrm\{train\}\}=\\bigcup\_\{i=1\}^\{I\}\\mathcal\{D\}\_\{i\},which are used across search rounds to evaluate candidate routes on fresh subsets of training data\. At search iterationii, candidates are evaluated on the current subset together with a small replay buffer of examples from previous rounds\. We denote the buffer byℬ\\mathcal\{B\}, initialize it asℬ0=∅\\mathcal\{B\}\_\{0\}=\\emptyset, and use

𝒮i=𝒟i∪ℬi−1\\mathcal\{S\}\_\{i\}=\\mathcal\{D\}\_\{i\}\\cup\\mathcal\{B\}\_\{i\-1\}as the route\-evaluation set\. After each iteration, the buffer is updated by adding a randomly sampled fractionλrep\\lambda\_\{\\mathrm\{rep\}\}of the current subset:

ℬi=ℬi−1∪Sampleλrep​\(𝒟i\)\.\\mathcal\{B\}\_\{i\}=\\mathcal\{B\}\_\{i\-1\}\\cup\\mathrm\{Sample\}\_\{\\lambda\_\{\\mathrm\{rep\}\}\}\(\\mathcal\{D\}\_\{i\}\)\.This schedule keeps each round focused mainly on fresh data while preserving limited coverage of examples seen in earlier iterations\.

### Markov Route Generator

We model route construction as a context\-dependent Markov policyπθ​\(at∣st\)\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\. A route is represented as a typed program that starts at the first transformer layer and is extended one action at a time until a terminalRejoinaction is selected\. The statests\_\{t\}is a compact summary of the partial program, including the current layer, the current length or budget phase, and optional context such as the previous layer, the incoming layer displacement, and whether the current hidden state was produced by a plain layer or by an add operation\. At each step, infeasible actions are masked so that the program stays within layer bounds and the maximum route length\.

### State Space

For a partial program, the state factorizes asst=\(ℓ,ϕ,δ,o\)s\_\{t\}=\(\\ell,\\phi,\\delta,o\), and the next\-action distribution is conditioned on all four components\.

1. 1\.The*current layer*ℓ∈\{0,…,L−1\}\\ell\\in\\\{0,\\dots,L\-1\\\}is the most recently produced layer in the partial program\.
2. 2\.The*budget phase*ϕ∈\{early,mid,late\}\\phi\\in\\\{\\textsc\{early\},\\textsc\{mid\},\\textsc\{late\}\\\}buckets the fraction of remaining route\-length budget into three bins, biasing the policy towardRejoinas the route nears its maximum length\.
3. 3\.The*incoming displacement*δ=ℓ−ℓprev∈\{−r,…,r\}\\delta=\\ell\-\\ell\_\{\\mathrm\{prev\}\}\\in\\\{\-r,\\dots,r\\\}, together with a dedicated start symbol, records how the current layer was reached\.
4. 4\.The binary*op\-context*o∈\{0,1\}o\\in\\\{0,1\\\}indicates whether the current hidden statehth\_\{t\}was produced by a plain layer or by an add operation\.

A single global policy is shared across these states \(no input\-dependent context\), giving a state space of size\|𝒮\|=L⋅3⋅\(2​r\+2\)⋅2\|\\mathcal\{S\}\|=L\\cdot 3\\cdot\(2r\{\+\}2\)\\cdot 2\.

### Action Space

At current layerℓ\\ell, the policy chooses from three action families:

1. 1\.Local layer move:apply layerℓ\+Δ\\ell\+\\Delta, whereΔ∈\{−r,…,r\}\\Delta\\in\\\{\-r,\\ldots,r\\\}and the destination layer is valid\. This allows local skips, repeats, and backward revisits while biasing the search toward small perturbations of the standard forward pass\.
2. 2\.Add\-and\-apply:add a previous hidden stateht−bh\_\{t\-b\}to the current hidden statehth\_\{t\}, usinght\+γ​ht−bh\_\{t\}\+\\gamma h\_\{t\-b\}withγ\>0\\gamma\>0, and then apply a local destination layerℓ\+Δ\\ell\+\\Delta\.
3. 3\.Rejoin:terminate the learned part of the route and execute the remaining standard suffix of transformer layers\.

At each routing step, we apply a feasibility mask that removes actions which would produce an invalid program, such as moving outside the layer range, exceeding the route\-length budget, or adding a hidden state that has not yet been produced\. The Markov policy is then renormalized over the remaining feasible actions, so sampling and Viterbi decoding always operate over valid routes\. During decoding, a route is executed as an ordered sequence of frozen block applications, and each program step gets its own attention cache slot\. Thus a revisited blockfℓf\_\{\\ell\}keeps a separate key/value history for each application, so a route ofTTapplied steps has the cache and decoding cost of aTT\-block network with tied weights\.

### Policy Initialization

The initial Markov policy is a forward\-biased prior rather than a uniform distribution over routes\. We make the baseline route most probable because accuracy\-improving programs are expected to be slight modifications of the standard forward pass rather than entirely random layer programs\. The route starts at the first layer and the final layer must terminate by rejoining the standard suffix:

ρ​\(ℓ\)=𝟏​\{ℓ=0\},πθ0​\(Rejoin∣s\)=1if​ℓ=L−1\.\\rho\(\\ell\)=\\mathbf\{1\}\\\{\\ell=0\\\},\\qquad\\pi\_\{\\theta\_\{0\}\}\(\\textsc\{Rejoin\}\\mid s\)=1\\quad\\text\{if \}\\ell=L\{\-\}1\.For any earlier layer, leta→a\_\{\\rightarrow\}be the standard forward moveℓ↦ℓ\+1\\ell\\mapsto\\ell\+1, let𝒜loc\\mathcal\{A\}\_\{\\mathrm\{loc\}\}be the local layer moves plusRejoin, and let𝒜op\\mathcal\{A\}\_\{\\mathrm\{op\}\}be the add\-and\-apply actions\. Before feasibility masking, we set

πθ0​\(a∣s\)=\\displaystyle\\pi\_\{\\theta\_\{0\}\}\(a\\mid s\)=\{\}\(1−εloc−εop\)​𝟏​\{a=a→\}\\displaystyle\(1\-\\varepsilon\_\{\\mathrm\{loc\}\}\-\\varepsilon\_\{\\mathrm\{op\}\}\)\\mathbf\{1\}\\\{a=a\_\{\\rightarrow\}\\\}\+εloc\|𝒜loc\|​𝟏​\{a∈𝒜loc\}\\displaystyle\+\\frac\{\\varepsilon\_\{\\mathrm\{loc\}\}\}\{\|\\mathcal\{A\}\_\{\\mathrm\{loc\}\}\|\}\\mathbf\{1\}\\\{a\\in\\mathcal\{A\}\_\{\\mathrm\{loc\}\}\\\}\+εop\|𝒜op\|​𝟏​\{a∈𝒜op\}\.\\displaystyle\+\\frac\{\\varepsilon\_\{\\mathrm\{op\}\}\}\{\|\\mathcal\{A\}\_\{\\mathrm\{op\}\}\|\}\\mathbf\{1\}\\\{a\\in\\mathcal\{A\}\_\{\\mathrm\{op\}\}\\\}\.Thusa→a\_\{\\rightarrow\}receives the large remaining mass, while all local deviations and add actions receive small exploration mass\. The most likely route underπθ0\\pi\_\{\\theta\_\{0\}\}is therefore the unmodified network,⟨0,1,…,L−1⟩\\langle 0,1,\\dots,L\{\-\}1\\rangle\. This prior is also the default for states that the search never visits: since the update uses pseudo\-countsα​πθi\\alpha\\pi\_\{\\theta\_\{i\}\}, the empirical rowp^i\\hat\{p\}\_\{i\}and weighted visit countmi​\(s\)m\_\{i\}\(s\)give

π^i\+1\(⋅∣s\)=αα\+mi​\(s\)πθi\(⋅∣s\)\+mi​\(s\)α\+mi​\(s\)p^i\(⋅∣s\)\.\\hat\{\\pi\}\_\{i\+1\}\(\\cdot\\mid s\)=\\frac\{\\alpha\}\{\\alpha\+m\_\{i\}\(s\)\}\\pi\_\{\\theta\_\{i\}\}\(\\cdot\\mid s\)\+\\frac\{m\_\{i\}\(s\)\}\{\\alpha\+m\_\{i\}\(s\)\}\\hat\{p\}\_\{i\}\(\\cdot\\mid s\)\.Unvisited states havemi​\(s\)=0m\_\{i\}\(s\)=0and keep the initialization exactly; visited states move away from it only in proportion to the evidence collected for that state\.

### Update Rule

We update the Markov policy with an estimation\-of\-distribution procedure\. At iterationii, we sampleKKcandidate programs\{rj\}j=1K\\\{r\_\{j\}\\\}\_\{j=1\}^\{K\}from the current policyπθi\\pi\_\{\\theta\_\{i\}\}, apply the feasibility mask at each step, and evaluate each candidate on𝒮i\\mathcal\{S\}\_\{i\}:

qj=Acc𝒮i⁡\(rj\)\.q\_\{j\}=\\operatorname\{Acc\}\_\{\\mathcal\{S\}\_\{i\}\}\(r\_\{j\}\)\.The scores are converted into normalized update weights\. Optionally, only the topmmcandidates are kept as an elite setEiE\_\{i\}; otherwiseEi=\{1,…,K\}E\_\{i\}=\\\{1,\\ldots,K\\\}:

wj=\{exp⁡\(β​\(qj−maxk∈Ei⁡qk\)\)∑k∈Eiexp⁡\(β​\(qk−maxℓ∈Ei⁡qℓ\)\)if​j∈Ei,0otherwise\.w\_\{j\}=\\begin\{cases\}\\dfrac\{\\exp\(\\beta\(q\_\{j\}\-\\max\_\{k\\in E\_\{i\}\}q\_\{k\}\)\)\}\{\\sum\_\{k\\in E\_\{i\}\}\\exp\(\\beta\(q\_\{k\}\-\\max\_\{\\ell\\in E\_\{i\}\}q\_\{\\ell\}\)\)\}&\\text\{if \}j\\in E\_\{i\},\\\\\[11\.99998pt\] 0&\\text\{otherwise\.\}\\end\{cases\}Hereβ\\betacontrols the sharpness of selection\.

LetNi​\(s,a\)N\_\{i\}\(s,a\)denote the weighted transition count for taking actionaafrom statess\. We initialize the counts with a smoothed copy of the current transition table and then add the weighted transitions observed in the sampled programs:

Ni​\(s,a\)=α​πθi​\(a∣s\)\+∑j=1Kwj​∑t𝟏​\{sj,t=s,aj,t=a\}\.N\_\{i\}\(s,a\)=\\alpha\\,\\pi\_\{\\theta\_\{i\}\}\(a\\mid s\)\+\\sum\_\{j=1\}^\{K\}w\_\{j\}\\sum\_\{t\}\\mathbf\{1\}\\\{s\_\{j,t\}=s,\\ a\_\{j,t\}=a\\\}\.The new policy is the normalized weighted maximum\-likelihood estimate,

π^i\+1​\(a∣s\)=Ni​\(s,a\)∑a′Ni​\(s,a′\)\.\\hat\{\\pi\}\_\{i\+1\}\(a\\mid s\)=\\frac\{N\_\{i\}\(s,a\)\}\{\\sum\_\{a^\{\\prime\}\}N\_\{i\}\(s,a^\{\\prime\}\)\}\.In the default update we setπθi\+1=π^i\+1\\pi\_\{\\theta\_\{i\+1\}\}=\\hat\{\\pi\}\_\{i\+1\}\. For a smoother update, we optionally use an exponential moving average,

πθi\+1=\(1−η\)​πθi\+η​π^i\+1,\\pi\_\{\\theta\_\{i\+1\}\}=\(1\-\\eta\)\\pi\_\{\\theta\_\{i\}\}\+\\eta\\hat\{\\pi\}\_\{i\+1\},followed by row\-wise normalization\. This update increases the probability of high\-scoring route programs while retaining probability mass on nearby alternatives through smoothing and feasibility\-masked sampling\.

### Final Selection

After the last update, the learned Markov policy defines a distribution over valid route programs\. Rather than selecting the best route seen during sampling, we decode the most likely programs under the final policy using a top\-kkViterbi procedure\. For a partial statess, let𝒜​\(s\)\\mathcal\{A\}\(s\)be the set of feasible actions after applying the length and layer\-bound masks\. The score of a complete programr=\(a1,…,aT\)r=\(a\_\{1\},\\ldots,a\_\{T\}\)is its log\-probability under the masked policy,

log⁡πθ​\(r\)=∑t=1Tlog⁡πθ​\(at∣st\)\.\\log\\pi\_\{\\theta\}\(r\)=\\sum\_\{t=1\}^\{T\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\.The Viterbi recursion keeps the topkksuffixes from each state:

Vk​\(s\)=TopKa∈𝒜​\(s\)⁡\[log⁡πθ​\(a∣s\)\+Vk​\(T​\(s,a\)\)\],V\_\{k\}\(s\)=\\operatorname\{TopK\}\_\{a\\in\\mathcal\{A\}\(s\)\}\\left\[\\log\\pi\_\{\\theta\}\(a\\mid s\)\+V\_\{k\}\(T\(s,a\)\)\\right\],whereT​\(s,a\)T\(s,a\)is the next state after actionaa\. IfaaisRejoin, the recursion terminates by appending the standard remaining suffix\.

This produces a small candidate set𝒞vit\\mathcal\{C\}\_\{\\mathrm\{vit\}\}of high probability valid programs\. We then evaluate these candidates on the validation set and select

r^=arg​maxr∈𝒞vit⁡Acc𝒟val⁡\(r\),\\hat\{r\}=\\operatorname\*\{arg\\,max\}\_\{r\\in\\mathcal\{C\}\_\{\\mathrm\{vit\}\}\}\\operatorname\{Acc\}\_\{\\mathcal\{D\}\_\{\\mathrm\{val\}\}\}\(r\),breaking ties by higher training accuracy and then by shorter route length\. The test set is used only once, to report the final accuracy of the selected router^\\hat\{r\}\.

## Experiments

In our main experiments we will use the following baselines:

- •Baseline:The baseline follows the standard transformer execution path, applying every layer sequentially from the first layer to the last\.
- •Dr\.LLM\(Heakl et al\.[2026](https://arxiv.org/html/2608.05872#bib.bib14)\): Dr\.LLM trains lightweight per\-layer routers from MCTS\-derived route labels, providing a learned layer\-routing baseline\.
- •Dr\.LLM\+Ext\.:Dr\.LLM\+Ext\. uses the same Dr\.LLM\-style router training procedure, but with the extendedMACROaction space\.
- •MACRO\(top\-1\):MACRO\(top\-1\) deploys the single most probable route under the learned policy, that is the rank\-1 Viterbi candidate\.
- •MACRO\(no Vit\.\):MACRO\(no Vit\.\) removes decoding altogether and deploys the highest training\-accuracy route the search ever sampled\.
- •MACRO:MACROlearns a task\-specific Markov policy over feasible layer\-routing programs, decodes top\-kkcandidates with Viterbi, and selects a single validation\-best route for test\-time evaluation\.

Additional details on dataset splits, generation settings, route\-search hyperparameters, and baseline training are provided in Appendix[C](https://arxiv.org/html/2608.05872#A3)\.

### Main Results

We compareMACROagainst baselines on mathematical reasoningGSM8K\(Cobbe et al\.[2021](https://arxiv.org/html/2608.05872#bib.bib4)\)andMATH500\(Lightman et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib22)\), knowledge\-intensive question answeringOpenBookQA\(Mihaylov et al\.[2018](https://arxiv.org/html/2608.05872#bib.bib29)\),MedQA\(Jin et al\.[2021](https://arxiv.org/html/2608.05872#bib.bib17)\),SciQ\(Welbl, Liu, and Gardner[2017](https://arxiv.org/html/2608.05872#bib.bib40)\),MMLU\-Pro\(Wang et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib39)\),SVAMP\(Patel, Bhattamishra, and Goyal[2021](https://arxiv.org/html/2608.05872#bib.bib32)\),ASDiv\(Miao, Liang, and Su[2020](https://arxiv.org/html/2608.05872#bib.bib28)\),MAWPS\(Koncel\-Kedziorski et al\.[2016](https://arxiv.org/html/2608.05872#bib.bib18)\),GSM8K\-Hard\(Gao et al\.[2023](https://arxiv.org/html/2608.05872#bib.bib10)\),GSM8K\-Plus\(Li et al\.[2024](https://arxiv.org/html/2608.05872#bib.bib20)\),multistep\-arithmetic, andobject counting\(Suzgun et al\.[2023](https://arxiv.org/html/2608.05872#bib.bib36)\)\. We additionally report results on meta\-llama/Llama\-3\.2\-3B\-Instruct in Appendix[A](https://arxiv.org/html/2608.05872#A1)\. Table[Main Results](https://arxiv.org/html/2608.05872#Sx4.SS0.SSS0.Px1)shows thatMACROattains the best average accuracy on every model except the reasoning\-distilled one, and that its margin over the unrouted baseline is widest exactly where that baseline is weakest, narrowing to a smaller but consistent gain once the model is already strong\. Dr\. LLM in its original skip\-and\-repeat action space frequently ends up below the unrouted baseline\. Dr\. LLM\+Ext\. keeps the same routers, supervision, and label budget and differs only in adopting theMACROaction space, yet is substantially stronger on every model we test, indicating that the action space itself is a decisive part of what makes layer routing help rather than hurt\.

SoftRule

Table 2:Performance comparison across prompting and optimization methods\. Colored bars denote different model families\. Within each model and benchmark column, cells are shaded by rank across methods \(red: best, orange: second, yellow: third\); ties share a color\. Values with gray subscripts report mean and standard deviation over 3 seeds\.### MACROWithout Viterbi or Validation Selection

MACRO\(no Vit\.\) keeps the learned policy but discards decoding, deploying the highest training\-accuracy route the search happened to sample\.MACRO\(top\-1\) instead keeps decoding but discards validation selection, deploying the single most probable route under the final policy\. FullMACROis the strongest of the three in Table[Main Results](https://arxiv.org/html/2608.05872#Sx4.SS0.SSS0.Px1), best on four of the six models and on the average over them \. Both stages appear to contribute, and each replaces a lucky draw with a more principled choice: Viterbi reconstructs candidates from the transition table rather than trusting whichever routes the sampler happened to visit, and validation then ranks those candidates on held\-out data rather than on the training score the search has already optimized\.

### Statistical Significance

Seed replicates of one benchmark share a test set and a deterministic baseline run, so they are not independent observations\. We therefore first average the accuracy differenced=AccMACRO−Accbased=\\operatorname\{Acc\}\_\{\\textsc\{MACRO\}\}\-\\operatorname\{Acc\}\_\{\\mathrm\{base\}\}over seeds within each benchmark, and treat the1313resulting per\-benchmark differences as the units of analysis\. Per model we run a one\-sided Wilcoxon signed\-rank test ofH0H\_\{0\}:MACROdoes not improve over the baseline \(no positive shift indd\) againstH1H\_\{1\}:MACROimproves accuracy\. Table[3](https://arxiv.org/html/2608.05872#Sx4.T3)reports the mean improvementd¯\\bar\{d\}, which equals the per\-model average gain in Table[Main Results](https://arxiv.org/html/2608.05872#Sx4.SS0.SSS0.Px1), a 95% confidence interval from a cluster bootstrap over benchmarks \(10510^\{5\}resamples\), the counts of benchmarks improved/tied/degraded, andpp\-values Holm\-corrected across the six models\. Every model improves significantly, degrades on no benchmark, and excludes zero\.

SoftRule

Table 3:Statistical significance ofMACROover the baseline, with benchmarks as the unit of analysis\.### State\-Space Ablation

We ablate the context used by the Markov transition model on Qwen3\-1\.7B by removing one state factor at a time: the budget phase, the incoming layer delta, or the operator context\. The full state performs best overall, with the largest drop coming from removing the budget\-phase factor, indicating that routing decisions depend strongly on where the program sits in its computation budget\.

SoftRule

Table 4:State\-space ablation on Qwen3\-1\.7B\. Averages are over all the 13 benchmarks used in Table[Main Results](https://arxiv.org/html/2608.05872#Sx4.SS0.SSS0.Px1)\.### Hyperparameter Analysis

We sweep one search hyperparameter at a time around our configuration on Qwen3\-1\.7B: the selection sharpnessβ\\betaand the elite countmm\(Figure[3](https://arxiv.org/html/2608.05872#Sx4.F3)\)\. Accuracy is essentially flat inβ\\beta, since the elite set already applies a hard filter before the update and leaves the softer reweighting little to decide\. The elite count matters more, and our setting sits between two failure modes:m=1m=1fits the transition counts to a single trajectory and commits to it before the evidence supports doing so, making it the only setting significantly worse than ours, whilem=12m=12keeps too broad a range of candidates for the update to stay selective and dilutes the counts with noise\. Every setting still improves over the unrouted model, soMACROdoes not depend on a narrow hyperparameter choice\.

![Refer to caption](https://arxiv.org/html/2608.05872v1/x3.png)Figure 3:Accuracy on Qwen3\-1\.7B when sweeping the selection sharpnessβ\\beta\(left\) and the elite countmm\(right\) one at a time around our configuration \(shaded, diamond\)\.### Mechanistic Interpretability ofMACRO

To understand*how*a route improves accuracy without any weight update, we apply a logit lens: at each layer we read the running hidden state through the model’s own output projection and track the probability assigned to the correct answer token\. Figure[4](https://arxiv.org/html/2608.05872#Sx4.F4)shows that the standard forward pass can represent the answer internally and then suppress it near the output, whileMACROpreserves this signal through the final readout\. Because the model’s own generated reasoning is held fixed in context, this suggests that routing improves how an already\-computed answer is exposed rather than changing the reasoning itself\.

![Refer to caption](https://arxiv.org/html/2608.05872v1/x4.png)Figure 4:Logit\-lens log\-probability of the correct answer at each layer of Qwen3\-1\.7B for baseline route andMACROrouteSoftRule

Table 5:Results for the*one model = one route*experiment\. A single shared route is trained on the combined training pool from the eight in\-domain benchmarks and evaluated on both the held\-out test splits of those in\-domain benchmarks and the five out\-of\-domain benchmarks\.### One Model, One Route

To test whether a single shared route can transfer across tasks, we train one route policy on a combined training pool built from MAWPS, ASDiv, SVAMP, multistep\-arithmetic, object counting, MATH500, OpenBookQA, and SciQ\. We then evaluate the resulting route on the held\-out test splits of those eight in\-domain benchmarks and on the five out\-of\-domain benchmarks GSM8K, GSM8K\-Hard, GSM8K\-Plus, MMLU\-Pro, and MedQA\. Results are summarized in Table[5](https://arxiv.org/html/2608.05872#Sx4.T5)\. For each model, we report the mean accuracy of the baseline forward pass and the mean accuracy of the routed candidate selected by the shared route, along with the aggregate in\-domain and out\-of\-domain averages\. Accuracies for all benchmarks and representative non\-standard route strings for the two Qwen models are provided in Appendix[B](https://arxiv.org/html/2608.05872#A2)\.

![Refer to caption](https://arxiv.org/html/2608.05872v1/x5.png)Figure 5:Route\-transfer heatmap on Qwen3\-1\.7B, with red\-to\-green shading from lower to higher target accuracy\.### Route Transfer Across Benchmarks

To test whether routes capture reusable computation or only benchmark\-specific quirks, we evaluate each Qwen3\-1\.7B route, unchanged, on every held\-out target benchmark\. Only the route transfers: the prompts, scoring, data, and baseline always come from the target benchmark\. Figure[5](https://arxiv.org/html/2608.05872#Sx4.F5)shows this accuracy matrix, where entry\(A,B\)\(A,B\)is the route searched on benchmarkAAevaluated on targetBB\. The highest\-accuracy cells are mostly on the diagonal, which is expected because each route is optimized for its home benchmark\. Off\-diagonal transfer still appears, especially among related mathematical\-reasoning tasks, suggesting that some routes encode reusable model computations rather than only task\-specific effects\.

## Conclusion

We have introduced a lightweight dynamic routing method that outperforms the unrouted base model and Dr\.LLM, the leading dynamic routing method across a plethora of benchmarks and LLM models\. It is an interesting further research question why routing improves performance: Is it because routing improves capabilities or because an internally already present answer is better elicited? We argue that some evidence points towards the latter, including our limited mechanistic interpretability study\. We argue that routing should be taken much further: we want models that profit from much more extensive input\-depending dynamic routing, allowing for another dimension of flexible test\-time scaling\.

## References

- Bae et al\. \(2026\)Bae, S\.; Kim, Y\.; Bayat, R\.; Kim, S\.; Ha, J\.; Schuster, T\.; Fisch, A\.; Harutyunyan, H\.; Ji, Z\.; Courville, A\.; et al\. 2026\.Mixture\-of\-recursions: Learning dynamic recursive depths for adaptive token\-level computation\.*Advances in Neural Information Processing Systems*, 38: 96572–96617\.
- Chen et al\. \(2024\)Chen, Y\.; Pan, X\.; Li, Y\.; Ding, B\.; and Zhou, J\. 2024\.EE\-LLM: Large\-Scale Training and Inference of Early\-Exit Large Language Models with 3D Parallelism\.In*International Conference on Machine Learning*, 7163–7189\.
- Chen et al\. \(2025\)Chen, Y\.; Shang, J\.; Zhang, Z\.; Xie, Y\.; Sheng, J\.; Liu, T\.; Wang, S\.; Sun, Y\.; Wu, H\.; and Wang, H\. 2025\.Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 28241–28259\.
- Cobbe et al\. \(2021\)Cobbe, K\.; Kosaraju, V\.; Bavarian, M\.; Chen, M\.; Jun, H\.; Kaiser, L\.; Plappert, M\.; Tworek, J\.; Hilton, J\.; Nakano, R\.; et al\. 2021\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*\.
- Dehghani et al\. \(2018\)Dehghani, M\.; Gouws, S\.; Vinyals, O\.; Uszkoreit, J\.; and Kaiser, Ł\. 2018\.Universal transformers\.*arXiv preprint arXiv:1807\.03819*\.
- Elbayad et al\. \(2020\)Elbayad, M\.; Gu, J\.; Grave, E\.; and Auli, M\. 2020\.Depth\-Adaptive Transformer\.In*International Conference on Learning Representations*\.
- Elhoushi et al\. \(2024\)Elhoushi, M\.; Shrivastava, A\.; Liskovich, D\.; Hosmer, B\.; Wasti, B\.; Lai, L\.; Mahmoud, A\.; Acun, B\.; Agarwal, S\.; Roman, A\.; Aly, A\.; Chen, B\.; and Wu, C\.\-J\. 2024\.LayerSkip: Enabling Early Exit Inference and Self\-Speculative Decoding\.In Ku, L\.\-W\.; Martins, A\.; and Srikumar, V\., eds\.,*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 12622–12642\. Bangkok, Thailand: Association for Computational Linguistics\.
- Fan, Grave, and Joulin \(2020\)Fan, A\.; Grave, E\.; and Joulin, A\. 2020\.Reducing Transformer Depth on Demand with Structured Dropout\.In*International Conference on Learning Representations*\.
- Fedus, Zoph, and Shazeer \(2022\)Fedus, W\.; Zoph, B\.; and Shazeer, N\. 2022\.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity\.*Journal of Machine Learning Research*, 23\(120\): 1–39\.
- Gao et al\. \(2023\)Gao, L\.; Madaan, A\.; Zhou, S\.; Alon, U\.; Liu, P\.; Yang, Y\.; Callan, J\.; and Neubig, G\. 2023\.Pal: Program\-aided language models\.In*International conference on machine learning*, 10764–10799\. PMLR\.
- Geiping et al\. \(2026\)Geiping, J\.; McLeish, S\. M\.; Jain, N\.; Kirchenbauer, J\.; Singh, S\.; Bartoldson, B\. R\.; Kailkhura, B\.; Bhatele, A\.; and Goldstein, T\. 2026\.Scaling up Test\-Time Compute with Latent Reasoning: A Recurrent Depth Approach\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Giannou et al\. \(2023\)Giannou, A\.; Rajput, S\.; Sohn, J\.\-y\.; Lee, K\.; Lee, J\. D\.; and Papailiopoulos, D\. 2023\.Looped transformers as programmable computers\.In*International Conference on Machine Learning*, 11398–11442\. PMLR\.
- He et al\. \(2025\)He, S\.; Ge, T\.; Sun, G\.; Tian, B\.; Wang, X\.; and Yu, D\. 2025\.Router\-Tuning: A Simple and Effective Approach for Dynamic Depth\.In Christodoulopoulos, C\.; Chakraborty, T\.; Rose, C\.; and Peng, V\., eds\.,*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 1925–1938\. Suzhou, China: Association for Computational Linguistics\.ISBN 979\-8\-89176\-332\-6\.
- Heakl et al\. \(2026\)Heakl, A\.; Gubri, M\.; Khan, S\.; Yun, S\.; and Oh, S\. J\. 2026\.Dr\.LLM: Dynamic Layer Routing in LLMs\.In*The Fourteenth International Conference on Learning Representations*\.
- Hou et al\. \(2020\)Hou, L\.; Huang, Z\.; Shang, L\.; Jiang, X\.; Chen, X\.; and Liu, Q\. 2020\.DynaBERT: Dynamic BERT with Adaptive Width and Depth\.In*Advances in Neural Information Processing Systems*, volume 33, 9782–9793\.
- Jazbec et al\. \(2024\)Jazbec, M\.; Timans, A\.; Veljković, T\. H\.; Sakmann, K\.; Zhang, D\.; Naesseth, C\. A\.; and Nalisnick, E\. 2024\.Fast yet Safe: Early\-Exiting with Risk Control\.In*Advances in Neural Information Processing Systems*, volume 37, 129825–129854\.
- Jin et al\. \(2021\)Jin, D\.; Pan, E\.; Oufattole, N\.; Weng, W\.\-H\.; Fang, H\.; and Szolovits, P\. 2021\.What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.*Applied Sciences*, 11\(14\): 6421\.
- Koncel\-Kedziorski et al\. \(2016\)Koncel\-Kedziorski, R\.; Roy, S\.; Amini, A\.; Kushman, N\.; and Hajishirzi, H\. 2016\.MAWPS: A Math Word Problem Repository\.In Knight, K\.; Nenkova, A\.; and Rambow, O\., eds\.,*Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 1152–1157\. San Diego, California: Association for Computational Linguistics\.
- Lepikhin et al\. \(2021\)Lepikhin, D\.; Lee, H\.; Xu, Y\.; Chen, D\.; Firat, O\.; Huang, Y\.; Krikun, M\.; Shazeer, N\.; and Chen, Z\. 2021\.GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\.In*International Conference on Learning Representations*\.
- Li et al\. \(2024\)Li, Q\.; Cui, L\.; Zhao, X\.; Kong, L\.; and Bi, W\. 2024\.Gsm\-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2961–2984\.
- Li, Li, and Zhou \(2025\)Li, Z\.; Li, Y\.; and Zhou, T\. 2025\.Skip a Layer or Loop it? Test\-Time Depth Adaptation of Pretrained LLMs\.*arXiv preprint arXiv:2507\.07996*\.
- Lightman et al\. \(2024\)Lightman, H\.; Kosaraju, V\.; Burda, Y\.; Edwards, H\.; Baker, B\.; Lee, T\.; Leike, J\.; Schulman, J\.; Sutskever, I\.; and Cobbe, K\. 2024\.Let’s Verify Step by Step\.In*The Twelfth International Conference on Learning Representations*\.
- Liu et al\. \(2020\)Liu, W\.; Zhou, P\.; Wang, Z\.; Zhao, Z\.; Deng, H\.; and Ju, Q\. 2020\.FastBERT: a Self\-distilling BERT with Adaptive Inference Time\.In Jurafsky, D\.; Chai, J\.; Schluter, N\.; and Tetreault, J\., eds\.,*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 6035–6044\. Online: Association for Computational Linguistics\.
- Liu et al\. \(2021a\)Liu, Y\.; Meng, F\.; Zhou, J\.; Chen, Y\.; and Xu, J\. 2021a\.Faster depth\-adaptive transformers\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 35, 13424–13432\.
- Liu et al\. \(2021b\)Liu, Z\.; Li, F\.; Li, G\.; and Cheng, J\. 2021b\.EBERT: Efficient BERT Inference with Dynamic Structured Pruning\.In Zong, C\.; Xia, F\.; Li, W\.; and Navigli, R\., eds\.,*Findings of the Association for Computational Linguistics: ACL\-IJCNLP 2021*, 4814–4823\. Online: Association for Computational Linguistics\.
- Luo, Wang, and Yan \(2025\)Luo, X\.; Wang, W\.; and Yan, X\. 2025\.Adaptive Layer\-skipping in Pre\-trained LLMs\.In*Second Conference on Language Modeling*\.
- Men et al\. \(2025\)Men, X\.; Xu, M\.; Zhang, Q\.; Yuan, Q\.; Wang, B\.; Lin, H\.; Lu, Y\.; Han, X\.; and Chen, W\. 2025\.Shortgpt: Layers in large language models are more redundant than you expect\.In*Findings of the Association for Computational Linguistics: ACL 2025*, 20192–20204\.
- Miao, Liang, and Su \(2020\)Miao, S\.\-Y\.; Liang, C\.\-C\.; and Su, K\.\-Y\. 2020\.A diverse corpus for evaluating and developing English math word problem solvers\.In*Proceedings of the 58th annual meeting of the Association for Computational Linguistics*, 975–984\.
- Mihaylov et al\. \(2018\)Mihaylov, T\.; Clark, P\.; Khot, T\.; and Sabharwal, A\. 2018\.Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering\.In Riloff, E\.; Chiang, D\.; Hockenmaier, J\.; and Tsujii, J\., eds\.,*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, 2381–2391\. Brussels, Belgium: Association for Computational Linguistics\.
- Ng \(2026\)Ng, D\. N\. 2026\.LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight\.*Online*\.
- Norris \(1998\)Norris, J\. R\. 1998\.*Markov chains*\.2\. Cambridge university press\.
- Patel, Bhattamishra, and Goyal \(2021\)Patel, A\.; Bhattamishra, S\.; and Goyal, N\. 2021\.Are NLP models really able to solve simple math word problems?In*Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies*, 2080–2094\.
- Raposo et al\. \(2024\)Raposo, D\.; Ritter, S\.; Richards, B\.; Lillicrap, T\.; Humphreys, P\. C\.; and Santoro, A\. 2024\.Mixture\-of\-depths: Dynamically allocating compute in transformer\-based language models\.*arXiv preprint arXiv:2404\.02258*\.
- Schuster et al\. \(2022\)Schuster, T\.; Fisch, A\.; Gupta, J\.; Dehghani, M\.; Bahri, D\.; Tran, V\.; Tay, Y\.; and Metzler, D\. 2022\.Confident Adaptive Language Modeling\.In*Advances in Neural Information Processing Systems*, volume 35, 17456–17472\.
- Shazeer et al\. \(2017\)Shazeer, N\.; Mirhoseini, A\.; Maziarz, K\.; Davis, A\.; Le, Q\.; Hinton, G\.; and Dean, J\. 2017\.Outrageously Large Neural Networks: The Sparsely\-Gated Mixture\-of\-Experts Layer\.In*International Conference on Learning Representations*\.
- Suzgun et al\. \(2023\)Suzgun, M\.; Scales, N\.; Schärli, N\.; Gehrmann, S\.; Tay, Y\.; Chung, H\. W\.; Chowdhery, A\.; Le, Q\.; Chi, E\.; Zhou, D\.; et al\. 2023\.Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.In*Findings of the Association for Computational Linguistics: ACL 2023*, 13003–13051\.
- Teerapittayanon, McDanel, and Kung \(2016\)Teerapittayanon, S\.; McDanel, B\.; and Kung, H\.\-T\. 2016\.Branchynet: Fast inference via early exiting from deep neural networks\.In*2016 23rd international conference on pattern recognition \(ICPR\)*, 2464–2469\. IEEE\.
- Wang et al\. \(2018\)Wang, X\.; Yu, F\.; Dou, Z\.\-Y\.; Darrell, T\.; and Gonzalez, J\. E\. 2018\.SkipNet: Learning Dynamic Routing in Convolutional Networks\.In*Proceedings of the European Conference on Computer Vision*, 409–424\.
- Wang et al\. \(2024\)Wang, Y\.; Ma, X\.; Zhang, G\.; Ni, Y\.; Chandra, A\.; Guo, S\.; Ren, W\.; Arulraj, A\.; He, X\.; Jiang, Z\.; et al\. 2024\.Mmlu\-pro: A more robust and challenging multi\-task language understanding benchmark\.*Advances in Neural Information Processing Systems*, 37: 95266–95290\.
- Welbl, Liu, and Gardner \(2017\)Welbl, J\.; Liu, N\. F\.; and Gardner, M\. 2017\.Crowdsourcing multiple choice science questions\.In*Proceedings of the 3rd Workshop on Noisy User\-generated Text*, 94–106\.
- Wu et al\. \(2025\)Wu, Q\.; Ke, Z\.; Zhou, Y\.; Sun, X\.; and Ji, R\. 2025\.Routing Experts: Learning to Route Dynamic Experts in Existing Multi\-modal Large Language Models\.In*The Thirteenth International Conference on Learning Representations*\.
- Wu et al\. \(2018\)Wu, Z\.; Nagarajan, T\.; Kumar, A\.; Rennie, S\.; Davis, L\. S\.; Grauman, K\.; and Feris, R\. 2018\.BlockDrop: Dynamic Inference Paths in Residual Networks\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 8817–8826\.
- Xin et al\. \(2020\)Xin, J\.; Tang, R\.; Lee, J\.; Yu, Y\.; and Lin, J\. 2020\.DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference\.In Jurafsky, D\.; Chai, J\.; Schluter, N\.; and Tetreault, J\., eds\.,*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2246–2251\. Online: Association for Computational Linguistics\.
- Yang et al\. \(2024\)Yang, L\.; Lee, K\.; Nowak, R\.; and Papailiopoulos, D\. 2024\.Looped transformers are better at learning learning algorithms\.In*International conference on learning representations*, volume 2024, 42195–42214\.
- Zhao et al\. \(2025\)Zhao, A\.; Ye, F\.; Fan, Y\.; Tong, J\.; Xiong, J\.; Fei, Z\.; Su, H\.; and Shen, X\. 2025\.SkipGPT: Each Token is One of a Kind\.In*International Conference on Machine Learning*, 77844–77863\.
- Zhou et al\. \(2020\)Zhou, W\.; Xu, C\.; Ge, T\.; McAuley, J\.; Xu, K\.; and Wei, F\. 2020\.Bert loses patience: Fast and robust inference with early exit\.*Advances in Neural Information Processing Systems*, 33: 18330–18341\.
- Zhu et al\. \(2025\)Zhu, R\.\-J\.; Wang, Z\.; Hua, K\.; Zhang, T\.; Li, Z\.; Que, H\.; Wei, B\.; Wen, Z\.; Yin, F\.; Xing, H\.; et al\. 2025\.Scaling Latent Reasoning via Looped Language Models\.*arXiv preprint arXiv:2510\.25741*\.

## Appendix AAdditional Experimental Results on Llama

We additionally evaluateMACROon meta\-llama/Llama\-3\.2\-3B\-Instruct\. For this model we report the standard forward\-pass baseline andMACROacross all benchmarks in Table[A\.1](https://arxiv.org/html/2608.05872#A1.T1)\. The results follow the same protocol and rank\-based color scheme as the main results table\. Consistent with the other models,MACROmatches or improves over the baseline on every benchmark; pooling the per\-dataset, per\-seed accuracy differences, the mean improvement isd¯=\+0\.35\\bar\{d\}=\+0\.35points \(95% bootstrap CI\[0\.03,0\.79\]\[0\.03,\\ 0\.79\]; one\-sided Wilcoxon signed\-rankp=3\.3×10−2p=3\.3\\times 10^\{\-2\}\)\.

black

Table A\.1:Additional results on meta\-llama/Llama\-3\.2\-3B\-Instruct\.
## Appendix BBest Discovered Routes

Tables[B](https://arxiv.org/html/2608.05872#A2.SS0.SSS0.Px2)through[B](https://arxiv.org/html/2608.05872#A2.SS0.SSS0.Px2)list, for every model and benchmark, the chosen route\. Routes built only from local moves dominate on the larger models, while the small and distilled models more often select routes that also contain an add operation\.

### Route notation

EachBest routeentry is read left to right as the sequence of transformer layers applied to the running hidden state\.Liiapplies layeriias a plain \(identity\) step, andLii\-Ljjabbreviates the consecutive runLiiLi\+1i\{\+\}1⋯\\cdotsLjj; a segment whose start index drops below the preceding layer marks a backward revisit or repeat\. A parenthesized suffix denotes an add\-and\-apply step:Lℓ\\ell\(\+h\-bb\)formsht\+γ​ht−bh\_\{t\}\+\\gamma\\,h\_\{t\-b\}from the hidden statebbproduced\-states back and then applies layerℓ\\ell, with the default coefficientγ=1\.0\\gamma=1\.0left implicit; the subtract variant carries an explicit sign and coefficient, e\.g\.L2\(\-0\.25h\-1\)appliesht−0\.25​ht−1h\_\{t\}\-0\.25\\,h\_\{t\-1\}\.RJisRejoin: it terminates the learned prefix and appends the remaining standard forward layers up to the last layer, so every route ends inRJ\. Rows marked*standard forward pass*indicate that the raw candidate is semantically equivalent to the unmodified baseline route afterRejoinexpansion, e\.g\.L0 RJorL0\-L2 RJ\. TheLayerscolumn reports the layer\-application count \(KV\-cache depth\) of the baseline and of the listed route asbaseline→\\toroute, andΔ\\Deltais the listedMACROtest accuracy minus the baseline\.

### Representative non\-standard routes for the one\-model\-one\-route experiment

Table[B](https://arxiv.org/html/2608.05872#A2.SS0.SSS0.Px2)reports, for each of the Qwen3\-1\.7B and Qwen3\-4B models, accuracy averaged over 3 seeds of the single shared policy trained over the combined pool of in\-domain benchmarks, evaluated on test splits of both in\-domain and out\-of\-domain benchmarks\. The layer range and route printed below the table are from one representative seed, given as an illustrative example of the kind of routing the policy discovers\.

SoftRule

Table B\.1:Best route found byMACROfor each model and benchmark\.SoftRule

Table B\.2:Best route found byMACROfor each model and benchmarkSoftRule

Table B\.3:Best route found byMACROfor each model and benchmark\.ModelLayersBest routeQwen3\-1\.7B:28→\\to34L0\-L4 L5\(\+h\-4\) L6\-L15 L10\-L11 RJQwen3\-4B:36→\\to37L0\-L5 L5 L6 RJ

Table B\.4:Performance ofMACROevaluated across all in\-domain and out\-of\-domain benchmarks for each model\. Reported accuracies are averaged over 3 seeds; the layer range and route shown below the table are from one representative seed, given as an illustrative example\.### Selected Route Composition

Table[B\.5](https://arxiv.org/html/2608.05872#A2.T5)summarizes which nonstandard actions appear in the final top\-1 routes selected byMACROand which Viterbi rank is ultimately chosen after validation\. Move actions are common across model families, while Add appears more selectively; every valid route eventually usesRejoin, so it is omitted from the table\.

SoftRule

Table B\.5:Frequency with which selected top\-1 routes contain each action at least once, together with relative route\-length change and the average raw Viterbi rank selected after validation among the final top\-5 candidates \(rank 1 being the most probable\)\. “Both” denotes routes that contain at least one Add and at least one Move\. Length change is100​\(L¯ours−L¯base\)/L¯base100\(\\overline\{L\}\_\{\\mathrm\{ours\}\}\-\\overline\{L\}\_\{\\mathrm\{base\}\}\)/\\overline\{L\}\_\{\\mathrm\{base\}\}, whereLLis the number of executed layer applications\.
## Appendix CExperimental Details

### Connection to CEM\.

Our search procedure can be viewed as a structured variant of the Cross\-Entropy Method\. The sampling distribution is a masked Markov policy over routing programs\. Each iteration samples candidate programs, evaluates them on training data, forms an elite or soft\-elite distribution using accuracy\-based weights, and updates the Markov transition table by weighted maximum likelihood\. Thus, the update minimizes cross\-entropy between the learned policy family and the empirical distribution induced by high\-performing sampled routes, while feasibility masks and smoothing adapt CEM to the constrained routing\-program space\.

### Splits and Decoding

For each model, benchmark, and random seed, we learn a separate task\-level route policy while keeping all pretrained model weights frozen\. Each run requests 1,000 training examples and a held\-out validation split for route selection \(100 examples in the standard runs and 200 examples in the baseline\-anchored 4\-bit/non\-Qwen runs; smaller benchmark train pools are split proportionally by the loader\)\. The final test score is computed after route selection using the available test split or held\-out remainder, with no test subsampling unless explicitly configured\. We report means and standard deviations over completed seeds from\{1,2,42\}\\\{1,2,42\\\}\. Generation is deterministic greedy decoding with temperature0, top\-p=1\.0p=1\.0, top\-k=0k=0, at most 400 thinking tokens, and at most 128 answer tokens\.

### MACROHyperparameters

The main experiments use 10 search iterations with 30 candidate programs per iteration\. Training examples are partitioned into iteration chunks; each iteration also replays one tenth\(λrep=0\.1\)\(\\lambda\_\{\\mathrm\{rep\}\}=0\.1\)of each previous chunk\. The Markov state uses three budget\-phase bins, incoming layer\-delta context, and operator context, but not the previous\-layer context\. The local move radius isr=5r=5\. The program action space contains identity layer moves, add\-and\-apply actions, andRejoin; subtraction and bare merge actions are disabled in the mainMACROruns\. Add actions draw from a history window of six previous hidden states and use coefficientγ=1\.0\\gamma=1\.0\. The initial policy setsεloc=0\.2\\varepsilon\_\{\\mathrm\{loc\}\}=0\.2for local layer moves andRejoin, andεop=0\.0867\\varepsilon\_\{\\mathrm\{op\}\}=0\.0867for add\-and\-apply actions\. The transition update uses smoothingα=0\.0397\\alpha=0\.0397, softmax sharpnessβ=1\.8262\\beta=1\.8262, and an elite set of the top five candidates per iteration\. Route sampling temperature is 1\.0\. Search stops early when all sampled programs become semantically identical after expandingRejoin; otherwise it stops at the 10\-iteration cap\.

SoftRule

Table C\.1:Model\-specific route\-length caps and final candidate construction\. Viterbi decoding returns the highest\-probability raw programs under the masked Markov policy\. For final evaluation we deduplicate programs that execute the same route after expandingRejoin; in baseline\-anchored runs, the standard route is inserted as the selection floor and the remaining candidates are the highest\-probability nonstandard neighbors\.
### Final Selection and Baselines

After the final Markov update, we decode five final candidates with top\-kkViterbi, evaluate them on the validation split, and select by validation accuracy, breaking ties by training accuracy and then shorter route length\. The reported test score is the selected route’s held\-out test accuracy\. The Dr\.LLM baseline trains one lightweight router per benchmark from MCTS\-generated route labels\. For Dr\.LLM\+Ext\., we keep the Dr\.LLM training procedure but replace its action space with the extendedMACROprogram action space\. The Program\-Dr\.LLM runs use 100 MCTS\-labeled training examples, 10 MCTS simulations per example, exploration constant 1\.8, length penalty 3\.0, router hidden dimension 128, eight windows, 15 epochs, AdamW learning rate10−310^\{\-3\}, weight decay 0\.01, gradient accumulation 16, and focal\-lossγ=2\.0\\gamma=2\.0\.

### Exactness of top\-kkdecoding

The five final candidates are the*exact*kkhighest\-probability programs under the learned policy, not a beam or sampling approximation\. Exactness is not automatic here: because the state conditions on the incoming layer\-delta \(the signed gap to the previously applied layer\) and on the operator context, the policy is*not*Markov in the current layer alone, so a decoder that merged all paths reaching a given layer and budget phase would be approximate\. We instead run the top\-kkdynamic program over the augmented state: current layer, budget phase, incoming delta, and operator context, keying on the previously applied layer \(which fixes the incoming delta exactly\) alongside the current layer, length, and operator context, so partial paths with differing continuations are never merged\. The augmentation is lossless and cheap: the incoming delta is a one\-step feature bounded by the local radiusrr, so it enlarges the state space by at most a factor2​r\+12r\+1and needs no longer\-range history, and since program length grows by one at every step the reachable set is a finite DAG on which retaining thekkbest continuations per state yields the exact global top\-kk\. After decoding we deduplicate programs that execute the same route onceRejoinis expanded, keeping the most probable representative of each route\.

## Appendix DInference\-Time Cost

BecauseMACROroutes revisit, repeat, or occasionally drop layers, the executed route changes the per\-token cost of decoding relative to the standard forward pass\. Figure[D\.1](https://arxiv.org/html/2608.05872#Ax1.F1)profiles this on Qwen3\-1\.7B, comparing the standard route with theMACROroute on decoding latency and peak GPU memory\. The overhead is modest and uneven: it is largest where the selected route is longest, negligible where the route stays close to baseline length, and occasionally negative, faster and lighter, when the route is shorter than the standard pass\. Since these are the routes that deliver the accuracy gains of Table[B](https://arxiv.org/html/2608.05872#A2.SS0.SSS0.Px2),MACROtrades a small, route\-dependent compute premium for higher accuracy at fixed model weights\.

## Use of LLM Assistance

Large language models were used only as auxiliary aids for editing, grammar, presentation, and implementation\-code support\. The authors made all substantive technical contributions, experimental design decisions, analyses, interpretations, and final research judgments\.

![Refer to caption](https://arxiv.org/html/2608.05872v1/x6.png)Figure D\.1:Per\-item decoding latency \(top\) and peak GPU memory \(bottom\) of the standard forward pass \(gray\) versus theMACROroute \(blue\) on Qwen3\-1\.7B, under the same two\-stage 400/128\-token generation budget with batched greedy decoding on a single A40 over 64 test items per benchmark\. The percentage above each pair is the route’s overhead relative to the standard route; the rightmost group averages over all thirteen benchmarks \(\+16\.1%\+16\.1\\%latency,\+4\.6%\+4\.6\\%/335335MB memory\)\. Peak memory is the maximum allocated during generation, not compute utilization\.