FARO：基于跨块等风险贡献预算分配的函数空间自适应收益风险比优化

# Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation
Source: [https://arxiv.org/html/2610.40858](https://arxiv.org/html/2610.40858)
Cehao YangAffiliation:The Hong Kong University of Science and Technology \(Guangzhou\)Xuhui JiangAffiliation:IDEA ResearchAffiliation:DataArcTech Ltd\.Chengjin XuAffiliation:IDEA ResearchAffiliation:DataArcTech Ltd\.Jia LiJian Guo

###### Abstract

Training a large language model is compute\-bound, and current recipes divide the work between optimizers by hand: orthogonalized momentum for attention and MLP matrices, Adam\-style updates for the rest\. This division rests on human priors about which layer type deserves which geometry, and it fixes a single global update budget that every parameter block shares\. We introduce FARO, a function\-space adaptive optimizer that makes the partition of parameters into blocks the unit of decision\. FARO gives each block a geometry by its realized counterfactual return, then freezes the routes and reallocates the frozen per\-block budget by cross\-block equal risk contribution \(ERC\), the risk\-parity principle from portfolio theory\. On a GPT\-style model trained on FineWeb, the framework gives two results\. First, its FARO\-Adaptive arm reaches 3\.2780 nats at the full schedule, matching NorMuonAndAdam \(3\.2794\), the best hand\-designed recipe: it recovers the human prior without hand\-setting per\-layer geometry\. Second, FARO\-ERC improves on that arm: it lowers the clean\-data loss at every one of 6 seeds, by 0\.0136 nats on average, a modest but robust gain\. Its sign test, paired test, and confidence interval all agree, and it exceeds the measured run\-to\-run noise at no added time or memory\. It works by trimming the riskiest blocks rather than expanding the safest, and it holds up under a corrupted stream, still lowering the mean loss and winning the seed majority\. The source code is available at[https://github\.com/wxj630/FARO](https://github.com/wxj630/FARO)\.

11footnotetext:Equal Contribution22footnotetext:Corresponding Author## 1Introduction

Optimizers decide how fast a language model reaches a given loss, and current practice spends that choice by hand\. A standard large\-scale recipe\([Jordan et al\., 2024a](https://arxiv.org/html/2610.40858#bib.bib15);[Jordan et al\., 2024b](https://arxiv.org/html/2610.40858#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib2)\)gives attention and MLP matrices an orthogonalized\-momentum rule such as Muon\. It gives embeddings, normalization scales, and the output head an Adam\-style rule\([Kingma and Ba, 2015](https://arxiv.org/html/2610.40858#bib.bib21);[Loshchilov and Hutter, 2017](https://arxiv.org/html/2610.40858#bib.bib22)\), then applies one global learning rate to the whole model\. Two implicit assumptions hold this recipe together\. First, the layer type determines the right geometry: matrices deserve spectral\-norm descent, vectors deserve coordinate\-wise adaptivity\. Second, a single scalar budget is an adequate summary of how much update the network should receive at each step\. Both assumptions come from human priors and small ablation grids, not from measurement on the model being trained\.

Recent matrix optimizers weaken neither assumption\. Muon and its extensions replace one component of the recipe with a better\-conditioned update, and report faster convergence at matched compute\([Jordan et al\., 2024b](https://arxiv.org/html/2610.40858#bib.bib1);[Liu et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib2);[Shah et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib3);[Wen et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib4)\)\. The family keeps growing: normalization\-aware rescaling\([Li et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib5)\), norm\-constrained oracles\([Pethick et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib6)\), rank shrinkage\([Ahn et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib7)\), rotation\-invariant coordinates\([Gong et al\., 2026](https://arxiv.org/html/2610.40858#bib.bib8)\), learning\-rate selection from gradient geometry\([Lau et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib9)\), and curvature\-aware preconditioning\([He et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib10)\)\. Each commits to one geometry and one update scale for every matrix it touches, and each is evaluated as a drop\-in replacement in the same hand\-split recipe\. Choosing among them becomes a benchmark race whose winner is model\-specific, while the assignment of geometry to block is still made by hand\.

We take the assignment itself as the object to be learned\. We presentFARO, a function\-space adaptive optimizer that decides, per parameter block, both the geometry of the update and the share of a fixed total update budget that the block receives\. FARO is a framework rather than a new update rule: Muon, NorMuon, and Adam\-style rules are candidate geometries inside it, so the family above becomes a set of arms FARO can select from instead of a set of competitors\. Figure[1](https://arxiv.org/html/2610.40858#S1.F1)contrasts this with current practice\.

Figure 1:The decision FARO makes, contrasted with current practice\. \(a\) A practitioner hand\-assigns a geometry to each layer type and applies one global learning rate, so every block receives the same budget regardless of the update it can absorb\. \(b\) FARO chooses each block’s geometry by its realized return and reallocates the frozen budget across blocks by risk parity, holding the total fixed\.The first decision is*routing*, which measures the return side of the trade\-off\. FARO partitions the parameters into 117 blocks\. For each block it applies every candidate geometry to a held\-out counterfactual probe and keeps the one with the largest loss decrease\. Routes change while the comparison is informative and freeze once the ranking settles, which bounds the search cost and removes routing as a source of variance\. Appendix[G](https://arxiv.org/html/2610.40858#A7)shows this per\-block decision composes with the joint one under a mild separability assumption, with a bound on the coupling it ignores\.

The second decision is*allocation*, the risk side of the trade\-off and where the measurement gets interesting\. After the routes freeze, each block still carries a budgetβb\\beta\_\{b\}that scales its update\. The obvious controller fails\. Three scalar controllers rescale one global budget: a fractional\-Kelly multiplier\([Kelly, 1956](https://arxiv.org/html/2610.40858#bib.bib38)\), its continuous relaxation, and a difficulty\-sensed rule\. All three are flat against the risk\-neutral FARO\-Adaptive arm, which matches the earlier finding that risk\-seeking step expansion is unproductive\([Moreira and Muir, 2017](https://arxiv.org/html/2610.40858#bib.bib37)\)\. The failure is structural\. A scalar controller multiplies everyβb\\beta\_\{b\}by a common factor, so it only moves along the diagonal of the budget simplex\. It cannot represent*which*blocks deserve more or fewer updates\. We therefore reallocate across blocks at fixed total budget, taking the rule from portfolio theory\. Cross\-block equal risk contribution \(ERC\), or risk parity\([Maillard et al\., 2010](https://arxiv.org/html/2610.40858#bib.bib35);[Roncalli, 2013](https://arxiv.org/html/2610.40858#bib.bib36)\), sets weights so each asset contributes the same share of realized portfolio risk\. Map a block to an asset and its update\-to\-weight ratio to a volatility, and ERC drives each block toward an equal share of update risk\. Raw per\-block risk spans orders of magnitude, so we winsorize it against the cross\-block median, cap the multiplier, and solve for the scale that keeps the total budget unchanged to numerical precision\. The reallocation reads only the previous step’s risk, so the current gradient never influences the budget that acts on it\.

#### Contributions\.

- •We formulate optimizer choice as a function\-space decision and build FARO, in which geometry and budget are separate per\-block decisions\. Routing by realized counterfactual return makes the choice measurable on the model at hand rather than assumed from layer type, so existing matrix optimizers enter as candidate arms\. We show that this per\-block decision composes with the joint decision over all blocks under a mild separability assumption, with a bounded gap \(Appendix[G](https://arxiv.org/html/2610.40858#A7)\)\.
- •We identify cross\-block reallocation as the axis a scalar controller cannot reach, and adapt equal risk contribution to the frozen\-budget problem\. Winsorization, a capped multiplier, and a bisection solve make the reallocation bounded and exactly budget\-neutral, so the comparison against FARO\-Adaptive isolates allocation from a global rescaling of the effective learning rate\. We show that the bisection returns the exact optimizer of a separable risk program whose budget coupling is carried by one scalar price, so equal risk contributions follow from the solve rather than being imposed \(Appendix[G](https://arxiv.org/html/2610.40858#A7)\)\.
- •On a GPT\-style model trained on FineWeb, the framework reproduces the human\-prior recipe without hand\-setting per\-layer geometry\. Its FARO\-Adaptive arm reaches 3\.2780 nats, matching NorMuonAndAdam \(3\.2794\), the strongest hand\-designed optimizer in our comparison\. FARO\-ERC then lowers the clean\-data final validation loss at 6/6 seeds by \-0\.0136 nats at matched compute\. The gain is modest but robust: the win rate, paired test, confidence interval, and repetition gap all agree, and adding it costs no extra time or memory\.

Section[2](https://arxiv.org/html/2610.40858#S2)places FARO against matrix optimizers and risk\-based controllers\. Section[3](https://arxiv.org/html/2610.40858#S3)develops the allocation problem, the per\-block risk signal, and the ERC solve\. Section[4](https://arxiv.org/html/2610.40858#S4)reports the main comparison, the ablations that localize the mechanism, the scalar negative controls, and the controller’s robustness under a corrupted stream\.

## 2Related Work

#### Matrix\-based and adaptive optimizers\.

AdamW remains the default for language\-model pretraining: a coordinate\-wise adaptive scale\([Duchi et al\., 2011](https://arxiv.org/html/2610.40858#bib.bib23);[Kingma and Ba, 2015](https://arxiv.org/html/2610.40858#bib.bib21)\)with decoupled weight decay\([Loshchilov and Hutter, 2017](https://arxiv.org/html/2610.40858#bib.bib22)\)\. A second line constrains the update of a linear layer with a matrix norm\. Muon orthogonalizes the momentum through a Newton–Schulz iteration\([Jordan et al\., 2024b](https://arxiv.org/html/2610.40858#bib.bib1)\), an instance of normalized steepest descent under the spectral norm\([Bernstein and Newhouse, 2024](https://arxiv.org/html/2610.40858#bib.bib13);[Cesista, 2024](https://arxiv.org/html/2610.40858#bib.bib34)\); it has since been scaled to billion\-parameter models\([Liu et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib2)\)and measured against AdamW at matched compute\([Shah et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib3);[Wen et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib4)\)\. Two\-sided preconditioning predates this line in Shampoo and SOAP\([Gupta et al\., 2018](https://arxiv.org/html/2610.40858#bib.bib24);[Vyas et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib25)\), and second\-order or variance\-reduced variants continue to appear\([Martens and Grosse, 2015](https://arxiv.org/html/2610.40858#bib.bib26);[Liu et al\., 2023](https://arxiv.org/html/2610.40858#bib.bib28);[Yuan et al\., 2024](https://arxiv.org/html/2610.40858#bib.bib29);[Zhao et al\., 2024](https://arxiv.org/html/2610.40858#bib.bib27);[Morwani et al\., 2024](https://arxiv.org/html/2610.40858#bib.bib20)\)\. Recent work adjusts the update rather than replacing it\. NorMuon rescales by the update RMS implied by the parameter shape\([Li et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib5)\); Scion studies general norm\-constrained linear\-minimization oracles\([Pethick et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib6)\); Dion2 shrinks the update rank\([Ahn et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib7)\); ARO rotates the update coordinates\([Gong et al\., 2026](https://arxiv.org/html/2610.40858#bib.bib8)\); PolarGrad derives the learning rate from gradient geometry\([Lau et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib9)\); and Newton–Muon right\-preconditions by the input second moment\([He et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib10)\)\. Every method in this family commits to one geometry for all matrices it optimizes, and inherits the hand\-split recipe that decides which parameters receive it\. FARO changes that interface\. The geometry per block becomes a decision made on the model at hand, so these methods enter as candidate arms rather than as competing replacements\.

A complementary line asks why discarding singular values helps\. Bernstein and Newhouse cast optimizer design as steepest descent under a chosen norm\([Bernstein and Newhouse, 2024](https://arxiv.org/html/2610.40858#bib.bib13)\), and the spectral reading of Muon was developed by Cesista\([Cesista, 2024](https://arxiv.org/html/2610.40858#bib.bib34);[Franz, 2024](https://arxiv.org/html/2610.40858#bib.bib33)\)and related analyses\([Riabinin et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib11);[Chen et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib12);[Carlson et al\., 2015](https://arxiv.org/html/2610.40858#bib.bib14)\)\. One\-step and isotropic\-curvature analyses study which directions descend fastest\([Su, 2025](https://arxiv.org/html/2610.40858#bib.bib16);[Davis and Drusvyatskiy, 2025](https://arxiv.org/html/2610.40858#bib.bib41)\), with convergence and implicit\-bias results extending this to stochastic settings\([Li and Hong, 2025](https://arxiv.org/html/2610.40858#bib.bib18);[Sato et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib17);[Fan et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib19)\)\. Our work is orthogonal: we claim no optimal geometry, taking the candidate set as given and asking how to spend the budget\.

#### Budgets, risk, and schedules\.

How much update to apply is normally set by a scalar schedule, from step\-size decay to warmup and cooldown\([Kaplan et al\., 2020](https://arxiv.org/html/2610.40858#bib.bib31);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.40858#bib.bib32)\)\. Scalar control has a financial analogue in bet sizing\. The Kelly criterion sets a fraction of capital from an estimated edge\([Kelly, 1956](https://arxiv.org/html/2610.40858#bib.bib38)\), and volatility management rescales exposure by recent risk\([Moreira and Muir, 2017](https://arxiv.org/html/2610.40858#bib.bib37)\)\. Risk parity instead allocates*across*assets so that each contributes an equal share of portfolio risk\([Maillard et al\., 2010](https://arxiv.org/html/2610.40858#bib.bib35);[Roncalli, 2013](https://arxiv.org/html/2610.40858#bib.bib36)\)\. Hierarchical variants relax the equal\-risk target\([López de Prado, 2016](https://arxiv.org/html/2610.40858#bib.bib39)\), and mean–variance allocation is the classical foundation\([Markowitz, 1952](https://arxiv.org/html/2610.40858#bib.bib40)\)\. We transfer the cross\-sectional version to training: blocks are the assets, the update\-to\-weight ratio is the risk, and the frozen budget is the capital reallocated at fixed total, while our negative controls instantiate the scalar side of the analogy\. Per\-group configurations such as separate learning rates for matrices, embeddings, and biases are common, but they are set by convention and stay static\. FARO learns both the per\-block geometry and the per\-block share, and it constrains the second so the total stays invariant\.

## 3Method

FARO frames optimizer choice as a return\-to\-risk decision and splits it into two decisions acting at different times: each block’s geometry \(measured by the return it realizes\) and its share of the update budget \(allocated by the risk it contributes\)\. Figure[2](https://arxiv.org/html/2610.40858#S3.F2)shows the two lanes and how they interact: routing settles and freezes, then allocation reweights the frozen budgets at every step\. This section states the problem \(§[3\.1](https://arxiv.org/html/2610.40858#S3.SS1)\), defines the per\-block risk signal \(§[3\.2](https://arxiv.org/html/2610.40858#S3.SS2)\), and derives the equal\-risk\-contribution reallocation with its budget\-neutral solve \(§[3\.3](https://arxiv.org/html/2610.40858#S3.SS3)\)\. Our terms are fixed throughout: a*block*is the unit FARO acts on, a*geometry*is the update rule a block runs, and a*budget*scales a block’s update\.

Figure 2:FARO\. Lane 1 routes every parameter block by comparing candidate geometries on a held\-out counterfactual probe, then freezes the routes\. Lane 2 reallocates the frozen budgets: it measures each block’s update\-to\-weight ratio, winsorizes and caps it, and solves for the scaleλ⋆\\lambda^\{\\star\}that preserves the total budget exactly\. The frozen budgets flow down from the routing lane; the risk signal is refreshed upward from the applied step\.### 3\.1Function\-space routing and the frozen\-budget problem

Let the parameters be partitioned intoBBblocks,θ=\(θ1,…,θB\)\\theta=\(\\theta\_\{1\},\\dots,\\theta\_\{B\}\), withB=117B=117in our model; Appendix[A](https://arxiv.org/html/2610.40858#A1)lists the notation\. A block is the coarsest unit FARO can act on\. It is an attention projection bank, an MLP up or down matrix, an embedding, or a group of normalization and bias parameters\. For each block, FARO chooses a*geometry*ϕb\\phi\_\{b\}from a candidate set\{coordinate,row\_column,spectral\}\\\{\\texttt\{coordinate\},\\texttt\{row\\\_column\},\\texttt\{spectral\}\\\}, which corresponds to a coordinate\-wise adaptive rule, a row\-and\-column rescaled matrix rule\([Li et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib5)\), and an orthogonalized matrix rule\([Jordan et al\., 2024b](https://arxiv.org/html/2610.40858#bib.bib1)\)\. The candidate set is open: NorMuon, Dion2, and preconditioned variants can be registered as arms without changing the rest of the method\([Li et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib5);[Ahn et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib7);[He et al\., 2025](https://arxiv.org/html/2610.40858#bib.bib10)\)\.

Routing is decided by measurement\. For a candidate updateub​\(ϕ\)u\_\{b\}\(\\phi\)on blockbb, FARO applies it to a held\-out counterfactual probe and records the realized*return*

ρb​\(ϕ\)=ℒprobe​\(θ\)−ℒprobe​\(θ−η​ub​\(ϕ\)\),\\rho\_\{b\}\(\\phi\)\\;=\\;\\mathcal\{L\}\_\{\\mathrm\{probe\}\}\(\\theta\)\\;\-\\;\\mathcal\{L\}\_\{\\mathrm\{probe\}\}\\big\(\\theta\-\\eta\\,u\_\{b\}\(\\phi\)\\big\),\(1\)the probe loss the step would remove\. Blockbbis routed toϕb=arg⁡maxϕ​ρb​\(ϕ\)\\phi\_\{b\}=\\arg\\max\_\{\\phi\}\\rho\_\{b\}\(\\phi\)\. Routes are held fixed once the ranking stops changing\. Freezing matters for two reasons\. It bounds the cost of an otherwise continuous search, and it removes geometry selection as a source of run\-to\-run variance, so the rest of training can be attributed to the budget decision alone\. Appendix[G](https://arxiv.org/html/2610.40858#A7)bounds how far this per\-block routing view can be from the joint one\. It also shows the allocation solve below is exact for its own risk program, with the budget coupling carried by one scalar price\.

After the freeze timeTfrT\_\{\\mathrm\{fr\}\}, geometry and a reference budgetβ¯b\\bar\{\\beta\}\_\{b\}are fixed, and each step applies

θb←θb−αβbdb\(ϕb\),b=1,…,B,\\theta\_\{b\}\\;\\leftarrow\\;\\theta\_\{b\}\\;\-\\;\\alpha\\,\\beta\_\{b\}\\,d\_\{b\}\(\\phi\_\{b\}\),\\qquad b=1,\\dots,B,\(2\)whereα\\alphais the base learning rate anddb​\(ϕb\)d\_\{b\}\(\\phi\_\{b\}\)is the direction the frozen geometry induces\. We callβb\\beta\_\{b\}the block’s budget\. The*FARO\-Adaptive*arm takesβb\\beta\_\{b\}equal to the geometry\-derived reference value and never changes it; it is the risk\-neutral reference point for every comparison in Section[4](https://arxiv.org/html/2610.40858#S4)\.

We treat the budgets as an allocation problem: holding the routing and the freeze timeline fixed, choose\{βb\}\\\{\\beta\_\{b\}\\\}to lower the expected final loss subject to a*fixed total*,

∑b=1Bβb=∑b=1Bβ¯b=:Bud\.\\sum\_\{b=1\}^\{B\}\\beta\_\{b\}\\;=\\;\\sum\_\{b=1\}^\{B\}\\bar\{\\beta\}\_\{b\}\\;=:\\;\\mathrm\{Bud\}\.\(3\)Constraint \([3](https://arxiv.org/html/2610.40858#S3.E3)\) is the design decision that separates FARO’s allocation from a schedule\. A scalar controller relaxes it by multiplying every budget by a common factor and so moves along the diagonal of the budget simplex; the allocation we build keeps the total fixed and moves mass*between*blocks\.

#### Why the constraint matters\.

Writehb​\(βb\)h\_\{b\}\(\\beta\_\{b\}\)for blockbb’s per\-step loss decrease, with block risk growing inβb\\beta\_\{b\}\. Moving a unit of budget from blockjjto blockiigains abouthi′​\(βi\)−hj′​\(βj\)h\_\{i\}^\{\\prime\}\(\\beta\_\{i\}\)\-h\_\{j\}^\{\\prime\}\(\\beta\_\{j\}\)against an added variance from the enlarged block\. A scalar controller forcesβi=βj\\beta\_\{i\}=\\beta\_\{j\}for every pair, so it cannot reach that term even when marginal returns differ\. That is why we allocate by risk\.

### 3\.2Per\-block risk

The allocation needs a per\-block quantity that plays the role of volatility\. We use the*update\-to\-weight ratio*, a scale\-free measure of how large a block’s update is relative to the parameter it acts on,

uwr^b\(t\)=∥Δb\(t\)∥2∥θb\(t\)∥2\+ϵ,Δb\(t\)=α​βb​db​\(ϕb\),\\widehat\{\\mathrm\{uwr\}\}\_\{b\}^\{\(t\)\}\\;=\\;\\frac\{\\lVert\\Delta\_\{b\}^\{\(t\)\}\\rVert\_\{2\}\}\{\\lVert\\theta\_\{b\}^\{\(t\)\}\\rVert\_\{2\}\+\\epsilon\},\\qquad\\Delta\_\{b\}^\{\(t\)\}=\\alpha\\,\\beta\_\{b\}\\,d\_\{b\}\(\\phi\_\{b\}\),\(4\)smoothed into an exponentially weighted risk signal

rb\(t\)=γ​rb\(t−1\)\+\(1−γ\)​uwr^b\(t\),r\_\{b\}^\{\(t\)\}\\;=\\;\\gamma\\,r\_\{b\}^\{\(t\-1\)\}\+\(1\-\\gamma\)\\,\\widehat\{\\mathrm\{uwr\}\}\_\{b\}^\{\(t\)\},\(5\)initialized atrb\(0\)=uwr^b\(0\)r\_\{b\}^\{\(0\)\}=\\widehat\{\\mathrm\{uwr\}\}\_\{b\}^\{\(0\)\}, with decayγ\\gamma\. The ratio is a relative displacement\. A block is low\-risk in portfolio terms when its parameters are large next to the step it receives, and high\-risk when it is moved hard relative to its own scale\. The ratio is scale\-free, sorbr\_\{b\}is comparable across blocks of different shapes and geometries, which is what a cross\-sectional rule needs\. We measure it once per frozen step\. The allocation at stepttusesrb\(t−1\)r\_\{b\}^\{\(t\-1\)\}, so the gradient the step will apply never sets its own budget\.

### 3\.3Cross\-block equal\-risk\-contribution allocation

Equal risk contribution sets weights so that each asset contributes an equal share of total portfolio risk\([Maillard et al\., 2010](https://arxiv.org/html/2610.40858#bib.bib35);[Roncalli, 2013](https://arxiv.org/html/2610.40858#bib.bib36)\)\. Readingβb\\beta\_\{b\}as a portfolio weight andrbr\_\{b\}as a volatility, the risk contributed by blockbbisβb​rb\\beta\_\{b\}r\_\{b\}, and equalizing contributions givesβb∝1/rb\\beta\_\{b\}\\propto 1/r\_\{b\}\. Two properties force changes\. Rawrbr\_\{b\}spans orders of magnitude, since scalar, normalization, and bias blocks have near\-zero parameter norms, so an unbounded inverse would hand almost the whole budget to a few tiny blocks; and an unbounded multiplier would let the allocation dominate the frozen geometry\. We bound both ends\.

#### Winsorization\.

Letm=medianb​rbm=\\mathrm\{median\}\_\{b\}\\,r\_\{b\}be the cross\-block median risk, and define the winsorized risk

r~b=clip⁡\(rb,mκ,κ​m\),\\tilde\{r\}\_\{b\}\\;=\\;\\mathrm\{clip\}\\\!\\Big\(r\_\{b\},\\;\\tfrac\{m\}\{\\kappa\},\\;\\kappa\\,m\\Big\),\(6\)with winsor boundκ\\kappa\. This keeps every block inside a bounded factor of the median, so a block can be judged relatively risky without being allowed to collapse the allocation\. The median, rather than the mean, is the reference because a few extreme blocks would otherwise set the scale\.

#### Budget neutrality\.

Ignoring bounds for a moment, the uncapped multiplier isλ/r~b\\lambda/\\tilde\{r\}\_\{b\}for a single scaleλ\>0\\lambda\>0\. The exact budget constraint \([3](https://arxiv.org/html/2610.40858#S3.E3)\) pins that scale in closed form,

λ=∑jβj∑jβj/r~j,\\lambda\\;=\\;\\frac\{\\sum\_\{j\}\\beta\_\{j\}\}\{\\sum\_\{j\}\\beta\_\{j\}/\\tilde\{r\}\_\{j\}\},\(7\)because then∑bβb​λ/r~b=∑bβb\\sum\_\{b\}\\beta\_\{b\}\\lambda/\\tilde\{r\}\_\{b\}=\\sum\_\{b\}\\beta\_\{b\}\. We then cap the multiplier into\[c−1,c\]\[c^\{\-1\},c\],

μb=clip⁡\(λr~b,c−1,c\),βb←βb​μb,\\mu\_\{b\}\\;=\\;\\mathrm\{clip\}\\\!\\Big\(\\tfrac\{\\lambda\}\{\\tilde\{r\}\_\{b\}\},\\;c^\{\-1\},\\;c\\Big\),\\qquad\\beta\_\{b\}\\leftarrow\\beta\_\{b\}\\,\\mu\_\{b\},\(8\)with allocation capcc\.

#### Solving the capped system\.

Capping and neutrality interact: renormalising after clamping would break the cap, while clamping after the closed\-form solve would break neutrality\. We therefore solve forλ\\lambdaon the monotone map

Λ⁡\(λ\)=∑bβ¯b​min⁡\(max⁡\(λr~b,c−1\),c\),\\Lambda\(\\lambda\)\\;=\\;\\sum\_\{b\}\\bar\{\\beta\}\_\{b\}\\,\\min\\\!\\Big\(\\max\\\!\\big\(\\tfrac\{\\lambda\}\{\\tilde\{r\}\_\{b\}\},\\,c^\{\-1\}\\big\),\\,c\\Big\),\(9\)The map is non\-decreasing inλ\\lambda, so bisection gives the uniqueλ⋆\\lambda^\{\\star\}withΛ⁡\(λ⋆\)=Bud\\Lambda\(\\lambda^\{\\star\}\)=\\mathrm\{Bud\}\. The allocation then respects the cap*and*preserves the total to numerical precision, so the arms cannot differ by a global rescaling of the effective learning rate\. Reallocation still changes how that fixed total is spread, so whether it also shifts the realized update norm is separate; the cap\-by\-LR ablation \(Section[4\.6](https://arxiv.org/html/2610.40858#S4.SS6)\) settles it empirically\. We may also clamp the product to ERC\-specific bounds\[βminerc,βmaxerc\]\[\\beta^\{\\mathrm\{erc\}\}\_\{\\min\},\\beta^\{\\mathrm\{erc\}\}\_\{\\max\}\], which are decoupled from the global routing bounds so that tightening one does not move the other\.

Algorithm 1ERC frozen\-budget allocation \(one frozen step\)1:frozen budgets

\{β¯b\}\\\{\\bar\{\\beta\}\_\{b\}\\\}, risk EWMA

\{rb\}\\\{r\_\{b\}\\\}, cap

cc, winsor

κ\\kappa, ERC clamp

\[βminerc,βmaxerc\]\[\\beta^\{\\mathrm\{erc\}\}\_\{\\min\},\\beta^\{\\mathrm\{erc\}\}\_\{\\max\}\], decay

γ\\gamma
2:effective budgets

\{βb\}\\\{\\beta\_\{b\}\\\}and multipliers

\{μb\}\\\{\\mu\_\{b\}\\\}
3:

Bud←∑bβ¯b\\mathrm\{Bud\}\\leftarrow\\sum\_\{b\}\\bar\{\\beta\}\_\{b\}
4:

m←medianb​rbm\\leftarrow\\mathrm\{median\}\_\{b\}\\,r\_\{b\}
5:for

b=1,…,Bb=1,\\dots,Bdo

6:

r~b←clip⁡\(rb,m/κ,κ​m\)\\tilde\{r\}\_\{b\}\\leftarrow\\mathrm\{clip\}\(r\_\{b\},\\;m/\\kappa,\\;\\kappa m\)⊳\\trianglerightwinsorize risk

7:endfor

8:define

Λ⁡\(λ\)←∑bβ¯b​clip​\(λ/r~b,c−1,c\)\\Lambda\(\\lambda\)\\leftarrow\\sum\_\{b\}\\bar\{\\beta\}\_\{b\}\\,\\mathrm\{clip\}\(\\lambda/\\tilde\{r\}\_\{b\},\\;c^\{\-1\},\\;c\)
9:

λ⋆←\\lambda^\{\\star\}\\leftarrowbisection root of

Λ⁡\(λ\)=Bud\\Lambda\(\\lambda\)=\\mathrm\{Bud\}⊳\\trianglerightbudget\-neutral scale

10:for

b=1,…,Bb=1,\\dots,Bdo

11:

μb←clip⁡\(λ⋆/r~b,c−1,c\)\\mu\_\{b\}\\leftarrow\\mathrm\{clip\}\(\\lambda^\{\\star\}/\\tilde\{r\}\_\{b\},\\;c^\{\-1\},\\;c\)
12:

βb←clip⁡\(β¯b​μb,βminerc,βmaxerc\)\\beta\_\{b\}\\leftarrow\\mathrm\{clip\}\(\\bar\{\\beta\}\_\{b\}\\,\\mu\_\{b\},\\;\\beta^\{\\mathrm\{erc\}\}\_\{\\min\},\\;\\beta^\{\\mathrm\{erc\}\}\_\{\\max\}\)
13:endfor

14:return

\{βb\},\{μb\}\\\{\\beta\_\{b\}\\\},\\ \\\{\\mu\_\{b\}\\\}

#### What the method leaves untouched, and its cost\.

Algorithm[1](https://arxiv.org/html/2610.40858#alg1)runs only after every route has frozen, leaving the geometries, router scores, freeze condition, and learning\-rate schedule unchanged\. It reads only prior\-step risk, so it cannot leak the current gradient into the allocation; with ERC disabled it is the identity \(μb≡1\\mu\_\{b\}\\equiv 1\) and FARO\-Adaptive is recovered exactly\. A solve costsO⁡\(B\)O\(B\)risk measurements plus the bisection and adds no parameters or optimizer state, which Table[1](https://arxiv.org/html/2610.40858#S4.T1)confirms by measurement\.

## 4Experiments

### 4\.1Setup

#### Model and data\.

We train a GPT\-style transformer with 11 layers, width 768, 6 heads, and vocabulary 50257 on FineWeb token shards\([Penedo et al\., 2024](https://arxiv.org/html/2610.40858#bib.bib30)\), following the reference trainer\([Jordan et al\., 2024a](https://arxiv.org/html/2610.40858#bib.bib15)\)\. Runs last 800 steps and we report the final validation cross\-entropy \(lower is better\); Appendix[A](https://arxiv.org/html/2610.40858#A1)lists the model, optimizer defaults, and schedule\.

#### Two data regimes\.

We evaluate under clean FineWeb, the default protocol, and a*scatter*regime that replaces60%60\\%of tokens at random positions \(per\-microbatch hash, so difficulty is not temporally clustered\), testing the controller when per\-block risk is noisier\.

#### Protocol\.

Every comparison is a matched pair: the pair shares the seed, data order, geometry routing, and freeze timeline, so it differs only in the budget controller\. On clean data we run 6 seeds×\\times2 repetitions per arm; on scatter, 6 seeds\. The run\-to\-run noise floor is0\.0040\.004–0\.010\.01nats, measured on repeated identical configurations, and we use it as the reference scale throughout\.

#### Arms and notation\.

TheFARO\-Adaptivearm is FARO with its geometry\-derived frozen budgets and no risk controller; theFARO\-ERCarm adds the reallocation of Algorithm[1](https://arxiv.org/html/2610.40858#alg1)on top, at the same total budget\. Every ablation reportsΔ=\\Delta=loss\(arm\)−\-loss\(FARO\-Adaptive\) on matched pairs, so negative favours the controller\.

### 4\.2Main result

Table 1:Main results: final validation loss at 800 steps under a matched\-pair protocol \(identical seed, data order, geometry routing, and freeze timeline; only the budget controller differs\), plus the per\-run cost the controller adds \(peak allocated memory in MiB, mean per\-step time in ms\)\.Δ=\\Delta=FARO\-ERC−\-FARO\-Adaptive \(negative is better\)\. Per\-arm loss and cost are±\\pmpopulation std over seeds;Δ\\Deltacarries the paired sample std used by the tests in Section[4\.4](https://arxiv.org/html/2610.40858#S4.SS4)\. Lower is better throughout\.clean FineWeb\(6 seeds×\\times2 reps\)cost\(clean\)scatter\(60% noise, 6 seeds\)SeedFARO\-Ad\.FARO\-ERCΔ\\Deltawinmemms Ad\.ms ERCFARO\-Ad\.FARO\-ERCΔ\\Deltawin*Per\-seed final validation loss and cost*seed03\.60193\.5917\-0\.0102✓42,495397\.0393\.23\.79563\.7690\-0\.0266✓seed13\.60813\.5850\-0\.0231✓42,495393\.7399\.33\.78983\.7871\-0\.0027✓seed23\.59973\.5865\-0\.0131✓42,495392\.3398\.33\.76993\.7811\+0\.0113—seed33\.60013\.5879\-0\.0122✓42,495395\.9398\.73\.77983\.7742\-0\.0056✓seed43\.60363\.5904\-0\.0132✓42,495392\.6394\.53\.79073\.7853\-0\.0054✓seed53\.59683\.5872\-0\.0096✓42,495396\.5397\.03\.76103\.7567\-0\.0043✓Mean3\.60173\.60173\.58813\.5881−0\.0136\-0\.01366/642,495394\.7396\.83\.78113\.78113\.77563\.7756−0\.0056\-0\.00565/6Std±0\.0036\\pm 0\.0036/±0\.0023\\pm 0\.0023±0\.0049\\pm 0\.0049—00±1\.9\\pm 1\.9±2\.3\\pm 2\.3—±0\.0121\\pm 0\.0121—Table[1](https://arxiv.org/html/2610.40858#S4.T1)is the central result\. On clean data FARO\-ERC lowers the final loss at 6/6 seeds, a modest but robust gain that the tests in Section[4\.4](https://arxiv.org/html/2610.40858#S4.SS4)confirm four ways\. On scatter the reallocation does not degrade: FARO\-ERC still lowers the mean and wins 5/6 seeds, keeping its footing where per\-block risk is noisiest\. The narrower margin there tracks the noisier risk signal, so we read scatter as a stress test of the same controller\.

#### Reading the margin\.

The baseline is NorMuonAndAdam, the best hand\-designed recipe in our comparison, so the reallocation acts on an already near\-optimal split: the question is whether the budget axis still carries structure once geometry is settled\. It does: FARO\-ERC improves on the human recipe at all 6/6 seeds, the confidence interval excluding zero, on the one axis a scalar controller cannot reach \(Section[4\.6](https://arxiv.org/html/2610.40858#S4.SS6)\)\. A near\-optimal start leaves little headroom, so a modest margin is expected; the improvement is systematic and the axis—not the magnitude—is the contribution\.

Figure 3:The paper’s main analysis; axis labels abbreviate FARO\-ERC and FARO\-Adaptive as*ERC*and*Adaptive*\. \(a\) Per\-seed loss change on clean data: FARO\-ERC is lower at all 6 seeds \(bars are matched\-pair differences; the shaded band is the±0\.01\\pm 0\.01\-nat noise floor\)\. \(b\) Per\-checkpoint gain: below zero from step 250, deepest at step 450, narrowing by the end, i\.e\. faster descent rather than earlier arrival\. \(c\) Allocation\-cap sweep: the gain grows with the cap and saturates\. \(d\) Clamp sweep: lowering the floor deepens the gain more than raising the ceiling, so the benefit concentrates on the risky tail\.

### 4\.3The framework recovers the human prior

Our first claim is that the framework works without hand\-set per\-layer geometry\. Against the strongest hand\-designed optimizers under one harness and schedule \(Table[8](https://arxiv.org/html/2610.40858#A4.T8)\), FARO’s adaptive arm \(n=3n\{=\}3\) reaches 3\.2780 nats, matching NorMuonAndAdam \(n=3n\{=\}3\) within0\.00140\.0014nats and clearly below Muon \(3\.5624\) and Adam \(3\.5144\); Figure[4](https://arxiv.org/html/2610.40858#S4.F4)shows the curves, indistinguishable from step250250on\. One framework that picks each block’s geometry by measured return lands on the best human\-designed recipe with no hand\-set split; FARO\-ERC then improves on that arm\.

Figure 4:Learning curves on the full schedule \(mean over seeds\)\. FARO\-Adaptive and NorMuonAndAdam overlap; Muon and Adam stay above\. The framework recovers the hand\-designed recipe without a hand\-set split\.
### 4\.4Statistical evidence for the clean\-data gain

We establish the clean\-data gain four independent ways\. The sign test \(6/6,p=0\.031p=0\.031\), the pairedt=−6\.76t=\-6\.76, and the95%95\\%CI\[−0\.0187,−0\.0084\]\[\-0\.0187,\-0\.0084\]all clear zero, and across the2×62\\times 6runs the groups do not overlap\. The effect is also3\.4×3\.4\\timesthe mean replicate noise \(0\.0039\) and1\.3×1\.3\\timesits worst case \(0\.0101\), so it sits above run\-to\-run variance rather than inside it\. These four checks establish robustness, which is the claim\.

#### Robustness\.

The comparison used a pre\-committed seed and repetition budget, and no outlier carries the win: all twelve FARO\-ERC runs beat all twelve FARO\-Adaptive runs \(Section[4\.4](https://arxiv.org/html/2610.40858#S4.SS4)\)\. Table[1](https://arxiv.org/html/2610.40858#S4.T1)extends this to a corrupted stream, where the reallocation still lowers the mean and wins 5/6 seeds, a narrower margin rather than a reversal—graceful under noise\.

### 4\.5Convergence behaviour

Lower final loss could mean reaching a target sooner or merely ending lower; the two differ, and the discrete measure cannot tell them apart \(validation is scored every5050steps, coarser than the gap\), so we read the trajectory\. FARO\-ERC sits below FARO\-Adaptive from step250250on, peaking at step450450\(\-0\.0471\) and narrowing to \-0\.0136 by step800800, with run\-mean loss lower by \-0\.0197 nats \(6/6 seeds\)\. That is a faster\-descent signature rather than an earlier arrival; Figure[3](https://arxiv.org/html/2610.40858#S4.F3)\(b\) shows it and Table[6](https://arxiv.org/html/2610.40858#A1.T6)gives the per\-checkpoint values\. The lead narrows toward the end as the schedule anneals\.

### 4\.6Ablations

Three single\-knob checks locate the mechanism\.*The allocation cap*is the dominant knob \(Table[4](https://arxiv.org/html/2610.40858#A1.T4)\): the gain rises with it \(about1\.7×1\.7\\timesfrom the default 1\.4 to 2\.6\) and then saturates, because the gain comes from clamping high\-risk blocks and once the downward clamp binds, a larger cap has nothing left to act on\. Moving the ERC clamp bounds independently \(Table[5](https://arxiv.org/html/2610.40858#A1.T5)\) shows the asymmetry: lowering the floor to0\.250\.25deepens the gain further \(about1\.8×1\.8\\times\) while raising the ceiling moves it less, so the benefit concentrates on trimming the risky tail\. A replay over logged per\-block risks agrees—the winsor boundκ\\kappaand EWMA decayγ\\gammaare near\-inert, while the cap sets the tilt directly\.

*Reallocation is not a learning\-rate effect\.*Crossing the cap with the base LR shows no interaction \(Table[3](https://arxiv.org/html/2610.40858#A1.T3)\): at both caps LR0\.7×0\.7\\timesand1\.0×1\.0\\timesare tied and0\.5×0\.5\\timesis worse, so cap and LR are not substitutes\.

*The signal is cross\-block, not global\.*We test the alternative directly: under the same protocol we run a fractional\-Kelly rule that rescales one global budget from a probe return\([Kelly, 1956](https://arxiv.org/html/2610.40858#bib.bib38)\), and DS\-FARO, which rescales it by the shortfall of a probe return below its trailing median\. Both land inside the noise floor \(\-0\.0002 and \-0\.0006 nats; 1/3 and 2/3 seed wins; Table[7](https://arxiv.org/html/2610.40858#A1.T7)\)\. A scalar controller only multiplies every budget by a common factor, so it cannot express which blocks should move less; its flat behaviour next to FARO\-ERC’s gain points to the cross\-block information rather than the size of the global step\.

## 5Conclusion

We posed optimizer choice as a function\-space decision and built FARO: each block gets its own geometry and its own share of a fixed update budget\. Geometry is chosen by realized counterfactual return and frozen; budget is reallocated by equal risk contribution, with winsorized risk, a capped multiplier, and a bisection solve that preserves the total exactly\. On FineWeb the FARO\-Adaptive arm matches the best hand\-designed recipe \(NorMuonAndAdam\) with no hand\-set split, and FARO\-ERC improves on it at all 6/6 seeds—below FARO\-Adaptive through mid\-training, robust under a corrupted stream and free in time and memory\. The ablations show the effect is cross\-block, from trimming high\-risk blocks rather than a learning\-rate change: geometry is no longer set by hand, and the budget axis carries a systematic gain no scalar controller can reach\.

## Limitations

Our study is at one model scale and one corpus: a 11\-layer, 768\-wide GPT\-style transformer on FineWeb\. Scaling up is the natural next step\. The framework’s premise only becomes stronger there, because a larger model offers a finer block structure to route over and a wider spread of per\-block risk to reallocate against\.

The comparison is taken at the 800\-step point of the schedule, so the reported losses are final losses under a fixed compute budget rather than converged values\. The allocation reads live per\-block risk, so its precision tracks that signal\. On clean data the gain is systematic\. On our corrupted stress stream per\-block risk is noisier, and the controller still lowers the mean loss and wins the seed majority, with a narrower margin\.

## Ethical Considerations

This paper studies a training optimizer, so the risks it raises are the ordinary ones of language\-model pretraining\. The main one is environmental: any method that lowers loss per unit of compute can invite larger training runs, and our own experiments used H800 GPU time\. We keep every comparison matched on seed, data order, and compute budget, and Table[1](https://arxiv.org/html/2610.40858#S4.T1)shows the reallocation adds no measurable time or memory, so the reported gain is not bought with extra compute\. All experiments use public FineWeb shards and a public trainer; no private data, human subjects, or personal data were involved\. Our corpus carries the usual web\-text biases\. An optimizer emits no text but shapes a model that will, so we present the method as a training\-time component and name the data source\.

## LLM Usage

We used large language models as a writing and coding aid: drafting and editing prose, implementing and refactoring the training and analysis code, and writing the plotting scripts\. The scientific content is our own, including the problem formulation, the routing\-by\-return design, the adaptation of equal risk contribution to the frozen\-budget setting, and the reading of the ablations\. Every reported number comes from our own runs, recomputed from the recorded training logs\. We verified all code that produces numbers in this paper and take responsibility for the results and claims\.

## References

- Ahnet al\.\(2025\)K\. Ahn, N\. Amsel, and J\. LangfordDion2: a simple method to shrink matrix in Muon\.arXiv preprint arXiv:2512\.16928\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.40858#S3.SS1.p1.1)\.
- Bernstein and Newhouse \(2024\)J\. Bernstein and L\. NewhouseOld optimizer, new norm: an anthology\.InOPT 2024: Optimization for Machine Learning,Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Carlsonet al\.\(2015\)D\. E\. Carlson, E\. Collins, Y\. Hsieh, L\. Carin, and V\. CevherPreconditioned spectral descent for deep learning\.Advances in neural information processing systems28\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Cesista \(2024\)F\. L\. CesistaDeep learning optimizers as steepest descent in normed spaces\.External Links:[Link](http://leloykun.github.io/ponder/steepest-descent-opt/)Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Chenet al\.\(2025\)L\. Chen, J\. Li, and Q\. LiuMuon optimizes under spectral norm constraints\.arXiv preprint arXiv:2506\.15054\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Davis and Drusvyatskiy \(2025\)D\. Davis and D\. DrusvyatskiyWhen do spectral gradient updates help in deep learning?\.arXiv preprint arXiv:2512\.04299\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Duchiet al\.\(2011\)J\. Duchi, E\. Hazan, and Y\. SingerAdaptive subgradient methods for online learning and stochastic optimization\.\.Journal of machine learning research12\(7\)\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Fanet al\.\(2025\)C\. Fan, M\. Schmidt, and C\. ThrampoulidisImplicit bias of spectral descent and Muon on multiclass separable data\.arXiv preprint arXiv:2502\.04664\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Franz \(2024\)L\. C\. FranzThe Case for Muon\.X \(Formerly Twitter\) thread\.External Links:[Link](https://x.com/leloykun/status/1846842887839125941)Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Gonget al\.\(2026\)W\. Gong, J\. Zazo, Q\. Luo, P\. Wang, J\. Hensman, and C\. MaARO: a new lens on matrix optimization for large models\.arXiv preprint arXiv:2602\.09006\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Guptaet al\.\(2018\)V\. Gupta, T\. Koren, and Y\. SingerShampoo: preconditioned stochastic tensor optimization\.InInternational Conference on Machine Learning,pp\. 1842–1850\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2025\)C\. He, Z\. Deng, and Z\. LuLow\-rank orthogonalization for large\-scale matrix optimization with applications to foundation model training\.arXiv preprint arXiv:2509\.11983\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.40858#S3.SS1.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. SifreTraining compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1)\.
- Jordanet al\.\(2024a\)K\. Jordan, J\. Bernstein, B\. Rappazzo, @fernbear\.bsky\.social, B\. Vlado, Y\. Jiacheng, F\. Cesista, B\. Koszarsky, and @Grad62304977modded\-nanogpt: speedrunning the NanoGPT baseline\.External Links:[Link](https://github.com/KellerJordan/modded-nanogpt)Cited by:[Appendix A](https://arxiv.org/html/2610.40858#A1.SS0.SSS0.Px2.p1.1),[Appendix B](https://arxiv.org/html/2610.40858#A2.SS0.SSS0.Px1.p1.1),[Appendix D](https://arxiv.org/html/2610.40858#A4.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2610.40858#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.40858#S4.SS1.SSS0.Px1.p1.1)\.
- Jordanet al\.\(2024b\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p1.1),[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.40858#S3.SS1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1)\.
- Kelly \(1956\)J\. L\. KellyA new interpretation of information rate\.Bell System Technical Journal35\(4\),pp\. 917–926\.External Links:[Document](https://dx.doi.org/10.1002/j.1538-7305.1956.tb03809.x)Cited by:[Appendix D](https://arxiv.org/html/2610.40858#A4.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.40858#S1.p5.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1),[§4\.6](https://arxiv.org/html/2610.40858#S4.SS6.p3.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: A method for stochastic optimization\.In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\-9, 2015, Conference Track Proceedings,Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://arxiv.org/abs/1412.6980)Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p1.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Lauet al\.\(2025\)T\. T\. Lau, Q\. Long, and W\. SuPolarGrad: A class of matrix\-gradient optimizers from a unifying preconditioning perspective\.arXiv preprint arXiv:2505\.21799\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Li and Hong \(2025\)J\. Li and M\. HongA note on the convergence of Muon\.arXiv preprint arXiv:2502\.02900\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Liet al\.\(2025\)Z\. Li, L\. Liu, C\. Liang, W\. Chen, and T\. ZhaoNorMuon: making Muon more efficient and scalable\.arXiv preprint arXiv:2510\.05491\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.40858#S3.SS1.p1.1)\.
- Liuet al\.\(2023\)H\. Liu, Z\. Li, D\. Hall, P\. Liang, and T\. MaSophia: a scalable stochastic second\-order optimizer for language model pre\-training\.arXiv preprint arXiv:2305\.14342\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan, Y\. Chen, H\. Zheng, Y\. Liu, S\. Liu, B\. Yin, W\. He, H\. Zhu, Y\. Wang, J\. Wang, M\. Dong, Z\. Zhang, Y\. Kang, H\. Zhang, X\. Xu, Y\. Zhang, Y\. Wu, X\. Zhou, and Z\. YangMuon is scalable for LLM training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p1.1),[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- López de Prado \(2016\)M\. López de PradoBuilding diversified portfolios that outperform out of sample\.The Journal of Portfolio Management42\(4\),pp\. 59–69\.External Links:[Document](https://dx.doi.org/10.3905/jpm.2016.42.4.059)Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p1.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Maillardet al\.\(2010\)S\. Maillard, T\. Roncalli, and J\. TeiletcheThe properties of equally weighted risk contribution portfolios\.The Journal of Portfolio Management36\(4\),pp\. 60–70\.External Links:[Document](https://dx.doi.org/10.3905/jpm.2010.36.4.060)Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p5.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2610.40858#S3.SS3.p1.1)\.
- Markowitz \(1952\)H\. MarkowitzPortfolio selection\.The Journal of Finance7\(1\),pp\. 77–91\.External Links:[Document](https://dx.doi.org/10.2307/2975974)Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1)\.
- Martens and Grosse \(2015\)J\. Martens and R\. GrosseOptimizing neural networks with Kronecker\-factored approximate curvature\.InInternational conference on machine learning,pp\. 2408–2417\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Moreira and Muir \(2017\)A\. Moreira and T\. MuirVolatility\-managed portfolios\.The Journal of Finance72\(4\),pp\. 1611–1644\.External Links:[Document](https://dx.doi.org/10.1111/jofi.12513)Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p5.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1)\.
- Morwaniet al\.\(2024\)D\. Morwani, I\. Shapira, N\. Vyas, E\. Malach, S\. Kakade, and L\. JansonA new perspective on Shampoo’s preconditioner\.arXiv preprint arXiv:2406\.17748\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, T\. Wolf,et al\.The fineweb datasets: decanting the web for the finest text data at scale\.arXiv preprint arXiv:2406\.17557\.Cited by:[Appendix B](https://arxiv.org/html/2610.40858#A2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.40858#S4.SS1.SSS0.Px1.p1.1)\.
- Pethicket al\.\(2025\)T\. Pethick, W\. Xie, K\. Antonakopoulos, Z\. Zhu, A\. Silveti\-Falls, and V\. CevherTraining deep learning models with norm\-constrained LMOs\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Riabininet al\.\(2025\)A\. Riabinin, E\. Shulgin, K\. Gruntkowska, and P\. RichtárikGluon: making Muon & Scion great again\!\(bridging theory and practice of LMO\-based optimizers for LLMs\)\.arXiv preprint arXiv:2505\.13416\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Roncalli \(2013\)T\. RoncalliIntroduction to risk parity and budgeting\.Chapman & Hall/CRC\.External Links:[Document](https://dx.doi.org/10.1201/b15151)Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p5.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2610.40858#S3.SS3.p1.1)\.
- Satoet al\.\(2025\)N\. Sato, H\. Naganuma, and H\. IidukaAnalysis of Muon’s convergence and critical batch size\.arXiv preprint arXiv:2507\.01598\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Shahet al\.\(2025\)I\. Shah, A\. M\. Polloreno, K\. Stratos, P\. Monk, A\. Chaluvaraju, A\. Hojel, A\. Ma, A\. Thomas, A\. Tanwer, D\. J\. Shah,et al\.Practical efficiency of Muon for pretraining\.arXiv preprint arXiv:2505\.02222\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Su \(2025\)W\. SuIsotropic curvature model for understanding deep learning optimization: is gradient orthogonalization optimal?\.arXiv preprint arXiv:2511\.00674\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p2.1)\.
- Vyaset al\.\(2025\)N\. Vyas, D\. Morwani, R\. Zhao, I\. Shapira, D\. Brandfonbrener, L\. Janson, and S\. KakadeSOAP: Improving and stabilizing Shampoo using Adam\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Wenet al\.\(2025\)K\. Wen, D\. Hall, T\. Ma, and P\. LiangFantastic pretraining optimizers and where to find them\.arXiv preprint arXiv:2509\.02046\.Cited by:[§1](https://arxiv.org/html/2610.40858#S1.p2.1),[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Yuanet al\.\(2024\)H\. Yuan, Y\. Liu, S\. Wu, X\. Zhou, and Q\. GuMars: unleashing the power of variance reduction for training large models\.arXiv preprint arXiv:2411\.10438\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2024\)J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. TianGaLore: memory\-efficient LLM training by gradient low\-rank projection\.arXiv preprint arXiv:2403\.03507\.Cited by:[§2](https://arxiv.org/html/2610.40858#S2.SS0.SSS0.Px1.p1.1)\.

## Supplementary Material Overview

This supplementary material documents the training configuration, the data protocol, the controller baselines, and the per\-block effect of the reallocation, so that the claims in the main paper can be checked in detail\.

- •Appendix[A](https://arxiv.org/html/2610.40858#A1): model, optimizer, and schedule settings\.
- •Appendix[B](https://arxiv.org/html/2610.40858#A2): corpus, validation and probe slices, and the synthetic corrupted regime\.
- •Appendix[C](https://arxiv.org/html/2610.40858#A3): hardware and software environment, and the cost of the reported runs\.
- •Appendix[D](https://arxiv.org/html/2610.40858#A4): the reference trainer, the risk\-neutral FARO\-Adaptive, and the scalar controllers used as negative controls\.
- •Appendix[E](https://arxiv.org/html/2610.40858#A5): what the allocation does to per\-block budgets, and the boundaries of the effect\.
- •Appendix[G](https://arxiv.org/html/2610.40858#A7): the decomposition argument for the routing decision, and a proof that the ERC allocation step solves its risk program exactly\.

## Appendix AExperimental setup

#### Model\.

A GPT\-style transformer with 11 layers, width 768, 6 heads of dimension128128, vocabulary 50257, and sequence length scheduled from896896to20482048\. FARO partitions the parameters intoB=117B=117blocks, so each block is an attention projection bank, an MLP matrix, an embedding, or a group of normalization and bias parameters\. Seed0\-5 are 0,1,2,7,42,1024\.

#### Optimizers\.

Base updates follow the reference trainer\([Jordan et al\., 2024a](https://arxiv.org/html/2610.40858#bib.bib15)\): a Muon\-style orthogonalized rule for matrix blocks, with an Adam\-style rule for the remaining parameters\. FARO adds per\-block geometry selection over the candidate set\{coordinate,row\_column,spectral\}\\\{\\texttt\{coordinate\},\\texttt\{row\\\_column\},\\texttt\{spectral\}\\\}together with the freeze\-after\-settling rule, plus the budgetβb\\beta\_\{b\}of Eq\. \([2](https://arxiv.org/html/2610.40858#S3.E2)\)\. The candidate set is a registry, so additional arms can be added without changing the allocation\.

#### Training schedule\.

The learning\-rate and weight\-decay schedules are defined over 1270 steps with a terminal cooldown to a0\.15×0\.15\\timesfloor\. The budget experiments run 800 steps, at which point the schedule has not fully cooled; the reported losses are therefore final losses under a fixed 800\-step compute budget rather than converged losses\. Both arms of every comparison share the schedule, so the comparison is unaffected\.

#### ERC defaults\.

Risk\-EWMA decayγ=0\.9\\gamma=0\.9, winsor boundκ=3\.0\\kappa=3\.0, allocation capc=1\.4c=1\.4for every reported result, with the larger2\.62\.6studied in the cap sweep \(Table[4](https://arxiv.org/html/2610.40858#A1.T4)\) and ERC clamp bounds inherited from the global routing bounds unless overridden\. The risk signal is refreshed once per frozen step and the allocation reads only the previous step’s value\.

#### Evaluation\.

Final validation cross\-entropy at the last step, computed on a fixed validation prefix so that all runs are directly comparable\. Each \(seed, arm\) is run 2 times and averaged; the standard deviations in Table[1](https://arxiv.org/html/2610.40858#S4.T1)are across seeds of those per\-seed averages\.

#### Noise floor\.

Repeating an identical configuration gives a spread of roughly0\.0040\.004–0\.010\.01nats\. We treat smaller differences as indistinguishable from noise and require the sign to agree across seeds before reporting a gain\.

#### Ablation protocol\.

Each ablation varies exactly one knob—the allocation cap, the floor or ceiling clamp, or the base learning rate—against the same FARO\-Adaptive baseline at the same 800\-step budget\. Per\-cell seed counts are given in the table captions\.

Table 2:Notation \(reference for Section[3](https://arxiv.org/html/2610.40858#S3)\)\.Table 3:Cap×\\timesbase\-LR grid,Δ\\Deltavs\. FARO\-Adaptive\. LR0\.5×0\.5\\timesuses33seeds, the remaining cells 6\.Table 4:Allocation\-cap sweep,Δ\\Deltavs\. FARO\-Adaptive\. Cap 1\.2 uses33seeds; the rest use 6\. A larger cap permits a stronger tilt\.Table 5:ERC clamp sweep at cap 2\.6\. Lowering the floor deepens the gain; raising the ceiling moves it less\. Floor rows use 6 seeds, ceiling rows33\.Table 6:Per\-checkpoint gain \(FARO\-ERC−\-FARO\-Adaptive, negative is better\), mean over 6 seeds\. FARO\-ERC is ahead at every sampled checkpoint from step 250; the advantage peaks mid\-training and narrows\.Figure 5:The two single\-knob sweeps of Tables[4](https://arxiv.org/html/2610.40858#A1.T4)and[5](https://arxiv.org/html/2610.40858#A1.T5),Δ\\Deltavs\. FARO\-Adaptive \(negative is better\)\. \(a\) Raising the allocation cap deepens the gain until the downward clamp binds\. \(b\) Lowering the floor \(red\) deepens the gain, while raising the ceiling \(blue\) moves it less\. Stars mark the default floor of0\.50\.5\.Table 7:Scalar global\-budget controllers under the same matched\-pair protocol \(33seeds\)\. Neither beats FARO\-Adaptive, and neither wins a majority of seed pairs\.

## Appendix BDataset and splits

#### Corpus\.

All experiments train on FineWeb token shards\([Penedo et al\., 2024](https://arxiv.org/html/2610.40858#bib.bib30)\)in the packeduint16format used by the reference trainer\([Jordan et al\., 2024a](https://arxiv.org/html/2610.40858#bib.bib15)\)\. The local corpus has99training shards \(≈9\.0×108\\approx 9\.0\\times 10^\{8\}tokens\) and one validation shard \(≈1\.0×108\\approx 1\.0\\times 10^\{8\}tokens\)\.

#### Validation and probe slices\.

Validation loss uses a fixed prefix of the validation shard \(10,485,76010\{,\}485\{,\}760tokens\), held constant across runs\. FARO’s counterfactual probe \(used for routing, Eq\. \([1](https://arxiv.org/html/2610.40858#S3.E1)\)\) and the risk measurement read a disjoint slice \(1,048,5761\{,\}048\{,\}576tokens\) immediately after that prefix, so no reported validation number is computed on probe tokens\.

#### Splits and fairness\.

The train/validation split is fixed\. Every matched pair consumes the identical data order and geometry/freeze timeline, and only the budget controller differs, so the pair is a controlled comparison\. No test split is used\.

#### Corrupted regime\.

The “scatter” regime used in Table[1](https://arxiv.org/html/2610.40858#S4.T1)replaces a fixed fraction \(60%60\\%\) of tokens at random positions, selected by a deterministic per\-microbatch hash\. Difficulty is therefore not temporally clustered, which stresses the controller under noisy risk\. We use this regime as a stress test of the controller, reported alongside the clean protocol rather than as a separate benchmark\.

## Appendix CHardware and software

#### Hardware\.

All runs used NVIDIA H800 \(80 GB\) GPUs, one GPU per run \(torchrunwithnproc\_per\_node=1=1\)\. Matched pairs were sometimes executed concurrently on distinct GPUs; per\-run wall\-clock was monitored so that GPU contention could not be mistaken for a method difference\.

#### Software\.

Python 3\.12, PyTorch 2\.10 with CUDA 12\.8, no distributed collectives inside the training loop, and the mixed\-precision settings of the reference trainer\.

#### Cost\.

A single 800\-step run takes roughly66–1010minutes of H800 time\. The main table is 6 seeds×\\times2 repetitions×\\timestwo arms, and the ablation grids multiply this on the same single\-GPU setting\.

#### Reproducibility\.

Runs are seeded\. The matched\-pair design fixes the data order, the geometry routing, and the freeze timeline across arms, so a pair differs only in the budget controller, and per\-step metrics and run configuration are written for every run\.

## Appendix DBaselines

#### Optimizer landscape\.

Table[8](https://arxiv.org/html/2610.40858#A4.T8)is not a head\-to\-head against the allocation axis; it locates the budget study in the optimizer landscape\. These optimizers were trained to the end of the schedule and evaluated identically, so the comparison is between whole optimizers under one recipe, not between budget controllers\. The result we draw on is that the FARO\-Adaptive arm \(3\.2780±\\pm0\.0008\) matches NorMuonAndAdam \(3\.2794±\\pm0\.0031\) and clearly beats Muon and Adam: under one framework, with no per\-layer geometry hand\-set, it reaches the level of the strongest hand\-designed recipe\. FARO\-ERC is the further step that improves on this arm \(Table[1](https://arxiv.org/html/2610.40858#S4.T1)\)\.

Table 8:Optimizer baselines on FineWeb at the end of the schedule, with the per\-seed values behind each mean\. Adam and Muon/NorMuonAndAdam use 4 and 3 seeds respectively; the FARO\-Adaptive baseline is the same framework with the budget controller off\. \(the matched\-pair study against it, Table[1](https://arxiv.org/html/2610.40858#S4.T1), uses 6\)\. Std is the population standard deviation\.
#### Why this comparison is not the paper’s claim\.

The numbers in Table[8](https://arxiv.org/html/2610.40858#A4.T8)come from different training lengths and seed counts than the main table, and involve different update rules\. They are not evidence that ERC beats a particular optimizer, and they are not a ranking of the optimizers against each other\. Each entry runs at its harness\-default setting\. The variants also differ in more than the update rule: they share neither an update cadence nor a weight\-decay scaling\. The table is therefore context, not a tuning\-matched comparison\. The row our claim rests on is FARO\-Adaptive, the same framework with routing frozen and the budget controller off\. It lands on NorMuonAndAdam, which we therefore take as the human\-prior reference\. The controlled evidence for ERC itself is the matched\-pair comparison in Table[1](https://arxiv.org/html/2610.40858#S4.T1)\. This is also why we treat the matrix\-optimizer family as candidate arms inside FARO rather than as competitors\. Improvements there change the geometry, and our question is how to allocate among geometries once they are fixed\.

#### FARO\-Adaptive \(risk\-neutral reference\)\.

The reference arm of the budget study is FARO\-Adaptive: FARO with its geometry\-derived frozen budgets and no risk controller\. It is the risk\-neutral point of the risk–reward spectrum, and everyΔ\\Deltain the main paper is measured against it\.

#### Scalar controllers\.

Two scalar controllers serve as negative controls, and both rescale a single global budget\. A fractional\-Kelly rule multiplies the global budget by a factor read from a probe return\([Kelly, 1956](https://arxiv.org/html/2610.40858#bib.bib38)\)\. DS\-FARO multiplies it by the shortfall of a held\-out probe return below its trailing median\. Both are evaluated as matched pairs \(Table[7](https://arxiv.org/html/2610.40858#A1.T7)\); neither beats FARO\-Adaptive, which is the evidence that the useful signal is cross\-block\.

#### Reference trainer\.

The base model, data pipeline, and matrix updates follow the publicmodded\-nanogpttrainer\([Jordan et al\., 2024a](https://arxiv.org/html/2610.40858#bib.bib15)\); its schedule and default hyperparameters are used unchanged for every part of training that is not the budget controller\.

## Appendix EWhat the allocation does to the budgets

The informative case study for an optimizer is not an input–output example but what the controller does to per\-block budgets during a run\. Table[9](https://arxiv.org/html/2610.40858#A5.T9)summarizes the multipliers actually applied in a representative run \(seed00, clean data\)\.

Table 9:FARO\-ERC allocation diagnostics for a representative run\. The multiplier is the factor applied to a block’s frozen budget:1\.01\.0means unchanged, less than11means shrunk\.#### Reading the distribution\.

The allocation is budget\-neutral by construction, since the capped solve \([9](https://arxiv.org/html/2610.40858#S3.E9)\) preserves the total, yet the realized distribution of multipliers is mildly asymmetric: at the default cap, 58\.7% of blocks are shrunk and 40\.8% are expanded, giving a mean applied multiplier of 0\.980, i\.e\. a small net trim of the risky tail\. The shrink side is the stronger lever: lowering the floor, which lets the shrink go further, deepens the gain more than raising the ceiling, which lets the expand go further \(Table[5](https://arxiv.org/html/2610.40858#A1.T5)\)\. The method therefore helps mostly by dampening the riskiest block updates rather than by amplifying the safest ones\.

#### Exactness of neutrality\.

The multiplier computation is exactly budget\-neutral, verified to below10−610^\{\-6\}in unit tests\. The effective budgets recorded at runtime can deviate slightly \(around10−210^\{\-2\}relative\) because the result is subsequently re\-clipped to the global routing bounds; the FARO\-ERC\-specific clamp exists to relax that coupling, so that tightening either bound does not silently move the other\.

#### Behaviour under a corrupted stream\.

On the corrupted stream the realized per\-block risks become noisier, so the reallocation separates high\-risk from low\-risk blocks less cleanly and the margin narrows \(Table[1](https://arxiv.org/html/2610.40858#S4.T1)\)\. The controller still lowers the mean loss and wins 5/6 seeds: it degrades gracefully under noise rather than breaking, which is the behaviour we want from a risk\-driven rule\.

## Appendix FSystem overhead: the exploration phase is not free

Automatic search costs something a fixed recipe does not\. FARO spends an exploration phase on shadow candidate updates, the held\-out probe, and the risk statistics, on top of compilation and warmup\. Table[10](https://arxiv.org/html/2610.40858#A6.T10)reports the full per\-run system cost of the four arms on the same schedule\.

Table 10:Per\-run system cost on the full 1270\-step schedule: throughput, training time \(optimizer steps\), wall time \(incl\. compilation and warmup\), and peak memory\.#### Against Muon\.

FARO\-Adaptive is not slower than Muon: throughput is 574,519\.6 vs 544,056\.2 tokens/s and training time 589\.748 vs 622\.770 s, so the extra machinery pays for itself in step throughput\. Its peak memory is higher \(42,489 vs 34,528 MiB\), the price of holding the exploration state\.

#### Against the human recipe\.

NorMuonAndAdam is the strongest hand\-designed baseline\. Against it, FARO\-Adaptive’s average training time is 5\.7% higher \(589\.748 vs 558\.039 s\) and its peak allocated memory 7\.7% higher \(42,489 vs 39,435 MiB\)\. Both sit in a range we consider manageable\. They also shrink as training stabilizes\. Past the exploration phase the steady\-state step time and memory of the two arms are close, so the gap is concentrated early rather than carried through the run\.

#### Where the overhead comes from, and what bounds it\.

The additional cost is an exploration\-phase cost\. The reported peak istorch\.cuda\.max\_memory\_allocated\(\)andreservedover the whole process, from compilation and warmup through the end of training, without a reset before the first optimizer step\. It therefore describes the peak the program has to accommodate in practice, not the frozen steady\-state footprint\. The added time is a small fraction of a full\-scale run and is acceptable for the training budgets we target, and the extra peak memory is a fixed resident term that can be reduced by offloading the exploration state\. We report both, rather than the frozen steady state alone, so the cost of the automation is not hidden\.

## Appendix GA decomposition argument for routing and allocation

FARO makes two decisions per block: a geometry, chosen by measured return, and a share of a fixed total budget, set by risk contribution\. Neither decision is made jointly over all blocks\. For routing, this appendix bounds how far the per\-block choice can be from the joint one: the linearized return decomposes exactly, and what the per\-block choice ignores enters only at second order, through a residual that the bound carries\. For allocation, it shows that the budget coupling is represented exactly rather than approximated, so the step is an exact solve of the program it comes from\. The two are different strengths of statement, and Section[G\.7](https://arxiv.org/html/2610.40858#A7.SS7)separates them\. Neither argument claims that the reduced decisions drive the original non\-convex training run to a global optimum\.

### G\.1Setup and standing assumptions

We keep the notation of Section[3](https://arxiv.org/html/2610.40858#S3)\. Parameters are split intoB=117B=117blocksθ=\(θ1,…,θB\)\\theta=\(\\theta\_\{1\},\\dots,\\theta\_\{B\}\)\. A decision for blockbbis a pairxb=\(ϕb,βb\)x\_\{b\}=\(\\phi\_\{b\},\\beta\_\{b\}\), a geometry and a budgetβb≥0\\beta\_\{b\}\\geq 0\. The step it induces is

Δ​θb​\(xb\)=α​βb​db​\(ϕb\),Δ​θ​\(x\)=∑b=1BΔ​θb​\(xb\),\\begin\{gathered\}\\Delta\\theta\_\{b\}\(x\_\{b\}\)=\\alpha\\,\\beta\_\{b\}\\,d\_\{b\}\(\\phi\_\{b\}\),\\\\ \\Delta\\theta\(x\)=\\sum\_\{b=1\}^\{B\}\\Delta\\theta\_\{b\}\(x\_\{b\}\),\\end\{gathered\}\(10\)with base learning rateα\\alphaand unit directiondb​\(ϕ\)d\_\{b\}\(\\phi\)\. The realized one\-step loss increment on microbatchξ\\xiis

Qξ​\(x\)=ℒ⁡\(θ\+Δ​θ​\(x\),ξ\)−ℒ⁡\(θ,ξ\)\.Q\_\{\\xi\}\(x\)=\\mathcal\{L\}\(\\theta\+\\Delta\\theta\(x\);\\xi\)\-\\mathcal\{L\}\(\\theta;\\xi\)\.\(11\)Four assumptions hold throughout\. They are mild, and each is checkable\.

- A1*Disjoint supports\.*Blocks act on non\-overlapping parameter coordinates, so⟨Δ​θb,Δ​θc⟩=0\\langle\\Delta\\theta\_\{b\},\\Delta\\theta\_\{c\}\\rangle=0forb≠cb\\neq c\.
- A2*Frozen risk\.*The per\-block riskrbr\_\{b\}is measured from the previous step and held fixed while the allocation is solved, so the current gradient never sets the budget that acts on it\.
- A3*Separable quadratic risk\.*Allocation scores a budget vector by the block\-diagonal quadratic riskRRof \([19](https://arxiv.org/html/2610.40858#A7.E19)\), with each block’s volatility read from its measured risk\. Cross\-block risk is not modeled\. This is the model the method uses, not a claim that the true one\-step objective is diagonal\.
- A4*Attainment\.*Each local feasible set𝒳b\\mathcal\{X\}\_\{b\}is nonempty and compact, and the joint set is the product𝒳=∏b𝒳b\\mathcal\{X\}=\\prod\_\{b\}\\mathcal\{X\}\_\{b\}when the guards are local\.

### G\.2The linearized return decomposes exactly

Letgξ,bg\_\{\\xi,b\}be the gradient block of blockbbon microbatchξ\\xi\. By A1 the first\-order term separates with no further hypothesis: one backward pass through the single global loss already delivers everygξ,bg\_\{\\xi,b\}\.

###### Proposition 1\(Exact decomposition of the linearized return\)\.

For every feasiblexx,−gξ⊤Δθ\(x\)=∑b−gξ,b⊤Δθb\(xb\)\-g\_\{\\xi\}^\{\\top\}\\Delta\\theta\(x\)=\\sum\_\{b\}\-g\_\{\\xi,b\}^\{\\top\}\\Delta\\theta\_\{b\}\(x\_\{b\}\)identically, and under A4

arg​maxx∈𝒳−gξ⊤​Δ​θ​\(x\)=∏b=1Barg​maxxb∈𝒳b−gξ,b⊤​Δ​θb​\(xb\)\.\\begin\{gathered\}\\operatorname\*\{arg\\,max\}\_\{x\\in\\mathcal\{X\}\}\\,\-g\_\{\\xi\}^\{\\top\}\\Delta\\theta\(x\)\\\\ =\\prod\_\{b=1\}^\{B\}\\;\\operatorname\*\{arg\\,max\}\_\{x\_\{b\}\\in\\mathcal\{X\}\_\{b\}\}\\,\-g\_\{\\xi,b\}^\{\\top\}\\Delta\\theta\_\{b\}\(x\_\{b\}\)\.\\end\{gathered\}\(12\)The joint maximum equals the sum of the per\-block maxima\.

###### Proof\.

Take anyxb⋆x\_\{b\}^\{\\star\}attaining thebb\-th block maximum\. The product domain makesx⋆=\(x1⋆,…,xB⋆\)x^\{\\star\}=\(x\_\{1\}^\{\\star\},\\dots,x\_\{B\}^\{\\star\}\)feasible, and per\-block optimality summed overbbgives−gξ⊤​Δ​θ​\(x\)≤−gξ⊤​Δ​θ​\(x⋆\)\-g\_\{\\xi\}^\{\\top\}\\Delta\\theta\(x\)\\leq\-g\_\{\\xi\}^\{\\top\}\\Delta\\theta\(x^\{\\star\}\)for every feasiblexx\. Conversely, if a component of a joint maximizer were not a block maximizer, replacing that component would raise the joint value\. ∎

The router scores the probe return of Eq\. \([1](https://arxiv.org/html/2610.40858#S3.E1)\) by finite difference, so it never forms the inner product below\. Writeρb,ξ​\(ϕ\)=−⟨gξ,b,db​\(ϕ\)⟩\\rho\_\{b,\\xi\}\(\\phi\)=\-\\langle g\_\{\\xi,b\},d\_\{b\}\(\\phi\)\\ranglefor the first\-order model of that return at the current parameters, withgξ,bg\_\{\\xi,b\}the gradient block of the loss on microbatchξ\\xi\. The two agree to first order when the probe and the training microbatch are drawn from the same distribution and the step is small, which is the regime the argument below covers\. Proposition[1](https://arxiv.org/html/2610.40858#Thmtheorem1)applies to this first\-order model, and through it to the routing decision as implemented\.

What such a decision ignores first appears beyond the linear term: cross\-block curvaturedb⊤​Hb​c​dcd\_\{b\}^\{\\top\}H\_\{bc\}\\,d\_\{c\}, cross\-block return covarianceΣb​c\\Sigma\_\{bc\}, the function\-space cross metricFb​cF\_\{bc\}, and the joint tail of the loss increments\. The next subsection bounds the first three\.

### G\.3A bounded cross\-block residual

Write the one\-step objective as separable local utilities plus a residual,

J⁡\(x\)=∑b=1BJb​\(xb\)\+Rt​\(x\),supx∈𝒳\|Rt​\(x\)\|≤εt,\\begin\{gathered\}J\(x\)=\\sum\_\{b=1\}^\{B\}J\_\{b\}\(x\_\{b\}\)\+R\_\{t\}\(x\),\\\\ \\sup\_\{x\\in\\mathcal\{X\}\}\|R\_\{t\}\(x\)\|\\leq\\varepsilon\_\{t\},\\end\{gathered\}\(13\)where the local utility is the linearized return the router actually scores,

Jb​\(ϕ,β\)=ρb​\(ϕ\)​β,J\_\{b\}\(\\phi,\\beta\)=\\rho\_\{b\}\(\\phi\)\\,\\beta,\(14\)so that at the reference budgets the per\-block maximizer ofJbJ\_\{b\}isarg⁡maxϕ​ρb​\(ϕ\)\\arg\\max\_\{\\phi\}\\rho\_\{b\}\(\\phi\), which is the routing rule of Section[3\.1](https://arxiv.org/html/2610.40858#S3.SS1)\. Routing is evaluated with the budget held at its reference valueβ¯b\\bar\{\\beta\}\_\{b\}, as in FARO\-Adaptive, so during routingxbx\_\{b\}reduces to the geometryϕb\\phi\_\{b\}and𝒳b\\mathcal\{X\}\_\{b\}to the geometry set; the budget only becomes a free variable in the allocation step of Section[G\.6](https://arxiv.org/html/2610.40858#A7.SS6)\. Everything the linearization drops sits inRtR\_\{t\}\. Its quadratic partRtquadR\_\{t\}^\{\\mathrm\{quad\}\}collects curvature, return variance, and the function\-space metric; its tail part collects the joint tail of the loss increments\. If the projected curvature, return covariance, and function\-space metric split into block\-diagonal and cross parts,CH=CHblk\+EHC\_\{H\}=C\_\{H\}^\{\\mathrm\{blk\}\}\+E\_\{H\},Σ=Σblk\+EΣ\\Sigma=\\Sigma^\{\\mathrm\{blk\}\}\+E\_\{\\Sigma\}, andCF=CFblk\+EFC\_\{F\}=C\_\{F\}^\{\\mathrm\{blk\}\}\+E\_\{F\}, then on a trust region‖β‖2≤ϱ\\\|\\beta\\\|\_\{2\}\\leq\\varrhothe quadratic part is bounded by the spectral norms of the cross parts,

\|Rtquad​\(x\)\|≤ϱ22​\(‖EH‖2\+ς​‖EΣ‖2\+ν​‖EF‖2\),\|R\_\{t\}^\{\\mathrm\{quad\}\}\(x\)\|\\leq\\tfrac\{\\varrho^\{2\}\}\{2\}\\bigl\(\\\|E\_\{H\}\\\|\_\{2\}\+\\varsigma\\\|E\_\{\\Sigma\}\\\|\_\{2\}\+\\nu\\\|E\_\{F\}\\\|\_\{2\}\\bigr\),\(15\)with weightsς≥0\\varsigma\\geq 0andν≥0\\nu\\geq 0on the variance and function\-space terms\. The tail part is not covered by \([15](https://arxiv.org/html/2610.40858#A7.E15)\) and is left insideεt\\varepsilon\_\{t\}\. Two points keep \([15](https://arxiv.org/html/2610.40858#A7.E15)\) honest\. First, the bound is informative only when the cross parts are small relative to the block\-diagonal ones; it does not make them small\. Second, the trust radiusϱ\\varrhois not free: a fixed total budget pins‖β‖2≤‖β‖1=Bud\\\|\\beta\\\|\_\{2\}\\leq\\\|\\beta\\\|\_\{1\}=\\mathrm\{Bud\}, so shrinkingϱ\\varrhois not available here, and the cross terms must be controlled by the block structure rather than by a small step\. Whether they are is empirical, andEHE\_\{H\},EΣE\_\{\\Sigma\}, andEFE\_\{F\}are measurable: finite differences, retained cross\-covariance, and activation or logit drift respectively\.

### G\.4Exact and approximate composition of the routing decision

###### Theorem 2\(Exact composition\)\.

IfRt≡0R\_\{t\}\\equiv 0in \([13](https://arxiv.org/html/2610.40858#A7.E13)\) and𝒳=∏b𝒳b\\mathcal\{X\}=\\prod\_\{b\}\\mathcal\{X\}\_\{b\}satisfies A4, then

arg​maxx∈𝒳⁡J​\(x\)=∏b=1Barg​maxxb∈𝒳b⁡Jb​\(xb\)\.\\operatorname\*\{arg\\,max\}\_\{x\\in\\mathcal\{X\}\}J\(x\)=\\prod\_\{b=1\}^\{B\}\\operatorname\*\{arg\\,max\}\_\{x\_\{b\}\\in\\mathcal\{X\}\_\{b\}\}J\_\{b\}\(x\_\{b\}\)\.\(16\)Every joint maximizer is a tuple of local maximizers, and conversely\.

The proof repeats Proposition[1](https://arxiv.org/html/2610.40858#Thmtheorem1)withJJin place of the linear term\. The theorem is the ideal case; the next bound covers the residual\.

###### Theorem 3\(2​ε2\\varepsilonsuboptimality\)\.

Suppose \([13](https://arxiv.org/html/2610.40858#A7.E13)\) holds on a common nonempty𝒳\\mathcal\{X\}\. Letx⋆x^\{\\star\}maximizeJJand letx^\\hat\{x\}maximize the factorized objectiveJfac=∑bJbJ\_\{\\mathrm\{fac\}\}=\\sum\_\{b\}J\_\{b\}\. Then

J⁡\(x⋆\)−J⁡\(x^\)≤2​εt\.J\(x^\{\\star\}\)\-J\(\\hat\{x\}\)\\leq 2\\varepsilon\_\{t\}\.\(17\)

###### Proof\.

From \([13](https://arxiv.org/html/2610.40858#A7.E13)\),J⁡\(x⋆\)≤Jfac​\(x⋆\)\+εtJ\(x^\{\\star\}\)\\leq J\_\{\\mathrm\{fac\}\}\(x^\{\\star\}\)\+\\varepsilon\_\{t\}\. Optimality ofx^\\hat\{x\}forJfacJ\_\{\\mathrm\{fac\}\}givesJfac​\(x⋆\)≤Jfac​\(x^\)J\_\{\\mathrm\{fac\}\}\(x^\{\\star\}\)\\leq J\_\{\\mathrm\{fac\}\}\(\\hat\{x\}\), and \([13](https://arxiv.org/html/2610.40858#A7.E13)\) again givesJfac​\(x^\)≤J⁡\(x^\)\+εtJ\_\{\\mathrm\{fac\}\}\(\\hat\{x\}\)\\leq J\(\\hat\{x\}\)\+\\varepsilon\_\{t\}\. Chaining the three givesJ⁡\(x⋆\)≤J⁡\(x^\)\+2​εtJ\(x^\{\\star\}\)\\leq J\(\\hat\{x\}\)\+2\\varepsilon\_\{t\}\. ∎

### G\.5When routing is safe: a margin condition

###### Proposition 4\(Route margin\)\.

Letx^\\hat\{x\}maximizeJfacJ\_\{\\mathrm\{fac\}\}, with joint geometryϕ^\\hat\{\\boldsymbol\{\\phi\}\}, and let

m=maxx∈𝒳Jfac\(x\)−maxx∈𝒳:ϕ⁡\(x\)≠ϕ^Jfac\(x\)m=\\max\_\{x\\in\\mathcal\{X\}\}J\_\{\\mathrm\{fac\}\}\(x\)\-\\max\_\{x\\in\\mathcal\{X\}:\\,\\boldsymbol\{\\phi\}\(x\)\\neq\\hat\{\\boldsymbol\{\\phi\}\}\}J\_\{\\mathrm\{fac\}\}\(x\)\(18\)be the factorized margin over every other joint geometry\. Ifm\>2​εtm\>2\\varepsilon\_\{t\}, then every maximizer of the true objectiveJJcarries the geometryϕ^\\hat\{\\boldsymbol\{\\phi\}\}\.

###### Proof\.

For anyxxwithϕ⁡\(x\)≠ϕ^\\boldsymbol\{\\phi\}\(x\)\\neq\\hat\{\\boldsymbol\{\\phi\}\}, \([13](https://arxiv.org/html/2610.40858#A7.E13)\) givesJ⁡\(x^\)≥Jfac​\(x^\)−εtJ\(\\hat\{x\}\)\\geq J\_\{\\mathrm\{fac\}\}\(\\hat\{x\}\)\-\\varepsilon\_\{t\}andJ⁡\(x\)≤Jfac​\(x\)\+εt≤Jfac​\(x^\)−m\+εtJ\(x\)\\leq J\_\{\\mathrm\{fac\}\}\(x\)\+\\varepsilon\_\{t\}\\leq J\_\{\\mathrm\{fac\}\}\(\\hat\{x\}\)\-m\+\\varepsilon\_\{t\}\. HenceJ⁡\(x^\)−J⁡\(x\)≥m−2​εt\>0J\(\\hat\{x\}\)\-J\(x\)\\geq m\-2\\varepsilon\_\{t\}\>0\. ∎

Proposition[4](https://arxiv.org/html/2610.40858#Thmtheorem4)gives the freeze rule a meaning: a route may be frozen when the incumbent–challenger margin covers the residual, not merely when the ranking stops moving\. Our implementation uses the settling window as a cheap proxy for that margin\.

### G\.6The allocation step is an exact dual decomposition

Routing fixes the geometry of every block\. Allocation then chooses the budget vectorβ\\beta, and unlike routing it faces a constraint that couples all blocks at once, the fixed total∑bβb=Bud\\sum\_\{b\}\\beta\_\{b\}=\\mathrm\{Bud\}of Eq\. \([3](https://arxiv.org/html/2610.40858#S3.E3)\)\. That constraint is not a product domain, so A4 does not apply to it\. The next result shows the coupling enters through exactly one scalar, and that solving for that scalar recovers the exact optimum rather than an approximation of it\.

Under A2 and A3, take the allocation objective to be the separable quadratic risk

R⁡\(β\)=∑b=1Br~b2​β¯b​βb2,R\(\\beta\)=\\sum\_\{b=1\}^\{B\}\\frac\{\\tilde\{r\}\_\{b\}\}\{2\\bar\{\\beta\}\_\{b\}\}\\,\\beta\_\{b\}^\{2\},\(19\)read as the risk of a diagonal portfolio whose position in blockbbisβb\\beta\_\{b\}, whose volatility is the winsorized riskr~b\\tilde\{r\}\_\{b\}of Eq\. \([6](https://arxiv.org/html/2610.40858#S3.E6)\), and whose reference weight isβ¯b\\bar\{\\beta\}\_\{b\}\. FARO\-ERC solves

minβ⁡R⁡\(β\)s\.t\.∑b=1Bβb=Bud,βb∈\[β¯b/c,c​β¯b\],\\begin\{gathered\}\\min\_\{\\beta\}\\;R\(\\beta\)\\quad\\text\{s\.t\.\}\\quad\\sum\_\{b=1\}^\{B\}\\beta\_\{b\}=\\mathrm\{Bud\},\\\\ \\beta\_\{b\}\\in\\bigl\[\\bar\{\\beta\}\_\{b\}/c,\\;c\\,\\bar\{\\beta\}\_\{b\}\\bigr\],\\end\{gathered\}\(20\)the box being the allocation cap of Eq\. \([8](https://arxiv.org/html/2610.40858#S3.E8)\)\.

###### Theorem 5\(ERC bisection solves \([20](https://arxiv.org/html/2610.40858#A7.E20)\) exactly\)\.

Under A2 and A3, withc\>1c\>1so that the box in \([20](https://arxiv.org/html/2610.40858#A7.E20)\) is nonempty, \([20](https://arxiv.org/html/2610.40858#A7.E20)\) has a unique solution, and it is

βb⋆=β¯b​clip⁡\(λ⋆r~b,c−1,c\),λ⋆​solves​Λ​\(λ⋆\)=Bud,\\begin\{gathered\}\\beta\_\{b\}^\{\\star\}=\\bar\{\\beta\}\_\{b\}\\,\\operatorname\{clip\}\\\!\\Big\(\\frac\{\\lambda^\{\\star\}\}\{\\tilde\{r\}\_\{b\}\},\\,c^\{\-1\},\\,c\\Big\),\\\\ \\lambda^\{\\star\}\\text\{ solves \}\\Lambda\(\\lambda^\{\\star\}\)=\\mathrm\{Bud\},\\end\{gathered\}\(21\)withΛ\\Lambdaas defined in Eq\. \([9](https://arxiv.org/html/2610.40858#S3.E9)\) andr~b\\tilde\{r\}\_\{b\}the winsorized risk of Eq\. \([6](https://arxiv.org/html/2610.40858#S3.E6)\)\. Algorithm[1](https://arxiv.org/html/2610.40858#alg1)therefore returns the exact optimizer of \([20](https://arxiv.org/html/2610.40858#A7.E20)\), not a heuristic approximation of it\.

###### Proof\.

The objective \([19](https://arxiv.org/html/2610.40858#A7.E19)\) is strictly convex on the box and the feasible set is convex and compact, so the solution exists and is unique\. Associate the multiplierλ\\lambdawith the total constraint andμb±≥0\\mu\_\{b\}^\{\\pm\}\\geq 0with the box\. The Lagrangian is separable acrossbb, and stationarity inβb\\beta\_\{b\}givesr~b​βb/β¯b=λ\+μb\+−μb−\\tilde\{r\}\_\{b\}\\beta\_\{b\}/\\bar\{\\beta\}\_\{b\}=\\lambda\+\\mu\_\{b\}^\{\+\}\-\\mu\_\{b\}^\{\-\}\. Solving the scalar problem per block yieldsβb=β¯b​clip⁡\(λ/r~b,c−1,c\)\\beta\_\{b\}=\\bar\{\\beta\}\_\{b\}\\operatorname\{clip\}\(\\lambda/\\tilde\{r\}\_\{b\},c^\{\-1\},c\): interior blocks satisfyβb=β¯b​λ/r~b\\beta\_\{b\}=\\bar\{\\beta\}\_\{b\}\\lambda/\\tilde\{r\}\_\{b\}, and blocks at a bound are pinned to it\. The total constraint reduces toΛ⁡\(λ\)=Bud\\Lambda\(\\lambda\)=\\mathrm\{Bud\}, andΛ\\Lambdais continuous and non\-decreasing inλ\\lambdawithΛ⁡\(0\)=c−1​Bud\\Lambda\(0\)=c^\{\-1\}\\mathrm\{Bud\}andΛ⁡\(λ\)→c​Bud\\Lambda\(\\lambda\)\\to c\\,\\mathrm\{Bud\}asλ→∞\\lambda\\to\\infty\. Sincec\>1c\>1, a root exists\.Λ\\Lambdais non\-decreasing, so the root may be non\-unique when the box saturates every block, butβ⋆\\beta^\{\\star\}is unique by strict convexity and every root returns it\. Bisection recovers one\. ∎

The theorem has three consequences that the paper relies on\. First, the budget coupling is carried by the single scalarλ⋆\\lambda^\{\\star\}, so the allocation decomposes intoBBclosed\-form scalar solves plus one one\-dimensional root; no cross\-block covariance is estimated\. This is not a scalar controller\. A scalar controller multiplies every budget by one common factor, so it leaves the ratiosβb/βc\\beta\_\{b\}/\\beta\_\{c\}unchanged and cannot move mass between blocks\. Hereλ⋆\\lambda^\{\\star\}is a price attached to the total constraint, and each block responds to it through its own risk,βb=β¯b​λ⋆/r~b\\beta\_\{b\}=\\bar\{\\beta\}\_\{b\}\\lambda^\{\\star\}/\\tilde\{r\}\_\{b\}; the ratios therefore change, in the direction the risks dictate\. Second, the optimum equalizes risk contributions\. Writesb​\(β\):=r~b​βb/β¯bs\_\{b\}\(\\beta\):=\\tilde\{r\}\_\{b\}\\beta\_\{b\}/\\bar\{\\beta\}\_\{b\}for blockbb’s contribution under the model, so that∂bR=sb\\partial\_\{b\}R=s\_\{b\}\. Every interior block is assigned the same contributionλ⋆\\lambda^\{\\star\}, and its budget is proportional toβ¯b/r~b\\bar\{\\beta\}\_\{b\}/\\tilde\{r\}\_\{b\}; when the reference budgets coincide this is the classical equal\-risk\-contribution conditionr~b​βb=const\\tilde\{r\}\_\{b\}\\beta\_\{b\}=\\text\{const\}\. Third, the contributions that are not equal are exactly the ones the cap pins: a block at its lower bound contributes less thanλ⋆\\lambda^\{\\star\}and one at its upper bound contributes more\. Equal risk contribution is thus derived from the KKT condition rather than imposed\.

Two features of the implementation follow from the same proof, and they act in different places\. Winsorizingrbr\_\{b\}intor~b\\tilde\{r\}\_\{b\}sets the coefficients ofRR, so it defines the model rather than restricting it; it is what bounds the ratioλ⋆/r~b\\lambda^\{\\star\}/\\tilde\{r\}\_\{b\}and hence how far one block’s budget can move relative to another’s\. The capccrestricts the feasible box instead\. Theorem[5](https://arxiv.org/html/2610.40858#Thmtheorem5)is exact for the model and the box the method actually uses, so neither feature is a correction applied after the fact\. The optional ERC\-specific clamp\[βminerc,βmaxerc\]\[\\beta^\{\\mathrm\{erc\}\}\_\{\\min\},\\beta^\{\\mathrm\{erc\}\}\_\{\\max\}\]of Section[3\.3](https://arxiv.org/html/2610.40858#S3.SS3)tightens the box further and is exact for the same program; when it binds, the applied budgets are re\-clipped and neutrality holds only to the tolerance measured in Appendix[E](https://arxiv.org/html/2610.40858#A5)\.

### G\.7What the two results cover, and what they do not

The two decisions are covered at different strengths, and it is worth separating them\.

For routing, the statement is a bound on the loss\. With budgets held at their reference values, as in FARO\-Adaptive, the decision is a geometry choice over a product domain, so Theorem[3](https://arxiv.org/html/2610.40858#Thmtheorem3)applies directly: the routed decision is2​εt2\\varepsilon\_\{t\}\-suboptimal for the joint one\-step objective, and Proposition[4](https://arxiv.org/html/2610.40858#Thmtheorem4)says a route whose margin clears2​εt2\\varepsilon\_\{t\}is not overturned\. The objective here is the one\-step loss increment that the update induces, not the final validation loss; the two are linked by the training dynamics, which this appendix does not model\.

For allocation, the statement is exactness, not a loss bound\. Theorem[5](https://arxiv.org/html/2610.40858#Thmtheorem5)shows that the bisection returns the exact optimizer of the risk program \([20](https://arxiv.org/html/2610.40858#A7.E20)\), so the allocation step is not a heuristic and the equal\-risk\-contribution property is a consequence rather than an assumption\. What the theorem does not do is identify the risk program with the loss:RRis a risk potential, not−Jfac\-J\_\{\\mathrm\{fac\}\}, and no argument here boundsmax⁡Jfac\\max J\_\{\\mathrm\{fac\}\}over the simplex bymin⁡R\\min R\. So the composition of the two steps does not inherit a2​εt2\\varepsilon\_\{t\}loss bound\. The evidence that minimizingRRalso lowers the final loss is the matched\-pair comparison in Section[4\.2](https://arxiv.org/html/2610.40858#S4.SS2)plus the cap\-by\-LR ablation, and that is where we rest it\.

Two points follow about the budget constraint itself\. The constraint∑bβb=Bud\\sum\_\{b\}\\beta\_\{b\}=\\mathrm\{Bud\}is a coupling, not something that disappears when the total is fixed; fixing a total is what creates it\. What Theorem[5](https://arxiv.org/html/2610.40858#Thmtheorem5)establishes is that this coupling is represented exactly, by one price, rather than approximated\. Had we instead split the total into fixed per\-block capacitiesAbA\_\{b\}with∑bAb≤Bud\\sum\_\{b\}A\_\{b\}\\leq\\mathrm\{Bud\}, the factorized feasible set would be a strict subset of the coupled one and we would pay a pre\-assignment loss that \([15](https://arxiv.org/html/2610.40858#A7.E15)\) does not see; the exact dual avoids that particular loss\. It does not avoid the proxy question above, which is a different gap\.

### G\.8Scope

What the argument supports\. The linearized return decomposes exactly, so scoring each block on its own probe return is not an approximation at first order\. Under a bounded cross\-block residual the per\-block routing optimum is2​εt2\\varepsilon\_\{t\}\-suboptimal for the joint one\-step objective, and a route whose margin clears2​εt2\\varepsilon\_\{t\}is not overturned by the coupling\. The allocation step is the exact solution of a separable risk program, with the budget constraint carried by one scalar price and equal risk contributions following from it\.

What the argument does not support\. It gives no end\-to\-end convergence claim for the non\-convex run, and no claim that FARO\-ERC beats a particular hand\-designed optimizer\. It does not establish thatεt\\varepsilon\_\{t\}is small in our runs; that is empirical, and the residual is measurable but we do not estimate it here\. The allocation result is exactness of the solve, not a loss bound: no step here relatesmin⁡R\\min Rtomax⁡Jfac\\max J\_\{\\mathrm\{fac\}\}, so the two decisions do not compose into a single2​εt2\\varepsilon\_\{t\}statement, and the case for allocation lowering the loss rests on the experiments\. The risk signal is the measured update\-to\-weight ratio of Eq\. \([4](https://arxiv.org/html/2610.40858#S3.E4)\), a proxy for volatility, not a modeled tail\. A tail\-aware variant would need its own analysis: CVaR is subadditive, so local tail budgets do give a conservative joint bound, but that bound is a different guarantee than the one proved here, and we do not claim it\.