LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment

# LoCA: Backprop-Free Post-Training via Local Credit Assignment
Source: [https://arxiv.org/html/2608.03020](https://arxiv.org/html/2608.03020)
Linhan Xia1,4,†Rui Liu2,†Zhaofeng Zhang3,5,†Yihao Wang6Binrui Shen7Shengxin Zhu7,8\* 1University of Oklahoma2Imperial College London3University of Michigan 4Tencent5University of Edinburgh6University of Southern California 7Beijing Normal University8Beijing Normal\-Hong Kong Baptist University †These authors contributed equally to this work\. \* Corresponding Author:shengxin\.zhu@bnu\.edu\.cn

###### Abstract

Parameter\-efficient post\-training reduces the number of trainable parameters, but still requires repeated end\-to\-end backpropagation through the frozen backbone\. Every adaptation step therefore needs backward\-capable hardware and must store or recompute activations\. We ask whether this repeated backward chain can be replaced by a one\-time calibration\. We introduce Local Credit Assignment \(LoCA\), a two\-stage method for small\-shift adaptation\. One probe backward pass fits a low\-rank map at each transformer block from the final prediction error to a local hidden\-state correction\. LoCA then reuses these maps to form blockwise regression targets from forward activations and fits low\-rank adapters with closed\-form ridge solves\. No further backbone backward pass is required\. We evaluate LoCA on five discriminative benchmarks with Qwen2\.5 models from 0\.5B to 14B\. In 16 of 25 reported task–scale comparisons, LoCA yields lower evaluation cross\-entropy than the corresponding LoRA run\. Its measured full\-run GPU peak, including calibration, is 26–29% lower than LoRA’s\. After calibration, its CPU steady\-state memory is 36–52% lower and its per\-pass time is 43–48% lower\. A shared scale\-normalized candidate set is reused across all tested Qwen2\.5 sizes and on SmolLM2\-1\.7B\. LoCA thus amortizes global credit assignment into one calibration and enables later forward\-only tuning when repeated backpropagation is impractical\. The code associated with this paper is available[here](https://github.com/Xia12121/LoCA)\.

*K*eywordsBackpropagation\-Free⋅\\cdotFeedback Alignment⋅\\cdotParameter\-Efficient Tuning⋅\\cdotResource\-Constrained Training

## 1Introduction

Post\-training includes instruction tuning\[[29](https://arxiv.org/html/2608.03020#bib.bib1)\], domain adaptation\[[10](https://arxiv.org/html/2608.03020#bib.bib2)\], and preference alignment\[bai2022constitutional\]\. Even when only adapters are updated, these methods usually backpropagate through the frozen backbone\. They must store or recompute activations, propagate gradients, and maintain optimizer states\[[15](https://arxiv.org/html/2608.03020#bib.bib3)\]\. Fine\-tuning can therefore use up to12×12\\timesthe memory of inference\[[24](https://arxiv.org/html/2608.03020#bib.bib4)\]\. The need for repeated backpropagation substantially increases the cost, making the method less practical than forward\-only inference\.

Parameter\-efficient fine\-tuning reduces trainable parameters and optimizer states, but it retains the backward chain\. LoRA\[[13](https://arxiv.org/html/2608.03020#bib.bib5)\]and adapter\- or prompt\-based methods\[[12](https://arxiv.org/html/2608.03020#bib.bib6),[19](https://arxiv.org/html/2608.03020#bib.bib7),[17](https://arxiv.org/html/2608.03020#bib.bib8)\]still propagate gradients through the backbone\. For example, PEFT for a 13B model can require6×6\\timesinference memory\[[24](https://arxiv.org/html/2608.03020#bib.bib4)\]\. These methods reduce several costs of backpropagation, but do not remove repeated backward execution\.

Forward\-only methods offer another trade\-off\. MeZO estimates update directions from function values\[[34](https://arxiv.org/html/2608.03020#bib.bib11),[24](https://arxiv.org/html/2608.03020#bib.bib4)\]\. Its memory use is close to inference, while its convergence and tuning can be affected by estimator variance\[[9](https://arxiv.org/html/2608.03020#bib.bib13),[26](https://arxiv.org/html/2608.03020#bib.bib12)\]\. Feedback alignment uses fixed feedback matrices\[[20](https://arxiv.org/html/2608.03020#bib.bib14),[28](https://arxiv.org/html/2608.03020#bib.bib15)\], but often relies on forward weights adapting to those matrices\[[31](https://arxiv.org/html/2608.03020#bib.bib17)\]\. A frozen backbone cannot use this mechanism\. Local learning avoids global propagation\[[3](https://arxiv.org/html/2608.03020#bib.bib18),[27](https://arxiv.org/html/2608.03020#bib.bib19),[11](https://arxiv.org/html/2608.03020#bib.bib20)\], but its targets usually support representation learning or compression\[[14](https://arxiv.org/html/2608.03020#bib.bib21)\]\. Taken together, this trade\-off motivates us to combine their complementary strengths by using a proper scheduling\.

We study small, targeted changes to a strong pretrained model, a more common practical setting than training a model from scratch\. Our goal is to avoid the high computational cost in this regime\. Prior work uses low\-rank parameter updates for this setting\[[18](https://arxiv.org/html/2608.03020#bib.bib23),[1](https://arxiv.org/html/2608.03020#bib.bib24),[13](https://arxiv.org/html/2608.03020#bib.bib5)\]\. We apply the same local view to credit assignment\. When the representation change is small, we separate two tasks\. The first estimates how each block output should change\. The second fits adapter parameters to that change\.

We introduceLocal Credit Assignment fine\-tuning \(LoCA\)for this two\-stage setting\. During calibration, one probe backward pass fits a low\-rank feedback operator for each block\. During adaptation, these operators turn the final\-layer error into local adapter targets\. Each adapter is fitted by ridge regression\. After calibration, an outer iteration needs forward execution and local linear solves, but no backbone backward pass\. A device without backward support must receive operators calibrated for the same checkpoint and target distribution\.

In the tested discriminative tasks, LoCA recovers most of LoRA’s cross\-entropy improvement and uses less measured memory\. We also reuse one normalized candidate set across model sizes and on a second model family\. Results from individual cells are treated as observations, not as evidence that LoCA is generally better than LoRA\.

Our contributions are as follows\.

1. 1\.Calibrated local credit\.LoCA reuses feedback operators from one probe backward pass during a later forward\-only tuning stage\.
2. 2\.Closed\-form local updates\.Each residual adapter is fitted by a ridge solve\. A scale\-normalized target lets us reuse the same candidate set across the tested models\.
3. 3\.Empirical study\.We compare quality, memory, and iteration count with LoRA and matched\-adapter MeZO on five benchmarks and 0\.5B–14B models\.

This paper is organized as follows\. Section[2](https://arxiv.org/html/2608.03020#S2)reviews related work on efficient fine\-tuning and local training methods\. Section[3](https://arxiv.org/html/2608.03020#S3)presents the LoCA framework and its closed\-form local update procedure\. Section[4](https://arxiv.org/html/2608.03020#S4)evaluates LoCA against representative baselines and analyzes its efficiency and limitations\. Section[5](https://arxiv.org/html/2608.03020#S5)discusses the scope of the method, and Section[6](https://arxiv.org/html/2608.03020#S6)concludes the paper\.

## 2Related Work

Parameter\- and memory\-efficient fine\-tuning\.Adapters\[[12](https://arxiv.org/html/2608.03020#bib.bib6)\], prompt methods\[[19](https://arxiv.org/html/2608.03020#bib.bib7),[17](https://arxiv.org/html/2608.03020#bib.bib8)\], and LoRA\[[13](https://arxiv.org/html/2608.03020#bib.bib5)\]reduce the number of trainable parameters\. Other methods reduce memory in different ways\. GaLore projects gradients\[[37](https://arxiv.org/html/2608.03020#bib.bib25)\], LOMO combines gradient computation and parameter updates\[[23](https://arxiv.org/html/2608.03020#bib.bib26)\], QLoRA quantizes frozen weights\[[8](https://arxiv.org/html/2608.03020#bib.bib10)\], and checkpointing recomputes activations\[[4](https://arxiv.org/html/2608.03020#bib.bib9)\]\. BAdam updates blocks in sequence\[[22](https://arxiv.org/html/2608.03020#bib.bib27)\]\. These methods reduce parts of the training cost, but still use backward differentiation through at least part of the model\. LoCA keeps a low\-rank adapter but changes how its update is obtained\.

Zeroth\-order and forward\-only optimization\.Zeroth\-order methods estimate update directions from function values\[[34](https://arxiv.org/html/2608.03020#bib.bib11)\]\. MeZO applies this idea to LLM fine\-tuning with inference\-level memory\[[24](https://arxiv.org/html/2608.03020#bib.bib4)\]\. Recent work uses low\-rank\[[35](https://arxiv.org/html/2608.03020#bib.bib29),[5](https://arxiv.org/html/2608.03020#bib.bib30)\], sparse\[[21](https://arxiv.org/html/2608.03020#bib.bib32)\], or curvature\-aware perturbations\[[38](https://arxiv.org/html/2608.03020#bib.bib31)\]\. These methods retain the memory benefit of forward\-only execution, but their iteration count and tuning can depend on estimator variance\[[9](https://arxiv.org/html/2608.03020#bib.bib13),[26](https://arxiv.org/html/2608.03020#bib.bib12)\]\. We compare with Adapter\-MeZO, which perturbs the same low\-rank adapter parameters as LoCA\. LoCA uses fixed local solves rather than stochastic perturbation updates\.

Feedback alignment\.Feedback alignment trains networks without exact transposed weights\[[20](https://arxiv.org/html/2608.03020#bib.bib14)\]\. Direct feedback alignment sends the top\-layer error to each layer\[[28](https://arxiv.org/html/2608.03020#bib.bib15)\], although transformers remain difficult\[[16](https://arxiv.org/html/2608.03020#bib.bib16)\]\. Its success often depends on forward weights adapting to the fixed feedback\[[31](https://arxiv.org/html/2608.03020#bib.bib17)\]\. This option is limited when the backbone is frozen\. LoCA instead fits the feedback once to the frozen model and then keeps it fixed\.

Local and layer\-wise learning\.Local learning uses separate objectives for different layers\[[3](https://arxiv.org/html/2608.03020#bib.bib18),[27](https://arxiv.org/html/2608.03020#bib.bib19),[11](https://arxiv.org/html/2608.03020#bib.bib20)\]\. Forward gradients have also been combined with local losses\[[32](https://arxiv.org/html/2608.03020#bib.bib34)\]\. Most of these targets support representation learning or compression\[[14](https://arxiv.org/html/2608.03020#bib.bib21)\]\. LoCA keeps the blockwise structure, but forms its targets from a first\-order change in the post\-training loss\. It then fits each adapter with a ridge solve\.

## 3Methodology

LoCA assumes that adaptation stays near the calibrated model state\. It separates credit calibration from adapter fitting\. Calibration uses one probe backward pass to fit low\-rank maps from the top\-layer error to blockwise credit signals\. After calibration, each outer iteration uses no backbone backward pass\. The block parameter is found by solving a local ridge objective\. This solve is exact for a fixed local target\. It does not imply that global cross\-entropy decreases at every outer iteration\. Figure[1](https://arxiv.org/html/2608.03020#S3.F1)shows the post\-calibration loop\.

![Refer to caption](https://arxiv.org/html/2608.03020v1/fig1-pipeline.png)Figure 1:LoRA and the post\-calibration LoCA loop\. One probe backward pass fitsFℓF\_\{\\ell\}before this loop\. Later iterations use forward propagation and local ridge solves\.### 3\.1Local Targets

Following the setting of LoRA\[[13](https://arxiv.org/html/2608.03020#bib.bib5)\], we consider a pre\-trained transformer withLLfrozen blocks\. Lethℓ∈ℝdh\_\{\\ell\}\\in\\mathbb\{R\}^\{d\}be the hidden state after blockℓ\\ell, with the hidden embeddingh0h\_\{0\}\. A frozen headWu∈ℝV×dW\_\{u\}\\in\\mathbb\{R\}^\{V\\times d\}mapshLh\_\{L\}to logitsz=Wu​hLz=W\_\{u\}h\_\{L\}over vocabulary of sizeVV\. We attach a low\-rank adapter to each block\. The adapted hidden state is

hℓ=h¯ℓ\+Bℓ​Aℓ​hℓ−1,h\_\{\\ell\}=\\bar\{h\}\_\{\\ell\}\+B\_\{\\ell\}A\_\{\\ell\}h\_\{\\ell\-1\},\(1\)whereh¯ℓ\\bar\{h\}\_\{\\ell\}is the frozen block output,Aℓ∈ℝr×dA\_\{\\ell\}\\in\\mathbb\{R\}^\{r\\times d\}is a fixed random projection, andBℓ∈ℝd×rB\_\{\\ell\}\\in\\mathbb\{R\}^\{d\\times r\}is fitted\. SettingBℓ=0B\_\{\\ell\}=0recovers the frozen model\. For one block solve, the current forward states and all other adapters are fixed\. Thus,h¯ℓ\\bar\{h\}\_\{\\ell\}andpℓ=Aℓ​hℓ−1p\_\{\\ell\}=A\_\{\\ell\}h\_\{\\ell\-1\}are fixed with respect toBℓB\_\{\\ell\}\. The statehℓh\_\{\\ell\}is then linear inBℓB\_\{\\ell\}, which makes the block objective a convex quadratic\.

Unlike LoRA\[[13](https://arxiv.org/html/2608.03020#bib.bib5)\], LoCA does not update every adapter from one global CE gradient\. It gives each block a regression targetτℓ∈ℝd\\tau\_\{\\ell\}\\in\\mathbb\{R\}^\{d\}and solves

minBℓ⁡𝔼x​‖hℓ−τℓ‖2\\min\_\{B\_\{\\ell\}\}\\;\\mathbb\{E\}\_\{x\}\\bigl\\\|\\,h\_\{\\ell\}\-\\tau\_\{\\ell\}\\,\\bigr\\\|^\{2\}\(2\)for each block\. The post\-calibration solve does not propagate gradients across blocks\. The remaining question is how to chooseτℓ\\tau\_\{\\ell\}\.

A perturbationδ​hℓ\\delta h\_\{\\ell\}changes the loss byδ​ℒ≈⟨gℓ,δ​hℓ⟩\\delta\\mathcal\{L\}\\approx\\langle g\_\{\\ell\},\\,\\delta h\_\{\\ell\}\\rangle, wheregℓ=∂ℒ/∂hℓg\_\{\\ell\}=\\partial\\mathcal\{L\}/\\partial h\_\{\\ell\}\. A first\-order target is

τ^ℓ=hℓ−η​gℓ\.\\hat\{\\tau\}\_\{\\ell\}=h\_\{\\ell\}\-\\eta\\,g\_\{\\ell\}\.\(3\)In the conventional setting, computinggℓg\_\{\\ell\}at every optimization step requires the repeated backward passes we aim to eliminate\. We approximate it during tuning by replacing the input\-dependent Jacobian product with a fixed matrixFℓ∈ℝd×dF\_\{\\ell\}\\in\\mathbb\{R\}^\{d\\times d\}:

τℓ=hℓ−η​Fℓ​e,\\tau\_\{\\ell\}\\;=\\;h\_\{\\ell\}\\;\-\\;\\eta\\,F\_\{\\ell\}\\,e,\(4\)wheree=Wu⊤​\(softmax​\(z\)−onehot​\(y\)\)e=W\_\{u\}^\{\\top\}\\bigl\(\\mathrm\{softmax\}\(z\)\-\\mathrm\{onehot\}\(y\)\\bigr\)is the top\-layer error from the frozen head\. OnceFℓF\_\{\\ell\}is fixed, this target does not require a backbone backward pass\.

### 3\.2The Feedback Operator

The operatorFℓF\_\{\\ell\}maps the top\-layer erroreeto an estimate ofgℓg\_\{\\ell\}\. Random feedback gives a mean cosine alignment near zero in our calibration test,αℓ=cos⁡∠​\(Fℓ​e,gℓ\)≈−0\.005\\alpha\_\{\\ell\}=\\cos\\angle\(F\_\{\\ell\}e,g\_\{\\ell\}\)\\approx\-0\.005\. It therefore provides little useful direction in this setting\. Feedback alignment can improve as forward weights change\[[31](https://arxiv.org/html/2608.03020#bib.bib17)\], but the LoCA backbone is frozen\. We instead fitFℓF\_\{\\ell\}to the frozen model\.

We fit an untruncated feedback mapF^ℓ\\widehat\{F\}\_\{\\ell\}once from samples of the backward mapping\. One probe backward pass through the frozen model yields per\-token gradientsgℓg\_\{\\ell\}and top\-layer errorsee, stacked asG~ℓ\\widetilde\{G\}\_\{\\ell\}andE~\\widetilde\{E\}\. We solve

F^ℓ=arg⁡minF⁡‖F​E~−G~ℓ‖F2\+β​‖F‖F2,\\widehat\{F\}\_\{\\ell\}=\\arg\\min\_\{F\}\\;\\big\\\|F\\widetilde\{E\}\-\\widetilde\{G\}\_\{\\ell\}\\big\\\|\_\{F\}^\{2\}\+\\beta\\\|F\\\|\_\{F\}^\{2\},\(5\)whereβ\>0\\beta\>0\. Writing its singular value decomposition asF^ℓ=U​Σ​V⊤\\widehat\{F\}\_\{\\ell\}=U\\Sigma V^\{\\top\}, we retainFℓ=Uk​Σk​Vk⊤F\_\{\\ell\}=U\_\{k\}\\Sigma\_\{k\}V\_\{k\}^\{\\top\}withk=8k=8\. Thus,FℓF\_\{\\ell\}is a rank\-kkapproximation to the regularized fit rather than the full fitted map\. On the probe data, its cosine alignment with the true hidden\-state gradients is0\.30\.3–0\.50\.5\. We treat this fixed map as a local approximation whose alignment may degrade as repeated updates move the model away from the calibrated state\. In the small\-shift regime, however, it can carry useful backward\-derived credit into forward\-only adaptation without requiring further backward passes\.

### 3\.3Closed\-Form Per\-Block Solve

For a fixedτℓ\\tau\_\{\\ell\}, each block gives a ridge problem\. Its residual target is

ρℓ:=τℓ−h¯ℓ=Bℓ\(t\)​pℓ−η​Fℓ​e,\\rho\_\{\\ell\}:=\\tau\_\{\\ell\}\-\\bar\{h\}\_\{\\ell\}=B\_\{\\ell\}^\{\(t\)\}p\_\{\\ell\}\-\\eta\\,F\_\{\\ell\}e,\(6\)whereBℓ\(t\)B\_\{\\ell\}^\{\(t\)\}is the current adapter\. OverNNpredicted tokens, blockℓ\\ellminimizes

∑n‖Bℓ​pℓ\(n\)−ρℓ\(n\)‖2\+λ​‖Bℓ‖F2\.\\sum\_\{n\}\\\|B\_\{\\ell\}p\_\{\\ell\}^\{\(n\)\}\-\\rho\_\{\\ell\}^\{\(n\)\}\\\|^\{2\}\+\\lambda\\\|B\_\{\\ell\}\\\|\_\{F\}^\{2\}\.\(7\)This strictly convex problem has the unique closed\-form minimizer

Bℓ⋆=Cℓ​\(Gℓ\+λ​Ir\)−1,B\_\{\\ell\}^\{\\star\}=C\_\{\\ell\}\\big\(G\_\{\\ell\}\+\\lambda I\_\{r\}\\big\)^\{\-1\},\(8\)whereλ\>0\\lambda\>0is the ridge regularization coefficient, and

Gℓ=∑npℓ\(n\)​pℓ\(n\)⊤,Cℓ=∑nρℓ\(n\)​pℓ\(n\)⊤\.G\_\{\\ell\}=\\sum\_\{n\}p\_\{\\ell\}^\{\(n\)\}p\_\{\\ell\}^\{\(n\)\\top\},\\qquad C\_\{\\ell\}=\\sum\_\{n\}\\rho\_\{\\ell\}^\{\(n\)\}p\_\{\\ell\}^\{\(n\)\\top\}\.\(9\)The block parameter uses no gradient optimizer, learning\-rate schedule, or optimizer state\. The coefficientη\\etaremains a target\-size hyperparameter\.

Three details are useful in practice\. First,GℓG\_\{\\ell\}andCℓC\_\{\\ell\}are additive over batches\. Their persistent storage isO​\(L​\(r2\+d​r\)\)O\(L\(r^\{2\}\+dr\)\)and does not grow with the number of training tokens\. Calibration creates one temporary backward graph\. Its peak is included in the GPU measurements in Table[2](https://arxiv.org/html/2608.03020#S4.T2)\. Second, we keep the frozen model as a candidate and return it when no adapted snapshot improves held\-out CE\. This fallback protects the selected validation objective, not unseen test metrics\. Third, the block solve is deterministic when the probe data, projection, data order, and numerical implementation are fixed\. These choices can still cause variation across runs\.

†indicates that the corresponding recovery ratio is not meaningful because the LoRA improvement over the frozen model is negligible\.‡\\ddaggerindicates that the accuracy recovery is omitted because the LoRA accuracy gain is zero or near zero\.

Table 1:Main benchmark results\. We report evaluation CE, where lower is better, together with CE and accuracy recovery ratios\.
### 3\.4Full Algorithm

The closed\-form solution above givesBℓ⋆B\_\{\\ell\}^\{\\star\}for a fixed target\. The target changes after an adapter update because later hidden states and the erroreealso change\. LoCA therefore repeats the procedure forTTouter steps\. Each step runs a forward pass, updates the targets, and solves the block objectives\. A Jacobi schedule uses one shared forward pass for all blocks\. A Gauss–Seidel schedule refreshes later states after each block\. Each solve minimizes its current local objective\. Global CE can still be non\-monotone, so we selectTTby held\-out CE\.

The step sizeη\\etain \([4](https://arxiv.org/html/2608.03020#S3.E4)\) is an absolute quantity, and the norm ofeevaries across model sizes\. In practice, a fixedη\\etathat works at 0\.5B returns the frozen model at 14B\. We reduce this sensitivity by normalizing the target correction to the residual\-stream scale:

τℓ=hℓ−η¯​Fℓ​e,η¯=η⋅RMS​\(hℓ−1\)RMS​\(Fℓ​e\),\\tau\_\{\\ell\}=h\_\{\\ell\}\-\\bar\{\\eta\}\\,F\_\{\\ell\}\\,e,\\qquad\\bar\{\\eta\}=\\eta\\cdot\\frac\{\\mathrm\{RMS\}\(h\_\{\\ell\-1\}\)\}\{\\mathrm\{RMS\}\(F\_\{\\ell\}\\,e\)\},\(10\)soη\\etais the relative change to the residual stream\. We reuse the same candidate set across all tested model sizes and architectures\.

## 4Experiments

### 4\.1Setup

Models and tasks\.We test Qwen2\.5 at 0\.5B, 1\.5B, 3B, 7B, and 14B parameters\[[30](https://arxiv.org/html/2608.03020#bib.bib42)\]\. We also test SmolLM2\-1\.7B\[[2](https://arxiv.org/html/2608.03020#bib.bib43)\]\. The tasks are SST\-2\[[33](https://arxiv.org/html/2608.03020#bib.bib36)\], BoolQ\[[6](https://arxiv.org/html/2608.03020#bib.bib37)\], ARC\-Easy and ARC\-Challenge\[[7](https://arxiv.org/html/2608.03020#bib.bib38)\], OpenBookQA\[[25](https://arxiv.org/html/2608.03020#bib.bib39)\], and HellaSwag\[[36](https://arxiv.org/html/2608.03020#bib.bib40)\]\.

Baselines and metrics\.We compare with the frozen model, LoRA\[[13](https://arxiv.org/html/2608.03020#bib.bib5)\], and MeZO\[[24](https://arxiv.org/html/2608.03020#bib.bib4)\]\. Full\-parameter MeZO perturbs a much larger parameter space and weakens with model size in supplementary Figure S1\. Our main comparison therefore uses Adapter\-MeZO, which perturbs the same low\-rank adapter parameters as LoCA and LoRA\. We report per\-token CE and ranking accuracy\. Recovery gives a normalized summary of the frozen\-to\-LoRA gap\.

Table[1](https://arxiv.org/html/2608.03020#S3.T1)reports the results\.

RCE=CEfrozen−CEoursCEfrozen−CELoRA,R\_\{\\mathrm\{CE\}\}=\\frac\{\\mathrm\{CE\}\_\{\\mathrm\{frozen\}\}\-\\mathrm\{CE\}\_\{\\mathrm\{ours\}\}\}\{\\mathrm\{CE\}\_\{\\mathrm\{frozen\}\}\-\\mathrm\{CE\}\_\{\\mathrm\{LoRA\}\}\},\(11\)Racc=accours−accfrozenaccLoRA−accfrozen,R\_\{\\mathrm\{acc\}\}=\\frac\{\\mathrm\{acc\}\_\{\\mathrm\{ours\}\}\-\\mathrm\{acc\}\_\{\\mathrm\{frozen\}\}\}\{\\mathrm\{acc\}\_\{\\mathrm\{LoRA\}\}\-\\mathrm\{acc\}\_\{\\mathrm\{frozen\}\}\},\(12\)soR=1R=1matches the reported LoRA value\. Recovery can be unstable when the frozen\-to\-LoRA gap is near zero\. We therefore treat raw CE and accuracy as the main metrics and use recovery as a descriptive summary\. We omit ratio claims for near\-zero denominators\.

Protocol\.LoCA uses rankr=32r=32, at most 40 outer iterations, and early stopping on held\-out CE\. We selectη\\etafrom the stated candidate set on the same held\-out metric\. MeZO scales its learning rate by1/D1/\\sqrt\{D\}and searches it at each model size\. Unless noted otherwise, table cells are single runs with seed 0\. We interpret cell\-level differences as descriptive rather than statistically significant\. Runs use NVIDIA L40S GPUs or 32 GB CPU\-only servers\. We do not include GaLore or LOMO because they retain backward differentiation and address a different cost trade\-off\.

Adapter\-MeZOLoCALoRALoCA/LoRA*GPU peak training memory \(MB\)*0\.5B2,1974,4206,2540\.71×0\.71\\times1\.5B5,6696,6469,1090\.73×0\.73\\times3B9,0799,24812,6120\.73×0\.73\\times7B9,15411,52015,6470\.74×0\.74\\times*CPU steady\-state memory \(MB\)*0\.5B8791,2792,6640\.48×0\.48\\times1\.5B1,7412,8455,6330\.51×0\.51\\times3B2,0152,5904,8750\.53×0\.53\\times7B3,5334,1866,5820\.64×0\.64\\times*CPU wall\-clock per pass \(s\)*0\.5B8\.95\.910\.30\.57×0\.57\\times1\.5B26\.014\.225\.50\.56×0\.56\\times3B28\.715\.029\.00\.52×0\.52\\times7B55\.429\.555\.20\.53×0\.53\\timesTable 2:Resources beyond the loaded model\. GPU peak includes probe calibration\. CPU steady\-state memory and per\-pass time describe post\-calibration tuning\.
### 4\.2Main Results

Experiment 1 compares the frozen model, LoRA, Adapter\-MeZO, and LoCA on five model sizes and five tasks\.

LoCA’s CE recovery ranges from 0\.81 to 2\.40 in the reported cells\. It obtains lower CE than the corresponding LoRA run in 16 of the 25 task–scale cells\. These cells are single runs, so the count is descriptive and does not show general superiority over LoRA\.

On BoolQ at 7B and 14B, the reported LoRA runs lower CE while ranking accuracy falls from 0\.824 to 0\.636 and from 0\.852 to 0\.364\. LoCA reaches 0\.840 and 0\.864 in these cells\. This CE–accuracy gap is consistent with completion\-format overfitting, but the current experiment does not isolate that cause\.

Table 3:Scale\-normalized targets with one shared candidate set across the tested models\.
### 4\.3Resource Use

Table[2](https://arxiv.org/html/2608.03020#S4.T2)reports memory beyond the loaded model\. LoCA uses0\.710\.71–0\.74×0\.74\\timesthe GPU peak memory of LoRA and0\.480\.48–0\.64×0\.64\\timesits CPU steady\-state memory\. The GPU peak is measured over the full LoCA run and includes the temporary graph used for calibration\. After calibration, LoCA stores the streaming statisticsGℓ∈ℝr×rG\_\{\\ell\}\\in\\mathbb\{R\}^\{r\\times r\}andCℓ∈ℝd×rC\_\{\\ell\}\\in\\mathbb\{R\}^\{d\\times r\}\. Their size isO​\(L​\(r2\+d​r\)\)O\(L\(r^\{2\}\+dr\)\)and does not grow with the number of training tokens\. On BoolQ at length 512, LoRA reaches 32\.3 GB and LoCA reaches 16\.3 GB\. LoRA is killed on the 32 GB device, while LoCA completes the run\.

Adapter\-MeZO uses the least memory in the table\. It perturbs parameters in place and stores no gradient graph or optimizer state\. Its update estimate is stochastic and needs10310^\{3\}–10410^\{4\}perturbation steps in our runs, together with a learning\-rate search at each scale\.

After calibration, one LoCA pass costs about one forward pass\. On CPU, its measured per\-pass time is1\.71\.7–1\.9×1\.9\\timeslower than LoRA’s\. On 1\.5B ARC\-Easy, 3,000 Adapter\-MeZO steps take 4\.2 hours\. On 1\.5B ARC\-Challenge, the measured LoCA run reaches lower CE in1\.7×1\.7\\timesless wall\-clock\. We did not record calibration latency separately, so the general timing claim is limited to post\-calibration cost\. The GPU peak\-memory result still includes calibration\.

### 4\.4Cross\-Family Results

The target\-size coefficientη\\etais sensitive to model scale\. We normalize it by the residual\-stream norm, which makes it a relative change\. We use the candidate set\{0\.003,0\.01,0\.03\}\\\{0\.003,0\.01,0\.03\\\}at every tested scale and select a value by held\-out CE\. The search range is not redesigned for each model\.

On SST\-2, the normalized candidate set gives recoveries of 1\.02, 1\.00, 0\.92, and 0\.94 across four sizes\. The hand\-tuned absolute values give 0\.94, 0\.91, 0\.89, and 0\.82\. Thus, one shared candidate set works across the tested scales, although held\-out selection is still used within that set\.

We also test SmolLM2\-1\.7B\. Reusing the absoluteη\\etagives recovery from 0 to 0\.07 across four tasks\. With target normalization, recovery ranges from 0\.93 to 1\.29 and averages about 1\.04\. These results support the use of relative target size on a second model family\. They do not show that normalization will transfer to every architecture\.

## 5Discussion

Across the 25 combinations of Qwen2\.5 model scales and tasks, LoCA reports lower evaluation CE than the corresponding LoRA run in 16 cases\. Although these cells are mainly single runs, and therefore do not establish that LoCA is generally better than LoRA, they show that LoCA is not merely an inaccurate copy of the LoRA update\. Whereas LoRA follows the global CE gradient, LoCA optimizes regularized low\-rank objectives and selects a snapshot by held\-out CE\. These choices define a different adaptation path\. Ridge regularization, the low\-rank constraint, and held\-out selection may jointly discourage large local changes, although our experiments do not isolate their effects\. BoolQ illustrates another difference\. At 7B and 14B, LoRA lowers CE while ranking accuracy falls from 0\.824 to 0\.636 and from 0\.852 to 0\.364\. LoCA reaches accuracies of 0\.840 and 0\.864 in the same cells\. While this pattern is consistent with completion\-format overfitting, the current experiments do not establish that cause\. More broadly, because the local objectives impose a different inductive bias from global gradient updates, they may also lead to different generalization behavior\.

Although both LoCA and Adapter\-MeZO reduce the need for repeated backward passes, they obtain update information in different ways\. MeZO applies zeroth\-order SGD to language\-model tuning, estimating an update direction from random perturbations and function\-value differences\[[24](https://arxiv.org/html/2608.03020#bib.bib4)\]\. This gives MeZO an inference\-level memory footprint, but each query provides a stochastic directional estimate rather than an exact solution to the current objective\. Classical zeroth\-order theory shows that, when gradients are replaced by random function\-value estimators, the resulting methods incur additional variance and may suffer dimension\-dependent query or convergence costs\[[9](https://arxiv.org/html/2608.03020#bib.bib13),[26](https://arxiv.org/html/2608.03020#bib.bib12)\]\. It’s also worth mentioning that, although random feedback is distinct from zeroth\-order estimation, the random\-map tested when constructing feedback operator exhibits near\-zero in\-probe alignment with the true hidden\-state gradient, whereas calibration produces substantially better\-aligned feedback\. These analyses rely on assumptions that do not directly describe non\-convex LLM tuning\. Nevertheless, they identify a general cost of random function\-value probes\. Recent work has therefore introduced variance\-reduced MeZO variants, which are designed to improve stability and convergence in language\-model tuning\[gautam2024variance\]\.

By contrast, once the forward states and local target are fixed, the ridge objective used by LoCA is a regularized convex quadratic\. Whenλ\>0\\lambda\>0, this objective has a unique closed\-form solution\. The block update uses sufficient statistics accumulated from the current data, rather than repeatedly estimating random directions\. It also avoids an iterative learning\-rate schedule inside the block solve\. The benefit of the closed form is not a guarantee of global optimality\. Instead, it removes optimization error within the current local subproblem\. Because other adapters and hidden states change after an update, the local target also changes across outer iterations\. Global CE therefore need not decrease at every step\.

Since the two methods pay for update information in different ways, their iteration counts also differ substantially in our experiments\. Adapter\-MeZO requires between10310^\{3\}and10410^\{4\}perturbation steps, whereas LoCA uses at most 40 outer iterations\. LoCA still requires the target\-size coefficient and ridge regularization to be selected, and its calibration cost is not zero\. Moreover, because calibration latency was not recorded separately, the current measurements do not support a universal end\-to\-end speed claim\. The observed trade\-off is more specific\. Adapter\-MeZO has the lowest memory use, while LoCA exchanges one probe backward pass for structured local updates and fewer later iterations\.

When the local correction is expressed relative to the residual\-stream scale, the same candidate set can be reused across Qwen2\.5 models from 0\.5B to 14B and on SmolLM2\-1\.7B\. An absolute correction, in contrast, changes substantially with model size and architecture\. This result suggests that relative representation change is more portable than an absolute hidden\-state correction\. It does not make LoCA hyperparameter\-free, since a value is still selected from the shared set by held\-out CE\. The supported claim is therefore that the candidate range transfers across the tested models, not that one fixed value works without validation\.

Because the fixed feedback maps are calibrated near the base model, the present results are limited to discriminative tasks that require relatively small representation changes\. The approximation may weaken when tuning moves far from the calibration point or when the task requires a larger shift\. Such settings may require a new calibration pass or an input\-dependent feedback map\. Long\-form generation and large distribution shifts are also outside the current study\. LoCA should therefore be viewed as a two\-stage method rather than a method that removes backward computation entirely\. One probe backward pass calibrates the feedback maps, after which adaptation uses forward execution and local closed\-form solves\. A device without backward support can receive the calibrated maps from a training\-capable host\. The reported GPU peak includes calibration, whereas CPU steady\-state memory and per\-pass time describe the later stage\. In this sense, LoCA amortizes repeated global credit assignment into one calibration pass, allowing subsequent tuning to proceed through forward computation and local solves\.

## 6Conclusion

We presented LoCA, a two\-stage method that separates global credit calibration from adapter fitting\. One probe backward pass fits low\-rank maps from the final\-layer error to blockwise corrections\. After calibration, these maps turn forward activations into local targets, and each adapter is fitted by a closed\-form ridge solve\. The main idea is therefore to reuse calibrated credit information rather than reconstruct it through every tuning step\.

Across five discriminative benchmarks and Qwen2\.5 models from 0\.5B to 14B, LoCA reports lower evaluation CE than the corresponding LoRA run in 16 of 25 comparisons\. Since most cells are single runs, this count is descriptive and does not establish general superiority\. The measured GPU peak over the full run, including calibration, is 26 to 29% lower than LoRA’s\. After calibration, CPU steady\-state memory is 36 to 52% lower and per\-pass time is 43 to 48% lower\. A shared scale\-normalized candidate set is also reused across the tested Qwen2\.5 sizes and on SmolLM2\-1\.7B, with held\-out selection within that set\.

The current evidence is limited to small\-shift adaptation on discriminative tasks\. Fixed feedback maps may become less accurate after a large model change or a long tuning path, and long\-form generation remains untested\. Future work should study when recalibration is needed and whether input\-dependent feedback can support larger shifts\. LoCA is not entirely free of backward computation\. Instead, it defines a practical boundary: a training\-capable host performs one calibration pass, while later adaptation can run through forward computation and local solves on an inference\-oriented device\.

## Acknowledgments

This work was supported in part by the National Key Technologies Research and Development Program \(2025YFG0202100; 2025YFG0202600\)\. The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]\(2021\)Intrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,pp\. 7319–7328\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p4.1)\.
- \[2\]L\. B\. Allal, A\. Lozhkov, E\. Bakouch, G\. M\. Blázquez, G\. Penedo, L\. Tunstall,et al\.\(2025\)SmolLM2: when smol goes big – data\-centric training of a small language model\.Note:arXiv:2502\.02737Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[3\]E\. Belilovsky, M\. Eickenberg, and E\. Oyallon\(2019\)Greedy layerwise learning can scale to ImageNet\.InProceedings of the 36th International Conference on Machine Learning,pp\. 583–593\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p4.1)\.
- \[4\]T\. Chen, B\. Xu, C\. Zhang, and C\. Guestrin\(2016\)Training deep nets with sublinear memory cost\.Note:arXiv:1604\.06174Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[5\]Y\. Chen, Y\. Zhang, L\. Cao, K\. Yuan, and Z\. Wen\(2025\)Enhancing zeroth\-order fine\-tuning for language models with low\-rank structures\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p2.1)\.
- \[6\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 2924–2936\.Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[7\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.Note:arXiv:1803\.05457Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[8\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[9\]J\. C\. Duchi, M\. I\. Jordan, M\. J\. Wainwright, and A\. Wibisono\(2015\)Optimal rates for zero\-order convex optimization: the power of two function evaluations\.IEEE Transactions on Information Theory61\(5\),pp\. 2788–2806\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p2.1),[§5](https://arxiv.org/html/2608.03020#S5.p2.1)\.
- \[10\]S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. Smith\(2020\)Don’t stop pretraining: adapt language models to domains and tasks\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8342–8360\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p1.1)\.
- \[11\]G\. Hinton\(2022\)The forward\-forward algorithm: some preliminary investigations\.arXiv preprint arXiv:2212\.13345\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p4.1)\.
- \[12\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for NLP\.InProceedings of the 36th International Conference on Machine Learning,pp\. 2790–2799\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p2.1),[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[13\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p2.1),[§1](https://arxiv.org/html/2608.03020#S1.p4.1),[§2](https://arxiv.org/html/2608.03020#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.03020#S3.SS1.p1.8),[§3\.1](https://arxiv.org/html/2608.03020#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p2.1)\.
- \[14\]X\. Jiao, Y\. Yin, L\. Shang, X\. Jiang, X\. Chen, L\. Li, F\. Wang, and Q\. Liu\(2020\)TinyBERT: distilling BERT for natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 4163–4174\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p4.1)\.
- \[15\]D\. P\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p1.1)\.
- \[16\]J\. Launay, I\. Poli, F\. Boniface, and F\. Krzakala\(2020\)Direct feedback alignment scales to modern deep learning tasks and architectures\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p3.1)\.
- \[17\]B\. Lester, R\. Al\-Rfou, and N\. Constant\(2021\)The power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p2.1),[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[18\]C\. Li, H\. Farkhoor, R\. Liu, and J\. Yosinski\(2018\)Measuring the intrinsic dimension of objective landscapes\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p4.1)\.
- \[19\]X\. L\. Li and P\. Liang\(2021\)Prefix\-Tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,pp\. 4582–4597\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p2.1),[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[20\]T\. P\. Lillicrap, D\. Cownden, D\. B\. Tweed, and C\. J\. Akerman\(2016\)Random synaptic feedback weights support error backpropagation for deep learning\.Nature Communications7,pp\. 13276\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p3.1)\.
- \[21\]Y\. Liu, Z\. Zhu, C\. Gong, M\. Cheng, C\. Hsieh, and Y\. You\(2024\)Sparse MeZO: less parameters for better performance in zeroth\-order LLM fine\-tuning\.arXiv preprint arXiv:2402\.15751\.Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p2.1)\.
- \[22\]Q\. Luo, H\. Yu, and X\. Li\(2024\)BAdam: a memory efficient full parameter optimization method for large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[23\]K\. Lv, Y\. Yang, T\. Liu, Q\. Gao, Q\. Guo, and X\. Qiu\(2024\)Full parameter fine\-tuning for large language models with limited resources\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,pp\. 8187–8198\.Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[24\]S\. Malladi, T\. Gao, E\. Nichani, A\. Damian, J\. D\. Lee, D\. Chen, and S\. Arora\(2023\)Fine\-tuning language models with just forward passes\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p1.1),[§1](https://arxiv.org/html/2608.03020#S1.p2.1),[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p2.1),[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p2.1),[§5](https://arxiv.org/html/2608.03020#S5.p2.1)\.
- \[25\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[26\]Y\. Nesterov and V\. Spokoiny\(2017\)Random gradient\-free minimization of convex functions\.Foundations of Computational Mathematics17\(2\),pp\. 527–566\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p2.1),[§5](https://arxiv.org/html/2608.03020#S5.p2.1)\.
- \[27\]A\. Nøkland and L\. H\. Eidnes\(2019\)Training neural networks with local error signals\.InProceedings of the 36th International Conference on Machine Learning,pp\. 4839–4850\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p4.1)\.
- \[28\]A\. Nøkland\(2016\)Direct feedback alignment provides learning in deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.29\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p3.1)\.
- \[29\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p1.1)\.
- \[30\]Qwen Team\(2024\)Qwen2\.5 technical report\.Note:arXiv:2412\.15115Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[31\]M\. Refinetti, S\. d’Ascoli, R\. Ohana, and S\. Goldt\(2021\)Align, then memorise: the dynamics of learning with feedback alignment\.InProceedings of the 38th International Conference on Machine Learning,pp\. 8925–8935\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p3.1),[§3\.2](https://arxiv.org/html/2608.03020#S3.SS2.p1.5)\.
- \[32\]M\. Ren, S\. Kornblith, R\. Liao, and G\. Hinton\(2023\)Scaling forward gradient with local losses\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p4.1)\.
- \[33\]R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Ng, and C\. Potts\(2013\)Recursive deep models for semantic compositionality over a sentiment treebank\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pp\. 1631–1642\.Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[34\]J\. C\. Spall\(1992\)Multivariate stochastic approximation using a simultaneous perturbation gradient approximation\.IEEE Transactions on Automatic Control37\(3\),pp\. 332–341\.Cited by:[§1](https://arxiv.org/html/2608.03020#S1.p3.1),[§2](https://arxiv.org/html/2608.03020#S2.p2.1)\.
- \[35\]Z\. Yu, P\. Zhou, S\. Wang, J\. Li, and H\. Huang\(2025\)Zeroth\-order fine\-tuning of LLMs in random subspaces\.InProceedings of the IEEE/CVF International Conference on Computer Vision,Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p2.1)\.
- \[36\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§4\.1](https://arxiv.org/html/2608.03020#S4.SS1.p1.1)\.
- \[37\]J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian\(2024\)GaLore: memory\-efficient LLM training by gradient low\-rank projection\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p1.1)\.
- \[38\]Y\. Zhao, S\. Dang, H\. Ye, G\. Dai, Y\. Qian, and I\. W\. Tsang\(2025\)Second\-order fine\-tuning without pain for LLMs: a Hessian informed zeroth\-order optimizer\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.03020#S2.p2.1)\.

## Appendix

## Appendix AScope and Roadmap

This section expands the opening of the main paper’s Methodology section\. It fixes the scope of the two\-stage claim and states how the remaining sections support the equations in the main paper\.

LoCA separates credit calibration from adapter fitting\. In Stage I, one probe backward pass through the frozen base model produces hidden\-state gradients used to fit the feedback maps\{Fℓ\}ℓ=1L\\\{F\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}\. In Stage II, these maps are fixed\. Each outer iteration then uses a forward pass, the analytic error at the frozen language\-model head, and blockwise ridge solves; it does not differentiate through a backbone block\.

Sections S2–S8 expand the Methodology section of the main paper\. The mathematical statements have three distinct scopes:

1. 1\.the language\-model\-head error and the two ridge minimizers are exact for the stated fixed data;
2. 2\.fixed feedback defines a virtual hidden\-state descent step only under explicit alignment and smoothness conditions;
3. 3\.the fitted adapter need not realize that virtual step exactly, and the full outer iteration is not claimed to decrease global cross\-entropy monotonically\.

Sections S9–S13 provide experimental context for the setup, main benchmark, resource study, scale normalization, and MeZO baselines\.

## Appendix BSequence\-Level Notation and the Affine Adapter

This section expands “Local Targets” and Eq\. \(1\) of the main paper\. Its purpose is to show why a block solve is linear inBℓB\_\{\\ell\}while retaining the cross\-token dependence created by self\-attention\.

Consider a decoder\-only transformer withLLfrozen blocks and hidden widthdd\. For a sequence withKKpositions, stack the hidden states after blockℓ\\ellby columns:

Hℓ=\[hℓ,1,…,hℓ,K\]∈ℝd×K\.H\_\{\\ell\}=\[h\_\{\\ell,1\},\\ldots,h\_\{\\ell,K\}\]\\in\\mathbb\{R\}^\{d\\times K\}\.\(13\)The vectorhℓh\_\{\\ell\}in the main paper denotes one column ofHℓH\_\{\\ell\}\. LoCA adds a direct low\-rank correction to the output residual stream:

Hℓ=H¯ℓ\+Bℓ​Aℓ​Hℓ−1=H¯ℓ\+Bℓ​Pℓall,H\_\{\\ell\}=\\bar\{H\}\_\{\\ell\}\+B\_\{\\ell\}A\_\{\\ell\}H\_\{\\ell\-1\}=\\bar\{H\}\_\{\\ell\}\+B\_\{\\ell\}P\_\{\\ell\}^\{\\mathrm\{all\}\},\(14\)where

H¯ℓ:=fℓbase​\(Hℓ−1\),Aℓ∈ℝr×d,Bℓ∈ℝd×r,Pℓall:=Aℓ​Hℓ−1∈ℝr×K\.\\bar\{H\}\_\{\\ell\}:=f\_\{\\ell\}^\{\\mathrm\{base\}\}\(H\_\{\\ell\-1\}\),\\qquad A\_\{\\ell\}\\in\\mathbb\{R\}^\{r\\times d\},\\qquad B\_\{\\ell\}\\in\\mathbb\{R\}^\{d\\times r\},\\qquad P\_\{\\ell\}^\{\\mathrm\{all\}\}:=A\_\{\\ell\}H\_\{\\ell\-1\}\\in\\mathbb\{R\}^\{r\\times K\}\.\(15\)For one block solve, the current forward states and all other adapters are fixed\. HenceH¯ℓ\\bar\{H\}\_\{\\ell\}andPℓallP\_\{\\ell\}^\{\\mathrm\{all\}\}are fixed with respect toBℓB\_\{\\ell\}, and

Hℓ=H¯ℓ\+Bℓ​PℓallH\_\{\\ell\}=\\bar\{H\}\_\{\\ell\}\+B\_\{\\ell\}P\_\{\\ell\}^\{\\mathrm\{all\}\}\(16\)is exactly affine inBℓB\_\{\\ell\}\. The block solve therefore does not linearize the adapter\.

Letwt≥0w\_\{t\}\\geq 0be the loss weight at positiontt; a masked position haswt=0w\_\{t\}=0\. For one sequence, the scalar training loss is

ℒ=∑t=1Kwt​ℓCE​\(hL,t,yt\)\.\\mathcal\{L\}=\\sum\_\{t=1\}^\{K\}w\_\{t\}\\,\\ell\_\{\\mathrm\{CE\}\}\(h\_\{L,t\},y\_\{t\}\)\.\(17\)The ridge objective in the main paper is accumulated overNNselected prediction positions from the training data\. Their projected features are denoted

Pℓ=\[pℓ\(1\),…,pℓ\(N\)\]∈ℝr×N\.P\_\{\\ell\}=\[p\_\{\\ell\}^\{\(1\)\},\\ldots,p\_\{\\ell\}^\{\(N\)\}\]\\in\\mathbb\{R\}^\{r\\times N\}\.\(18\)Using only selected prediction positions is part of the local surrogate, not an exact reduction of the global gradient\. A masked context position can still have a nonzero lower\-layer gradient because it can affect later unmasked losses\.

## Appendix CExact Head Error and Full Hidden\-State Credit

This section expands the top\-layer error in “Local Targets” and clarifies what the feedback operator in “The Feedback Operator” approximates\. It separates the exact head derivative from the cross\-token, cross\-layer gradient supplied by the probe backward pass\.

The frozen language\-model head isWu∈ℝV×dW\_\{u\}\\in\\mathbb\{R\}^\{V\\times d\}\. At positiontt,

zt=Wu​hL,t,ℓCE​\(hL,t,yt\)=−zt,yt\+log​∑j=1Vexp⁡\(zt,j\)\.z\_\{t\}=W\_\{u\}h\_\{L,t\},\\qquad\\ell\_\{\\mathrm\{CE\}\}\(h\_\{L,t\},y\_\{t\}\)=\-z\_\{t,y\_\{t\}\}\+\\log\\sum\_\{j=1\}^\{V\}\\exp\(z\_\{t,j\}\)\.\(19\)
###### Lemma C\.1\(Top\-layer error\)\.

The contribution of positionttto the derivative with respect to the last hidden state is

et:=wt​∇hL,tℓCE=wt​Wu⊤​\(softmax⁡\(zt\)−onehot⁡\(yt\)\)\.e\_\{t\}:=w\_\{t\}\\nabla\_\{h\_\{L,t\}\}\\ell\_\{\\mathrm\{CE\}\}=w\_\{t\}W\_\{u\}^\{\\top\}\\bigl\(\\operatorname\{softmax\}\(z\_\{t\}\)\-\\operatorname\{onehot\}\(y\_\{t\}\)\\bigr\)\.\(20\)Thuset=0e\_\{t\}=0at a zero\-weight position\.

The vectoreein the main paper is the single\-position case with unit weight\. Herewtw\_\{t\}also covers masking and loss reduction\.

###### Proof\.

For logit coordinateii,

∂ℓCE∂zt,i=−𝟏​\[i=yt\]\+exp⁡\(zt,i\)∑jexp⁡\(zt,j\)\.\\frac\{\\partial\\ell\_\{\\mathrm\{CE\}\}\}\{\\partial z\_\{t,i\}\}=\-\\mathbf\{1\}\[i=y\_\{t\}\]\+\\frac\{\\exp\(z\_\{t,i\}\)\}\{\\sum\_\{j\}\\exp\(z\_\{t,j\}\)\}\.\(21\)Multiplication bywtw\_\{t\}and the chain rule throughzt=Wu​hL,tz\_\{t\}=W\_\{u\}h\_\{L,t\}give \([20](https://arxiv.org/html/2608.03020#A3.E20)\)\. ∎

StackE=\[e1,…,eK\]∈ℝd×KE=\[e\_\{1\},\\ldots,e\_\{K\}\]\\in\\mathbb\{R\}^\{d\\times K\}and define the exact gradient of the scalar sequence loss with respect to the block output by

𝒢ℓ:=∇Hℓℒ=\[gℓ,1,…,gℓ,K\]∈ℝd×K\.\\mathcal\{G\}\_\{\\ell\}:=\\nabla\_\{H\_\{\\ell\}\}\\mathcal\{L\}=\[g\_\{\\ell,1\},\\ldots,g\_\{\\ell,K\}\]\\in\\mathbb\{R\}^\{d\\times K\}\.\(22\)If𝒥ℓ\+1:L\\mathcal\{J\}\_\{\\ell\+1:L\}is the Jacobian of the current downstream adapted network, with all current adapters held fixed, then

vec⁡\(𝒢ℓ\)=𝒥ℓ\+1:L⊤​vec⁡\(E\)\.\\operatorname\{vec\}\(\\mathcal\{G\}\_\{\\ell\}\)=\\mathcal\{J\}\_\{\\ell\+1:L\}^\{\\top\}\\operatorname\{vec\}\(E\)\.\(23\)Equivalently, causal attention gives the positionwise expression

gℓ,s=∑t=sK\(∂hL,t∂hℓ,s\)⊤​et\.g\_\{\\ell,s\}=\\sum\_\{t=s\}^\{K\}\\left\(\\frac\{\\partial h\_\{L,t\}\}\{\\partial h\_\{\\ell,s\}\}\\right\)^\{\\top\}e\_\{t\}\.\(24\)Consequently,es=0e\_\{s\}=0does not generally implygℓ,s=0g\_\{\\ell,s\}=0\.

One probe backward pass computes all matrices\{𝒢ℓ\}\\\{\\mathcal\{G\}\_\{\\ell\}\\\}in \([23](https://arxiv.org/html/2608.03020#A3.E23)\)\. During Stage II, LoCA instead applies the samed×dd\\times dmap independently to the top errors:

Qℓ:=Fℓ​E\.Q\_\{\\ell\}:=F\_\{\\ell\}E\.\(25\)This is a tokenwise predictor of hidden\-state credit, not an exact factorization of the full sequence Jacobian\. The approximation begins at \([25](https://arxiv.org/html/2608.03020#A3.E25)\), not at the language\-model head\.

## Appendix DIdeal Targets and Conditional Virtual Descent

This section expands Eqs\. \(3\)–\(4\) in “Local Targets” and the alignment discussion in “The Feedback Operator\.” It explains why the ideal target matches the global gradient locally and gives the precise condition under which fixed feedback defines a virtual descent step\.

For a fixed current forward pass, the differential of the global loss caused by a perturbationΔ​Hℓ\\Delta H\_\{\\ell\}is

d​ℒ=⟨𝒢ℓ,Δ​Hℓ⟩F\.d\\mathcal\{L\}=\\left\\langle\\mathcal\{G\}\_\{\\ell\},\\Delta H\_\{\\ell\}\\right\\rangle\_\{F\}\.\(26\)Sinced​Hℓ=\(d​Bℓ\)​PℓalldH\_\{\\ell\}=\(dB\_\{\\ell\}\)P\_\{\\ell\}^\{\\mathrm\{all\}\}, the exact global adapter gradient is

∇Bℓℒ=𝒢ℓ​\(Pℓall\)⊤\.\\nabla\_\{B\_\{\\ell\}\}\\mathcal\{L\}=\\mathcal\{G\}\_\{\\ell\}\(P\_\{\\ell\}^\{\\mathrm\{all\}\}\)^\{\\top\}\.\(27\)If𝒢ℓ\\mathcal\{G\}\_\{\\ell\}were available at every iteration, the ideal hidden\-state target with absolute representation\-space stepsℓ\>0s\_\{\\ell\}\>0would be

H^ℓ=Hℓ−sℓ​𝒢ℓ\.\\widehat\{H\}\_\{\\ell\}=H\_\{\\ell\}\-s\_\{\\ell\}\\mathcal\{G\}\_\{\\ell\}\.\(28\)At the currentBℓB\_\{\\ell\}, the gradient of12​‖H¯ℓ\+Bℓ​Pℓall−H^ℓ‖F2\\frac\{1\}\{2\}\\left\\lVert\\bar\{H\}\_\{\\ell\}\+B\_\{\\ell\}P\_\{\\ell\}^\{\\mathrm\{all\}\}\-\\widehat\{H\}\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\}equalssℓ​∇Bℓℒs\_\{\\ell\}\\nabla\_\{B\_\{\\ell\}\}\\mathcal\{L\}\. Thus the ideal full\-state target reproduces the global adapter\-gradient direction at the current forward state\.

LoCA replaces𝒢ℓ\\mathcal\{G\}\_\{\\ell\}byQℓ=Fℓ​EQ\_\{\\ell\}=F\_\{\\ell\}Eand forms the virtual target

Tℓ=Hℓ−sℓ​Qℓ\.T\_\{\\ell\}=H\_\{\\ell\}\-s\_\{\\ell\}Q\_\{\\ell\}\.\(29\)The following result concerns this virtual change in one block output\. It does not yet concern the fitted adapter\.

###### Proposition D\.1\(Single\-block virtual descent under alignment\)\.

Consider one block and one batch, sosℓs\_\{\\ell\}is a scalar\. Fix the current downstream adapted network and suppose that the upper\-network loss, viewed as a function ofHℓH\_\{\\ell\}, hasMℓM\_\{\\ell\}\-Lipschitz gradient on a convex set containing\{Hℓ−a​sℓ​Qℓ:0≤a≤1\}\\\{H\_\{\\ell\}\-as\_\{\\ell\}Q\_\{\\ell\}:0\\leq a\\leq 1\\\}, whereMℓ\>0M\_\{\\ell\}\>0\. Then

ℒ​\(Hℓ−sℓ​Qℓ\)−ℒ​\(Hℓ\)≤−sℓ​⟨𝒢ℓ,Qℓ⟩F\+Mℓ​sℓ22​‖Qℓ‖F2\.\\mathcal\{L\}\(H\_\{\\ell\}\-s\_\{\\ell\}Q\_\{\\ell\}\)\-\\mathcal\{L\}\(H\_\{\\ell\}\)\\leq\-s\_\{\\ell\}\\left\\langle\\mathcal\{G\}\_\{\\ell\},Q\_\{\\ell\}\\right\\rangle\_\{F\}\+\\frac\{M\_\{\\ell\}s\_\{\\ell\}^\{2\}\}\{2\}\\left\\lVert Q\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\}\.\(30\)IfQℓ≠0Q\_\{\\ell\}\\neq 0and

⟨𝒢ℓ,Qℓ⟩F≥γℓ​‖Qℓ‖F2\(γℓ\>0\),\\left\\langle\\mathcal\{G\}\_\{\\ell\},Q\_\{\\ell\}\\right\\rangle\_\{F\}\\geq\\gamma\_\{\\ell\}\\left\\lVert Q\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\}\\qquad\(\\gamma\_\{\\ell\}\>0\),\(31\)then every

0<sℓ<2​γℓMℓ0<s\_\{\\ell\}<\\frac\{2\\gamma\_\{\\ell\}\}\{M\_\{\\ell\}\}\(32\)gives a strict decrease for the virtual single\-block step\.

###### Proof\.

The descent lemma on the stated convex set gives

ℒ​\(Hℓ\+Δ\)≤ℒ​\(Hℓ\)\+⟨𝒢ℓ,Δ⟩F\+Mℓ2​‖Δ‖F2\.\\mathcal\{L\}\(H\_\{\\ell\}\+\\Delta\)\\leq\\mathcal\{L\}\(H\_\{\\ell\}\)\+\\left\\langle\\mathcal\{G\}\_\{\\ell\},\\Delta\\right\\rangle\_\{F\}\+\\frac\{M\_\{\\ell\}\}\{2\}\\left\\lVert\\Delta\\right\\rVert\_\{F\}^\{2\}\.\(33\)SubstituteΔ=−sℓ​Qℓ\\Delta=\-s\_\{\\ell\}Q\_\{\\ell\}and then use \([31](https://arxiv.org/html/2608.03020#A4.E31)\)\. The resulting upper bound is

−sℓ​\(γℓ−Mℓ​sℓ2\)​‖Qℓ‖F2,\-s\_\{\\ell\}\\left\(\\gamma\_\{\\ell\}\-\\frac\{M\_\{\\ell\}s\_\{\\ell\}\}\{2\}\\right\)\\left\\lVert Q\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\},\(34\)which is negative under \([32](https://arxiv.org/html/2608.03020#A4.E32)\)\. ∎

When both matrices are nonzero, the diagnostic cosine is

αℓ=⟨𝒢ℓ,Qℓ⟩F‖𝒢ℓ‖F​‖Qℓ‖F\.\\alpha\_\{\\ell\}=\\frac\{\\left\\langle\\mathcal\{G\}\_\{\\ell\},Q\_\{\\ell\}\\right\\rangle\_\{F\}\}\{\\left\\lVert\\mathcal\{G\}\_\{\\ell\}\\right\\rVert\_\{F\}\\left\\lVert Q\_\{\\ell\}\\right\\rVert\_\{F\}\}\.\(35\)Positive cosine makes−Qℓ\-Q\_\{\\ell\}a first\-order descent direction, but the admissible step also depends on relative magnitude and local curvature\.

###### Corollary D\.2\(Relative feedback error\)\.

Suppose𝒢ℓ≠0\\mathcal\{G\}\_\{\\ell\}\\neq 0and‖Qℓ−𝒢ℓ‖F≤ε​‖𝒢ℓ‖F\\left\\lVert Q\_\{\\ell\}\-\\mathcal\{G\}\_\{\\ell\}\\right\\rVert\_\{F\}\\leq\\varepsilon\\left\\lVert\\mathcal\{G\}\_\{\\ell\}\\right\\rVert\_\{F\}for0≤ε<10\\leq\\varepsilon<1\. Then the virtual step is strictly descending whenever

0<sℓ<2​\(1−ε\)Mℓ​\(1\+ε\)2\.0<s\_\{\\ell\}<\\frac\{2\(1\-\\varepsilon\)\}\{M\_\{\\ell\}\(1\+\\varepsilon\)^\{2\}\}\.\(36\)

###### Proof\.

Cauchy–Schwarz and the triangle inequality give

⟨𝒢ℓ,Qℓ⟩F\\displaystyle\\left\\langle\\mathcal\{G\}\_\{\\ell\},Q\_\{\\ell\}\\right\\rangle\_\{F\}≥\(1−ε\)​‖𝒢ℓ‖F2,\\displaystyle\\geq\(1\-\\varepsilon\)\\left\\lVert\\mathcal\{G\}\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\},\(37\)‖Qℓ‖F\\displaystyle\\left\\lVert Q\_\{\\ell\}\\right\\rVert\_\{F\}≤\(1\+ε\)​‖𝒢ℓ‖F\.\\displaystyle\\leq\(1\+\\varepsilon\)\\left\\lVert\\mathcal\{G\}\_\{\\ell\}\\right\\rVert\_\{F\}\.\(38\)Substitution into \([30](https://arxiv.org/html/2608.03020#A4.E30)\) proves the claim\. ∎

## Appendix EOne\-Shot Calibration of the Feedback Map

This section derives Eq\. \(5\) in “The Feedback Operator\.” It shows the exact ridge solution used during Stage I and the precise effect of the subsequent rank\-kktruncation\.

The probe loss uses the same masking, weighting, and reduction convention as the adaptation loss\. One backward pass gives the exact hidden\-state gradients in \([23](https://arxiv.org/html/2608.03020#A3.E23)\)\. Select theNpN\_\{p\}probe positions used by the calibration procedure and stack their top errors and exact gradient columns as

E~=\[e\(1\),…,e\(Np\)\]∈ℝd×Np,G~ℓ=\[gℓ\(1\),…,gℓ\(Np\)\]∈ℝd×Np\.\\widetilde\{E\}=\[e^\{\(1\)\},\\ldots,e^\{\(N\_\{p\}\)\}\]\\in\\mathbb\{R\}^\{d\\times N\_\{p\}\},\\qquad\\widetilde\{G\}\_\{\\ell\}=\[g\_\{\\ell\}^\{\(1\)\},\\ldots,g\_\{\\ell\}^\{\(N\_\{p\}\)\}\]\\in\\mathbb\{R\}^\{d\\times N\_\{p\}\}\.\(39\)Because of \([24](https://arxiv.org/html/2608.03020#A3.E24)\), a column ofG~ℓ\\widetilde\{G\}\_\{\\ell\}may already aggregate credit from several later losses\. The calibration therefore fits a predictor from a same\-position top error to that exact gradient column; it does not identify ad×dd\\times dsequence\-level Jacobian\.

The alignment diagnostic in the main paper uses the stacked prediction and exact probe gradient for each block\. The value−0\.005\-0\.005is the mean blockwise cosine for random feedback, while0\.30\.3–0\.50\.5is the observed range after fitting\. These are in\-probe diagnostics, not guarantees for later iterations\.

The full ridge fit is

F^ℓ=arg⁡minF∈ℝd×d⁡‖F​E~−G~ℓ‖F2\+β​‖F‖F2,β\>0\.\\widehat\{F\}\_\{\\ell\}=\\arg\\min\_\{F\\in\\mathbb\{R\}^\{d\\times d\}\}\\left\\lVert F\\widetilde\{E\}\-\\widetilde\{G\}\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\}\+\\beta\\left\\lVert F\\right\\rVert\_\{F\}^\{2\},\\qquad\\beta\>0\.\(40\)
###### Theorem E\.1\(Feedback ridge solution\)\.

Problem \([40](https://arxiv.org/html/2608.03020#A5.E40)\) is strictly convex and has the unique solution

F^ℓ=G~ℓ​E~⊤​\(E~​E~⊤\+β​Id\)−1\.\\widehat\{F\}\_\{\\ell\}=\\widetilde\{G\}\_\{\\ell\}\\widetilde\{E\}^\{\\top\}\\bigl\(\\widetilde\{E\}\\widetilde\{E\}^\{\\top\}\+\\beta I\_\{d\}\\bigr\)^\{\-1\}\.\(41\)Equivalently,

F^ℓ=G~ℓ​\(E~⊤​E~\+β​INp\)−1​E~⊤\.\\widehat\{F\}\_\{\\ell\}=\\widetilde\{G\}\_\{\\ell\}\\bigl\(\\widetilde\{E\}^\{\\top\}\\widetilde\{E\}\+\\beta I\_\{N\_\{p\}\}\\bigr\)^\{\-1\}\\widetilde\{E\}^\{\\top\}\.\(42\)

###### Proof\.

The gradient of the objective is

2​\(F​E~−G~ℓ\)​E~⊤\+2​β​F\.2\(F\\widetilde\{E\}\-\\widetilde\{G\}\_\{\\ell\}\)\\widetilde\{E\}^\{\\top\}\+2\\beta F\.\(43\)The first\-order condition is

F​\(E~​E~⊤\+β​Id\)=G~ℓ​E~⊤\.F\(\\widetilde\{E\}\\widetilde\{E\}^\{\\top\}\+\\beta I\_\{d\}\)=\\widetilde\{G\}\_\{\\ell\}\\widetilde\{E\}^\{\\top\}\.\(44\)For nonzeroxx,

x⊤​\(E~​E~⊤\+β​Id\)​x=‖E~⊤​x‖2\+β​‖x‖2\>0,x^\{\\top\}\(\\widetilde\{E\}\\widetilde\{E\}^\{\\top\}\+\\beta I\_\{d\}\)x=\\left\\lVert\\widetilde\{E\}^\{\\top\}x\\right\\rVert^\{2\}\+\\beta\\left\\lVert x\\right\\rVert^\{2\}\>0,\(45\)so the matrix is positive definite, proving uniqueness and \([41](https://arxiv.org/html/2608.03020#A5.E41)\)\. The push\-through identity

E~⊤​\(E~​E~⊤\+β​Id\)−1=\(E~⊤​E~\+β​INp\)−1​E~⊤\\widetilde\{E\}^\{\\top\}\(\\widetilde\{E\}\\widetilde\{E\}^\{\\top\}\+\\beta I\_\{d\}\)^\{\-1\}=\(\\widetilde\{E\}^\{\\top\}\\widetilde\{E\}\+\\beta I\_\{N\_\{p\}\}\)^\{\-1\}\\widetilde\{E\}^\{\\top\}\(46\)gives \([42](https://arxiv.org/html/2608.03020#A5.E42)\)\. In implementation, these expressions are evaluated by solving a linear system rather than forming an explicit inverse\. ∎

LetF^ℓ=U​Σ​V⊤\\widehat\{F\}\_\{\\ell\}=U\\Sigma V^\{\\top\}, withσ1≥σ2≥⋯≥0\\sigma\_\{1\}\\geq\\sigma\_\{2\}\\geq\\cdots\\geq 0\. The map retained by LoCA is

Fℓ=Uk​Σk​Vk⊤,k=8​in the experiments\.F\_\{\\ell\}=U\_\{k\}\\Sigma\_\{k\}V\_\{k\}^\{\\top\},\\qquad k=8\\ \\text\{in the experiments\}\.\(47\)
###### Lemma E\.2\(Rank\-truncation error\)\.

With the conventionσk\+1=0\\sigma\_\{k\+1\}=0whenkkis at least the rank ofF^ℓ\\widehat\{F\}\_\{\\ell\},

‖F^ℓ−Fℓ‖F2\\displaystyle\\left\\lVert\\widehat\{F\}\_\{\\ell\}\-F\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\}=∑j\>kσj2,\\displaystyle=\\sum\_\{j\>k\}\\sigma\_\{j\}^\{2\},\(48\)‖\(F^ℓ−Fℓ\)​e‖\\displaystyle\\left\\lVert\(\\widehat\{F\}\_\{\\ell\}\-F\_\{\\ell\}\)e\\right\\rVert≤σk\+1​‖e‖\.\\displaystyle\\leq\\sigma\_\{k\+1\}\\left\\lVert e\\right\\rVert\.\(49\)

###### Proof\.

The first identity is the Eckart–Young truncated\-SVD property\. The second follows from‖F^ℓ−Fℓ‖2=σk\+1\\left\\lVert\\widehat\{F\}\_\{\\ell\}\-F\_\{\\ell\}\\right\\rVert\_\{2\}=\\sigma\_\{k\+1\}\. ∎

The truncation is the best rank\-kkFrobenius approximation to the fitted matrixF^ℓ\\widehat\{F\}\_\{\\ell\}\. It is not, in general, the exact optimizer of a rank\-constrained version of \([40](https://arxiv.org/html/2608.03020#A5.E40)\), whose data\-fit term is weighted by the probe covariance\. Nor does the calibration theorem provide an out\-of\-distribution or long\-horizon alignment guarantee\.

## Appendix FClosed\-Form Block Fit and Target Realization

This section expands “Closed\-Form Per\-Block Solve” and Eqs\. \(6\)–\(9\) of the main paper\. It derives the exact ridge minimizer and separates exact optimization of the local quadratic from exact realization of the virtual hidden\-state target\.

At outer iterationtt, letPℓ∈ℝr×NP\_\{\\ell\}\\in\\mathbb\{R\}^\{r\\times N\}stack the selected projected features\. For positionnn, writeqℓ\(n\)=Fℓ​e\(n\)q\_\{\\ell\}^\{\(n\)\}=F\_\{\\ell\}e^\{\(n\)\}and letsℓ\(n\)s\_\{\\ell\}^\{\(n\)\}be the absolute step used for its mini\-batch\. Its residual target is

ρℓ\(n\)=Bℓ\(t\)​pℓ\(n\)−sℓ\(n\)​qℓ\(n\),Rℓ=\[ρℓ\(1\),…,ρℓ\(N\)\]\.\\rho\_\{\\ell\}^\{\(n\)\}=B\_\{\\ell\}^\{\(t\)\}p\_\{\\ell\}^\{\(n\)\}\-s\_\{\\ell\}^\{\(n\)\}q\_\{\\ell\}^\{\(n\)\},\\qquad R\_\{\\ell\}=\[\\rho\_\{\\ell\}^\{\(1\)\},\\ldots,\\rho\_\{\\ell\}^\{\(N\)\}\]\.\(50\)All positions in one mini\-batch share the same step\. During the solve,PℓP\_\{\\ell\}andRℓR\_\{\\ell\}are fixed\.

The local objective is

Φℓ​\(B\)=‖B​Pℓ−Rℓ‖F2\+λ​‖B‖F2,λ\>0\.\\Phi\_\{\\ell\}\(B\)=\\left\\lVert BP\_\{\\ell\}\-R\_\{\\ell\}\\right\\rVert\_\{F\}^\{2\}\+\\lambda\\left\\lVert B\\right\\rVert\_\{F\}^\{2\},\\qquad\\lambda\>0\.\(51\)Define

Gℓ:=Pℓ​Pℓ⊤∈ℝr×r,Cℓ:=Rℓ​Pℓ⊤∈ℝd×r\.G\_\{\\ell\}:=P\_\{\\ell\}P\_\{\\ell\}^\{\\top\}\\in\\mathbb\{R\}^\{r\\times r\},\\qquad C\_\{\\ell\}:=R\_\{\\ell\}P\_\{\\ell\}^\{\\top\}\\in\\mathbb\{R\}^\{d\\times r\}\.\(52\)
###### Theorem F\.1\(Unique closed\-form block minimizer\)\.

Problem \([51](https://arxiv.org/html/2608.03020#A6.E51)\) is strictly convex and has the unique minimizer

Bℓ⋆=Cℓ​\(Gℓ\+λ​Ir\)−1\.B\_\{\\ell\}^\{\\star\}=C\_\{\\ell\}\(G\_\{\\ell\}\+\\lambda I\_\{r\}\)^\{\-1\}\.\(53\)Moreover, for everyB∈ℝd×rB\\in\\mathbb\{R\}^\{d\\times r\},

Φℓ​\(B\)−Φℓ​\(Bℓ⋆\)=tr⁡\(\(B−Bℓ⋆\)​\(Gℓ\+λ​Ir\)​\(B−Bℓ⋆\)⊤\)≥0\.\\Phi\_\{\\ell\}\(B\)\-\\Phi\_\{\\ell\}\(B\_\{\\ell\}^\{\\star\}\)=\\operatorname\{tr\}\\left\(\(B\-B\_\{\\ell\}^\{\\star\}\)\(G\_\{\\ell\}\+\\lambda I\_\{r\}\)\(B\-B\_\{\\ell\}^\{\\star\}\)^\{\\top\}\\right\)\\geq 0\.\(54\)The inequality is strict unlessB=Bℓ⋆B=B\_\{\\ell\}^\{\\star\}\.

###### Proof\.

Expanding the data\-fit term gives

Φℓ​\(B\)\\displaystyle\\Phi\_\{\\ell\}\(B\)=tr⁡\(B​Pℓ​Pℓ⊤​B⊤\)−2​tr⁡\(Rℓ​Pℓ⊤​B⊤\)\+tr⁡\(Rℓ​Rℓ⊤\)\+λ​tr⁡\(B​B⊤\)\.\\displaystyle=\\operatorname\{tr\}\(BP\_\{\\ell\}P\_\{\\ell\}^\{\\top\}B^\{\\top\}\)\-2\\operatorname\{tr\}\(R\_\{\\ell\}P\_\{\\ell\}^\{\\top\}B^\{\\top\}\)\+\\operatorname\{tr\}\(R\_\{\\ell\}R\_\{\\ell\}^\{\\top\}\)\+\\lambda\\operatorname\{tr\}\(BB^\{\\top\}\)\.\(55\)Thus

∇BΦℓ​\(B\)=2​B​\(Gℓ\+λ​Ir\)−2​Cℓ\.\\nabla\_\{B\}\\Phi\_\{\\ell\}\(B\)=2B\(G\_\{\\ell\}\+\\lambda I\_\{r\}\)\-2C\_\{\\ell\}\.\(56\)For nonzerox∈ℝrx\\in\\mathbb\{R\}^\{r\},

x⊤​\(Gℓ\+λ​Ir\)​x=‖Pℓ⊤​x‖2\+λ​‖x‖2\>0\.x^\{\\top\}\(G\_\{\\ell\}\+\\lambda I\_\{r\}\)x=\\left\\lVert P\_\{\\ell\}^\{\\top\}x\\right\\rVert^\{2\}\+\\lambda\\left\\lVert x\\right\\rVert^\{2\}\>0\.\(57\)ThereforeGℓ\+λ​IrG\_\{\\ell\}\+\\lambda I\_\{r\}is positive definite, the objective is strictly convex, and the first\-order condition gives \([53](https://arxiv.org/html/2608.03020#A6.E53)\)\. SubstitutingB=Bℓ⋆\+ΔB=B\_\{\\ell\}^\{\\star\}\+\\Deltaand usingBℓ⋆​\(Gℓ\+λ​Ir\)=CℓB\_\{\\ell\}^\{\\star\}\(G\_\{\\ell\}\+\\lambda I\_\{r\}\)=C\_\{\\ell\}gives \([54](https://arxiv.org/html/2608.03020#A6.E54)\)\. ∎

The desired displacement at fitted positionnnis−sℓ\(n\)​qℓ\(n\)\-s\_\{\\ell\}^\{\(n\)\}q\_\{\\ell\}^\{\(n\)\}\. Stack these displacements asDℓtarD\_\{\\ell\}^\{\\mathrm\{tar\}\}\. The fitted adapter realizes

Δℓfit=\(Bℓ⋆−Bℓ\(t\)\)​Pℓ\.\\Delta\_\{\\ell\}^\{\\mathrm\{fit\}\}=\(B\_\{\\ell\}^\{\\star\}\-B\_\{\\ell\}^\{\(t\)\}\)P\_\{\\ell\}\.\(58\)Their difference is exactly the data residual

Eℓfit:=Δℓfit−Dℓtar=Bℓ⋆​Pℓ−Rℓ\.E\_\{\\ell\}^\{\\mathrm\{fit\}\}:=\\Delta\_\{\\ell\}^\{\\mathrm\{fit\}\}\-D\_\{\\ell\}^\{\\mathrm\{tar\}\}=B\_\{\\ell\}^\{\\star\}P\_\{\\ell\}\-R\_\{\\ell\}\.\(59\)It need not vanish because the target can lie outside the adapter feature span and because ridge regularization trades fit against‖B‖F2\\left\\lVert B\\right\\rVert\_\{F\}^\{2\}\. Moreover, Section S4 uses all sequence positions, whereas the implemented fit uses selected prediction positions\. This restriction is another approximation\.

For an isolated update ofBℓB\_\{\\ell\}with its cached input held fixed, the direct change at all sequence positions is

Δ​Hℓdir=\(Bℓ⋆−Bℓ\(t\)\)​Pℓall\.\\Delta H\_\{\\ell\}^\{\\mathrm\{dir\}\}=\(B\_\{\\ell\}^\{\\star\}\-B\_\{\\ell\}^\{\(t\)\}\)P\_\{\\ell\}^\{\\mathrm\{all\}\}\.\(60\)Under the smoothness assumption of Section S4,

ℒ​\(Hℓ\+Δ​Hℓdir\)−ℒ​\(Hℓ\)≤⟨𝒢ℓ,Δ​Hℓdir⟩F\+Mℓ2​‖Δ​Hℓdir‖F2\.\\mathcal\{L\}\(H\_\{\\ell\}\+\\Delta H\_\{\\ell\}^\{\\mathrm\{dir\}\}\)\-\\mathcal\{L\}\(H\_\{\\ell\}\)\\leq\\left\\langle\\mathcal\{G\}\_\{\\ell\},\\Delta H\_\{\\ell\}^\{\\mathrm\{dir\}\}\\right\\rangle\_\{F\}\+\\frac\{M\_\{\\ell\}\}\{2\}\\left\\lVert\\Delta H\_\{\\ell\}^\{\\mathrm\{dir\}\}\\right\\rVert\_\{F\}^\{2\}\.\(61\)This bound applies to the isolated cached\-input change\. It is not the full state change after a Jacobi update, because simultaneous upstream updates also changeHℓ−1H\_\{\\ell\-1\}andPℓallP\_\{\\ell\}^\{\\mathrm\{all\}\}\. Neither this bound nor the ridge theorem guarantees descent of the outer iteration\.

## Appendix GStreaming Statistics and Numerical Solution

This section expands the practical paragraph after Eq\. \(9\) in “Closed\-Form Per\-Block Solve\.” It proves that the exact sufficient statistics can be accumulated by mini\-batch and states the numerical solve actually required\.

Partition theNNfitted positions into mini\-batchesℐ1,…,ℐM\\mathcal\{I\}\_\{1\},\\ldots,\\mathcal\{I\}\_\{M\}\. Then

Gℓ\\displaystyle G\_\{\\ell\}=∑m=1M∑n∈ℐmpℓ\(n\)​pℓ\(n\)⊤,\\displaystyle=\\sum\_\{m=1\}^\{M\}\\sum\_\{n\\in\\mathcal\{I\}\_\{m\}\}p\_\{\\ell\}^\{\(n\)\}p\_\{\\ell\}^\{\(n\)\\top\},\(62\)Cℓ\\displaystyle C\_\{\\ell\}=∑m=1M∑n∈ℐmρℓ\(n\)​pℓ\(n\)⊤\.\\displaystyle=\\sum\_\{m=1\}^\{M\}\\sum\_\{n\\in\\mathcal\{I\}\_\{m\}\}\\rho\_\{\\ell\}^\{\(n\)\}p\_\{\\ell\}^\{\(n\)\\top\}\.\(63\)Each mini\-batch can be discarded after its contribution has been accumulated\. For explicit nonnegative weightsana\_\{n\}, the same derivation uses

Gℓ\\displaystyle G\_\{\\ell\}=∑nan​pℓ\(n\)​pℓ\(n\)⊤,\\displaystyle=\\sum\_\{n\}a\_\{n\}p\_\{\\ell\}^\{\(n\)\}p\_\{\\ell\}^\{\(n\)\\top\},\(64\)Cℓ\\displaystyle C\_\{\\ell\}=∑nan​ρℓ\(n\)​pℓ\(n\)⊤\.\\displaystyle=\\sum\_\{n\}a\_\{n\}\\rho\_\{\\ell\}^\{\(n\)\}p\_\{\\ell\}^\{\(n\)\\top\}\.\(65\)The corresponding objective is∑nan​‖B​pℓ\(n\)−ρℓ\(n\)‖2\+λ​‖B‖F2\\sum\_\{n\}a\_\{n\}\\left\\lVert Bp\_\{\\ell\}^\{\(n\)\}\-\\rho\_\{\\ell\}^\{\(n\)\}\\right\\rVert^\{2\}\+\\lambda\\left\\lVert B\\right\\rVert\_\{F\}^\{2\}\. The ridge term keeps the system positive definite even when the observed features do not spanℝr\\mathbb\{R\}^\{r\}\.

The persistent statistics requirer2\+d​rr^\{2\}\+drvalues per block and do not grow with the total number of accumulated tokens\. Numerically, LoCA solves

\(Gℓ\+λ​Ir\)​X=Cℓ⊤,Bℓ⋆=X⊤,\(G\_\{\\ell\}\+\\lambda I\_\{r\}\)X=C\_\{\\ell\}^\{\\top\},\\qquad B\_\{\\ell\}^\{\\star\}=X^\{\\top\},\(66\)using a positive\-definite linear solver\. No explicit matrix inverse is required\.

## Appendix HScale\-Normalized Targets and the Outer Iteration

This section expands “Full Algorithm” and Eq\. \(10\) of the main paper\. It defines RMS precisely, distinguishes the dimensionless target coefficient from the absolute step used in Section S4, and explains why exact block solves do not imply convergence of the outer loop\.

For a nonempty matrixX∈ℝa×bX\\in\\mathbb\{R\}^\{a\\times b\}, define

RMS⁡\(X\):=‖X‖Fa​b\.\\operatorname\{RMS\}\(X\):=\\frac\{\\left\\lVert X\\right\\rVert\_\{F\}\}\{\\sqrt\{ab\}\}\.\(67\)The implementation uses a separate scalar for each block and mini\-batch\. For one such batch, letη≥0\\eta\\geq 0be the dimensionless candidate selected by held\-out CE and define

η¯ℓ=\{η​RMS⁡\(Hℓ−1\)RMS⁡\(Qℓ\),RMS⁡\(Qℓ\)\>0,0,Qℓ=0,Dℓtar=−η¯ℓ​Qℓ\.\\bar\{\\eta\}\_\{\\ell\}=\\begin\{cases\}\\displaystyle\\eta\\frac\{\\operatorname\{RMS\}\(H\_\{\\ell\-1\}\)\}\{\\operatorname\{RMS\}\(Q\_\{\\ell\}\)\},&\\operatorname\{RMS\}\(Q\_\{\\ell\}\)\>0,\\\\\[5\.16663pt\] 0,&Q\_\{\\ell\}=0,\\end\{cases\}\\qquad D\_\{\\ell\}^\{\\mathrm\{tar\}\}=\-\\bar\{\\eta\}\_\{\\ell\}Q\_\{\\ell\}\.\(68\)Hereη¯ℓ\\bar\{\\eta\}\_\{\\ell\}is the absolute stepsℓs\_\{\\ell\}used in Section S4\. When Section S6 accumulates several mini\-batches, each target column uses the step computed for its own mini\-batch\.

###### Proposition H\.1\(Target magnitude and positive scale invariance\)\.

IfQℓ≠0Q\_\{\\ell\}\\neq 0, then

RMS⁡\(Dℓtar\)=η​RMS⁡\(Hℓ−1\)\.\\operatorname\{RMS\}\(D\_\{\\ell\}^\{\\mathrm\{tar\}\}\)=\\eta\\,\\operatorname\{RMS\}\(H\_\{\\ell\-1\}\)\.\(69\)ReplacingQℓQ\_\{\\ell\}bya​QℓaQ\_\{\\ell\}for anya\>0a\>0leavesDℓtarD\_\{\\ell\}^\{\\mathrm\{tar\}\}unchanged\.

###### Proof\.

Absolute homogeneity of RMS gives

RMS⁡\(Dℓtar\)=η¯ℓ​RMS⁡\(Qℓ\)=η​RMS⁡\(Hℓ−1\)\.\\operatorname\{RMS\}\(D\_\{\\ell\}^\{\\mathrm\{tar\}\}\)=\\bar\{\\eta\}\_\{\\ell\}\\operatorname\{RMS\}\(Q\_\{\\ell\}\)=\\eta\\operatorname\{RMS\}\(H\_\{\\ell\-1\}\)\.\(70\)Fora\>0a\>0, both the denominator ofη¯ℓ\\bar\{\\eta\}\_\{\\ell\}andQℓQ\_\{\\ell\}acquire the same factoraa, which cancels\. ∎

Equation \([69](https://arxiv.org/html/2608.03020#A8.E69)\) concerns the virtual target, not the direct cached\-input change in \([60](https://arxiv.org/html/2608.03020#A6.E60)\)\. Moreover, the sufficient descent bound in \([32](https://arxiv.org/html/2608.03020#A4.E32)\) applies toη¯ℓ\\bar\{\\eta\}\_\{\\ell\}, not directly to the shared coefficientη\\eta\. Scale normalization supports transfer of a candidate range; it is not a descent theorem and does not make LoCA hyperparameter\-free\.

Let𝒟​\(B\(t\)\)\\mathcal\{D\}\(B^\{\(t\)\}\)denote all local data produced by the current forward pass, including\{Pℓ,H¯ℓ,Qℓ,Rℓ\}ℓ=1L\\\{P\_\{\\ell\},\\bar\{H\}\_\{\\ell\},Q\_\{\\ell\},R\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}, and let𝒮\\mathcal\{S\}apply the blockwise ridge solutions\. A Jacobi outer update is

B\(t\+1\)=𝒮​\(𝒟​\(B\(t\)\)\)\.B^\{\(t\+1\)\}=\\mathcal\{S\}\\bigl\(\\mathcal\{D\}\(B^\{\(t\)\}\)\\bigr\)\.\(71\)Theorem S6\.1 proves that every component of𝒮\\mathcal\{S\}minimizes its current local quadratic\. It does not imply that𝒮∘𝒟\\mathcal\{S\}\\circ\\mathcal\{D\}is a contraction\. After an update, the projected features, frozen\-block outputs, top error, local targets, and realization errors can all change\. Consequently, the next iteration solves different quadratics and global CE can be non\-monotone\. The main paper therefore selects the returned snapshot by held\-out CE and retains the frozen model as a candidate\.

A Jacobi schedule forms all local data from one shared forward pass and then solves blocks independently\. A Gauss–Seidel schedule refreshes downstream states after a block update and therefore requires additional partial forward computation\. No convergence theorem is claimed for either schedule on the nonlinear adapted transformer\.

## Appendix IExperimental Setup and Recovery Metrics

This section corresponds to “Experiments–Setup\.” It records the comparison boundary, explains the recovery ratios, and states which experimental conclusions are descriptive\.

The main comparison uses the frozen model, LoRA, Adapter\-MeZO, and LoCA\. Adapter\-MeZO perturbs the same low\-rank adapter parameterization as LoCA and LoRA; full\-parameter MeZO is reported separately in Section S13\. The primary reported metrics are per\-token cross\-entropy \(CE\) and ranking accuracy\.

The recovery ratios in the main paper are

RCE\\displaystyle R\_\{\\mathrm\{CE\}\}=CEfrozen−CELoCACEfrozen−CELoRA,\\displaystyle=\\frac\{\\mathrm\{CE\}\_\{\\mathrm\{frozen\}\}\-\\mathrm\{CE\}\_\{\\mathrm\{LoCA\}\}\}\{\\mathrm\{CE\}\_\{\\mathrm\{frozen\}\}\-\\mathrm\{CE\}\_\{\\mathrm\{LoRA\}\}\},\(72\)Racc\\displaystyle R\_\{\\mathrm\{acc\}\}=accLoCA−accfrozenaccLoRA−accfrozen\.\\displaystyle=\\frac\{\\mathrm\{acc\}\_\{\\mathrm\{LoCA\}\}\-\\mathrm\{acc\}\_\{\\mathrm\{frozen\}\}\}\{\\mathrm\{acc\}\_\{\\mathrm\{LoRA\}\}\-\\mathrm\{acc\}\_\{\\mathrm\{frozen\}\}\}\.\(73\)A value of one matches the reported LoRA value\. When the frozen\-to\-LoRA denominator is positive, a value above one means that LoCA improves more than LoRA in that cell\. The ratio is unstable near a zero denominator, so such cells are flagged or omitted and raw CE and accuracy remain primary\.

LoCA uses adapter rankr=32r=32, at most4040outer iterations, and held\-out CE for early stopping and for selectingη\\etafrom the stated candidate set\. Unless otherwise noted, table cells are single runs at seed 0\. Cell\-level differences and counts are therefore descriptive rather than estimates of population\-level superiority\.

## Appendix JExperiment 1: Main Benchmark Comparison

This section corresponds to “Experiments–Main Results” and Table 1\. It explains the two observations that need qualification: the count of lower\-CE cells and the CE–accuracy discrepancy on BoolQ\.

Across the2525reported Qwen2\.5 task–scale cells, LoCA has lower evaluation CE than the corresponding LoRA run in1616cells\. This count shows that the local ridge path can differ materially from the LoRA optimization path\. Because most cells are single runs, it is not evidence that LoCA is generally superior to LoRA\.

For cells with a positive frozen\-to\-LoRA denominator, recovery above one means that LoCA improves more than the reported LoRA run\. It is not a significance test, and it can become large when the denominator is small\. The main paper therefore also reports the raw metrics\.

On BoolQ at 7B and 14B, the LoRA runs reduce CE while ranking accuracy falls from0\.8240\.824to0\.6360\.636and from0\.8520\.852to0\.3640\.364\. The LoCA runs reach0\.8400\.840and0\.8640\.864\. This establishes a mismatch between token\-level CE and answer ranking in those runs\. Completion\-format overfitting is a plausible interpretation, but the experiment does not isolate it causally\.

## Appendix KExperiment 2: Resource Use

This section corresponds to “Experiments–Resource Use” and Table 2\. It explains the measurement boundary and connects the observed storage to the sufficient statistics derived in Section S7\.

For each block andNNfitted positions, the principal Stage II operations are

Pℓ=Aℓ​Hℓ−1\\displaystyle P\_\{\\ell\}=A\_\{\\ell\}H\_\{\\ell\-1\}:\\displaystyle:O​\(N​d​r\),\\displaystyle O\(Ndr\),\(74\)Qℓ=Fℓ​E\\displaystyle Q\_\{\\ell\}=F\_\{\\ell\}E:\\displaystyle:O​\(N​d​k\),\\displaystyle O\(Ndk\),\(75\)Gℓ=Pℓ​Pℓ⊤\\displaystyle G\_\{\\ell\}=P\_\{\\ell\}P\_\{\\ell\}^\{\\top\}:\\displaystyle:O​\(N​r2\),\\displaystyle O\(Nr^\{2\}\),\(76\)Cℓ=Rℓ​Pℓ⊤\\displaystyle C\_\{\\ell\}=R\_\{\\ell\}P\_\{\\ell\}^\{\\top\}:\\displaystyle:O​\(N​d​r\),\\displaystyle O\(Ndr\),\(77\)Bℓ⋆\\displaystyle B\_\{\\ell\}^\{\\star\}:\\displaystyle:O​\(r3\+d​r2\)\.\\displaystyle O\(r^\{3\}\+dr^\{2\}\)\.\(78\)The low\-rank feedback cost usesFℓ​e=Uk​\(Σk​\(Vk⊤​e\)\)F\_\{\\ell\}e=U\_\{k\}\(\\Sigma\_\{k\}\(V\_\{k\}^\{\\top\}e\)\)\. The exact head error additionally appliesWu⊤W\_\{u\}^\{\\top\}once per top error; this shared head computation is not multiplied by the number of blocks\.

Beyond the frozen checkpoint and ordinary inference buffers, the persistent Stage II storage is

O​\(L​r2\+L​d​r\+L​d​k\),O\\bigl\(Lr^\{2\}\+Ldr\+Ldk\\bigr\),\(79\)up to constant factors for the fixed projections, adapters, and two feedback factors\. The sufficient\-statistic term does not grow with the total number of processed tokens, although batch size and sequence length still affect transient forward buffers\.

The GPU peak in Table 2 is measured over the full LoCA run and includes the temporary backward graph used in Stage I\. The CPU steady\-state memory and per\-pass time describe Stage II after calibration\. These are different measurement boundaries and should not be combined into a claim that the complete method is forward\-only or that calibration has zero cost\.

Adapter\-MeZO has the lowest memory in the table because it stores neither a backward graph nor ridge statistics\. In the reported runs, it instead uses10310^\{3\}–10410^\{4\}perturbation steps\. LoCA uses at most4040outer iterations, but still pays for calibration and for selectingη\\etaandλ\\lambda\. The measurements support the stated memory and per\-pass comparisons; because calibration latency was not separately recorded, they do not establish a universal end\-to\-end speed advantage\.

## Appendix LExperiment 3: Scale\-Normalized Cross\-Family Transfer

This section corresponds to “Experiments–Cross\-Family Results” and Table 3\. It explains exactly what transfers across model sizes and what is still selected on held\-out data\.

The normalized experiment reuses the candidate set

η∈\{0\.003,0\.01,0\.03\}\\eta\\in\\\{0\.003,0\.01,0\.03\\\}\(80\)for each model reported in Table 3 of the main paper\. A value is still selected from this set by held\-out CE\. Thus the candidate range is reused without redesign, but one fixed value is not used without validation\.

On Qwen2\.5 SST\-2, the normalized candidate set yields the recoveries reported in Table 3 across four model sizes\. The comparison with separately tuned absolute coefficients tests whether relative residual\-stream change is more portable than an absolute hidden\-state correction\. It does not prove scale invariance outside the tested models\.

The SmolLM2\-1\.7B rows provide the cross\-family check\. Reusing the absolute coefficient gives recovery between0and0\.070\.07on the four reported tasks, whereas the normalized target gives recovery between0\.930\.93and1\.291\.29\. This supports the relative target parameterization on a second family\. It does not make the method hyperparameter\-free or establish transfer to arbitrary architectures\.

## Appendix MPrecise Boundary of the Forward\-Only Claim

After calibration, assume that the following objects are available on the adaptation device: the frozen checkpoint, fixed projectionsAℓA\_\{\\ell\}, low\-rank feedback factors forFℓF\_\{\\ell\}, and the current adaptersBℓB\_\{\\ell\}\. Then one outer iteration consists of:

1. 1\.a forward pass through the adapted frozen model;
2. 2\.evaluation of \([20](https://arxiv.org/html/2608.03020#A3.E20)\) at the frozen head;
3. 3\.application ofFℓ​eF\_\{\\ell\}eusing fixed low\-rank factors;
4. 4\.streaming accumulation ofGℓG\_\{\\ell\}andCℓC\_\{\\ell\};
5. 5\.positive\-definite linear solves forBℓB\_\{\\ell\}\.

None of these steps differentiates through a backbone block\. Therefore the adaptation loop is backward\-free after calibration\. The complete method is not backpropagation\-free, because producing the calibrated feedback maps requires the one probe backward pass described in Section S5\. A backward\-capable host may perform calibration and transfer the resulting maps to an inference\-oriented adaptation device, provided the checkpoint and calibration assumptions match\.

## Appendix NMeZO Comparison and Full\-Parameter Context

This section corresponds to the Adapter\-MeZO discussion in “Related Work,” “Experiments–Setup,” and the Discussion section\. It explains why Adapter\-MeZO is the matched baseline and reports the separate full\-parameter MeZO scaling observation used in the main paper\.

Adapter\-MeZO is the main comparison because it perturbs the same low\-rank adapter parameterization as LoCA and LoRA\. The methods therefore act on comparable parameter spaces even though they obtain their updates differently: Adapter\-MeZO uses random function\-value perturbations, whereas LoCA solves the current local ridge objectives\. In the reported runs, Adapter\-MeZO has the smallest memory footprint but uses10310^\{3\}–10410^\{4\}perturbation steps and a learning\-rate search at each scale\.

![Refer to caption](https://arxiv.org/html/2608.03020v1/x1.png)Figure 2:Full\-parameter MeZO on BoolQ across the tested model sizes\. Left: best evaluation CE within the reported runs\. Right: evaluation CE at step 10,000\. The full\-parameter result worsens with scale in this setting, motivating the matched Adapter\-MeZO baseline in the main benchmark\.Figure[2](https://arxiv.org/html/2608.03020#A14.F2)is limited to BoolQ on the tested Qwen2\.5 sizes\. It does not establish algorithmic divergence, a long\-context result, or a general claim about all tasks or zeroth\-order methods\.