ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling

# ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling
Source: [https://arxiv.org/html/2608.07974](https://arxiv.org/html/2608.07974)
Wentao Dai∗, Xuanran Li∗, Yuxiang Zhang, Ming Tang, Chao HuangWentao Dai, Yuxiang Zhang, and Ming Tang are with the Department of Computer Science and Engineering, Southern University of Science and Technology, China\. Xuanran Li is with the Department of Mathematics, Southern University of Science and Technology, China\. Chao Huang is with the School of Computing, Montclair State University, New Jersey, USA\. Email: \{12311217,12312110,12410823\}@mail\.sustech\.edu\.cn, tangm3@sustech\.edu\.cn, huangch@montclair\.edu\. \(Corresponding Author: Ming Tang\)\* Equal Contribution

###### Abstract

Large language model \(LLM\) fine\-tuning at the edge adapts the model to scenario\-specific data while preserving privacy\. Although existing studies proposed pipeline parallelism to address the limited memory and computing resources of edge devices, they commonly rely on backpropagation \(BP\) training, which has a fundamental limitation of update locking and could experience severe throughput and memory bottlenecks\. In this work, we propose a BP\-free algorithm, called ZeroLock, that decouples the model updates into independent chunk updates by local objective construction\. It breaks the update locking of BP and hence can improve throughput at the algorithm level and lower memory usage by reducing activation storage\. To the best of our knowledge, we provide the first theoretical framework for such local objective construction\-based approaches under general model chunk division by mapping local objectives to the global objective\. We prove that ZeroLock has a convergence rate of𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(1/\\sqrt\{T\}\), which differs from BP only by polylogarithmic factors\. We design a system for ZeroLock and build real\-world prototypes, incorporating techniques such as early forwarding and failure recovery for efficient and robust implementation\. Experiments on the prototype show that compared to BP\-based baselines, ZeroLock reduces the memory by 26\.5% and improves throughput by 4\.9%\.

## IIntroduction

Edge intelligence deploys large language models \(LLMs\) at the network edge, achieving privacy\-preserving and providing real\-time inference services\. In practice, it is important to fine\-tune LLMs at the edge for two reasons\. First, data distributions are usually scenario\- and individual\-specific, making it important to adapt the models to maintain inference accuracy\. Second, data privacy concern and possibly frequent model adaptation requirements make it difficult to upload the data to the cloud for fine\-tuning\. There are many such examples requiring both model adaptation and privacy preserving\. For example, test\-time training\[[27](https://arxiv.org/html/2608.07974#bib.bib41)\]adapts LLMs to maintain user\-specific memory for personalized service provision\. Electroencephalogram signal\-based tasks\[[2](https://arxiv.org/html/2608.07974#bib.bib45)\]\(e\.g\., emotion recognition, sleep staging\) require model adaptation for individuals, as those signals demonstrate strong individual\-dependent patterns\.

However, devices at the network edge usually have limited computation and memory resources, so it is difficult for a device or a GPU to fine\-tune the entire LLM\. To address this, existing studies have proposed*pipeline parallelism*approaches\[[13](https://arxiv.org/html/2608.07974#bib.bib10),[20](https://arxiv.org/html/2608.07974#bib.bib11),[6](https://arxiv.org/html/2608.07974#bib.bib1)\]\. The main idea is to partition the model vertically into chunks of consecutive layers\. The update of each model chunk corresponds to a*stage*and is assigned to a different device, and these devices collaborate to update the chunks in a pipeline fashion\. GPipe\[[13](https://arxiv.org/html/2608.07974#bib.bib10)\]is a typical approach and processes micro\-batches sequentially\. 1F1B\[[20](https://arxiv.org/html/2608.07974#bib.bib11)\]interleaves forward pass and backward update to improve pipeline utilization\. Building upon 1F1B, PipeDream\[[20](https://arxiv.org/html/2608.07974#bib.bib11),[21](https://arxiv.org/html/2608.07974#bib.bib12)\]enables asynchronous pipeline training via weight stashing\. Recent studies\[[7](https://arxiv.org/html/2608.07974#bib.bib13),[22](https://arxiv.org/html/2608.07974#bib.bib14),[16](https://arxiv.org/html/2608.07974#bib.bib15),[24](https://arxiv.org/html/2608.07974#bib.bib16)\]reduce bubbles in pipeline through placement search, virtual or bidirectional stages, and finer backward decomposition\. Other studies\[[17](https://arxiv.org/html/2608.07974#bib.bib53),[8](https://arxiv.org/html/2608.07974#bib.bib54),[29](https://arxiv.org/html/2608.07974#bib.bib55),[5](https://arxiv.org/html/2608.07974#bib.bib52),[25](https://arxiv.org/html/2608.07974#bib.bib20)\]focused on pipeline planning considering preemptions, stragglers, device heterogeneity, or communication efficiency\. Confidant\[[6](https://arxiv.org/html/2608.07974#bib.bib1)\]implements 1F1B on smartphones\.

![Refer to caption](https://arxiv.org/html/2608.07974v1/x1.png)

\(a\)

![Refer to caption](https://arxiv.org/html/2608.07974v1/x2.png)

\(b\) ![Refer to caption](https://arxiv.org/html/2608.07974v1/x3.png)![Refer to caption](https://arxiv.org/html/2608.07974v1/x4.png)

\(c\) \(d\)

Figure 1:Throughput with \(a\) BP baseline and \(b\) ZeroLock; memory with \(c\) BP baseline and \(d\) ZeroLock\. In this figure, we use the typical 1F1B framework as the BP baseline, because many existing studies \(e\.g\.,\[[6](https://arxiv.org/html/2608.07974#bib.bib1)\]\) are built upon it\. S0, S1, and S2 are stages, each corresponding to the updates of a model chunk and assigned to a device for execution\. Fnn, Bnn, and Anndenote the forward pass, backward pass, and activations of thenn\-th microbathch, repetitively\. The stripes in \(c\) and \(d\) indicate the duration that the corresponding activations are kept in memory\. For example, at time slot 3 of S0, A1\-A3 are kept in memory in \(c\), while only A2 is kept in memory in \(d\)\.Most of these pipeline parallelism works \(e\.g\.,\[[13](https://arxiv.org/html/2608.07974#bib.bib10),[20](https://arxiv.org/html/2608.07974#bib.bib11),[21](https://arxiv.org/html/2608.07974#bib.bib12),[7](https://arxiv.org/html/2608.07974#bib.bib13),[22](https://arxiv.org/html/2608.07974#bib.bib14),[16](https://arxiv.org/html/2608.07974#bib.bib15),[24](https://arxiv.org/html/2608.07974#bib.bib16),[17](https://arxiv.org/html/2608.07974#bib.bib53),[8](https://arxiv.org/html/2608.07974#bib.bib54),[29](https://arxiv.org/html/2608.07974#bib.bib55),[6](https://arxiv.org/html/2608.07974#bib.bib1),[5](https://arxiv.org/html/2608.07974#bib.bib52),[4](https://arxiv.org/html/2608.07974#bib.bib31),[25](https://arxiv.org/html/2608.07974#bib.bib20)\]\) rely on backpropagation \(BP\) training and focus system\-level scheduling optimization\.BP training has a fundamental limitation of update locking\[[14](https://arxiv.org/html/2608.07974#bib.bib47),[30](https://arxiv.org/html/2608.07974#bib.bib8),[31](https://arxiv.org/html/2608.07974#bib.bib46)\]\. That is, the model update contains a forward pass followed by a backward pass, so the update of upstream layers needs to wait for the forward and backward computation of downstream layers\. As a result, the approaches based on BP could have crucial limitations:

- •Throughput Bottleneck:As shown in Fig\.[1](https://arxiv.org/html/2608.07974#S1.F1)\(a\), due to update locking, bubbles arise because an upstream stage \(e\.g\., S0\) needs to wait for the backward pass of all downstream stages \(e\.g\., S1, S2\) to accomplish its own backward pass\. Although existing works \(e\.g\.,\[[20](https://arxiv.org/html/2608.07974#bib.bib11),[21](https://arxiv.org/html/2608.07974#bib.bib12)\]\) have proposed approaches to eliminate the bubbles, the updates of stages remain coupled due to the BP training, and hence the fundamental issue has not been resolved\.
- •Memory Bottleneck:As shown in Fig\.[1](https://arxiv.org/html/2608.07974#S1.F1)\(c\), due to update locking, an upstream stage \(e\.g\., S0\) needs to keep its activations until all its downstream stages and itself \(e\.g\., S0, S1, S2\) accomplish their backward passes\. This leads to significant memory waste for upstream stages\.

The pervasive use of BP training and the resulting throughput and memory bottlenecks motivate the main question:

###### Main Question\.

How can we design a system that fundamentally overcome the throughput and memory bottlenecks at the algorithm level by breaking the update locking in BP?

BP\-free training was proposed to break the update locking in BP\. There are two categories\. \(I\) Backward Gradient Estimation\. For example, direct feedback alignment\[[23](https://arxiv.org/html/2608.07974#bib.bib2),[1](https://arxiv.org/html/2608.07974#bib.bib3)\]bypasses the chain rule by propagating target errors directly to all layers\. Zero\-order optimization\[[18](https://arxiv.org/html/2608.07974#bib.bib4),[28](https://arxiv.org/html/2608.07974#bib.bib5)\]computes perturbation\-based loss differences during the forward pass to estimate gradients\. However, this category usually leads to either significant model accuracy degradation for complex tasks or significant computing overhead\. \(II\) Objective Reconstruction\. For example, NoProp\[[15](https://arxiv.org/html/2608.07974#bib.bib6)\]reframes neural modules as independent denoising units that map noisy target embeddings back to clean targets\. Predictive coding\[[19](https://arxiv.org/html/2608.07974#bib.bib7)\]alternates between minimizing chunk\-wise prediction errors and updating weights using the local errors\. Local objective construction \(e\.g\., depth\-progressive monotonic learning\[[30](https://arxiv.org/html/2608.07974#bib.bib8)\]\) equips each layer with an independent local objective, allowing purely local gradient computation\. Within category \(II\),local objective construction decouples the modular updates across stagesand has comparable model accuracy to BP approaches, so it is a promising candidate to address the Main Question\. Specifically, since each chunk is updated independently using its local loss, its update does not need to wait for the forward and backward passes of its downstream chunks, so bubbles due to waiting can be removed; meanwhile, each chunk needs to keep its activations only until its own updates, reducing memory usage\.

In this work, based on local objective construction\-based BP\-free approaches, we aim to propose a pipeline parallelism LLM training system to fundamentally overcome the throughput and memory bottlenecks\. However, this design is not straightforward and needs to address the following questions:

- Q1How to fine\-tune LLM with local objective construction?
- Q2Does the modular decoupling of local objective construction theoretically harm the model convergence?
- Q3How to design and construct real\-world prototype systems \(for both multi\-GPU server and Android phone scenarios\) to achieve efficient and robust implementation?

Answering Q1 requires concrete design to incorporate LLM\-specific characteristics \(e\.g\., token sequence, low\-rank adaptation for efficient fine\-tuning\) into local objective construction framework\. Answering Q2 is very challenging\. This is because the modular update decoupling breaks global objective into local objectives of model chunks, while recent studies lack frameworks to quantifying how optimizing the local objectives improves the global objective\. Answering Q3 is non\-trivial due to the real\-world engineering requirements of light\-weight, high\-throughput, and robust implementation\.

Existing works tried to address one or two of Q1–Q3\. PPLL\[[9](https://arxiv.org/html/2608.07974#bib.bib39)\]places gradient\-isolated vision blocks on different GPUs and transfers their features for block\-wise updates\. FluidPipe\[[3](https://arxiv.org/html/2608.07974#bib.bib40)\]focuses on LLM and adds an auxiliary task head to the first part of a two\-stage model, avoiding the local updates on the first part from waiting for the second part’s gradient\. SCPL\[[11](https://arxiv.org/html/2608.07974#bib.bib42)\]decouples BP via per\-segment supervised contrastive losses for synchronized multi\-GPU model parallelism\. However, these works\[[9](https://arxiv.org/html/2608.07974#bib.bib39),[3](https://arxiv.org/html/2608.07974#bib.bib40),[11](https://arxiv.org/html/2608.07974#bib.bib42)\]considered hidden state alignment for Q1, failing to characterize task\-specific information in local objective construction, and they did not address Q2\. For Q3, although they propose high\-level pipeline logic, their designs remain at a conceptual level, lacking critical implementation details such as continuous cross\-batch pipelining, stage I/O queues, and explicit RPC primitives\. Meanwhile, they did not provide deployable prototype for real\-world execution\. Although recent work LoPT\[[26](https://arxiv.org/html/2608.07974#bib.bib48)\]addressed Q2, its analysis limits to two chunks and fails to provide a general analytical framework for connecting local objective to global objective\.

We address those challenges and answer Q1–Q3\. Our main contributions are summarized as follows:

- •ZeroLock Algorithm: To answer Q1, we propose a local objective construction\-based BP\-free algorithm, called ZeroLock, for LLM fine\-tuning, which incorporates low\-rank adaptation \(LoRA\)\[[12](https://arxiv.org/html/2608.07974#bib.bib44)\]and LLM\-compatible readout head and loss function\. It breaks the update coupling across model chunks, and hence reduces the pipeline bubbles and activation storage at the algorithm level\.
- •Theoretical Analysis: To answer Q2, we establish theoretical equivalences to represent the local optimum of model chunks in a form of global objective\. Then, the local updates of model chunks can be equivalently represented as global updates, with which the global convergence can be derived\. To the best of our knowledge, this is the first analytical framework for the local objective construction algorithms under general model chunk division, which provides a systematic approach for analyzing global convergence given decoupled local updates\. We prove that ZeroLock has a convergence rate of𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(1/\\sqrt\{T\}\), which differs from BP by only polylogarithmic factors\.
- •Real\-World System Design: To answer Q3, based on ZeroLock, we design systems and build prototypes for both multi\-GPU server and Android phone scenarios\. This system can achieve concurrent throughput \(see Fig\.[1](https://arxiv.org/html/2608.07974#S1.F1)\(b\)\) and lower memory usage by reducing activation storage \(see Fig\.[1](https://arxiv.org/html/2608.07974#S1.F1)\(d\)\)\. Techniques such as early forwarding and failure recovery are proposed to ensure light\-weight, high\-throughput, and robust implementation\. To the best of our knowledge, we build the first prototype on Android system for local objective construction\-based BP\-free algorithm\.
- •Evaluation on Real\-World Prototype: Experiments on multi\-GPU server show that when compared with BP\-based baselines, ZeroLock system reduces the memory usage by 26\.5% and improves the throughput by 4\.9%\. The fine\-tuning of TinyLlama on Android phones experiences a peak PSS of less than 4000 MiB, a battery temperature of around 37∘C, and a wall\-clock time of 1644\.1 seconds, indicating that the implementation is practically feasible\. Our code is available at[https://anonymous\.4open\.science/r/unlock\_trainer\-105B](https://anonymous.4open.science/r/unlock_trainer-105B)\.

The rest of this paper is organized as follows\. Sections[II](https://arxiv.org/html/2608.07974#S2)and[III](https://arxiv.org/html/2608.07974#S3)present ZeroLock algorithm and its system design, respectively\. Experiments are in Section[IV](https://arxiv.org/html/2608.07974#S4)\. Section[V](https://arxiv.org/html/2608.07974#S5)concludes this work\.

## IIZeroLock Algorithm and Analysis

### II\-AZeroLock Algorithm

Consider an LLM with an embedding operationE​\(x\)E\(x\)and a set of transformer layersℒ\\mathcal\{L\}\. We introduce LoRA\[[12](https://arxiv.org/html/2608.07974#bib.bib44)\]to reduce the trainable parameters in fine\-tuning\. Letwl∈ℝd1×d2w\_\{l\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{2\}\}denote the base weights of thell\-th transformer layer, which is frozen during fine\-tuning\. LetAl∈ℝd3×d2A\_\{l\}\\in\\mathbb\{R\}^\{d\_\{3\}\\times d\_\{2\}\}andBl∈ℝd1×d3B\_\{l\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\_\{3\}\}denote the trainable low\-rank decomposition matrices in LoRA\. Then, the parameter updates of transformer layerl∈ℒl\\in\\mathcal\{L\}is represented byΔ​wl=Bl​Al\\Delta w\_\{l\}=B\_\{l\}A\_\{l\}, so the weights after fine\-tuning arewl\+Δ​wlw\_\{l\}\+\\Delta w\_\{l\}\. To achieve modular update decoupling in LLM fine\-tuning, we partition the entire model into model chunks and introduce local objectives to these chunks for independent updates\.111The main idea is inspired by the local objective construction approach in\[[30](https://arxiv.org/html/2608.07974#bib.bib8)\]\. Different from\[[30](https://arxiv.org/html/2608.07974#bib.bib8)\]on classification tasks, ZeroLock handles LLM fine\-tuning by incorporating LoRA and LLM\-compatible head and loss design\.

Chunk Partitioning:Consider an LLM split into an embedding operationE​\(⋅\)E\(\\cdot\)andKKchunks, where chunkkkcontains one or multiple consecutive transformer layers, denoted by setℒk\\mathcal\{L\}\_\{k\}\. LetWk=\(wl\|l∈ℒk\)W\_\{k\}=\(w\_\{l\}~\|~l\\in\\mathcal\{L\}\_\{k\}\)andΔ​Wk=\(Δ​wl\|l∈ℒk\)\\Delta W\_\{k\}=\(\\Delta w\_\{l\}~\|~l\\in\\mathcal\{L\}\_\{k\}\)denote the frozen base weights and the trainable LoRA parameters in chunkkk, respectively, leading to a mappingfk​\(⋅;Wk\+Δ​Wk\)f\_\{k\}\(\\cdot;W\_\{k\}\+\\Delta W\_\{k\}\)from the chunk input to output\. Letxxbe the LLM input\. Define𝒉0∈ℝS×d\\boldsymbol\{h\}\_\{0\}\\in\\mathbb\{R\}^\{S\\times d\}and𝒉k∈ℝS×d\\boldsymbol\{h\}\_\{k\}\\in\\mathbb\{R\}^\{S\\times d\}as the token embedding and the output of chunkkk, respectively, withSSas the sequence length:

𝒉0=E​\(x\),𝒉k=fk​\(𝒉k−1;Wk\+Δ​Wk\),k=1,⋯,K\.\\boldsymbol\{h\}\_\{0\}=E\(x\),~\\boldsymbol\{h\}\_\{k\}=f\_\{k\}\(\\boldsymbol\{h\}\_\{k\-1\};W\_\{k\}\+\\Delta W\_\{k\}\),k=1,\\cdots,K\.\(1\)
Readout Head:To enable local updates, we introduce a frozen local readout head after chunkk=1,⋯,Kk=1,\\cdots,K:

𝒛k=Norm​\(𝒉k\)​Wlm⊤,𝒑k,i=softmax​\(𝒛k,i\)\.\\boldsymbol\{z\}\_\{k\}=\\mathrm\{Norm\}\(\\boldsymbol\{h\}\_\{k\}\)W\_\{\\mathrm\{lm\}\}^\{\\top\},~\\boldsymbol\{p\}\_\{k,i\}=\\mathrm\{softmax\}\(\\boldsymbol\{z\}\_\{k,i\}\)\.\(2\)In \([2](https://arxiv.org/html/2608.07974#S2.E2)\),WlmW\_\{\\mathrm\{lm\}\}is the head weight that maps the normalized hidden representationNorm​\(𝒉k\)\\mathrm\{Norm\}\(\\boldsymbol\{h\}\_\{k\}\)to vocabulary logits𝒛k∈ℝS×V\\boldsymbol\{z\}\_\{k\}\\in\\mathbb\{R\}^\{S\\times V\}, withVVbeing the vocabulary size\. Vector𝒛k,i\\boldsymbol\{z\}\_\{k,i\}is theii\-th row of𝒛k∈ℝS×V\\boldsymbol\{z\}\_\{k\}\\in\\mathbb\{R\}^\{S\\times V\}\. Withsoftmax​\(⋅\)\\mathrm\{softmax\}\(\\cdot\),𝒑k,i∈ℝ1×V\\boldsymbol\{p\}\_\{k,i\}\\in\\mathbb\{R\}^\{1\\times V\}is the predictive distribution of next token over the vocabulary space\.

Local Objective:With the output of the readout head, the local loss of each chunkkkconsists of a task\-dependent termLTask​\(⋅\)L\_\{\\mathrm\{Task\}\}\(\\cdot\)and a consistency termLConsisk→k−1​\(⋅\)L\_\{\\mathrm\{Consis\}\}^\{k\\rightarrow k\-1\}\(\\cdot\):

Lk​\(𝒑k\)=α​LTask​\(𝒑k\)\+\(1−α\)​LConsisk→k−1​\(𝒑k\),L\_\{k\}\(\\boldsymbol\{p\}\_\{k\}\)=\\alpha L\_\{\\mathrm\{Task\}\}\(\\boldsymbol\{p\}\_\{k\}\)\+\(1\-\\alpha\)L\_\{\\mathrm\{Consis\}\}^\{k\\rightarrow k\-1\}\(\\boldsymbol\{p\}\_\{k\}\),\(3\)whereα∈\(0,1\]\\alpha\\in\(0,1\]is a weight, and𝒑k=\(𝒑k,i,i∈Ω\)\\boldsymbol\{p\}\_\{k\}=\(\\boldsymbol\{p\}\_\{k,i\},i\\in\\Omega\)withΩ\\Omegaas the token sequence removing prompt or padding positions\.

\(i\) Task\-dependent termLTaskL\_\{\\mathrm\{Task\}\}is a local version of the global objective\. It aligns the output of the readout head of each chunkkkto the global ground\-truth target:

LTask​\(𝒑k\)=−1\|Ω\|​∑i∈ΩDψ​\(𝒑k,i,𝒑y\),L\_\{\\mathrm\{Task\}\}\(\\boldsymbol\{p\}\_\{k\}\)=\-\\frac\{1\}\{\|\\Omega\|\}\\sum\_\{i\\in\\Omega\}D\_\{\\psi\}\\\!\\left\(\\boldsymbol\{p\}\_\{k,i\},\\boldsymbol\{p\}\_\{y\}\\right\),\(4\)where𝒑y∈ℛ1×V\\boldsymbol\{p\}\_\{y\}\\in\\mathcal\{R\}^\{1\\times V\}is the one\-hot target\. Let𝒫\\mathcal\{P\}denote the space of𝒑\\boldsymbol\{p\}\.Dψ​\(⋅\)D\_\{\\psi\}\(\\cdot\)is the Bregman divergence induced by a strictly convex and differentiable potential functionψ:𝒫→ℝ\\psi:\\mathcal\{P\}\\rightarrow\\mathbb\{R\}, i\.e\.,Dψ​\(𝒖,𝒗\)=ψ​\(𝒖\)−ψ​\(𝒗\)−⟨∇ψ​\(𝒗\),𝒖−𝒗⟩D\_\{\\psi\}\(\\boldsymbol\{u\},\\boldsymbol\{v\}\)=\\psi\(\\boldsymbol\{u\}\)\-\\psi\(\\boldsymbol\{v\}\)\-\\langle\\nabla\\psi\(\\boldsymbol\{v\}\),\\boldsymbol\{u\}\-\\boldsymbol\{v\}\\rangle\. KL divergence is a specific example of Bregman divergence by selecting negative entropy as the potential function\. Importantly, since minimizing KL divergence is equivalent to minimizing cross\-entropy, \([4](https://arxiv.org/html/2608.07974#S2.E4)\) can be replaced with cross\-entropy loss in practice\.

\(ii\) Consistency term aligns the outputs of chunkkkto those of chunkk−1k\-1, ensuring coherent outputs across chunks\.

LConsisk→k−1​\(𝒑k\)=1\|Ω\|​∑i∈ΩDψ​\(𝒑k,i,sg​\(𝒑k−1,i\)\),L\_\{\\mathrm\{Consis\}\}^\{k\\rightarrow k\-1\}\(\\boldsymbol\{p\}\_\{k\}\)=\\frac\{1\}\{\|\\Omega\|\}\\sum\_\{i\\in\\Omega\}D\_\{\\psi\}\\\!\\left\(\\boldsymbol\{p\}\_\{k,i\},\\mathrm\{sg\}\(\\boldsymbol\{p\}\_\{k\-1,i\}\)\\right\),\(5\)wheresg​\(⋅\)\\mathrm\{sg\}\(\\cdot\)is stop\-gradient operator withsg​\(𝒑\)=𝒑\\mathrm\{sg\}\(\\boldsymbol\{p\}\)=\\boldsymbol\{p\}and∇sg​\(𝒑\)=𝟎\\nabla\\mathrm\{sg\}\(\\boldsymbol\{p\}\)=\\boldsymbol\{0\}\. Similarly, KL divergence can be used in practice\.

Fine\-Tuning Process:This process containsTTiterations\. In each iteration, the low rank matricesAlA\_\{l\}andBlB\_\{l\}of layerl∈ℒkl\\in\\mathcal\{L\}\_\{k\}are fine\-tuned using stochastic gradient descents:

Al←Al−η​∂fk∂Al,Bl←Bl−η​∂fk∂Bl,A\_\{l\}\\leftarrow A\_\{l\}\-\\eta\\frac\{\\partial f\_\{k\}\}\{\\partial A\_\{l\}\},~B\_\{l\}\\leftarrow B\_\{l\}\-\\eta\\frac\{\\partial f\_\{k\}\}\{\\partial B\_\{l\}\},\(6\)whereη\\etais the learning rate\.

### II\-BTheoretical Analysis: Chunk\-Wise Performance

With chunk\-wise performance analysis, we aim to provide insights into how the global objective value changes across model chunks\. We focus on one token in the sequence and omit the token subscript\. Recall that𝒑\\boldsymbol\{p\}is the readout head output of a chunk and is a distribution over vocabulary space\. LetL:𝒫→ℝL:\\mathcal\{P\}\\rightarrow\\mathbb\{R\}denote the global objective given𝒑∈𝒫\\boldsymbol\{p\}\\in\\mathcal\{P\}, i\.e\.,L​\(𝒑\)≜LTask​\(𝒑\)L\(\\boldsymbol\{p\}\)\\triangleq L\_\{\\mathrm\{Task\}\}\(\\boldsymbol\{p\}\), which is the loss function characterizing how much𝒑\\boldsymbol\{p\}deviates from global ground\-truth target\. Let𝒑k∗\\boldsymbol\{p\}\_\{k\}^\{\*\}denote the optimal𝒑k\\boldsymbol\{p\}\_\{k\}of chunkkkthat minimizes the local lossLk​\(⋅\)L\_\{k\}\(\\cdot\), i\.e\.,𝒑k∗≜arg⁡min𝒑∈𝒫⁡Lk​\(𝒑\)\\boldsymbol\{p\}\_\{k\}^\{\*\}\\triangleq\\arg\\min\_\{\\boldsymbol\{p\}\\in\\mathcal\{P\}\}L\_\{k\}\(\\boldsymbol\{p\}\):

###### Lemma 1\(Local Optimum\)\.

The local optimum𝐩k∗\\boldsymbol\{p\}\_\{k\}^\{\*\}can be equivalently represented by

𝒑k∗=arg⁡min𝒑∈𝒫⁡\{⟨∇L​\(𝒑k−1\),𝒑−𝒑k−1⟩\+1α​Dψ​\(𝒑,𝒑k−1\)\}\.\\boldsymbol\{p\}\_\{k\}^\{\*\}\\\!=\\\!\\arg\\min\_\{\\boldsymbol\{p\}\\in\\mathcal\{P\}\}\\left\\\{\\langle\\nabla L\(\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\),\\boldsymbol\{p\}\\\!\-\\\!\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\\rangle\\\!\+\\\!\\frac\{1\}\{\\alpha\}D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\)\\right\\\}\.\(7\)

###### Proof\.

Recall the definition of the Bregman divergence,

Dψ​\(𝒖,𝒗\)=ψ​\(𝒖\)−ψ​\(𝒗\)−⟨∇ψ​\(𝒗\),𝒖−𝒗⟩\.D\_\{\\psi\}\(\\boldsymbol\{u\},\\boldsymbol\{v\}\)=\\psi\(\\boldsymbol\{u\}\)\-\\psi\(\\boldsymbol\{v\}\)\-\\langle\\nabla\\psi\(\\boldsymbol\{v\}\),\\boldsymbol\{u\}\-\\boldsymbol\{v\}\\rangle\.\(8\)Substituting\(𝒖=𝒑,𝒗=𝒑k−1\)\(\\boldsymbol\{u\}=\\boldsymbol\{p\},\\boldsymbol\{v\}=\\boldsymbol\{p\}\_\{k\-1\}\)and\(𝒖=𝒑,𝒗=𝒑y\)\(\\boldsymbol\{u\}=\\boldsymbol\{p\},\\boldsymbol\{v\}=\\boldsymbol\{p\}\_\{y\}\)into \([8](https://arxiv.org/html/2608.07974#S2.E8)\) respectively and rearranging the former, we haveψ​\(𝒑\)=ψ​\(𝒑k−1\)\+⟨∇ψ​\(𝒑k−1\),𝒑−𝒑k−1⟩\+Dψ​\(𝒑,𝒑k−1\)\\psi\(\\boldsymbol\{p\}\)\\\!=\\\!\\psi\(\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\)\\\!\+\\\!\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\)\\\!,\\\!\\boldsymbol\{p\}\-\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\\rangle\\\!\+\\\!D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\),Dψ​\(𝒑,𝒑y\)=ψ​\(𝒑\)−ψ​\(𝒑y\)−⟨∇ψ​\(𝒑y\),𝒑−𝒑y⟩D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{y\}\)=\\psi\(\\boldsymbol\{p\}\)\-\\psi\(\\boldsymbol\{p\}\_\{y\}\)\-\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{y\}\),\\boldsymbol\{p\}\-\\boldsymbol\{p\}\_\{y\}\\rangle\. Thus,

Dψ​\(𝒑,𝒑y\)=\\displaystyle D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{y\}\)\{=\}ψ​\(𝒑k−1\)−ψ​\(𝒑y\)−⟨∇ψ​\(𝒑y\),𝒑k−1−𝒑y⟩\\displaystyle\\psi\(\\boldsymbol\{p\}\_\{k\-1\}\)\-\\psi\(\\boldsymbol\{p\}\_\{y\}\)\-\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{y\}\),\\boldsymbol\{p\}\_\{k\-1\}\-\\boldsymbol\{p\}\_\{y\}\\rangle−⟨∇ψ​\(𝒑y\),𝒑−𝒑y⟩\+⟨∇ψ​\(𝒑y\),𝒑k−1−𝒑y⟩\\displaystyle\-\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{y\}\),\\boldsymbol\{p\}\-\\boldsymbol\{p\}\_\{y\}\\rangle\+\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{y\}\),\\boldsymbol\{p\}\_\{k\-1\}\-\\boldsymbol\{p\}\_\{y\}\\rangle\+⟨∇ψ​\(𝒑k−1\),𝒑−𝒑k−1⟩\+Dψ​\(𝒑,𝒑k−1\)\\displaystyle\+\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{k\-1\}\),\\boldsymbol\{p\}\-\\boldsymbol\{p\}\_\{k\-1\}\\rangle\+D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{k\-1\}\)=\(a\)\\displaystyle\\overset\{\(a\)\}\{=\}Dψ​\(𝒑k−1,𝒑y\)\+⟨∇L​\(𝒑k−1\),𝒑−𝒑k−1⟩\\displaystyle D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\-1\},\\boldsymbol\{p\}\_\{y\}\)\+\\langle\\nabla L\(\\boldsymbol\{p\}\_\{k\-1\}\),\\boldsymbol\{p\}\-\\boldsymbol\{p\}\_\{k\-1\}\\rangle\+Dψ​\(𝒑,𝒑k−1\),\\displaystyle\+D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{k\-1\}\),\(9\)where \(a\) holds as the definition ofDψ​\(𝒑k−1,𝒑y\)D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\-1\},\\boldsymbol\{p\}\_\{y\}\)andL​\(𝒑i−1\)L\(\\boldsymbol\{p\}\_\{i\-1\}\)\.

Substituting \([II\-B](https://arxiv.org/html/2608.07974#S2.Ex1)\) into the local loss functionLk​\(𝒑\)L\_\{k\}\(\\boldsymbol\{p\}\)leads toLk​\(𝒑\)=α​Dψ​\(𝒑k−1,𝒑y\)\+α​⟨∇L​\(𝒑k−1\),𝒑−𝒑k−1⟩\+Dψ​\(𝒑,𝒑k−1\)L\_\{k\}\(\\boldsymbol\{p\}\)=\\alpha D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\-1\},\\boldsymbol\{p\}\_\{y\}\)\+\\alpha\\langle\\nabla L\(\\boldsymbol\{p\}\_\{k\-1\}\),\\boldsymbol\{p\}\-\\boldsymbol\{p\}\_\{k\-1\}\\rangle\+D\_\{\\psi\}\(\\boldsymbol\{p\},\\boldsymbol\{p\}\_\{k\-1\}\)\. Given𝒑k−1\\boldsymbol\{p\}\_\{k\-1\}, termα​Dψ​\(𝒑k−1,𝒑y\)\\alpha D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\-1\},\\boldsymbol\{p\}\_\{y\}\)is a constant, so it can be omitted in optimization\. Proof completes\. ∎

Lemma[1](https://arxiv.org/html/2608.07974#Thmlemma1)represents the local optimum𝒑⋆\\boldsymbol\{p\}^\{\\star\}in a form of the global objectiveL​\(⋅\)L\(\\cdot\), connecting local and global objectives\. Meanwhile, it demonstrates a balance between moving toward the steepest descent direction of global objectiveL​\(𝒑\)L\(\\boldsymbol\{p\}\)and penalizing deviations from the previous distribution𝒑k−1\\boldsymbol\{p\}\_\{k\-1\}\.

As in many existing convergence analysis on BP or BP\-free \(e\.g\.,\[[10](https://arxiv.org/html/2608.07974#bib.bib43),[26](https://arxiv.org/html/2608.07974#bib.bib48)\]\), we assume the smoothness of global objective\.

###### Assumption 1\(Global Objective Smoothness\)\.

The global objectiveL​\(⋅\)L\(\\cdot\)isβ\\beta\-smooth relative toψ\\psiin Bregman geometry:

L​\(𝒖\)≤L​\(𝒗\)\+⟨∇L​\(𝒗\),𝒖−𝒗⟩\+β​Dψ​\(𝒖,𝒗\),𝒖,𝒗∈𝒫\.L\(\\boldsymbol\{u\}\)\\leq L\(\\boldsymbol\{v\}\)\+\\langle\\nabla L\(\\boldsymbol\{v\}\),\\boldsymbol\{u\}\-\\boldsymbol\{v\}\\rangle\+\\beta D\_\{\\psi\}\(\\boldsymbol\{u\},\\boldsymbol\{v\}\),~\\boldsymbol\{u\},\\boldsymbol\{v\}\\in\\mathcal\{P\}\.\(10\)

Then, we derive the chunk\-wise performance of ZeroLock\.

###### Proposition 1\(Chunk\-Wise Performance\)\.

Suppose𝐩k−1\\boldsymbol\{p\}\_\{k\-1\},𝐩k⋆\\boldsymbol\{p\}\_\{k\}^\{\\star\}, and𝐩k\\boldsymbol\{p\}\_\{k\}lie in a compact set of the relative interior of the simplex, andψ​\(⋅\)\\psi\(\\cdot\)is locally strongly convex\. Defineδk≜Dψ​\(𝐩k,𝐩k∗\)\\delta\_\{k\}\\triangleq D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\},\\boldsymbol\{p\}\_\{k\}^\{\*\}\)\. Under Assumption[1](https://arxiv.org/html/2608.07974#Thmassumption1)andα<1β\\alpha<\\frac\{1\}\{\\beta\},

L​\(𝒑K\)≤L​\(𝒑0\)−∑k=1K\(1α−β\)​Dψ​\(𝒑k,𝒑k−1\)\+1α​∑k=1Kδk\.L\(\\boldsymbol\{p\}\_\{K\}\)\\\!\\leq\\\!L\(\\boldsymbol\{p\}\_\{0\}\)\\\!\-\\\!\\sum\_\{k=1\}^\{K\}\\left\(\\frac\{1\}\{\\alpha\}\\\!\-\\\!\\beta\\right\)D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\},\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}\)\\\!\+\\\!\\frac\{1\}\{\\alpha\}\\sum\_\{k=1\}^\{K\}\\delta\_\{k\}\.\(11\)

###### Proof\.

Substituting𝒖=𝒑k\\boldsymbol\{u\}=\\boldsymbol\{p\}\_\{k\}and𝒗=𝒑k−1\\boldsymbol\{v\}=\\boldsymbol\{p\}\_\{k\-1\}in Assumption[1](https://arxiv.org/html/2608.07974#Thmassumption1),

L​\(𝒑k\)≤⟨∇L​\(𝒑k−1\),𝒑k−𝒑k∗⟩⏟\(i\)\+⟨∇L​\(𝒑k−1\),𝒑k∗−𝒑k−1⟩⏟\(i​i\)\+L​\(𝒑k−1\)\+β​Dψ​\(𝒑k,𝒑k−1\)\.L\(\\boldsymbol\{p\}\_\{k\}\)\\leq\\underbrace\{\\langle\\nabla L\(\\boldsymbol\{p\}\_\{k\-1\}\),\\boldsymbol\{p\}\_\{k\}\-\\boldsymbol\{p\}\_\{k\}^\{\*\}\\rangle\}\_\{\(i\)\}\+\\underbrace\{\\langle\\nabla L\(\\boldsymbol\{p\}\_\{k\-1\}\),\\boldsymbol\{p\}\_\{k\}^\{\*\}\-\\boldsymbol\{p\}\_\{k\-1\}\\rangle\}\_\{\(ii\)\}\\\\ \+L\(\\boldsymbol\{p\}\_\{k\-1\}\)\+\\beta D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\},\\boldsymbol\{p\}\_\{k\-1\}\)\.\(12\)
Since𝒑k⋆\\boldsymbol\{p\}\_\{k\}^\{\\star\}lies in the relative interior of the probability simplex, the KKT condition of \([7](https://arxiv.org/html/2608.07974#S2.E7)\) gives∇L​\(𝒑k−1\)\+1α​\(∇ψ​\(𝒑k⋆\)−∇ψ​\(𝒑k−1\)\)\+λk​𝟏=0\\nabla L\(\\boldsymbol\{p\}\_\{k\-1\}\)\+\\frac\{1\}\{\\alpha\}\\bigl\(\\nabla\\psi\(\\boldsymbol\{p\}\_\{k\}^\{\\star\}\)\-\\nabla\\psi\(\\boldsymbol\{p\}\_\{k\-1\}\)\\bigr\)\+\\lambda\_\{k\}\\mathbf\{1\}=0\. Accordingly, term \(i\) satisfies

α​⟨∇L​\(𝒑k−1\),𝒑k−𝒑k⋆⟩\\displaystyle\\alpha\\langle\\nabla L\(\\boldsymbol\{p\}\_\{k\-1\}\),\\boldsymbol\{p\}\_\{k\}\-\\boldsymbol\{p\}\_\{k\}^\{\\star\}\\rangle=\(a\)\\displaystyle\\overset\{\(a\)\}\{=\}⟨∇ψ​\(𝒑k−1\)−∇ψ​\(𝒑k⋆\),𝒑k−𝒑k⋆⟩\\displaystyle\\langle\\nabla\\psi\(\\boldsymbol\{p\}\_\{k\-1\}\)\-\\nabla\\psi\(\\boldsymbol\{p\}\_\{k\}^\{\\star\}\),\\boldsymbol\{p\}\_\{k\}\-\\boldsymbol\{p\}\_\{k\}^\{\\star\}\\rangle=\(b\)\\displaystyle\\overset\{\(b\)\}\{=\}δk\+Dψ​\(𝒑k⋆,𝒑k−1\)−Dψ​\(𝒑k,𝒑k−1\)\.\\displaystyle\\delta\_\{k\}\+D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\}^\{\\star\},\\boldsymbol\{p\}\_\{k\-1\}\)\-D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\},\\boldsymbol\{p\}\_\{k\-1\}\)\.\(13\)Here, \(a\) holds due to the KKT condition and𝟏⊤​\(𝒑k−𝒑k⋆\)=0\\mathbf\{1\}^\{\\top\}\(\\boldsymbol\{p\}\_\{k\}\-\\boldsymbol\{p\}\_\{k\}^\{\\star\}\)=0, where the latter holds since𝒑k−𝒑k⋆\\boldsymbol\{p\}\_\{k\}\-\\boldsymbol\{p\}\_\{k\}^\{\\star\}lies in the simplex tangent space\. \(b\) holds based on Bregman three\-point identity\. According to the definition of𝒑k∗\\boldsymbol\{p\}\_\{k\}^\{\*\}, term \(ii\) satisfies⟨𝒈k,𝒑k∗−𝒑k−1⟩\+1α​Dψ​\(𝒑k∗,𝒑k−1\)≤0\\langle\\boldsymbol\{g\}\_\{k\},\\boldsymbol\{p\}\_\{k\}^\{\*\}\-\\boldsymbol\{p\}\_\{k\-1\}\\rangle\+\\frac\{1\}\{\\alpha\}D\_\{\\psi\}\(\\boldsymbol\{p\}\_\{k\}^\{\*\},\\boldsymbol\{p\}\_\{k\-1\}\)\\leq 0\. Substituting \(i\) and \(ii\) into \([12](https://arxiv.org/html/2608.07974#S2.E12)\) and summing overk=1,…,Kk=1,\\dots,Kcomplete the proof\. ∎

Proposition[1](https://arxiv.org/html/2608.07974#Thmproposition1)shows that the global lossL​\(𝒑K\)L\(\\boldsymbol\{p\}\_\{K\}\)of the final chunk tends to decrease as the number of chunks increases, with a bounded error∑k=1Kδk/α\\sum\_\{k=1\}^\{K\}\\delta\_\{k\}/\\alpharesulting from the fine\-tuning suboptimality of model chunks due to finite parameters\.222Although a relevant analysis on chunk\-wise performance is presented in\[[30](https://arxiv.org/html/2608.07974#bib.bib8)\], our Proposition[1](https://arxiv.org/html/2608.07974#Thmproposition1)generalizes that in\[[30](https://arxiv.org/html/2608.07974#bib.bib8)\]\(i\) from KL divergence to general Bregman divergence and \(ii\) by explicitly deriving the specific form of the errorδk\\delta\_\{k\}\(rather than simply assuming an error\)\.

### II\-CTheoretical Analysis: Algorithm Convergence

We provide the first analytical framework for local objective construction\-based BP\-free approach under general chunk division\. As mentioned earlier, the major challenge comes from characterizing global convergence based on the local updates of each model chunk\. To overcome this, we equivalently map local updates to global updates \(Lemma[3](https://arxiv.org/html/2608.07974#Thmlemma3)\); based on this, we prove that the convergence rate of ZeroLock is𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(1/\\sqrt\{T\}\)\(Theorem[1](https://arxiv.org/html/2608.07974#Thmtheorem1)\), differing from𝒪​\(1/T\)\\mathcal\{O\}\(1/\\sqrt\{T\}\)of BP by a polylogarithmic factor\.333f​\(T\)=𝒪~​\(g​\(T\)\)f\(T\)=\\tilde\{\\mathcal\{O\}\}\(g\(T\)\)meansf​\(T\)=𝒪​\(g​\(T\)​logc⁡T\)f\(T\)=\\mathcal\{O\}\(g\(T\)\\log^\{c\}T\)for some constantcc\. The extralog⁡T\\log Tis a slow\-varying factor that becomes negligible for largeTT\.

For iterationtt, let𝝎t=\(𝝎kt,k=0,1,⋯,K\)\\boldsymbol\{\\omega\}^\{t\}=\(\\boldsymbol\{\\omega\}\_\{k\}^\{t\},k=0,1,\\cdots,K\)denote the parameters of chunks to be updated\. We introduce this notation to generalize the analysis for various scenarios \(e\.g\., LLM, CNN\)\. Following the notation in Section[II\-A](https://arxiv.org/html/2608.07974#S2.SS1),𝝎0t\\boldsymbol\{\\omega\}\_\{0\}^\{t\}is the fixed parameters of operatorE​\(⋅\)E\(\\cdot\), and𝝎kt≜Δ​Wkt\+Wk\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\\triangleq\\Delta W\_\{k\}^\{t\}\+W\_\{k\}for chunkkk\. Let𝒑0t,𝒑1t,…,𝒑Kt\\boldsymbol\{p\}^\{t\}\_\{0\},\\boldsymbol\{p\}^\{t\}\_\{1\},\\dots,\\boldsymbol\{p\}^\{t\}\_\{K\}denote the readout head output of the associated model chunks\. For analytical simplicity, we set the Bregman divergence in \([4](https://arxiv.org/html/2608.07974#S2.E4)\) and \([5](https://arxiv.org/html/2608.07974#S2.E5)\) as KL divergence\. Then, the local loss is equivalent to

Lkt​\(𝒑;α\)=α​DKL​\(𝒑∥𝒑y\)\+\(1−α\)​DKL​\(𝒑∥𝒑k−1t\)\.L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\)=\\alpha D\_\{\\mathrm\{KL\}\}\(\\boldsymbol\{p\}~\\\|~\\boldsymbol\{p\}\_\{y\}\)\+\(1\-\\alpha\)D\_\{\\mathrm\{KL\}\}\(\\boldsymbol\{p\}~\\\|~\\boldsymbol\{p\}\_\{k\-1\}^\{t\}\)\.\(14\)
Mapping from Local to Global Updates:First, we show the equivalence on local objective as follows\.

###### Lemma 2\(Objective Equivalence\)\.

MinimizingLkt​\(𝐩;α\)L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\)is equivalent to minimizingDKL​\(𝐩∥𝐩k∗,t​\(α\)\)D\_\{\\mathrm\{KL\}\}\\left\(\\boldsymbol\{p\}~\\\|~\\boldsymbol\{p\}\_\{k\}^\{\*,t\}\(\\alpha\)\\right\), where

𝒑k∗,t​\(l;α\)=𝒑y​\(l\)α​\(𝒑k−1t​\(l\)\)1−α∑l′=1m𝒑y​\(l′\)α​\(𝒑k−1t​\(l′\)\)1−α,\\boldsymbol\{p\}\_\{k\}^\{\*,t\}\(l;\\alpha\)=\\frac\{\\boldsymbol\{p\}\_\{y\}\(l\)^\{\\alpha\}\\left\(\\boldsymbol\{p\}\_\{k\-1\}^\{t\}\(l\)\\right\)^\{1\-\\alpha\}\}\{\\sum\_\{l^\{\\prime\}=1\}^\{m\}\\boldsymbol\{p\}\_\{y\}\(l^\{\\prime\}\)^\{\\alpha\}\\left\(\\boldsymbol\{p\}\_\{k\-1\}^\{t\}\(l^\{\\prime\}\)\\right\)^\{1\-\\alpha\}\},\(15\)withlldenoting thell\-th element of the associated vector andmmdenoting the length of the vector\.

###### Proof\.

Local lossLkt​\(𝒑;α\)L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\)can be equivalently represented as

Lkt​\(𝒑;α\)=∑l=1m𝒑​\(l\)​log⁡𝒑​\(l\)𝒑y​\(l\)α​\(𝒑kt​\(l\)\)1−α\.L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\)=\\sum\_\{l=1\}^\{m\}\\boldsymbol\{p\}\(l\)\\log\\frac\{\\boldsymbol\{p\}\(l\)\}\{\\boldsymbol\{p\}\_\{y\}\(l\)^\{\\alpha\}\\left\(\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)\\right\)^\{1\-\\alpha\}\}\.\(16\)DefineZkt​\(α\)=∑l=1m𝒑y​\(l\)α​\(𝒑k−1t​\(l\)\)1−αZ\_\{k\}^\{t\}\(\\alpha\)=\\sum\_\{l=1\}^\{m\}\\boldsymbol\{p\}\_\{y\}\(l\)^\{\\alpha\}\\left\(\\boldsymbol\{p\}\_\{k\-1\}^\{t\}\(l\)\\right\)^\{1\-\\alpha\}\. Then,

Lkt​\(𝒑;α\)=DKL​\(𝒑∥𝒑k∗,t​\(α\)\)−log⁡Zkt​\(α\)\.L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\)=D\_\{\\mathrm\{KL\}\}\(\\boldsymbol\{p\}~\\\|~\\boldsymbol\{p\}\_\{k\}^\{\*,t\}\(\\alpha\)\)\-\\log Z\_\{k\}^\{t\}\(\\alpha\)\.\(17\)Note that for any chunkkk, vectors𝒑y\\boldsymbol\{p\}\_\{y\}and𝒑k−1t\\boldsymbol\{p\}\_\{k\-1\}^\{t\}are given, soZkt​\(α\)Z\_\{k\}^\{t\}\(\\alpha\)is a constant\. Hence, this lemma is proved\. ∎

Note that𝒑k∗,t\(α\)≜\(𝒑k∗,t\(l;α\),l=1,⋯,m\)\\boldsymbol\{p\}\_\{k\}^\{\\ast,t\}\(\\alpha\)\\triangleq\(\\boldsymbol\{p\}\_\{k\}^\{\\ast,t\}\(l;\\alpha\),l=1,\\cdots,m\)is essentially the optimal solution that minimizesLkt​\(𝒑;α\)L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\), with proof omitted here\. Thus, minimizingDKL​\(𝒑∥𝒑k∗,t​\(α\)\)D\_\{\\mathrm\{KL\}\}\\left\(\\boldsymbol\{p\}~\\\|~\\boldsymbol\{p\}\_\{k\}^\{\*,t\}\(\\alpha\)\\right\)is equivalent to minimizing the difference between𝒑\\boldsymbol\{p\}and the optimal solution\.

Then, we can represent global updates in ZeroLock as a form of local updates\. We define𝒑kt=f^k​\(𝒑k−1t;𝝎kt\)\\boldsymbol\{p\}\_\{k\}^\{t\}=\\hat\{f\}\_\{k\}\(\\boldsymbol\{p\}\_\{k\-1\}^\{t\};\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\)as the mapping from the readout head output𝒑k−1t\\boldsymbol\{p\}\_\{k\-1\}^\{t\}of chunkkkto𝒑kt\\boldsymbol\{p\}\_\{k\}^\{t\}, given parameter𝝎kt\\boldsymbol\{\\omega\}\_\{k\}^\{t\}of chunkkk\. Although this mapping cannot be directly obtained using the readout head operation \(as it is not a one\-to\-one correspondence\), it can be approximated given the statistics of𝒑kt\\boldsymbol\{p\}\_\{k\}^\{t\}across chunks\. Also, this mapping is introduced for theoretical analysis, and it does not need to be obtained in practice\. Define the Jacobian matrix𝑱θk,kt≜∂f^k​\(𝒑k−1t;𝝎kt\)/∂𝝎kt\\boldsymbol\{J\}\_\{\\theta\_\{k\},k\}^\{t\}\\triangleq\\partial\\hat\{f\}\_\{k\}\(\\boldsymbol\{p\}\_\{k\-1\}^\{t\};\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\)/\\partial\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\. The stop\-gradient operation induces the following Jacobian matrix for all parameters:

𝑱t=blkdiag⁡\(𝑱𝝎1,1t,𝑱𝝎2,2t,⋯,𝑱𝝎K,Kt\),\\boldsymbol\{J\}^\{t\}=\\operatorname\{blkdiag\}\(\\boldsymbol\{J\}\_\{\\boldsymbol\{\\omega\}\_\{1\},1\}^\{t\},\\boldsymbol\{J\}\_\{\\boldsymbol\{\\omega\}\_\{2\},2\}^\{t\},\\cdots,\\boldsymbol\{J\}\_\{\\boldsymbol\{\\omega\}\_\{K\},K\}^\{t\}\),\(18\)whereblkdiag⁡\(⋅\)\\operatorname\{blkdiag\}\(\\cdot\)denotes block\-diagonal matrix with all elements outside the diagonal blocks being zero\. Define𝒆kt​\(α\)=log⁡𝒑kt−log⁡𝒑k∗,t​\(α\)\\boldsymbol\{e\}\_\{k\}^\{t\}\(\\alpha\)=\\log\\boldsymbol\{p\}\_\{k\}^\{t\}\-\\log\\boldsymbol\{p\}\_\{k\}^\{\\ast,t\}\(\\alpha\), which is the gap between the recent𝒑kt\\boldsymbol\{p\}\_\{k\}^\{t\}and the optimal𝒑k∗,t​\(α\)\\boldsymbol\{p\}\_\{k\}^\{\\ast,t\}\(\\alpha\)in logarithmic form\.

Based on Lemma[2](https://arxiv.org/html/2608.07974#Thmlemma2)and𝒑kt=f^k​\(𝒑k−1t;𝝎kt\)\\boldsymbol\{p\}\_\{k\}^\{t\}=\\hat\{f\}\_\{k\}\(\\boldsymbol\{p\}\_\{k\-1\}^\{t\};\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\), minimizingLkt​\(𝒑;α\)L\_\{k\}^\{t\}\(\\boldsymbol\{p\};\\alpha\)for all chunks over dataset𝒟\\mathcal\{D\}is equivalent to finding the parameter𝝎=\(𝝎k,k=0,1,…,K\)\\boldsymbol\{\\omega\}=\(\\boldsymbol\{\\omega\}\_\{k\},k=0,1,\\dots,K\)that minimizes

ℛt​\(𝝎;α\)=𝔼\(x,y\)∼𝒟​\[∑k=1KDKL​\(f^k​\(𝒑k−1t;𝝎k\)∥𝒑k∗,t​\(α\)\)\]\.\\\!\\\!\\\!\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\\\!=\\\!\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\}\\\!\\\!\\left\[\\sum\_\{k=1\}^\{K\}\\\!D\_\{\\mathrm\{KL\}\}\\\!\\\!\\left\(\\hat\{f\}\_\{k\}\(\\boldsymbol\{p\}\_\{k\\\!\-\\\!1\}^\{t\};\\boldsymbol\{\\omega\}\_\{k\}\)\\\|\\boldsymbol\{p\}\_\{k\}^\{\\ast,t\}\(\\alpha\)\\\!\\right\)\\\!\\right\]\\\!\.\(19\)
###### Lemma 3\(Global Update Equivalence\)\.

Based on Lemma[2](https://arxiv.org/html/2608.07974#Thmlemma2), the update rule of ZeroLock can be represented as follows:444The algorithm in\[[30](https://arxiv.org/html/2608.07974#bib.bib8)\]differs from ZeroLock by maintaining an encoder and having its outputc​\(x\)c\(x\)as the input for each chunk\. To generalize it, we can append a column of𝐉𝛚c,kt\\boldsymbol\{J\}^\{t\}\_\{\\boldsymbol\{\\omega\}\_\{c\},k\}fork=1,⋯,Kk=1,\\cdots,Kbefore the columns in𝐉t\\boldsymbol\{J\}^\{t\}, where𝐉𝛚c,kt\\boldsymbol\{J\}^\{t\}\_\{\\boldsymbol\{\\omega\}\_\{c\},k\}is the Jacobian matrix offkt​\(⋅\)f\_\{k\}^\{t\}\(\\cdot\)with respect to the parameters𝛚c\\boldsymbol\{\\omega\}\_\{c\}of the encoder\. Under this modification, Lemma[3](https://arxiv.org/html/2608.07974#Thmlemma3)still holds\.

𝝎t\+1←𝝎t−ηt​∇𝝎ℛt​\(𝝎t;α\),\\boldsymbol\{\\omega\}^\{t\+1\}\\leftarrow\\boldsymbol\{\\omega\}^\{t\}\-\\eta\_\{t\}\\nabla\_\{\\boldsymbol\{\\omega\}\}\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\};\\alpha\),\(20\)where∇𝛚ℛt​\(𝛚;α\)=\(𝐉t\)⊤​𝐞t\\nabla\_\{\\boldsymbol\{\\omega\}\}\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)=\(\\boldsymbol\{J\}^\{t\}\)^\{\\top\}\\boldsymbol\{e\}^\{t\},𝐞t≜\(𝐞kt,k=0,1,⋯,K\)\\boldsymbol\{e\}^\{t\}\\triangleq\(\\boldsymbol\{e\}^\{t\}\_\{k\},k=0,1,\\cdots,K\)\.

###### Proof\.

According to \([17](https://arxiv.org/html/2608.07974#S2.E17)\) in Lemma[2](https://arxiv.org/html/2608.07974#Thmlemma2), given output𝒑kt\\boldsymbol\{p\}\_\{k\}^\{t\},

∇𝒑ktLkt​\(𝒑kt;α\)=log⁡𝒑kt−log⁡𝒑k∗,t​\(α\)\+𝟏\.\\nabla\_\{\\boldsymbol\{p\}\_\{k\}^\{t\}\}L\_\{k\}^\{t\}\(\\boldsymbol\{p\}\_\{k\}^\{t\};\\alpha\)=\\log\\boldsymbol\{p\}\_\{k\}^\{t\}\-\\log\\boldsymbol\{p\}\_\{k\}^\{\\ast,t\}\(\\alpha\)\+\\mathbf\{1\}\.\(21\)We apply the chain rule\. Letℓkt​\(𝝎kt\)≜Lkt​\(𝒑kt\)\\ell\_\{k\}^\{t\}\(\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\)\\triangleq L\_\{k\}^\{t\}\(\\boldsymbol\{p\}\_\{k\}^\{t\}\), and we omitα\\alphafor presentation simplicity\. For theuu\-th coordinate of𝝎k\\boldsymbol\{\\omega\}\_\{k\},

∂ℓkt​\(𝝎kt\)∂𝝎k,ut=∑l=1m∂Lkt​\(𝒑kt\)∂𝒑kt​\(l\)​∂𝒑kt​\(l\)∂𝝎k,ut​=\(a\)​∑l=1m𝒆kt​\(l\)​∂𝒑kt​\(l\)∂𝝎k,u\.\\frac\{\\partial\\ell\_\{k\}^\{t\}\(\\boldsymbol\{\\omega\}\_\{k\}^\{t\}\)\}\{\\partial\\boldsymbol\{\\omega\}\_\{k,u\}^\{t\}\}=\\sum\_\{l=1\}^\{m\}\\frac\{\\partial L\_\{k\}^\{t\}\(\\boldsymbol\{p\}\_\{k\}^\{t\}\)\}\{\\partial\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)\}\\frac\{\\partial\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)\}\{\\partial\\boldsymbol\{\\omega\}\_\{k,u\}^\{t\}\}\\overset\{\(a\)\}\{=\}\\sum\_\{l=1\}^\{m\}\\boldsymbol\{e\}\_\{k\}^\{t\}\(l\)\\frac\{\\partial\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)\}\{\\partial\\boldsymbol\{\\omega\}\_\{k,u\}\}\.\(22\)Equality \(a\) holds because∑l=1m∂𝒑kt​\(l\)∂𝝎ut=∂∂𝝎ut​∑l=1m𝒑kt​\(l\)=0\\sum\_\{l=1\}^\{m\}\\frac\{\\partial\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)\}\{\\partial\\boldsymbol\{\\omega\}\_\{u\}^\{t\}\}=\\frac\{\\partial\}\{\\partial\\boldsymbol\{\\omega\}\_\{u\}^\{t\}\}\\sum\_\{l=1\}^\{m\}\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)=0, since∑l=1m𝒑kt​\(l\)=1\\sum\_\{l=1\}^\{m\}\\boldsymbol\{p\}\_\{k\}^\{t\}\(l\)=1, and \([21](https://arxiv.org/html/2608.07974#S2.E21)\)\. Finally, stacking all coordinates and the gradients of all chunks yields this lemma\. ∎

Convergence:Define‖𝒂‖𝒟2≜𝔼x∼𝒟X​\[∑i=1I‖ai​\(x\)‖22\]\\\|\\boldsymbol\{a\}\\\|\_\{\\mathcal\{D\}\}^\{2\}\\triangleq\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{X\}\}\[\\sum\_\{i=1\}^\{I\}\\\|a\_\{i\}\(x\)\\\|\_\{2\}^\{2\}\],⟨𝒂,𝒃⟩𝒟≜𝔼x∼𝒟X​\[∑i=1I⟨ai​\(x\),bi​\(x\)⟩\]\\langle\\boldsymbol\{a\},\\boldsymbol\{b\}\\rangle\_\{\\mathcal\{D\}\}\\triangleq\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{X\}\}\[\\sum\_\{i=1\}^\{I\}\\langle a\_\{i\}\(x\),b\_\{i\}\(x\)\\rangle\]\. Here,IIis the size of𝒂\\boldsymbol\{a\}and𝒃\\boldsymbol\{b\},xxis the input of the model, and𝒟X\\mathcal\{D\}\_\{X\}is the distribution ofxxin dataset𝒟\\mathcal\{D\}\. Let𝒑​\(𝝎\)=\(𝒑k​\(𝝎\)≜f^k​\(𝒑k−1;𝝎k\),k=0,⋯,K\)\\boldsymbol\{p\}\(\\boldsymbol\{\\omega\}\)=\(\\boldsymbol\{p\}\_\{k\}\(\\boldsymbol\{\\omega\}\)\\triangleq\\hat\{f\}\_\{k\}\(\\boldsymbol\{p\}\_\{k\-1\};\\boldsymbol\{\\omega\}\_\{k\}\),k=0,\\cdots,K\)denote the readout head output given𝝎\\boldsymbol\{\\omega\}\. We rewriteℛt​\(𝝎;α\)=ℛ​\(𝝎;α,𝒑~t\)\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)=\\mathcal\{R\}\(\\boldsymbol\{\\omega\};\\alpha,\\tilde\{\\boldsymbol\{p\}\}^\{t\}\)where𝒑t=𝒑​\(𝝎t\)\\boldsymbol\{p\}^\{t\}=\\boldsymbol\{p\}\(\\boldsymbol\{\\omega\}^\{t\}\)for presentation simplicity\. The following Assumptions[2](https://arxiv.org/html/2608.07974#Thmassumption2)–[4](https://arxiv.org/html/2608.07974#Thmassumption4)are commonly considered in existing analysis \(e\.g\.,\[[26](https://arxiv.org/html/2608.07974#bib.bib48),[10](https://arxiv.org/html/2608.07974#bib.bib43)\]\)\. Assumption[5](https://arxiv.org/html/2608.07974#Thmassumption5)is reasonable given the bounded gradient in Assumption[2](https://arxiv.org/html/2608.07974#Thmassumption2)and the probability simplex space of𝒑~t\\tilde\{\\boldsymbol\{p\}\}^\{t\}\. Assumption[6](https://arxiv.org/html/2608.07974#Thmassumption6)is reasonable under simple random sampling and given the bounded gradient\.

###### Assumption 2\(Bounded Gradient\)\.

The gradient is upper bounded, i\.e\.,𝔼​\[‖∇𝛚ℛt​\(𝛚;α\)‖2\]≤M​\(α\)\\mathbb\{E\}\[\\\|\\nabla\_\{\\boldsymbol\{\\omega\}\}\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\\\|^\{2\}\]\\leq M\(\\alpha\)\.

###### Assumption 3\(Lipschitz Continuity\)\.

Readout output𝐩​\(𝛚\)\\boldsymbol\{p\}\(\\boldsymbol\{\\omega\}\)is Lipschitz continuous with respect to𝛚\\boldsymbol\{\\omega\}, i\.e\.,‖𝐩​\(𝛚\)−𝐩​\(𝛚′\)‖𝒟≤ρ​‖𝛚−𝛚′‖\\\|\\boldsymbol\{p\}\(\\boldsymbol\{\\omega\}\)\-\\boldsymbol\{p\}\(\\boldsymbol\{\\omega\}^\{\\prime\}\)\\\|\_\{\\mathcal\{D\}\}\\leq\\rho\\\|\\boldsymbol\{\\omega\}\-\\boldsymbol\{\\omega\}^\{\\prime\}\\\|, for all𝛚\\boldsymbol\{\\omega\}and𝛚′\\boldsymbol\{\\omega\}^\{\\prime\}in parameter space\.

###### Assumption 4\(Smoothness\)\.

For any𝛚\\boldsymbol\{\\omega\}in parameter space,ℛt​\(𝛚;α\)\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)isβR​\(α\)\\beta\_\{R\}\(\\alpha\)\-smooth, i\.e\.,ℛt​\(𝛚′;α\)≤ℛt​\(𝛚;α\)\+⟨∇ℛt​\(𝛚;α\),𝛚′−𝛚⟩\+βR​\(α\)2​‖𝛚′−𝛚‖2\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{\\prime\};\\alpha\)\\leq\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\+\\langle\\nabla\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\),\\boldsymbol\{\\omega\}^\{\\prime\}\-\\boldsymbol\{\\omega\}\\rangle\+\\frac\{\\beta\_\{R\}\(\\alpha\)\}\{2\}\\\|\\boldsymbol\{\\omega\}^\{\\prime\}\-\\boldsymbol\{\\omega\}\\\|^\{2\}\.

###### Assumption 5\(Stability\)\.

The first\-order objective variation is quadratically controlled along the distribution update, i\.e\.,\[⟨∇𝐩~tℛ​\(𝛚t;α,𝐩~t\),𝐩~t\+1−𝐩~t⟩𝒟\]\+≤Cfo​\(α\)​‖𝐩~t\+1−𝐩~t‖𝒟2\\left\[\\left\\langle\\nabla\_\{\\tilde\{\\boldsymbol\{p\}\}^\{t\}\}\\mathcal\{R\}\(\\boldsymbol\{\\omega\}^\{t\};\\alpha,\\tilde\{\\boldsymbol\{p\}\}^\{t\}\),\\tilde\{\\boldsymbol\{p\}\}^\{t\+1\}\-\\tilde\{\\boldsymbol\{p\}\}^\{t\}\\right\\rangle\_\{\\mathcal\{D\}\}\\right\]\_\{\+\}\\leq C\_\{\\rm fo\}\(\\alpha\)\\\|\\tilde\{\\boldsymbol\{p\}\}^\{t\+1\}\-\\tilde\{\\boldsymbol\{p\}\}^\{t\}\\\|\_\{\\mathcal\{D\}\}^\{2\}\.

###### Assumption 6\(Sampling\)\.

The mini\-batch sampling is unbiased and has a bounded variance, i\.e\.,‖∇𝛚ℛ^t​\(𝛚;α\)−∇𝛚ℛt​\(𝛚;α\)‖2≤σ2​\(α\)/B\|\|\\nabla\_\{\\boldsymbol\{\\omega\}\}\\hat\{\\mathcal\{R\}\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\-\\nabla\_\{\\boldsymbol\{\\omega\}\}\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\|\|^\{2\}\\leq\\sigma^\{2\}\(\\alpha\)/B, where∇𝛚ℛ^t​\(𝛚;α\)\\nabla\_\{\\boldsymbol\{\\omega\}\}\\hat\{\\mathcal\{R\}\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)is the gradient under sampled mini\-batch andBBis the mini\-batch size\.

Define driftδtref​\(α\)≜\[ℛt\+1​\(𝝎t\+1;α\)−ℛt​\(𝝎t\+1;α\)\]\+\\delta\_\{t\}^\{\\mathrm\{ref\}\}\(\\alpha\)\\triangleq\\left\[\\mathcal\{R\}\_\{t\+1\}\(\\boldsymbol\{\\omega\}^\{t\+1\};\\alpha\)\-\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\+1\};\\alpha\)\\right\]\_\{\+\}\. This drift exists as distribution𝒑k−1t\\boldsymbol\{p\}^\{t\}\_\{k\-1\}from chunkk−1k\-1changes, and characterizes the change ofℛt​\(⋅\)\\mathcal\{R\}\_\{t\}\(\\cdot\)across iterations\.

###### Lemma 4\(Bounded Drift\)\.

The cumulative drift ofℛt​\(⋅\)\\mathcal\{R\}\_\{t\}\(\\cdot\)is upper bounded, i\.e\.,𝔇T​\(α\)≜∑t=0T−1𝔼​\[δtref​\(α\)\]≤\(Cfo​\(α\)\+βR​\(α\)/2\)​ρ2​M​\(α\)​∑t=0T−1ηt2\\mathfrak\{D\}\_\{T\}\(\\alpha\)\\triangleq\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\[\\delta\_\{t\}^\{\\mathrm\{ref\}\}\(\\alpha\)\]\\leq\(C\_\{\\rm fo\}\(\\alpha\)\+\\beta\_\{R\}\(\\alpha\)/2\)\\rho^\{2\}M\(\\alpha\)\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{2\}\.

###### Proof\.

According to the definition ofδtref​\(α\)\\delta\_\{t\}^\{\\mathrm\{ref\}\}\(\\alpha\), by substituting𝝎′=𝝎t\+1\\boldsymbol\{\\omega\}^\{\\prime\}=\\boldsymbol\{\\omega\}^\{t\+1\}and𝝎=𝝎t\\boldsymbol\{\\omega\}=\\boldsymbol\{\\omega\}\_\{t\}in Assumption[4](https://arxiv.org/html/2608.07974#Thmassumption4),δtref​\(α\)≤\(Cfo​\(α\)\+βR​\(α\)2\)​‖𝒑~t\+1−𝒑~t‖𝒟2\\delta\_\{t\}^\{\\rm ref\}\(\\alpha\)\\leq\(C\_\{\\rm fo\}\(\\alpha\)\+\\frac\{\\beta\_\{R\}\(\\alpha\)\}\{2\}\)\\\|\\tilde\{\\boldsymbol\{p\}\}^\{t\+1\}\-\\tilde\{\\boldsymbol\{p\}\}^\{t\}\\\|\_\{\\mathcal\{D\}\}^\{2\}\. Based on Lipschitz continuity of𝒑~t≜𝒑~​\(𝝎t\)\\tilde\{\\boldsymbol\{p\}\}^\{t\}\\triangleq\\tilde\{\\boldsymbol\{p\}\}\(\\boldsymbol\{\\omega\}^\{t\}\)in Assumption[3](https://arxiv.org/html/2608.07974#Thmassumption3),‖𝒑~t\+1−𝒑~t‖𝒟2≤ρ2​‖𝝎t\+1−𝝎t‖2\\\|\\tilde\{\\boldsymbol\{p\}\}^\{t\+1\}\-\\tilde\{\\boldsymbol\{p\}\}^\{t\}\\\|\_\{\\mathcal\{D\}\}^\{2\}\\leq\\rho^\{2\}\\\|\\boldsymbol\{\\omega\}^\{t\+1\}\-\\boldsymbol\{\\omega\}^\{t\}\\\|^\{2\}\. LetCref≜\(Cfo​\(α\)\+βR​\(α\)/2\)​ρ2C\_\{\\rm ref\}\\triangleq\(C\_\{\\rm fo\}\(\\alpha\)\+\\beta\_\{R\}\(\\alpha\)/2\)\\rho^\{2\}\. Based on Lemma[3](https://arxiv.org/html/2608.07974#Thmlemma3)and Assumption[2](https://arxiv.org/html/2608.07974#Thmassumption2),𝔼​\[δtref\]≤Cref​\(α\)​𝔼​\[‖𝝎t\+1−𝝎t‖2\]=Cref​\(α\)​ηt2​𝔼​\[‖∇𝝎ℛt​\(𝝎;α\)‖2\]≤Cref​\(α\)​M​\(α\)​ηt2\\mathbb\{E\}\[\\delta\_\{t\}^\{\\rm ref\}\]\\leq C\_\{\\rm ref\}\(\\alpha\)\\mathbb\{E\}\[\\\|\\boldsymbol\{\\omega\}^\{t\+1\}\-\\boldsymbol\{\\omega\}^\{t\}\\\|^\{2\}\]=C\_\{\\rm ref\}\(\\alpha\)\\eta\_\{t\}^\{2\}\\mathbb\{E\}\[\\\|\\nabla\_\{\\boldsymbol\{\\omega\}\}\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\\\|^\{2\}\]\\leq C\_\{\\rm ref\}\(\\alpha\)M\(\\alpha\)\\eta\_\{t\}^\{2\}\. Summing overt=0,…,T−1t=0,\\ldots,T\-1completes the proof\. ∎

###### Theorem 1\(Convergence\)\.

Letηt=η0/t\+γ≤1/βR​\(α\)\\eta\_\{t\}=\\eta\_\{0\}/\\sqrt\{t\+\\gamma\}\\leq 1/\\beta\_\{R\}\(\\alpha\), whereγ\>0\\gamma\>0\. Based on Lemma[4](https://arxiv.org/html/2608.07974#Thmlemma4)and Assumptions[4](https://arxiv.org/html/2608.07974#Thmassumption4)–[6](https://arxiv.org/html/2608.07974#Thmassumption6),

1T​∑t=0T−1𝔼​‖∇𝝎ℛt​\(𝝎;α\)‖2≤2​\(ℛ0−ℛo​p​t\+𝔇T​\(α\)\)T​η0/T\+γ\+βR​\(α\)​σ​\(α\)2​∑t=0T−1ηt2/BT​η0/T\+γ=𝒪~​\(1T\),\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla\_\{\\boldsymbol\{\\omega\}\}\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\};\\alpha\)\\\|^\{2\}\\leq\\frac\{2\(\\mathcal\{R\}\_\{0\}\-\\mathcal\{R\}\_\{opt\}\+\\mathfrak\{D\}\_\{T\}\(\\alpha\)\)\}\{T\\eta\_\{0\}/\\sqrt\{T\+\\gamma\}\}\\\\ \+\\frac\{\\beta\_\{R\}\(\\alpha\)\\sigma\(\\alpha\)^\{2\}\\sum\_\{t=0\}^\{T\-1\}\\eta\_\{t\}^\{2\}/B\}\{T\\eta\_\{0\}/\\sqrt\{T\+\\gamma\}\}=\\tilde\{\\mathcal\{O\}\}\\left\(\\frac\{1\}\{\\sqrt\{T\}\}\\right\),\(23\)whereℛ0\\mathcal\{R\}\_\{0\}andℛo​p​t\\mathcal\{R\}\_\{opt\}are the initial and optimalℛ​\(⋅\)\\mathcal\{R\}\(\\cdot\)\.

###### Proof\.

For presentation simplicity, we omit notationα\\alphainℛt​\(⋅\)\\mathcal\{R\}\_\{t\}\(\\cdot\)\. By Assumption[4](https://arxiv.org/html/2608.07974#Thmassumption4)and considering update with sampled mini\-batch, i\.e\.,𝝎t\+1←𝝎t−ηt​∇ℛ^t​\(𝝎t\)\\boldsymbol\{\\omega\}^\{t\+1\}\\leftarrow\\boldsymbol\{\\omega\}^\{t\}\-\\eta^\{t\}\\nabla\\hat\{\\mathcal\{R\}\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\), we have

ℛt​\(𝝎t\+1\)≤ℛt​\(𝝎t\)−ηt​⟨∇ℛt​\(𝝎t\),∇ℛ^t​\(𝝎\)⟩\+βR​\(α\)​ηt22​‖∇ℛ^t​\(𝝎t\)‖2\.\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\+1\}\)\\leq\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\)\-\\eta^\{t\}\\langle\\nabla\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\),\\nabla\\hat\{\\mathcal\{R\}\}\_\{t\}\(\\boldsymbol\{\\omega\}\)\\rangle\\\\ \+\\frac\{\\beta\_\{R\}\(\\alpha\)\\eta\_\{t\}^\{2\}\}\{2\}\\\|\\nabla\\hat\{\\mathcal\{R\}\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\)\\\|^\{2\}\.\(24\)Taking conditional expectation and using Assumption[6](https://arxiv.org/html/2608.07974#Thmassumption6)andηt≤1/βR​\(α\)\\eta\_\{t\}\\leq 1/\\beta\_\{R\}\(\\alpha\), we have

𝔼​\[ℛt​\(𝝎t\+1\)\]≤𝔼​\[ℛt​\(𝝎t\)\]−ηt2​‖∇ℛt​\(𝝎t\)‖2\+βR​\(α\)​ηt2​σ​\(α\)22​B\.\\mathbb\{E\}\[\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\+1\}\)\]\\leq\\mathbb\{E\}\[\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\)\]\-\\frac\{\\eta\_\{t\}\}\{2\}\\\|\\nabla\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\)\\\|^\{2\}\+\\frac\{\\beta\_\{R\}\(\\alpha\)\\eta\_\{t\}^\{2\}\\sigma\(\\alpha\)^\{2\}\}\{2B\}\.\(25\)Based on the definition ofδtref​\(α\)\\delta\_\{t\}^\{\\mathrm\{ref\}\}\(\\alpha\),ℛt​\(𝝎t\+1\)≥ℛt\+1​\(𝝎t\+1\)−δtref​\(α\)\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\+1\}\)\\geq\\mathcal\{R\}\_\{t\+1\}\(\\boldsymbol\{\\omega\}^\{t\+1\}\)\-\\delta\_\{t\}^\{\\mathrm\{ref\}\}\(\\alpha\)\. By substitutingℛt​\(𝝎t\+1\)\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\+1\}\)into \([25](https://arxiv.org/html/2608.07974#S2.E25)\) and rearranging,

ηt2​𝔼​‖∇ℛt​\(𝝎t\)‖2≤𝔼​\[ℛt​\(𝝎t\)\]−𝔼​\[ℛt\+1​\(𝝎t\+1\)\]\+βR​\(α\)​ηt2​σ​\(α\)22​B\+𝔼​\[δtref​\(α\)\]\.\\frac\{\\eta\_\{t\}\}\{2\}\\mathbb\{E\}\\\|\\nabla\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\)\\\|^\{2\}\\leq\\mathbb\{E\}\[\\mathcal\{R\}\_\{t\}\(\\boldsymbol\{\\omega\}^\{t\}\)\]\-\\mathbb\{E\}\[\\mathcal\{R\}\_\{t\+1\}\(\\boldsymbol\{\\omega\}^\{t\+1\}\)\]\\\\ \+\\frac\{\\beta\_\{R\}\(\\alpha\)\\eta\_\{t\}^\{2\}\\sigma\(\\alpha\)^\{2\}\}\{2B\}\+\\mathbb\{E\}\[\\delta\_\{t\}^\{\\mathrm\{ref\}\}\(\\alpha\)\]\.\(26\)Summing fromt=0t=0toT−1T\-1and based on Lemma[4](https://arxiv.org/html/2608.07974#Thmlemma4)and the definition ofℛo​p​t\\mathcal\{R\}\_\{opt\}, the proof is complete\. ∎

Theorem[1](https://arxiv.org/html/2608.07974#Thmtheorem1)shows that ZeroLock algorithm has a convergence rate of𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(1/\\sqrt\{T\}\), which differs from the convergence rate𝒪​\(1/T\)\{\\mathcal\{O\}\}\(1/\\sqrt\{T\}\)of conventional BP by only polylogarithmic factor\. Meanwhile, the study in this section provides the first analytical framework for the local objective construction\-based approach under general model chunk division\.

## IIISystem Design

We propose a pipeline training system for ZeroLock\. It is applicable to both multi\-GPU server and multi\-device scenarios\. We first present the overview and detail the system design\. Then, we discuss the specific design for parallelism at mobile devices\.

![Refer to caption](https://arxiv.org/html/2608.07974v1/x5.png)Figure 2:System overview with one coordinator and multiple executors, each corresponding to a stage \(i\.e\., the update of a model chunk\)\.### III\-ASystem Overview

As shown in Fig\.[2](https://arxiv.org/html/2608.07974#S3.F2), this system contains one coordinator and multiple executors \(i\.e\., GPUs or devices\), and is separated into control plane and data plane\. The coordinator falls in the control plane and has three components:*membership*for tracking active executors;*scheduler*for managing the scheduling of micro\-batch updates across stages;*failure recovery*for tracking the state of committed requests and assigning replay requests when failure detected\. In the*scheduler*, update window defines a contiguous sequence of micro\-batches that executors are allowed to and must handle before advancing its process and serving as the unit of checkpoint; micro\-batch scheduler indicates the sequence of micro\-batches in the update window and will assist with the micro\-batch replay after failure detected\.

Each executor contains both control plane and data plane\. The control plane contains three components:*membership*for reporting its registration and hearbeat;*execution monitor*for monitoring its execution status;*failure recovery*for execution replay\. The data plane at an executor is responsible for handing a stage, i\.e\., the update of a model chunk\. It retrieves either the model input or the hidden state of the upstream stage from its buffer, performs forward pass to determine its hidden state, transfers the hidden state to the buffer of the downstream stage, and performs a backward update based on its local objective\.

We propose the following techniques to achieve lightweight, high\-throughput, and robust implementation\.\(I\) Early Forwarding\.The forwarding of the hidden state is designed to perform ahead of the local updates, avoiding unnecessary inter\-stage waiting\.\(II\) Independent Execution and Checkpoint\.Each executor maintains only its own trainable parameters, optimizer state, and checkpoint, enabling recovery at the stage granularity, and independently constructs its local signal and perform local updates\.\(III\) State\-Only Inter\-Stage Exchange\.Executors exchange only the hidden state \(but not the gradients or optimizer states\) from other stages\.\(IV\) Buffer\-Assisted State Exchange\.Each executor maintains a bounded hidden state buffer, storing the hidden states of its upstream stage for iterations, enabling replay after failure\. Overall, \(I\)\-\(III\) contribute to lightweight and high\-throughput; \(II\) and \(IV\) contribute to failure recovery capability\.

### III\-BSystem Design Details

We first present the execution interfaces\. Then, we introduce inter\-stage runtime, in\-stage execution, and failure recovery\.

#### III\-B1Stage\-Execution\-Related Interface

The following defines the unified interfaces related to stage execution at the executors\.

Input Interface\.For stage 0 executor, it inputs either token IDs or pre\-computed initial hidden representation \(constructed via a local input embedding layer\)\. For each of the subsequent stages, it inputs the hidden states of the upstream stage from the hidden state buffer, filled by its upstream executor\. Meanwhile, all stages input the attention mask, position IDs, and labels associated with a specific micro\-batch to facilitate local training\.

Forward\-and\-Output Interface\.The executor performs forward pass using its chunk to produce its hidden state, which is then detached from the computation graph and written to the outbound buffer for sending to the downstream stage\.

Local Update Interface\.The executor utilizes the input hidden states and labels to construct a local loss and performs a backward pass and optimizer step to update its trainable parameters\. This interface supports two ways to construct loss:

\(a\) Static Model Head\. It reuses the pre\-trained model’s final normalization layer and frozen model head as the readout head, and the token\-level cross entropy and task\-specific labels are used to construct local loss, as in Section[II\-A](https://arxiv.org/html/2608.07974#S2.SS1)\. This design does not induce new trainable parameters, while it assumes the intermediate hidden states can be interpretable by the final head, which may not hold for complex generative tasks\.

\(b\) Readout Adapter\. To bridge the gap between the intermediate hidden states and the frozen model head’s readout space, executor can insert a stage\-wise readout adapter \(i\.e\., a small residual MLP initialized approximate the identity mapping\)\. During training, this adapter aligns the intermediate hidden space to be more amenable to the frozen model head\.

Besides \(a\) and \(b\), this interface also accommodates other lightweight head, e\.g\., bottleneck projection head or restricted vocabulary head, to reduce computation and memory overhead\.

#### III\-B2Inter\-Stage Runtime and In\-Stage Execution

The inter\-stage runtime \(acting as a scheduling engine\) organizes multiple micro\-batches into an update window at the coordinator\. These micro\-batches enter the stage pipeline sequentially\. For each micro\-batch, the stage worker performs*in\-stage execution*:

- S1Input: Retrieve hidden states of the upstream stage from the hidden state buffer; load the attention mask, position IDs, and labels related to the micro\-batch\.
- S2Forward\-and\-Output: Perform forward pass to produce its hidden state; output it to the outbound buffer\.
- S3Local Training: Backpropagate the local loss and accumulate gradients for the current update window\. The optimizer step is applied after the final micro\-batch in that window\.

Note that downstream execution can start immediately after S2 without waiting for the upstream’s local backward in S3, leading to a higher degree of parallelism\.

Across stages,*inter\-stage runtime*is responsible for managing the hidden state flow\. The management is in the form of an*entry*, which contains the hidden state tensor of a micro\-batch and the related metadata for sending, receiving, and status\. The runtime tracks each entry’s transition from the outbound buffer of upstream stage to the hidden state buffer of downstream stage, allowing the coordinator to accurately control the buffer occupancy of each stage without tracking each tensor\. Meanwhile, we introduce an in\-flight depth to restrict the number of entries between each pair of stages, preventing upstream stages from overwhelming downstream stages\. Furthermore, on GPU, preposted receive is allowed, such that downstream stage can submitreceive​\_​entry\\operatorname\{receive\\\_entry\}request in advance\. Each entry binds to a readiness event\. Compute stream waits only for this specific event, avoiding unrelated communication operations from blocking the compute stream\.

#### III\-B3Failure Recovery

The failure recovery capability comes from two facts: each stage checkpoints its state independently; the hidden state buffer introduces resilience to failure\. Consequently, after a failure, only the failed stages need to roll back, which retrieves hidden state from the buffer and replaying the requests\. The brief failure recovery mechanism is as follows:

- •Proactive Checkpointing: Each executor periodically captures local checkpoints, including trainable parameters, optimizer states, and progress metadata\.
- •Reactive Recovery: Upon detecting a failure, the runtime restores the failed stage from its latest checkpoint and replays necessary update windows to catch up to the pipeline frontier\. Successful stages retain their execution\.

### III\-CMobile Execution Backend

For deployment at mobile devices, we use the ExecuTorch framework, Meta’s official PyTorch\-native edge runtime\. It performs ahead\-of\-time \(AOT\) compilation to produce a static computational representation, serialized as a\.pteprogram, which enables deterministic execution on resource\-constrained end devices without a Python interpreter\. However, ExecuTorch bundles forward and backward computations into a single Python ExecuTorch \(PTE\) method, where the forward hidden state is exposed only after the local backward gradient computation\. This prohibit the early\-forwarding opportunity\.

To address this, we insert a lightweight pipeline marker operator within the PTE method—specifically after the detached hidden state output but before the parameter\-gradient subgraph\. This operator performs no tensor computation; instead, it signals the runtime to capture the hidden outputs and suspend the PTE method while preserving its execution state \(stack, tensors, optimizer buffers\)\. We extend the ExecuTorch training runtime with a two\-phase Android interface\. The first phase executes up to the marker and returns the hidden state, which is immediately transferred to the downstream device\. The second phase resumes the same method to complete the local backward pass, after which an on\-device AdamW optimizer updates the stage’s LoRA parameters\. In this case, the transfer to the downstream device overlaps with the local backward computation, making the resulting stage program support local training and early\-forwarding without incurring cross\-stage gradient Remote Procedure Call \(RPC\) overhead\.

## IVExperiments

We build prototype systems using both multi\-GPU/CPU server and Android devices respectively\. The evaluation on multi\-GPU/CPU server shows \(E1\) the method comparison on memory and throughput and \(E2\) failure recovery\. The evaluation on Android devices shows \(E3\) on\-device performance\.

### IV\-AExperimental Settings

Model and data\.We use TinyLlama as the pre\-trained model\. Its Transformer layers are partitioned into three consecutive chunks, each assigned to one NVIDIA L40 GPU or one Andriod phone\. We apply LoRA with rank44and scaling factor1616\. On default, experiments are conducted on a fixed 10,000\-example subset of AG News; all sequences are padded or truncated to 128 tokens\. Letbbdenote the number of physical batches processed by one microbatch call;mmdenotes the number of such microbatch calls before an optimizer update; thus, batchB=b​mB=bm\. We repeat the experiment with three random seeds\.

Methods\.We compare ZeroLock with three approaches\. \(i\) GPipe\[[13](https://arxiv.org/html/2608.07974#bib.bib10)\]completes all forward passes in a logical batch before the backward pass\. \(ii\) 1F1B\[[20](https://arxiv.org/html/2608.07974#bib.bib11)\]interleaves forward and backward within a logical batch and flushes the pipeline before the shared optimizer update\. \(iii\) PipeDream\[[20](https://arxiv.org/html/2608.07974#bib.bib11)\]maintains 1F1B execution across update windows via weight stashing \(i\.e\., storing multiple copies\), ensuring forward and backward passes use consistent parameter versions\.555Although existing works\[[9](https://arxiv.org/html/2608.07974#bib.bib39),[3](https://arxiv.org/html/2608.07974#bib.bib40),[11](https://arxiv.org/html/2608.07974#bib.bib42),[26](https://arxiv.org/html/2608.07974#bib.bib48)\]considered modular decoupling in pipeline parallelism,\[[11](https://arxiv.org/html/2608.07974#bib.bib42)\]is not for LLM fine\-tuning;\[[9](https://arxiv.org/html/2608.07974#bib.bib39),[3](https://arxiv.org/html/2608.07974#bib.bib40)\]do not provide open\-sourced code; the open\-sourced code of\[[26](https://arxiv.org/html/2608.07974#bib.bib48)\]does not implement system\-level pipeline parallelism \(e\.g\., it places all chunks of a certain batch in one GPU\), so comparing it with those deployed in multiple GPUs may not be fair\.

### IV\-BE1: Memory and Throughput

Despite the modular update decoupling, ZeroLock has a comparable accuracy and negative log\-likelyhood \(NLL\) to baseline approaches \(see Fig\.[3](https://arxiv.org/html/2608.07974#S4.F3)\)\. Build upon this, we show its memory reduction and throughput improvement as follows\.

![Refer to caption](https://arxiv.org/html/2608.07974v1/x6.png)

![Refer to caption](https://arxiv.org/html/2608.07974v1/x7.png)

Figure 3:\(a\) Accuracy and \(b\) negative\-likelihood loss\.![Refer to caption](https://arxiv.org/html/2608.07974v1/x8.png)Figure 4:Peak GPU memory usage and the memory usage of each component under the peak \(b=8b=8andm=4m=4\)\.#### IV\-B1Memory

As shown in Fig\.[4](https://arxiv.org/html/2608.07974#S4.F4), when compared with GPipe, 1F1B, and PipeDream, ZeroLock reduces the mean per\-stage peak memory by 47\.8%, 14\.7%, and 14\.6%, respectively, and by 55\.3%, 26\.6%, and 26\.5% for the maximum stage usage, respectively\. Most of the reduction comes from the elimination of activations, where the reduction are 75\.4%, 40\.8%, and 48\.8% on per\-stage average, respectively\. This reduction results from the modular update decoupling of ZeroLock; in comparison, the other approaches implement end\-to\-end backpropagation, which need to retain activations for backward gradient computation\. Meanwhile, givenm=4m=4, GPipe, 1F1B, PipeDream, and ZeroLock experience out\-of\-memory whenb=10,20,20,28b=10,~20,~20,~28, respectively, showing that the system with ZeroLock can afford a larger batch size\.

Figs\.[5](https://arxiv.org/html/2608.07974#S4.F5)\(a\) and \(c\) show that ZeroLock is more beneficial when the physical batch to microbatch ratio \(i\.e\.,b/mb/m\) is larger\. Specifically, more physical batches \(in a microbatch call\) indicates larger forward graphs retained by baseline approaches, leading to more obvious benefits for modular decoupling\.

![Refer to caption](https://arxiv.org/html/2608.07974v1/x9.png)

![Refer to caption](https://arxiv.org/html/2608.07974v1/x10.png)

\(a\) \(b\) ![Refer to caption](https://arxiv.org/html/2608.07974v1/x11.png)![Refer to caption](https://arxiv.org/html/2608.07974v1/x12.png)

\(c\) \(d\)

Figure 5:\(a\) and \(c\), maximum stage memory usage underB=32B=32andB=128B=128, respectively; \(b\) and \(d\), throughput underB=32B=32andB=128B=128, respectively\. Whenb/m=8/4b/m=8/4, when compared with PipeDream, ZeroLock reduces the memory by 26\.5% and improves throughput by 4\.9%\.![Refer to caption](https://arxiv.org/html/2608.07974v1/x13.png)

![Refer to caption](https://arxiv.org/html/2608.07974v1/x14.png)

\(a\) \(b\)

Figure 6:Throughput under diverse scenarios: \(a\) number of stages, each executed by a GPU; \(b\) diverse sender\-side links\.![Refer to caption](https://arxiv.org/html/2608.07974v1/x15.png)Figure 7:CUDA activity: \(a\) GPipe; \(b\) 1F1B; \(c\) PipeDream; \(d\) ZeroLock\.
#### IV\-B2Throughput

Figs\.[5](https://arxiv.org/html/2608.07974#S4.F5)\(b\) and \(d\) and Fig\.[6](https://arxiv.org/html/2608.07974#S4.F6)show the throughput comparison\. The default setting contains three stages, withb=8b=8andm=4m=4\. First, in Fig\.[5](https://arxiv.org/html/2608.07974#S4.F5)\(b\), under the default setting, when compared with GPipe, 1F1B, and PipeDream, ZeroLock improves the throughput by 55\.8%, 62\.8%, and 4\.9%, respectively\. Second, in Figs\.[5](https://arxiv.org/html/2608.07974#S4.F5)\(b\) and \(d\) and Fig\.[6](https://arxiv.org/html/2608.07974#S4.F6)\(a\), as the increase of the physical batches in a microbatch call and stages, ZeroLock and PipeDream demonstrate more significant increase in throughput, showing the better potentials for reducing bubbles in each call and exploiting additional GPUs, respectively\. Finally, Fig\.[6](https://arxiv.org/html/2608.07974#S4.F6)\(b\) shows the throughput under simulated sender\-side communication link: Wi\-Fi, 2 ms/1000 Mbps \(latency/bandwidth\); mobile, 10 ms/200 Mbps; constrained, 30 ms/50 Mbps\. ZeroLock is more beneficial when the link is slower\. This is because baselines need to send both forward hidden state and backward hidden gradients, while ZeroLock transfers only forward hidden state\.

Fig\.[7](https://arxiv.org/html/2608.07974#S4.F7)visualizes the CUDA activity traces underb=8b=8,m=4m=4, 512 data samples, and 16 optimizer updates\. The plots show a one\-second steady\-state interval, during which all three GPUs exhibit compute activity; this interval selection was fixed prior to inspecting method\-specific behavior to avoid bias\. The Kernel and device copy timestamps are measured externally, without injecting device\-side synchronization\. As shown in the figure, ZeroLock and PipeDream demonstrate more concurrent stage activities and less GPU idle time\. Furthermore, when compared with PipeDream, ZeroLock shows a more balanced activity load across stages and a higher active fraction, with an improvement of 13\.7%\. Note that this active fraction serves as a diagnostic indicator of pipeline scheduling rather than a proxy for throughput, as idle periods also encompass host dispatch and Gloo communication overhead\.

TABLE I:Failure recovery\.MethodRecovery \(ms\)Transfer \(MiB\)Synchronous 1F1B2381\.2±66\.62381\.2\\pm 66\.6192ZeroLock2013\.1±6\.42013\.1\\pm 6\.496TABLE II:On\-device evaluation with Andriod phones\.Stage/devicePTEQueueTotalPSSTemp\.S0/NX809J11\.43/13\.5612\.44/17\.5638\.28/46\.903423\.537\.0S1/Lenovo L710917\.56/8\.320/0\.0215\.61/27\.513408\.236\.0S2/Pixel 10 Pro XL9\.53/11\.400/5\.7910\.30/18\.993824\.336\.7![Refer to caption](https://arxiv.org/html/2608.07974v1/x16.png)

![Refer to caption](https://arxiv.org/html/2608.07974v1/x17.png)

\(a\) \(b\)

Figure 8:\(a\) Latency breakdown; \(b\) Local loss convergence\.

### IV\-CE2: Failure Recovery

We evaluate the outage recovery capability of ZeroLock\. It is compared with 1F1B\. Specifically, 1F1B recovers via global rollback and full replay\. In contrast, ZeroLock enables local recovery by reusing buffered hidden states and retaining committed previous updates, eliminating redundant recomputation\.

We consider three stages, and setb=1b=1andm=8m=8\. The failure setting is as follows: W0–W3, four prelude windows; W4–W7, four failure windows at Stage 1; W8–W11, four resumed windows\. The worker processes and GPU contexts remain alive during the failure\. Recovery latency is defined as the during between when Stage 1 resumes and when Stage 2 commits W7\. In Table[I](https://arxiv.org/html/2608.07974#S4.T1), ZeroLock reduces the recovery latency by 368 ms and the transfer traffic from 192 to 96 MiB\.

### IV\-DE3: On\-Device Evaluation with Andriod Phones

For implementation on Android devices, we export three TinyLlama training PTEs containing transformer layers \[0,6\], \[7,13\], and \[14,21\] and deploy them on NX809J, Lenovo L71091, and Pixel 10 Pro XL, respectively\. We use 128 data samples for training with sequence length 128,b=1b=1, LoRA rank 8, scale 16, and learning rate10−410^\{\-4\}\. Each stage commits 128 optimizer steps and writes nine checkpoints\.

Table[II](https://arxiv.org/html/2608.07974#S4.T2)shows the median/95th percentile PTE, queuing, and total duration \(in seconds\) for a single batch request with the latency break down shown in Fig\.[8](https://arxiv.org/html/2608.07974#S4.F8)\(a\), the application peak PSS in MiB, and battery temperature inC∘\{\}^\{\\circ\}C\. The total wall\-clock time of the entire fine\-tuning is 1644\.1 s with a throughput of 0\.0779 records/s\. Fig\.[8](https://arxiv.org/html/2608.07974#S4.F8)\(b\) shows the loss convergence\. These results indicate the implementation is practically feasible\.

## VConclusion

In this work, we propose a ZeroLock algorithm, which achieves modular update decoupling by local objective construction, for pipeline parallelism in LLM fine\-tuning\. It effectively removes pipeline bubbles by alleviating the inter\-stage waiting and reduces memory usage by mitigating unnecessary activation storage at the algorithm level\. We provide the first analytical framework for local objective construction\-based approaches under general chunk division and prove that ZeroLock has a convergence rate of𝒪~​\(1/T\)\\tilde\{\\mathcal\{O\}\}\(1/\\sqrt\{T\}\), comparable to BP training\. Meanwhile, we design a system for deploying ZeroLock, incorporating techniques such as early forwarding and failure recovery to improve system throughout and robustness\. We build real\-world prototype and show that when compared with BP\-based baselines, ZeroLock reduces the memory by 26\.5% and improves throughput by 4\.9%\. For future direction, it is meaningful to further incorporate operator\-level optimization to further improve throughput and reduce memory usage\.

## VIUse of AI Disclosure

ChatGPT was used to assist with code development and experimental setup\. Specifically, it was used to configure and troubleshoot the environments required for benchmark reproduction, run and migrate benchmark implementations across different environments, generate Kotlin code for deploying and executing PTE models on mobile devices, and generate batch experiment scripts using torchrun\. The AI\-generated code and configuration instructions were reviewed, adapted where necessary, tested, and validated by the author\. The author takes full responsibility for the final implementation, experimental results, and their interpretation\.

## References

- \[1\]\(2019\)Deep learning without weight transport\.InProc\. NeurIPS,Vol\.32\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p5.1)\.
- \[2\]A\. M\. Alghamdi, M\. U\. Ashraf, A\. A\. Bahaddad, K\. A\. Almarhabi, W\. A\. Al Shehri, and A\. Daraz\(2025\)Cross\-subject eeg signals\-based emotion recognition using contrastive learning\.Scientific Reports15\(1\),pp\. 28295\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p1.1)\.
- \[3\]M\. Aljahdali and M\. Canini\(2025\-07\)Idle no more: boosting distributed pipeline training via FluidPipe\.InProc\. IEEE ICDCSW,Glasgow, U\.K\.\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p8.1),[footnote 5](https://arxiv.org/html/2608.07974#footnote5)\.
- \[4\]A\. Borzunov, D\. Baranchuk, T\. Dettmers, M\. Ryabinin, Y\. Belkada, A\. Chumachenko, P\. Samygin, and C\. Raffel\(2023\-07\)Petals: collaborative inference and fine\-tuning of large models\.InProc\. ACL Syst\. Demonstrations,Toronto, ON, Canada\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[5\]J\. Chen, X\. Deng, Z\. Xiong, S\. Guo, X\. Qiu, P\. Wang, and D\. Niyato\(2025\-09\)CollaPipe: adaptive segment\-optimized pipeline parallelism for collaborative LLM training in heterogeneous edge networks\.arXiv:2509\.19855\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[6\]Y\. Chen, Y\. Yan, S\. Ge, Y\. Qin, Y\. Zheng, Q\. Yang, S\. He, Z\. Shi, J\. Chen, and Y\. Shu\(2025\)Confidant: customizing transformer\-based llms via collaborative training on mobile devices\.InProc\. ACM MobiCom,Cited by:[Figure 1](https://arxiv.org/html/2608.07974#S1.F1),[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[7\]S\. Fan, Y\. Rong, C\. Meng, Z\. Cao, S\. Wang, Z\. Zheng, C\. Wu, G\. Long, J\. Yang, L\. Xia, L\. Diao, X\. Liu, and W\. Lin\(2021\-Feb\.–Mar\.\)DAPPLE: a pipelined data parallel approach for training large models\.InProc\. ACM PPoPP,Virtual Conf\.\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[8\]S\. Gandhi and C\. Kozyrakis\(2026\-05\)Sparse checkpointing for fast and reliable MoE training\.InProc\. USENIX NSDI,Renton, WA, USA\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[9\]X\. Guo, C\. Xu, G\. Guo, F\. Zhu, C\. Cai, P\. Wang, X\. Wei, J\. Su, and J\. Gao\(2024\-11\)Faster multi\-GPU training with PPLL: a pipeline parallelism framework leveraging local learning\.arXiv:2411\.12780\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p8.1),[footnote 5](https://arxiv.org/html/2608.07974#footnote5)\.
- \[10\]P\. Han, C\. Huang, G\. Tian, M\. Tang, and X\. Liu\(2024\)Convergence analysis of split federated learning on heterogeneous data\.Advances in Neural Information Processing Systems37,pp\. 103476–103544\.Cited by:[§II\-B](https://arxiv.org/html/2608.07974#S2.SS2.p3.1),[§II\-C](https://arxiv.org/html/2608.07974#S2.SS3.p7.14)\.
- \[11\]M\. Ho, C\. Wang, Y\. Lin, and H\. Chen\(2026\-06\)SCPL: enhancing neural network training throughput with decoupled local losses and model parallelism\.ACM Trans\. Manage\. Inf\. Syst\.17\(2\)\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p8.1),[footnote 5](https://arxiv.org/html/2608.07974#footnote5)\.
- \[12\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\-04\)LoRA: low\-rank adaptation of large language models\.InProc\. ICLR,Virtual Conf\.\.Cited by:[1st item](https://arxiv.org/html/2608.07974#S1.I3.i1.p1.1),[§II\-A](https://arxiv.org/html/2608.07974#S2.SS1.p1.9)\.
- \[13\]Y\. Huang, Y\. Cheng, A\. Bapna, O\. Firat, M\. X\. Chen, D\. Chen, H\. Lee, J\. Ngiam, Q\. V\. Le, Y\. Wu, and Z\. Chen\(2019\-12\)GPipe: efficient training of giant neural networks using pipeline parallelism\.InProc\. NeurIPS,Vancouver, BC, Canada\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1),[§IV\-A](https://arxiv.org/html/2608.07974#S4.SS1.p2.1)\.
- \[14\]Z\. Huo, B\. Gu, qian Yang, and H\. Huang\(2018\-Jul\.\)Decoupled parallel backpropagation with convergence guarantee\.InProc\. ICML,Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p3.1.1)\.
- \[15\]Q\. Li, Y\. W\. Teh, and R\. Pascanu\(2026\)Noprop: training neural networks without back\-propagation or forward\-propagation\.InConference on Lifelong Learning Agents,pp\. 525–544\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p5.1)\.
- \[16\]S\. Li and T\. Hoefler\(2021\-11\)Chimera: efficiently training large\-scale neural networks with bidirectional pipelines\.InProc\. ACM/IEEE SC,St\. Louis, MO, USA\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[17\]X\. Lian, S\. A\. Jacobs, L\. Kurilenko, M\. Tanaka, S\. Bekman, O\. Ruwase, and M\. Zhang\(2025\-07\)Universal checkpointing: a flexible and efficient distributed checkpointing system for large\-scale DNN training with reconfigurable parallelism\.InProc\. USENIX ATC,Boston, MA, USA\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[18\]S\. Malladi, T\. Gao, E\. Nichani, A\. Damian, J\. D\. Lee, D\. Chen, and S\. Arora\(2023\)Fine\-tuning language models with just forward passes\.InProc\. NeurIPS,External Links:[Document](https://dx.doi.org/10.52202/075280-2308)Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p5.1)\.
- \[19\]B\. Millidge, A\. Tschantz, and C\. L\. Buckley\(2022\)Predictive coding approximates backprop along arbitrary computation graphs\.Neural Computation34\(6\),pp\. 1329–1368\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p5.1)\.
- \[20\]D\. Narayanan, A\. Harlap, A\. Phanishayee, V\. Seshadri, N\. R\. Devanur, G\. R\. Ganger, P\. B\. Gibbons, and M\. Zaharia\(2019\-10\)PipeDream: generalized pipeline parallelism for DNN training\.InProc\. ACM SOSP,Huntsville, ON, Canada\.Cited by:[1st item](https://arxiv.org/html/2608.07974#S1.I1.i1.p1.1),[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1),[§IV\-A](https://arxiv.org/html/2608.07974#S4.SS1.p2.1)\.
- \[21\]D\. Narayanan, A\. Phanishayee, K\. Shi, X\. Chen, and M\. Zaharia\(2021\-07\)Memory\-efficient pipeline\-parallel DNN training\.InProc\. ICML,Virtual Conf\.\.Cited by:[1st item](https://arxiv.org/html/2608.07974#S1.I1.i1.p1.1),[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[22\]D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, V\. A\. Korthikanti, D\. Vainbrand, P\. Kashinkunti, J\. Bernauer, B\. Catanzaro, A\. Phanishayee, and M\. Zaharia\(2021\-11\)Efficient large\-scale language model training on GPU clusters using Megatron\-LM\.InProc\. ACM/IEEE SC,St\. Louis, MO, USA\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[23\]A\. Nøkland\(2016\)Direct feedback alignment provides learning in deep neural networks\.Proc\. NeurIPS\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p5.1)\.
- \[24\]P\. Qi, X\. Wan, G\. Huang, and M\. Lin\(2024\-05\)Zero bubble \(almost\) pipeline parallelism\.InProc\. ICLR,Vienna, Austria\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[25\]M\. Ryabinin, T\. Dettmers, M\. Diskin, and A\. Borzunov\(2023\-07\)SWARM parallelism: training large models can be surprisingly communication\-efficient\.InProc\. ICML,Honolulu, HI, USA\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[26\]H\. Shi, T\. Han, P\. Wang, Z\. Wang, X\. Yang, and J\. Su\(2026\-05\)Rethinking local learning: a cheaper and faster recipe for LLM post\-training\.arXiv:2605\.04913\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p8.1),[§II\-B](https://arxiv.org/html/2608.07974#S2.SS2.p3.1),[§II\-C](https://arxiv.org/html/2608.07974#S2.SS3.p7.14),[footnote 5](https://arxiv.org/html/2608.07974#footnote5)\.
- \[27\]A\. Tandon, K\. Dalal, X\. Li, D\. Koceja, M\. Rød, S\. Buchanan, X\. Wang, J\. Leskovec, S\. Koyejo, T\. Hashimoto,et al\.\(2025\)End\-to\-end test\-time training for long context\.arXiv preprint arXiv:2512\.23675\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p1.1)\.
- \[28\]S\. Wang, Z\. Chen, and M\. Tang\(2026\)CurvZO: adaptive curvature\-guided sparse zeroth\-order optimization for efficient llm fine\-tuning\.InProc\. ICML,Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p5.1)\.
- \[29\]T\. Wu, L\. Cao, H\. Lu, X\. Jiang, Y\. Yu, S\. Yang, G\. Yang, J\. Wang, L\. Qu, L\. Zhang, and W\. Wang\(2026\-05\)Attack of the bubbles: straggler\-resilient pipeline parallelism for large model training\.InProc\. USENIX NSDI,Renton, WA, USA\.Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p2.1),[§I](https://arxiv.org/html/2608.07974#S1.p3.1)\.
- \[30\]C\. Ye, R\. Ye, Y\. Zhang, and M\. Tang\(2026\)Depth\-progressive monotonic learning without global backpropagation\.InProc\. ICML,Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p3.1.1),[§I](https://arxiv.org/html/2608.07974#S1.p5.1),[footnote 1](https://arxiv.org/html/2608.07974#footnote1),[footnote 2](https://arxiv.org/html/2608.07974#footnote2),[footnote 4](https://arxiv.org/html/2608.07974#footnote4)\.
- \[31\]R\. Ye, C\. Ye, C\. Huang, M\. Tang, and Y\. Liu\(2026\)Beyond\-backpropagation training: methods, applications, and perspectives\.TechRxiv2026\(0103\),pp\.\.External Links:[Document](https://dx.doi.org/10.36227/techrxiv.176740426.63642005/v1),[Link](https://www.techrxiv.org/doi/abs/10.36227/techrxiv.176740426.63642005/v1),https://www\.techrxiv\.org/doi/pdf/10\.36227/techrxiv\.176740426\.63642005/v1Cited by:[§I](https://arxiv.org/html/2608.07974#S1.p3.1.1)\.