ToPA：用于表达力与高效微调的分块Toeplitz适应方法

# ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning
Source: [https://arxiv.org/html/2610.22883](https://arxiv.org/html/2610.22883)
Sicong Li Qianqian Xu Zhiyong Yang Zitai Wang Longtao Huang Xiaochun Cao Qingming Huang Institute of Information Engineering, CASSchool of Cyber Security, University of Chinese Academy of SciencesState Key Laboratory of AI Safety, Institute of Computing Technology, CASBeijing Academy of Artificial IntelligenceSchool of Computer Science and Tech\., University of Chinese Academy of SciencesAlibaba GroupSchool of Cyber Science and Tech\., Shenzhen Campus of Sun Yat\-sen Universitylisicong@iie\.ac\.cn, \{xuqianqian,wangzitai\}@ict\.ac\.cn, \{yangzhiyong21,qmhuang\}@ucas\.ac\.cn, kaiyang\.hlt@alibaba\-inc\.com, caoxiaochun@mail\.sysu\.edu\.cn††thanks:Corresponding authors\.

###### Abstract

Parameter\-Efficient Fine\-Tuning \(PEFT\) has emerged as a prevalent strategy for adapting large pre\-trained models\. Among PEFT techniques, reparameterization\-based methods, particularly those that employ low\-rank or sparsity structures, have attracted significant attention\. However, these approaches often underperform full fine\-tuning\. Specifically, low\-rank methods constrain the update rank, which is misaligned with the inherently high\-rank nature of full fine\-tuning\. In contrast, sparsity\-based techniques limit the flexibility of the parameter space and compromise connectivity across weights\. To overcome these limitations, we explore the use of Toeplitz matrices, whose entries are constant along each diagonal, therefore providing a compact parameterization without imposing explicit rank or sparsity constraints\. A straightforward approach is to update weights via a product of Toeplitz matrices\. However, similar to RNNs, long multiplicative chains often lead to gradient instability\. To address this issue, we proposeToPA, a more refined version of this idea that replaces standard Toeplitz matrices with block\-wise Toeplitz, greatly increasing each factor’s capacity and shortening the chain, thereby stabilizing training and improving expressiveness\. We theoretically show that products of block\-wise Toeplitz matrices can approximate arbitrary matrices under shorter chain length, justifying the structural design of ToPA\. Furthermore, we demonstrate that ToPA offers greater expressivity than existing low\-rank and sparse parameterizations\. Empirical evaluations across 21 NLP and CV datasets, spanning 5 model architectures, consistently validate the effectiveness of ToPA\.

### 1Introduction

In recent years, large language models \(LLMs\) have made significant advances in a wide range of fields[Brown et al\. \(2020\)](https://arxiv.org/html/2610.22883#bib.bib17);[Thoppilan et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib18);[Chen et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib19);[Li et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib20);[Wei et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib21)\. Despite their strong generalization capabilities, applying LLMs to downstream tasks often requires additional fine\-tuning\. Fully fine\-tuning all parameters, however, is computationally expensive and memory\-intensive\. To mitigate this challenge, parameter\-efficient fine\-tuning \(PEFT\)[Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.22883#bib.bib12);[Lester et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib13);[Zaken et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib15);[Zhou et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib16)has emerged as a practical paradigm that adapts pretrained models by updating only a small subset of parameters\.

Among PEFT approaches, one of the most representative methods is based on reparameterization, which represents the update matrix into an efficient structure based on a specific inductive bias\. According to the selected structure, existing techniques typically follow two directions\. One line islow\-rank decomposition, exemplified by LoRA and its variants[Hu et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib1);[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.22883#bib.bib11);[Meng et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib10), which adds the product of two low\-rank matrices\. The other line adoptssparse structure, such as OFT[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib2)and BOFT[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22883#bib.bib5), which multiplies one or multiple sparse orthogonal matrices to modulate pretrained weights\.

![Refer to caption](https://arxiv.org/html/2610.22883v1/framework-v11.png)Figure 1:Low\-rank methods \(e\.g\., LoRA\) adapt pretrained weights via additive low\-rank updates, while sparsity\-structured methods \(e\.g\., OFT/BOFT\) apply multiplicative sparse adapters\. In contrast, ToPA adapts the model based on the block\-wise Toeplitz matrices with identity residual design, enabling dense and high\-rank adaptation while maintaining parameter efficiency\.Despite their empirical success, both paradigms stillfall short of full fine\-tuning\. Low\-rank methods are motivated by the assumption that over\-parameterized models admit low intrinsic dimension[Li et al\. \(2018\)](https://arxiv.org/html/2610.22883#bib.bib40);[Aghajanyan et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib38)\. However, some recent work suggests thatlow intrinsic dimension does not imply low rank[Huang et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib3)and in fact full fine\-tuning typically yields a high\-rank update[Jiang et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib8);[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib22)\. Hence, the low\-rank constraint does not lead to the desired inductive bias and thus could become a structural bottleneck, limiting expressiveness and potentially leading to underfitting\. Sparsity structure methods impose afixed, predetermined sparsity patterncoupled with orthogonality constraints on the adapter\. Under this design, each column of the adapted weight is restricted to a predefined low\-dimensional subspace spanned byonly a small set oforiginal columns\. Moreover, sparsity masks most entries, disrupting connectivity among parameters and preventing interactions across dimensions, thereby blocking information propagation\. Seeing the weakness of these two mainstream paradigms, we come to the following question:

Can we design a new compact parameterization to unleash the expressiveness of the PEFT method to catch up to full fine\-tuning?

To this end, we turn to theToeplitz matrix, a structured matrix whose entries are identical along each descending diagonal\. As shown in Fig\.[1](https://arxiv.org/html/2610.22883#S1.F1), it inherently exhibits a shifting pattern with only2​n−12n\-1free parameters to form ann×nn\\times nmatrix\. Intuitively, the shifting structure ensures that each row introduces an independent parameter, which almost surely avoids linear dependence and achieves full\-rank\. The absence of a fixed zero pattern ensures that efficiency herein is not tied to sparsity\. Overall, Toeplitz matrices arefree from rank or sparsity constraintsyet maintain compactness\. Moreover, matrix–vector multiplication with a Toeplitz matrix can be implemented efficiently in𝒪⁡\(n​log⁡n\)\\mathcal\{O\}\(n\\log n\)time via Fast Fourier Transform \(FFT\)\. A classical result further shows that any matrix can be represented as a product of Toeplitz matrices\. These properties make the Toeplitz matrix a promising candidate for expressive and efficient fine\-tuning\.

Building on these advantages, a naive implementation is to reparameterize pretrained weights by multiplying them with a chain of learnable Toeplitz matrices, whose parameter capacity can only be increased by stacking more Toeplitz factors\. However, our theoretical and empirical analyses indicate that an overly long multiplication chain tends to cause gradient explosion, resembling the situation in RNNs[Pascanu et al\. \(2013\)](https://arxiv.org/html/2610.22883#bib.bib7)\. To address this issue, we introduce two key improvements\. First, we replace the plain Toeplitz matrix with itsblock\-wise variant\. By partitioning the matrix intop2p^\{2\}blocks, each factor containsp⁡\(2​n−p\)p\(2n\-p\)learnable parameters, which is approximatelypptimes that of a standard Toeplitz matrix\. Consequently, under the same parameter budget, the required chain length can be reduced by roughly a factor ofpp\. Second, we incorporate anidentity matrixto enable residual learning, which further stabilizes optimization\. Together, these refinements lead to our proposedToPAframework, as illustrated in Fig\.[1](https://arxiv.org/html/2610.22883#S1.F1)\.

Theoretically, we show that ToPA enables high\-rank and dense adaptation, effectively addressing the limitations of previous methods\. Moreover, we prove that the product of block\-wise Toeplitz matrices can represent any matrix under a shorter chain length than the classical ones\. Empirically, we conduct extensive experiments on diverse tasks, including natural language understanding, subject\-driven generation, commonsense reasoning, and math & code generation, using multiple models such as DeBERTaV3, Stable\-Diffusion, LLaMA\-2/3, and Qwen3\. The results show that ToPA consistently outperforms strong baselines, validating its effectiveness\.

In summary, our key contributions are as follows:

- •We propose ToPA, a new PEFT technique that exploits block\-wise Toeplitz reparameterization to achieve high expressiveness with a compact parameter budget\.
- •Theoretical analysis shows that ToPA can achieve high\-rank and dense adaptation, enabling stronger expressiveness than prior approaches\.
- •Extensive experiments across multiple benchmarks and models show that ToPA consistently surpasses multiple baselines, highlighting its effectiveness\.

### 2Preliminaries

This section briefly reviews two mainstream structural constraints used by existing reparameterization approaches,i\.e\., low\-rank and sparsity\. Meanwhile, we also highlight their inherent limitations that may hinder expressiveness and adaptation flexibility\.

\(a\)Rank of the update matrix\(b\)Sparsity of adapter𝑹\\bm\{R\}
Figure 2:Fine\-tuning LLaMA\-2\-7B for mathematical reasoning under different parameter budgets\.\(a\)Average rank of the weight update matrixΔ​𝑾\\Delta\\bm\{W\}\.\(b\)Average density of the adapter𝑹\\bm\{R\}across different methods, measured by the non\-zero ratio \(NNZ\)\.#### 2\.1Low\-rank Decomposition and its Limitation

As one of the most representative PEFT methods, LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib1)decomposes the update parameter as the multiplication of two low\-rank matrices𝑩∈ℝm×r,𝑨∈ℝr×n\\bm\{B\}\\in\\mathbb\{R\}^\{m\\times r\},\\bm\{A\}\\in\\mathbb\{R\}^\{r\\times n\}:

𝑾=𝑾0\+𝑩​𝑨,\\bm\{W\}=\\bm\{W\}\_\{0\}\+\\bm\{B\}\\bm\{A\},\(1\)where𝑾0∈ℝm×n\\bm\{W\}\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}denotes the pretrained weights, and the rankr≪min⁡\(m,n\)r\\ll\\min\(m,n\)\.

The success of low\-rank stems from the inductive bias that the over\-parameterized models possess low intrinsic dimension[Li et al\. \(2018\)](https://arxiv.org/html/2610.22883#bib.bib40);[Aghajanyan et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib38)\. However, prior arts point out that low dimension is not equivalent to low rank\. In fact, full fine\-tuning generally produces update matrices with high rank\. Therefore, therank constraintinherently restricts the model’s expressiveness, leading to a performance gap relative to full fine\-tuning\.

To substantiate this, we compute the rank of the updated query and value attention matrices of the Llama2\-7B model fine\-tuned via LoRA for math reasoning task with varying parameter ratios\. As illustrated in Fig\.[2\(a\)](https://arxiv.org/html/2610.22883#S2.F2.sf1), the rank of full fine\-tuning is about37583758, significantly higher than those of LoRA \(e\.g\.,16,32,6416,32,64\)\. Hence, this low\-rank structure creates a severe bottleneck that constrains the model from adequately modeling complex parameter transformations, as corroborated by prior studies[Jiang et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib8);[Huang et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib3);[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib22)\.

#### 2\.2Sparsity Structure and its Limitations

Sparsity structure approaches \(e\.g\., OFT[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib2)and BOFT[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22883#bib.bib5), whose details are provided in App\.[B](https://arxiv.org/html/2610.22883#A2)\) reparameterize the updated weights by multiplying a series of sparse orthogonal matrices𝑩i∈ℝn×n\\bm\{B\}\_\{i\}\\in\\mathbb\{R\}^\{n\\times n\}:

𝑾=𝑾0⋅𝑹=𝑾0⋅\(∏i=1m𝑩i\),\\bm\{W\}=\\bm\{W\}\_\{0\}\\cdot\\bm\{R\}=\\bm\{W\}\_\{0\}\\cdot\(\\prod\_\{i=1\}^\{m\}\\bm\{B\}\_\{i\}\),\(2\)Despite their effectiveness, these methods enforce a fixed, predetermined sparsity pattern where non\-zero entries are specifieda priorirather than learned\. To illustrate this, we have the following reformulation:

\(𝑾0𝑹\):,j=∑i:ri,j≠0ri,j\(𝑾0\):,i∈span\{\(𝑾0\):,i\|ri,j≠0\}\.\(\\bm\{W\}\_\{0\}\\bm\{R\}\)\_\{:,j\}=\\sum\_\{i:r\_\{i,j\}\\neq 0\}r\_\{i,j\}\(\\bm\{W\}\_\{0\}\)\_\{:,i\}\\in\\mathrm\{span\}\\big\\\{\(\\bm\{W\}\_\{0\}\)\_\{:,i\}\\big\|r\_\{i,j\}\\neq 0\\big\\\}\.\(3\)First, each adapted column\(𝑾0𝑹\):,j\(\\bm\{W\}\_\{0\}\\bm\{R\}\)\_\{:,j\}is restricted to a subspacespan\{\(𝑾0\):,i\|ri,j≠0\}\\mathrm\{span\}\\big\\\{\(\\bm\{W\}\_\{0\}\)\_\{:,i\}\\big\|r\_\{i,j\}\\neq 0\\big\\\}, which is manually fixed before training instead of being learned\. Second, due to the masked entries in𝑾0\\bm\{W\}\_\{0\}, the imposed sparsity pattern severs a large fraction of potential cross\-dimensional interactions, greatly reducing parameter connectivity and restricting information flow\.

These issues are particularly evident in practical PEFT settings where the parameter budget is tight\. As shown in Fig\.[2\(b\)](https://arxiv.org/html/2610.22883#S2.F2.sf2), when fine\-tuning the query and value projections of LLaMA\-2\-7B using OFT and BOFT for the math reasoning task, we observe that theNoN\-Zero ratio \(NNZ\) ofR\\bm\{R\}is exceedingly low, ranging from approximately0\.5%0\.5\\%to6\.5%6\.5\\%\. This highly sparse𝑹\\bm\{R\}can substantially hinder adaptation flexibility and thus degrade downstream performance\.

### 3Methodology

Based on the analysis in Sec\.[2](https://arxiv.org/html/2610.22883#S2), this section explores an alternative reparameterization beyond low\-rank and sparsity\.

#### 3\.1Toeplitz Matrix

Originating from Otto Toeplitz’s early 20th\-century work on spectral theory[Toeplitz \(1911\)](https://arxiv.org/html/2610.22883#bib.bib64), a Toeplitz matrix is a structured matrix in which each descending diagonal from left to right is constant:

𝑻⁡\(𝒘\)=\[w0w1⋯wn−1w−1w0⋱⋱⋱w1w−n\+1⋯w−1w0\]∈ℝn×n,\\bm\{T\}\(\\bm\{w\}\)=\\begin\{bmatrix\}w\_\{0\}&w\_\{1\}&\\cdots&w\_\{n\-1\}\\\\ w\_\{\-1\}&w\_\{0\}&\\ddots&\\vdots\\\\ \\vdots&\\ddots&\\ddots&w\_\{1\}\\\\ w\_\{\-n\+1\}&\\cdots&w\_\{\-1\}&w\_\{0\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{n\\times n\},\(4\)where𝒘=\(w−n\+1,…,w0,…,wn−1\)⊤\\bm\{w\}=\(w\_\{\-n\+1\},\\ldots,w\_\{0\},\\ldots,w\_\{n\-1\}\)^\{\\top\}defines the elements along the diagonals\.

An appealing property of the Toeplitz matrix is that its structure isefficientin nature\. For parameter efficiency, ann×nn\\times nToeplitz matrix is fully determined by its defining vector𝒘∈ℝ2​n−1\\bm\{w\}\\in\\mathbb\{R\}^\{2n\-1\}, significantly lower thann2n^\{2\}for general matrices\. For computational efficiency, its matrix–vector multiplication can be efficiently implemented using the Fast Fourier Transform \(FFT\)\. This reduces the computational complexity from𝒪⁡\(n2\)\\mathcal\{O\}\(n^\{2\}\)to𝒪⁡\(n​log⁡n\)\\mathcal\{O\}\(n\\log n\)\. Details on the acceleration mechanism are provided in App\.[D](https://arxiv.org/html/2610.22883#A4)\.

Another desired property is itsexpressiveness\. Intuitively, this shifting structure endows each row with a distinct parameter, thereby almost surely eliminating linear dependence and resulting in a full\-rank matrix\. Moreover, since no deterministic zero pattern is enforced, the compactness does not rely on sparsity\. Overall, this structured parameter matrix isneither restricted by rank nor by sparsity\. Moreover, a classical result in matrix theory establishes that any matrix can be factored as a product of Toeplitz matrices:

###### Theorem 3\.1\.

[Ye and Lim \(2016\)](https://arxiv.org/html/2610.22883#bib.bib4)Everyn×nn\\times nmatrix is generically a product of⌊n2⌋\+1\\lfloor\\frac\{n\}\{2\}\\rfloor\+1Toeplitz matrices and always a product of at most2​n\+52n\+5Toeplitz matrices\.

This result indicates that its expressiveness matches that of full fine\-tuning under a sufficient chain length\.

#### 3\.2A Naive Implementation

Given the aforementioned properties, it is natural to introduce the Toeplitz matrix into PEFT\. A naive implementation is to reparameterize the adapted weight𝑾\\bm\{W\}by multiplying the pretrained weight𝑾0\\bm\{W\}\_\{0\}with a chain of learnable Toeplitz matrices:

𝑾=𝑾0⋅𝑹=𝑾0⋅\(∏i=1t𝑻i\),\\bm\{W\}=\\bm\{W\}\_\{0\}\\cdot\\bm\{R\}=\\bm\{W\}\_\{0\}\\cdot\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{i\}\),\(5\)where each𝑻i\\bm\{T\}\_\{i\}is initialized as the identity matrix𝑰\\bm\{I\}, ensuring𝑾=𝑾0\\bm\{W\}=\\bm\{W\}\_\{0\}at the start of training\. However, this design suffers from a key practical limitation: repeatedly multiplying an overly long chain of Toeplitz matrices can lead to numerical instability, typically manifesting as gradient explosion\. To illustrate this issue, we fine\-tune LLaMA\-3\-8B on the commonsense reasoning task with the chain lengtht=16t=16\(with the matched number of parameters as LoRAr=16\)\. As shown in Fig\.[3](https://arxiv.org/html/2610.22883#S3.F3), we track both the Frobenius norm of gradients and the training loss during the first 700 steps\. Both metrics exhibit substantial fluctuations, indicating unstable optimization and degraded learning dynamics\.

\(a\)Gradient Norm\(b\)Train Loss
Figure 3:Training dynamics over the first 700 steps of fine\-tuning LLaMA\-3\-8B on commonsense reasoning tasks using the naive method \(t=16t=16\) and ToPAp=8t=2\{\}^\{t=2\}\_\{p=8\}with matched parameters\. As training proceeds, the naive method collapses, so only the first 700 steps are shown\.To provide theoretical insights, we investigate this instability stems from the perspective of the gradient structure\. In particular, the gradient with respect to𝑻i\\bm\{T\}\_\{i\}contains a product of all preceding and following Toeplitz factors:

###### Proposition 3\.2\(Gradient of Toeplitz chain\)\.

Letℒ\\mathcal\{L\}denote the scalar loss, and define the gradient with respect to𝐖\\bm\{W\}as𝐆:=∇𝐖ℒ∈ℝm×n\\bm\{G\}:=\\nabla\_\{\\bm\{W\}\}\\mathcal\{L\}\\in\\mathbb\{R\}^\{m\\times n\}\. Then, for eachi∈\{1,…,t\}i\\in\\\{1,\\dots,t\\\}, the gradient with respect to𝐓i\\bm\{T\}\_\{i\}is given by:

∇𝑻iℒ=\(∏k=1i−1𝑻k\)⊤​𝑾0⊤​𝑮​\(∏k=i\+1t𝑻k\)⊤\.\\nabla\_\{\\bm\{T\}\_\{i\}\}\\mathcal\{L\}=\\big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\big\)^\{\\top\}\\bm\{W\}\_\{0\}^\{\\top\}\\bm\{G\}\\big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\big\)^\{\\top\}\.\(6\)

Eq\. \([6](https://arxiv.org/html/2610.22883#S3.E6)\) reveals that the gradient computation involves a cumulative product of matrices across the chain\. Consequently, if the spectral norms of these intermediate matrices exceed unity, which is common during training, the gradient tends to grow exponentially as the chain lengthttincreases\. This phenomenon is analogous to the well\-known gradient explosion in recurrent neural networks \(RNNs\)[Pascanu et al\. \(2013\)](https://arxiv.org/html/2610.22883#bib.bib7)\.

#### 3\.3ToPA: Block\-wise Toeplitz Adaptation

The instability discussed in Sec\.[3\.2](https://arxiv.org/html/2610.22883#S3.SS2)stems from the fixed parameter budget in a single Toeplitz matrix\. As a result, a long chain becomes the only choice for greater model capacity, inevitably amplifying the instability\. To avoid an overly long chain, a simple yet effective method is to increase the budget in a single matrix\. In view of this, we replace the plain Toeplitz matrix with theblock\-wise Toeplitz matrixto:

𝑻blk=\[𝑻11⋯𝑻1​p⋱𝑻p​1⋯𝑻p​p\]∈ℝn×n\.\\bm\{T\}\_\{\\mathrm\{blk\}\}=\\begin\{bmatrix\}\\bm\{T\}\_\{11\}&\\cdots&\\bm\{T\}\_\{1p\}\\\\ \\vdots&\\ddots&\\vdots\\\\ \\bm\{T\}\_\{p1\}&\\cdots&\\bm\{T\}\_\{pp\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{n\\times n\}\.\(7\)Here, each sub\-block𝑻i​j∈ℝnp×np\\bm\{T\}\_\{ij\}\\in\\mathbb\{R\}^\{\\frac\{n\}\{p\}\\times\\frac\{n\}\{p\}\}is an independent Toeplitz matrix, whereppdenotes the number of blocks along each dimension\. On top of this partition,𝑻blk\\bm\{T\}\_\{\\mathrm\{blk\}\}yieldsp⁡\(2​n−p\)p\(2n\-p\)parameters, approximatelypptimes that of the standard Toeplitz matrix\. This reformulation not only allows the chain length to be reduced by a factor ofppunder the same parameter budget, but also maintains the same gradientw\.r\.t\.𝑻blki\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}as that in Eq\. \([6](https://arxiv.org/html/2610.22883#S3.E6)\), significantly promoting training stability\.

To further stabilize optimization, we introduce a residual connection to our final framework,ToPA:

𝑾=𝑾0⋅𝑹=𝑾0⋅\(∏i=1t𝑻blki\+𝑰\),\\bm\{W\}=\\bm\{W\}\_\{0\}\\cdot\\bm\{R\}=\\bm\{W\}\_\{0\}\\cdot\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\+\\bm\{I\}\),\(8\)where𝑰\\bm\{I\}is the identity matrix\. This residual term centers the adaptation around the identity mapping, effectively stabilizing optimization[He et al\. \(2016\)](https://arxiv.org/html/2610.22883#bib.bib9)\. Moreover, it imposes an inductive bias that keeps the adapted weights close to the pretrained initialization, which aligns with the fine\-tuning settings\.

As shown in Fig\.[3](https://arxiv.org/html/2610.22883#S3.F3), ToPA demonstrates markedly smoother training dynamics compared to the naive implementation under the same parameter budget, confirming the effectiveness of our approach\.

#### 3\.4Initialization Strategy of ToPA

To ensure numerical stability during fine\-tuning, the model initialization should preserve the exact behavior of the pretrained network\. Formally, this requires that the learnable modules contribute zero output at the start of training,i\.e\.,∏i=1t𝑻blki=0\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}=0\. We implement this via an asymmetric initialization strategy analogous to LoRA, where𝑩=0\\bm\{B\}=0and𝑨∼𝒩⁡\(0,σ2\)\\bm\{A\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)\. Formally, the first block\-wise Toeplitz matrix is initialized by𝟎\\bm\{0\}, while the others are drawn from the normal distribution:

𝒘i​j1=0,𝒘i​j2,…,𝒘i​jt∼𝒩⁡\(0,σ2\),i,j∈\[1,p\],\\bm\{w\}\_\{ij\}^\{1\}=0,\\ \\bm\{w\}\_\{ij\}^\{2\},\\ldots,\\bm\{w\}\_\{ij\}^\{t\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\),\\ i,j\\in\[1,p\],\(9\)where𝒘i​jk∈ℝ2​n/p−1\\bm\{w\}\_\{ij\}^\{k\}\\in\\mathbb\{R\}^\{2n/p\-1\}denotes the defining vector of\(i,j\)\(i,j\)block ofkk\-th block\-wise Toeplitz matrix\.

### 4Theoretical Analysis

#### 4\.1Achieving High\-Rank and Dense Adaptation

High\-rank Update\.Owing to the block\-wise Toeplitz structure in Eq\. \([7](https://arxiv.org/html/2610.22883#S3.E7)\), each factor𝑻blki\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}in ToPA is almost surely invertible\. Thus, the resulting update matrixΔ​𝑾:=𝑾0⋅\(∏i=1t𝑻blki\)\\Delta\\bm\{W\}:=\\bm\{W\}\_\{0\}\\cdot\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\)is capable ofachieving high or even full rank\. This property is formally stated in the following proposition:

###### Proposition 4\.1\.

Assume thatrank⁡\(𝐖0\)=s\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\)=s\. Then,Δ​𝐖\\Delta\\bm\{W\}has rankssalmost surely, i\.e\.,

ℙ⁡\(rank⁡\(Δ​𝑾\)=s\)=1\.\\mathbb\{P\}\\\!\\left\(\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=s\\right\)=1\.\(10\)

Since the pretrained weights𝑾0\\bm\{W\}\_\{0\}typically exhibit high rank, ToPA naturally performs expressive, high\-rank updates, effectively circumventing the representational bottlenecks of low\-rank adaptation methods discussed in Sec\.[2](https://arxiv.org/html/2610.22883#S2)\. Empirically, Fig\.[2\(a\)](https://arxiv.org/html/2610.22883#S2.F2.sf1)confirms our analysis, showing that ToPA yields substantially higher rank updates than LoRA, under comparable parameter budgets\.

Dense Transformation with Parameter Efficiency\.Unlike sparsity structured methods, each block\-wise Toeplitz matrix𝑻blki\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}remains dense while requiring onlyp⁡\(2​n−p\)p\(2n\-p\)parameters, as shown in Eq\. \([4](https://arxiv.org/html/2610.22883#S3.E4)\)\. This dense parameterization yields two critical advantages\. First, it restores cross\-dimensional interactions severed by rigid sparsity masks, facilitating global information flow\. Second, it liberates the adapted columns from being confined to a predetermined subspace, enabling the model to explore the complete column space of𝑾0\\bm\{W\}\_\{0\}\. By effectively circumventing these limitations, ToPA preserves high expressive power\. As shown in Fig\.[2\(b\)](https://arxiv.org/html/2610.22883#S2.F2.sf2), the resulting adapter𝑹\\bm\{R\}is substantially denser than those produced by OFT and BOFT, reinforcing our analysis\.

#### 4\.2Expressive Power of ToPA

❶Compositional Expressiveness\.Recall that Thm\.[3\.1](https://arxiv.org/html/2610.22883#S3.Thmtheorem1)substantiates the strong expressiveness of standard Toeplitz matrices\. Hence, it is natural to extend this compelling result to the block\-wise Toeplitz matrix:

###### Theorem 4\.2\(Informal\)\.

Everyn×nn\\times nmatrix is generically a product of⌊n2​p⌋\+1\\lfloor\\frac\{n\}\{2p\}\\rfloor\+1block\-wise Toeplitz matrices and always a product of at most2​np\+5\\frac\{2n\}\{p\}\+5block\-wise Toeplitz matrices \(with each Toeplitz block of sizen/pn/p\)\.

This theorem suggests that a composition of block\-wise Toeplitz matrices can represent any arbitrary matrix under a shorter chain length than the classical results\.

❷Stronger Expressive Power\.Beyond compositional expressiveness, ToPA can exhibit stronger expressive power than low\-rank decomposition and sparse structure methods for certain transformations\. Specifically, we show that there exists a family of adaptations that can be compactly represented by ToPA but cannot be efficiently captured by either individual approach:

###### Theorem 4\.3\.

There exists a class of adapted weight matrices𝐖\\bm\{W\}, that can be realized by ToPA with𝒪⁡\(n\)\\mathcal\{O\}\(n\)trainable parameters, while achieving the same𝐖\\bm\{W\}with LoRA at least requires𝒪⁡\(n2\)\\mathcal\{O\}\(n^\{2\}\)parameters and with OFT or BOFT requires at least𝒪⁡\(n​log⁡n\)\\mathcal\{O\}\(n\\log n\)parameters\.

This theorem indicates that ToPA can realize certain transformations with significantly fewer parameters than low\-rank or sparsity methods, which highlights ToPA’s superior expressive capacity\.

Table 1:Performance comparison on commonsense reasoning tasks with LLaMA\-2\-7B and LLaMA\-3\-8B backbones\.Table 2:Performance comparison on math reasoning and code generation tasks with the LLaMA\-3\-8B backbone\.
Table 3:Performance comparison on math reasoning and code generation tasks with the Qwen3\-4B backbone\.

### 5Experiments

#### 5\.1Experiments Protocols

Datasets\.We evaluate ToPA on a broad range of NLP and CV tasks\.\(1\)Commonsense Reasoning: We train on Commonsense170K[Hu et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib24)dataset and evaluate on eight commonsense benchmarks\.\(2\)Math Reasoning & Code Generation: For math reasoning, we train on a 100k subset of MetaMathQA[Yu et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib25)dataset and evaluate on GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib26)and MATH[Yu et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib25)datasets\. For code generation, we train on CodeFeedback[Zheng et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib27)dataset, and evaluate on HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib19)and MBPP[Austin et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib28)\.\(3\)Subject\-Driven Generation: We adopt the DreamBooth[Ruiz et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib29)dataset\.\(4\)Natural Language Understanding: We use the GLUE[Wang et al\. \(2019\)](https://arxiv.org/html/2610.22883#bib.bib23)benchmark\. For details of these datasets, please refer to App\.[E\.1](https://arxiv.org/html/2610.22883#A5.SS1)\.

Competitors\.We compare ToPA with representative methods, including low\-rank techniques such as LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib1), DoRA[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.22883#bib.bib11), and PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib10)\. We also consider high\-rank variants of LoRA, such as HiRA[Huang et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib3)and RidgeLoRA[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib22)\. In addition, we include sparsity approaches, including OFT[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib2)and BOFT[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22883#bib.bib5), and their variant HRA[Yuan et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib6)\. Details of these baselines can be found in App\.[E\.2](https://arxiv.org/html/2610.22883#A5.SS2)\.

Implementation Details\.We use task\-specific configurations across different evaluation settings\. For commonsense reasoning, we fine\-tune LLaMA\-2\-7B[Touvron et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib31)and LLaMA\-3\-8B[Team \(2024\)](https://arxiv.org/html/2610.22883#bib.bib32)\. For mathematical reasoning and code generation, we evaluate on LLaMA\-2\-7B and Qwen3\-4B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib33)\. For subject\-driven generation, we adopt Stable Diffusion v1\.5[Rombach et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib34)\. Finally, for NLU tasks, we utilize DeBERTaV3\-base[He et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib30)\. Additional details are provided in App\.[E\.3](https://arxiv.org/html/2610.22883#A5.SS3)\.

#### 5\.2Commonsense Reasoning

Tab\.[1](https://arxiv.org/html/2610.22883#S4.T1)reports results on commonsense reasoning tasks\. Across both LLaMA\-2\-7B and LLaMA\-3\-8B backbones, ToPA achieves the best average performance while maintaining a comparable or smaller parameter count\. Specifically, ToPA attains a leading average score of 81\.4% on LLaMA\-2\-7B and 86\.3% on LLaMA\-3\-8B, consistently surpassing all baseline methods\. These findings demonstrate that ToPA enables highly effective and efficient adaptation for commonsense reasoning tasks\.

![Refer to caption](https://arxiv.org/html/2610.22883v1/Subject-Driven_Generation-v2.png)Figure 4:Qualitative comparison of results on subject\-driven generation tasks\.
#### 5\.3Math Reasoning & Code Generation

Tab\.[3](https://arxiv.org/html/2610.22883#S4.T3)and Tab\.[3](https://arxiv.org/html/2610.22883#S4.T3)report ToPA’s performance on math and code generation tasks with LLaMA\-3\-8B and Qwen3\-4B\. Overall, ToPA consistently surpasses all baseline methods on GSM8K, MATH, and HumanEval benchmarks with comparable or fewer parameters\. Specifically, with LLaMA\-3\-8B, ToPA achieves 74\.7% on GSM8K and 51\.8% on HumanEval, exceeding the strongest baseline by 0\.9% and 0\.6%, respectively\. On Qwen3\-4B, ToPA achieves the best performance, with 84\.3% on GSM8K and 73\.2% on HumanEval\. Although ToPA is slightly below the leading method on MBPP, the gap is marginal\. These results confirm ToPA’s effectiveness in both math and code generation\.

#### 5\.4Subject\-Driven Generation

Table 4:Performance comparison on subject\-driven generation\.Tab\.[4](https://arxiv.org/html/2610.22883#S5.T4)presents the quantitative evaluation of subject\-driven generation on Stable Diffusion v1\.5\. Under a comparable or reduced parameter budget, ToPA achieves superior performance on DINO[Caron et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib35), CLIP\-I[Radford et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib36), and CLIP\-T[Radford et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib36)metrics\. Given that DINO and CLIP\-I quantify subject fidelity while CLIP\-T measures prompt alignment, these results indicate that ToPA excels at preserving subject identity and adhering to textual instructions\. Although there is a marginal trade\-off in sample diversity as indicated by the LPIPS[Zhang et al\. \(2018\)](https://arxiv.org/html/2610.22883#bib.bib37)metric, the overall performance remains robust\. These findings are visually corroborated in Fig\.[4](https://arxiv.org/html/2610.22883#S5.F4), where ToPA consistently outperforms baselines like LoRA and OFT, yielding images with higher fidelity and superior prompt alignment across diverse subjects\.

Table 5:Performance comparison on the GLUE benchmark using the DeBERTaV3\-base model\. We report Matthew’s correlation for CoLA, Pearson correlation for STS\-B, matched accuracy for MNLI, and accuracy for the other datasets\.\(a\)Impact ofpp\(b\)Impact oftt
Figure 5:Ablation study on the effects ofppandttin ToPA\. We varypporttand fine\-tune LLaMA\-3\-8B using ToPA on commonsense reasoning tasks\.
\(a\)LLaMA\-2\-7b\(b\)LLaMA\-3\-8b
Figure 6:Ablation of the identity residual in ToPA on commonsense reasoning tasks when fine\-tuning LLaMA\-2\-7B and LLaMA\-3\-8B\.

#### 5\.5Natural Language Understanding

Tab\.[5](https://arxiv.org/html/2610.22883#S5.T5)demonstrates the performance of different methods for fine\-tuning the DeBERTaV3\-base model on the GLUE benchmark, with all methods limited to about 0\.44M parameters\. The results indicate that ToPA achieves the best average score of 88\.75, outperforming several baselines\. In particular, ToPA yields a 0\.29% gain in the average score and consistently ranks first or second across eight tasks\. These results show that ToPA has a stronger capability under the same parameter budget\.

\(a\)GSM8K\(b\)MATH
Figure 7:Performance across parameter scales\. We compare ToPA and LoRA on math reasoning tasks using LLaMA\-3\-8B\.
#### 5\.6Performance Comparison across Parameter Scales

To evaluate ToPA across different parameter scales, we compare it with LoRA on mathematical reasoning tasks using LLaMA\-3\-8B\. LoRA uses ranksr∈\{8,16,32\}r\\in\\\{8,16,32\\\}, while ToPA fixest=3t=3and variesp∈\{2,4,8\}p\\in\\\{2,4,8\\\}for comparable parameter budgets\. As shown in Fig\.[7](https://arxiv.org/html/2610.22883#S5.F7.fig1), ToPA outperforms LoRA at each matched scale on both GSM8K and MATH, demonstrating its effectiveness\.

#### 5\.7Influence ofppandtt

Impact ofpp:Fig\.[5\(a\)](https://arxiv.org/html/2610.22883#S5.F5.sf1)examines the impact of the block numberpp\. Concretely, we fix the chain lengthttto 3 and varyppover\{1,2,4,8\}\\\{1,2,4,8\\\}\. Asppincreases, the number of learnable parameters grows accordingly, leading to a stronger expressive capacity for ToPA\. Consequently, we observe a consistent performance gain on all three datasets, where largerppgenerally yields higher accuracy\.

Impact oftt:Fig\.[5\(b\)](https://arxiv.org/html/2610.22883#S5.F5.sf2)investigates the impact of chain lengthtt\. We fix the block numberppto 8 and varyttin\{1,2,3,4\}\\\{1,2,3,4\\\}\. Asttincreases from 1 to 3, performance consistently improves on all three benchmarks, indicating stronger expressiveness with longer chains\. However, whenttreaches 4, accuracy slightly drops on OBQA and PIQA, which may be attributed to the optimization instability of longer chains, leading to suboptimal training, as discussed in Sec\.[3](https://arxiv.org/html/2610.22883#S3)\.

#### 5\.8Ablation Study of Identity Residual

To evaluate the contribution of the identity residual in ToPA, we conducted an ablation study by fine\-tuning LLaMA\-2\-7B and LLaMA\-3\-8B on commonsense reasoning benchmarks\. As illustrated in Fig\.[6](https://arxiv.org/html/2610.22883#S5.F6), incorporating the identity matrix consistently enhances performance across both backbones and all evaluated datasets, confirming its effectiveness\.

### 6Related Work

#### 6\.1Reparameterization\-Based Methods

Among PEFT approaches, reparameterization\-based methods are widely adopted\. These methods reparameterize large weight matrices into compact auxiliary components, such that only the lightweight parameters are updated during fine\-tuning\. After training, the auxiliary parameters can be merged back into the original weights without increasing inference latency, enabling efficient adaptation with minimal computational overhead\.

Most existing methods typically follow two main directions\. The first is low\-rank decomposition, such as LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib1), which factorizes weight updates into the products of two low\-rank matrices to achieve parameter efficiency\. To further enhance performance, numerous variants have been proposed\. For instance, DoRA[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.22883#bib.bib11)decomposes the weight into magnitude and direction components, directly tuning the magnitude while applying LoRA to the direction, thereby improving learning capacity\. PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2610.22883#bib.bib10)applies SVD to decompose each pretrained weight matrix into principal and residual components, updating the principal part while freezing the residual part to enable faster convergence\. HiRA[Huang et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib3)expresses the update matrix as the Hadamard product between the frozen pretrained weight and a low\-rank adapter, thereby achieving a higher\-rank and more expressive adaptation\. RidgeLoRA[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.22883#bib.bib22)replaces LoRA’s parallel structure with a series connection and augments the low\-rank adapter with a diagonal ridge term, enabling high\-rank weight updates with improved representation capacity\. The second category introduces sparsity\-based reparameterization\. OFT[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.22883#bib.bib2)reparameterizes the pretrained matrix using a sparse orthogonal matrix to preserve pretrained knowledge\. BOFT[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22883#bib.bib5)extends OFT by replacing the single sparse orthogonal matrix with a product of multiple sparse butterfly matrices, thereby further enhancing model expressiveness\. PSOFT[Wu et al\. \(2026\)](https://arxiv.org/html/2610.22883#bib.bib70)confines orthogonal transformations to a low\-rank principal subspace, effectively preserving pretrained geometry while improving efficiency and expressiveness\.

#### 6\.2Other PEFT Methods

Beyond reparameterization strategies, a number of other PEFT approaches have been proposed\. The prompt\-based family learns task\-specific conditioning signals without updating backbone weights\. Prompt Tuning[Lester et al\. \(2021\)](https://arxiv.org/html/2610.22883#bib.bib13);[Zhou et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib16)optimizes a small set of continuous embeddings prepended to the input, whereas Prefix Tuning[Li and Liang \(2021\)](https://arxiv.org/html/2610.22883#bib.bib14)trains key–value prefixes in attention layers to guide the model at the hidden\-state level\. Adapter\-based methods instead insert lightweight bottleneck modules into each transformer block\. The Adapter framework[Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.22883#bib.bib12)captures task\-specific transformations while keeping the pretrained weights frozen, and AdaptFormer[Chen et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib41)enhances this design with residual adapter branches that increase adaptation capacity at minimal parameter cost\. Finally, selective\-parameter tuning methods update only a very small subset of existing parameters: BitFit[Zaken et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib15)adjusts only bias terms to obtain competitive performance with negligible overhead, while IA3[Liu et al\. \(2022\)](https://arxiv.org/html/2610.22883#bib.bib42)learns multiplicative gating vectors over attention and feed\-forward activations to enable highly parameter\-efficient task conditioning\.

### 7Conclusion

In this work, we identify the limitations of existing reparameterization paradigms that rely on low\-rank or sparse structures\. To address these challenges, we introduce ToPA\. This block\-wise Toeplitz framework enables high\-rank, dense parameter updates while maintaining parameter efficiency and optimization stability\. Theoretically, we prove that ToPA possesses compositional expressiveness and can offer superior expressive capacity compared to low\-rank and sparse methods\. Extensive experiments across diverse models and tasks validate the effectiveness of our approach\.

### Acknowledgements

This work was supported in part by National Natural Science Foundation of China: 62525212, U23B2051, 62236008, 62441232, 62521007, U2541229, U21B2038, 62576332 and 62502500, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No\. XDB0680201, in part by the project ZR2025ZD01 supported by Shandong Provincial Natural Science Foundation, in part by Ningbo Science and Technology Innovation 2025 Major Project 2025Z027, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384, in part by Beijing Natural Science Foundation–Xiaomi Innovation Joint Fund \(Grant No\. 26L030072\), in part by General Program of the Chinese Postdoctoral Science Foundation under Grant No\. 2025M771558, and in part by Beijing Natural Science Foundation under Grant No\. L252144\.

### References

- A\. Aghajanyan, S\. Gupta, and L\. ZettlemoyerIntrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 7319–7328\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22883#S2.SS1.p2.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. I\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. J\. Cai, M\. Terry, Q\. V\. Le, and C\. SuttonProgram synthesis with large language models\.CoRR\.Cited by:[4th item](https://arxiv.org/html/2610.22883#A5.I3.i4.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. ChoiPIQA: reasoning about physical commonsense in natural language\.InThe AAAI Conference on Artificial Intelligence,pp\. 7432–7439\.Cited by:[2nd item](https://arxiv.org/html/2610.22883#A5.I2.i2.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAnnual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1)\.
- Caronet al\.\(2021\)M\. Caron, H\. Touvron, I\. Misra, H\. Jégou, J\. Mairal, P\. Bojanowski, and A\. JoulinEmerging properties in self\-supervised vision transformers\.InIEEE/CVF International Conference on Computer Vision,pp\. 9630–9640\.Cited by:[§5\.4](https://arxiv.org/html/2610.22883#S5.SS4.p1.1)\.
- Caron and Traynor \(2005\)R\. Caron and T\. TraynorThe zero set of a polynomial\.WSMR Report,pp\. 05–02\.Cited by:[§A\.2](https://arxiv.org/html/2610.22883#A1.SS2.p4.3.1),[§A\.4](https://arxiv.org/html/2610.22883#A1.SS4.p8.1.1)\.
- Ceret al\.\(2017\)D\. M\. Cer, M\. T\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. SpeciaSemEval\-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation\.InInternational Workshop on Semantic Evaluation,pp\. 1–14\.Cited by:[4th item](https://arxiv.org/html/2610.22883#A5.I1.i4.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.CoRRabs/2107\.03374\.Cited by:[3rd item](https://arxiv.org/html/2610.22883#A5.I3.i3.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Chenet al\.\(2022\)S\. Chen, C\. Ge, Z\. Tong, J\. Wang, Y\. Song, J\. Wang, and P\. LuoAdaptFormer: adapting vision transformers for scalable visual recognition\.InAnnual Conference on Neural Information Processing Systems,Cited by:[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: exploring the surprising difficulty of natural yes/no questions\.InConference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,,pp\. 2924–2936\.Cited by:[1st item](https://arxiv.org/html/2610.22883#A5.I2.i1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the AI2 reasoning challenge\.CoRR\.Cited by:[4th item](https://arxiv.org/html/2610.22883#A5.I2.i4.p1.1),[5th item](https://arxiv.org/html/2610.22883#A5.I2.i5.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.CoRR\.Cited by:[1st item](https://arxiv.org/html/2610.22883#A5.I3.i1.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Coxet al\.\(1997\)D\. A\. Cox, J\. Little, and D\. O’SheaIdeals, varieties, and algorithms \- an introduction to computational algebraic geometry and commutative algebra \(2\. ed\.\)\.Springer\.Cited by:[§A\.2](https://arxiv.org/html/2610.22883#A1.SS2.p4.3.1)\.
- Dolan and Brockett \(2005\)W\. B\. Dolan and C\. BrockettAutomatically constructing a corpus of sentential paraphrases\.InInternational Workshop on Paraphrasing,Cited by:[3rd item](https://arxiv.org/html/2610.22883#A5.I1.i3.p1.1)\.
- Giampiccoloet al\.\(2007\)D\. Giampiccolo, B\. Magnini, I\. Dagan, and B\. DolanThe third PASCAL recognizing textual entailment challenge\.InProceedings of the ACL\-PASCAL@ACL 2007 Workshop on Textual Entailment and Paraphrasing,pp\. 1–9\.Cited by:[8th item](https://arxiv.org/html/2610.22883#A5.I1.i8.p1.1)\.
- Grayet al\.\(2006\)R\. M\. Grayet al\.Toeplitz and circulant matrices: a review\.Foundations and Trends® in Communications and Information Theory2\(3\),pp\. 155–239\.Cited by:[Appendix D](https://arxiv.org/html/2610.22883#A4.p1.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§3\.3](https://arxiv.org/html/2610.22883#S3.SS3.p2.2)\.
- Heet al\.\(2023\)P\. He, J\. Gao, and W\. ChenDeBERTaV3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.InInternational Conference on Learning Representations,Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p3.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InAnnual Conference on Neural Information Processing Systems,Cited by:[2nd item](https://arxiv.org/html/2610.22883#A5.I3.i2.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for NLP\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 2790–2799\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1),[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[1st item](https://arxiv.org/html/2610.22883#A5.I5.i1.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.22883#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Huet al\.\(2023\)Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. K\. LeeLLM\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InConference on Empirical Methods in Natural Language Processing,pp\. 5254–5276\.Cited by:[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Huanget al\.\(2025\)Q\. Huang, T\. Ko, Z\. Zhuang, L\. Tang, and Y\. ZhangHiRA: parameter\-efficient hadamard high\-rank adaptation for large language models\.InInternational Conference on Learning Representations,Cited by:[4th item](https://arxiv.org/html/2610.22883#A5.I5.i4.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22883#S2.SS1.p3.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Jianget al\.\(2024\)T\. Jiang, S\. Huang, S\. Luo, Z\. Zhang, H\. Huang, F\. Wei, W\. Deng, F\. Sun, Q\. Zhang, D\. Wang, and F\. ZhuangMoRA: high\-rank updating for parameter\-efficient fine\-tuning\.CoRR\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22883#S2.SS1.p3.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InConference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1),[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Liet al\.\(2018\)C\. Li, H\. Farkhoor, R\. Liu, and J\. YosinskiMeasuring the intrinsic dimension of objective landscapes\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22883#S2.SS1.p2.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 4582–4597\.Cited by:[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Liet al\.\(2022\)Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. D\. Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. S\. Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. VinyalsCompetition\-level code generation with alphacode\.Science378\(6624\),pp\. 1092–1097\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1)\.
- Liuet al\.\(2022\)H\. Liu, D\. Tam, M\. Muqeeth, J\. Mohta, T\. Huang, M\. Bansal, and C\. RaffelFew\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.InAnnual Conference on Neural Information Processing Systems,Cited by:[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Liuet al\.\(2024a\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning,Cited by:[2nd item](https://arxiv.org/html/2610.22883#A5.I5.i2.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Liuet al\.\(2024b\)W\. Liu, Z\. Qiu, Y\. Feng, Y\. Xiu, Y\. Xue, L\. Yu, H\. Feng, Z\. Liu, J\. Heo, S\. Peng, Y\. Wen, M\. J\. Black, A\. Weller, and B\. SchölkopfParameter\-efficient orthogonal finetuning via butterfly factorization\.InInternational Conference on Learning Representations,Cited by:[§A\.4](https://arxiv.org/html/2610.22883#A1.SS4.p16.1.1),[7th item](https://arxiv.org/html/2610.22883#A5.I5.i7.p1.1),[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.22883#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p1.1)\.
- Meiet al\.\(2026\)J\. Mei, M\. Sun, J\. Chen, P\. Qin, Y\. Li, D\. Chen, and B\. ByrneExPO\-hm: learning to explain\-then\-detect for hateful meme detection\.InInternational Conference on Learning Representations,Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2610.22883#Ax2.p2.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPiSSA: principal singular values and singular vectors adaptation of large language models\.InAnnual Conference on Neural Information Processing Systems,Cited by:[3rd item](https://arxiv.org/html/2610.22883#A5.I5.i3.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? A new dataset for open book question answering\.InConference on Empirical Methods in Natural Language Processing,pp\. 2381–2391\.Cited by:[6th item](https://arxiv.org/html/2610.22883#A5.I2.i6.p1.1)\.
- Pascanuet al\.\(2013\)R\. Pascanu, T\. Mikolov, and Y\. BengioOn the difficulty of training recurrent neural networks\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p6.1),[§3\.2](https://arxiv.org/html/2610.22883#S3.SS2.p3.1)\.
- Penget al\.\(2026\)Y\. Peng, Z\. Wang, G\. Li, X\. Zheng, S\. Yin, and H\. HeA survey on fine\-grained multimodal large language models\.Chinese Journal of Electronics35\(2\),pp\. 771–803\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2610.22883#Ax2.p2.1)\.
- Qinet al\.\(2023\)Z\. Qin, X\. Han, W\. Sun, B\. He, D\. Li, D\. Li, Y\. Dai, L\. Kong, and Y\. ZhongToeplitz neural network for sequence modeling\.InInternational Conference on Learning Representations,Cited by:[Appendix D](https://arxiv.org/html/2610.22883#A4.p1.1)\.
- Qiuet al\.\(2023\)Z\. Qiu, W\. Liu, H\. Feng, Y\. Xue, Y\. Feng, Z\. Liu, D\. Zhang, A\. Weller, and B\. SchölkopfControlling text\-to\-image diffusion by orthogonal finetuning\.InAnnual Conference on Neural Information Processing Systems,Cited by:[6th item](https://arxiv.org/html/2610.22883#A5.I5.i6.p1.1),[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p2.1),[§1](https://arxiv.org/html/2610.22883#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.22883#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. SutskeverLearning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,pp\. 8748–8763\.Cited by:[§5\.4](https://arxiv.org/html/2610.22883#S5.SS4.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100, 000\+ questions for machine comprehension of text\.InConference on Empirical Methods in Natural Language Processing,pp\. 2383–2392\.Cited by:[7th item](https://arxiv.org/html/2610.22883#A5.I1.i7.p1.1)\.
- Rombachet al\.\(2022\)R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. OmmerHigh\-resolution image synthesis with latent diffusion models\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10674–10685\.Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p2.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p3.1)\.
- Ronnebergeret al\.\(2015\)O\. Ronneberger, P\. Fischer, and T\. BroxU\-net: convolutional networks for biomedical image segmentation\.InMedical Image Computing and Computer\-Assisted Intervention,Vol\.9351,pp\. 234–241\.Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p2.1)\.
- Ruizet al\.\(2023\)N\. Ruiz, Y\. Li, V\. Jampani, Y\. Pritch, M\. Rubinstein, and K\. AbermanDreamBooth: fine tuning text\-to\-image diffusion models for subject\-driven generation\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22500–22510\.Cited by:[1st item](https://arxiv.org/html/2610.22883#A5.I4.i1.p1.1),[§E\.1](https://arxiv.org/html/2610.22883#A5.SS1.p7.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.Commun\. ACM64\(9\),pp\. 99–106\.Cited by:[8th item](https://arxiv.org/html/2610.22883#A5.I2.i8.p1.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. L\. Bras, and Y\. ChoiSocial iqa: commonsense reasoning about social interactions\.InConference on Empirical Methods in Natural Language Processing,pp\. 4462–4472\.Cited by:[3rd item](https://arxiv.org/html/2610.22883#A5.I2.i3.p1.1)\.
- Socheret al\.\(2013\)R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Y\. Ng, and C\. PottsRecursive deep models for semantic compositionality over a sentiment treebank\.InConference on Empirical Methods in Natural Language Processing,pp\. 1631–1642\.Cited by:[2nd item](https://arxiv.org/html/2610.22883#A5.I1.i2.p1.1)\.
- Team \(2024\)L\. TeamThe llama 3 herd of models\.CoRRabs/2407\.21783\.Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p3.1),[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p4.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p3.1)\.
- Thoppilanet al\.\(2022\)R\. Thoppilan, D\. D\. Freitas, J\. Hall, N\. Shazeer, A\. Kulshreshtha, H\. Cheng, A\. Jin, T\. Bos, L\. Baker, Y\. Du, Y\. Li, H\. Lee, H\. S\. Zheng, A\. Ghafouri, M\. Menegali, Y\. Huang, M\. Krikun, D\. Lepikhin, J\. Qin, D\. Chen, Y\. Xu, Z\. Chen, A\. Roberts, M\. Bosma, Y\. Zhou, C\. Chang, I\. Krivokon, W\. Rusch, M\. Pickett, K\. S\. Meier\-Hellstern, M\. R\. Morris, T\. Doshi, R\. D\. Santos, T\. Duke, J\. Soraker, B\. Zevenbergen, V\. Prabhakaran, M\. Diaz, B\. Hutchinson, K\. Olson, A\. Molina, E\. Hoffman\-John, J\. Lee, L\. Aroyo, R\. Rajakumar, A\. Butryna, M\. Lamm, V\. Kuzmina, J\. Fenton, A\. Cohen, R\. Bernstein, R\. Kurzweil, B\. A\. y Arcas, C\. Cui, M\. Croak, E\. H\. Chi, and Q\. LeLaMDA: language models for dialog applications\.CoRRabs/2201\.08239\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1)\.
- Tianet al\.\(2023\)Y\. Tian, Y\. Fu, and J\. ZhangTransformer\-based under\-sampled single\-pixel imaging\.Chinese Journal of Electronics32\(5\),pp\. 1151–1159\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2610.22883#Ax2.p2.1)\.
- Toeplitz \(1911\)O\. ToeplitzZur theorie der quadratischen und bilinearen formen von unendlichvielen veränderlichen: i\. teil: theorie der l\-formen\.Mathematische Annalen70\(3\),pp\. 351–376\.Cited by:[§3\.1](https://arxiv.org/html/2610.22883#S3.SS1.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. Canton\-Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. ScialomLlama 2: open foundation and fine\-tuned chat models\.CoRR\.Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p3.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p3.1)\.
- Wanget al\.\(2019\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. BowmanGLUE: A multi\-task benchmark and analysis platform for natural language understanding\.InInternational Conference on Learning Representations,Cited by:[§E\.1](https://arxiv.org/html/2610.22883#A5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Wanget al\.\(2025\)D\. Wang, J\. Ren, H\. Huang, and J\. QiaoParticle swarm optimization for adaptive\-critic feedback control with power system applications\.Chinese Journal of Electronics34\(4\),pp\. 1265–1274\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2610.22883#Ax2.p2.1)\.
- Wanget al\.\(2017\)Z\. Wang, W\. Hamza, and R\. FlorianBilateral multi\-perspective matching for natural language sentences\.InInternational Joint Conference on Artificial Intelligence,pp\. 4144–4150\.Cited by:[5th item](https://arxiv.org/html/2610.22883#A5.I1.i5.p1.1)\.
- Warstadtet al\.\(2019\)A\. Warstadt, A\. Singh, and S\. R\. BowmanNeural network acceptability judgments\.Trans\. Assoc\. Comput\. Linguistics7,pp\. 625–641\.Cited by:[1st item](https://arxiv.org/html/2610.22883#A5.I1.i1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAnnual Conference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1)\.
- Williamset al\.\(2018\)A\. Williams, N\. Nangia, and S\. R\. BowmanA broad\-coverage challenge corpus for sentence understanding through inference\.InConference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 1112–1122\.Cited by:[6th item](https://arxiv.org/html/2610.22883#A5.I1.i6.p1.1)\.
- Wuet al\.\(2026\)F\. Wu, J\. Hu, G\. Min, and S\. WangEfficient orthogonal fine\-tuning with principal subspace adaptation\.InInternational Conference on Learning Representations,Cited by:[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.CoRR\.Cited by:[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p4.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p3.1)\.
- Ye and Lim \(2016\)K\. Ye and L\. LimEvery matrix is a product of toeplitz matrices\.Found\. Comput\. Math\.16\(3\),pp\. 577–598\.Cited by:[§A\.3](https://arxiv.org/html/2610.22883#A1.SS3.p4.2.1),[§A\.3](https://arxiv.org/html/2610.22883#A1.SS3.p6.1.1),[Lemma A\.4](https://arxiv.org/html/2610.22883#A1.Thmtheorem4.p1.1),[Theorem 3\.1](https://arxiv.org/html/2610.22883#S3.Thmtheorem1.p1.1)\.
- Yeet al\.\(2022\)Z\. Ye, X\. He, and Y\. PengUnsupervised cross\-media hashing learning via knowledge graph\.Chinese Journal of Electronics31\(6\),pp\. 1081–1091\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2610.22883#Ax2.p2.1)\.
- Yuet al\.\(2024\)L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetaMath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Yuanet al\.\(2024\)S\. Yuan, H\. Liu, and H\. XuBridging the gap between low\-rank and orthogonal adaptation via householder reflection adaptation\.InAnnual Conference on Neural Information Processing Systems,Cited by:[8th item](https://arxiv.org/html/2610.22883#A5.I5.i8.p1.1),[§E\.3](https://arxiv.org/html/2610.22883#A5.SS3.p1.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1)\.
- Zakenet al\.\(2022\)E\. B\. Zaken, Y\. Goldberg, and S\. RavfogelBitFit: simple parameter\-efficient fine\-tuning for transformer\-based masked language\-models\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 1–9\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1),[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[7th item](https://arxiv.org/html/2610.22883#A5.I2.i7.p1.1)\.
- Zhanget al\.\(2018\)R\. Zhang, P\. Isola, A\. A\. Efros, E\. Shechtman, and O\. WangThe unreasonable effectiveness of deep features as a perceptual metric\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 586–595\.Cited by:[§5\.4](https://arxiv.org/html/2610.22883#S5.SS4.p1.1)\.
- Zhanget al\.\(2024\)T\. Zhang, Y\. Fu, and J\. ZhangDeep guided attention network for joint denoising and demosaicing in real image\.Chinese Journal of Electronics33\(1\),pp\. 303–312\.Cited by:[NeurIPS Paper Checklist](https://arxiv.org/html/2610.22883#Ax2.p2.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics,pp\. 12834–12859\.Cited by:[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p1.1)\.
- Zhouet al\.\(2022\)K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. LiuLearning to prompt for vision\-language models\.Int\. J\. Comput\. Vis\.130\(9\),pp\. 2337–2348\.Cited by:[§1](https://arxiv.org/html/2610.22883#S1.p1.1),[§6\.2](https://arxiv.org/html/2610.22883#S6.SS2.p1.1)\.
- Zhuet al\.\(2025\)J\. Zhu, J\. Ai, Y\. Li, Y\. Yin, Y\. Wang, L\. Shang, and Q\. LiuRidgeLoRA: matrix ridge enhanced low\-rank adaptation of large language models\.InAnnual Conference on Neural Information Processing Systems,Cited by:[5th item](https://arxiv.org/html/2610.22883#A5.I5.i5.p1.1),[§1](https://arxiv.org/html/2610.22883#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22883#S2.SS1.p3.1),[§5\.1](https://arxiv.org/html/2610.22883#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2610.22883#S6.SS1.p2.1)\.

## Appendix

### Contents

### Appendix AProof

#### A\.1Proof of Prop\.[3\.2](https://arxiv.org/html/2610.22883#S3.Thmtheorem2)

###### Proposition A\.1\(Restate of Prop\.[3\.2](https://arxiv.org/html/2610.22883#S3.Thmtheorem2)\)\.

Letℒ\\mathcal\{L\}denote the scalar loss, and define the gradient with respect to𝐖\\bm\{W\}as𝐆:=∇𝐖ℒ∈ℝm×n\\bm\{G\}:=\\nabla\_\{\\bm\{W\}\}\\mathcal\{L\}\\in\\mathbb\{R\}^\{m\\times n\}\. Then, for eachi∈\{1,…,t\}i\\in\\\{1,\\dots,t\\\}, the gradient with respect to𝐓i\\bm\{T\}\_\{i\}is given by:

∇𝑻iℒ=\(∏k=1i−1𝑻k\)⊤​𝑾0⊤​𝑮​\(∏k=i\+1t𝑻k\)⊤\.\\nabla\_\{\\bm\{T\}\_\{i\}\}\\mathcal\{L\}=\\big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\big\)^\{\\top\}\\bm\{W\}\_\{0\}^\{\\top\}\\bm\{G\}\\big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\big\)^\{\\top\}\.\(11\)

###### Proof\.

Fix an indexi∈\{1,…,t\}i\\in\\\{1,\\dots,t\\\}\. Letd​𝑻i\\mathrm\{d\}\\bm\{T\}\_\{i\}denote the differential of𝑻i\\bm\{T\}\_\{i\}\. Then the induced differential of𝑾\\bm\{W\}can be expressed as:

d​𝑾=𝑾0​\(∏k=1i−1𝑻k\)​\(d​𝑻i\)​\(∏k=i\+1t𝑻k\)\.\\mathrm\{d\}\\bm\{W\}=\\bm\{W\}\_\{0\}\\big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\big\)\(\\mathrm\{d\}\\bm\{T\}\_\{i\}\)\\big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\big\)\.\(12\)Accordingly, the differential of the loss functionℒ\\mathcal\{L\}is given by:

d​ℒ=tr⁡\(𝑮⊤​d​𝑾\)=tr⁡\(𝑮⊤​𝑾0​\(∏k=1i−1𝑻k\)​\(d​𝑻i\)​\(∏k=i\+1t𝑻k\)\)\.\\mathrm\{d\}\\mathcal\{L\}=\\operatorname\{tr\}\\big\(\\bm\{G\}^\{\\top\}\\mathrm\{d\}\\bm\{W\}\\big\)=\\operatorname\{tr\}\\Big\(\\bm\{G\}^\{\\top\}\\bm\{W\}\_\{0\}\\big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\big\)\(\\mathrm\{d\}\\bm\{T\}\_\{i\}\)\\big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\big\)\\Big\)\.\(13\)By the cyclic property of the trace, we can rearrange the terms as:

d​ℒ=tr⁡\(\(∏k=i\+1t𝑻k\)​𝑮⊤​𝑾0​\(∏k=1i−1𝑻k\)​d​𝑻i\)\.\\mathrm\{d\}\\mathcal\{L\}=\\operatorname\{tr\}\\Big\(\\Big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\Big\)\\bm\{G\}^\{\\top\}\\bm\{W\}\_\{0\}\\Big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\Big\)\\mathrm\{d\}\\bm\{T\}\_\{i\}\\Big\)\.\(14\)Recalling thatdℒ=tr\(\(∇𝑻iℒ\)⊤d𝑻i\)\\mathrm\{d\}\\mathcal\{L\}=\\operatorname\{tr\}\\big\(\(\\nabla\{\\bm\{T\}\_\{i\}\}\\mathcal\{L\}\)^\{\\top\}\\mathrm\{d\}\\bm\{T\}\_\{i\}\\big\), we identify the gradient with respect to𝑻i\\bm\{T\}\_\{i\}as

∇𝑻iℒ=\(\(∏k=i\+1t𝑻k\)​𝑮⊤​𝑾0​\(∏k=1i−1𝑻k\)​d​𝑻i\)⊤=\(∏k=1i−1𝑻k\)⊤​𝑾0⊤​𝑮​\(∏k=i\+1t𝑻k\)⊤\.\\nabla\_\{\\bm\{T\}\_\{i\}\}\\mathcal\{L\}=\\Big\(\\big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\big\)\\bm\{G\}^\{\\top\}\\bm\{W\}\_\{0\}\\big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\big\)\\mathrm\{d\}\\bm\{T\}\_\{i\}\\Big\)^\{\\top\}=\\big\(\\prod\_\{k=1\}^\{i\-1\}\\bm\{T\}\_\{k\}\\big\)^\{\\top\}\\bm\{W\}\_\{0\}^\{\\top\}\\bm\{G\}\\big\(\\prod\_\{k=i\+1\}^\{t\}\\bm\{T\}\_\{k\}\\big\)^\{\\top\}\.\(15\)
∎

#### A\.2Proof of Prop\.[4\.1](https://arxiv.org/html/2610.22883#S4.Thmtheorem1)

###### Proposition A\.2\(Restate of Prop\.[4\.1](https://arxiv.org/html/2610.22883#S4.Thmtheorem1)\)\.

Let𝐖0∈ℝm×n\\bm\{W\}\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}satisfyrank⁡\(𝐖0\)=s\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\)=s\. Let\{𝐓blki\}i=1t\\\{\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\\}\_\{i=1\}^\{t\}be a sequence ofn×nn\\times nblock\-wise Toeplitz matrices and define the update

Δ​𝑾=𝑾0​\(∏i=1t𝑻blki\)\.\\Delta\\bm\{W\}\\;=\\;\\bm\{W\}\_\{0\}\\Big\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\Big\)\.\(16\)Thenrank⁡\(Δ​𝐖\)=s\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=salmost surely, i\.e\.,

ℙ⁡\(rank⁡\(Δ​𝑾\)=s\)=1\.\\mathbb\{P\}\\\!\\left\(\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=s\\right\)=1\.\(17\)

###### Proof\.

We first derive an upper bound onrank⁡\(Δ​𝑾\)\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)that holds for all admissible choices of\{𝑻blki\}i=1t\\\{\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\\}\_\{i=1\}^\{t\}\. We then show that this bound is achieved with probability11by noting that the set of singular block\-wise Toeplitz matrices has Lebesgue measure zero, and hence the product∏i=1t𝑻blki\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}is invertible with probability11\.

Upper bound\.For any block\-wise Toeplitz matrices𝑻blki\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}, we have

Δ​𝑾=𝑾0​\(∏i=1t𝑻blki\)\.\\Delta\\bm\{W\}=\\bm\{W\}\_\{0\}\\Big\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\Big\)\.\(18\)Since right multiplication cannot increase matrix rank, it follows that

rank⁡\(Δ​𝑾\)=rank⁡\(𝑾0​∏i=1t𝑻blki\)≤min⁡\(rank⁡\(𝑾0\),rank⁡\(∏i=1t𝑻blki\)\)≤s\.\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=\\operatorname\{rank\}\\\!\\left\(\\bm\{W\}\_\{0\}\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\right\)\\leq\\min\\Big\(\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\),\\ \\operatorname\{rank\}\\\!\\Big\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\Big\)\\Big\)\\leq s\.\(19\)
Almost\-sure equality\.Let𝑷≔∏i=1t𝑻blki∈ℝn×n\\bm\{P\}\\coloneqq\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\in\\mathbb\{R\}^\{n\\times n\}\. If𝑷\\bm\{P\}is invertible, then right multiplication by𝑷\\bm\{P\}preserves rank, and hence

rank⁡\(Δ​𝑾\)=rank⁡\(𝑾0​𝑷\)=rank⁡\(𝑾0\)=s\.\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\\bm\{P\}\)=\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\)=s\.\(20\)Therefore, it suffices to show that𝑷\\bm\{P\}is invertible with probability11\.

Each block\-wise Toeplitz matrix𝑻blki\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}is determined by finitely many parameters\. Denote its parameter vector by𝜽i\\bm\{\\theta\}\_\{i\}, and write𝑻blk​\(𝜽i\)\\bm\{T\}\_\{\\mathrm\{blk\}\}\(\\bm\{\\theta\}\_\{i\}\)for the corresponding block\-wise Toeplitz matrix\. Define

f⁡\(𝜽1,…,𝜽t\)≔det\(∏i=1t𝑻blk​\(𝜽i\)\)\.f\(\\bm\{\\theta\}\_\{1\},\\ldots,\\bm\{\\theta\}\_\{t\}\)\\;\\coloneqq\\;\\det\\\!\\Big\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}\(\\bm\{\\theta\}\_\{i\}\)\\Big\)\.\(21\)Thenffis a polynomial in\(𝜽1,…,𝜽t\)\(\\bm\{\\theta\}\_\{1\},\\ldots,\\bm\{\\theta\}\_\{t\}\)\. Moreover,ffis not identically zero: since𝑰n\\bm\{I\}\_\{n\}is itself a block\-wise Toeplitz matrix, choosing𝑻blki=𝑰n\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}=\\bm\{I\}\_\{n\}for alliiyields𝑷=𝑰n\\bm\{P\}=\\bm\{I\}\_\{n\}and hencef=1≠0f=1\\neq 0\. Consequently, the degenerate set

𝒟≔\{\(𝜽1,…,𝜽t\):f⁡\(𝜽1,…,𝜽t\)=0\}\\mathcal\{D\}\\;\\coloneqq\\;\\\{\(\\bm\{\\theta\}\_\{1\},\\ldots,\\bm\{\\theta\}\_\{t\}\):f\(\\bm\{\\theta\}\_\{1\},\\ldots,\\bm\{\\theta\}\_\{t\}\)=0\\\}\(22\)is a proper algebraic subset of the block\-wise Toeplitz parameter space\[[13](https://arxiv.org/html/2610.22883#bib.bib61)\], and in particular has Lebesgue measure00\[[6](https://arxiv.org/html/2610.22883#bib.bib39)\]\. Equivalently, the set of singular block\-wise Toeplitz products𝑷\\bm\{P\}has measure00, and thus𝑷\\bm\{P\}is invertible with probability11\.

Hence, with probability11,𝑷\\bm\{P\}is invertible, and Eq\. \([20](https://arxiv.org/html/2610.22883#A1.E20)\) impliesrank⁡\(Δ​𝑾\)=s\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=s\. Combining this with the upper bound in Eq\. \([19](https://arxiv.org/html/2610.22883#A1.E19)\) completes the proof\.

∎

#### A\.3Proof of Thm\.[4\.2](https://arxiv.org/html/2610.22883#S4.Thmtheorem2)

###### Theorem A\.3\(Restatement of Thm\.[4\.2](https://arxiv.org/html/2610.22883#S4.Thmtheorem2)\)\.

Letm,p∈ℕm,p\\in\\mathbb\{N\}and denoten:=m​pn:=mp\. Consider thep×pp\\times pblock partition ofℂn×n\\mathbb\{C\}^\{n\\times n\}in which each block has sizem×mm\\times m\. LetToepm​\(ℂ\)⊂ℂm×m\\mathrm\{Toep\}\_\{m\}\(\\mathbb\{C\}\)\\subset\\mathbb\{C\}^\{m\\times m\}denote the linear space ofm×mm\\times mToeplitz matrices, and define the set of block\-wise Toeplitz matrices as:

𝒯ℬm,p:=\{Tblk=\[\(Tblk\)i​j\]i,j=1p∈ℂn×n\|\(Tblk\)i​j∈Toepm\(ℂ\)for alli,j\}\.\\mathcal\{TB\}\_\{m,p\}:=\\Bigl\\\{T\_\{\\mathrm\{blk\}\}=\[\(T\_\{\\mathrm\{blk\}\}\)\_\{ij\}\]\_\{i,j=1\}^\{p\}\\in\\mathbb\{C\}^\{n\\times n\}\\ \\big\|\\ \(T\_\{\\mathrm\{blk\}\}\)\_\{ij\}\\in\\mathrm\{Toep\}\_\{m\}\(\\mathbb\{C\}\)\\ \\text\{for all \}i,j\\Bigr\\\}\.\(23\)For any integerr≥1r\\geq 1, define the polynomial multiplication map:

ρrTB:𝒯ℬm,pr→ℂn×n,ρrTB\(Tblk\(1\),…,Tblk\(r\)\)=Tblk\(1\)⋯Tblk\(r\)\.\\rho\_\{r\}^\{\\mathrm\{TB\}\}:\\mathcal\{TB\}\_\{m,p\}^\{r\}\\to\\mathbb\{C\}^\{n\\times n\},\\qquad\\rho\_\{r\}^\{\\mathrm\{TB\}\}\(T\_\{\\mathrm\{blk\}\}^\{\(1\)\},\\dots,T\_\{\\mathrm\{blk\}\}^\{\(r\)\}\)=T\_\{\\mathrm\{blk\}\}^\{\(1\)\}\\cdots T\_\{\\mathrm\{blk\}\}^\{\(r\)\}\.\(24\)Let

r0:=⌊m2⌋\+1\.r\_\{0\}:=\\Big\\lfloor\\frac\{m\}\{2\}\\Big\\rfloor\+1\.\(25\)Then, the following statements hold\.

1. \(i\)\(Generic factorization\.\) The mapρr0TB\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}is dominant\. Equivalently, there exists a Zariski\-open dense set𝒰⊂ℂn×n\\mathcal\{U\}\\subset\\mathbb\{C\}^\{n\\times n\}such that every matrixA∈𝒰A\\in\\mathcal\{U\}admits a factorization A=Tblk\(1\)⋯Tblk\(r0\),Tblk\(i\)∈𝒯ℬm,p\.A=T\_\{\\mathrm\{blk\}\}^\{\(1\)\}\\cdots T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\},\\qquad T\_\{\\mathrm\{blk\}\}^\{\(i\)\}\\in\\mathcal\{TB\}\_\{m,p\}\.\(26\)
2. \(ii\)\(Generic minimality\.\) Ifs<r0s<r\_\{0\}, thenρsTB\\rho\_\{s\}^\{\\mathrm\{TB\}\}is not dominant\. Consequently, a generic matrixA∈ℂn×nA\\in\\mathbb\{C\}^\{n\\times n\}cannot be expressed as a product ofssblock\-wise Toeplitz matrices\.
3. \(iii\)\(Worst\-case invertible matrices\.\) Every invertible matrixA∈GLn​\(ℂ\)A\\in\\mathrm\{GL\}\_\{n\}\(\\mathbb\{C\}\)admits a factorization into a product of at most2​r02r\_\{0\}block\-wise Toeplitz matrices\.
4. \(iv\)\(Worst\-case arbitrary matrices\.\) Every matrixA∈ℂn×nA\\in\\mathbb\{C\}^\{n\\times n\}admits a factorization into a product of at most4​r0\+14r\_\{0\}\+1block\-wise Toeplitz matrices\.

###### Proof\.

We structure the proof into multiple parts, corresponding to claims \(i\) through \(iv\)\.

A linear\-algebraic model for𝒯​ℬm,p\\mathcal\{TB\}\_\{m,p\}\.We begin by establishing a tensor product representation of the space𝒯​ℬm,p\\mathcal\{TB\}\_\{m,p\}\. LetU:=ℂp×pU:=\\mathbb\{C\}^\{p\\times p\}andV:=Toepm​\(ℂ\)V:=\\mathrm\{Toep\}\_\{m\}\(\\mathbb\{C\}\)\. Consider the map

Φ:U⊗V→ℂn×n,Φ⁡\(A⊗T\):=A⊗T,\\Phi:U\\otimes V\\to\\mathbb\{C\}^\{n\\times n\},\\qquad\\Phi\(A\\otimes T\):=A\\otimes T,\(27\)whereA⊗TA\\otimes Tis viewed as ap×pp\\times pblock matrix whose\(i,j\)\(i,j\)\-block is given byAi​j​TA\_\{ij\}T\. We claim thatΦ\\Phirestricts to a linear isomorphism fromU⊗VU\\otimes Vonto𝒯​ℬm,p\\mathcal\{TB\}\_\{m,p\}\.

To establish surjectivity, consider an arbitrary elementTblk=\[\(Tblk\)i​j\]∈𝒯​ℬm,pT\_\{\\mathrm\{blk\}\}=\[\(T\_\{\\mathrm\{blk\}\}\)\_\{ij\}\]\\in\\mathcal\{TB\}\_\{m,p\}\. For each\(i,j\)\(i,j\), since block\(Tblk\)i​j\(T\_\{\\mathrm\{blk\}\}\)\_\{ij\}lies inVV, we may expand it relative to a fixed basis\{Bk\}k=12​m−1\\\{B\_\{k\}\\\}\_\{k=1\}^\{2m\-1\}ofVV:

\(Tblk\)i​j=∑k=12​m−1ci​j,k​Bk\.\(T\_\{\\mathrm\{blk\}\}\)\_\{ij\}=\\sum\_\{k=1\}^\{2m\-1\}c\_\{ij,k\}B\_\{k\}\.\(28\)Letting\{Ei​j\}\\\{E\_\{ij\}\\\}denote the standard basis ofUU, we can rewrite the global matrix as:

Tblk=∑i,j=1p∑k=12​m−1ci​j,k​\(Ei​j⊗Bk\)∈Im⁡\(Φ\),T\_\{\\mathrm\{blk\}\}=\\sum\_\{i,j=1\}^\{p\}\\sum\_\{k=1\}^\{2m\-1\}c\_\{ij,k\}\\,\(E\_\{ij\}\\otimes B\_\{k\}\)\\in\\mathrm\{Im\}\(\\Phi\),\(29\)Thus,𝒯​ℬm,p⊆Im⁡\(Φ\)\\mathcal\{TB\}\_\{m,p\}\\subseteq\\mathrm\{Im\}\(\\Phi\)\. The reverse inclusion is immediate from the definition of𝒯​ℬm,p\\mathcal\{TB\}\_\{m,p\}, implyingIm⁡\(Φ\)=𝒯​ℬm,p\\mathrm\{Im\}\(\\Phi\)=\\mathcal\{TB\}\_\{m,p\}\. Injectivity follows from the fact that the basis elements\{Ei​j⊗Bk\}i,j,k\\\{E\_\{ij\}\\otimes B\_\{k\}\\\}\_\{i,j,k\}map to block matrices with disjoint supports and linearly independent within each block\. Therefore,Φ\\Phiis an isomorphism, and the dimension of the space is given by:

dim𝒯​ℬm,p=dim\(U⊗V\)=dim\(U\)​dim\(V\)=p2​\(2​m−1\)\.\\dim\\mathcal\{TB\}\_\{m,p\}=\\dim\(U\\otimes V\)=\\dim\(U\)\\dim\(V\)=p^\{2\}\(2m\-1\)\.\(30\)
Proof of \(i\): dominance via a full\-rank differential\.Consider the Toeplitz multiplication map

ρrtoep:Vr→ℂm×m,ρrtoep\(T1,…,Tr\)=T1⋯Tr\.\\rho\_\{r\}^\{\\mathrm\{toep\}\}:V^\{r\}\\to\\mathbb\{C\}^\{m\\times m\},\\qquad\\rho\_\{r\}^\{\\mathrm\{toep\}\}\(T\_\{1\},\\dots,T\_\{r\}\)=T\_\{1\}\\cdots T\_\{r\}\.\(31\)By the results from\[[61](https://arxiv.org/html/2610.22883#bib.bib4)\], forr=r0r=r\_\{0\}there existsσ=\(T1,…,Tr0\)∈Vr0\\sigma=\(T\_\{1\},\\dots,T\_\{r\_\{0\}\}\)\\in V^\{r\_\{0\}\}such that the differential\(d​ρr0toep\)σ:Vr0→ℂm×m\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{toep\}\}\)\_\{\\sigma\}:V^\{r\_\{0\}\}\\to\\mathbb\{C\}^\{m\\times m\}is surjective\.

We lift this configuration to the block setting\. Define the factors

Tblk\(i\):=Ip⊗Ti∈𝒯​ℬm,p,τ:=\(Tblk\(1\),…,Tblk\(r0\)\)∈𝒯​ℬm,pr0\.T\_\{\\mathrm\{blk\}\}^\{\(i\)\}:=I\_\{p\}\\otimes T\_\{i\}\\in\\mathcal\{TB\}\_\{m,p\},\\qquad\\tau:=\(T\_\{\\mathrm\{blk\}\}^\{\(1\)\},\\dots,T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\}\)\\in\\mathcal\{TB\}\_\{m,p\}^\{\\,r\_\{0\}\}\.\(32\)For any perturbation tuple\(Δ​Tblk\(1\),…,Δ​Tblk\(r0\)\)∈𝒯​ℬm,pr0\(\\Delta T\_\{\\mathrm\{blk\}\}^\{\(1\)\},\\dots,\\Delta T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\}\)\\in\\mathcal\{TB\}\_\{m,p\}^\{r\_\{0\}\}, the product rule yields

\(dρr0TB\)τ\(ΔTblk\(1\),…,ΔTblk\(r0\)\)=∑i=1r0Tblk\(1\)⋯Tblk\(i−1\)ΔTblk\(i\)Tblk\(i\+1\)⋯Tblk\(r0\)\.\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}\)\_\{\\tau\}\(\\Delta T\_\{\\mathrm\{blk\}\}^\{\(1\)\},\\dots,\\Delta T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\}\)=\\sum\_\{i=1\}^\{r\_\{0\}\}T\_\{\\mathrm\{blk\}\}^\{\(1\)\}\\cdots T\_\{\\mathrm\{blk\}\}^\{\(i\-1\)\}\\,\\Delta T\_\{\\mathrm\{blk\}\}^\{\(i\)\}\\,T\_\{\\mathrm\{blk\}\}^\{\(i\+1\)\}\\cdots T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\}\.\(33\)Using the isomorphism𝒯​ℬm,p≅U⊗V\\mathcal\{TB\}\_\{m,p\}\\cong U\\otimes V, we decompose the perturbations as:

Δ​Tblk\(i\)=∑α=1p2Eα⊗Δ​Ti\(α\),Δ​Ti\(α\)∈V,\\Delta T\_\{\\mathrm\{blk\}\}^\{\(i\)\}=\\sum\_\{\\alpha=1\}^\{p^\{2\}\}E\_\{\\alpha\}\\otimes\\Delta T\_\{i\}^\{\(\\alpha\)\},\\qquad\\Delta T\_\{i\}^\{\(\\alpha\)\}\\in V,\(34\)where\{Eα\}α=1p2\\\{E\_\{\\alpha\}\\\}\_\{\\alpha=1\}^\{p^\{2\}\}is a basis ofUU\. Observing the identity\(Ip⊗A\)​\(Eα⊗B\)​\(Ip⊗C\)=Eα⊗\(A​B​C\)\(I\_\{p\}\\otimes A\)\(E\_\{\\alpha\}\\otimes B\)\(I\_\{p\}\\otimes C\)=E\_\{\\alpha\}\\otimes\(ABC\), we substituteTblk\(j\)=Ip⊗TjT\_\{\\mathrm\{blk\}\}^\{\(j\)\}=I\_\{p\}\\otimes T\_\{j\}into Eq\. \([33](https://arxiv.org/html/2610.22883#A1.E33)\) to obtain the tensorized differential form:

\(d​ρr0TB\)τ​\(Δ​Tblk\(1\),…,Δ​Tblk\(r0\)\)=∑α=1p2Eα⊗\(d​ρr0toep\)σ​\(Δ​T1\(α\),…,Δ​Tr0\(α\)\)\.\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}\)\_\{\\tau\}\(\\Delta T\_\{\\mathrm\{blk\}\}^\{\(1\)\},\\dots,\\Delta T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\}\)=\\sum\_\{\\alpha=1\}^\{p^\{2\}\}E\_\{\\alpha\}\\otimes\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{toep\}\}\)\_\{\\sigma\}\\bigl\(\\Delta T\_\{1\}^\{\(\\alpha\)\},\\dots,\\Delta T\_\{r\_\{0\}\}^\{\(\\alpha\)\}\\bigr\)\.\(35\)We now verify the surjectivity of\(d​ρr0TB\)τ\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}\)\_\{\\tau\}\. LetY∈ℂn×nY\\in\\mathbb\{C\}^\{n\\times n\}be an arbitrary target matrix\. Under the identificationℂn×n≅U⊗ℂm×m\\mathbb\{C\}^\{n\\times n\}\\cong U\\otimes\\mathbb\{C\}^\{m\\times m\}, we may uniquely writeY=∑α=1p2Eα⊗XαY=\\sum\_\{\\alpha=1\}^\{p^\{2\}\}E\_\{\\alpha\}\\otimes X\_\{\\alpha\}, withXα∈ℂm×mX\_\{\\alpha\}\\in\\mathbb\{C\}^\{m\\times m\}\. By the surjectivity of\(d​ρr0toep\)σ\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{toep\}\}\)\_\{\\sigma\}, for each indexα\\alpha, there exists a tuple\(Δ​T1\(α\),…,Δ​Tr0\(α\)\)∈Vr0\(\\Delta T\_\{1\}^\{\(\\alpha\)\},\\dots,\\Delta T\_\{r\_\{0\}\}^\{\(\\alpha\)\}\)\\in V^\{r\_\{0\}\}satisfying\(d​ρr0toep\)σ​\(Δ​T1\(α\),…,Δ​Tr0\(α\)\)=Xα\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{toep\}\}\)\_\{\\sigma\}\(\\Delta T\_\{1\}^\{\(\\alpha\)\},\\dots,\\Delta T\_\{r\_\{0\}\}^\{\(\\alpha\)\}\)=X\_\{\\alpha\}\. Constructing the global perturbationΔ​Tblk\(i\):=∑α=1p2Eα⊗Δ​Ti\(α\)\\Delta T\_\{\\mathrm\{blk\}\}^\{\(i\)\}:=\\sum\_\{\\alpha=1\}^\{p^\{2\}\}E\_\{\\alpha\}\\otimes\\Delta T\_\{i\}^\{\(\\alpha\)\}and substituting it into Eq\. \([35](https://arxiv.org/html/2610.22883#A1.E35)\) yields\(d​ρr0TB\)τ​\(Δ​Tblk\(1\),…,Δ​Tblk\(r0\)\)=Y\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}\)\_\{\\tau\}\(\\Delta T\_\{\\mathrm\{blk\}\}^\{\(1\)\},\\dots,\\Delta T\_\{\\mathrm\{blk\}\}^\{\(r\_\{0\}\)\}\)=Y\. Hence\(d​ρr0TB\)τ\(d\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}\)\_\{\\tau\}is surjective ontoℂn×n\\mathbb\{C\}^\{n\\times n\}\.

Since𝒯​ℬm,pr0\\mathcal\{TB\}\_\{m,p\}^\{r\_\{0\}\}is irreducible andρr0TB\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}is a polynomial map, the existence of a pointτ\\tauwith surjective differential implies thatρr0TB\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}is dominant\[[61](https://arxiv.org/html/2610.22883#bib.bib4)\]\. This proves \(i\)\.

Proof of \(ii\): generic minimality by a dimension obstruction\.IfρsTB\\rho\_\{s\}^\{\\mathrm\{TB\}\}were dominant for somess\. This requires the dimension of the domain to be at least the dimension of the codomain:

dim\(𝒯​ℬm,ps\)≥dim\(ℂn×n\)=n2\.\\dim\(\\mathcal\{TB\}\_\{m,p\}^\{s\}\)\\geq\\dim\(\\mathbb\{C\}^\{n\\times n\}\)=n^\{2\}\.\(36\)Substituting Eq\. \([30](https://arxiv.org/html/2610.22883#A1.E30)\) and recallingn=m​pn=mp, we obtain the inequality:

sp2\(2m−1\)≥m2p2⟺s≥m22​m−1\.s\\,p^\{2\}\(2m\-1\)\\geq m^\{2\}p^\{2\}\\quad\\Longleftrightarrow\\quad s\\geq\\frac\{m^\{2\}\}\{2m\-1\}\.\(37\)Direct calculation shows thatm22​m−1\>m2\\frac\{m^\{2\}\}\{2m\-1\}\>\\frac\{m\}\{2\}\. Sincessmust be an integer, the minimal possible value iss=⌈m22​m−1⌉=⌊m2⌋\+1=r0s=\\left\\lceil\\frac\{m^\{2\}\}\{2m\-1\}\\right\\rceil=\\left\\lfloor\\frac\{m\}\{2\}\\right\\rfloor\+1=r\_\{0\}\. Therefore, no integers<r0s<r\_\{0\}can yield dominance, proving \(ii\)\.

Auxiliary lemma\.LetG:=GLn​\(ℂ\)G:=\\mathrm\{GL\}\_\{n\}\(\\mathbb\{C\}\)\. We recall a standard result for affine algebraic groups\.

###### Lemma A\.4\.

\[[61](https://arxiv.org/html/2610.22883#bib.bib4)\]LetGGbe an algebraic group andU,V⊆GU,V\\subseteq Gbe two Zariski\-open and Zariski\-dense subsets\. Then,U​V=GUV=G\.

Proof of \(iii\): worst\-case factorization overGLn​\(ℂ\)\\mathrm\{GL\}\_\{n\}\(\\mathbb\{C\}\)\.From \(i\), the image ofρr0TB\\rho\_\{r\_\{0\}\}^\{\\mathrm\{TB\}\}contains a Zariski\-open dense subset𝒰⊂ℂn×n\\mathcal\{U\}\\subset\\mathbb\{C\}^\{n\\times n\}\. Let𝒰G:=𝒰∩G\\mathcal\{U\}\_\{G\}:=\\mathcal\{U\}\\cap G\. SinceGGis open and dense inℂn×n\\mathbb\{C\}^\{n\\times n\}, their intersection𝒰G\\mathcal\{U\}\_\{G\}remains Zariski\-open dense inGG\. Applying Lemma[A\.4](https://arxiv.org/html/2610.22883#A1.Thmtheorem4)withU=V=𝒰GU=V=\\mathcal\{U\}\_\{G\}yields𝒰G​𝒰G=G\\mathcal\{U\}\_\{G\}\\mathcal\{U\}\_\{G\}=G\. Thus, for anyA∈GA\\in G, there existX,Y∈𝒰GX,Y\\in\\mathcal\{U\}\_\{G\}such thatA=X​YA=XY\.

By the definition of𝒰\\mathcal\{U\}, each ofXXandYYcan be written as a product ofr0r\_\{0\}block\-wise Toeplitz matrices:X=∏i=1r0XiX=\\prod\_\{i=1\}^\{r\_\{0\}\}X\_\{i\}andY=∏i=1r0YiY=\\prod\_\{i=1\}^\{r\_\{0\}\}Y\_\{i\}withXi,Yi∈𝒯​ℬm,pX\_\{i\},Y\_\{i\}\\in\\mathcal\{TB\}\_\{m,p\}\. Consequently,

A=X​Y=\(∏i=1r0Xi\)​\(∏i=1r0Yi\)A=XY=\\Big\(\\prod\_\{i=1\}^\{r\_\{0\}\}X\_\{i\}\\Big\)\\Big\(\\prod\_\{i=1\}^\{r\_\{0\}\}Y\_\{i\}\\Big\)\(38\)is a product of2​r02r\_\{0\}block\-wise Toeplitz matrices, proving \(iii\)\.

Proof of \(iv\): reduction to a rank\-normal form\.LetA∈ℂn×nA\\in\\mathbb\{C\}^\{n\\times n\}be an arbitrary matrix withrank⁡\(A\)=s\\mathrm\{rank\}\(A\)=s\. By Gaussian elimination, there exist invertible matricesP0,Q0∈GP\_\{0\},Q\_\{0\}\\in Gsuch that

P0​A​Q0=\[Is000\]=:Js\.P\_\{0\}\\,A\\,Q\_\{0\}=\\begin\{bmatrix\}I\_\{s\}&0\\\\ 0&0\\end\{bmatrix\}=:J\_\{s\}\.\(39\)We observe thatJsJ\_\{s\}can be realized as a product of a permutation a specific Toeplitz shift\. LetBk:=\(δi\+k,j\)i,j=1nB\_\{k\}:=\(\\delta\_\{i\+k,j\}\)\_\{i,j=1\}^\{n\}be the shift matrix with ones on thekk\-th diagonal\. Note thatBkB\_\{k\}is a Toeplitz matrix\. Fork=n−sk=n\-s, there exists a permutation matrixΠ\\Pisuch that:

Js=Bn−s​Π−1\.J\_\{s\}=B\_\{n\-s\}\\Pi^\{\-1\}\.\(40\)Substituting this back into Eq\. \([39](https://arxiv.org/html/2610.22883#A1.E39)\), we have

A=P0−1​Js​Q0−1=P0−1​Bn−s​\(Π−1​Q0−1\)=:P​B​Q,A=P\_\{0\}^\{\-1\}J\_\{s\}Q\_\{0\}^\{\-1\}=P\_\{0\}^\{\-1\}B\_\{n\-s\}\(\\Pi^\{\-1\}Q\_\{0\}^\{\-1\}\)=:P\\,B\\,Q,\(41\)whereP=P0−1P=P\_\{0\}^\{\-1\}andQ=Π−1​Q0−1Q=\\Pi^\{\-1\}Q\_\{0\}^\{\-1\}belong toGG, andB:=Bn−sB:=B\_\{n\-s\}\.

A crucial observation is thatB∈𝒯​ℬm,pB\\in\\mathcal\{TB\}\_\{m,p\}\. This holds becauseBBis globally Toeplitz, and the restriction of any Toeplitz matrix to sub\-blocks formed by consecutive indices preserves the Toeplitz structure\.

Finally, applying result \(iii\), the invertible matricesPPandQQeach factor into2​r02r\_\{0\}block\-wise Toeplitz matrices\. Thus, the factorization

is a product of\(2​r0\)\+1\+\(2​r0\)=4​r0\+1\(2r\_\{0\}\)\+1\+\(2r\_\{0\}\)=4r\_\{0\}\+1block\-wise Toeplitz matrices, proving \(iv\)\. ∎

#### A\.4Proof of Thm\.[4\.3](https://arxiv.org/html/2610.22883#S4.Thmtheorem3)

###### Theorem A\.5\(Restate of Thm\.[4\.3](https://arxiv.org/html/2610.22883#S4.Thmtheorem3)\)\.

Let𝐖0∈ℝm×n\\bm\{W\}\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}be a pretrained weight matrix with full column rank, i\.e\.,rank⁡\(𝐖0\)=n\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\)=n\. Then there exists a class of adapted weight matrices𝐖\\bm\{W\}, that can be realized by ToPA with𝒪⁡\(n\)\\mathcal\{O\}\(n\)trainable parameters, while achieving the same𝐖\\bm\{W\}with LoRA requires at least𝒪⁡\(n2\)\\mathcal\{O\}\(n^\{2\}\)parameters and with OFT or BOFT requires at least𝒪⁡\(n​log⁡n\)\\mathcal\{O\}\(n\\log n\)parameters\.

###### Proof\.

\.

1\. Construction of the Adaptation via ToPA\.

Choose constantst=𝒪⁡\(1\)t=\\mathcal\{O\}\(1\)andp=𝒪⁡\(1\)p=\\mathcal\{O\}\(1\), both independent ofnn\. For eachk=1,…,tk=1,\\dots,t, define a block\-wise Toeplitz matrix as:

𝑻blk\(k\)=\[𝑻11\(k\)⋯𝑻1​p\(k\)⋱𝑻p​1\(k\)⋯𝑻p​p\(k\)\],\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}=\\begin\{bmatrix\}\\bm\{T\}\_\{11\}^\{\(k\)\}&\\cdots&\\bm\{T\}\_\{1p\}^\{\(k\)\}\\\\ \\vdots&\\ddots&\\vdots\\\\ \\bm\{T\}\_\{p1\}^\{\(k\)\}&\\cdots&\\bm\{T\}\_\{pp\}^\{\(k\)\}\\end\{bmatrix\},\(43\)where each sub\-block𝑻i​j\(k\)∈ℝnp×np\\bm\{T\}\_\{ij\}^\{\(k\)\}\\in\\mathbb\{R\}^\{\\frac\{n\}\{p\}\\times\\frac\{n\}\{p\}\}is a Toeplitz matrix parameterized by a vector𝒘i​j\(k\)∈ℝ2​n/p−1\\bm\{w\}\_\{ij\}^\{\(k\)\}\\in\\mathbb\{R\}^\{2n/p\-1\}\. We assume the entries of all parameter vectors are drawn independently from a continuous distribution\.

Define the adapter𝑹\\bm\{R\}as:

𝑹=∏k=1t𝑻blk\(k\)\+𝑰n,\\bm\{R\}=\\prod\_\{k=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}\+\\bm\{I\}\_\{n\},\(44\)and let the adapted weight:

𝑾=𝑾0​𝑹\.\\bm\{W\}=\\bm\{W\}\_\{0\}\\bm\{R\}\.\(45\)
The total number of trainable parameters in ToPA for this adaptation is

\#params\(ToPA\)=t⋅p2⋅\(2​np−1\)=t⁡\(2​p​n−p2\)=𝒪⁡\(n\),\\textbf\{\\\#\\text\{params\}\}\_\{\(\\text\{ToPA\}\)\}=t\\cdot p^\{2\}\\cdot\\left\(\\frac\{2n\}\{p\}\-1\\right\)=t\(2pn\-p^\{2\}\)=\\mathcal\{O\}\(n\),\(46\)given thatttandppare constants\.

2\. Properties of the Adaptation\.

Rank of the Update Matrix\.Let𝒘\(k\)∈ℝN\\bm\{w\}^\{\(k\)\}\\in\\mathbb\{R\}^\{N\}denote the concatenated vector of all parameters in𝑻blk\(k\)\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}, whereN=p2​\(2​n/p−1\)N=p^\{2\}\(2n/p\-1\)\. The determinantdet\(𝑻blk\(k\)\)\\det\(\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}\)constitutes a multivariate polynomialfk:ℝN→ℝf\_\{k\}:\\mathbb\{R\}^\{N\}\\to\\mathbb\{R\}with respect to𝒘\(k\)\\bm\{w\}^\{\(k\)\}\. Since the Toeplitz structure subsumes the identity matrix, there exists a parameter configuration𝒘0\(k\)\\bm\{w\}\_\{0\}^\{\(k\)\}yielding𝑻blk\(k\)=𝑰n\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}=\\bm\{I\}\_\{n\}, which impliesfk​\(𝒘0\(k\)\)=1f\_\{k\}\(\\bm\{w\}\_\{0\}^\{\(k\)\}\)=1\. Consequently,fkf\_\{k\}is not identically zero\.

Since the zero set of a nonzero polynomial has Lebesgue measure zero\[[6](https://arxiv.org/html/2610.22883#bib.bib39)\], it follows that under a continuous parameter distribution,

ℙ⁡\(𝑻blk\(k\)​is singular\)=0\.\\mathbb\{P\}\\big\(\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}\\text\{ is singular\}\\big\)=0\.\(47\)
Sincettis constant, all𝑻blk\(k\)\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}are invertible almost surely, and so is their product𝑻:=∏k=1t𝑻blk\(k\)\\bm\{T\}:=\\prod\_\{k=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}\. Consider the update matrixΔ​𝑾=𝑾−𝑾0=𝑾0​𝑻\\Delta\\bm\{W\}=\\bm\{W\}\-\\bm\{W\}\_\{0\}=\\bm\{W\}\_\{0\}\\bm\{T\}, we have the following equation holds almost surely:

rank⁡\(Δ​𝑾\)=rank⁡\(𝑾0​𝑻\)=rank⁡\(𝑾0\)=s,\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\\bm\{T\}\)=\\operatorname\{rank\}\(\\bm\{W\}\_\{0\}\)=s,\(48\)where the last equality uses the fact that right multiplication by an invertible matrix preserves rank\.

Density ofR\\bm\{R\}\.Each entry of𝑹\\bm\{R\}is a polynomial in the parameters\. For any fixed\(i,j\)\(i,j\), if this polynomial is not identically zero, its zero set has Lebesgue measure zero, and thusℙ⁡\(\(𝑹\)i​j=0\)=0\\mathbb\{P\}\(\(\\bm\{R\}\)\_\{ij\}=0\)=0\. Because𝑻\\bm\{T\}is invertible and generic, one can verify that for each\(i,j\)\(i,j\)the polynomial is indeed nonzero with probability11\. A union bound over the finitely many entries yields

ℙ⁡\(𝑹​is dense\)=1\.\\mathbb\{P\}\\big\(\\bm\{R\}\\text\{ is dense\}\\big\)=1\.\(49\)Consequently, the set of parameters yielding a dense and full\-rank𝑹\\bm\{R\}almost surely\.

Uniqueness ofR\\bm\{R\}\.Because𝑾0\\bm\{W\}\_\{0\}has full column rank, the linear map𝑿↦𝑾0​𝑿\\bm\{X\}\\mapsto\\bm\{W\}\_\{0\}\\bm\{X\}is injective onℝn×n\\mathbb\{R\}^\{n\\times n\}\. Therefore, for a given adapted weight𝑾\\bm\{W\}, there is a unique𝑹\\bm\{R\}satisfying𝑾=𝑾0​𝑹\\bm\{W\}=\\bm\{W\}\_\{0\}\\bm\{R\}\. Consequently, any method that produces the same𝑾\\bm\{W\}must effectively represent this specific𝑹\\bm\{R\}\.

3\. Parameter Lower Bounds for LoRA and OFT/BOFT\.

LoRA Requires𝒪⁡\(n2\)\\mathcal\{O\}\(n^\{2\}\)Parameters\.Suppose LoRA realizes the same adaptation, i\.e\.,𝑾=𝑾0\+𝑩​𝑨\\bm\{W\}=\\bm\{W\}\_\{0\}\+\\bm\{B\}\\bm\{A\}with𝑩∈ℝm×r\\bm\{B\}\\in\\mathbb\{R\}^\{m\\times r\},𝑨∈ℝr×n\\bm\{A\}\\in\\mathbb\{R\}^\{r\\times n\}\. ThenΔ​𝑾=𝑩​𝑨\\Delta\\bm\{W\}=\\bm\{B\}\\bm\{A\}, and becauserank⁡\(Δ​𝑾\)=s\\operatorname\{rank\}\(\\Delta\\bm\{W\}\)=s, we must haver≥sr\\geq s\. The number of trainable parameters isr⁡\(m\+n\)≥s⁡\(m\+n\)r\(m\+n\)\\geq s\(m\+n\)\. Withs=Θ⁡\(n\)s=\\Theta\(n\)andm=𝒪⁡\(n\)m=\\mathcal\{O\}\(n\), we have:

\#params\(LoRA\)≳n⁡\(m\+n\)=𝒪⁡\(n2\)\.\\textbf\{\\\#\\text\{params\}\}\_\{\(\\text\{LoRA\}\)\}\\gtrsim n\(m\+n\)=\\mathcal\{O\}\(n^\{2\}\)\.\(50\)
OFT/BOFT RequireΩ⁡\(n​log⁡n\)\\Omega\(n\\log n\)Parameters\.Both OFT and BOFT construct the adapter as a product of sparse matrices𝑩i\\bm\{B\}\_\{i\}with butterfly structures\. Since the adapter𝑹\\bm\{R\}is unique, realizing the same adaptation with OFT or BOFT requires

∏i=1m𝑩i=𝑹\.\\prod\_\{i=1\}^\{m\}\\bm\{B\}\_\{i\}=\\bm\{R\}\.\(51\)Note that𝑹\\bm\{R\}is a dense matrix\.

For OFT, the multiplication chain consists of only one factor, i\.e\.,m=1m=1\. Therefore,𝑩\\bm\{B\}itself must be dense, which leads to

\#params\(OFT\)≳𝒪⁡\(n2\)\.\\textbf\{\\\#\\text\{params\}\}\_\{\(\\text\{OFT\}\)\}\\gtrsim\\mathcal\{O\}\(n^\{2\}\)\.\(52\)
For BOFT, a fundamental result in butterfly factorization theory states that representing a dense matrix with such a factorization requires at leastΩ⁡\(n​log⁡n\)\\Omega\(n\\log n\)parameters\[[31](https://arxiv.org/html/2610.22883#bib.bib5)\]\. Consequently, to represent the same adaptation with BOFT, the required number of parameters satisfies

\#params\(BOFT\)≳𝒪⁡\(n​log⁡n\)\.\\textbf\{\\\#\\text\{params\}\}\_\{\(\\text\{BOFT\}\)\}\\gtrsim\\mathcal\{O\}\(n\\log n\)\.\(53\)
∎

### Appendix BDetailed Introduction of OFT and BOFT

Both OFT and BOFT adapt a pretrained weight matrix through a multiplicative orthogonal update:

𝑾=𝑾0​𝑹=𝑾0​\(∏i=1m𝑩i\),𝑩i∈ℝn×n,𝑩i⊤​𝑩i=𝑰\.\\bm\{W\}=\\bm\{W\}\_\{0\}\\bm\{R\}=\\bm\{W\}\_\{0\}\\Big\(\\prod\_\{i=1\}^\{m\}\\bm\{B\}\_\{i\}\\Big\),\\qquad\\bm\{B\}\_\{i\}\\in\\mathbb\{R\}^\{n\\times n\},\\ \\ \\bm\{B\}\_\{i\}^\{\\top\}\\bm\{B\}\_\{i\}=\\bm\{I\}\.\(54\)OFT\.OFT uses a single \(m=1m=1\) structured sparse orthogonal matrix with a block\-diagonal form:

𝑹=𝑩1=diag⁡\(𝑹1,𝑹2,…,𝑹r\),𝑹j∈ℝb×b,\\bm\{R\}=\\bm\{B\}\_\{1\}=\\mathrm\{diag\}\(\\bm\{R\}\_\{1\},\\bm\{R\}\_\{2\},\\dots,\\bm\{R\}\_\{r\}\),\\qquad\\bm\{R\}\_\{j\}\\in\\mathbb\{R\}^\{b\\times b\},\(55\)wherer​b=nrb=nand each block is constrained to be orthogonal:

𝑹j⊤​𝑹j=𝑰b\.\\bm\{R\}\_\{j\}^\{\\top\}\\bm\{R\}\_\{j\}=\\bm\{I\}\_\{b\}\.\(56\)The resulting transformation matrix explicitly manifests as:

𝑩1=\[𝑹10⋯00𝑹2⋯0⋱00⋯𝑹r\]\.\\bm\{B\}\_\{1\}=\\begin\{bmatrix\}\\bm\{R\}\_\{1\}&0&\\cdots&0\\\\ 0&\\bm\{R\}\_\{2\}&\\cdots&0\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ 0&0&\\cdots&\\bm\{R\}\_\{r\}\\end\{bmatrix\}\.\(57\)
BOFT\.BOFT generalizes OFT by parameterizing the orthogonal matrix𝑹\\bm\{R\}as a product ofmmsparse orthogonal factors:

𝑹=∏i=1m𝑩i,\\bm\{R\}=\\prod\_\{i=1\}^\{m\}\\bm\{B\}\_\{i\},\(58\)Here, each𝑩i\\bm\{B\}\_\{i\}is structured as a block\-diagonal matrix, exclusively constituted by multiple butterfly factors𝑭⁡\(ki\)∈ℝki×ki\\bm\{F\}\(k\_\{i\}\)\\in\\mathbb\{R\}^\{k\_\{i\}\\times k\_\{i\}\}:

𝑩i=diag⁡\(𝑭1​\(ki\),…,𝑭n/ki​\(ki\)\),ki=b⋅2i\.\\bm\{B\}\_\{i\}=\\mathrm\{diag\}\\big\(\\bm\{F\}\_\{1\}\(k\_\{i\}\),\\dots,\\bm\{F\}\_\{n/k\_\{i\}\}\(k\_\{i\}\)\\big\),\\quad k\_\{i\}=b\\cdot 2^\{i\}\.\(59\)Specifically, each𝑭⁡\(ki\)\\bm\{F\}\(k\_\{i\}\)is defined as a structured sparse matrix populated by learnableb×bb\\times bsub\-blocks:

𝑭⁡\(ki\)=\[diag⁡\(𝑬1\(1\),…,𝑬p\(1\)\)diag⁡\(𝑬1\(2\),…,𝑬p\(2\)\)diag⁡\(𝑬1\(3\),…,𝑬p\(3\)\)diag⁡\(𝑬1\(4\),…,𝑬p\(4\)\)\],\\bm\{F\}\(k\_\{i\}\)=\\begin\{bmatrix\}\\mathrm\{diag\}\(\\bm\{E\}\_\{1\}^\{\(1\)\},\\dots,\\bm\{E\}\_\{p\}^\{\(1\)\}\)&\\mathrm\{diag\}\(\\bm\{E\}\_\{1\}^\{\(2\)\},\\dots,\\bm\{E\}\_\{p\}^\{\(2\)\}\)\\\\ \\mathrm\{diag\}\(\\bm\{E\}\_\{1\}^\{\(3\)\},\\dots,\\bm\{E\}\_\{p\}^\{\(3\)\}\)&\\mathrm\{diag\}\(\\bm\{E\}\_\{1\}^\{\(4\)\},\\dots,\\bm\{E\}\_\{p\}^\{\(4\)\}\)\\end\{bmatrix\},\(60\)wherep=ki/\(2​b\)p=k\_\{i\}/\(2b\), and𝑬\(j\)∈ℝb×b\\bm\{E\}^\{\(j\)\}\\in\\mathbb\{R\}^\{b\\times b\}represents the learnable block entry\.

### Appendix CAnalysis of Parameter and Computation Cost

#### C\.1Parameter Cost

For ToPA, the fine\-tuning process is formulated as

𝑾=𝑾0⋅\(∏i=1t𝑻blki\+𝑰\),\\bm\{W\}=\\bm\{W\}\_\{0\}\\cdot\\left\(\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\+\\bm\{I\}\\right\),\(61\)where each𝑻blki\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}denotes a block\-wise Toeplitz matrix in the transformation chain\. For a block\-wise Toeplitz matrix containingp2p^\{2\}sub\-blocks, the total number of learnable parameters isp⁡\(2​n−p\)p\(2n\-p\)\. Consequently, a chain of lengthttintroducest​p​\(2​n−p\)tp\(2n\-p\)parameters in total\.

For comparison, LoRA employs a pair of rank\-rrmatrices, resulting inr⁡\(m\+n\)r\(m\+n\)parameters\. In many practical settings, the input and output dimensions are of similar scale, i\.e\.,m≈nm\\approx n\. Under this common assumption, whent​p≈rtp\\approx r, the parameter count of ToPA becomes broadly comparable to that of LoRA, while still enabling richer and more expressive updates to the underlying weight matrix\.

#### C\.2Computation Cost

In the forward computation, we consider the form𝒚=𝑾​𝒙\\bm\{y\}=\\bm\{Wx\}, where𝒙∈ℝn\\bm\{x\}\\in\\mathbb\{R\}^\{n\}is an input vector and𝑾\\bm\{W\}denotes the adapted weight matrix after fine\-tuning\. For ease of exposition, we assume that the pretrained weight matrix is square with sizen×nn\\times n\. Under ToPA, the adaptation is applied multiplicatively to the pretrained weight matrix𝑾0\\bm\{W\}\_\{0\}, and the resulting parameter is expressed as

𝑾=𝑾0​\(∏k=1t𝑻blk\(k\)\+𝑰\),\\bm\{W\}=\\bm\{W\}\_\{0\}\\left\(\\prod\_\{k=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}\+\\bm\{I\}\\right\),\(62\)where each𝑻blk\(k\)\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{\(k\)\}is a block\-wise Toeplitz matrix and𝑰\\bm\{I\}denotes the identity matrix\. To analyze the computational cost, we consider the intermediate computation

𝒛=∏i=1t𝑻blki​𝒙\.\\bm\{z\}=\\prod\_\{i=1\}^\{t\}\\bm\{T\}\_\{\\mathrm\{blk\}\}^\{i\}\\bm\{x\}\.\(63\)
We partition the input vector intoppequal\-length sub\-vectors, i\.e\.,

𝒙=\[𝒙1⊤⋯𝒙p⊤\]⊤,𝒙j∈ℝq,n=q​p,\\bm\{x\}=\\begin\{bmatrix\}\\bm\{x\}\_\{1\}^\{\\top\}&\\cdots&\\bm\{x\}\_\{p\}^\{\\top\}\\end\{bmatrix\}^\{\\top\},\\quad\\bm\{x\}\_\{j\}\\in\\mathbb\{R\}^\{q\},\\quad n=qp,\(64\)and each block\-wise Toeplitz matrix is ap×pp\\times pblock matrix

𝑻blk=\[𝑻11⋯𝑻1​p⋱𝑻p​1⋯𝑻p​p\],𝑻i​j∈ℝq×q\.\\bm\{T\}\_\{\\mathrm\{blk\}\}=\\begin\{bmatrix\}\\bm\{T\}\_\{11\}&\\cdots&\\bm\{T\}\_\{1p\}\\\\ \\vdots&\\ddots&\\vdots\\\\ \\bm\{T\}\_\{p1\}&\\cdots&\\bm\{T\}\_\{pp\}\\end\{bmatrix\},\\quad\\bm\{T\}\_\{ij\}\\in\\mathbb\{R\}^\{q\\times q\}\.\(65\)where every block𝑻i​j\\bm\{T\}\_\{ij\}is a Toeplitz matrix\. The matrix–vector multiplication𝑻blk​𝒙\\bm\{T\}\_\{\\mathrm\{blk\}\}\\bm\{x\}is then carried out in a block\-wise way:

\(𝑻blk𝒙\)i=∑j=1p𝑻i​j𝒙j,i=1,…,p,\(\\bm\{T\}\_\{\\mathrm\{blk\}\}\\bm\{x\}\)\_\{i\}=\\sum\_\{j=1\}^\{p\}\\bm\{T\}\_\{ij\}\\bm\{x\}\_\{j\},\\qquad i=1,\\dots,p,\(66\)so that the output vector is likewise obtained as

𝑻blk​𝒙=\[∑j=1p𝑻1​j​𝒙j∑j=1p𝑻p​j​𝒙j\]\.\\bm\{T\}\_\{\\mathrm\{blk\}\}\\bm\{x\}=\\begin\{bmatrix\}\\sum\_\{j=1\}^\{p\}\\bm\{T\}\_\{1j\}\\bm\{x\}\_\{j\}\\\\ \\vdots\\\\ \\sum\_\{j=1\}^\{p\}\\bm\{T\}\_\{pj\}\\bm\{x\}\_\{j\}\\end\{bmatrix\}\.\(67\)Each block–vector product𝑻i​j​𝒙j\\bm\{T\}\_\{ij\}\\bm\{x\}\_\{j\}can be computed using FFT\-based Toeplitz multiplication with computational cost of𝒪⁡\(q​log⁡q\)\\mathcal\{O\}\(q\\log q\)\. Since there arep2p^\{2\}Toeplitz blocks in total, a single block\-wise Toeplitz matrix–vector multiplication incurs a cost of

𝒪⁡\(p2​q​log⁡q\)=𝒪⁡\(p​n​log⁡q\)\.\\mathcal\{O\}\\left\(p^\{2\}q\\log q\\right\)=\\mathcal\{O\}\(pn\\log q\)\.\(68\)In ToPA, this operation is applied sequentially along a chain of lengthtt, yielding a total cost of

𝒪⁡\(t​p​n​log⁡q\)\\mathcal\{O\}\\left\(tpn\\log q\\right\)\(69\)
To evaluate the practical computational efficiency of ToPA, we measure both the peak GPU memory consumption and the average wall\-clock time per training step\. All methods are evaluated under aligned settings on eight RTX 4090 GPUs, withper\_device\_batch\_size=1and gradient accumulation enabled\. We report results on Qwen3\-4B for math reasoning and LLaMA\-2\-7B for commonsense reasoning\.Notably, on LLaMA\-2\-7B, BOFT runs out of memory without gradient checkpointing and thus cannot be executed under the same setting\.

As shown in Tab\.[6](https://arxiv.org/html/2610.22883#A3.T6)and Tab\.[7](https://arxiv.org/html/2610.22883#A3.T7), ToPA introduces moderate overhead compared with low\-rank methods such as LoRA, DoRA, and HiRA, while remaining substantially more efficient than sparsity structured methods such as OFT and BOFT\. Considering the consistent performance gains achieved by ToPA, this overhead is practically acceptable\.

Table 6:Efficiency comparison on Qwen3\-4B for math reasoning\. Time denotes the average wall\-clock time per training step, and memory denotes the peak GPU memory consumption\.Table 7:Efficiency comparison on LLaMA\-2\-7B for commonsense reasoning\. Time denotes the average wall\-clock time per training step, and memory denotes the peak GPU memory consumption\. BOFT runs out of memory under the same settings without gradient checkpointing\.

### Appendix DFast Matrix–Vector Multiplication for Toeplitz Matrices

In this section, we provide the details of the fast matrix–vector multiplication for Toeplitz matrices\. The method relies on embedding the Toeplitz matrix into a larger circulant matrix and computing the resulting product via the fast Fourier transform \(FFT\), see, e\.g\.,\[[16](https://arxiv.org/html/2610.22883#bib.bib43),[38](https://arxiv.org/html/2610.22883#bib.bib44)\]\. We first recall the classical circulant case, which serves as the algorithmic backbone of our construction\.

Circulant matrices and Fourier diagonalization\.LetC∈ℝn×nC\\in\\mathbb\{R\}^\{n\\times n\}be a circulant matrix whose structure is fully determined by its first columnc=\(c0,…,cn−1\)⊤c=\(c\_\{0\},\\dots,c\_\{n\-1\}\)^\{\\top\}\. Each row is obtained by a cyclic shift ofcc,

C=\(c0cn−1⋯c1c1c0⋯c2⋱cn−1cn−2⋯c0\),Ci​j=c\(i−j\)modn\.C=\\begin\{pmatrix\}c\_\{0\}&c\_\{n\-1\}&\\cdots&c\_\{1\}\\\\ c\_\{1\}&c\_\{0\}&\\cdots&c\_\{2\}\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ c\_\{n\-1\}&c\_\{n\-2\}&\\cdots&c\_\{0\}\\end\{pmatrix\},\\qquad C\_\{ij\}=c\_\{\(i\-j\)\\bmod n\}\.\(70\)LetFFdenote the unitary discrete Fourier transform \(DFT\) matrix\. A standard structural fact is that every circulant matrix is diagonalized byFF:

C=F∗​Λ​F,Λ=diag⁡\(F​c\)\.C=F^\{\\ast\}\\Lambda F,\\qquad\\Lambda=\\mathrm\{diag\}\(Fc\)\.\(71\)Consequently, the matrix–vector productC​xCxmay be written as

C​x=F∗​\(Λ⁡\(F​x\)\)=F∗​\(\(F​c\)⊙\(F​x\)\),Cx=F^\{\\ast\}\\\!\\left\(\\Lambda\(Fx\)\\right\)=F^\{\\ast\}\\\!\\left\(\(Fc\)\\odot\(Fx\)\\right\),\(72\)where⊙\\odotdenotes elementwise multiplication\. SinceFFandF∗F^\{\\ast\}can be computed using FFT and inverse FFT, the resulting complexity is𝒪⁡\(n​log⁡n\)\\mathcal\{O\}\(n\\log n\), in contrast to the naive𝒪⁡\(n2\)\\mathcal\{O\}\(n^\{2\}\)cost\.

Embedding a Toeplitz matrix into a circulant matrix\.Consider now a Toeplitz matrixT∈ℝn×nT\\in\\mathbb\{R\}^\{n\\times n\}whose entries depend only on the diagonal offset:

Ti​j=ti−j,i,j=0,…,n−1\.T\_\{ij\}=t\_\{i\-j\},\\qquad i,j=0,\\dots,n\-1\.\(73\)AlthoughTTis not circulant, it can be embedded into a\(2​n\)×\(2​n\)\(2n\)\\times\(2n\)circulant matrixC~\\widetilde\{C\}whose top\-leftn×nn\\times nblock coincides withTT\. We construct the generating columnc~∈ℝ2​n\\tilde\{c\}\\in\\mathbb\{R\}^\{2n\}as

c~k=\{tk,0≤k≤n−1,0,k=n,tk−2​n,n\+1≤k≤2​n−1,\\tilde\{c\}\_\{k\}=\\begin\{cases\}t\_\{k\},&0\\leq k\\leq n\-1,\\\\ 0,&k=n,\\\\ t\_\{k\-2n\},&n\+1\\leq k\\leq 2n\-1,\\end\{cases\}\(74\)and defineC~\\widetilde\{C\}to be the circulant matrix generated byc~\\tilde\{c\}\. Partitioning

C~=\(TC12C21C22\),Cs​t∈ℝn×n,\\widetilde\{C\}=\\begin\{pmatrix\}T&C\_\{12\}\\\\ C\_\{21\}&C\_\{22\}\\end\{pmatrix\},\\qquad C\_\{st\}\\in\\mathbb\{R\}^\{n\\times n\},\(75\)identifiesTTexactly as the upper\-left block ofC~\\widetilde\{C\}\.

Recovering the Toeplitz product via zero padding\.Givenx∈ℝnx\\in\\mathbb\{R\}^\{n\}, we form the zero\-padded vector

x~=\(x0n\)∈ℝ2​n\.\\tilde\{x\}=\\begin\{pmatrix\}x\\\\ 0\_\{n\}\\end\{pmatrix\}\\in\\mathbb\{R\}^\{2n\}\.\(76\)Using the block structure above, we obtain

C~​x~=\(T​xC21​x\)∈ℝ2​n,\\widetilde\{C\}\\,\\tilde\{x\}=\\begin\{pmatrix\}Tx\\\\ C\_\{21\}x\\end\{pmatrix\}\\in\\mathbb\{R\}^\{2n\},\(77\)so the desired Toeplitz product is recovered simply by selecting the firstnncoordinates:

T​x=P⁡\(C~​x~\),P=\[In​0n×n\]\.Tx=P\\,\(\\widetilde\{C\}\\,\\tilde\{x\}\),\\qquad P=\\bigl\[\\,I\_\{n\}\\;\\;0\_\{n\\times n\}\\,\\bigr\]\.\(78\)
FFT\-based implementation\.SinceC~\\widetilde\{C\}is circulant, it admits the\(2​n\)\(2n\)\-point Fourier diagonalization

C~=F2​n∗​Λ~​F2​n,Λ~=diag⁡\(F2​n​c~\),\\widetilde\{C\}=F\_\{2n\}^\{\\ast\}\\,\\widetilde\{\\Lambda\}\\,F\_\{2n\},\\qquad\\widetilde\{\\Lambda\}=\\mathrm\{diag\}\(F\_\{2n\}\\tilde\{c\}\),\(79\)which yields the practical computation rule

T​x=P​F2​n∗​\(\(F2​n​c~\)⊙\(F2​n​x~\)\)\.Tx=P\\,F\_\{2n\}^\{\\ast\}\\\!\\Bigl\(\(F\_\{2n\}\\tilde\{c\}\)\\odot\(F\_\{2n\}\\tilde\{x\}\)\\Bigr\)\.\(80\)Thus, the Toeplitz matrix–vector product can be implemented via fast Fourier transform \(FFT\), achieving an overall computational complexity of𝒪⁡\(n​log⁡n\)\\mathcal\{O\}\(n\\log n\)\.

### Appendix EMore Experiment Protocols

#### E\.1Datasets

Natural Language Understanding\.For natural language understanding evaluation, we conduct experiments on eight benchmark datasets from the GLUE\[[53](https://arxiv.org/html/2610.22883#bib.bib23)\]benchmark: CoLA, SST\-2, MRPC, STS\-B, QQP, MNLI, QNLI, and RTE\. These datasets cover a diverse range of sentence\-level and sentence\-pair tasks, including linguistic acceptability, sentiment classification, paraphrase detection, semantic similarity, and natural language inference\. Below is a brief description of these datasets:

- •CoLA\[[56](https://arxiv.org/html/2610.22883#bib.bib45)\]\. The Corpus of Linguistic Acceptability is a binary classification dataset for grammatical acceptability judgment\. Each instance consists of a single sentence annotated as either acceptable or unacceptable according to linguistic criteria\.
- •SST\-2\[[47](https://arxiv.org/html/2610.22883#bib.bib46)\]\. The Stanford Sentiment Treebank \(binary version\) is a sentence\-level sentiment analysis dataset\. The task is to classify whether a movie review sentence expresses positive or negative sentiment\.
- •MRPC\[[14](https://arxiv.org/html/2610.22883#bib.bib47)\]\. The Microsoft Research Paraphrase Corpus is a paraphrase identification dataset composed of sentence pairs\. The goal is to determine whether two sentences are semantically equivalent\.
- •STS\-B\[[7](https://arxiv.org/html/2610.22883#bib.bib48)\]\. The Semantic Textual Similarity Benchmark contains sentence pairs annotated with real\-valued similarity scores\. The task is formulated as a regression problem measuring the degree of semantic similarity between the two sentences\.
- •QQP\[[55](https://arxiv.org/html/2610.22883#bib.bib49)\]\. The Quora Question Pairs dataset contains pairs of user\-submitted questions\. The objective is to detect whether two questions are duplicates that convey the same underlying meaning\.
- •MNLI\[[58](https://arxiv.org/html/2610.22883#bib.bib50)\]\. The Multi\-Genre Natural Language Inference dataset is a large\-scale textual entailment benchmark spanning multiple domains\. Given a premise and a hypothesis, the task is to classify the relation as entailment, contradiction, or neutral\.
- •QNLI\[[41](https://arxiv.org/html/2610.22883#bib.bib51)\]\. The Question\-answering Natural Language Inference dataset is derived from SQuAD and reformulated as a binary classification task\. Given a question and a candidate sentence, the model predicts whether the sentence contains the answer\.
- •RTE\[[15](https://arxiv.org/html/2610.22883#bib.bib52)\]\. The Recognizing Textual Entailment dataset aggregates several small\-scale entailment corpora\. Each sample consists of a premise–hypothesis pair labeled as entailment or non\-entailment\.

Commonsense Reasoning\.For commonsense reasoning tasks, we evaluate ToPA’s effectiveness on eight widely used benchmark datasets: BoolQ, PIQA, SocialIQA \(SIQA\), ARC\-c, ARC\-e, OpenBookQA \(OBQA\), HellaSwag \(HellaS\), and WinoGrande \(WinoG\)\. These datasets cover diverse forms of commonsense inference, including physical reasoning, social interaction understanding, science question answering, narrative plausibility judgment, and pronoun resolution\. Below is a brief description of these datasets:

- •BoolQ\[[10](https://arxiv.org/html/2610.22883#bib.bib53)\]\. BoolQ is a yes/no question answering dataset collected from real user queries paired with supporting passages from Wikipedia\. The task requires models to determine whether the answer to the question is entailed by the given passage, often involving non\-trivial reading comprehension and commonsense inference rather than direct lexical matching\.
- •PIQA\[[3](https://arxiv.org/html/2610.22883#bib.bib54)\]\. The Physical Interaction Question Answering \(PIQA\) dataset focuses on physical commonsense reasoning about how people interact with everyday objects\. Given a short description of a goal or situation, the model must choose the more plausible of two candidate solutions, which typically requires knowledge of object affordances and intuitive physical constraints\.
- •SIQA\[[46](https://arxiv.org/html/2610.22883#bib.bib55)\]\. SocialIQA is a benchmark for social commonsense reasoning centered on human intentions, emotions, and social behaviors\. Each example presents a short description of a social scenario followed by a multiple\-choice question, and the task is to infer the most plausible motivation, reaction, or outcome based on implicit social norms and everyday experiences\.
- •ARC\-c\[[11](https://arxiv.org/html/2610.22883#bib.bib56)\]\. The ARC Challenge \(ARC\-c\) subset consists of grade\-school science multiple\-choice questions that cannot be solved by simple information\-retrieval or co\-occurrence baselines\. These questions require multi\-step reasoning that integrates background science knowledge with commonsense inference, making ARC\-c a difficult reasoning\-oriented benchmark\.
- •ARC\-e\[[11](https://arxiv.org/html/2610.22883#bib.bib56)\]\. The ARC Easy \(ARC\-e\) subset contains science questions from the same source as ARC\-c but with relatively simpler reasoning requirements\. Many questions can be answered using direct retrieval or short factual associations, and this subset is commonly used to evaluate performance on more knowledge\-oriented science QA tasks\.
- •OBQA\[[35](https://arxiv.org/html/2610.22883#bib.bib57)\]\. OpenBookQA is an open\-book question answering dataset that provides a small collection of elementary science facts together with multiple\-choice questions\. Solving each question typically requires combining one core fact from the open book with additional commonsense or world knowledge, encouraging models to perform multi\-hop reasoning across explicit and implicit information\.
- •HellaS\[[66](https://arxiv.org/html/2610.22883#bib.bib58)\]\. HellaSwag is a commonsense inference dataset formulated as a sentence completion task\. Given a short context describing an everyday event, the model must select the most plausible continuation among four candidates\. The dataset is adversarially constructed so that superficial statistical cues are ineffective, highlighting gaps in models’ understanding of everyday scenarios\.
- •WinoG\[[45](https://arxiv.org/html/2610.22883#bib.bib59)\]\. WinoGrande is a large\-scale benchmark inspired by the Winograd Schema Challenge\. Each instance consists of a short sentence with an ambiguous reference, and the task is to choose the correct entity based on commonsense reasoning\. Compared to earlier Winograd datasets, WinoGrande is adversarially filtered and substantially larger, providing a more robust evaluation of commonsense reasoning abilities\.

Math Reasoning & Code Generation\.For mathematical reasoning and code generation tasks, we evaluate the effectiveness of our method on four widely used benchmark datasets: GSM8K, MATH, HumanEval, and MBPP\. These datasets cover a broad spectrum of problem\-solving skills, including grade\-school arithmetic reasoning, competition\-level mathematics, functional code synthesis, and programming\-by\-example tasks\. Below is a brief description of these datasets:

- •GSM8K\[[12](https://arxiv.org/html/2610.22883#bib.bib26)\]\. The GSM8K dataset is a collection of grade\-school math word problems that primarily assess multi\-step arithmetic and quantitative reasoning\. Each question requires reasoning through intermediate steps rather than applying a single direct formula, making it widely used to evaluate chain\-of\-thought and step\-by\-step problem\-solving ability in language models\.
- •MATH\[[19](https://arxiv.org/html/2610.22883#bib.bib60)\]\. The MATH dataset consists of competition\-style math problems spanning subjects such as algebra, geometry, number theory, and combinatorics\. Compared with GSM8K, the problems in MATH are substantially more challenging and often require formal mathematical reasoning, symbolic manipulation, and multi\-stage derivations to arrive at correct solutions\.
- •HumanEval\[[8](https://arxiv.org/html/2610.22883#bib.bib19)\]\. HumanEval is a benchmark for code generation and functional correctness, where each sample provides a natural\-language specification and a partially implemented Python function\. Models are required to synthesize executable code that satisfies hidden unit tests, and performance is measured using pass@1 metrics that evaluate correctness across multiple samples\.
- •MBPP\[[2](https://arxiv.org/html/2610.22883#bib.bib28)\]\. The MBPP \(Mostly Basic Python Programming\) dataset contains short, self\-contained Python programming tasks designed for programming\-by\-example evaluation\. Each problem includes a textual description and a few input–output test cases, and the goal is to generate Python code that correctly implements the described functionality across unseen tests\.

Subject\-Driven Generation\.For evaluating subject\-driven image generation, we conduct experiments on the dataset introduced in DreamBooth\[[44](https://arxiv.org/html/2610.22883#bib.bib29)\]\. This dataset is designed to assess a model’s ability to personalize a generative model to a specific subject using only a few reference images, while preserving the subject’s identity and appearance across novel contexts and backgrounds\. It enables systematic evaluation of fidelity, editability, and compositional generalization in personalized image synthesis\. Below is a brief description of this dataset:

- •DreamBooth\[[44](https://arxiv.org/html/2610.22883#bib.bib29)\]\. The DreamBooth dataset contains a collection of small subject\-specific image sets, where each subject is represented by only a few high\-quality images captured under different viewpoints and environments\. During training, the model learns to bind a unique identifier token to the target subject, enabling personalized generation conditioned on text prompts\. The dataset is used to evaluate whether the generated images maintain identity consistency while allowing the subject to be rendered in diverse scenes, artistic styles, and compositional settings\.

#### E\.2Competitors

To evaluate the effectiveness of our method, we select several representative methods as competitors\. Below is a brief description of the competitors considered in our experiments:

- •LoRA\[[21](https://arxiv.org/html/2610.22883#bib.bib1)\]\. LoRA \(Low\-Rank Adaptation\) factorizes the weight update into the product of two low\-rank matrices that are learned during fine\-tuning, while the original pretrained weights remain frozen\. This design substantially reduces the number of trainable parameters and allows the updates to be merged back into the base model at inference time\.
- •DoRA\[[30](https://arxiv.org/html/2610.22883#bib.bib11)\]\. DoRA decomposes each weight matrix into a magnitude component and a direction component\. It directly tunes the magnitude component while employing LoRA to the direction component, thereby increasing the adapter’s learning capacity and optimization stability\.
- •PiSSA\[[34](https://arxiv.org/html/2610.22883#bib.bib10)\]\. PiSSA performs the SVD decomposition to split each pretrained weight matrix into principal and residual components, initializing LoRA with the principal component while assigning the residual component to the pretrained weight, thereby accelerating the convergence\.
- •HiRA\[[23](https://arxiv.org/html/2610.22883#bib.bib3)\]\. HiRA factorizes the update matrix as the Hadamard product between the pretrained weight matrix and a low\-rank matrix, thereby yielding a high\-rank and more expressive adaptation\.
- •RidgeLoRA\[[71](https://arxiv.org/html/2610.22883#bib.bib22)\]\. RidgeLoRA reformulates the LoRA update by replacing the parallel additive residual branch with a series, multiplicative transformation\. In addition, it enriches the low\-rank adapter with a learnable diagonal ridge matrix, achieving a high\-rank update and further strengthening the expressiveness\.
- •OFT\[[39](https://arxiv.org/html/2610.22883#bib.bib2)\]\. OFT adapts the model weights by multiplying the pretrained weights with a carefully designed sparse orthogonal matrix, which enables lightweight transformation while preserving the hyperspherical energy of the network and maintaining the pretrained semantic knowledge of the model\.
- •BOFT\[[31](https://arxiv.org/html/2610.22883#bib.bib5)\]\. BOFT extends OFT by replacing the single sparse orthogonal matrix with a product of multiple sparse butterfly matrices\. This structured parameterization improves the expressiveness of the transformation while preserving parameter efficiency\.
- •HRA\[[64](https://arxiv.org/html/2610.22883#bib.bib6)\]\. HRA fine\-tunes each pretrained matrix by right\-multiplying it with an orthogonal matrix parameterized as a product of learnable Householder reflections\. By regularizing the mutual orthogonality of the reflection planes, HRA balances adaptation capacity and regularity, thereby preserving pretrained knowledge while achieving strong performance\.

#### E\.3Implementation Details

Natural Language Understanding\. Following\[[31](https://arxiv.org/html/2610.22883#bib.bib5),[64](https://arxiv.org/html/2610.22883#bib.bib6)\], we adopt DeBERTaV3\-Base\[[18](https://arxiv.org/html/2610.22883#bib.bib30)\]as the backbone model\. We use the same batch size of 32 and maximum sequence length as in the prior work, while tuning the learning rate and the number of training epochs\. We employ the AdamW\[[32](https://arxiv.org/html/2610.22883#bib.bib62)\]optimizer for training\. During fine\-tuning, we freeze the pretrained backbone parameters and insert adapters into the query, key, and value projection matrices in the attention modules\. Detailed hyperparameter settings are reported in Tab\.[8](https://arxiv.org/html/2610.22883#A5.T8)\.

Table 8:The hyperparameter settings on the GLUE benchmarksSubject\-Driven Generation\. Following\[[39](https://arxiv.org/html/2610.22883#bib.bib2)\], we adopt Stable Diffusion v1\.5\[[42](https://arxiv.org/html/2610.22883#bib.bib34)\]as the base model and inject the adapter into the query, key, value, and output projection matrices of the U\-Net\[[43](https://arxiv.org/html/2610.22883#bib.bib63)\]module\. We use the AdamW optimizer with a learning rate of6×10−56\\times 10^\{\-5\}and a batch size of 1, training the model for approximately 1,000 steps\.

Commonsense Reasoning\. For the commonsense reasoning tasks, we employ the LLaMA\-2\-7B\[[52](https://arxiv.org/html/2610.22883#bib.bib31)\]and LLaMA\-3\-8B\[[48](https://arxiv.org/html/2610.22883#bib.bib32)\]as the backbone models\. During fine\-tuning, all pretrained weights are kept frozen, and adapters are inserted into the query and value projection matrices\. Optimization is performed with AdamW optimizer using a learning rate of3×10−53\\times 10^\{\-5\}, a warm up steps of100100, and a batch size of1616\. The training extends for33epochs\.

Math Reasoning & Code Generation\. For math reasoning and code generation tasks, we use LLaMA\-3\-8B\[[48](https://arxiv.org/html/2610.22883#bib.bib32)\]and Qwen3\-4B\[[60](https://arxiv.org/html/2610.22883#bib.bib33)\]as the base models\. During training, adapters are applied to fine\-tune the query and value projection matrices within the attention modules\. Optimization is carried out using AdamW with a learning rate of1×10−41\\times 10^\{\-4\}and a batch size of3232\. We further employ a cosine learning rate scheduler, and the models are trained for a single epoch\.

### Appendix FMore Experiment Results

#### F\.1Additional Ablation Study of the Identity Residual

To further demonstrate the crucial role of the identity residual, we perform an additional ablation analysis\. As detailed in Tab\.[9](https://arxiv.org/html/2610.22883#A6.T9), we compare the performance impact of the identity matrix when fine\-tuning LLaMA\-3\-8B with ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}on math reasoning and code generation tasks\. The result suggests that incorporating the identity residual yields consistent performance gains across all evaluated datasets\. These results further confirm the effectiveness of the residual design\.

Table 9:Performance comparison of LLaMA\-3\-8B fine\-tuned with ToPAt=3p=8\{\}\_\{p=8\}^\{t=3\}on math & code generation tasks, with and without the identity residual\.
#### F\.2Additional Results on Training Instability in the Naive Method

Fig\.[8](https://arxiv.org/html/2610.22883#A6.F8.fig1)illustrates the training dynamics of the Frobenius norm of gradients and the training loss for the naive method \(t=32t=32\) and ToPAp=16t=2\{\}^\{t=2\}\_\{p=16\}when fine\-tuning LLaMA\-2\-7B on commonsense reasoning tasks\. As shown in the figures, the naive method suffers from pronounced instability: both the gradient norm and the training loss exhibit substantial fluctuations and attain relatively large values, indicating a tendency toward gradient explosion\. In contrast, ToPA yields significantly more stable behavior, with both metrics remaining consistently smoother and markedly smaller than those of the naive baseline\. These observations further corroborate the effectiveness of ToPA in mitigating gradient explosion, thereby leading to more stable training dynamics\.

\(a\)Gradient Norm\(b\)Train Loss
Figure 8:Training dynamics over the first 300 steps of fine\-tuning LLaMA\-2\-7B on commonsense reasoning tasks using the naive method \(t=32t=32\) and ToPAp=16t=2\{\}^\{t=2\}\_\{p=16\}with matched parameters\. As training proceeds, the naive method collapses, so only the first 300 steps are shown\.
#### F\.3Multi\-Seed Evaluation on Math Reasoning and Code Generation

To further verify the robustness of ToPA, we repeat the math reasoning and code generation experiments on the LLaMA\-3\-8B backbone with three random seeds\. As shown in Tab\.[10](https://arxiv.org/html/2610.22883#A6.T10), ToPA consistently achieves the best or tied\-best average performance under a comparable parameter budget\. These results further validate the effectiveness of ToPA\.

Table 10:Multi\-seed performance comparison on math reasoning and code generation tasks with the LLaMA\-3\-8B backbone\. Results are averaged over three random seeds\.
#### F\.4Additional Qualitative Results for Subject\-Driven Generation

Fig\.[9](https://arxiv.org/html/2610.22883#A6.F9), Fig\.[10](https://arxiv.org/html/2610.22883#A6.F10), and Fig\.[11](https://arxiv.org/html/2610.22883#A6.F11)demonstrate additional qualitative results for subject\-driven generation tasks\. From the results, we can observe that ToPA consistently produces images with higher subject fidelity and stronger instruction\-following ability compared with LoRA and OFT\. Overall, these qualitative comparisons further support the effectiveness of ToPA for subject\-driven generation, demonstrating its ability to balance subject preservation with generation diversity\.

![Refer to caption](https://arxiv.org/html/2610.22883v1/appendix_subject_driven_generation_1.png)Figure 9:Additional qualitative comparison of results on subject\-driven generation tasks\.![Refer to caption](https://arxiv.org/html/2610.22883v1/appendix_subject_driven_generation_2.png)Figure 10:Additional qualitative comparison of results on subject\-driven generation tasks\.![Refer to caption](https://arxiv.org/html/2610.22883v1/appendix_subject_driven_generation_3.png)Figure 11:Additional qualitative comparison of results on subject\-driven generation tasks\.
#### F\.5Qualitative Results for Math Reasoning

In this section, we present several qualitative examples comparing the questions and corresponding responses generated by the LLaMA\-3\-8B model fine\-tuned with LoRA and ToPA on math reasoning tasks\. From these cases, we observe that ToPA consistently produces more accurate and reliable answers than LoRA\. These qualitative results further support the effectiveness of ToPA in enhancing the model’s math reasoning capability\.

Case StudyQuestion:I have 10 liters of orange drink that are two\-thirds water and I wish to add it to 15 liters of pineapple drink that is three\-fifths water\. But as I pour it, I spill one liter of the orange drink\. How much water is in the remaining 24 liters?Response of LoRAr=32:First, we need to find out how much water is in the orange drink\. Two\-thirds of 10 liters is 10 \* 2/3 = 6\.67 liters\. So, there are 6\.67 liters of water in the orange drink\. Next, we need to find out how much water is in the pineapple drink\. Three\-fifths of 15 liters is 15 \* 3/5 = 9 liters\. So, there are 9 liters of water in the pineapple drink\. Now, we need to add the water from the orange drink to the pineapple drink\. The total amount of water in the pineapple drink is 9 \+ 6\.67 = 15\.67 liters\. Since we spilled 1 liter of the orange drink, the remaining amount of water in the pineapple drink is 15\.67 \- 1 = 14\.67 liters\. Therefore, there are 14\.67 liters of water in the remaining 24 liters\. \#\#\#\# 14\.67 The answer is: 14\.67Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:We have 10 liters of orange drink that is two\-thirds water, so 10 \* \(2/3\) = 6\.67 liters of water in the orange drink\. We also have 15 liters of pineapple drink that is three\-fifths water, so 15 \* \(3/5\) = 9 liters of water in the pineapple drink\. When we pour the orange drink, we spill 1 liter, so we have 10 \- 1 = 9 liters of orange drink remaining\. The remaining orange drink is two\-thirds water, so 9 \* \(2/3\) = 6 liters of water in the remaining orange drink\. The total amount of water in the remaining 24 liters is 6 \+ 9 = 15 liters\. \#\#\#\# 15 The answer is: 15Correct Answer:15

Case StudyQuestion:Meredith is a freelance blogger who writes about health topics and submits to clients each day as her permanent job\. A blog article takes an average of 4 hours to research and write about\. Last week, she wrote 5 articles on Monday and 2/5 times more articles on Tuesday than on Monday\. On Wednesday, she wrote twice the number of articles she wrote on Tuesday\. Calculate the total number of hours she spent writing articles in the three days\.Response of LoRAr=32:On Monday, Meredith wrote 5 articles, so she spent 5 \* 4 = 20 hours writing articles\. On Tuesday, she wrote 2/5 \* 5 = 2 \* 5 = 10 articles\. So, she spent 10 \* 4 = 40 hours writing articles on Tuesday\. On Wednesday, she wrote twice the number of articles she wrote on Tuesday, so she wrote 10 \* 2 = 20 articles\. Therefore, she spent 20 \* 4 = 80 hours writing articles on Wednesday\. In total, she spent 20 \+ 40 \+ 80 = 140 hours writing articles in the three days\. \#\#\#\# 140 The answer is: 140Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:On Monday, Meredith wrote 5 articles, so she spent 5 \* 4 = 20 hours writing articles\. On Tuesday, she wrote 2/5 \* 5 = 2 more articles than on Monday, so she wrote 5 \+ 2 = 7 articles\. Therefore, she spent 7 \* 4 = 28 hours writing articles on Tuesday\. On Wednesday, she wrote twice the number of articles she wrote on Tuesday, so she wrote 7 \* 2 = 14 articles\. Therefore, she spent 14 \* 4 = 56 hours writing articles on Wednesday\. In total, Meredith spent 20 \+ 28 \+ 56 = 104 hours writing articles in the three days\. \#\#\#\# 104 The answer is: 104Correct Answer:104

Case StudyQuestion:John plans to sell all his toys and use the money to buy video games\. He has 13 lego sets and he sells them for $15 each\. He ends up buying 8 video games for $20 each and has $5 left\. How many lego sets does he still have?Response of LoRAr=32:John sells 13 lego sets for $15 each, so he earns 13 \* $15 = $195\. He buys 8 video games for $20 each, so he spends 8 \* $20 = $160\. He has $5 left, so he has $195 \- $160 = $35 from selling the lego sets\. Therefore, John still has 13 \- 8 = 5 lego sets\. \#\#\#\# 5 The answer is: 5Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:John sells 13 lego sets for $15 each, so he earns 13 \* $15 = $195\. He then buys 8 video games for $20 each, so he spends 8 \* $20 = $160\. He has $5 left, so he has $195 \- $160 \- $5 = $30 from selling the lego sets\. Since each lego set is worth $15, he still has $30 / $15 = 2 lego sets left\. \#\#\#\# 2 The answer is: 2Correct Answer:2

Case StudyQuestion:Gloria is shoe shopping when she comes across a pair of boots that fit her shoe budget\. However, she has to choose between the boots and two pairs of high heels that together cost five dollars less than the boots\. If one pair of heels costs $33 and the other costs twice as much, how many dollars are the boots?Response of LoRAr=32:One pair of heels costs $33\. The other pair of heels costs twice as much, so it costs 2 \* $33 = $66\. The total cost of the two pairs of heels is $33 \+ $66 = $99\. The boots cost five dollars less than the two pairs of heels, so they cost $99 \- $5 = $94\. Therefore, the boots cost $94\. \#\#\#\# 94 The answer is: 94Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:One pair of heels costs $33, and the other costs twice as much, so the second pair of heels costs 2 \* $33 = $66\. The total cost of the two pairs of heels is $33 \+ $66 = $99\. The boots cost five dollars more than the two pairs of heels, so the boots cost $99 \+ $5 = $104\. Therefore, the boots cost $104\. \#\#\#\# 104 The answer is: 104Correct Answer:104

Case StudyQuestion:A translation of the plane takes5\+5​i5\+5ito6\+8​i\.6\+8i\.Find the complex number that the translation takes−6\-6to\.Response of LoRAr=32:We are given that the translation takes5\+5​i5\+5ito6\+8​i6\+8i\. To find the complex number that the translation takes−6\-6to, we need to find the complex number that is−6\-6units away from5\+5​i5\+5iin the direction of6\+8​i6\+8i\. We can use the formula for the distance between two complex numbers to find the distance between5\+5​i5\+5iand−6\-6\. The distance between5\+5​i5\+5iand−6\-6is\(−6−5\)2\+\(−6−5​i\)2=121\+121=242=14\\sqrt\{\(\-6\-5\)^\{2\}\+\(\-6\-5i\)^\{2\}\}=\\sqrt\{121\+121\}=\\sqrt\{242\}=14\. Now, we need to find the complex number that is1414units away from5\+5​i5\+5iin the direction of6\+8​i6\+8i\. We can use the formula for the direction of a complex number to find the direction of6\+8​i6\+8i\. The direction of6\+8​i6\+8iistan−1⁡\(86\)=tan−1⁡\(43\)\\tan^\{\-1\}\\left\(\\frac\{8\}\{6\}\\right\)=\\tan^\{\-1\}\\left\(\\frac\{4\}\{3\}\\right\)\. Now, we can use the formula for the complex number that is1414units away from5\+5​i5\+5iin the direction of6\+8​i6\+8i\. The complex number is5\+5​i\+14​\(cos⁡\(tan−1⁡\(43\)\)\+i​sin⁡\(tan−1⁡\(43\)\)\)5\+5i\+14\\left\(\\cos\\left\(\\tan^\{\-1\}\\left\(\\frac\{4\}\{3\}\\right\)\\right\)\+i\\sin\\left\(\\tan^\{\-1\}\\left\(\\frac\{4\}\{3\}\\right\)\\right\)\\right\)\. Simplifying, we get5\+5​i\+14​\(45\+i​35\)=5\+5​i\+56​\(45\+i​35\)5\+5i\+14\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)=5\+5i\+56\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)\. Further simplifying, we get5\+5​i\+224​\(45\+i​35\)=5\+5​i\+896​\(45\+i​35\)5\+5i\+224\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)=5\+5i\+896\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)\. Finally, we get5\+5​i\+896​\(45\+i​35\)=5\+5​i\+3584​\(45\+i​35\)5\+5i\+896\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)=5\+5i\+3584\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)\. Therefore, the complex number that the translation takes−6\-6to is3584​\(45\+i​35\)\\boxed\{3584\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)\}\.The answer is:3584​\(45\+i​35\)3584\\left\(\\frac\{4\}\{5\}\+i\\frac\{3\}\{5\}\\right\)Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:The translation takes5\+5​i5\+5ito6\+8​i\.6\+8i\.This means that the translation adds11to the real part and33to the imaginary part\. So, if we want to find the complex number that the translation takes−6\-6to, we need to add11to the real part and33to the imaginary part\. Therefore, the complex number that the translation takes−6\-6to is−6\+1\+3​i=−5\+3​i\.\-6\+1\+3i=\\boxed\{\-5\+3i\}\.The answer is:−5\+3​i\-5\+3iCorrect Answer:−5\+3​i\-5\+3i

Case StudyQuestion:Compute\(310−74205−1\)​\(4−1−2\)\.\\begin\{pmatrix\}3&1&0\\\\ \-7&4&2\\\\ 0&5&\-1\\end\{pmatrix\}\\begin\{pmatrix\}4\\\\ \-1\\\\ \-2\\end\{pmatrix\}\.Response of LoRAr=32:We want to multiply the matrix by the vector\. First, we multiply the first row of the matrix by the first element of the vector, the second row by the second element, and the third row by the third element\. So, we have3⋅4\+1⋅\(−1\)\+0⋅\(−2\)=12−1\+0=113\\cdot 4\+1\\cdot\(\-1\)\+0\\cdot\(\-2\)=12\-1\+0=11\. Next, we multiply the first row of the matrix by the second element of the vector, the second row by the third element, and the third row by the first element\. So, we have−7⋅4\+4⋅\(−1\)\+0⋅\(−2\)=−28\+4\+0=−24\-7\\cdot 4\+4\\cdot\(\-1\)\+0\\cdot\(\-2\)=\-28\+4\+0=\-24\. Finally, we multiply the first row of the matrix by the third element of the vector, the second row by the first element, and the third row by the second element\. So, we have0⋅4\+5⋅\(−1\)\+\(−1\)⋅\(−2\)=0−5\+2=−30\\cdot 4\+5\\cdot\(\-1\)\+\(\-1\)\\cdot\(\-2\)=0\-5\+2=\-3\. Therefore, the product of the matrix and the vector is\(11−24−3\)\\begin\{pmatrix\}11\\\\ \-24\\\\ \-3\\end\{pmatrix\}\. The answer is:\(11−24−3\)\\begin\{pmatrix\}11\\\\ \-24\\\\ \-3\\end\{pmatrix\}Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:We have a3×33\\times 3matrix and a3×13\\times 1matrix\. To multiply these matrices, we need to multiply each element of the first matrix by the corresponding element of the second matrix and then sum the products\. So, we have\(310−74205−1\)​\(4−1−2\)=\(3⋅4\+1⋅\(−1\)\+0⋅\(−2\)−7⋅4\+4⋅\(−1\)\+2⋅\(−2\)0⋅4\+5⋅\(−1\)\+\(−1\)⋅\(−2\)\)=\(12−1−0−28−4−40−5\+2\)=\(11−36−3\)\.\\begin\{pmatrix\}3&1&0\\\\ \-7&4&2\\\\ 0&5&\-1\\end\{pmatrix\}\\begin\{pmatrix\}4\\\\ \-1\\\\ \-2\\end\{pmatrix\}=\\begin\{pmatrix\}3\\cdot 4\+1\\cdot\(\-1\)\+0\\cdot\(\-2\)\\\\ \-7\\cdot 4\+4\\cdot\(\-1\)\+2\\cdot\(\-2\)\\\\ 0\\cdot 4\+5\\cdot\(\-1\)\+\(\-1\)\\cdot\(\-2\)\\end\{pmatrix\}=\\begin\{pmatrix\}12\-1\-0\\\\ \-28\-4\-4\\\\ 0\-5\+2\\end\{pmatrix\}=\\begin\{pmatrix\}11\\\\ \-36\\\\ \-3\\end\{pmatrix\}\.So, the product is\(11−36−3\)\\boxed\{\\begin\{pmatrix\}11\\\\ \-36\\\\ \-3\\end\{pmatrix\}\}\.The answer is:\(11−36−3\)\\begin\{pmatrix\}11\\\\ \-36\\\\ \-3\\end\{pmatrix\}Correct Answer:\(11−36−3\)\\begin\{pmatrix\}11\\\\ \-36\\\\ \-3\\end\{pmatrix\}

Case StudyQuestion:Computearccos⁡\(−12\)\.\\arccos\\left\(\-\\frac\{1\}\{2\}\\right\)\.Express your answer in radians\.Response of LoRAr=32:First, we need to find the angle whose cosine is−12\-\\frac\{1\}\{2\}\. We know that the cosine of an angle is negative when the angle is in the second or fourth quadrant\. In the second quadrant, the cosine of an angle is negative when the angle is between2​π3\\frac\{2\\pi\}\{3\}andπ\\pi\. So, we need to find the angle between2​π3\\frac\{2\\pi\}\{3\}andπ\\piwhose cosine is−12\-\\frac\{1\}\{2\}\. We can use a calculator to find that the angle is5​π6\\boxed\{\\frac\{5\\pi\}\{6\}\}\. The answer is:5​π6\\frac\{5\\pi\}\{6\}Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:The cosine function is positive in the first and fourth quadrants, so the angleθ\\thetathat satisfiescos⁡θ=−12\\cos\\theta=\-\\frac\{1\}\{2\}must be in the second or third quadrant\. In the second quadrant, the reference angle isθ=2​π3\\theta=\\frac\{2\\pi\}\{3\}, and in the third quadrant, the reference angle isθ=4​π3\\theta=\\frac\{4\\pi\}\{3\}\. Sincearccos\\arccosgives the angle in the principal branch, we take the angle in the second quadrant, soarccos⁡\(−12\)=2​π3\\arccos\\left\(\-\\frac\{1\}\{2\}\\right\)=\\boxed\{\\frac\{2\\pi\}\{3\}\}\. The answer is:2​π3\\frac\{2\\pi\}\{3\}Correct Answer:2​π3\\frac\{2\\pi\}\{3\}

Case StudyQuestion:Given‖𝐯‖=5\\\|\\mathbf\{v\}\\\|=5and‖𝐰‖=8,\\\|\\mathbf\{w\}\\\|=8,find the largest possible value of‖proj𝐰⁡𝐯‖\.\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|\.Response of LoRAr=32:First, we need to find the projection of𝐯\\mathbf\{v\}onto𝐰\\mathbf\{w\}\. The projection of𝐯\\mathbf\{v\}onto𝐰\\mathbf\{w\}is given byproj𝐰⁡𝐯=𝐯⋅𝐰‖𝐰‖2​𝐰\.\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}=\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{\\\|\\mathbf\{w\}\\\|^\{2\}\}\\mathbf\{w\}\.We know that‖𝐯‖=5\\\|\\mathbf\{v\}\\\|=5and‖𝐰‖=8\\\|\\mathbf\{w\}\\\|=8, so we can substitute these values into the formula:proj𝐰⁡𝐯=𝐯⋅𝐰82​𝐰\.\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}=\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{8^\{2\}\}\\mathbf\{w\}\.Now, we need to find the largest possible value of‖proj𝐰⁡𝐯‖\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|\. To do this, we need to maximize the magnitude of the projection, which is given by‖proj𝐰⁡𝐯‖=‖𝐯⋅𝐰82​𝐰‖\.\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|=\\left\\\|\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{8^\{2\}\}\\mathbf\{w\}\\right\\\|\.Since𝐰\\mathbf\{w\}is a unit vector, we can simplify this expression to‖proj𝐰⁡𝐯‖=‖𝐯⋅𝐰82​𝐰‖=‖𝐯⋅𝐰64​𝐰‖=‖𝐯⋅𝐰64​𝐰‖\.\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|=\\left\\\|\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{8^\{2\}\}\\mathbf\{w\}\\right\\\|=\\left\\\|\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{64\}\\mathbf\{w\}\\right\\\|=\\left\\\|\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{64\}\\mathbf\{w\}\\right\\\|\.To maximize this expression, we need to maximize the magnitude of𝐯⋅𝐰\\mathbf\{v\}\\cdot\\mathbf\{w\}\. Since𝐯\\mathbf\{v\}and𝐰\\mathbf\{w\}are vectors, we can think of𝐯⋅𝐰\\mathbf\{v\}\\cdot\\mathbf\{w\}as the dot product of𝐯\\mathbf\{v\}and𝐰\\mathbf\{w\}\. The dot product of two vectors is equal to the product of their magnitudes times the cosine of the angle between them\. So, we want to maximize the magnitude of𝐯⋅𝐰\\mathbf\{v\}\\cdot\\mathbf\{w\}, which is equal to the product of the magnitudes of𝐯\\mathbf\{v\}and𝐰\\mathbf\{w\}times the cosine of the angle between them\. Since‖𝐯‖=5\\\|\\mathbf\{v\}\\\|=5and‖𝐰‖=8\\\|\\mathbf\{w\}\\\|=8, the magnitude of𝐯⋅𝐰\\mathbf\{v\}\\cdot\\mathbf\{w\}is equal to5⋅8⋅cos⁡θ5\\cdot 8\\cdot\\cos\\theta, whereθ\\thetais the angle between𝐯\\mathbf\{v\}and𝐰\\mathbf\{w\}\. To maximize this expression, we want to maximize the value ofcos⁡θ\\cos\\theta\. The maximum value ofcos⁡θ\\cos\\thetais 1, which occurs whenθ=0\\theta=0\. So, the largest possible value of‖proj𝐰⁡𝐯‖\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|is‖𝐯⋅𝐰‖=5⋅8⋅cos⁡0=40\\\|\\mathbf\{v\}\\cdot\\mathbf\{w\}\\\|=5\\cdot 8\\cdot\\cos 0=40\. Therefore, the largest possible value of‖proj𝐰⁡𝐯‖\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|is40\\boxed\{40\}\.The answer is: 40Response of ToPAp=8t=3\{\}^\{t=3\}\_\{p=8\}:The projection of𝐯\\mathbf\{v\}onto𝐰\\mathbf\{w\}is given byproj𝐰⁡𝐯=𝐯⋅𝐰‖𝐰‖2​𝐰\.\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}=\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{\\\|\\mathbf\{w\}\\\|^\{2\}\}\\mathbf\{w\}\.We want to find the largest possible value of‖proj𝐰⁡𝐯‖\.\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|\.Since‖𝐯‖=5\\\|\\mathbf\{v\}\\\|=5and‖𝐰‖=8,\\\|\\mathbf\{w\}\\\|=8,we have‖proj𝐰⁡𝐯‖=‖𝐯⋅𝐰‖𝐰‖2​𝐰‖=‖𝐯⋅𝐰64​𝐰‖=\|𝐯⋅𝐰\|64​‖𝐰‖=\|𝐯⋅𝐰\|64⋅8=\|𝐯⋅𝐰\|8\.\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|=\\left\\\|\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{\\\|\\mathbf\{w\}\\\|^\{2\}\}\\mathbf\{w\}\\right\\\|=\\left\\\|\\frac\{\\mathbf\{v\}\\cdot\\mathbf\{w\}\}\{64\}\\mathbf\{w\}\\right\\\|=\\frac\{\|\\mathbf\{v\}\\cdot\\mathbf\{w\}\|\}\{64\}\\\|\\mathbf\{w\}\\\|=\\frac\{\|\\mathbf\{v\}\\cdot\\mathbf\{w\}\|\}\{64\}\\cdot 8=\\frac\{\|\\mathbf\{v\}\\cdot\\mathbf\{w\}\|\}\{8\}\.To maximize this expression, we want to maximize\|𝐯⋅𝐰\|\.\|\\mathbf\{v\}\\cdot\\mathbf\{w\}\|\.Since‖𝐯‖=5\\\|\\mathbf\{v\}\\\|=5and‖𝐰‖=8,\\\|\\mathbf\{w\}\\\|=8,the largest possible value of\|𝐯⋅𝐰\|\|\\mathbf\{v\}\\cdot\\mathbf\{w\}\|occurs when𝐯\\mathbf\{v\}and𝐰\\mathbf\{w\}are parallel\. In this case,𝐯⋅𝐰=‖𝐯‖⋅‖𝐰‖=5⋅8=40\.\\mathbf\{v\}\\cdot\\mathbf\{w\}=\\\|\\mathbf\{v\}\\\|\\cdot\\\|\\mathbf\{w\}\\\|=5\\cdot 8=40\.Therefore, the largest possible value of‖proj𝐰⁡𝐯‖\\\|\\operatorname\{proj\}\_\{\\mathbf\{w\}\}\\mathbf\{v\}\\\|is408=5\.\\frac\{40\}\{8\}=\\boxed\{5\}\.The answer is: 5Correct Answer:5

### Appendix GLimitations

Although ToPA shows consistent effectiveness across diverse NLP and CV tasks, our current evaluation is still mainly focused on single\-modal backbone models\. Extending ToPA to multimodal large models, such as vision\-language models, remains an important direction for future work, where cross\-modal interactions may introduce new adaptation requirements\.

In addition, due to computational resource constraints, our experiments are conducted on models up to the 8B scale\. While the results demonstrate the effectiveness of ToPA on multiple architectures, further investigation on larger\-scale models would be valuable to better understand its scalability and practical behavior in more demanding settings\.

### Appendix HBroader Impacts

This work aims to improve the efficiency and effectiveness of parameter\-efficient fine\-tuning\. Its positive impact lies in reducing the computational and memory cost of adapting large pretrained models, which may make fine\-tuning more accessible to researchers and practitioners with limited resources\. It may also help reduce the environmental cost associated with repeatedly adapting large models\.

This work is primarily methodological and does not target a specific high\-risk application\. We do not identify direct negative societal impacts beyond those generally associated with improving the adaptability of large models\. In practice, downstream deployment should still consider task\-specific risks, such as misuse, bias, or unreliable generations, depending on the application domain\.

### NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The claims in the abstract and introduction accurately reflect the proposed method, theoretical analysis, and empirical results presented in the paper\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: The paper includes a dedicated Limitations section discussing the current evaluation scope and future extensions to multimodal and larger\-scale models\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification: The theoretical results are stated with the necessary assumptions and are supported by formal proofs in the appendix\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: The paper provides detailed descriptions of the model architectures, datasets, baselines, hyperparameter settings, and evaluation protocols used in the experiments\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[No\]
24. Justification: If the paper is accepted, we will release the complete codebase along with detailed reproduction instructions\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: The paper specifies the main datasets, model backbones, baselines, evaluation protocols, and key training details\. Additional hyperparameters and implementation settings are provided in the appendix\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: The appendix includes a subsection reporting results over multiple random seeds\. The paper reports the mean and standard deviation to reflect the stability and statistical variation of the main experimental results\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: The paper reports the compute resources used for the experiments, including the GPU type, memory usage, and wall\-clock training time\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The research conforms to the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: The paper includes a Broader Impacts section discussing the potential benefits of more efficient model adaptation and the general downstream risks associated with adapting large models\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: The paper does not release new high\-risk pretrained models, image generators, or scraped datasets\. Therefore, this item is not applicable to the proposed methodological work\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: The paper properly cites the existing datasets, models, and baselines used in the experiments\. We use publicly available assets and follow their corresponding licenses and terms of use\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: The paper does not introduce or release new datasets, model, or other standalone assets, so this item is not applicable\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: The paper does not involve crowdsourcing, human\-subject studies, or user experiments, so this item is not applicable\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: The paper does not involve human\-subject studies, crowdsourcing, or user experiments, so IRB approval is not applicable\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: The core methodology does not use LLMs as an important, original, or non\-standard research component\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.

[37](https://arxiv.org/html/2610.22883#bib.bib65),[62](https://arxiv.org/html/2610.22883#bib.bib66),[50](https://arxiv.org/html/2610.22883#bib.bib67),[68](https://arxiv.org/html/2610.22883#bib.bib68),[54](https://arxiv.org/html/2610.22883#bib.bib69)[33](https://arxiv.org/html/2610.22883#bib.bib71)