Fine-Tuning of Transformer models with Frames

# Fine-Tuning of Transformer models with Frames
Source: [https://arxiv.org/html/2608.26430](https://arxiv.org/html/2608.26430)
Harshavardhan AdepuLi ZhangAffiliation:Google DeepMindSanjiv KumarAffiliation:Google DeepMindVikas SinghAffiliation:University of Wisconsin\-Madison

###### Abstract

Parameter\-Efficient Fine\-Tuning \(PEFT\) strategies such as Low\-Rank Adaptation \(LoRA\) are effective solutions for fine\-tuning large\-scale pre\-trained models; however, their memory requirements scale with the size of the model,𝒪⁡\(d​r\)\\mathcal\{O\}\(dr\), whereddis the model’s hidden dimension andrris the rank\. Our proposal, FrameFT, models the parameter updateΔ​W\\Delta Wwith a sparse coefficient matrix in a Fusion Frame basis\. Fusion Frames can be generatedalgorithmicallyand shared across model layers, enabling very efficient updates\. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint\. The sparse structure of the coefficient matrix in FrameFTandthe sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results\. We evaluate the idea across a suite of supervised fine\-tuning benchmarks, focusing on language tasks, but also report application to vision models\. Our experiments show that FrameFT achieves performance on par with/exceeding state\-of\-the\-art PEFT techniques, but needs far fewer trainable parameters\.

###### Keywords:

Machine Learning, ICML

## 1Introduction

Foundation models show impressive capabilities across a range of domains, including language\([Touvron et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib25);[Team et al\., 2024a](https://arxiv.org/html/2608.26430#bib.bib26);[OpenAI, 2025](https://arxiv.org/html/2608.26430#bib.bib27)\), vision\([Dehghani et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib55)\), and other modalities\([Łajszczak et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib56);[Fang et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib64)\)\. While increasing model size gives improved performance, many downstream applications benefit from an additional fine\-tuning step to specialize the model for specific tasks\. However, fine\-tuning all parameters of a large model is expensive\. This becomes worse when fine\-tuning is required for multiple tasks, each needing to store its own dedicated copy of the model\. To address these difficulties, Parameter\-Efficient Fine\-Tuning \(PEFT\) methods provide an alternative\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28);[Zhang et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib65);[Dettmers et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib39)\)\. These techniques seek to minimize the number of trainable parameters during fine\-tuning, significantly reducing resource needs\. Recent work has shown that PEFT methods can match the performance of full\-model fine\-tuning while updating only a small subset of the parameters\([Bensaïd et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib74);[Zhang et al\., 2026](https://arxiv.org/html/2608.26430#bib.bib75)\)\.

#### LoRA and Sparse Fine\-tuning\.

The most widely used PEFT technique is Low\-Rank Adaptation \(LoRA\)\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28)\), which keeps the pre\-trained weights frozen and injects task\-specific low\-rank matrices into each layer\. These matrices provide the flexibility to adapt the model while maintaining low memory/compute costs\. LoRA’s success has inspired a range of variants that offer improvements in convergence, efficiency, and storage overhead\([Dettmers et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib39);[Hayou et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib43);[Zhu et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib44);[Yaras et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib52);[Xia et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib53)\)\. One line of work uses the Singular Value Decomposition \(SVD\) of pre\-trained weights to construct fixed bases for adaptation\([Lingam et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib70);[Bałazy et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib71)\)\. However, this needs dense singular vectors, which increases the memory footprint and affects inference throughput\. This overhead is problematic in multi\-tenant serving scenarios where there is a single pretrained model and its many fine\-tuned versions\.

In contrast, sparse fine\-tuning \(SFT\) methods use a different strategy: they selectively update a sparse subset of the model’s original parameters\. The selection rule varies: some approaches rely on magnitude\-based pruning\([Lu et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib54)\), while others use sensitivity measures such as Fisher information\([Guo et al\., 2021](https://arxiv.org/html/2608.26430#bib.bib30)\), Fourier\-domain analyses\([Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29)\), or parameter change measures\([Ansell et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib31)\)\. SFT methods can face scalability issues when applied to large models, especially in identifying sparse patterns and tuning hyperparameters\([Guo et al\., 2021](https://arxiv.org/html/2608.26430#bib.bib30)\)\. Some proposals also rely on access to training dynamics\([Ansell et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib31)\)that may not be easily available\.

#### Motivation\.

LoRA’s dense singular vectors and asymmetricA/BA/Bstructure were inherited from matrix approximation methods and shown to work well for fine\-tuning\. SFT methods similarly inherited ideas from model pruning, i\.e\., identify task\-relevant parameters through gradient or magnitude\-based proxies\. We instead take a use case driven approach and ask: how should we parameterize weight updates if we want expressivity, parameter efficiency, stable optimization, and compute scalability simultaneously for fine\-tuning \(in particular\)?Frame Theoryprovides a natural answer\. Briefly, Finite Frames and Fusion Frames are used to study the spanning sets and spanning subspaces of a given finite\-dimensional vector space, respectively\. They have many applications in coding theory, compressed sensing\([Boufounos et al\., 2009](https://arxiv.org/html/2608.26430#bib.bib5)\), quantization\([Adepu et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib32);[Czaja and Na, 2024](https://arxiv.org/html/2608.26430#bib.bib33)\), and dictionary learning\([Hwang et al\., 2019](https://arxiv.org/html/2608.26430#bib.bib62);[Cai et al\., 2014](https://arxiv.org/html/2608.26430#bib.bib63)\)\. Their ability to encode updates in overlapping subspaces offers robustness, flexibility, and efficiency – properties that are relevant for model adaptation\. FrameFT partitions the model’s parameter space into fusion\-frame\-based subspaces, within which updates are learned in a highly sparse and structured manner\.

#### Our proposal: Fine\-tuning with Frames\.

We propose a new PEFT strategy inspired by Fusion Frames\([Christensen, 2018](https://arxiv.org/html/2608.26430#bib.bib3);[Casazza et al\., 2008](https://arxiv.org/html/2608.26430#bib.bib4);[Waldron, 2019](https://arxiv.org/html/2608.26430#bib.bib7)\)\. Fusion Frames allow structured representations by decomposing a space into overlapping subspaces at multiple scales\. Unlike strategies that rely on either low\-rank or sparse updates, FrameFT performs fine\-tuning across multiple structured subspaces, naturally combining the global adaptability of LoRA with the precision of sparse updates\.

#### Contributions\.

Our main contributions are:

\(a\)A new fine\-tuning framework,FrameFT, that utilizes highly structured subspaces \(Fusion Frames\) to capture parameter updates efficiently;\(b\)Extensive empirical evaluations across multiple foundation models and tasks, demonstrating FrameFT’s effectiveness compared to state\-of\-the\-art PEFT baselines;\(c\)A technical analysis of the numerical stability, convergence, and efficiency benefits offered by FrameFT\.

#### Conflict of Interest Disclosure\.

Two of the authors \(LZ, SK\) work at Google DeepMind, where the Gemma models were developed\. The other two authors \(HA, VS\) previously worked at Google DeepMind\.

## 2Frames and Subspace decompositions

We provide a brief review of Finite Frame theory for Hilbert spaces\([Waldron, 2019](https://arxiv.org/html/2608.26430#bib.bib7);[Christensen, 2018](https://arxiv.org/html/2608.26430#bib.bib3)\)\. Readers familiar with these concepts can skip to the next section, where we describe how we apply this idea to fine\-tuning\.

#### Spanning sets and Frames\.

A spanning set is a collection of vectors that spans a finite\-dimensional Hilbert space or vector space, and orthonormal bases are a canonical example\. Frames generalize this concept by allowing redundancy, enabling stable and often more robust representations\.

Figure 1:Examples of tight frames withk=5k=5vectors inℝ3\\mathbb\{R\}^\{3\}\.Letℋd\\mathcal\{H\}^\{d\}denote add\-dimensional Hilbert space\. A sequence of vectorsϕ=\(φi\)i=1k\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}\\upphi\}=\(\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}\\varphi\}\_\{i\}\)\_\{i=1\}^\{k\}inℋd\\mathcal\{H\}^\{d\}is called a frame if there exist constants0<A≤B<∞0<A\\leq B<\\inftysuch that for allx∈ℋdx\\in\\mathcal\{H\}^\{d\},

A​‖x‖2≤∑i=1k\|⟨x,φi⟩\|2≤B​‖x‖2,\\displaystyle A\|\|x\|\|^\{2\}\\leq\\sum\_\{i=1\}^\{k\}\|\\langle x,\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}\\varphi\}\_\{i\}\\rangle\|^\{2\}\\leq B\|\|x\|\|^\{2\},\(1\)where⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangleis the inner product inℋd\\mathcal\{H\}^\{d\}, andAAandBBare known as the frame bounds\. Typically,k≥dk\\geq dallows the frame to represent vectors with redundancy\. An example is illustrated in Figure[1](https://arxiv.org/html/2608.26430#S2.F1)withk=5k=5vectors inℝ3\\mathbb\{R\}^\{3\}\. We can project any vector inℝ3\\mathbb\{R\}^\{3\}onto these frame vectors without distorting the signal\([Casazza and Kutyniok, 2012](https://arxiv.org/html/2608.26430#bib.bib24)\)\. Throughout the paper, we operate in real\-valued Euclidean spacesℝd\\mathbb\{R\}^\{d\}instead of a more general Hilbert space for simplicity\.

Fusion frames extend the above idea to settings where subspace decomposition is also desired \(instead of redundancy alone\)\. Given a collection of subspaces\{𝒲i\}i=1k\\\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\_\{i\}\\\}\_\{i=1\}^\{k\}inℝd\\mathbb\{R\}^\{d\}and a corresponding set of positive weights\{wi\}i=1k\\\{w\_\{i\}\\\}\_\{i=1\}^\{k\}, the collection\(𝒲i,wi\)i=1k\{\(\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\_\{i\},w\_\{i\}\)\}\_\{i=1\}^\{k\}forms a fusion frame if there exist constants0<A≤B<∞0<A\\leq B<\\inftysuch that for allx∈ℝdx\\in\\mathbb\{R\}^\{d\},

A​‖x‖2≤∑i=1kwi2​‖Pi​x‖2≤B​‖x‖2,\\displaystyle A\|\|x\|\|^\{2\}\\leq\\sum\_\{i=1\}^\{k\}w\_\{i\}^\{2\}\|\|\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}x\|\|^\{2\}\\leq B\|\|x\|\|^\{2\},wherePi\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}denotes the orthogonal projection onto the subspace𝒲i\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}\\mathcal\{W\}\}\_\{i\}\. The weightswiw\_\{i\}adjust the influence of each subspace, andAA,BBare fusion frame bounds\. A fusion frame is tight ifA=BA=B, and Parseval ifA=B=1A=B=1\. We will focus on Parseval fusion frames\. If all weightswiw\_\{i\}are equal to11, we write the fusion frame simply as\{𝒲i\}i=1k\\\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\_\{i\}\\\}\_\{i=1\}^\{k\}\. Frame bounds will help later in our convergence analysis\.

### 2\.1Operators in Fusion Frames

Let\(\(𝒲i,wi\)\)i=1k\{\(\(\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\_\{i\},w\_\{i\}\)\)\}\_\{i=1\}^\{k\}be a Fusion Frame forℝd\\mathbb\{R\}^\{d\}, with orthogonal projection matrices\(Pi\)i=1k\(\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}\)\_\{i=1\}^\{k\}respectively\. We define three operators:

The analysis operator𝒯𝒲:ℝd→⨁i=1k𝒲i\\mathcal\{T\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}:\\mathbb\{R\}^\{d\}\\to\\bigoplus\_\{i=1\}^\{k\}\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\_\{i\}maps a vector to its projections across all subspaces:

𝒯𝒲​\(x\)=\(wi​PiT​x\)i=1k\.\\mathcal\{T\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}\(x\)=\(w\_\{i\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}^\{T\}x\)\_\{i=1\}^\{k\}\.\(2\)The synthesis operator𝒯𝒲∗:⨁i=1k𝒲i→ℝd\\mathcal\{T\}^\{\*\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}:\\bigoplus\_\{i=1\}^\{k\}\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\_\{i\}\\to\\mathbb\{R\}^\{d\}“re\-builds” a vector from its fusion frame representation:

𝒯𝒲∗​\(\(yi\)i=1k\)=∑i=1kwi​Pi​yi\.\\mathcal\{T\}^\{\*\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}\(\(\{\\color\[rgb\]\{0,1,0\}y\}\_\{i\}\)\_\{i=1\}^\{k\}\)=\\sum\_\{i=1\}^\{k\}w\_\{i\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}\{\\color\[rgb\]\{0,1,0\}y\}\_\{i\}\.\(3\)The fusion frame operator is the composition of the two operators above,𝒮𝒲=𝒯𝒲∗​𝒯𝒲\\mathcal\{S\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}=\\mathcal\{T\}^\{\*\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}\\mathcal\{T\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}, defined by:

𝒮𝒲​\(x\)=∑i=1kwi2​Pi​PiT​x\.\\mathcal\{S\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}\(x\)=\\sum\_\{i=1\}^\{k\}w\_\{i\}^\{2\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{i\}^\{T\}x\.\(4\)This operator is self\-adjoint, positive semi\-definite, and bounded\. For Parseval fusion frames, we have𝒮𝒲=Id\\mathcal\{S\}\_\{\{\\color\[rgb\]\{0,0,1\}\\mathcal\{W\}\}\}=I\_\{d\}\(allow exact reconstruction\)\. These operators help in mapping vectors inℝd\\mathbb\{R\}^\{d\}to their fusion frame representations and back\.

## 3Fine\-tuning in Fusion Frame Subspaces

We will start by analyzing the subspace updates in LoRA\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28)\)\. The key idea is representing weight updates through a low\-rank decomposition\. For a layerllwith pre\-trained parametersWl∈ℝm×nW\_\{l\}\\in\\mathbb\{R\}^\{m\\times n\}, LoRA decomposes the weight updateδ​Wl\\delta W\_\{l\}as a product of two rank\-deficient matrices:

Wl′=Wl\+δ​Wl=Wl\+Bl​Al\\displaystyle W\_\{l\}^\{\\prime\}=W\_\{l\}\+\\delta W\_\{l\}=W\_\{l\}\+B\_\{l\}A\_\{l\}\(5\)whereBl∈ℝm×rB\_\{l\}\\in\\mathbb\{R\}^\{m\\times r\},Al∈ℝr×nA\_\{l\}\\in\\mathbb\{R\}^\{r\\times n\}, andr<min⁡\(m,n\)r<\\min\(m,n\)\.

To better understand the structure of these updates, we can check the Singular Value Decomposition \(SVD\) ofδ​Wl\\delta W\_\{l\}:

δ​Wl=U​Σ​VT=Ur​Σr​VrT\\displaystyle\\delta W\_\{l\}=U\\Sigma V^\{T\}=U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}\(6\)whereU∈ℝm×mU\\in\\mathbb\{R\}^\{m\\times m\}andV∈ℝn×nV\\in\\mathbb\{R\}^\{n\\times n\}are best viewed as orthogonal matrices spanning theoutputandinputspaces respectively, andΣ∈ℝm×n\\Sigma\\in\\mathbb\{R\}^\{m\\times n\}is a rectangular diagonal matrix\. Since the rank ofδ​Wl\\delta W\_\{l\}is constrained torr, we can simplify this product toUr​Σr​VrTU\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{T\}, whereΣr\\Sigma\_\{r\}contains only the toprrsingular values andUrU\_\{r\},VrV\_\{r\}are the corresponding left and right singular vectors\. This decomposition provides insight into LoRA updates:VrV\_\{r\}defines a subspace in theinput spaceℝn\\mathbb\{R\}^\{n\}where the input is first projected\. Then,Σr\\Sigma\_\{r\}scales these projections, andUrU\_\{r\}maps them to a subspace of theoutput spaceℝm\\mathbb\{R\}^\{m\}\. These input and output subspaces emergeimplicitlyas a by\-product of training the matricesBBandAA, without direct control over their properties or interactions\.

#### Asymmetricity\.

Recent results indicate that the contribution of matricesAAandBBisasymmetric, which can lead to instability\([Hayou et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib43)\), or failure to converge to the optimal solution\([Malinovsky et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib34)\)\. This suggests two key opportunities for improvement\.

\(i\)Can we explicitly choose and work with multiple subspaces in both input spaceℝn\\mathbb\{R\}^\{n\}and output spaceℝm\\mathbb\{R\}^\{m\}to better capture parameter updates?\(ii\)Can we maintain precise control over how these subspaces interact during training?

While existing work has explored defining subspaces using Orthogonal bases derived from the pretrained weights\([Lingam et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib70);[Sun et al\., 2024a](https://arxiv.org/html/2608.26430#bib.bib58)\)or fixed spectral bases such as the Fourier basis\([Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29)\), there is a significant cost in storage and impact on inference throughput due to dense operations\. We will see how Fusion frames addressboth questionsdirectly\. Specifically, we can decompose the input and output spaces into potentially overlapping subspaces with controlled relationships\. Further, we can achieve good storage efficiency and inference throughput due to the inherent sparsity of the generated frames for free\.

### 3\.1Fusion Frames for Finetuning Weight Updates

For the first problem, we want to decompose the input and output spaces into multiple subspaces\. The Fusion Frame operators in §[2\.1](https://arxiv.org/html/2608.26430#S2.SS1)can be used\. In particular, let\(Pn,iT\)i=1k\(\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}^\{T\}\_\{n,i\}\)\_\{i=1\}^\{k\}and\(Pm,iT\)i=1k\(\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}^\{T\}\_\{m,i\}\)\_\{i=1\}^\{k\}be the orthogonal projection matrices for decomposing the input \(ℝn\\mathbb\{R\}^\{n\}\) and output \(ℝm\\mathbb\{R\}^\{m\}\) spaces intokksubspaces with dimensionsρn\\rho\_\{n\}andρm\\rho\_\{m\}, respectively\. Let,

Pn=\[Pn,1,Pn,2,…,Pn,k\]Pm=\[Pm,1,Pm,2,…,Pm,k\]\\displaystyle\\begin\{split\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}=\\left\[\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n,1\},\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n,2\},\\dots,\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n,k\}\\right\]\\\\ \{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\}=\\left\[\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m,1\},\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m,2\},\\dots,\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m,k\}\\right\]\\end\{split\}\(7\)We can model rich interactions between these subspaces by using trainable coefficient matrices, in contrast to the diagonal matrixΣr\\Sigma\_\{r\}in LoRA\. To be specific, we define\(Cl,i\)i=1k\(C\_\{l,i\}\)\_\{i=1\}^\{k\}where eachCl,i∈ℝρm×ρnC\_\{l,i\}\\in\\mathbb\{R\}^\{\\rho\_\{m\}\\times\\rho\_\{n\}\}encodes the relationships within theii\-th input and output subspaces\. These matrices can be neatly organized into a block diagonal structure:Cl=diag​\(Cl,1,Cl,2,…,Cl,k\)C\_\{l\}=\\text\{diag\}\(C\_\{l,1\},C\_\{l,2\},\\dots,C\_\{l,k\}\)which yields our simple parameter update equation:

Wl′=Wl\+δ​Wl=Wl\+αm​n​Pm​Cl​PnTW\_\{l\}^\{\\prime\}=W\_\{l\}\+\\delta W\_\{l\}=W\_\{l\}\+\\frac\{\\alpha\}\{\\sqrt\{mn\}\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\}C\_\{l\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}^\{T\}Expanding this expression with the subspaces, we get:

Wl′=Wl\+αm​n​∑i=1kPm,i​Cl,i​Pn,iTW\_\{l\}^\{\\prime\}=W\_\{l\}\+\\frac\{\\alpha\}\{\\sqrt\{mn\}\}\\sum\_\{i=1\}^\{k\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m,i\}C\_\{l,i\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n,i\}^\{T\}Here,α\\alphais a scaling hyperparameter, normalized bym​n\\sqrt\{mn\}\(for dimension independence\)\. Fig\.[2](https://arxiv.org/html/2608.26430#S3.F2)\(left\) shows the difference between SVD in LoRA and FrameFT\. The projection matricesPn\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}andPm\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\}are obtained from Tight Fusion Frames \(TFF\) discussed in §[3\.2](https://arxiv.org/html/2608.26430#S3.SS2)\. Under certain conditions, these TFFs exhibit Equichordal and Equi\-Isoclinic properties\([Fickus et al\., 2023a](https://arxiv.org/html/2608.26430#bib.bib1)\), maximizing subspaceseparationfor efficient representations\. These TFFs remainfixedandonly the coefficient matrices\(Cl,i\)i=1k\(C\_\{l,i\}\)\_\{i=1\}^\{k\}are trainable\. Fig\.[2](https://arxiv.org/html/2608.26430#S3.F2)\(right\) shows FrameFT update for a single layer \(also see Alg\.[1](https://arxiv.org/html/2608.26430#alg1)\)\.

0:Frozen Parameters

WlW\_\{l\}, Fusion Frame number of subspaces

kk, subspace dimensions

ρm,ρn\\rho\_\{m\},\\rho\_\{n\}, number of non\-zero coefficients

ncn\_\{c\}, non\-zero coefficients

GG, scalar

α\\alpha
1:

mm,

nn= shape\(

WlW\_\{l\}\)// get the shape of the parameter matrix

2:

Pm\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\},

Pn\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}= TFF\(

k,ρm,mk,\\rho\_\{m\},m\), TFF\(

k,ρn,nk,\\rho\_\{n\},n\)// generate the TFFs

3: Coeffs = Init\(m, n, size=

ncn\_\{c\}\)// initialize the coefficients matrix

4:

ClC\_\{l\}= rearrange\(Coeffs,

mm,

nn,

GG\)// re\-arrange the coefficients in dense format

5:

δ​Wl=αm​n​Pm​Cl​PnT\\delta W\_\{l\}=\\frac\{\\alpha\}\{\\sqrt\{mn\}\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\}C\_\{l\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}^\{T\}//compute the update term

6:

Wl′=Wl\+δWlW\_\{l\}^\{\{\}^\{\\prime\}\}=W\_\{l\}\+\\delta W\_\{l\}// update the parameter matrix

Algorithm 1FrameFT: Finetuning with Frames![Refer to caption](https://arxiv.org/html/2608.26430v1/fig/svd_vs_FF_FrameFT_merge.png)Figure 2:\(left\) How parameter updateδ​W\\delta Wis modeled by LoRA and by FrameFT\. \(right\) The parameter update rule in FrameFT\. We freeze the Fusion Frame projection matrices and train only the coefficientsClC\_\{l\}\.
### 3\.2Constructing Tight Fusion Frames

We use uniform tight fusion frames, where all subspaces have thesamedimension\. In practice, one can easily use different subspace dimensions across layers\. To construct a\(k,ρ,d\)\(k,\\rho,d\)uniform tight fusion frame wherekkis the number of subspaces,ρ\\rhois the dimension of each subspace, andddis the dimension of the entire space, we use the Spectral Tetris algorithm described in\([Casazza et al\., 2011](https://arxiv.org/html/2608.26430#bib.bib6)\), which is summarized as:

\(1\)Construct a unit norm tight frame \(UNTF\) inℂρ\\mathbb\{C\}^\{\\rho\}withddvectors\.\(2\)Modulate these vectors using thekthk^\{\\text\{th\}\}roots of unity to formkksubspaces of dimensionρ\\rhoinℂd\\mathbb\{C\}^\{d\}\.\(3\)Use the method in\([Fickus et al\., 2023b](https://arxiv.org/html/2608.26430#bib.bib8)\)to extend the result to real\-valued spaces\.

This process ensures that the resultant set satisfies the tight fusion frame conditions\. An example is included in Appendix §[H](https://arxiv.org/html/2608.26430#A8)\.

### 3\.3Structured Coefficient Matrix

Prior work on Sparse Fine\-Tuning methods\([Ansell et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib31)\)suggests that we can boost parameter efficiency by introducing sparsity into the coefficient matrices\(Cl,i\)i=1k\(C\_\{l,i\}\)\_\{i=1\}^\{k\}\. This sparsification nicely complements our fusion frame architecture: while fusion frames providestructured subspace decompositions of the parameter space, sparse coefficients identify themost useful subspace interactions\(see off\-diagonal terms in Fig\.[2](https://arxiv.org/html/2608.26430#S3.F2)\)\. Note that the Fusion Frames themselves are sparse, which isdistinctfrom the sparsity in the coefficient matrix\. Both are exploited in our implementation\. Our experiments indicate that a simple random sparsity pattern performs well: we randomly designate a small subset of entries inClC\_\{l\}as non\-zero, distributing them uniformly across the block\-diagonal matrices\(Cl,i\)i=1k\(C\_\{l,i\}\)\_\{i=1\}^\{k\}\. Figure[3](https://arxiv.org/html/2608.26430#S3.F3)shows this sparse structure with an example from RoBERTa\-Large\.

![Refer to caption](https://arxiv.org/html/2608.26430v1/fig/c3.png)Figure 3:An example of the coefficient matrixClC\_\{l\}from a Query layer \(RoBERTa\-L\) with different subspace dimensionsρm=ρn=ρ\\rho\_\{m\}=\\rho\_\{n\}=\\rho\.The memory requirements for storing the updates reduce fromO⁡\(k​ρm​ρn\)O\(k\\rho\_\{m\}\\rho\_\{n\}\)toO⁡\(s\)O\(s\), wheres≪k​ρm​ρns\\ll k\\rho\_\{m\}\\rho\_\{n\}\. To reduce memory overhead even further, we share these sparsity patterns across layers, i\.e\.,supp​\(Cl\)=supp​\(Cl′\)\\text\{supp\}\(C\_\{l\}\)=\\text\{supp\}\(C\_\{l^\{\\prime\}\}\)for layersllandl′l^\{\\prime\}, wheresupp​\(⋅\)\\text\{supp\}\(\\cdot\)represents the sets of index locations\. The cross\-layer sharing of sparsity patterns not only reduces memory requirements but also suggests that subspace interactions necessary for task adaptation may share structural patterns across network layers\. An ablation study we present in §[4\.6](https://arxiv.org/html/2608.26430#S4.SS6)confirms this empirically: shared and per\-layer random masks perform similarly \(plus/minus noise of each other\)\.

### 3\.4System Efficiency and Scalability

Our design choices, specifically the use of algorithmically generated Fusion Frames and sparse coefficients, give a number of advantages over alternatives\.

#### Storage efficiency\.

Since Spectral Tetris generates Fusion Frames algorithmically, we eliminate the need to store basis matrices\. Only the sparse coefficient matrices must be serialized\. For a Llama\-2\-7B model, this reduces the checkpoint size to just 1\.28 MB \(vs\. 67\.1 MB for LoRA rank\-32\), facilitating ultra\-low\-latency “hot\-swapping” of adapters in multi\-tenant serving scenarios where a single backbone supports thousands of fine\-tuned models for different users\. Both numbers above assume FP32 precision\. The frozen backbone operates in FP16/BF16 during training, but adapter parameters can be maintained in FP32 to preserve dynamic range/gradient fidelity\.

#### Memory Overhead\.

Unlike SVD\-based methods such as SVFit\([Sun et al\., 2024a](https://arxiv.org/html/2608.26430#bib.bib58)\)and LoRA\-XS\([Bałazy et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib71)\), which require unique basis vectors for each layer, Fusion Frames generated by the Spectral Tetris algorithm dependonlyon the layer dimensions\. This allows a single set of frames to beshared across all layerswith the same dimensions, significantly reducing the initialization time \(see Table[11](https://arxiv.org/html/2608.26430#A8.T11)\) and runtime memory footprint\.

#### Inference throughput\.

While the orthogonal bases generated by SVD based methods are dense, Spectral Tetris directly constructs frames with inherent block\-sparsity \(see §[10](https://arxiv.org/html/2608.26430#A8.T10)\)\. This structural advantage directly translates to higher inference throughput \(see Table[2](https://arxiv.org/html/2608.26430#S4.T2)\)\.

Beyond these practical efficiencies, the construction of FrameFT also offers useful theoretical advantages for optimization\. Next, we analyze how preserving the smoothness of the loss landscape allows stable convergence properties\.

### 3\.5Analysis of FrameFT Convergence properties

Our formulation of the parameter updates using Tight Fusion Frames and sparse coefficients enjoys theoretical benefits compared to LoRA\. The analysis in[Sun et al\. \(2024b\)](https://arxiv.org/html/2608.26430#bib.bib35)describes how, for Lipschitz\-smooth loss functions, LoRA can create a non\-smooth landscape when projected onto the parameters inAAandBB\. Via Lemma[3\.1](https://arxiv.org/html/2608.26430#S3.EGx6), we show that FrameFT preserves Lipschitz\-smoothness: when a function exhibits Lipschitz smoothness w\.r\.t\.WW, this property holds when remapped onto our coefficient spaceClC\_\{l\}\.

###### Lemma 3\.1\.

For any differentiable functionf⁡\(W\)f\(W\)that isLL\-Lipschitz smooth under Frobenius norm, with fusion frames characterized by frame bounds\(Am,Bm\)\(A\_\{m\},B\_\{m\}\)and\(An,Bn\)\(A\_\{n\},B\_\{n\}\)for output and input spaces, respectively, the transformed functionf⁡\(Cl\)=f⁡\(W0\+Pm​Cl​PnT\)f\(C\_\{l\}\)=f\(W\_\{0\}\+\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\}C\_\{l\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}^\{T\}\)obeys:

‖∇f​\(Cl1\)−∇f​\(Cl2\)‖F≤L​Bm​Bn​‖Cl1−Cl2‖F\\displaystyle\|\|\\nabla f\(C\_\{l\}^\{1\}\)\-\\nabla f\(C\_\{l\}^\{2\}\)\|\|\_\{F\}\\leq LB\_\{m\}B\_\{n\}\|\|C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\|\|\_\{F\}

Note thatf⁡\(W\)f\(W\)denotes the global loss function of the entire deep neural network, defined over the collection of all layer weightsW=\{W1,…,WD\}W=\\\{W\_\{1\},\\dots,W\_\{D\}\\\}just like in\([Sun et al\., 2024b](https://arxiv.org/html/2608.26430#bib.bib35), see p\. 13\)\. Thus,ffimplicitly encodes the composition of all network layers\. During fine\-tuning, with pre\-trained weights frozen, the loss landscape is reparameterized purely as a function of the coefficient matricesf⁡\(C\)f\(C\), whereC=\{C1,…,CD\}C=\\\{C\_\{1\},\\dots,C\_\{D\}\\\}represents the set of learnable Frame coefficients across all layers\.

Since FrameFT preserves smoothness, we can immediately invoke a broad set of convergence results for Lipschitz\-smooth functions\([Bubeck, 2015](https://arxiv.org/html/2608.26430#bib.bib19);[Zhou and Cong, 2017](https://arxiv.org/html/2608.26430#bib.bib18)\)\. We highlight one result: gradient descent constrained to learning ratesη≤1/L~\\eta\\leq 1/\\tilde\{L\}achieves the expected1/T1/Tconvergence rate to first\-order stationary points \(in Theorem[3\.2](https://arxiv.org/html/2608.26430#S3.Thmtheorem2)below\)\. Additional details are provided in §[B](https://arxiv.org/html/2608.26430#A2)\.

###### Theorem 3\.2\.

Consider minimizingf⁡\(Cl\)=f⁡\(W0\+Pm​Cl​PnT\)f\(C\_\{l\}\)=f\(W\_\{0\}\+\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{m\}C\_\{l\}\{\\color\[rgb\]\{0\.1,0\.05,0\.96\}P\}\_\{n\}^\{T\}\)where:f⁡\(Cl\)f\(C\_\{l\}\)isL~\\tilde\{L\}\-Lipschitz smooth \(but potentially non\-convex\) andf⁡\(Cl\)f\(C\_\{l\}\)is lower bounded byf∗f^\{\*\}\. For gradient descent with step sizeη≤1/L~\\eta\\leq 1/\\tilde\{L\}, running gradient descent forTTiterations satisfies:

\(1/T\)​∑t=0T−1‖∇f​\(Clt\)‖F2≤2​\(f⁡\(Cl0\)−f∗\)/\(η​T\)\\displaystyle\(1/T\)\\sum\_\{t=0\}^\{T\-1\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\\leq 2\(f\(C\_\{l\}^\{0\}\)\-f^\{\*\}\)/\(\\eta T\)

This analysis offers a unified view of subspace\-based fine\-tuning\. Our results can be extended to methods such as SVFit and FourierFT by substituting the appropriate projection matrices and bounds\. Lemma[3\.1](https://arxiv.org/html/2608.26430#S3.EGx6)explicitly links the Lipschitz constant of the reparameterized lossf⁡\(C\)f\(C\)to the upper frame boundsBm,BnB\_\{m\},B\_\{n\}\. This result directly guides our design choice: to maximize the smoothness of the loss landscape, we normalize our Frame bounds to unity \(Parseval frames\) for all experiments\. To summarize, Lemma[3\.1](https://arxiv.org/html/2608.26430#S3.EGx6)informs the design choices via the boundL~=L​Bm​Bn\\tilde\{L\}=LB\_\{m\}B\_\{n\}\.

## 4Experiments

In this section, we compare the performance of Transformer\-based models fine\-tuned with FrameFT\. All our experiments are conducted on two NVIDIA A100 GPUs with 40GB of memory each\. Individual subsections include more details on the experimental setup\. Additional experiments on varying hyperparameters and discussions on subspace dimensions are available in the Appendix §[D](https://arxiv.org/html/2608.26430#A4), §[E](https://arxiv.org/html/2608.26430#A5), and §[F](https://arxiv.org/html/2608.26430#A6)\.

### 4\.1Natural Language Understanding Capabilities

Table 1:Fine\-Tuning RoBERTa Base and Large models on GLUE benchmark\. \* indicates the results reported in prior work\. FrameFT performs better than full fine\-tuning and LoRA, using10×10\\timesfewer parameters\.#### Evaluation framework:

We evaluate the performance of FrameFT by fine\-tuning the base and large variants of RoBERTa\([Liu et al\., 2019](https://arxiv.org/html/2608.26430#bib.bib38)\)across multiple GLUE benchmark tasks\([Wang et al\., 2019](https://arxiv.org/html/2608.26430#bib.bib36)\)\. This suite of benchmarks covers sentiment classification, paraphrase detection, and entailment recognition\. It is a standardized setup for testing\.

#### Fine\-tuning strategy:

We use the original LoRA\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28)\)recipe by fine\-tuning the Query and Value matrices across network layers\. For FrameFT, we use10001000non\-zero coefficient entries with their positions determined randomly and shared across layers\. For tight fusion frame construction, we use a subspace dimensionρ=2\\rho=2, and the subspace countkkis calibrated so thatk​ρ=nk\\rho=n\(nnis layer dimension\)\.

#### Performance analysis:

We report the Pearson correlation coefficient \(PCC\) for the STS\-B task, Matthews correlation coefficient \(MCC\) for CoLA and accuracy for the remaining tasks\. For the baseline methods, we report LoRA\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28)\), AdaLoRA\([Zhang et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib69)\), FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29)\), SVFit\([Sun et al\., 2024a](https://arxiv.org/html/2608.26430#bib.bib58)\), SMT\([He et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib72)\), SVFT\([Lingam et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib70)\)and RoseLoRA\([Wang et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib61)\)\. The results are presented in Table[1](https://arxiv.org/html/2608.26430#S4.T1)\. We see that FrameFT performs on par or better than LoRA and full fine\-tuning on individual tasks, and performs better than all the baselines on average\. FrameFT achieves this with10×10\\timesfewer parameters when compared to LoRA\. We also note that even though the number of parameters of SVFit\([Sun et al\., 2024a](https://arxiv.org/html/2608.26430#bib.bib58)\)is slightly lower than FrameFT, SVFit trains the singular values, keeping the singular vectors fixed\. So, in practice, one would need to save the singular vectors for each layer after training, which increases the storage requirements\. Also, for RoseLoRA\([Wang et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib61)\), only the parameter count is reported in Table[1](https://arxiv.org/html/2608.26430#S4.T1), but they use a different mask for each layer, so the storage cost is3×3\\timesthe number of parameters if we account for the location of the non\-zero coefficients\.

### 4\.2Instruction Tuning

#### Benchmarking Framework:

We evaluate FrameFT for fine\-tuning LLMs to follow instructions\. We finetune 7B and 13B variants of Llama2 models\([Touvron et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib25)\), Gemma2 models\([Team et al\., 2024b](https://arxiv.org/html/2608.26430#bib.bib45)\)with 2B and 9B parameters and Llama 3\.1\([Grattafiori et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib46)\)8B model on the Alpaca instruction dataset\([Taori et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib37)\)\. We evaluate the performance of the fine\-tuned models on the LM\-evaluation harness\([Gao et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib57)\)by Eleuther AI\. We use eight distinct challenge categories spanning reasoning, world knowledge, and generalization capabilities\.

#### FrameFT configuration:

We apply FrameFT withn=5000n=5000non\-zero coefficients and adapt the Query and Value matrices for all the Transformer blocks\. We determine the position of the non\-zero coefficients at random and share them across all the layers\. We useα=200\\alpha=200across all of our experiments\. A sensitivity analysis sweepingα∈\{10,100,400,600\}\\alpha\\in\\\{10,100,400,600\\\}for Llama\-2\-7B is in the Appendix[I](https://arxiv.org/html/2608.26430#A9)\. We set the subspace dimensionρ=2\\rho=2and as before, calculate the number of subspaceskksuch thatk​ρ=nk\\rho=nfor each layer\. For all baselines, we choose the hyperparameters suggested for the respective method\.

#### Performance Analysis:

We compare FrameFT with LoRA\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28)\),\(IA\)3\(\\text\{IA\}\)^\{3\}\([Liu et al\., 2022a](https://arxiv.org/html/2608.26430#bib.bib41)\), DoRA\([Liu et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib42)\),S2​F​T\\text\{S\}^\{2\}\{FT\}\([Yang et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib40)\), SVFT\([Lingam et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib70)\), and FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29)\)\. Our results are shown in Table[3](https://arxiv.org/html/2608.26430#S4.T3)\. We observe that FrameFT performs on par or better compared to the baselines while using the fewest number of parameters across all models\. In addition, FrameFT also provides other compute benefits we discuss in Section[4\.4](https://arxiv.org/html/2608.26430#S4.SS4)\.

### 4\.3Performance on Vision transformer models

We performed a set of experiments to determine whether FrameFT was effective only for Language models\. To check this, we applied FrameFT to fine\-tune Vision Transformers on 8 image classification tasks, which include remote sensing, fine\-grained classification, and texture recognition\.

#### Performance Analysis:

Section[C](https://arxiv.org/html/2608.26430#A3)in the appendix shows the performance of FrameFT across these tasks for ViT\-L and ViT\-B models\.

Table 2:Tokens per second for different PEFT methods\.We observe that FrameFT performs better than the baseline methods \(which include full\-finetuning\) on average\. Moreover, we observe performance improvements across the majority of the tasks when we increase the number of parameters for FrameFT\. These results indicate that FrameFT generalizes well across both Vision and Language models\. More experimental details are presented in Appendix[C](https://arxiv.org/html/2608.26430#A3)and Table[8](https://arxiv.org/html/2608.26430#A7.T8)\.

![Refer to caption](https://arxiv.org/html/2608.26430v1/fig/frameFT_vs_n_ff_coeffs.png)Figure 4:Performance of RoBERTa base model fine\-tuned with FrameFT versus the number of non\-zero coefficients\. The red line indicates the performance of full fine\-tuning\.

### 4\.4Operational Efficiency/Latency Analysis

In this section, we analyze the operational efficiency of FrameFT in terms of storage requirements and inference throughput\. In practice, when serving LLMs to a large user base, the pretrained model is held fixed, and various specialized variants are maintained simultaneously\. Hence, we measure the throughput of the adapter layer introduced by different PEFT methods, with the idea that the pretrained layer throughput remains the same in the pretrained and fine\-tuned models\.

#### Inference throughput:

Table[2](https://arxiv.org/html/2608.26430#S4.T2)displays the tokens per second for various methods across different model families\. All measurements are performed on an NVIDIA A100 machine\. We observe that FrameFT performs better than LoRA, which in turn performs better than FourierFT and other methods\. These measurements are near\-deterministic on fixed hardware\. Across 100 iterations the coefficient of variation is below 12%, and the FrameFT–LoRA gap is large enough that variance does not affect the conclusion \(e\.g\., on Llama\-2\-7B:37\.9±4\.537\.9\\pm 4\.5k for FrameFT vs\.24\.4±2\.724\.4\\pm 2\.7k tokens/sec for LoRA\)\. This highlights FrameFT’s efficiency benefits due to sparse coefficients and sparse projection matrices as described in §[3\.3](https://arxiv.org/html/2608.26430#S3.SS3)and Appendix §[H](https://arxiv.org/html/2608.26430#A8)

#### Storage and Memory Footprint\.

A key advantage of FrameFT is its minimal storage footprint for task\-specific adapter checkpoints\. Since Fusion Frames are generated algorithmically once and shared across layers of the same dimension, the basis vectors \(Pm,PnP\_\{m\},P\_\{n\}\) do not need to be stored or transmitted\. We only store the sparse coefficient matrices\. This is a clear advantage compared to SVD\-based methods such as SVFT and LoRA\-XS, where one needs to store the dense orthogonal basis for each layer, increasing their storage/memory requirements\.

We note that constructing the Fusion Frames takes a fixed amount of time\. Specifically, the time complexity for constructing a\(k,ρ,d\)\(k,\\rho,d\)TFF is𝒪⁡\(k​d\)\\mathcal\{O\}\(kd\)\. However, this generation step is performed onlyonceduring initialization, and the resulting projection matrices are shared across all network layers, amortizing the one\-time compute cost across multiple layers and forward calls \(see §[H\.3](https://arxiv.org/html/2608.26430#A8.SS3)for wall clock time\)\. So, we focus on per\-layer throughput measurements\.

Table 3:Performance of LLMs fine\-tuned on the Alpaca dataset by various methods and then evaluated on the LM\-evaluation\-harness\. FrameFT performs competitively with all the baseline methods under comparison, using a30×30\\timesfewer number of parameters than LoRA\. FrameFT needs minimal hyperparameter tuning\.

### 4\.5Performance vis\-a\-vis non\-zero coefficients

We evaluate the performance of FrameFT as we increase the number of non\-zero coefficients\. We choose the RoBERTa base model for this experiment and check the performance of the model fine\-tuned with FrameFT on different tasks in the GLUE benchmark\. Figure[4](https://arxiv.org/html/2608.26430#S4.F4)shows expected trends: the performance of the fine\-tuned model on different tasks increases as we increase the number of non\-zero terms in the coefficient matrix\. Table[6](https://arxiv.org/html/2608.26430#A3.T6)also confirms this behavior as we increasennfrom10001000to50005000\. These results show that FrameFT is able to utilize the additional parameters to improve the fine\-tuning performance for both vision and language tasks\. Additional experiments on more tasks from the GLUE benchmark are included in appendix[E](https://arxiv.org/html/2608.26430#A5)\.

### 4\.6Sparsity Pattern and Cross\-Layer Sharing Ablation

In this experiment, we ablate the choice of random shared sparsity by comparing four masking strategies – a random shared mask across all layers, a per layer random mask, magnitude\-based non\-zero coefficients selection and a gradient\-based mask for FrameFT\. We evaluate the performance of each of these methods for finetuning RoBERTA\-base model on the GLUE benchmark\. For the random mask strategies, we average the performance accross five different seeds each\. Results are shown in Table[4](https://arxiv.org/html/2608.26430#S4.T4)\.

Table 4:Sparsity mask ablation on RoBERTa\-Base/GLUE \(averaged over five seeds\)\. Shared and per\-layer random masks perform within noise of each other while adaptive heuristics perform strictly worse\.We see that cross\-layer sharing incurs no expressivity cost: shared and per\-layer masks are indistinguishable within variance\. This aligns with Lemma[3\.1](https://arxiv.org/html/2608.26430#S3.EGx6): the optimization geometry depends on the Fusion Frame bounds and not on the specific entries of the coefficient matrix, which are non\-zero\. Interestingly, adaptive heuristics perform strictly worse\. Gradient\-based masks collapse to 74\.83 average, lower than random\. We suspect this is due to task\-relevant subspace interactions being broadly distributed across the coefficient matrix rather than concentrated in high\-magnitude or high\-gradient pretrained weights\. Random sampling provides better coverage of this distributed structure than greedy selection\.

### 4\.7Training Time and GPU Memory

We measured training time and peak GPU memory across all five models evaluated in Section[4\.2](https://arxiv.org/html/2608.26430#S4.SS2)\. Table[5](https://arxiv.org/html/2608.26430#S4.T5)reports these measurements\. FrameFT trains 5\-7% faster than LoRA and within 0\.6 GiB of its peak GPU memory\. The modest memory gap reflects the fact that the frozen pretrained backbone dominates the total GPU footprint\. Both these measurements use standard PyTorch without custom kernels\. Optimized kernels will be available on the GitHub repository shortly\.

Table 5:Training time \(minutes\) and peak GPU memory \(GiB\) for LoRA and FrameFT on the Alpaca instruction tuning task\.

## 5Related work

Model adaptation for downstream tasks has been studied extensively in recent years, and has provided various efficient methods that reduce computation and storage needs while maintaining performance\. Here, we describe different variants of PEFT methods briefly introduced in §[1](https://arxiv.org/html/2608.26430#S1)\.

#### Adapters:

Adapters introduce specialized modules between pre\-existing layers within the pretrained model\. These adapter layers are trained during fine\-tuning while keeping the pretrained model parameters frozen\([Houlsby et al\., 2019](https://arxiv.org/html/2608.26430#bib.bib2);[Karimi Mahabadi et al\., 2021](https://arxiv.org/html/2608.26430#bib.bib15);[He et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib16)\)\. By keeping the parameters of the original model frozen, they preserve the knowledge acquired during pretraining while reducing the risk of overfitting\.

#### Low\-Rank Matrix Factorization:

LoRA techniques reparameterize the weight updates on selected layers of the model through low\-rank factorizations\([Hu et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib28)\)\. This framework of training only the decomposition matrices while freezing pretrained parameters has led to many variants exploring asymmetric chaining\([Malinovsky et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib34)\), quantization\-aware formulations\([Dettmers et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib39)\), different learning rates for the update terms\([Hayou et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib43)\), and compressing the adapters into a single\-matrix update\([Bensaïd et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib74)\)\.

#### Prefix and Prompt Tuning\.

Prefix\-tuning strategies prepend learnable vector sequences before transformer layer inputs, creating controllable input modifications while keeping the pretrained model fixed\([Wang et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib66);[Li and Liang, 2021](https://arxiv.org/html/2608.26430#bib.bib17);[Qin and Eisner, 2021](https://arxiv.org/html/2608.26430#bib.bib20);[Lester et al\., 2021b](https://arxiv.org/html/2608.26430#bib.bib47);[Liu et al\., 2022c](https://arxiv.org/html/2608.26430#bib.bib48)\)\. Closely related, Prompt\-tuning approaches learn different prompt representations to guide model behaviors without architectural modification\([Xiao et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib67);[Lester et al\., 2021a](https://arxiv.org/html/2608.26430#bib.bib21);[Liu et al\., 2022b](https://arxiv.org/html/2608.26430#bib.bib22);[Ge et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib23)\)\. Unlike prefix tuning, prompt tuning operates exclusively at the input embedding level, making it particularly efficient and easy to integrate\([Lester et al\., 2021a](https://arxiv.org/html/2608.26430#bib.bib21)\)\. This approach has been widely extended to domain adaptation\([Ge et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib23)\), vision\-language models\([Zhou et al\., 2022](https://arxiv.org/html/2608.26430#bib.bib49)\), and diffusion models\([Dong et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib50);[Chung et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib51)\)\.

#### Sparse Fine\-Tuning:

Sparse fine\-tuning methodologies exploit natural parameter redundancy by targeting only critical components while freezing the rest\([Iurada et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib76);[Khaki et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib68);[He et al\., 2025](https://arxiv.org/html/2608.26430#bib.bib72);[Lu et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib54);[Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29);[Guo et al\., 2021](https://arxiv.org/html/2608.26430#bib.bib30)\)\. These approaches, whether through low\-rank operations\([Lu et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib54);[Dragomir et al\., 2026](https://arxiv.org/html/2608.26430#bib.bib73)\)or spectral compression via Fourier transformations\([Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29);[Zhang et al\., 2026](https://arxiv.org/html/2608.26430#bib.bib75)\)– achieve competitive/superior performance compared to full fine\-tuning but reducing trainable parameter count\.

## 6Conclusions

We describe FrameFT, a parameter\-efficient fine\-tuning framework that leverages structured subspace decompositions based on Fusion Frames to fine\-tune transformer models for vision and language tasks\. Our extensive empirical validation across both vision transformers and state\-of\-the\-art language models \(including the Llama and Gemma families\), shows that substantial compute and parameter efficiency gains are achievable without sacrificing performance across many evaluation benchmarks\. We also provide analysis showing that FrameFT preserves the Lipschitz smoothness of the loss landscape, and so achieves desirable convergence properties\. We believe that one key advantage of parameter efficiency of FrameFT will be in situations where the use case requires asetof fine\-tuned models, each fine\-tuned on a specific task and invoked on a case\-by\-case basis\. We note that support for structured sparsity \(beyond2:42:4sparsity\) remains limited but this provides a concrete opportunity for higher efficiency gains if specialized kernels are implemented\. The code is available at[https://github\.com/vsingh\-group/FrameFT](https://github.com/vsingh-group/FrameFT)\.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning by introducing more efficient methods for adapting foundation models\. FrameFT aims to lower the computational cost and storage requirements for fine\-tuning, which can facilitate broader accessibility and reduced energy usage in AI applications\. We are not aware of any specific negative societal consequences or ethical issues unique to this work that require further highlighting\.

## References

- Adepuet al\.\(2024\)H\. Adepu, Z\. Zeng, L\. Zhang, and V\. SinghFrameQuant: flexible low\-bit quantization for transformers\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px2.p1.1)\.
- Ansellet al\.\(2022\)A\. Ansell, E\. Ponti, A\. Korhonen, and I\. VulićComposable sparse fine\-tuning for cross\-lingual transfer\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p2.1),[§3\.3](https://arxiv.org/html/2608.26430#S3.SS3.p1.1)\.
- Bałazyet al\.\(2025\)K\. Bałazy, M\. Banaei, K\. Aberer, and J\. TaborLoRA\-xs: low\-rank adaptation with extremely small number of parameters\.External Links:2405\.17604,[Link](https://arxiv.org/abs/2405.17604)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2608.26430#S3.SS4.SSS0.Px2.p1.1)\.
- Bensaïdet al\.\(2025\)D\. Bensaïd, N\. Rotstein, R\. Velich, D\. Bensaïd, and R\. KimmelSingLoRA: low rank adaptation using a single matrix\.External Links:2507\.05566,[Link](https://arxiv.org/abs/2507.05566)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px2.p1.1)\.
- Boufounoset al\.\(2009\)P\. Boufounos, G\. Kutyniok, and H\. RauhutCompressed sensing for fusion frames\.Proceedings of SPIE \- The International Society for Optical Engineering\.External Links:[Document](https://dx.doi.org/10.1117/12.826327)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px2.p1.1)\.
- Bubeck \(2015\)S\. BubeckConvex optimization: algorithms and complexity\.External Links:1405\.4980Cited by:[§3\.5](https://arxiv.org/html/2608.26430#S3.SS5.p3.1)\.
- Caiet al\.\(2014\)J\. Cai, H\. Ji, Z\. Shen, and G\. YeData\-driven tight frame construction and image denoising\.Applied and Computational Harmonic Analysis\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px2.p1.1)\.
- Casazzaet al\.\(2008\)P\. G\. Casazza, G\. Kutyniok, and S\. LiFusion frames and distributed processing\.Applied and computational harmonic analysis25\(1\)\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px3.p1.1)\.
- Casazzaet al\.\(2011\)P\. G\. Casazza, M\. Fickus, D\. G\. Mixon, Y\. Wang, and Z\. ZhouConstructing tight fusion frames\.Applied and Computational Harmonic Analysis30\(2\)\.External Links:ISSN 1063\-5203,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.acha.2010.05.002),[Link](https://www.sciencedirect.com/science/article/pii/S1063520310000850)Cited by:[§H\.2](https://arxiv.org/html/2608.26430#A8.SS2.p1.2),[Appendix H](https://arxiv.org/html/2608.26430#A8.p1.1),[§3\.2](https://arxiv.org/html/2608.26430#S3.SS2.p1.1)\.
- Casazza and Kutyniok \(2012\)P\. G\. Casazza and G\. KutyniokFinite frames, theory and applications\.External Links:[Link](https://link.springer.com/book/10.1007/978-0-8176-8373-3)Cited by:[§2](https://arxiv.org/html/2608.26430#S2.SS0.SSS0.Px1.p2.2)\.
- Chenget al\.\(2017\)G\. Cheng, J\. Han, and X\. LuRemote sensing image scene classification: benchmark and state of the art\.Proceedings of the IEEE105\(10\),pp\. 1865–1883\.Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p1.1)\.
- Christensen \(2018\)O\. ChristensenAn introduction to frames and riesz bases\.External Links:[Link](https://link.springer.com/book/10.1007/978-3-319-25613-9)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.26430#S2.p1.1.1)\.
- Chunget al\.\(2023\)H\. Chung, J\. C\. Ye, P\. Milanfar, and M\. DelbracioPrompt\-tuning latent diffusion models for inverse problems\.External Links:2310\.01110Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Cimpoiet al\.\(2014\)M\. Cimpoi, S\. Maji, I\. Kokkinos, S\. Mohamed, and A\. VedaldiDescribing textures in the wild\.InProceedings of the IEEE Conf\. on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p1.1)\.
- Czaja and Na \(2024\)W\. Czaja and S\. NaFrame quantization of neural networks\.arXiv preprint arXiv:2404\.08131\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px2.p1.1)\.
- Dehghaniet al\.\(2023\)M\. Dehghani, J\. Djolonga, B\. Mustafa,et al\.Scaling vision transformers to 22 billion parameters\.External Links:2302\.05442Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: efficient finetuning of quantized llms\.arXiv preprint arXiv:2305\.14314\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.26430#S1.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px2.p1.1)\.
- Donget al\.\(2023\)W\. Dong, S\. Xue, X\. Duan, and S\. HanPrompt tuning inversion for text\-driven image editing using diffusion models\.External Links:2305\.04441Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Dosovitskiyet al\.\(2021\)A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. HoulsbyAn image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p1.1)\.
- Dragomiret al\.\(2026\)A\. Dragomir, I\. Pintilie, A\. Barbalau, M\. Dragoi, F\. Brad, C\. D\. Paduraru, A\. Tifrea, E\. Burceanu, and R\. T\. IonescuJumpLoRA: sparse adapters for continual learning in large language models\.External Links:2604\.16171,[Link](https://arxiv.org/abs/2604.16171)Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Fanget al\.\(2025\)W\. Fang, J\. Wang, Y\. Lu, S\. Liu, Y\. Wu, Y\. Ma, and Z\. XieA survey of circuit foundation model: foundation ai models for vlsi circuit design and eda\.External Links:2504\.03711,[Link](https://arxiv.org/abs/2504.03711)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1)\.
- Fickuset al\.\(2023a\)M\. Fickus, J\. W\. Iverson, J\. Jasper, and D\. G\. MixonHarmonic grassmannian codes\.Applied and Computational Harmonic Analysis65,pp\. 1–39\.Cited by:[item 3](https://arxiv.org/html/2608.26430#A8.I1.i3.p1.1),[Appendix H](https://arxiv.org/html/2608.26430#A8.p1.1),[§3\.1](https://arxiv.org/html/2608.26430#S3.SS1.p1.4)\.
- Fickuset al\.\(2023b\)M\. Fickus, J\. W\. Iverson, J\. Jasper, and D\. G\. MixonHarmonic grassmannian codes\.Applied and Computational Harmonic Analysis65\.External Links:ISSN 1063\-5203,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.acha.2023.01.009),[Link](https://www.sciencedirect.com/science/article/pii/S1063520323000106)Cited by:[item 3](https://arxiv.org/html/2608.26430#S3.I2.i3.2)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Tow, B\. Abbasi,et al\.A framework for few\-shot language model evaluation\.Zenodo\.External Links:[Link](https://zenodo.org/records/10256836)Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2024\)Z\. Gao, Q\. Wang, A\. Chen, Z\. Liu, B\. Wu, L\. Chen, and J\. LiParameter\-efficient fine\-tuning with discrete fourier transform\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p3.1),[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p2.1),[§3](https://arxiv.org/html/2608.26430#S3.SS0.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Geet al\.\(2022\)C\. Ge, R\. Huang, M\. Xie, Z\. Lai, S\. Song, S\. Li, and G\. HuangDomain adaptation via prompt learning\.External Links:2202\.06687Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.The llama 3 herd of models\.External Links:2407\.21783Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px1.p1.1)\.
- Guoet al\.\(2021\)D\. Guo, A\. M\. Rush, and Y\. KimParameter\-efficient transfer learning with diff pruning\.External Links:2012\.07463Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuLoRA\+: efficient low rank adaptation of large models\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.26430#S3.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px2.p1.1)\.
- Heet al\.\(2025\)H\. He, J\. B\. Li, X\. Jiang, and H\. MillerSMT: fine\-tuning large language models with sparse matrices\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Heet al\.\(2022\)J\. He, C\. Zhou, X\. Ma, T\. Berg\-Kirkpatrick, and G\. NeubigTowards a unified view of parameter\-efficient transfer learning\.External Links:2110\.04366Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px1.p1.1)\.
- Helberet al\.\(2018\)P\. Helber, B\. Bischke, A\. Dengel, and D\. BorthIntroducing eurosat: a novel dataset and deep learning benchmark for land use and land cover classification\.InIGARSS 2018\-2018 IEEE International Geoscience and Remote Sensing Symposium,pp\. 204–207\.Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski,et al\.Parameter\-efficient transfer learning for NLP\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research\.Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.26430#S1.p1.1),[§3](https://arxiv.org/html/2608.26430#S3.p1.1),[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px2.p1.1)\.
- Hwanget al\.\(2019\)W\. Hwang, P\. Huang, B\. Kung, J\. Ho, and T\. JongFrame\-based sparse analysis and synthesis signal representations and parseval k\-svd\.IEEE Transactions on Signal Processing67\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px2.p1.1)\.
- Iuradaet al\.\(2025\)L\. Iurada, M\. Ciccone, and T\. TommasiEfficient model editing with task\-localized sparse fine\-tuning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TDyE2iuvyc)Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Karimi Mahabadiet al\.\(2021\)R\. Karimi Mahabadi, S\. Ruder, M\. Dehghani, and J\. HendersonParameter\-efficient multi\-task fine\-tuning for transformers via shared hypernetworks\.Association for Computational Linguistics\.Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px1.p1.1)\.
- Khakiet al\.\(2025\)S\. Khaki, X\. Li, J\. Guo, L\. Zhu, C\. Xu, K\. N\. Plataniotis, A\. Yazdanbakhsh, K\. Keutzer, S\. Han, and Z\. LiuSparseLoRA: accelerating llm fine\-tuning with contextual sparsity\.External Links:2506\.16500,[Link](https://arxiv.org/abs/2506.16500)Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Lesteret al\.\(2021a\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Lesteret al\.\(2021b\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.External Links:2104\.08691Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Lingamet al\.\(2024\)V\. Lingam A\. Tejaswiet al\.SVFT: parameter\-efficient fine\-tuning with singular vectors\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.26430#S3.SS0.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1),[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px3.p1.1)\.
- Liuet al\.\(2022a\)H\. Liu, D\. Tam, M\. Muqeeth, J\. Mohta, T\. Huang, M\. Bansal, and C\. RaffelFew\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.arXiv preprint arXiv:2205\.05638\.Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px3.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDoRA: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px3.p1.1)\.
- Liuet al\.\(2022b\)X\. Liu, K\. Ji, Y\. Fu,et al\.P\-tuning: prompt tuning can be comparable to fine\-tuning across scales and tasks\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2022c\)X\. Liu, K\. Ji, Y\. Fu, W\. L\. Tam, Z\. Du, Z\. Yang, and J\. TangP\-tuning v2: prompt tuning can be comparable to fine\-tuning universally across scales and tasks\.External Links:2110\.07602Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2019\)Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. StoyanovRoBERTa: a robustly optimized bert pretraining approach\.External Links:1907\.11692,[Link](https://arxiv.org/abs/1907.11692)Cited by:[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px1.p1.1)\.
- Luet al\.\(2024\)X\. Lu, A\. Zhou, Y\. Xu,et al\.SPP: sparsity\-preserved parameter\-efficient fine\-tuning for large language models\.External Links:2405\.16057Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Majiet al\.\(2013\)S\. Maji, J\. Kannala, E\. Rahtu, M\. Blaschko, and A\. VedaldiFine\-grained visual classification of aircraft\.External Links:1306\.5151Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p1.1)\.
- Malinovskyet al\.\(2024\)G\. Malinovsky, U\. Michieli, H\. A\. A\. K\. Hammoud,et al\.Randomized asymmetric chain of lora: the first meaningful theoretical framework for low\-rank adaptation\.External Links:2410\.08305Cited by:[§3](https://arxiv.org/html/2608.26430#S3.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px2.p1.1)\.
- OpenAI \(2025\)OpenAIGPT\-4 technical report\.External Links:2303\.08774Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1)\.
- Parkhiet al\.\(2012\)O\. M\. Parkhi, A\. Vedaldi, A\. Zisserman, and C\. V\. JawaharCats and dogs\.InIEEE Conference on Computer Vision and Pattern Recognition,Cited by:[Appendix C](https://arxiv.org/html/2608.26430#A3.p1.1)\.
- Puet al\.\(2023\)G\. Pu, A\. Jain, J\. Yin, and R\. KaplanEmpirical analysis of the strengths and weaknesses of peft techniques for llms\.External Links:2304\.14999,[Link](https://arxiv.org/abs/2304.14999)Cited by:[item a](https://arxiv.org/html/2608.26430#A6.I1.i1.p1.1)\.
- Qin and Eisner \(2021\)G\. Qin and J\. EisnerLearning how to ask: querying lms with mixtures of soft prompts\.arXiv preprint arXiv:2104\.06599\.Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Sunet al\.\(2024a\)C\. Sun, J\. Wei, Y\. Wu, Y\. Shi, S\. He, Z\. Ma, N\. Xie, and Y\. YangSVFit: parameter\-efficient fine\-tuning of large pre\-trained models using singular values\.External Links:2409\.05926,[Link](https://arxiv.org/abs/2409.05926)Cited by:[§3](https://arxiv.org/html/2608.26430#S3.SS0.SSS0.Px1.p2.1),[§3\.4](https://arxiv.org/html/2608.26430#S3.SS4.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1)\.
- Sunet al\.\(2024b\)Y\. Sun, Z\. Li, Y\. Li, and B\. DingImproving loRA in privacy\-preserving federated learning\.InThe Twelfth International Conference on Learning Representations,Cited by:[§3\.5](https://arxiv.org/html/2608.26430#S3.SS5.p1.1),[§3\.5](https://arxiv.org/html/2608.26430#S3.SS5.p2.1)\.
- Taoriet al\.\(2023\)R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, and P\. LiangAlpaca: a strong, instruction\-following language model\.Note:[https://crfm\.stanford\.edu/2023/03/13/alpaca\.html](https://crfm.stanford.edu/2023/03/13/alpaca.html)Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px1.p1.1)\.
- Teamet al\.\(2024a\)G\. Team, R\. Anil, S\. Borgeaud,et al\.Gemini: a family of highly capable multimodal models\.External Links:2312\.11805Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1)\.
- Teamet al\.\(2024b\)G\. Team, M\. Riviere, S\. Pathak,et al\.Gemma 2: improving open language models at a practical size\.External Links:2408\.00118Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px1.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone,et al\.Llama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1),[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px1.p1.1)\.
- Waldron \(2019\)S\. F\. D\. WaldronAn introduction to finite tight frames\.External Links:[Link](https://link.springer.com/book/10.1007/978-0-8176-4815-2)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.26430#S2.p1.1.1)\.
- Wanget al\.\(2019\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InInternational Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)H\. Wang, B\. Chen, S\. Li, X\. Liang, H\. K\. Lee, K\. Kawaguchi, and T\. HuPrefix\-tuning\+: modernizing prefix\-tuning by decoupling the prefix from attention\.External Links:2506\.13674,[Link](https://arxiv.org/abs/2506.13674)Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2024\)H\. Wang, T\. Liu, R\. Li, M\. Cheng, T\. Zhao, and J\. GaoRoseLoRA: row and column\-wise sparse low\-rank adaptation of pre\-trained language model for knowledge editing and fine\-tuning\.External Links:2406\.10777,[Link](https://arxiv.org/abs/2406.10777)Cited by:[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1)\.
- Wistubaet al\.\(2024\)M\. Wistuba, P\. T\. Sivaprasad, L\. Balles, and G\. ZappellaChoice of peft technique in continual learning: prompt tuning is not all you need\.External Links:2406\.03216,[Link](https://arxiv.org/abs/2406.03216)Cited by:[item a](https://arxiv.org/html/2608.26430#A6.I1.i1.p1.1)\.
- Xiaet al\.\(2024\)W\. Xia, C\. Qin, and E\. HazanChain of loRA: efficient fine\-tuning of language models via residual learning\.InICML 2024 Workshop on LLMs and Cognition,Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1)\.
- Xiaoet al\.\(2025\)Z\. Xiao S\. Yanet al\.DynaPrompt: dynamic test\-time prompt tuning\.External Links:2501\.16404,[Link](https://arxiv.org/abs/2501.16404)Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2024\)X\. Yang, J\. Leng, G\. Guo, J\. Zhao, R\. Nakada, L\. Zhang, H\. Yao, and B\. ChenS2FT: efficient, scalable and generalizable llm fine\-tuning by structured sparsity\.InThe 38th Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§4\.2](https://arxiv.org/html/2608.26430#S4.SS2.SSS0.Px3.p1.1)\.
- Yaraset al\.\(2024\)C\. Yaras, P\. Wang, L\. Balzano, and Q\. QuCompressible dynamics in deep overparameterized low\-rank learning & adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)B\. Zhang, Z\. Yu, L\. Zhang, K\. Lin, T\. Chen, Y\. Sun, Y\. Ye, and Y\. HeS2FT: parameter\-efficient fine\-tuning in sparse spectrum domain\.External Links:2605\.08589,[Link](https://arxiv.org/abs/2605.08589)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1),[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2025\)D\. Zhang, T\. Feng, L\. Xue, Y\. Wang, Y\. Dong, and J\. TangParameter\-efficient fine\-tuning for foundation models\.External Links:2501\.13787,[Link](https://arxiv.org/abs/2501.13787)Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1)\.
- Zhanget al\.\(2023\)Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.External Links:2303\.10512,[Link](https://arxiv.org/abs/2303.10512)Cited by:[§4\.1](https://arxiv.org/html/2608.26430#S4.SS1.SSS0.Px3.p1.1)\.
- Zhou and Cong \(2017\)F\. Zhou and G\. CongOn the convergence properties of aKK\-step averaging stochastic gradient descent algorithm for nonconvex optimization\.arXiv preprint arXiv:1708\.01012\.Cited by:[§3\.5](https://arxiv.org/html/2608.26430#S3.SS5.p3.1)\.
- Zhouet al\.\(2022\)K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. LiuLearning to prompt for vision\-language models\.International Journal of Computer Vision\.Cited by:[§5](https://arxiv.org/html/2608.26430#S5.SS0.SSS0.Px3.p1.1)\.
- Zhuet al\.\(2024\)J\. Zhu K\. Greenewaldet al\.Asymmetry in low\-rank adapters of foundation models\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2608.26430#S1.SS0.SSS0.Px1.p1.1)\.
- Łajszczaket al\.\(2024\)M\. Łajszczak, G\. Cámbara, and Y\. L\. othersBASE tts: lessons from building a billion\-parameter text\-to\-speech model on 100k hours of data\.External Links:2402\.08093Cited by:[§1](https://arxiv.org/html/2608.26430#S1.p1.1)\.

## Appendix

In this appendix, we provide proofs for the theorems and additional details on the experiments presented in the main paper\. In Section[A](https://arxiv.org/html/2608.26430#A1), we provide a detailed proof for the lemma showing that FrameFT preserves the Lipschitz smoothness of the loss function\. Section[B](https://arxiv.org/html/2608.26430#A2)presents the results for the convergence guarantee of Gradient Descent to a stationary point for Lipschitz smooth functions\. Section[C](https://arxiv.org/html/2608.26430#A3)evaluates the performance of FrameFT for Image classification tasks\. In Section[D](https://arxiv.org/html/2608.26430#A4), we measure the performance of FrameFT as we vary the subspace dimension\. In Section[E](https://arxiv.org/html/2608.26430#A5), we show the performance of FrameFT with an increasing number of non\-zero coefficients\. We observe that FrameFT can leverage the additional parameters to improve the performance\. We provide some additional discussions in Section[F](https://arxiv.org/html/2608.26430#A6)on the Frames and different subspace dimensions in Fusion Frames\. Section[G](https://arxiv.org/html/2608.26430#A7)lists the hyperparameters used for our experiments\. In Section[H](https://arxiv.org/html/2608.26430#A8), we describe an algorithm for constructing Tight Fusion Frames along with an example\. Finally, in Section[I](https://arxiv.org/html/2608.26430#A9), we analyze the sensitivity of FrameFT to the scaling factorα\\alphaand observe that FrameFT performs well above the baselines across a wide range ofα\\alphavalues\.

## Appendix AFrameFT preserves Lipschitz smoothness

Here, for working through the proof, assume the following dimensions for the matrices involved:PmP\_\{m\}ism×pm\\times p,PnP\_\{n\}isn×qn\\times q,Cl⋅C\_\{l\}^\{\\cdot\}isp×qp\\times qandW0W\_\{0\}ism×nm\\times n\.

Proof\.Let us consider how the loss function behaves at two different points\. Take any two coefficient matricesCl1C\_\{l\}^\{1\}andCl2C\_\{l\}^\{2\}\. From our composition rule, these map to:W1=W0\+Pm​Cl1​PnTW^\{1\}=W\_\{0\}\+P\_\{m\}C\_\{l\}^\{1\}P\_\{n\}^\{T\}andW2=W0\+Pm​Cl2​PnTW^\{2\}=W\_\{0\}\+P\_\{m\}C\_\{l\}^\{2\}P\_\{n\}^\{T\}\. Scalar is omitted for notation simplicity\.

By theLL\-Lipschitz smoothness assumption onf⁡\(W\)f\(W\):

‖∇f​\(W1\)−∇f​\(W2\)‖F≤L​‖W1−W2‖F\\displaystyle\|\|\\nabla f\(W^\{1\}\)\-\\nabla f\(W^\{2\}\)\|\|\_\{F\}\\leq L\|\|W^\{1\}\-W^\{2\}\|\|\_\{F\}where we use the notation∇f​\(W\)=∂f∂W\\nabla f\(W\)=\\frac\{\\partial f\}\{\\partial W\}, the derivative offfwith respect toWW\. We also haveW1−W2=Pm​\(Cl1−Cl2\)​PnTW^\{1\}\-W^\{2\}=P\_\{m\}\(C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\)P\_\{n\}^\{T\}and we will show shortly that‖W1−W2‖F\|\|W^\{1\}\-W^\{2\}\|\|\_\{F\}is upper\-bounded by terms involving\(Cl1−Cl2\)\(C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\)and other constants\. The chain rule for matrix derivatives means the derivative offfwith respect to the coefficientsClC\_\{l\}is,

∇f​\(Cl\)\\displaystyle\\nabla f\(C\_\{l\}\)=∂f∂W⋅∂W∂Cl=∇f​\(W\)⋅∇W​\(Cl\)\.\\displaystyle=\\frac\{\\partial f\}\{\\partial W\}\\cdot\\frac\{\\partial W\}\{\\partial C\_\{l\}\}=\\nabla f\(W\)\\cdot\\nabla W\(C\_\{l\}\)\.The term∇W​\(Cl\)\\nabla W\(C\_\{l\}\)denotes how the weight matrixWWchanges with respect to a change in the coefficient matrixClC\_\{l\}\. We will use directional derivatives to compute this\.

Consider a matrixζΔ∈ℝk​ρm×k​ρn\\zeta\_\{\\Delta\}\\in\\mathbb\{R\}^\{k\\rho\_\{m\}\\times k\\rho\_\{n\}\}which represents a direction\. The directional derivative ofWWwith respect toClC\_\{l\}in the direction ofζΔ\\zeta\_\{\\Delta\}is given by

∇ζΔW​\(Cl\)\\displaystyle\\nabla\_\{\\zeta\_\{\\Delta\}\}W\(C\_\{l\}\)=limt→0W⁡\(Cl\+t​ζΔ\)−W⁡\(Cl\)t=limt→0Pm​ζΔ​PnT​tt=Pm​ζΔ​PnT\\displaystyle=\\lim\_\{t\\rightarrow 0\}\\frac\{W\(C\_\{l\}\+t\\zeta\_\{\\Delta\}\)\-W\(C\_\{l\}\)\}\{t\}=\\lim\_\{t\\rightarrow 0\}\\frac\{P\_\{m\}\\zeta\_\{\\Delta\}P\_\{n\}^\{T\}t\}\{t\}=P\_\{m\}\\zeta\_\{\\Delta\}P\_\{n\}^\{T\}Hence, the derivative offfwith respect toClC\_\{l\}in the direction ofζΔ\\zeta\_\{\\Delta\}is given by

∇ζΔf​\(Cl\)\\displaystyle\\nabla\_\{\\zeta\_\{\\Delta\}\}f\(C\_\{l\}\)=∇f​\(W\)⋅∇ζΔW​\(Cl\)\\displaystyle=\\nabla f\(W\)\\cdot\\nabla\_\{\\zeta\_\{\\Delta\}\}W\(C\_\{l\}\)=Tr​\(\(∇f​\(W\)\)T​\(Pm​ζΔ​PnT\)\)\\displaystyle=\\text\{Tr\}\(\(\\nabla f\(W\)\)^\{T\}\(P\_\{m\}\\zeta\_\{\\Delta\}P\_\{n\}^\{T\}\)\)=\(a\)Tr\(PnT∇f\(W\)TPmζΔ\)\\displaystyle\\overset\{\(a\)\}\{=\}\\text\{Tr\}\(P\_\{n\}^\{T\}\\nabla f\(W\)^\{T\}P\_\{m\}\\zeta\_\{\\Delta\}\)=\(b\)Tr\(\(PmT∇f\(W\)Pn\)TζΔ\)\\displaystyle\\overset\{\(b\)\}\{=\}\\text\{Tr\}\(\(P\_\{m\}^\{T\}\\nabla f\(W\)P\_\{n\}\)^\{T\}\\zeta\_\{\\Delta\}\)=⟨PmT∇f\(W\)Pn,ζΔ⟩\\displaystyle=\\langle P\_\{m\}^\{T\}\\nabla f\(W\)P\_\{n\},\\zeta\_\{\\Delta\}\\rangleThe equality \(a\) follows fromTr​\(A​B\)=Tr​\(B​A\)\\text\{Tr\}\(AB\)=\\text\{Tr\}\(BA\)and the equality \(b\) follows from\(A​B​C\)T=CT​BT​AT\(ABC\)^\{T\}=C^\{T\}B^\{T\}A^\{T\}\.

From the definition of the directional gradient, we have that∇ζΔf​\(Cl\)=⟨∇f​\(Cl\),ζΔ⟩\\nabla\_\{\\zeta\_\{\\Delta\}\}f\(C\_\{l\}\)=\\langle\\nabla f\(C\_\{l\}\),\\zeta\_\{\\Delta\}\\rangle\. But from above, we have∇ζΔf\(Cl\)=Tr\(\(PmT∇f\(W\)Pn\)TζΔ\)\\nabla\_\{\\zeta\_\{\\Delta\}\}f\(C\_\{l\}\)=\\text\{Tr\}\(\(P\_\{m\}^\{T\}\\nabla f\(W\)P\_\{n\}\)^\{T\}\\zeta\_\{\\Delta\}\)\. These two expressions are equal for allζΔ\\zeta\_\{\\Delta\}\. Therefore,

∇f\(Cl\)=PmT∇f\(W\)Pn\\nabla f\(C\_\{l\}\)=P\_\{m\}^\{T\}\\nabla f\(W\)P\_\{n\}
Hence, for two coefficient matricesCl1C\_\{l\}^\{1\},Cl2C\_\{l\}^\{2\}:

‖∇f​\(Cl1\)−∇f​\(Cl2\)‖F\\displaystyle\|\|\\nabla f\(C\_\{l\}^\{1\}\)\-\\nabla f\(C\_\{l\}^\{2\}\)\|\|\_\{F\}=‖PmT​\(∇f​\(W1\)−∇f​\(W2\)\)​Pn‖F\\displaystyle=\|\|P\_\{m\}^\{T\}\(\\nabla f\(W^\{1\}\)\-\\nabla f\(W^\{2\}\)\)P\_\{n\}\|\|\_\{F\}≤‖Pm‖o​p​‖\(∇f​\(W1\)−∇f​\(W2\)\)​Pn‖F\\displaystyle\\leq\|\|P\_\{m\}\|\|\_\{\\text\{o\}p\}\|\|\(\\nabla f\(W^\{1\}\)\-\\nabla f\(W^\{2\}\)\)P\_\{n\}\|\|\_\{F\}≤‖∇f​\(W1\)−∇f​\(W2\)‖F​‖Pm‖o​p\|\|Pn\|\|o​p\\displaystyle\\leq\|\|\\nabla f\(W^\{1\}\)\-\\nabla f\(W^\{2\}\)\|\|\_\{F\}\|\|P\_\{m\}\|\|\_\{\\text\{o\}p\}\|\|P\_\{n\}\|\|\_\{\\text\{o\}p\}≤L​‖W1−W2‖F​Bm​Bn\\displaystyle\\leq L\|\|W^\{1\}\-W^\{2\}\|\|\_\{F\}\\sqrt\{B\_\{m\}\}\\sqrt\{B\_\{n\}\}\(8\)
We can bound the operator norm ofPmP\_\{m\}using the upper frame bounds\.

‖PmT​x‖22≤Bm​‖x‖2⟹\|\|Pm\|\|o​p=‖PmT‖o​p≤Bm\|\|P\_\{m\}^\{T\}x\|\|\_\{2\}^\{2\}\\leq B\_\{m\}\|\|x\|\|^\{2\}\\implies\|\|P\_\{m\}\|\|\_\{op\}=\|\|P\_\{m\}^\{T\}\|\|\_\{op\}\\leq\\sqrt\{B\_\{m\}\}
Now, we need to bound‖W1−W2‖F\|\|W^\{1\}\-W^\{2\}\|\|\_\{F\}\. We know thatW1=W0\+Pm​Cl1​PnTW^\{1\}=W\_\{0\}\+P\_\{m\}C\_\{l\}^\{1\}P\_\{n\}^\{T\}andW2=W0\+Pm​Cl2​PnTW^\{2\}=W\_\{0\}\+P\_\{m\}C\_\{l\}^\{2\}P\_\{n\}^\{T\}\. Therefore:

W1−W2=Pm​\(Cl1−Cl2\)​PnT\\displaystyle W^\{1\}\-W^\{2\}=P\_\{m\}\(C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\)P\_\{n\}^\{T\}\(9\)Now

‖W1−W2‖F\\displaystyle\|\|W^\{1\}\-W^\{2\}\|\|\_\{F\}=‖Pm​\(Cl1−Cl2\)​PnT‖F\\displaystyle=\|\|P\_\{m\}\(C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\)P\_\{n\}^\{T\}\|\|\_\{F\}≤\(a\)​‖Pm‖o​p⋅‖\(Cl1−Cl2\)​PnT‖F\\displaystyle\\overset\{\(a\)\}\{\\leq\}\|\|P\_\{m\}\|\|\_\{op\}\\cdot\|\|\(C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\)P\_\{n\}^\{T\}\|\|\_\{F\}≤\(b\)​‖Pm‖o​p⋅‖Cl1−Cl2‖F⋅‖PnT‖o​p\\displaystyle\\overset\{\(b\)\}\{\\leq\}\|\|P\_\{m\}\|\|\_\{op\}\\cdot\|\|C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\|\|\_\{F\}\\cdot\|\|P\_\{n\}^\{T\}\|\|\_\{op\}=\(Bm​Bn\)⋅‖Cl1−Cl2‖F\\displaystyle=\\sqrt\{\(B\_\{m\}B\_\{n\}\)\}\\cdot\|\|C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\|\|\_\{F\}\(10\)\(a\) and \(b\) follow from‖A​B‖F≤‖A‖o​p\|\|B\|\|F\|\|AB\|\|\_\{F\}\\leq\|\|A\|\|\_\{op\}\|\|B\|\|\_\{F\}and‖A​B‖F≤‖A‖F\|\|B\|\|o​p\|\|AB\|\|\_\{F\}\\leq\|\|A\|\|\_\{F\}\|\|B\|\|\_\{op\}respectively\. Substituting this bound in \([8](https://arxiv.org/html/2608.26430#A1.Ex18)\), we get

‖∇f​\(Cl1\)−∇f​\(Cl2\)‖F\\displaystyle\|\|\\nabla f\(C\_\{l\}^\{1\}\)\-\\nabla f\(C\_\{l\}^\{2\}\)\|\|\_\{F\}≤Bm​Bn​L​\(Bm​Bn​‖Cl1−Cl2‖F\)\\displaystyle\\leq\\sqrt\{B\_\{m\}B\_\{n\}\}L\\left\(\\sqrt\{B\_\{m\}B\_\{n\}\}\|\|C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\|\|\_\{F\}\\right\)=L​Bm​Bn​‖Cl1−Cl2‖F\\displaystyle=LB\_\{m\}B\_\{n\}\|\|C\_\{l\}^\{1\}\-C\_\{l\}^\{2\}\|\|\_\{F\}
Hence,f⁡\(Cl\)f\(C\_\{l\}\)isL~\\tilde\{L\}\-Lipschitz smooth withL~=L​Bm​Bn\\tilde\{L\}=LB\_\{m\}B\_\{n\}\.

## Appendix BConvergence Guarantee based on Lipschitz smoothness

The gradient descent update at iterationttis:Clt\+1=Clt−η∇f\(Clt\)C\_\{l\}^\{t\+1\}=C\_\{l\}^\{t\}\-\\eta\\nabla f\(C\_\{l\}^\{t\}\)\.

LetL~\\tilde\{L\}be the Lipschitz constants derived in Section[A](https://arxiv.org/html/2608.26430#A1)\. ByL~\\tilde\{L\}\-Lipschitz smoothness:

f⁡\(Clt\+1\)≤f⁡\(Clt\)\+⟨∇f​\(Clt\),Clt\+1−Clt⟩\+\(L~/2\)​‖Clt\+1−Clt‖F2f\(C\_\{l\}^\{t\+1\}\)\\leq f\(C\_\{l\}^\{t\}\)\+\\langle\\nabla f\(C\_\{l\}^\{t\}\),C\_\{l\}^\{t\+1\}\-C\_\{l\}^\{t\}\\rangle\+\(\\tilde\{L\}/2\)\|\|C\_\{l\}^\{t\+1\}\-C\_\{l\}^\{t\}\|\|\_\{F\}^\{2\}SubstitutingClt\+1=Clt−η∇f\(Clt\)C\_\{l\}^\{t\+1\}=C\_\{l\}^\{t\}\-\\eta\\nabla f\(C\_\{l\}^\{t\}\)we get:

f⁡\(Clt\+1\)\\displaystyle f\(C\_\{l\}^\{t\+1\}\)≤f\(Clt\)\+⟨∇f\(Clt\),−η∇f\(Clt\)⟩\+\(L~/2\)η2\|\|∇f\(Clt\)\|\|F2\\displaystyle\\leq f\(C\_\{l\}^\{t\}\)\+\\langle\\nabla f\(C\_\{l\}^\{t\}\),\-\\eta\\nabla f\(C\_\{l\}^\{t\}\)\\rangle\+\(\\tilde\{L\}/2\)\\eta^\{2\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\(11\)=f⁡\(Clt\)−η​‖∇f​\(Clt\)‖F2\+\(L~/2\)​η2​‖∇f​\(Clt\)‖F2\\displaystyle=f\(C\_\{l\}^\{t\}\)\-\\eta\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\+\(\\tilde\{L\}/2\)\\eta^\{2\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\(12\)Progress per step is

f⁡\(Clt\)−f⁡\(Clt\+1\)≥η​‖∇f​\(Clt\)‖F2​\(1−\(L~​η\)/2\)f\(C\_\{l\}^\{t\}\)\-f\(C\_\{l\}^\{t\+1\}\)\\geq\\eta\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\(1\-\(\\tilde\{L\}\\eta\)/2\)Summing fromt=0t=0toT−1T\-1:

∑t=0T−1\[f⁡\(Clt\)−f⁡\(Clt\+1\)\]≥η⁡\(1−\(L~​η\)/2\)​∑t=0T−1‖∇f​\(Clt\)‖F2\\sum\_\{t=0\}^\{T\-1\}\[f\(C\_\{l\}^\{t\}\)\-f\(C\_\{l\}^\{t\+1\}\)\]\\geq\\eta\(1\-\(\\tilde\{L\}\\eta\)/2\)\\sum\_\{t=0\}^\{T\-1\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}The left side telescopes:

f⁡\(Cl0\)−f⁡\(ClT\)≥η⁡\(1−\(L~​η\)/2\)​∑t=0T−1‖∇f​\(Clt\)‖F2f\(C\_\{l\}^\{0\}\)\-f\(C\_\{l\}^\{T\}\)\\geq\\eta\(1\-\(\\tilde\{L\}\\eta\)/2\)\\sum\_\{t=0\}^\{T\-1\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}Sincef⁡\(ClT\)≥f∗f\(C\_\{l\}^\{T\}\)\\geq f^\{\*\}, we have:

f⁡\(Cl0\)−f∗≥η⁡\(1−\(L~​η\)/2\)​∑t=0T−1‖∇f​\(Clt\)‖F2f\(C\_\{l\}^\{0\}\)\-f^\{\*\}\\geq\\eta\(1\-\(\\tilde\{L\}\\eta\)/2\)\\sum\_\{t=0\}^\{T\-1\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}Dividing both sides byTTand rearranging:

\(1/T\)​∑t=0T−1‖∇f​\(Clt\)‖F2≤\(f⁡\(Cl0\)−f∗\)/\(η⁡\(1−\(L~​η\)/2\)​T\)\(1/T\)\\sum\_\{t=0\}^\{T\-1\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\\leq\(f\(C\_\{l\}^\{0\}\)\-f^\{\*\}\)/\(\\eta\(1\-\(\\tilde\{L\}\\eta\)/2\)T\)Forη≤1/L~\\eta\\leq 1/\\tilde\{L\}, we have1−\(L~​η\)/2≥1/21\-\(\\tilde\{L\}\\eta\)/2\\geq 1/2, and so

\(1/T\)​∑t=0T−1‖∇f​\(Clt\)‖F2≤2​\(f⁡\(Cl0\)−f∗\)/\(η​T\)\.\(1/T\)\\sum\_\{t=0\}^\{T\-1\}\|\|\\nabla f\(C\_\{l\}^\{t\}\)\|\|\_\{F\}^\{2\}\\leq 2\(f\(C\_\{l\}^\{0\}\)\-f^\{\*\}\)/\(\\eta T\)\.So, this descent reaches anϵ\\epsilon\-first order stationary point\.

## Appendix CImage Classification with Vision Transformers

Evaluation Framework:We investigate the effectiveness of FrameFT for fine\-tuning Vision Transformers\. To this end, we fine\-tune the base and large variants of the Vision Transformer \(ViT\) architecture\([Dosovitskiy et al\., 2021](https://arxiv.org/html/2608.26430#bib.bib9)\)\. We chose the ImageNet\-21K pre\-trained ViT models available on the Hugging Face Hub for our base model and evaluated across a diverse set of image classification challenges\. The test suite includes fine\-grained classification tasks – Oxford Pets\([Parkhi et al\., 2012](https://arxiv.org/html/2608.26430#bib.bib10)\), Stanford Cars, FGVC Aircraft\([Maji et al\., 2013](https://arxiv.org/html/2608.26430#bib.bib11)\), CIFAR10, CIFAR100, texture recognition – DTD\([Cimpoi et al\., 2014](https://arxiv.org/html/2608.26430#bib.bib12)\), and remote sensing applications – EuroSAT\([Helber et al\., 2018](https://arxiv.org/html/2608.26430#bib.bib13)\), RESISC45\([Cheng et al\., 2017](https://arxiv.org/html/2608.26430#bib.bib14)\)\.

![Refer to caption](https://arxiv.org/html/2608.26430v1/fig/cola_mrpc_vs_l.png)Figure 5:Performance of FrameFT on CoLA and MRPC for different values of subspace dimensionρ\\rhoFrameFT configuration:For this experiment, we utilized sparse block diagonal coefficient matrices withn=1000n=1000andn=5000n=5000non\-zero elements, and a scaling factorα=250\\alpha=250across all the tasks\. We share the positions of non\-zero positions across all the layers\. Similar to the natural language understanding experiment, we setρ=2\\rho=2while adjusting the number of subspaceskkto satisfyk​ρ=nk\\rho=nthroughout the network\. Again, following LoRA, we adapt only the Query and Value matrices in the self\-attention blocks in all layers of the model\.

Performance analysis:Table[6](https://arxiv.org/html/2608.26430#A3.T6)shows the performance of FrameFT for fine\-tuning Vision Transformers on diverse datasets\. On average, with50005000non\-zero coefficients per layer, FrameFT performs better than all the baseline methods except full fine\-tuning\. FrameFT is able to achieve this with the lowest number of parameters compared to all the baseline methods, which is5−20×5\-20\\timeslower than LoRA\. We also see an improvement in performance as we increase the number of non\-zero coefficients from10001000to50005000\. As an additional note, the hyperparameters for FrameFT are held fixed for all the tasks in this benchmark\. The hyperparameters for FourierFT were adjusted on a task\-by\-task basis, as noted by\([Gao et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib29)\)\.

Table 6:Performance of different methods for Fine\-Tuning ViT Base and Large models on different datasets\. \* indicates the results reported in previous work\. We highlight non PEFT methods ingray\. FrameFT performs better than all the baseline methods on average, using a 30x lower number of parameters than LoRA\.
## Appendix DPerformance versus subspace dimension

In this experiment, we measure the performance of FrameFT for different values of subspace dimension,ρ\\rho\. For this purpose, we fine\-tune the RoBERTa\-base model on two of the tasks in the GLUE benchmark \- CoLA and MRPC, for different values ofρ\\rho\. We keep the hyperparameters that we obtain forρ=2\\rho=2as shown in Table[7](https://arxiv.org/html/2608.26430#A7.T7)\. Figure[5](https://arxiv.org/html/2608.26430#A3.F5)shows the performance of FrameFT as the subspace dimension is varied from22to256256\. We observe that FrameFT performs well even with a subspace dimension of22\. We do not see a strong relationship between the performance and the subspace dimension\. Given that the two key choices we have for adjustments – the number of coefficients and the subspace dimension – to fill the space, the number of coefficients is a more consistent way of improving performance as indicated in Figure[6](https://arxiv.org/html/2608.26430#A5.F6)\. An alternative and promising direction for future work is to make the subspace dimension a trainable parameter, potentially enabling more adaptive and efficient optimization\.

## Appendix EPerformance as a function of number of non\-zero coefficients

Figure[6](https://arxiv.org/html/2608.26430#A5.F6)shows the performance of RoBERTa base fine\-tuned with FrameFT on two of the tasks in the GLUE benchmark, in addition to the tasks shown in the main text\. As noted earlier, we observe that FrameFT improves as the number of non\-zero coefficients are increased, performing on par or better than full\-finetuning\.

![Refer to caption](https://arxiv.org/html/2608.26430v1/fig/frameFT_vs_n_ff_coeffs_2.png)Figure 6:Performance of RoBERTa base model fine\-tuned with FrameFT versus the number of non\-zero coefficients on the GLUE benchmark\. The red line indicates the performance of full fine\-tuning\.
## Appendix FAdditional Discussions

We cover a few relevant points not discussed in detail so far\.

1. \(a\)*Comparison with Prompt Tuning\.*Prior literature has extensively analyzed the trade\-offs between prompt tuning and Parameter\-Efficient Fine\-Tuning \(PEFT\) methods\([Wistuba et al\., 2024](https://arxiv.org/html/2608.26430#bib.bib59);[Pu et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib60)\)\. Notably,\([Pu et al\., 2023](https://arxiv.org/html/2608.26430#bib.bib60)\)provides a quantitative comparison of prompt tuning against methods like LoRA and\(I​A\)3\(IA\)^\{3\}, reporting that PEFT strategies generally yield superior performance\. Since FrameFT demonstrates performance comparable to or exceeding that of LoRA, we infer that it maintains this advantage over prompt tuning\. Hence, FrameFT serves as a more robust alternative to prompt tuning, delivering the reliability of PEFT methods while requiring a very low number of parameters\.
2. \(b\)*Standard Frames versus Fusion Frames\.*A natural question is whether standard Frames \(characterized by a subspace dimension of11\) could be employed instead of Fusion Frames\. While feasible in principle, the specific construction algorithm we outline in §[3\.2](https://arxiv.org/html/2608.26430#S3.SS2)enforces a minimum subspace dimension of22\. This constraint arises because we first generate the frame in the Complex domain; when transforming these components to the Real domain, the dimension inherently doubles\. Consequently, if an application strictly requires a subspace dimension of11\(while maintainingk​ρ=nk\\rho=n\), the complexity of fusion frames is unnecessary\. In such scalar cases, one can simply rely on classical orthonormal bases, such as Fourier bases or Wavelets\.
3. \(c\)*How does performance vary as a function of subspace dimension?*We performed experiments changing the subspace dimension and report the results in §[D](https://arxiv.org/html/2608.26430#A4)\. We observe that as the subspace dimension is increased \(withk​ρ=nk\\rho=n\), the sparsity in Fusion Frames increases, thereby reducing the degrees of freedom for the parameter update\. This leads to a small drop in performance\.

## Appendix GHyperparameters used for the experiments

We present the hyperparameters used for different experiments in Tables[7](https://arxiv.org/html/2608.26430#A7.T7),[8](https://arxiv.org/html/2608.26430#A7.T8),[9](https://arxiv.org/html/2608.26430#A7.T9)\. As for the baselines, we use the hyperparameters suggested by the respective authors\.

Table 7:Hyperparameters for the GLUE benchmarkTable 8:Hyperparameters for the Image classification taskTable 9:Hyperparameters for the Instruction tuning task
## Appendix HFusion Frames Construction: an example

In this section, we briefly outline an algorithm for generating Tight Fusion Frames with the help of an example\.\([Casazza et al\., 2011](https://arxiv.org/html/2608.26430#bib.bib6)\)formalized a systematic framework for identifying the\(k,ρ,d\)\(k,\\rho,d\)values for which a Tight Fusion Frame exists and generating the TFF whenever it exists\. Their algorithm generates a TFF inℂd\\mathbb\{C\}^\{d\}\. Since we are mainly interested in real vector spaces, we use the simple extension in\([Fickus et al\., 2023a](https://arxiv.org/html/2608.26430#bib.bib1)\)to adapt the TFF to the real domain\. The overall algorithm can be divided into three parts\.

1. 1\.Construct a UNTF forℂρ\\mathbb\{C\}^\{\\rho\}withddelements by playing Spectral Tetris\.
2. 2\.Modulate these vectors withkthk^\{\\text\{th\}\}roots of unity to formkksubspaces of dimensionρ\\rhoinℂd\\mathbb\{C\}^\{d\}\.
3. 3\.Use the method in\([Fickus et al\., 2023a](https://arxiv.org/html/2608.26430#bib.bib1)\)to extend the result to real\-valued spaces\.

We describe these steps in detail by walking through the construction of a \(6,3,11\) TFF, a Tight Fusion Frame spanningℂ11\\mathbb\{C\}^\{11\}withk=6k=6subspaces where each subspace has a dimension ofρ=3\\rho=3\.

### H\.1Spectral Tetris

The first step is to generate a “smaller” frame and in the next step, we modulate the smaller frame to generate a “larger” Tight Fusion Frame\. After generating a TFF forℂd\\mathbb\{C\}^\{d\}we can easily extend it to the Real Field by applying the entrywise mapx\+i​y↦\[x−yyx\]x\+iy\\mapsto\\begin\{bmatrix\}x&\-y\\\\ y&x\\end\{bmatrix\}\. So,k=6,ρ=3,d=11k=6,\\rho=3,d=11\. As the name suggests UNTFs are Tight frames where each frame vector has a unit norm\. We construct a4×114\\times 11matrixFFwhose columns are the frame vectors forℂ4\\mathbb\{C\}^\{4\}which satisfies

In this first step, we generate a “smaller” frame \- a Unit Norm Tight Frame \(UNTF\) forℂ3\\mathbb\{C\}^\{3\}with1111vectors\. We arrange these vectors in the columns of a matrixFF\. This UNTF is characterized by

- •Columns with unit norm
- •Rows are Orthogonal and have a constant norm, that isF​F∗FF^\{\*\}is a constant multiple of the Identity matrix \( here the constant being113\\frac\{11\}\{3\}\)

We start by filling the first two entries inFFwith 1

F=\[11???????????????????????????????\]F=\\begin\{bmatrix\}1&1&?&?&?&?&?&?&?&?&?\\\\ ?&?&?&?&?&?&?&?&?&?&?\\\\ ?&?&?&?&?&?&?&?&?&?&?\\\\ \\end\{bmatrix\}The remaining norm left to be filled is113−2=53\\frac\{11\}\{3\}\-2=\\frac\{5\}\{3\}\. We continue to fill in 1s until the required norm is less than 1\. Here, we can do this only once, yielding

F=\[111??????????????????????????????\]F=\\begin\{bmatrix\}1&1&1&?&?&?&?&?&?&?&?\\\\ ?&?&?&?&?&?&?&?&?&?&?\\\\ ?&?&?&?&?&?&?&?&?&?&?\\\\ \\end\{bmatrix\}This leaves a norm of23\\frac\{2\}\{3\}to be filled\. This can be added with a2×22\\times 2matrixT⁡\(x\)T\(x\)\.T⁡\(x\)T\(x\)here is defined as follows:

T⁡\(x\):=12​\[xx2−x−2−x\],T⁡\(x\)​T∗​\(x\)=\[x002−x\]T\(x\)\\vcentcolon=\\frac\{1\}\{\\sqrt\{2\}\}\\begin\{bmatrix\}\\sqrt\{x\}&\\sqrt\{x\}\\\\ \\sqrt\{2\-x\}&\-\\sqrt\{2\-x\}\\end\{bmatrix\},\\quad\\quad T\(x\)T^\{\*\}\(x\)=\\begin\{bmatrix\}x&0\\\\ 0&2\-x\\end\{bmatrix\}After substitutingT⁡\(x\)T\(x\)withx=23x=\\frac\{2\}\{3\},FFis now

F=\[111131300000000023−23??????0000???????\]F=\\begin\{bmatrix\}1&1&1&\\frac\{1\}\{\\sqrt\{3\}\}&\\frac\{1\}\{\\sqrt\{3\}\}&0&0&0&0&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&?&?&?&?&?&?\\\\ 0&0&0&0&?&?&?&?&?&?&?\\\\ \\end\{bmatrix\}Now, we continue adding ones in row two until the norm becomes less than 1 again\.

F=\[111131300000000023−2311????0000???????\]F=\\begin\{bmatrix\}1&1&1&\\frac\{1\}\{\\sqrt\{3\}\}&\\frac\{1\}\{\\sqrt\{3\}\}&0&0&0&0&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&1&1&?&?&?&?\\\\ 0&0&0&0&?&?&?&?&?&?&?\\\\ \\end\{bmatrix\}Now we insertT⁡\(x\)T\(x\)with the remaining norm\. We repeat this process until all the rows are filled\. The FinalFFis given by

F=\[111131300000000023−2311161600000000056−5611\]F=\\begin\{bmatrix\}1&1&1&\\frac\{1\}\{\\sqrt\{3\}\}&\\frac\{1\}\{\\sqrt\{3\}\}&0&0&0&0&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&1&1&\\frac\{1\}\{\\sqrt\{6\}\}&\\frac\{1\}\{\\sqrt\{6\}\}&0&0\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}&\-\\frac\{5\}\{\\sqrt\{6\}\}&1&1\\\\ \\end\{bmatrix\}

### H\.2Modulation

In the second step of TFF construction, theFFmatrix is modulated with complex roots of unity, one subspace at a time\. For eachki=0,1,2,…​k−1k\_\{i\}=0,1,2,\\dots k\-1, we construct a row vector

wki=\[\(ei​2​π​kik\)0​\(ei​2​π​kik\)1​\(ei​2​π​kik\)2​…​\(ei​2​π​kik\)d−1\]w\_\{k\_\{i\}\}=\\left\[\\left\(e^\{\\frac\{i2\\pi k\_\{i\}\}\{k\}\}\\right\)^\{0\}\\left\(e^\{\\frac\{i2\\pi k\_\{i\}\}\{k\}\}\\right\)^\{1\}\\left\(e^\{\\frac\{i2\\pi k\_\{i\}\}\{k\}\}\\right\)^\{2\}\\dots\\left\(e^\{\\frac\{i2\\pi k\_\{i\}\}\{k\}\}\\right\)^\{d\-1\}\\right\]Each row ofFFis multiplied bywkiw\_\{k\_\{i\}\}to produce the orthogonal basis for the subspace indexed bykik\_\{i\}\. Theorem1414by[Casazza et al\. \(2011\)](https://arxiv.org/html/2608.26430#bib.bib6)proves that the Fusion Frames generated by this algorithm are Tight\. The Final Fusion Frame generated is shown in Table[10](https://arxiv.org/html/2608.26430#A8.T10)\.

\[11113130000001ωω213​ω313​ω40000001ω2ω41313​ω20000001ω3113​ω3130000001ω4ω21313​ω40000001ω5ω413​ω313​ω200000000023−231116160000023​ω3−23​ω4ω5116​ω16​ω20000023−23​ω2ω4116​ω216​ω40000023​ω3−23ω3116​ω3160000023−23​ω4ω2116​ω416​ω20000023​ω3−23​ω2ω1116​ω516​ω400000000056−5611000000056​ω−56​ω2ω3ω4000000056​ω2−56​ω41ω2000000056​ω3−56ω31000000056​ω4−56​ω21ω4000000056​ω5−56​ω4ω3ω2\]\\begin\{bmatrix\}\[l\]1&1&1&\\frac\{1\}\{\\sqrt\{3\}\}&\\frac\{1\}\{\\sqrt\{3\}\}&0&0&0&0&0&0\\\\ 1&\\omega&\\omega^\{2\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{3\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{4\}&0&0&0&0&0&0\\\\ 1&\\omega^\{2\}&\\omega^\{4\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{2\}&0&0&0&0&0&0\\\\ 1&\\omega^\{3\}&1&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{3\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}&0&0&0&0&0&0\\\\ 1&\\omega^\{4\}&\\omega^\{2\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{4\}&0&0&0&0&0&0\\\\ 1&\\omega^\{5\}&\\omega^\{4\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{3\}&\\frac\{\\sqrt\{1\}\}\{\\sqrt\{3\}\}\\omega^\{2\}&0&0&0&0&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&1&1&\\frac\{1\}\{\\sqrt\{6\}\}&\\frac\{1\}\{\\sqrt\{6\}\}&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{3\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{4\}&\\omega^\{5\}&1&\\frac\{1\}\{\\sqrt\{6\}\}\\omega&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{2\}&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{2\}&\\omega^\{4\}&1&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{2\}&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{4\}&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{3\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\\omega^\{3\}&1&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{3\}&\\frac\{1\}\{\\sqrt\{6\}\}&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{4\}&\\omega^\{2\}&1&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{4\}&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{2\}&0&0\\\\ 0&0&0&\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{3\}&\-\\frac\{\\sqrt\{2\}\}\{\\sqrt\{3\}\}\\omega^\{2\}&\\omega^\{1\}&1&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{5\}&\\frac\{1\}\{\\sqrt\{6\}\}\\omega^\{4\}&0&0\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}&\-\\frac\{5\}\{\\sqrt\{6\}\}&1&1\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}\\omega&\-\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{2\}&\\omega^\{3\}&\\omega^\{4\}\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{2\}&\-\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{4\}&1&\\omega^\{2\}\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{3\}&\-\\frac\{5\}\{\\sqrt\{6\}\}&\\omega^\{3\}&1\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{4\}&\-\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{2\}&1&\\omega^\{4\}\\\\ 0&0&0&0&0&0&0&\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{5\}&\-\\frac\{5\}\{\\sqrt\{6\}\}\\omega^\{4\}&\\omega^\{3\}&\\omega^\{2\}\\\\ \\end\{bmatrix\}Table 10:\(𝟔,𝟑,𝟏𝟏\)\\bm\{\(6,3,11\)\}\-TFFforℂ11\\mathbb\{C\}^\{11\}\. Here,ω=ei​π/3\\omega=e^\{i\\pi/3\}\. A pair of rows belongs to the same subspace if their indices differ by a multiple of 6
### H\.3Fusion Frames construction time

Table[11](https://arxiv.org/html/2608.26430#A8.T11), presents the time taken to construct the Fusion Frames using the Spectral Tetris method\. As a reference, we provide the time taken to construct a random orthonormal basis using two methods, namely the Gram\-Schmidt algorithm and QR decomposition on a random matrix\. These times are measured on an NVIDIA A100 machine\. Since the Gram\-Schmidt process is sequential, it cannot be parallelized on GPUs/TPUs\. Hence, the time taken to perform this process is quite high\. We can see that Spectral Tetris takes a fraction of the time to construct than QR decomposition on modern GPUs\. Though there is some latency in generating Fusion Frames, it is a minor issue, as it is amortized across multiple model calls\.

Table 11:Basis generation time\. Spectral Tetris takes a fraction of time to construct the Basis on an NVDIA A100 gpu

## Appendix IScaling Factor Sensitivity

Table[12](https://arxiv.org/html/2608.26430#A9.T12)reports performance of Llama\-2\-7B instruction\-tuned on the Alpaca dataset with FrameFT as the scaling factorα\\alphais varied from 10 to 600\. FrameFT outperforms LoRA’s average of 63\.18 across the entire sweep\. Forα≥100\\alpha\\geq 100, it consistently matches or exceeds the full fine\-tuning upper bound of 63\.39\. The results demonstrate a major practical advantage: FrameFT is highly insensitive to this factor\. Unlike PEFT methods requiring exhaustive searches to prevent instability, FrameFT operates reliably out\-of\-the\-box\.

Table 12:Scaling factorα\\alphasensitivity on Llama\-2\-7B/Alpaca\. FrameFT outperforms LoRA \(63\.18\) across the full sweep\.