Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs

# Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
Source: [https://arxiv.org/html/2608.04488](https://arxiv.org/html/2608.04488)
\\UseRawInputEncoding

Kuanysh Akhmetzhanov and Jurn\-Gyu ParkCorresponding author: Jurn\-Gyu Park \(jurn\.park@nu\.edu\.kz\)\.K\. Akhmetzhanov and J\-\.G\. Park are with the Department of Computer Science, Nazarbayev University, Astana 010000, Kazakhstan \(e\-mail: kuanysh\.akhmetzhanov@nu\.edu\.kz, jurn\.park@nu\.edu\.kz\)\.The authors gratefully acknowledge Saltanat Ami for her editorial assistance and valuable feedback on this manuscript\.

###### Abstract

Despite rapid advances in large language models \(LLMs\), deploying and personalizing them on resource\-constrained devices remains impractical due to high VRAM, time, and energy costs\. Parameter\-Efficient Fine\-Tuning \(PEFT\) of Small Language Models \(SLMs\) offers a promising alternative, yet few studies compare PEFT methods across architectures using both general and personalization benchmarks while accounting for energy consumption\. We compare five fine\-tuning approaches \(Full Fine\-Tuning, LoRA, LoRA\+, QLoRA, and BitFit\) on four SLMs from two families \(Transformer\-based: TinyLlama\-1\.1B, Qwen3\-1\.7B; SSM\-based: Mamba\-1\.4B, Mamba\-2\-1\.3B\) across three GLUE tasks \(SST\-2, QNLI, STS\-B\) and three LaMP personalization tasks \(LaMP\-1, LaMP\-2, LaMP\-3\)\. Each configuration is evaluated with the energy\-focused NetScore\-E and the memory\-focused NetScore\-M, the two variants that reflect the constraints binding on\-device deployment\. Methods are selected with a strict energy\-first rule \(highest NetScore\-E, ties broken by NetScore\#\)\. LoRA\+ achieves the highest NetScore\-E in 19 of 24 configurations and the highest NetScore\-M in 13 of 24, and is the selected method in 18 of 24\. QLoRA, available only for the Transformer models, cuts peak finetuning VRAM by up to3\.9×3\.9\\timesrelative to LoRA and therefore takes the best NetScore\-M in 5 of the 12 Transformer configurations, although its de\-quantization overhead leaves it selected in only one of them once energy decides\. BitFit and full fine\-tuning are almost never competitive on either variant, and TinyLlama\-1\.1B leads the energy\-focused NetScore\-E on five of the six benchmarks and the memory\-focused NetScore\-M on four\. These results show that compact SLMs paired with PEFT provide a practical, energy\-aware path to personalized on\-device deployment, with the optimal method set by the dominant constraint: LoRA\+ for energy and QLoRA for memory\.

###### Index Terms:

Small Language Models \(SLMs\), Parameter\-Efficient Fine\-Tuning \(PEFT\), LoRA, On\-Device AI, Model Personalization, Resource\-Constrained Deployment\.

## IIntroduction

There have been major developments in the area of Natural Language Processing \(NLP\) during the last ten years due to the increasing scalability of neural language models\. It started with the emergence of the Transformer architecture\[[30](https://arxiv.org/html/2608.04488#bib.bib21)\], and then with the large\-scale pre\-training models of BERT\[[8](https://arxiv.org/html/2608.04488#bib.bib22)\]and GPT\[[39](https://arxiv.org/html/2608.04488#bib.bib23)\]\. With that came the ”larger is better” trend\[[13](https://arxiv.org/html/2608.04488#bib.bib24)\], which led to the exponential increase in model sizes from hundreds of millions of parameters to hundreds of billions\. This resulted in LLMs that could generate code, reason for multiple steps and engage users in open ended dialogues with extremely high levels of accuracy\. While there was an enormous amount of attention given to LLMs that could perform complex reasoning, there also existed a growing demand towards on\-device and personalized AI where language models will operate independently on a consumer single GPU, smart phones, embedded platforms, and various types of edge computing hardware\. They would need to adapt to the unique needs and preferences of individual users without having access to remote cloud resources\.

These two trends represent contrasting ideas and philosophies\. While state of the art LLMs can often require hundreds of GB memory to operate efficiently\[[22](https://arxiv.org/html/2608.04488#bib.bib25)\], they cannot effectively be deployed on consumer grade GPUs, let alone on smaller mobile or embedded systems\. To support such deployments and fine\-tuning, organizations may be required to utilize expensive cloud APIs or proprietary servers to provide the necessary computational capacity to run large language models\. Each option creates additional costs for latency, privacy and operational expenses that create barriers for widespread adoption\[[15](https://arxiv.org/html/2608.04488#bib.bib27)\]\. Additionally, the fine\-tuning process for large language models further complicates many of these issues\. Updating billions of parameters requires increased amounts of VRAM, longer periods of training time and greater amounts of energy usage\[[37](https://arxiv.org/html/2608.04488#bib.bib26)\]\. All of these factors combine to make large language models difficult to deploy and personalize with fine\-tuning in resource\-constrained environments\.

To help bridge this gap between the abilities of current frontier LLMs and those that can currently be realistically used and personalized within resource constraints, researchers have proposed two approaches\. One method involves developing what are referred to as SLMs, compact architectures with model sizes ranging from 1 – 2 Billion parameters, which when trained on very large datasets containing many diverse samples, can produce similar results to larger models on specific tasks\[[2](https://arxiv.org/html/2608.04488#bib.bib14),[21](https://arxiv.org/html/2608.04488#bib.bib15),[26](https://arxiv.org/html/2608.04488#bib.bib16),[19](https://arxiv.org/html/2608.04488#bib.bib28),[27](https://arxiv.org/html/2608.04488#bib.bib29)\]\. Examples of SLMs include: TinyLlama\-1\.1B and Qwen3\-1\.7B, which are members of the transformer family, and Mamba\-1\.4B and Mamba\-2\-1\.3B, which are structured state space models \(SSMs\), which offer linear time sequence processing as an alternative to the quadratic time complexity associated with self\-attention\.

Another strategy involves Parameter Efficient Fine Tuning \(PEFT\)\. This strategy modifies only a fraction of the parameters in a pre\-trained model\. Examples include: LoRA, which adds low rank updates to the weights, LoRA\+, an enhanced version of LoRA, QLoRA, a quantized version of LoRA, and BitFit, which only updates the biases of the model\. Many recent studies have demonstrated how PEFT can achieve competitive results while significantly reducing memory and compute requirements for fine\-tuning large language models\[[23](https://arxiv.org/html/2608.04488#bib.bib18),[5](https://arxiv.org/html/2608.04488#bib.bib19),[1](https://arxiv.org/html/2608.04488#bib.bib30),[41](https://arxiv.org/html/2608.04488#bib.bib31)\]\.

A key limitation however is that most studies that investigate PEFT parameters \(not VRAM sizes\) do so based on one type of architecture family \(e\.g\., transformers\), one or two benchmark sets or do not account for trade offs related to energy efficiency alongside improvements in task accuracy\. Secondly, evaluations typically rely on general language understanding benchmark tasks, often neglecting personalization tasks, despite the fact that adapting a model to individual user preferences is a primary motivation for on\-device fine\-tuning\[[5](https://arxiv.org/html/2608.04488#bib.bib19)\]\. Additionally, most studies report only task accuracy and memory footprint, while failing to report the energy consumed during training, which is one of the most important constraints for battery powered deployments\.

To do this, we develop a comprehensive experimental framework that unifies both strategies \(SLMs \+ PEFTs\) and examines them together across metrics including performance, efficiency, and sustainability\. We employed full fine\-tuning as well as four PEFT methods \(LoRA, LoRA\+, QLoRA and BitFit\) to fine\-tune four SLMs from two architectural families \(Transformer\-based: TinyLlama\-1\.1B and Qwen3\-1\.7B, SSM\-based: Mamba\-1\.4B and Mamba\-2\-1\.3B\)\. Each configuration was evaluated on three General Language Understanding Evaluation \(GLUE\) benchmarks \(SST\-2, QNLI, STS\-B\) for general language understanding, and three LaMP benchmarks \(LaMP\-1, LaMP\-2, LaMP\-3\) for personalization\. This gives 24 model–task configurations and 108 fine\-tuning runs in total, as QLoRA is not supported for the two SSM models\. Alongside traditional evaluation metrics \(i\.e\., test accuracy\), we measured training time, VRAM usage, average power consumption, total energy consumption, and TFLOPs\. Those measurements then were used to calculate a metric called NetScore\[[36](https://arxiv.org/html/2608.04488#bib.bib43),[35](https://arxiv.org/html/2608.04488#bib.bib44)\], alongside its extensions: NetScore\-E\[[29](https://arxiv.org/html/2608.04488#bib.bib45),[33](https://arxiv.org/html/2608.04488#bib.bib46)\], NetScore\-M\[[28](https://arxiv.org/html/2608.04488#bib.bib47)\], NetScore\#\[[28](https://arxiv.org/html/2608.04488#bib.bib47)\]\.

The main contributions of this paper are as follows:

- •Demonstrate various PEFT methods on SLMs to match or surpass full fine\-tuning performance on multiple tasks\.
- •Provide empirical comparisons of Transformer\- and SSM\-based models, analyzing their trade\-offs across accuracy, energy consumption, memory usage, and training time on both general GLUE and personalized LaMP benchmarks\.
- •Utilize multiple quantitative metrics and NetScores, composite score metrics that balance task performance against efficiency\.
- •

## IIMotivation and Related Work

### II\-AMotivation

#### II\-A1Problem Statement

The use of SLMs and PEFT techniques provides theoretically efficient solutions\. However, the challenges associated with implementing and fine\-tuning personalized language models on resource\-constrained edge devices have yet to be resolved\. There are no comprehensive data or standardized metrics available to assist developers in determining the best combination of model architecture \(i\.e\. Transformers vs\. SSM\) and fine\-tuning strategies \(LoRA, LoRA\+, QLoRA, and BitFit\)\.

As long as there are no quantitative assessments of the trade\-offs between model performance \(specifically for user\-personalized tasks\) and operating expenses \(such as VRAM usage, time required to train, etc\.\), it will be hard for companies to effectively expand their on\-device AI deployments\. Therefore, the problem is the absence of an evaluation framework and a unified metric that balances model performance with sustainable energy consumption across modern architectures\.

### II\-BMotivation for Model and Task Selections

We focused on SLMs that range from 1B to 2B parameters based on our used hardware, which has 24 GB of VRAM\. Under this setup, a 2B\-parameter model represents the upper limit for full fine\-tuning\. Additionally, we selected models from two distinct architectural families to examine their trade\-offs in on\-device deployment and fine\-tuning:

- •Transformer\-based \(TinyLlama\-1\.1B, Qwen3\-1\.7B\):TinyLlama serves as a widely adopted, compact, dense baseline for SLMs, while Qwen3 represents the state\-of\-the\-art SLM within this parameter range\.
- •SSM\-based \(Mamba\-1\.4B, Mamba\-2\-1\.3B\):Structured State Space Models are among the most promising alternatives to Transformer based models, since they offer linear\-time sequence processing and a constant memory footprint during inference\. Evaluating such models against traditional Transformers is important to determine if the theoretical advantages of SSM architectures apply in resource\-constrained environments\.

We evaluated every configuration on six tasks drawn from two complementary benchmark suites: three GLUE tasks for general language understanding and three LaMP tasks for user personalization\. This pairing lets us verify that a PEFT method preserves the core NLP skills of a pre\-trained SLM while also adapting it to an individual user’s profile, the two capabilities that on\-device fine\-tuning must deliver at once\. The six tasks further cover binary classification, multi\-class classification, and regression, so the comparison across architectures and PEFT methods is not tied to a single task format\.

- •General Understanding \(GLUE\):We selected SST\-2 \(sentiment analysis\), QNLI \(question\-answering natural language inference\) and STS\-B \(semantic textual similarity\) as a standardized baseline\. This ensures that the applied PEFT methods do not cause the models to suffer catastrophic forgetting or lose core NLP skills\.
- •User Personalization \(LaMP\):Because one of the primary motivations of using on\-device fine\-tuning is to preserve privacy, we used benchmark tasks related to personalization, particularly we selected LaMP\-1 \(personalized citation identification\), LaMP\-2 \(personalized movie tagging\), and LaMP\-3 \(personalized product rating\)\. These tasks simulate real\-world scenarios in which a model must adjust its responses based on a specific user’s historical profile\.

Research Objectives: To address the gap identified in previous studies regarding the evaluation of on\-device AI personalization and deployment efficiencies, this research is driven by the following questions:

1. 1\.Efficiency\-Accuracy Trade\-off:To what degree are PEFT methods able to produce results comparable to those obtained through full fine\-tuning of SLMs on GLUE and LaMP benchmarks?
2. 2\.Architecture Comparison:What differences are there in regards to performance on tasks and computational efficiency between transformer\-based models \(TinyLlama\-1\.1B and Qwen3\-1\.7B\) and state\-space models \(Mamba\-1\.4B and Mamba\-2\-1\.3B\) when fine\-tuned using the same PEFT methods?
3. 3\.Method Selection for Constraints:Using the NetScore metric and its extensions, what specific combinations of PEFT approaches and model architectures provide the optimal balance between task accuracy and low energy consumption for resource\-constrained deployments?

### II\-CRelated Work

Parameter\-Efficient Fine\-Tuning\. The full fine\-tuning performs complete update of all the weights of a pre\-trained model; therefore, a significant amount of memory is needed to store the entire model copy\. This adds a need for more efficient methods of adapting pre\-trained models\. The Parameter Efficient Fine Tuning \(PEFT\) method\[[23](https://arxiv.org/html/2608.04488#bib.bib18)\]is used to address this problem\. In PEFT, the majority of the weights of a pre\-trained model are frozen and only a few of the weights are allowed to be adapted during fine\-tuning\. As a result, a relatively small number of weights needs to be stored and modified\. There are several variants of the PEFT method including LoRA\[[14](https://arxiv.org/html/2608.04488#bib.bib4)\], LoRA\+\[[12](https://arxiv.org/html/2608.04488#bib.bib9)\]and QLoRA\[[7](https://arxiv.org/html/2608.04488#bib.bib11)\]\. The LoRA variant represents each weight updateΔ​W\\Delta Was a product of two low\-rank matricesB​ABA\. These two matrices account for only a small fraction of the total number of parameters\. In addition, the LoRa\+ variant introduces different learning rates for the two matricesAAandBB\. This leads to better convergence speed and stability\. The QLoRA variant uses 4\-bit quantization for the weights to further reduce the memory usage\. Finally, the BitFit\[[3](https://arxiv.org/html/2608.04488#bib.bib10)\]method freezes all the weights except biases, offering the smallest trainable parameter count at the cost of limited performance\. However, none of these methods was evaluated against the others under a common framework based on a single set of small models, which is what we address here\.

Small Language Models \(SLMs\)\. Another approach to efficiency is to start from a small but competent model rather than adapt a large one\. Wang et al\.\[[32](https://arxiv.org/html/2608.04488#bib.bib41)\]define an SLM by two bounds: the smallest size that still shows emergent ability on a specialized task and the largest size that remains sustainable under the available resource constraints\. Their survey covers SLM architectures \(Transformers, Mamba, Hymba, xLSTM\), methods for deriving SLMs from LLMs \(pruning, knowledge distillation, quantization\), and the PEFT methods commonly applied to them\. Within this range, TinyLlama\-1\.1B\[[42](https://arxiv.org/html/2608.04488#bib.bib5)\]shows that 1\.1 billion parameters trained on three trillion tokens can match or surpass considerably larger models trained on less data, challenging earlier scaling assumptions\. Qwen3\-1\.7B\[[38](https://arxiv.org/html/2608.04488#bib.bib1)\]extends this with roughly 36 trillion tokens across 119 languages, GQA to reduce the key\-value cache footprint, and a hybrid thinking mode that trades inference cost for output quality on a per\-query basis, reaching performance comparable to Qwen2\.5\-3B at about half the parameter count\. On the deployment side, MobileLLM\[[17](https://arxiv.org/html/2608.04488#bib.bib6)\]combines hardware\-friendly architecture changes with post\-training quantization to run sub\-billion models on mobile hardware, and Lu et al\.\[[18](https://arxiv.org/html/2608.04488#bib.bib42)\]benchmark over 60 SLMs on edge devices, reporting that they can outperform 7B models on general tasks with limited in\-context learning\.

These works establish SLMs as viable for on\-device deployment, but they stop at inference\. Neither the survey nor the edge benchmark empirically compares PEFT methods across architecturally distinct models under an energy\-aware metric during fine\-tuning, which is the gap this work addresses\.

State Space Models \(SSMs\) based SLMs\. The Mamba\[[10](https://arxiv.org/html/2608.04488#bib.bib7)\]approach substitutes quadratic self\-attention from transformers with selective structured SSMs, whose input\-dependent gating provides content\-aware reasoning and maintains linear time\-complexity\. At 1\.4 B scale, Mamba produces equivalent language modeling quality as transformers of double size at five times faster throughput\. To improve over the original design, Mamba\-2\[[6](https://arxiv.org/html/2608.04488#bib.bib8)\]replaces sequential scanning operation with matrix multiplication, leading to 2\-8x faster computation than Mamba while keeping competitive performance\.

## IIIMethodology

### III\-AMethodology Overview

As illustrated in Figure[1](https://arxiv.org/html/2608.04488#S3.F1), our methodology pipeline consists of five sequential stages \(Benchmarks, Models, PEFT Methods, Performance Metrics, and Comparison\):

![Refer to caption](https://arxiv.org/html/2608.04488v1/x1.png)Figure 1:Methodology Overview\.Benchmarks\.We select six tasks from two different benchmarks\. Three of the six tasks are selected from each category of the popular GLUE \(General Language Understanding Evaluation\)\[[31](https://arxiv.org/html/2608.04488#bib.bib2)\]benchmark: SST\-2 \(sentiment analysis\), QNLI \(natural language inference\), and STS\-B \(Semantic Textual Similarity Benchmark\), which are used to assess general\-purpose language understanding\. The remaining three tasks come from the simple and representative LaMP \(Language Models Personalization\)\[[25](https://arxiv.org/html/2608.04488#bib.bib12)\]benchmark: LaMP\-1 \(Personalized Citation Identification\), LaMP\-2 \(Personalized Movie Tagging\), and LaMP\-3 \(Personalized Product Rating\), to evaluate the model’s capacity for user\-level personalization, which requires fine\-tuning strategies\.

Models\.Four SLMs are evaluated from two architectural families\. First, on the Transformer side, we select TinyLlama\-1\.1B and Qwen3\-1\.7B\. On the state space model \(SSM\) side, we choose Mamba\-1\.4B and Mamba2\-1\.3B\. TinyLlama\-1\.1B was chosen because it represents a highly optimized, compact, dense transformer trained on an extensive dataset \(3 trillion tokens\), while Qwen3\-1\.7B represents the state\-of\-the\-art sub\-2B\-parameter transformer\[[38](https://arxiv.org/html/2608.04488#bib.bib1)\]\. SSM\-based models were chosen to evaluate their theoretical improved computational and memory efficiency over Transformer models\[[10](https://arxiv.org/html/2608.04488#bib.bib7)\]\.

Fine\-Tuning Methods\.Each model is adapted to every benchmark task using five fine\-tuning methods: Full Fine\-Tuning \(Full\-FT\), which updates all parameters, and four PEFT methods, which are LoRA, LoRA\+, QLoRA \(4\-bit quantization\), and BitFit\. This results in 18 configurations per task \(4 models×\\times5 methods, minus the two QLoRA runs that the available libraries do not support for the SSM models\)\. BitFit was chosen as a lower bound of trainable parameters which represents a minimal baseline for PEFT method\. LoRA was chosen as a standard baseline for a PEFT method due to its popularity and widespread use, while LoRA\+ was chosen as its improvement with the same parameter budget but faster theoretical convergence and fine\-tuning stability\. QLoRA was chosen to measure the trade\-off between saved VRAM and the potential energy/time costs associated with de\-quantization overhead during training\.\[[7](https://arxiv.org/html/2608.04488#bib.bib11)\]

Evaluation and Metrics\.Every configuration is evaluated along two axes\. Task performance is measured by accuracy for most tasks \(SST\-2, QNLI, LaMP\-1, LaMP\-2\), Pearson\-Spearman correlation for STS\-B, and Root Mean Squared Error \(RMSE\) for LaMP\-3\. Note that, since some tasks have performance metrics where higher scores indicates poorer performance \(i\.e\., errors such as RMSE and MAE\), we transform their respective error values prior to generating the NetScore\. Specifically,perf=1/\(1\+error\)\\text\{perf\}=\{1\}/\{\(1\+\\text\{error\}\)\}\[[20](https://arxiv.org/html/2608.04488#bib.bib35)\], which is expressed as a percentage \(i\.e\.,a=100/\(1\+error\)a=100/\(1\+\\text\{error\}\)\) in the NetScore computation, consistent with the percentage accuracies used for the other tasks\. Resource efficiency is captured through training time, peak VRAM usage, and total energy consumption \(Wh\)\. The total energy consumption is obtained through multiplying total training time in hours and average power draw in Watts during training\.

In order to provide a single interpretable value that jointly represents performance and efficiency, we adopt the NetScore metric\[[36](https://arxiv.org/html/2608.04488#bib.bib43),[35](https://arxiv.org/html/2608.04488#bib.bib44)\], originally proposed to summarize the trade\-off between accuracy and model size, and extend it to additionally penalize variables related to efficiency\. The metric is defined as

NetScore=S​log10⁡\(aα\(p​m\)β​vγ​tδ​wλ\),\\text\{NetScore\}=S\\,\\log\_\{10\}\\\!\\left\(\\frac\{a^\{\\alpha\}\}\{\(p\\,m\)^\{\\beta\}\\;v^\{\\gamma\}\\;t^\{\\delta\}\\;w^\{\\lambda\}\}\\right\),whereaadenotes task performance,p​mp\\,mthe product of the number of parametersppand the FLOPsmm,vvthe peak VRAM,ttthe fine\-tuning \(or inference\) time, andwwthe average power draw\. The scaleSSand the performance exponentα\\alphacontrol how much emphasis is placed on performance relative to cost\. Following common practice\[[36](https://arxiv.org/html/2608.04488#bib.bib43),[35](https://arxiv.org/html/2608.04488#bib.bib44)\], we setS=20S=20andα=2\\alpha=2\. The efficiency exponentsβ,γ,δ,λ\\beta,\\gamma,\\delta,\\lambdaact as switches that determine which cost variables are included, and we use a1/81/8exponent for the efficiency terms \(rather than the1/41/4used in prior NetScore extensions\) to give higher weight to the task performance\.

Rather than a single value, we report a family of NetScore variants per configuration, each prioritizing a different efficiency variable by toggling the corresponding coefficient \(Table[I](https://arxiv.org/html/2608.04488#S3.T1)\): the baseNetScore\[[36](https://arxiv.org/html/2608.04488#bib.bib43),[35](https://arxiv.org/html/2608.04488#bib.bib44)\]penalizes only model size \(parameters and FLOPs\);NetScore\-E\[[29](https://arxiv.org/html/2608.04488#bib.bib45),[33](https://arxiv.org/html/2608.04488#bib.bib46)\]\(energy\) penalizes only energy and time;NetScore\-M\[[28](https://arxiv.org/html/2608.04488#bib.bib47)\]\(memory\) penalizes only peak memory; andNetScore\#\[[28](https://arxiv.org/html/2608.04488#bib.bib47)\]combines all efficiency terms\. This lets us examine sustainability under several priority levels\.

TABLE I:Coefficient settings \(α=2\\alpha=2,S=20S=20\) defining the NetScore variants\. Non\-zero efficiency exponents use1/8=0\.1251/8=0\.125\.Variantβ\\beta\(P×\\timesFLOPs\)γ\\gamma\(VRAM\)δ\\delta\(Time\)λ\\lambda\(Power\)NS\[[36](https://arxiv.org/html/2608.04488#bib.bib43),[35](https://arxiv.org/html/2608.04488#bib.bib44)\]0\.5000NS\-E\[[29](https://arxiv.org/html/2608.04488#bib.bib45),[33](https://arxiv.org/html/2608.04488#bib.bib46)\]000\.1250\.125NS\-M\[[28](https://arxiv.org/html/2608.04488#bib.bib47)\]00\.12500NS\#\[[28](https://arxiv.org/html/2608.04488#bib.bib47)\]00\.1250\.1250\.125Selection and Comparison\.The final stage uses NetScores to identify the optimal configuration in three steps: \(1\)Best PEFT Method across Models: for each model and task, select the fine\-tuning method with the highest energy\-focused NS\-E, breaking ties on NS\-E with the higher NS\#; \(2\)Best Model across PEFT Methods: compare the best model across all PEFT methods for each benchmark; and \(3\)Best PEFT−\-Model Combination across Benchmarks: determine the most optimal PEFT−\-model combination for each benchmark\.

## IVExperimental Setup

### IV\-AHardware and Software Configuration

As shown in Table[II](https://arxiv.org/html/2608.04488#S4.T2), the hardware and software configuration with specific library versions are detailed\. All of the experiments were conducted using the same single NVIDIA RTX 4090 \(with 24GB of VRAM\) and the same software configuration to ensure that variations in train times, VRAM use, and GPU utilization are solely due to the model architecture and the methods used for fine\-tuning as opposed to other variables such as environmental variation\. Library versions are provided as even small changes to libraries may significantly affect training performance, and therefore impact metrics related to training efficiency\.

TABLE II:Experimental setup\.ComponentSpecificationHardwareOperating SystemLinux \(Ubuntu 22\.04\)GPUNVIDIA RTX 4090 \(24 GB VRAM\)CPU / RAMIntel Core i9\-13900 / 64 GB DDR5Core SoftwarePython3\.11PyTorch2\.9\.1 \(CUDA 12\.8, cuDNN 8\.9\)Hugging Face Transformers4\.36 \(model loading & tokenisation\)Datasets2\.14 \(data pipeline & preprocessing\)PEFT0\.7 \(LoRA / QLoRA adapter training\)Evaluate0\.4 \(metric computation\)QuantizationBitsAndBytes0\.41 \(4\-bit NF4 quantization\)SSM\-specific Librariescausal\-conv1d1\.4\.0 \(depthwise causal convolution kernel\)mamba\-ssm2\.30 \(selective\-scan CUDA kernel\)Monitoringpynvml11\.5 \(GPU utilisation & VRAM sampled at 100 ms\)
### IV\-BModels

We evaluate four SLMs that belong to two different architectural families\. One of them is transformers: TinyLlama\-1\.1B\[[42](https://arxiv.org/html/2608.04488#bib.bib5)\]\(TinyLlama/TinyLlama\-1\.1B\-Chat\-v1\.0\) and Qwen3\-1\.7B\[[38](https://arxiv.org/html/2608.04488#bib.bib1)\]\(Qwen/Qwen3\-1\.7B\)\. The SSM\-family contains the models Mamba\-1\.4B\[[10](https://arxiv.org/html/2608.04488#bib.bib7)\]\(state\-spaces/mamba\-1\.4b\-hf\) and Mamba2\-1\.3B\[[6](https://arxiv.org/html/2608.04488#bib.bib8)\]\(state\-spaces/mamba2\-1\.3b\)\. All models were loaded in bfloat16 precision\.

### IV\-CDatasets

#### IV\-C1GLUE Benchmark

Three tasks have been selected from the General Language Understanding Evaluation \(GLUE\) benchmark\[[31](https://arxiv.org/html/2608.04488#bib.bib2)\]\. These tasks have been directly loaded from Hugging Face Datasets: 1\) SST\-2: Stanford Sentiment Treebank \(binary sentiment classification, evaluation metric: accuracy\)\. 2\) QNLI: Question NLI derived from SQuAD\[[24](https://arxiv.org/html/2608.04488#bib.bib33)\]\(entailment vs\. not entailment, evaluation metric: accuracy\)\. 3\) STS\-B: Semantic Textual Similarity Benchmark \(evaluation of regression on similarity scores 0 – 5, evaluation metric: average of Pearson and Spearman correlation coefficients\)\.

#### IV\-C2LaMP Benchmark

Three tasks have been selected from the Language Models Personalization \(LaMP\) benchmark\[[25](https://arxiv.org/html/2608.04488#bib.bib12)\]to assess user\-level personalization: 1\) LaMP\-1: Personalized Citation Identification \(classification problem, evaluation metric: accuracy\)\. 2\) LaMP\-2: Personalized Movie Tagging \(classification problem; evaluation metric: accuracy\)\. 3\) LaMP\-3: Personalized Product Rating \(regression problem, evaluation metric: Root Mean Squared Error \(RMSE\)\)\. To retrieve user\-profile information for each LaMP task, we use Contriever\[[16](https://arxiv.org/html/2608.04488#bib.bib32)\]\(facebook/contriever\) withk=4k=4retrieved profile entries per query\.

### IV\-DMethods for Fine\-Tuning

Each model are fine\-tuned using five methods: 1\) Full Fine\-Tuning \(Full\-FT\): All model parameters will be updated during training\. 2\) LoRA\[[14](https://arxiv.org/html/2608.04488#bib.bib4)\]: Low\-Rank Adapters will be inserted in the attention projections \(q\_proj,k\_proj,v\_proj,o\_proj\) for transformer\-based models, a configuration that showed better performance than the standard query\-and\-value\-only configuration\[[7](https://arxiv.org/html/2608.04488#bib.bib11)\]\. The low\-rank adapters will be added toin\_projandx\_projfor Mamba\-based models following best practices for SSM fine\-tuning\[[9](https://arxiv.org/html/2608.04488#bib.bib38)\]\. For both architectures, we apply a rankr=16r=16, a scaling factorα=32\\alpha=32, and a dropout of0\.10\.1\. 3\) LoRA\+\[[12](https://arxiv.org/html/2608.04488#bib.bib9)\]: The same adapter configuration as LoRA will be applied\. However, in contrast to LoRA, this method uses a differentiated learning rate for theAAandBBmatrices\. Specifically, a learning rate ratio of16×16\\timeswill be used \(i\.e\., lr=B16⋅\{\}\_\{B\}=16\\cdotlrA\), which results in faster convergence speed and improved stability\[[12](https://arxiv.org/html/2608.04488#bib.bib9)\]\. 4\) QLoRA\[[7](https://arxiv.org/html/2608.04488#bib.bib11)\]: The base model will first be quantized to NF4 precision with double quantization\. Afterward, the same adapter configuration as LoRA and LoRA\+ will be applied on top\. 5\) BitFit\[[3](https://arxiv.org/html/2608.04488#bib.bib10)\]: Only the bias terms, layer normalization weights, and the classification heads of the model will be trainable\. All other model weights remain frozen\.

### IV\-EHyperparameter settings

Table[III](https://arxiv.org/html/2608.04488#S4.T3)shows the hyperparameter settings for the training of both benchmark tasks\. For the GLUE experiments, models are trained as sequence classification models with the Hugging Face Trainer\[[34](https://arxiv.org/html/2608.04488#bib.bib40)\]and a linear learning rate schedule with warm\-up\[[8](https://arxiv.org/html/2608.04488#bib.bib22)\]\. A fixed random seed of 42 has been used for all training runs for reproducibility purposes\. For the GLUE experiments, model selection is done based on the best validation metric over every 200 training steps\.

TABLE III:Training HyperparametersHyperparameterGLUELaMPBatch size \(per device\)324Gradient accumulation steps14Effective batch size3216Learning rate1×10−51\{\\times\}10^\{\-5\}2×10−42\{\\times\}10^\{\-4\}Epochs510Max sequence length128512Warmup ratio0\.10\.05Weight decay0\.010\.01Precisionbf16bf16LoRA rank \(rr\)1616LoRA alpha \(α\\alpha\)3232LoRA dropout0\.10\.1LoRA\+ ratio1616Random seed4242For the LaMP experiments, models are trained as causal language models using generative fine\-tuning with beam search \(num\_beams=4\\text\{num\\\_beams\}=4\) at test time, following the evaluation protocol of\[[25](https://arxiv.org/html/2608.04488#bib.bib12)\]\. The hyperparameter and training settings for the LaMP experiments were taken from the original LaMP paper\[[25](https://arxiv.org/html/2608.04488#bib.bib12)\]\. For the LaMP experiments, up to 128 new tokens can be generated with beam search decoding with 4 beams\.

TABLE IV:Merged results onGLUE \(SST\-2, QNLI, STS\-B\)benchmark \(finetuning \+ inference\)\. “Perf\.” denotes accuracy for SST\-2 and QNLI and the Pearson–Spearman correlation for STS\-B;Commoncolumns are identical for finetuning and inference\. NS = NetScore: NS\-E uses time and power; NS\-M uses VRAM; NS\# uses VRAM, time, and power, all without the parameter/FLOPs term\. All variants use the1/81/8coefficient\. Finetuning Time/Energy are in min/Wh; Inference Time/Energy are in s/mWh\. QLoRA results for Mamba and Mamba2 are omitted because currently available libraries do not support their native quantization\. Bold numeric values mark the best value within each model group, computed separately for finetuning and inference\. For finetuning, red highlights the highest NS\-E and blue highlights the highest NS\-M\. The selected finetuning method \(bold, red\-shaded method name\) follows a strict energy\-first rule: the method with the highest NS\-E is selected, and a tie on NS\-E is broken by the higher NS\#\. This is the same rule used in Table[VI](https://arxiv.org/html/2608.04488#S5.T6)\.TaskModelMethodCommonFinetuningInferenceParams↓\\downarrowPerf\.↑\\uparrowTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrowTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrow\(M\)\(TF\)\(min\)\(MB\)\(W\)\(Wh\)\(TF\)\(s\)\(MB\)\(W\)\(mWh\)SST\-2TinyLlama\-1\.1BFull\-FT1104\.550\.95424\.0934\.0122850\.4314\.50178\.2734\.969\.168\.358\.28\.0315\.003876\.090\.81378\.3839\.771\.370\.262\.4LoRA4\.510\.95916\.1719\.1816479\.8261\.4383\.5760\.670\.068\.759\.58\.0318\.733654\.094\.00489\.0663\.771\.270\.462\.3LoRA\+4\.510\.95716\.1719\.3315957\.0268\.8486\.6160\.669\.968\.759\.48\.0316\.653765\.094\.50437\.0663\.671\.270\.362\.3QLoRA4\.510\.95216\.1731\.054225\.2284\.56147\.2660\.569\.370\.160\.28\.0329\.363243\.0116\.70951\.7563\.670\.370\.461\.5BitFit0\.090\.92016\.0618\.8517950\.0288\.4190\.6177\.069\.267\.958\.68\.0314\.863608\.091\.76378\.7680\.070\.769\.761\.8Qwen3\-1\.7BFull\-FT1720\.570\.95535\.0051\.5622586\.7338\.59290\.9631\.468\.668\.357\.711\.6717\.004812\.098\.47465\.0036\.271\.170\.061\.9LoRA6\.420\.95523\.4921\.4623977\.5360\.31128\.8757\.469\.568\.358\.511\.6723\.004467\.0102\.00651\.6760\.570\.870\.161\.6LoRA\+6\.420\.96223\.4922\.3020047\.6350\.26130\.1857\.569\.668\.658\.811\.6723\.004678\.0101\.00645\.2860\.670\.970\.261\.7QLoRA6\.420\.95823\.4934\.177808\.1376\.07214\.1757\.569\.069\.559\.311\.6734\.003120\.0138\.001303\.3360\.570\.170\.561\.3BitFit0\.120\.90023\.3322\.2718683\.6333\.06123\.6273\.768\.567\.557\.811\.6717\.004884\.099\.00467\.5076\.770\.168\.960\.9Mamba\-1\.4BFull\-FT1383\.310\.95831\.4571\.4124419\.7379\.54451\.7132\.968\.268\.357\.210\.4817\.004151\.099\.00467\.5037\.671\.270\.262\.1LoRA11\.130\.95721\.2448\.7110795\.8387\.37314\.4855\.568\.569\.258\.510\.4823\.504012\.0101\.00659\.3158\.670\.870\.261\.8LoRA\+11\.130\.95821\.2449\.3710779\.6383\.87315\.8655\.568\.669\.258\.510\.4823\.503970\.0105\.00685\.4258\.670\.870\.361\.8BitFit0\.390\.80920\.9644\.9322646\.9394\.41295\.3567\.265\.765\.454\.810\.4817\.003852\.098\.00462\.7870\.268\.367\.459\.3Mamba2\-1\.3BFull\-FT1351\.870\.93633\.2646\.1612990\.9330\.82254\.5132\.368\.468\.658\.111\.0929\.004190\.098\.00789\.4437\.170\.269\.861\.2LoRA8\.110\.95522\.3834\.2412903\.4338\.00192\.8856\.669\.068\.958\.811\.0931\.804225\.0104\.00918\.6759\.770\.470\.161\.3LoRA\+8\.110\.96722\.3831\.7012805\.1357\.50188\.8856\.869\.369\.159\.011\.0931\.704356\.0104\.00915\.7859\.970\.670\.361\.5BitFit0\.210\.50022\.1832\.849029\.5331\.83181\.6261\.357\.958\.148\.011\.0929\.004368\.0101\.00813\.6164\.359\.358\.950\.2QNLITinyLlama\-1\.1BFull\-FT1104\.550\.93424\.09109\.3323200\.1325\.32592\.7934\.667\.467\.956\.58\.03141\.003768\.085\.163335\.4339\.368\.669\.959\.7LoRA4\.510\.92416\.1778\.1316699\.1280\.30364\.9960\.067\.868\.157\.28\.03153\.003168\.087\.273708\.9863\.068\.369\.959\.6LoRA\+4\.510\.93216\.1781\.4616728\.6283\.19384\.4960\.167\.968\.257\.38\.03153\.003158\.086\.533677\.5363\.268\.570\.059\.7QLoRA4\.510\.91716\.17124\.594797\.4273\.32567\.5559\.967\.269\.358\.08\.03184\.002120\.0110\.325638\.5862\.967\.770\.259\.4BitFit0\.090\.83916\.0675\.0719450\.2308\.12385\.5175\.466\.066\.255\.38\.03141\.003132\.084\.823322\.1278\.466\.868\.258\.0Qwen3\-1\.7BFull\-FT1720\.570\.94235\.00170\.3024089\.7390\.011106\.9931\.266\.968\.056\.011\.67153\.005678\.094\.234004\.7835\.968\.669\.659\.2LoRA6\.420\.94323\.49102\.8023674\.0395\.16677\.0457\.267\.568\.056\.511\.67166\.005684\.097\.574499\.0660\.268\.569\.659\.1LoRA\+6\.420\.94623\.49104\.2023605\.8391\.14679\.2857\.367\.568\.156\.611\.67144\.005668\.099\.503980\.0060\.368\.669\.759\.3QLoRA6\.420\.93923\.49159\.4011114\.3387\.681029\.9357\.166\.968\.856\.811\.67213\.004040\.0135\.598022\.4160\.267\.869\.958\.7BitFit0\.120\.85923\.3399\.4023339\.0388\.94644\.3572\.965\.966\.455\.011\.67152\.005592\.094\.794002\.2475\.967\.068\.057\.6Mamba\-1\.4BFull\-FT1383\.310\.90231\.45197\.6624531\.9413\.361361\.7331\.865\.967\.255\.010\.48112\.004120\.094\.432937\.8236\.668\.169\.259\.1LoRA11\.130\.92021\.24147\.8313688\.1422\.051039\.8754\.866\.668\.256\.210\.48148\.003986\.097\.524009\.1657\.968\.269\.659\.2LoRA\+11\.130\.92821\.24147\.9019639\.4421\.401038\.7655\.066\.768\.056\.010\.48148\.003942\.097\.614012\.8658\.068\.369\.759\.3BitFit0\.390\.67520\.96132\.5019153\.5424\.45937\.3264\.061\.362\.550\.610\.48153\.003840\.096\.284091\.9067\.162\.864\.253\.8Mamba2\-1\.3BFull\-FT1351\.870\.82433\.26119\.5621769\.0392\.03781\.1830\.165\.065\.854\.111\.09187\.004567\.092\.774818\.8934\.966\.067\.556\.9LoRA8\.110\.91722\.38100\.2417756\.4413\.83691\.3655\.967\.067\.956\.311\.09199\.004442\.099\.695510\.6459\.067\.869\.458\.6LoRA\+8\.110\.92622\.38101\.3517726\.1406\.87687\.2756\.167\.168\.056\.511\.09197\.004390\.098\.755403\.8259\.167\.969\.658\.8BitFit0\.210\.50022\.1891\.9616521\.3406\.89623\.6362\.457\.658\.547\.111\.09187\.004304\.093\.734868\.7565\.458\.560\.049\.4STS\-BTinyLlama\-1\.1BFull\-FT1104\.550\.90924\.094\.7512444\.4371\.2429\.3934\.170\.268\.160\.08\.0322\.005850\.0105\.59645\.2738\.969\.968\.960\.5LoRA4\.510\.83916\.172\.5024039\.3374\.8815\.6258\.369\.566\.058\.68\.0333\.005574\.0102\.22937\.0261\.468\.167\.658\.8LoRA\+4\.510\.89316\.172\.5619197\.1367\.7315\.6959\.470\.667\.359\.98\.0333\.005676\.0102\.38938\.4862\.469\.268\.659\.8QLoRA4\.510\.82916\.174\.208893\.5373\.2926\.1358\.168\.866\.958\.98\.0347\.003350\.0118\.901552\.3161\.267\.467\.958\.6BitFit0\.090\.31016\.062\.4424209\.6357\.7914\.5558\.152\.348\.741\.38\.0321\.005144\.0100\.28584\.9761\.151\.350\.442\.1Qwen3\-1\.7BFull\-FT1720\.570\.90335\.0012\.6324482\.4228\.5548\.1130\.469\.667\.358\.611\.6730\.0010293\.098\.50820\.8335\.269\.668\.259\.5LoRA6\.420\.79523\.493\.3123809\.7367\.0720\.2554\.268\.365\.157\.411\.6741\.009592\.0117\.811341\.7357\.366\.866\.156\.9LoRA\+6\.420\.89323\.493\.4223622\.6358\.6020\.4456\.270\.367\.159\.411\.6741\.009536\.0118\.151345\.6059\.368\.868\.158\.9QLoRA6\.420\.81823\.495\.417545\.5370\.4333\.4054\.768\.366\.858\.611\.6758\.006689\.0141\.412278\.2757\.866\.766\.957\.2BitFit0\.120\.31623\.333\.3723435\.9345\.4019\.4055\.552\.349\.141\.411\.6736\.007728\.0115\.001150\.0058\.550\.950\.341\.2Mamba\-1\.4BFull\-FT1383\.310\.81431\.457\.5710887\.5408\.5151\.5430\.067\.766\.357\.610\.4832\.004450\.099\.37883\.2934\.867\.767\.358\.5LoRA11\.130\.80721\.245\.1218525\.9422\.4636\.0552\.567\.965\.657\.310\.4843\.004120\.0102\.941229\.5655\.667\.267\.258\.1LoRA\+11\.130\.86921\.245\.1818798\.0418\.6136\.1453\.869\.266\.958\.510\.4843\.004526\.0104\.001242\.2256\.968\.468\.459\.3BitFit0\.390\.14620\.964\.8319158\.6417\.1433\.5837\.438\.335\.927\.610\.4831\.003840\.098\.58848\.8840\.537\.937\.628\.9Mamba2\-1\.3BFull\-FT1351\.870\.63833\.264\.93516998\.34356\.2329\.3025\.764\.161\.653\.511\.0951\.004777\.098\.751398\.9630\.462\.963\.053\.7LoRA8\.110\.70522\.383\.5015044\.3384\.1722\.4151\.366\.163\.555\.711\.0953\.004566\.0100\.891485\.3354\.464\.664\.855\.5LoRA\+8\.110\.84722\.383\.5414986\.6381\.3622\.5054\.569\.366\.758\.911\.0952\.004488\.0102\.001473\.3357\.667\.868\.058\.7BitFit0\.210\.02322\.183\.5312893\.7366\.9721\.597\.86\.74\.2\-3\.611\.0947\.004368\.099\.711301\.7710\.85\.35\.4\-3\.8

TABLE V:Merged results onLaMP \(LaMP\-1/2/3\)benchmark \(finetuning \+ inference\)\. “Perf\.” denotes accuracy for LaMP\-1 and LaMP\-2 \(higher is better\) and RMSE for LaMP\-3 \(lower is better\)\.Commoncolumns are identical for finetuning and inference\. NS = NetScore: NS\-E uses time and power; NS\-M uses VRAM; NS\# uses VRAM, time, and power, all without the parameter/FLOPs term\. All variants use the1/81/8coefficient\. Finetuning Time/Energy are in min/Wh; inference Time/Energy are in min/Wh for LaMP\-1 and LaMP\-3 and in s/mWh for LaMP\-2\. QLoRA results for Mamba and Mamba2 are omitted because currently available libraries do not support their native quantization\. Bold numeric values mark the best value within each model group, computed separately for finetuning and inference\. For finetuning, red highlights the highest NS\-E and blue highlights the highest NS\-M\. The selected finetuning method \(bold, red\-shaded method name\) follows a strict energy\-first rule: the method with the highest NS\-E is selected, and a tie on NS\-E is broken by the higher NS\#\. This is the same rule used in Table[VII](https://arxiv.org/html/2608.04488#S5.T7)\.TaskModelMethodCommonFinetuningInferenceParams↓\\downarrowPerf\.↑⁣/⁣↓\\uparrow/\\downarrowTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrowTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrow\(M\)\(TF\)\(min\)\(MB\)\(W\)\(Wh\)\(TF\)\(min/s\)\(MB\)\(W\)\(Wh/mWh\)LaMP\-1TinyLlama\-1\.1BFull\-FT1104\.550\.70725\.9940\.912424\.3376\.28256\.529\.463\.563\.753\.38\.6611\.933944\.0323\.2164\.2834\.265\.065\.056\.0LoRA4\.510\.73017\.4423\.48039\.6369\.49144\.155\.664\.764\.854\.98\.6612\.453966\.0314\.0065\.1658\.665\.665\.556\.6LoRA\+4\.510\.73217\.4427\.57608\.5324\.22148\.655\.664\.764\.955\.08\.6612\.523966\.0313\.5065\.4058\.765\.665\.656\.6QLoRA4\.510\.72817\.4439\.87451\.8282\.81187\.655\.564\.464\.854\.78\.6613\.822580\.0292\.4467\.3458\.665\.566\.056\.9BitFit0\.090\.48017\.3320\.97361\.7384\.11133\.865\.357\.557\.647\.88\.6611\.733944\.0331\.8564\.9068\.358\.358\.349\.3Qwen3\-1\.7BFull\-FT1720\.570\.73943\.0061\.222690\.6358\.92366\.126\.163\.963\.953\.014\.3312\.335080\.0321\.0065\.9830\.865\.865\.556\.5LoRA6\.420\.76028\.8334\.319097\.9367\.35210\.052\.665\.064\.554\.314\.3313\.335458\.0305\.0067\.7855\.666\.265\.956\.9LoRA\+6\.420\.75828\.8340\.618992\.9324\.24219\.452\.564\.964\.554\.214\.3313\.225490\.0307\.0067\.6355\.566\.265\.856\.8QLoRA6\.420\.73628\.8354\.816402\.2288\.72263\.752\.064\.264\.153\.614\.3315\.273916\.0281\.0071\.5055\.065\.665\.756\.6BitFit0\.120\.43928\.6731\.718392\.2382\.15201\.960\.355\.555\.044\.814\.3323\.775014\.0261\.00103\.3963\.356\.256\.447\.0Mamba\-1\.4BFull\-FT1383\.310\.67033\.9867\.014600\.5392\.60438\.426\.362\.062\.651\.611\.3330\.534294\.0215\.00109\.4131\.163\.564\.054\.4LoRA11\.130\.57022\.9248\.011260\.7381\.00304\.846\.259\.660\.149\.511\.3340\.654326\.0185\.00125\.3449\.260\.561\.151\.5LoRA\+11\.130\.69822\.9250\.811124\.9368\.74312\.249\.763\.163\.653\.011\.3336\.934326\.0195\.00120\.0352\.764\.164\.755\.0BitFit0\.390\.50122\.6540\.610701\.4417\.19282\.358\.557\.457\.947\.311\.3329\.904254\.0216\.00107\.6461\.558\.558\.949\.4Mamba2\-1\.3BFull\-FT1351\.870\.68635\.7977\.915290\.4230\.91299\.826\.662\.863\.052\.411\.9331\.084178\.0221\.00114\.4931\.463\.964\.454\.8LoRA8\.110\.70024\.0644\.78013\.4286\.44213\.450\.963\.564\.053\.811\.9339\.174212\.0189\.00123\.3853\.964\.164\.755\.1LoRA\+8\.110\.72224\.0678\.07680\.3198\.92258\.651\.463\.964\.654\.211\.9338\.334145\.0190\.00121\.3954\.564\.765\.355\.6BitFit0\.210\.51723\.8669\.87131\.7187\.31217\.961\.558\.258\.948\.611\.9329\.084200\.0217\.00105\.1864\.659\.059\.550\.0LaMP\-2TinyLlama\-1\.1BFull\-FT1104\.550\.69625\.9931\.111298\.1412\.28213\.729\.163\.463\.653\.38\.6667\.003825\.0200\.113724\.2733\.963\.464\.754\.4LoRA4\.510\.66617\.4418\.79399\.3414\.87129\.354\.063\.263\.053\.38\.6675\.003845\.0193\.604033\.3357\.062\.564\.053\.6LoRA\+4\.510\.69217\.4420\.09143\.4389\.40129\.854\.663\.963\.754\.08\.6675\.003854\.0192\.004000\.0057\.763\.264\.654\.2QLoRA4\.510\.70317\.4425\.58784\.4369\.88157\.254\.963\.964\.054\.18\.6692\.002580\.0169\.214324\.2658\.063\.465\.354\.9BitFit0\.090\.10117\.3317\.08606\.9415\.76117\.838\.230\.630\.320\.78\.6649\.003834\.0230\.213133\.4141\.330\.031\.221\.1Qwen3\-1\.7BFull\-FT1720\.570\.72343\.0051\.620397\.1398\.49342\.725\.763\.663\.652\.814\.3376\.004986\.0207\.804386\.8930\.463\.965\.154\.6LoRA6\.420\.57928\.8331\.622318\.8406\.71214\.247\.860\.259\.649\.414\.3376\.005494\.0216\.004560\.0050\.960\.061\.250\.6LoRA\+6\.420\.69928\.8333\.522799\.5388\.30216\.851\.163\.562\.952\.614\.3386\.005526\.0201\.814821\.0254\.163\.264\.453\.8QLoRA6\.420\.61228\.8340\.318678\.6371\.17249\.348\.861\.060\.850\.414\.33114\.003952\.0183\.335805\.4551\.860\.762\.551\.7BitFit0\.120\.04328\.6729\.521240\.9408\.20200\.720\.015\.114\.54\.314\.33237\.004954\.0205\.6313537\.3123\.013\.616\.14\.4Mamba\-1\.4BFull\-FT1383\.310\.59133\.9854\.014758\.6426\.78384\.124\.160\.060\.449\.511\.33371\.004582\.0157\.6516246\.7128\.958\.961\.749\.8LoRA11\.130\.48822\.9240\.312291\.6411\.96276\.743\.557\.057\.346\.811\.33652\.004700\.0116\.5021099\.4446\.555\.358\.446\.2LoRA\+11\.130\.63522\.9242\.712347\.2396\.39282\.148\.061\.561\.951\.311\.33484\.004690\.0140\.4818886\.7651\.160\.062\.950\.9BitFit0\.390\.36722\.6534\.611158\.9434\.57250\.653\.152\.152\.542\.011\.33373\.004244\.0153\.6115915\.7056\.150\.753\.541\.6Mamba2\-1\.3BFull\-FT1351\.870\.57635\.7932\.417391\.0402\.59217\.423\.660\.159\.849\.511\.93384\.004170\.0140\.0014933\.3328\.358\.661\.449\.5LoRA8\.110\.32624\.0625\.511626\.2431\.06183\.237\.650\.450\.440\.311\.93540\.004232\.0137\.0020550\.0040\.748\.451\.539\.3LoRA\+8\.110\.57224\.0631\.711514\.6365\.30193\.047\.460\.160\.150\.011\.93570\.004345\.0142\.0022483\.3350\.458\.061\.248\.9BitFit0\.210\.32623\.8627\.710482\.9373\.21172\.353\.550\.550\.540\.411\.93346\.004300\.0155\.0014897\.2256\.548\.751\.439\.6LaMP\-3TinyLlama\-1\.1BFull\-FT1104\.550\.67625\.99160\.511523\.2412\.711104\.026\.459\.060\.948\.88\.6616\.373964\.0306\.0083\.4731\.261\.862\.052\.8LoRA4\.510\.61117\.4496\.38662\.1411\.78660\.952\.860\.261\.950\.48\.6617\.453982\.0293\.0085\.2155\.862\.462\.753\.4LoRA\+4\.510\.61817\.44103\.68529\.1384\.27663\.552\.760\.161\.850\.38\.6617\.453982\.0294\.0285\.5155\.762\.462\.653\.4QLoRA4\.510\.63117\.44129\.87876\.8371\.46803\.652\.559\.861\.850\.18\.6619\.902582\.0265\.1987\.9555\.662\.263\.053\.7BitFit0\.091\.08117\.3387\.47866\.9411\.56599\.565\.355\.957\.546\.18\.6615\.153966\.0324\.6181\.9668\.458\.058\.349\.0Qwen3\-1\.7BFull\-FT1720\.570\.62643\.00270\.220442\.7400\.151802\.022\.959\.060\.848\.214\.3317\.755120\.0300\.5488\.9127\.662\.262\.353\.0LoRA6\.420\.63128\.83165\.615248\.5403\.881114\.748\.859\.461\.049\.014\.3317\.505494\.0299\.5987\.3851\.962\.262\.252\.9LoRA\+6\.420\.61428\.83175\.616325\.3386\.551131\.349\.059\.661\.249\.114\.3318\.675526\.0285\.7388\.8952\.062\.462\.353\.0QLoRA6\.420\.63328\.83215\.714345\.4366\.591317\.948\.859\.261\.148\.814\.3322\.533948\.0249\.7793\.8051\.862\.162\.553\.1BitFit0\.123\.09328\.67153\.714192\.5407\.511043\.950\.243\.545\.133\.114\.3325\.725054\.0269\.00115\.3053\.245\.946\.336\.7Mamba\-1\.4BFull\-FT1383\.310\.62033\.98276\.814751\.9426\.741968\.724\.958\.961\.248\.511\.3334\.104450\.0219\.96125\.0129\.761\.962\.552\.8LoRA11\.130\.64422\.92203\.910747\.7410\.441394\.847\.359\.161\.349\.011\.3340\.704366\.0201\.52136\.7050\.461\.662\.352\.5LoRA\+11\.130\.59822\.92214\.710998\.6397\.391422\.047\.859\.561\.849\.411\.3340\.854526\.0198\.21134\.9550\.862\.162\.752\.9BitFit0\.390\.66522\.65173\.29622\.2434\.831255\.261\.759\.061\.249\.011\.3333\.854245\.0220\.28124\.2764\.761\.562\.152\.4Mamba2\-1\.3BFull\-FT1351\.870\.58135\.79169\.217144\.5406\.701146\.925\.259\.961\.549\.411\.9332\.754228\.0227\.00123\.9130\.062\.463\.053\.3LoRA8\.110\.57524\.06134\.411063\.5429\.02961\.049\.260\.262\.050\.111\.9339\.624169\.0196\.00129\.4152\.362\.463\.153\.3LoRA\+8\.110\.59924\.06165\.311150\.6368\.861016\.248\.959\.961\.749\.811\.9341\.484068\.0188\.00129\.9852\.062\.162\.853\.1BitFit0\.210\.72123\.86147\.09902\.2376\.20921\.763\.658\.760\.648\.711\.9335\.034445\.0205\.00119\.7066\.660\.961\.451\.8

## VResults and Analysis

### V\-AStep 1: Best PEFT Method across Models

For each model and benchmark, we select the finetuning method with a strict energy\-first rule: the method with the highest finetuning NS\-E is selected, and if two or more methods share the same NS\-E, the one with the higher NS\# wins\. NS\-E measures energy efficiency using training time and power consumption, which is the binding constraint for battery\-powered on\-device fine\-tuning\. The memory\-focused NS\-M \(peak VRAM\) and the combined NS\# are still reported and discussed, since NS\-M identifies the method to prefer when memory rather than energy is the dominant constraint, but only NS\-E and the NS\# tie\-break determine the selection\. Although the parameter\-based NS is reported in the tables for completeness, it is not used for method selection or for the following analysis because its strong dependence on the number of trainable parameters can favor methods with poor task performance\. Table[IV](https://arxiv.org/html/2608.04488#S4.T4)reports the GLUE results, while Table[V](https://arxiv.org/html/2608.04488#S4.T5)reports the LaMP results\.

#### V\-A1Perspective on PEFT Methods

A key observation is thatLoRA\+ is by far the most frequently selected PEFT method, being chosen in 18 of the 24 model–task pairs\. Once energy alone decides the selection, the choice is also much less architecture\-dependent than it is under a memory\-aware rule: LoRA\+ wins for both SSMs and for both Transformers on almost every task, and QLoRA, whose advantage lies in VRAM rather than energy, survives in a single configuration\.

LoRA\+, specifically, is ranked the top fine\-tuning method in 3 out of 6 tasks for TinyLlama\-1\.1B, 4 out of 6 for Qwen3\-1\.7B, 6 out of 6 for Mamba\-1\.4B, and 5 out of 6 for Mamba2\-1\.3B\. One of the reasons why LoRA\+ is the best method is that LoRA\+ modifies only the optimizer and not the adapter itself\. It keeps the same rank, the same trainable\-parameter count and the same FLOPs as LoRA, and simply assigns a larger learning rate to the up\-projectionBBthan to the down\-projectionAA\[[12](https://arxiv.org/html/2608.04488#bib.bib9)\]\. Any accuracy it gains is therefore obtained at essentially zero additional cost, and since NetScore multiplies a performance term by cost terms that remain unchanged, the gain translates directly into a higher NS\-E and NS\-M\. This is visible in the tables, where LoRA and LoRA\+ share identical parameter counts and TFLOPs and differ only marginally in time, VRAM and power \(e\.g\., Mamba2\-1\.3B on SST\-2: 34\.24 vs\. 31\.70 min and 12,903 vs\. 12,805 MB\), while accuracy increases from 0\.955 to 0\.967\. Secondly, becauseBBis initialized to zero whileAAis random, a single shared learning rate under\-trainsBBrelative toAA; the enlarged learning rate onBBmainly accelerates convergence\. Under the short training budget used here, faster convergence appears as higher accuracy at the same time and energy, which is exactly what NS\-E rewards\. Third, the benefit is largest where the pretrained model is furthest from the target task: the largest LoRA→\\rightarrowLoRA\+ jumps occur on the personalization benchmarks and on the SSMs \(Mamba2\-1\.3B on LaMP\-2, 0\.326→\\rightarrow0\.572\), whereas TinyLlama on SST\-2 is already near ceiling and gains nothing \(0\.959 vs\. 0\.957\)\.

LoRA, the standard PEFT method, is selected in 4 of the 24 pairs, always with an almost negligible NS\-E difference from LoRA\+: SST\-2 for TinyLlama, LaMP\-1 for Qwen3, and LaMP\-3 for both TinyLlama and Mamba2\. For example, on SST\-2 TinyLlama reaches an NS\-E of 70\.0 with LoRA against 69\.9 with LoRA\+, and on LaMP\-3 it reaches 60\.2 against 60\.1\.

Full\-FTis selected in only 1 of the 24 model–task pairs\. On LaMP\-2, Qwen3\-1\.7B attains its highest accuracy \(0\.723\) with Full\-FT, which is enough to give it the highest NS\-E of that group \(63\.6, against 63\.5 for LoRA\+\) even though its base NS collapses to 25\.7 under the parameter penalty\. A second near\-miss is TinyLlama on STS\-B, where the whole finetuning run lasts under five minutes \(4\.75 min\) so the energy penalty of updating all parameters nearly disappears; Full\-FT gives the best correlation \(0\.909\) and the highest NS\-M of that group \(68\.1\), but LoRA\+ still takes the higher NS\-E \(70\.6 vs\. 70\.2\) and is therefore selected under the energy\-first rule\. Outside these cases full finetuning is never cost\-efficient: from Table[V](https://arxiv.org/html/2608.04488#S4.T5), Qwen3\-1\.7B on LaMP\-1 achieved a lower accuracy of 0\.739 under full fine\-tuning versus 0\.760 under LoRA, while consuming 366\.1 Wh against 210\.0 Wh\. This trend is seen in 23 of the 24 model–task pairs, showing that updating all parameters is generally not cost\-efficient at this model size\.

BitFit, which updates only the bias terms in the fine\-tuned model, failed to generate competitive results for any of the models examined and is never selected\. The largest declines were found on STS\-B \(Mamba2: 0\.023 correlation\), LaMP\-2 \(Qwen3: 0\.043 accuracy\), and LaMP\-3 \(Qwen3: 3\.093 RMSE\), demonstrating that BitFit is not the optimal choice for fine\-tuning model in general understanding and user personalization tasks\. On STS\-B, BitFit even produces a negative NS\# \(−3\.6\-3\.6\) for Mamba2\-1\.3B, confirming that bias\-only adaptation cannot fit the regression objective\.

QLoRA, available only for the Transformer models, trades training time for memory\. On SST\-2 it reduces TinyLlama’s peak finetuning VRAM from 16,480 MB \(LoRA\) to 4,225 MB \(Table[IV](https://arxiv.org/html/2608.04488#S4.T4)\), but the de\-quantization overhead lengthens training from 19\.2 to 31\.1 min and raises energy from 83\.6 to 147\.3 Wh\. It therefore almost never achieves the best NS\-E within a model group \(its only case is a tie with LoRA\+ at 63\.9 for TinyLlama on LaMP\-2\), yet it consistently achieves the best NS\-M: it holds the highest NS\-M for both Transformers on SST\-2 \(70\.1 and 69\.5\) and QNLI \(69\.3 and 68\.8\), as well as for TinyLlama on LaMP\-2 \(64\.0\), that is, in 5 of the 12 Transformer model–task pairs\. Under the energy\-first rule this memory advantage no longer drives the selection, so QLoRA is the selected method in only 1 of those 12 pairs, namely TinyLlama on LaMP\-2, where it ties LoRA\+ on NS\-E and wins the NS\# tie\-break \(54\.1 vs\. 54\.0\)\. QLoRA thus remains the method of choice only when VRAM, not energy, is the binding constraint\. The reason why QLoRA is unavailable for Mamba models is because SSM blocks have highly sensitive feature maps within the selective scan mechanism and massive outliers in output activations not present in Transformer attention modules\[[4](https://arxiv.org/html/2608.04488#bib.bib39)\], causing standard quantization to fail\.

#### V\-A2Perspective on Models

A key observation is thatSSM\-based models show a greater improvement from LoRA to LoRA\+ than the Transformer\-based models\.

Transformer Models: For Transformer\-based models, the gap between LoRA and LoRA\+ is much smaller than for SSMs\. On TinyLlama\-1\.1B, standard LoRA already matches or slightly exceeds LoRA\+ on several tasks \(e\.g\., an NS\-E of 70\.0 vs\. 69\.9 on SST\-2 in Table[IV](https://arxiv.org/html/2608.04488#S4.T4), and 60\.2 vs\. 60\.1 on LaMP\-3 in Table[V](https://arxiv.org/html/2608.04488#S4.T5)\), while Qwen3\-1\.7B shows only marginal LoRA\+ gains on SST\-2 \(69\.5 vs\. 69\.6\) and none on QNLI \(67\.5 for both\)\. Transformers also uniquely support QLoRA, which is decisive whenever VRAM is weighted: it holds the highest NS\-M for both Transformers on SST\-2 and QNLI and for TinyLlama on LaMP\-2, where it is also the best configuration on the base NetScore \(NS 54\.9, Table[VII](https://arxiv.org/html/2608.04488#S5.T7)\) and the only pair in which it is the selected method under the energy\-first rule\. Overall, Transformer models require less PEFT specialization than SSMs to reach competitive NetScore values under the same training budget\.

SSM Models: On LaMP\-1, similar to LaMP\-2 and LaMP\-3, Mamba\-1\.4B improves from an accuracy of 0\.570 with LoRA to 0\.698 with LoRA\+, a 22% gain, while TinyLlama improves from an accuracy of 0\.730 to an accuracy of 0\.732\. This aligns with recent findings that PEFT methods are generally more effective for Mamba than for Transformers\[[40](https://arxiv.org/html/2608.04488#bib.bib36)\]and that Mamba exhibits greater stability under PEFT\[[11](https://arxiv.org/html/2608.04488#bib.bib37)\]\.

We speculate that two properties of Mamba explain this\. First, the adapters are attached to the projections that feed the selective scan, which controls how information is carried across the whole sequence\. Therefore, a small number of trainable weights can change the behaviour of the model much more than the same number of weights inside an attention block\[[40](https://arxiv.org/html/2608.04488#bib.bib36)\]\. Second, Mamba has been shown to stay stable when adapted with PEFT\[[11](https://arxiv.org/html/2608.04488#bib.bib37)\], allowing it to absorb larger weight updates without destabilizing the training process\. This is exactly what LoRA\+ does with its larger learning rate, which would explain why moving from LoRA to LoRA\+ helps the SSMs far more \(up to\+75%\+75\\%relative accuracy for Mamba2\-1\.3B on LaMP\-2\) than the Transformers, where the gain is a few percent at most\. As SSM\-specific PEFT methods mature\[[9](https://arxiv.org/html/2608.04488#bib.bib38)\], the gap between SSM and Transformer\-based models may decrease\.

#### V\-A3Perspective on Benchmarks

A key observation is thatthe benchmark family and not the fine\-tuning method, sets the efficiency level that can be reached, while LoRA\+ is a safer default on GLUE than on LaMP\. Every selected GLUE configuration attains a higher finetuning NS\-E \(66\.7–70\.6, Table[VI](https://arxiv.org/html/2608.04488#S5.T6)\) than every selected LaMP configuration \(59\.5–65\.0, Table[VII](https://arxiv.org/html/2608.04488#S5.T7)\), a separation of 7\.9% between the two maxima that no choice of method is able to close, because the LaMP tasks are generative, use longer inputs \(512 versus 128 tokens\) and therefore consume far more time and energy per run\.

Within each family however, the winning method is not equally consistent\. LoRA\+ is the selected method in 11 of the 12 GLUE configurations \(all four models on QNLI and STS\-B, and both SSMs together with Qwen3\-1\.7B on SST\-2\), the single exception being TinyLlama\-1\.1B on SST\-2, and by the smallest possible margin \(NS\-E 70\.0 for LoRA against 69\.9 for LoRA\+\)\. On LaMP it is selected in only 7 of the 12 configurations: plain LoRA takes three of the remaining slots \(Qwen3\-1\.7B on LaMP\-1, TinyLlama\-1\.1B and Mamba2\-1\.3B on LaMP\-3\), while QLoRA and Full\-FT take one each on LaMP\-2\. The short, high\-accuracy GLUE classification runs leave little room for accuracy differences between methods, so the small but consistent convergence benefit of LoRA\+ decides the outcome, whereas the much longer generative LaMP runs produce larger accuracy spreads in which plain LoRA, QLoRA, or even Full\-FT can occasionally take the highest NS\-E\.

We present the full per\-task breakdown across three benchmarks \(SST2, QNLI, STS\-B\) in Tables[IV](https://arxiv.org/html/2608.04488#S4.T4)and the three personalization benchmarks \(LaMP\-1, LaMP\-2, LaMP\-3\) in Table[V](https://arxiv.org/html/2608.04488#S4.T5)\. Each table reports efficiency \(Time, VRAM, Power, Energy\), task performance, and the four NetScore variants \(NS, NS\-E, NS\-M, NS\#\) for both finetuning and inference\. For each model, the optimal tuning method \(the one with the highest NS\-E, with NS\# breaking ties\) is marked inboldwith a red\-shaded method name\. Below we analyze each detailed table in turn\.

GLUE Tasks:SST\-2\.On this sentiment\-classification task, all four models reach high accuracy under LoRA and LoRA\+ \(0\.955–0\.967\), with Mamba2\-1\.3B\+LoRA\+ attaining the single highest accuracy \(0\.967\)\. Because SST\-2 is comparatively easy, even bias\-only BitFit stays competitive for the Transformers \(TinyLlama 0\.920, Qwen3 0\.900\) and therefore takes the top base NetScore \(NS 77\.0 and 73\.7\) owing to its negligible number of trainable parameters \(0\.09–0\.12 million\)\. However, BitFit performs poorly on Mamba2\-1\.3B \(0\.500 accuracy\), showing that bias\-only tuning is unreliable for SSMs even on relatively easy tasks\. Once the objective shifts to energy \(NS\-E\), LoRA and LoRA\+ overtake BitFit for every model\. QLoRA shows the expected memory benefit, cutting TinyLlama’s finetuning VRAM from 16,480 MB \(LoRA\) to 4,225 MB, but its de\-quantization overhead lengthens training time \(19\.2 min→\\rightarrow31\.1 min\) and raises energy, so it never leads on NS or NS\-E even though it takes the highest NS\-M for both Transformers \(70\.1 and 69\.5\); under the energy\-first rule it is therefore not selected for either, the choice falling to LoRA for TinyLlama \(NS\-E 70\.0\) and to LoRA\+ for Qwen3 \(69\.6\)\. Inference VRAM is 1\.3–5\.9×\\timessmaller than finetuning across all configurations, the smallest ratio being QLoRA, which is already memory\-frugal during training\.QNLI\.QNLI is the longest\-running GLUE task, which further increases the energy gap between architectures\. LoRA\+ gives the best accuracy for three of the four models \(0\.946, 0\.928, 0\.926\), the exception being TinyLlama, where Full\-FT is marginally ahead \(0\.934 vs\. 0\.932\), while BitFit again wins the base NetScore \(e\.g\., TinyLlama NS 75\.4\) but at a steep accuracy cost \(TinyLlama 0\.839, Mamba2\-1\.3B 0\.500\)\. The training\-time difference is noticeable: TinyLlama finetunes in≈\\approx78–81 min versus≈\\approx148 min for Mamba\-1\.4B, so despite comparable VRAM the SSMs record lower NS\-E\. QLoRA again minimizes VRAM \(TinyLlama 4,797 MB, Qwen3 11,114 MB\) at the price of the longest training times \(TinyLlama 124\.6 min\), reproducing the memory–energy trade\-off observed on SST\-2; because those long runs depress its NS\-E, LoRA\+ is instead the selected method for all four models on this task \(67\.9, 67\.5, 66\.7, and 67\.1\)\.STS\-B\.STS\-B is the one GLUE task where BitFit fails completely: as a correlation \(regression\) objective, bias\-only tuning yields near\-zero scores \(TinyLlama 0\.310, Qwen3 0\.316, Mamba\-1\.4B 0\.146, Mamba2\-1\.3B 0\.023\) and even a negative NS\# \(−3\.6\-3\.6\) for Mamba2\-1\.3B\. Consequently LoRA\+ becomes the best method per model on the base NetScore and on NS\-E, unlike the other GLUE tasks where BitFit topped the base NS, and it is the selected method for all four models; the two Transformers are the exception on NS\-M, where Full\-FT retains the highest value \(68\.1 for TinyLlama and 67\.3 for Qwen3\) because the extremely short runs \(4\.75 and 12\.63 min\) leave little room for an efficiency penalty\. LoRA\+ also recovers most of the accuracy that plain LoRA loses \(e\.g\., Mamba2\-1\.3B 0\.705→\\rightarrow0\.847\), highlighting the benefit of its differential learning rates\. Because all configurations train in under 13 min, STS\-B is the most efficient GLUE task and shows the largest NetScore separation between strong and weak methods\.

TABLE VI:GLUE Summary for Best Training Configurations \(finetuning \+ inference\)\.The performance metric \(Perf\.\) is accuracy for SST\-2 and QNLI and Pearson–Spearman correlation for STS\-B; it is shared across both sections\. TheBest Methodis selected using the finetuning results, including Full\-FT, under a strict energy\-first rule: the method with the highest NS\-E is selected, and a tie on NS\-E is broken by the higher NS\#\. NS = NetScore: NS\-E uses time and power; NS\-M uses VRAM; NS\# uses VRAM, time, and power\. All variants use the1/81/8coefficient\. Finetuning Time/Energy are in min/Wh; Inference Time/Energy are in s/mWh\.Boldmarks the best value within each task, computed separately for the finetuning and inference sections\.TaskModelArchitectureBest MethodPerf\.FinetuningInferenceTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrowTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrow\(TF\)\(min\)\(MB\)\(W\)\(Wh\)\(TF\)\(s\)\(MB\)\(W\)\(mWh\)SST\-2TinyLlama\-1\.1BTransformerLoRA0\.95916\.1719\.1816479\.8261\.4383\.5760\.670\.068\.759\.58\.0318\.733654\.094\.00489\.0663\.771\.270\.462\.3Qwen3\-1\.7BLoRA\+0\.96223\.4922\.3020047\.6350\.26130\.1857\.569\.668\.658\.811\.6723\.004678\.0101\.00645\.2860\.670\.970\.261\.7Mamba\-1\.4BSSMLoRA\+0\.95821\.2449\.3710779\.6383\.87315\.8655\.568\.669\.258\.510\.4823\.503970\.0105\.00685\.4258\.670\.870\.361\.8Mamba2\-1\.3BLoRA\+0\.96722\.3831\.7012805\.1357\.50188\.8856\.869\.369\.159\.011\.0931\.704356\.0104\.00915\.7859\.970\.670\.361\.5QNLITinyLlama\-1\.1BTransformerLoRA\+0\.93216\.1781\.4616728\.6283\.19384\.4960\.167\.968\.257\.38\.03153\.003158\.086\.533677\.5363\.268\.570\.059\.7Qwen3\-1\.7BLoRA\+0\.94623\.49104\.2023605\.8391\.14679\.2857\.367\.568\.156\.611\.67144\.005668\.099\.503980\.0060\.368\.669\.759\.3Mamba\-1\.4BSSMLoRA\+0\.92821\.24147\.9019639\.4421\.401038\.7655\.066\.768\.056\.010\.48148\.003942\.097\.614012\.8658\.068\.369\.759\.3Mamba2\-1\.3BLoRA\+0\.92622\.38101\.3517726\.1406\.87687\.2756\.167\.168\.056\.511\.09197\.004390\.098\.755403\.8259\.167\.969\.658\.8STS\-BTinyLlama\-1\.1BTransformerLoRA\+0\.89316\.172\.5619197\.1367\.7315\.6959\.470\.667\.359\.98\.0333\.005676\.0102\.38938\.4862\.469\.268\.659\.8Qwen3\-1\.7BLoRA\+0\.89323\.493\.4223622\.6358\.6020\.4456\.270\.367\.159\.411\.6741\.009536\.0118\.151345\.6059\.368\.868\.158\.9Mamba\-1\.4BSSMLoRA\+0\.86921\.245\.1818798\.0418\.6136\.1453\.869\.266\.958\.510\.4843\.004526\.0104\.001242\.2256\.968\.468\.459\.3Mamba2\-1\.3BLoRA\+0\.84722\.383\.5414986\.6381\.3622\.5054\.569\.366\.758\.911\.0952\.004488\.0102\.001473\.3357\.667\.868\.058\.7

TABLE VII:LaMP Summary for Best Training Configurations \(finetuning \+ inference\)\.The performance metric \(Perf\.\) is accuracy for LaMP\-1 and LaMP\-2 and RMSE \(↓\\downarrow\) for LaMP\-3; it is shared across both sections\. TheBest Methodis selected using the finetuning results, including Full\-FT, under a strict energy\-first rule: the method with the highest NS\-E is selected, and a tie on NS\-E is broken by the higher NS\#\. NS = NetScore: NS\-E uses time and power; NS\-M uses VRAM; NS\# uses VRAM, time, and power\. All variants use the1/81/8coefficient\. Finetuning Time/Energy are in min/Wh\. Inference Energy is in mWh, while inference Time units are given inline \(min for LaMP\-1/LaMP\-3 and s for LaMP\-2\)\.Boldmarks the best value within each task, computed separately for finetuning and inference\.TaskModelArchitectureBest MethodPerf\.FinetuningInferenceTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrowTFLOPs↓\\downarrowTime↓\\downarrowVRAM↓\\downarrowPower↓\\downarrowEnergy↓\\downarrowNS↑\\uparrowNS\-E↑\\uparrowNS\-M↑\\uparrowNS\#↑\\uparrow\(TF\)\(min\)\(MB\)\(W\)\(Wh\)\(TF\)\(MB\)\(W\)\(mWh\)LaMP\-1TinyLlama\-1\.1BTransformerLoRA\+0\.73217\.4427\.57608\.5324\.22148\.655\.664\.764\.955\.08\.6612\.52min3966313\.5065400\.0058\.765\.665\.656\.6Qwen3\-1\.7BLoRA0\.76028\.8334\.319097\.9367\.35210\.052\.665\.064\.554\.314\.3313\.33 min5458305\.0067780\.0055\.666\.265\.956\.9Mamba\-1\.4BSSMLoRA\+0\.69822\.9250\.811124\.9368\.74312\.249\.763\.163\.653\.011\.3336\.93 min4326195\.00120030\.0052\.764\.164\.755\.0Mamba2\-1\.3BLoRA\+0\.72224\.0678\.07680\.3198\.92258\.651\.463\.964\.654\.211\.9338\.33 min4145190\.00121390\.0054\.564\.765\.355\.6LaMP\-2TinyLlama\-1\.1BTransformerQLoRA0\.70317\.4425\.58784\.4369\.88157\.254\.963\.964\.054\.18\.6692\.00 s2580169\.214324\.2658\.063\.465\.354\.9Qwen3\-1\.7BFull\-FT0\.72343\.0051\.620397\.1398\.49342\.725\.763\.663\.652\.814\.3376\.00s4986207\.804386\.8930\.463\.965\.154\.6Mamba\-1\.4BSSMLoRA\+0\.63522\.9242\.712347\.2396\.39282\.148\.061\.561\.951\.311\.33484\.00 s4690140\.4818886\.7651\.160\.062\.950\.9Mamba2\-1\.3BLoRA\+0\.57224\.0631\.711514\.6365\.30193\.047\.460\.160\.150\.011\.93570\.00 s4345142\.0022483\.3350\.458\.061\.248\.9LaMP\-3TinyLlama\-1\.1BTransformerLoRA0\.61117\.4496\.38662\.1411\.78660\.952\.860\.261\.950\.48\.6617\.45min3982293\.0085210\.0055\.862\.462\.753\.4Qwen3\-1\.7BLoRA\+0\.61428\.83175\.616325\.3386\.551131\.349\.059\.661\.249\.114\.3318\.67 min5526285\.7388890\.0052\.062\.462\.353\.0Mamba\-1\.4BSSMLoRA\+0\.59822\.92214\.710998\.6397\.391422\.047\.859\.561\.849\.411\.3340\.85 min4526198\.21134950\.0050\.862\.162\.752\.9Mamba2\-1\.3BLoRA0\.57524\.06134\.411063\.5429\.02961\.049\.260\.262\.050\.111\.9339\.62 min4169196\.00129410\.0052\.362\.463\.153\.3

LaMP Tasks:LaMP\-1\.On personalized citation identification, plain LoRA achieves the highest accuracy for Qwen3\-1\.7B \(0\.760\), while for TinyLlama LoRA\+ is marginally ahead \(0\.732 vs\. 0\.730\) and LoRA\+ is clearly needed to reach higher results on the SSMs \(Mamba\-1\.4B 0\.570→\\rightarrow0\.698; Mamba2\-1\.3B 0\.700→\\rightarrow0\.722\)\. Unlike the GLUE tasks, BitFit remains moderately usable here \(0\.439–0\.517 accuracy\) and therefore still tops the base NetScore \(e\.g\., TinyLlama NS 65\.3\), but LoRA/LoRA\+ dominate once energy \(NS\-E\) is considered\. This is also the benchmark where Qwen3\-1\.7B overtakes TinyLlama at performance\-weighted operating points, because its accuracy advantage \(0\.760 vs\. 0\.732\) outweighs the energy penalty only when accuracy is heavily emphasized\.LaMP\-2\.Personalized movie tagging exposes the largest LoRA→\\rightarrowLoRA\+ gains: Qwen3\-1\.7B rises from 0\.579 to 0\.699, Mamba\-1\.4B from 0\.488 to 0\.635, and Mamba2\-1\.3B from 0\.326 to 0\.572\. Here TinyLlama\+QLoRA is the best configuration on the base NetScore \(NS 54\.9\) while also reaching the top accuracy among the PEFT methods \(0\.703\), a rare case where quantization beats other approaches\. This is also the only task in the whole study where Full\-FT is selected, for Qwen3\-1\.7B, whose 0\.723 accuracy is enough to give it the highest NS\-E of the group \(63\.6, against 63\.5 for LoRA\+\) despite a base NS of only 25\.7\. BitFit is catastrophic for Qwen3\-1\.7B \(0\.043 accuracy, NS 20\.0\), confirming that bias\-only tuning cannot handle the larger label space of this task\. The generative nature of LaMP\-2 also makes SSM inference expensive \(Mamba\-1\.4B 484 s vs\. TinyLlama 75 s\), reducing the theoretical latency advantage of SSMs in practice\.LaMP\-3\.Personalized product rating is measured by RMSE and is the most energy\-intensive benchmark \(599–1968 Wh of finetuning energy\)\. LoRA and LoRA\+ give the lowest RMSE for every model \(e\.g\., TinyLlama 0\.611, Mamba2\-1\.3B 0\.575\), whereas BitFit degrades poorly \(Qwen3\-1\.7B RMSE 3\.093\), which pulls its NetScore down sharply once the accuracy term is applied\. Because every configuration consumes a large amount of energy, the NetScore denominators are similar and the spread between models is the smallest among all six tasks\. This makes LaMP\-3 a low\-separation benchmark in which secondary constraints \(VRAM, latency\) can reasonably drive the choice\.

### V\-BStep 2: Best Model across PEFT Methods

#### V\-B1Model Perspective

Having selected the best method per model for each of the task in Step 1, Step 2 compares these four final configurations of the models with respect to their efficiency\. The comparison connects the per\-task summaries of the best training configurations in Table[VI](https://arxiv.org/html/2608.04488#S5.T6)\(GLUE\) and Table[VII](https://arxiv.org/html/2608.04488#S5.T7)\(LaMP\) with the detailed per\-configuration results in Tables[IV](https://arxiv.org/html/2608.04488#S4.T4)and[V](https://arxiv.org/html/2608.04488#S4.T5)\.

##### Transformer\-based models\.

TinyLlama\-1\.1B has the highest base NetScore \(NS\) among the best\-method configurations on all six benchmarks \(Tables[VI](https://arxiv.org/html/2608.04488#S5.T6)and[VII](https://arxiv.org/html/2608.04488#S5.T7)\): 60\.6 on SST\-2, 60\.1 on QNLI, 59\.4 on STS\-B, 55\.6 on LaMP\-1, 54\.9 on LaMP\-2, and 52\.8 on LaMP\-3\. STS\-B deserves a note: the energy\-first rule gives TinyLlama LoRA\+ \(NS\-E 70\.6\) rather than Full\-FT, which holds the highest NS\-M of that group; had Full\-FT been selected, the parameter penalty of updating all 1\.1B weights would have cut TinyLlama’s base NS to 34\.1 and left Qwen3 \(56\.2\) in front\. TinyLlama leads or ties the energy\-focused variant \(NS\-E\) on five of the six benchmarks \(70\.0 on SST\-2, 67\.9 on QNLI, 70\.6 on STS\-B, 63\.9 on LaMP\-2, and 60\.2 on LaMP\-3, tied with Mamba2\-1\.3B\), losing only on LaMP\-1 to Qwen3 \(65\.0 vs\. 64\.7\)\. It leads the memory\-focused variant \(NS\-M\) on four of the six benchmarks, losing on SST\-2 to Mamba\-1\.4B with LoRA\+ \(69\.2 vs\. 68\.7\) and on LaMP\-3 to Mamba2\-1\.3B with LoRA \(62\.0 vs\. 61\.9\)\. One of the reasons that explains why TinyLlama\-1\.1B outperformed other transformer\-based model is that TinyLlama has fewer parameters \(1\.1 billion\) than Qwen3 \(1\.7 billion\), resulting in lower energy costs and lower VRAM requirements\.

In addition, TinyLlama has an optimized Transformer inference/training stack compared to SSM\-based models\. For instance, on STS\-B \(Table[IV](https://arxiv.org/html/2608.04488#S4.T4)\), TinyLlama with LoRA\+ achieved an NS of 59\.4 and an NS\-E of 70\.6 at only 15\.69 Wh, while Qwen3 with LoRA\+ achieved an NS of 56\.2 and an NS\-E of 70\.3 at 20\.44 Wh—meaning that Qwen3 used 30% more energy \(with a 56% larger model\) to reach the same 0\.893 correlation\. Qwen3\-1\.7B, despite its larger size and higher raw accuracy on several tasks, is penalized by the model\-size and energy terms of the NetScore variants\. Its only leading position is under the energy\-focused NS\-E on LaMP\-1, where Qwen3 with LoRA reaches 65\.0 versus 64\.7 for TinyLlama with LoRA\+ \(Table[V](https://arxiv.org/html/2608.04488#S4.T5)\), because its accuracy advantage of 0\.760 versus TinyLlama’s 0\.732 is large enough to outweigh its 41% higher training energy \(210\.0 vs\. 148\.6 Wh\)\. This shows how the choice of NetScore variant can shift the optimal model: under the base NS, NS\-M, and NS\# \(size\-, memory\-, and all\-inclusive priorities\), TinyLlama wins even on LaMP\-1, whereas under the energy\-focused NS\-E, Qwen3 prevails\.

##### SSM\-based models\.

Under their best LoRA\-family configurations, Mamba2\-1\.3B outperformed Mamba\-1\.4B in NetScore values on five of the six tasks \(all except LaMP\-2\) despite having fewer parameters, e\.g\., an NS\-E of 69\.3 versus 68\.6 on SST\-2 \(Table[IV](https://arxiv.org/html/2608.04488#S4.T4)\) and of 63\.9 versus 63\.1 on LaMP\-1 \(Table[V](https://arxiv.org/html/2608.04488#S4.T5)\)\. On SST\-2, Mamba2 achieved an accuracy of 0\.967 – the highest among all models – while consuming 189 Wh, whereas Mamba\-1\.4B reached an accuracy of 0\.958 with 316 Wh\. Similarly, on LaMP\-1, Mamba2 required only 259 Wh to reach a higher accuracy \(0\.722\) than Mamba\-1\.4B, which consumed 312 Wh for an accuracy of 0\.698\. This shows a generation\-to\-generation improvement in SSM architectures\. However, neither Mamba model demonstrated the ability to keep pace with the Transformers in terms of base NetScore values on any of the six benchmarks: on STS\-B, for instance, TinyLlama’s selected LoRA\+ configuration reaches a base NS of 59\.4 against 54\.5 for Mamba2\-1\.3B and 53\.8 for Mamba\-1\.4B\. The SSMs lead only on the memory\-focused NS\-M, and only twice: Mamba\-1\.4B on SST\-2 \(69\.2\) and Mamba2\-1\.3B on LaMP\-3 \(62\.0\)\. The main reason for this disparity lies in the training duration: on QNLI, Mamba\-1\.4B requires approximately 148 minutes to train, whereas TinyLlama requires just 81 minutes under the same LoRA\+ configuration; similarly, on LaMP\-3, Mamba\-1\.4B takes 215 minutes to train, whereas TinyLlama takes 104 minutes to train\. These longer training durations can be attributed to a variety of factors related to the immature nature of the SSM software ecosystem, which contribute to increased energy utilization and reduced NetScore values\.

Although Mamba’s selective scan hasO​\(N\)O\(N\)complexity in the sequence lengthNN, compared withO​\(N2\)O\(N^\{2\}\)for self\-attention\[[10](https://arxiv.org/html/2608.04488#bib.bib7)\], Mamba\-1\.4B takes longer to train than the larger Qwen3\-1\.7B on five of the six tasks \(49\.4 versus 22\.3 minutes on SST\-2 and 147\.9 versus 104\.2 minutes on QNLI\)\. It is also slower during inference on all six tasks \(484 versus 76 seconds on LaMP\-2\), despite having about 18% fewer parameters and a lower TFLOPs cost \(21\.2–22\.9 TF versus 23\.5–28\.8 TF\)\. This suggests that the difference comes from actual GPU use rather than theoretical complexity\. One possible reason is that our sequence lengths \(128 tokens on GLUE and 512 on LaMP\) are too short for the quadratic cost of attention to become dominant\. In addition, Mamba’s sequential scan requires frequent memory access and cannot use GPU tensor cores as effectively as the dense matrix operations in Transformers\[[10](https://arxiv.org/html/2608.04488#bib.bib7)\]\. Its beam\-search decoding also uses less optimized kernels to copy and reorder the recurrent state at each step\[[4](https://arxiv.org/html/2608.04488#bib.bib39)\]\. These are mainly software and implementation limits, as Mamba\-2 uses a matrix\-based scan that is reported to be 2–8×\\timesfaster than Mamba\[[6](https://arxiv.org/html/2608.04488#bib.bib8)\]and trains in 31\.7 versus 49\.4 minutes on SST\-2 in our results\.

#### V\-B2VRAM Perspective

While VRAM is not directly accounted for within the base NetScore, it is an important constraint for edge deployments\. Mamba models most of the times offer a VRAM advantage over Transformer\-based models of comparable capability: on LaMP\-1, for example, Mamba2 utilizes only 7,680 MB of VRAM, essentially matching the much smaller TinyLlama \(7,609 MB with LoRA\+\) while using only 40% of the 19,098 MB required by Qwen3\. Therefore, in environments where memory is constrained, SSM\-based models may be preferred regardless of their lower NetScore values\. Additionally, while number of parameters directly affects the VRAM used, it is also influenced by architectural differences, PEFT method used and specific task tested\. For example, even though TinyLlama is the smallest model tested, intuitively it should use the least VRAM, however due to architecture differences, both Mamba models can use substaintially less VRAM even though they have more parameters\. In STS\-B benchmark, TinyLlama uses at most 24209 MB of VRAM, while Mamba1 and Mamba2 use at most 19158 and 16998 MB of VRAM respectively\. Note, that this trend can change depending on the task, as for example on LaMP\-3 benchmark task, TinyLlama manages to use on average the least VRAM compared to other models\.

#### V\-B3Benchmark Perspective

The relative rankings of models were consistent within each benchmark family\. Across all three GLUE tasks, the base NetScore ranking of the best configurations is identical, TinyLlama\>\>Qwen3\>\>Mamba2\-1\.3B\>\>Mamba\-1\.4B \(60\.6\>\>57\.5\>\>56\.8\>\>55\.5 on SST\-2, 60\.1\>\>57\.3\>\>56\.1\>\>55\.0 on QNLI, and 59\.4\>\>56\.2\>\>54\.5\>\>53\.8 on STS\-B, Table[VI](https://arxiv.org/html/2608.04488#S5.T6)\)\. STS\-B joins this pattern only because the energy\-first rule selects LoRA\+ for TinyLlama; under a memory\-weighted criterion Full\-FT would be chosen there instead, dropping TinyLlama’s base NS to 34\.1 and placing it last\. On the LaMP tasks, TinyLlama kept its lead on the base NetScore for all three benchmarks\. Qwen3 remains second on LaMP\-1 \(52\.6\), but falls to last on LaMP\-2 \(25\.7\) because Full\-FT is its selected method there, and Mamba2\-1\.3B moves ahead of Qwen3 into second place on LaMP\-3 \(52\.8\>\>49\.2\>\>49\.0\>\>47\.8, Table[VII](https://arxiv.org/html/2608.04488#S5.T7)\), reflecting the heavier size and energy penalties of Qwen3’s 1\.7B parameters on the personalization workloads\. LaMP\-3 stands out as the task where model differences are smallest: among the best LoRA\-family configurations, the finetuning NS\-E values range only from 59\.5 \(Mamba\-1\.4B with LoRA\+\) to 60\.2 \(TinyLlama with LoRA and Mamba2\-1\.3B with LoRA\), a spread of roughly 1% \(Table[V](https://arxiv.org/html/2608.04488#S4.T5)\)\. Furthermore, LaMP\-3 was found to be the most energy intensive task evaluated\. It consumed between 661–1422 Wh across all four models, thus, due to the larger denominators used in the NetScore formula, the differences among scores are reduced and make selections regarding method/model less decisive for this specific benchmark\.

#### V\-B4Training Time Perspective

Training time follows a consistent pattern across the six benchmarks: under a matched LoRA\+ configuration, TinyLlama\-1\.1B is the fastest of the four models on every task\. For example, on QNLI \(Table[IV](https://arxiv.org/html/2608.04488#S4.T4)\) TinyLlama requires only about 81 minutes to train, compared to 104 minutes for Qwen3\-1\.7B and 148 minutes for Mamba\-1\.4B; on LaMP\-3 \(Table[V](https://arxiv.org/html/2608.04488#S4.T5)\) it takes 104 minutes against 176 minutes for Qwen3 and 215 minutes for Mamba\-1\.4B\. This is mainly due to its lower parameter count and the more mature Transformer training stack compared to the SSM\-specific kernels\. Because the energy\-first rule selects LoRA\+ for all four models on QNLI, this ordering carries over directly to the selected configurations there \(81\.5 minutes for TinyLlama against 101\.4 for Mamba2\-1\.3B and 147\.9 for Mamba\-1\.4B\), and it also survives on the tasks where the selected methods differ: on LaMP\-2, TinyLlama with QLoRA still trains in 25\.5 minutes against 31\.7 for Mamba2\-1\.3B with LoRA\+\. Within the SSM family, Mamba2\-1\.3B trains substantially faster than Mamba\-1\.4B on the GLUE tasks \(31\.7 vs\. 49\.4 minutes on SST\-2, and 101 vs\. 148 minutes on QNLI\), which reflects the matrix\-multiplication\-based scan introduced in Mamba\-2, replacing the sequential selective scan used in the original Mamba\. However, this advantage is not preserved on the personalization tasks: on LaMP\-1, Mamba2 \(78\.0 minutes\) actually trains longer than Mamba\-1\.4B \(50\.8 minutes\), showing that the gap between the two SSM generations also depends on the specific task and sequence length used\. When the training times are compared between the Transformer and SSM families, the gap is most visible on long\-running tasks \(QNLI, LaMP\-3\), and is very small on STS\-B, where all four selected configurations finish training in under six minutes\. Because the total energy consumption is essentially a product of training time and average power draw, these longer training durations of SSM\-based models translate directly into reduced NetScore values, even when their VRAM footprint is competitive with the Transformer\-based models\.

#### V\-B5Power Perspective

The differences in average power draw between models during training are smaller than the differences in training time, however, the power draw is still a relevant factor in total energy consumption\. On SST\-2 and QNLI, TinyLlama\-1\.1B records the lowest average power draw: under a matched LoRA\+ configuration it draws 269 W on SST\-2 against 350 W for Qwen3\-1\.7B, 384 W for Mamba\-1\.4B and 358 W for Mamba2\-1\.3B; on QNLI it draws 283 W against 391 W, 421 W, and 407 W respectively\. Mamba\-1\.4B has the highest average power draw on most of the GLUE tasks \(384\-421 W\), which combined with its longer training time is the main reason for it having the highest energy consumption out of the four models\. The differences in power draw become noticeably smaller on LaMP\-2 and LaMP\-3, where nearly all configurations fall within the 365\-435 W range\. This is likely a result of the smaller per\-device batch size \(4 vs\. 32\) but longer sequence length \(512 vs\. 128\) used for the LaMP experiments, which increases compute intensity per step and pushes the GPU to similar utilization levels regardless of architecture\. LaMP\-1 is the exception, spanning 187\-417 W, mainly because Mamba2\-1\.3B runs at an unusually low draw on this task \(199 W with LoRA\+ and 187 W with BitFit\) despite its long training time\.

#### V\-B6Inference Analysis

Across all model\-task configurations, the amount of memory required for inference was significantly less than what was used for training\. Inference memory ranged from 2,120 to 10,293 MB, a 23 to 84% reduction compared to training memory, with the smallest reductions observed for QLoRA configurations whose training footprint is already quantized\. For example, TinyLlama LoRA on SST\-2 required only 3,654 MB for inference, while 16,480 MB was required for training\. There is a substantial difference in inference cost when comparing the GLUE classification task and the LaMP generative task\. On the GLUE classification tasks, which produce one label or score for each input, all of the models took anywhere from seconds to just a couple minutes to finish an inference\. Specifically, SST\-2 took 15\-34 seconds, STS\-B took 21\-58 seconds and QNLI took 112\-213 seconds\. The LaMP generative tasks, however, are much more costly as they perform autoregressive generation with beam search \(up to four beams and up to 128 tokens\), while also having higher input size due to having in context the user profiles\. On LaMP\-1, inference took from about 12 minutes for TinyLlama up to 41 minutes for Mamba\-1\.4B\. Similarly, inference on LaMP\-3 took from 15 to 41 minutes\. One of the most interesting observations is that even though SSM\-based models have been theoretically shown to have linear time advantage, this does not hold true for autoregressive generation\. For example, on LaMP\-2, Mamba\-1\.4B performs an inference in 484 seconds, while TinyLlama performed the same inference in 75 seconds\. The choice of fine\-tuning method did not significantly affect the latency of inference for the Full\-FT, LoRA and LoRA\+ configurations, because the LoRA adapters are merged into the base weights prior to deployment and therefore introduce no additional overhead\. However, QLoRA was found to always incur additional latency costs resulting from dequantizing adapters during inference, while having significant decrease in VRAM used during inference\. As an example, on SST\-2, TinyLlama QLoRA has a 76% increase in inference latency compared to LoRA\+ and a 14% decrease in VRAM used\.

### V\-CStep 3: Best PEFT–Model Combination across Benchmarks

In this section, we find the best combination of model and method for each of the six benchmarks, connecting the per\-model selections summarized in Table[VI](https://arxiv.org/html/2608.04488#S5.T6)\(GLUE\) and Table[VII](https://arxiv.org/html/2608.04488#S5.T7)\(LaMP\) with the detailed per\-configuration results in Tables[IV](https://arxiv.org/html/2608.04488#S4.T4)and[V](https://arxiv.org/html/2608.04488#S4.T5)\. The method inside the best combination depends on which efficiency dimension is prioritized\. Under the base NetScore, which penalizes trainable parameters and FLOPs, BitFit forms the winning combination on four of the six benchmarks \(SST\-2, QNLI, LaMP\-1, and LaMP\-3\) owing to its minimal 0\.09–0\.39 M trainable parameters\. The two exceptions are STS\-B, where BitFit’s near\-zero correlation \(e\.g\., 0\.310 for TinyLlama\) makes TinyLlama with LoRA\+ the best pairing \(NS 59\.4, Table[IV](https://arxiv.org/html/2608.04488#S4.T4)\), and LaMP\-2, where TinyLlama with QLoRA combines the best PEFT accuracy \(0\.703\) with reduced VRAM \(NS 54\.9, Table[V](https://arxiv.org/html/2608.04488#S4.T5)\)\. Under the energy\-focused NS\-E, LoRA\-based methods always form the best combination \(e\.g\., TinyLlama with LoRA reaches 70\.0 on SST\-2 and TinyLlama with LoRA\+ reaches 70\.6 on STS\-B\), whereas under the memory\-focused NS\-M and the combined NS\#, QLoRA pairs best on SST\-2, QNLI, and LaMP\-2 \(NS\-M values of 70\.1, 69\.3, and 64\.0, all with TinyLlama\) owing to its 4\-bit quantized base weights\. Full\-FT never forms the best combination under the base or energy\-focused variants; its only competitive cases are on STS\-B under NS\-M \(68\.1\) and NS\# \(60\.0, vs\. 59\.9 for LoRA\+\), where training takes under five minutes and the efficiency penalty nearly vanishes\. Even there the energy\-first rule prefers LoRA\+ for TinyLlama, so the single configuration in which Full\-FT is selected is Qwen3 on LaMP\-2, where its high accuracy gives it the highest NS\-E of the group\. As such, we conclude that updating all parameters is generally not a cost\-efficient choice at this model scale\. From a practical standpoint, an organization deploying SLMs on edge devices can therefore select the fine\-tuning method according to its dominant constraint: BitFit when the trainable\-parameter budget matters most \(and the task tolerates its accuracy drop\), LoRA or LoRA\+ when energy is the constraint, and QLoRA when VRAM is the constraint\. TinyLlama\-1\.1B dominates the final selection, providing the best combination on all six benchmarks under the base NetScore: TinyLlama with BitFit on SST\-2 \(77\.0\), QNLI \(75\.4\), LaMP\-1 \(65\.3\), and LaMP\-3 \(65\.3\); TinyLlama with LoRA\+ on STS\-B \(59\.4\); and TinyLlama with QLoRA on LaMP\-2 \(54\.9\)\. TinyLlama loses in two cases, but even then, it depends on the chosen priority\. On LaMP\-1, when energy is prioritized \(NS\-E\), Qwen3 with LoRA wins with 65\.0 versus 64\.7 for TinyLlama with LoRA\+ \(a 0\.5% relative difference\), since Qwen3’s higher accuracy \(0\.760 vs\. 0\.732, a 3\.7% relative difference\) outweighs its 41% higher training energy \(210\.0 vs\. 148\.6 Wh\); under the base NS, however, TinyLlama with LoRA\+ still leads with 55\.6 versus 52\.6, a 5\.4% relative improvement\. On LaMP\-3, when memory is prioritized \(NS\-M\), Mamba2 with LoRA \(62\.0\) narrowly edges out TinyLlama with LoRA \(61\.9\), a 0\.2% relative difference\. The six benchmarks can be grouped into three types based on how decisive the model\-method selection is:

1. 1\.High\-Separation Benchmarks\(STS\-B, LaMP\-2\): There is a very large difference in the NetScore values between the best and worst model\-method pairs, driven mainly by BitFit collapses on these tasks: on STS\-B, NS\-E spans from 70\.6 \(TinyLlama with LoRA\+\) down to 6\.7 \(Mamba2 with BitFit\), with NS\# even turning negative \(−3\.6\-3\.6\); on LaMP\-2, Qwen3 with BitFit falls to an NS\-E of 15\.1\. High\-separation benchmarks have high sensitivity to both model quality and efficiency, so choosing between configurations is very important\.
2. 2\.Medium\-Separation Benchmarks\(SST\-2, QNLI, LaMP\-1\): There is some difference between the top and bottom model\-method pairs in terms of their NetScore values \(for example, on SST\-2 the base NS of the best configurations ranges from 77\.0 for TinyLlama down to 61\.3 for Mamba2, a 20% spread computed as100×\(max−min\)/max100\\times\(\\text\{max\}\-\\text\{min\}\)/\\text\{max\}\)\. In these cases, TinyLlama is a clear winner, however a user who has strict memory limitations may choose an SSM\-based model at a small penalty in terms of NetScore\.
3. 3\.Low\-Separation Benchmarks\(LaMP\-3\): The NetScore values of all reasonable configurations are close together \(the finetuning NS\-E of the best LoRA\-family configurations ranges only from 59\.5 to 60\.2, a spread of about 1%\) since all configurations require significant amounts of energy \(661–1422 Wh\)\. For low\-separation benchmarks, the choice of model and method has relatively little impact, thus users can prioritize other constraints such as VRAM and latency, without a significant loss in NetScore\.

## VIConclusion

This paper investigated whether SLMs can be fine\-tuned with Parameter\-Efficient Fine\-Tuning as a practical, energy\-aware alternative to full fine\-tuning of large language models\. By assessing four models from two families \(Transformer\-based: TinyLlama\-1\.1B and Qwen3\-1\.7B; SSM\-based: Mamba\-1\.4B and Mamba2\-1\.3B\) with five fine\-tuning methods \(Full Fine\-Tuning, LoRA, LoRA\+, QLoRA, and BitFit\) across six benchmarks \(three GLUE and three LaMP\), we reach two main conclusions\. First, PEFT methods match or surpass full fine\-tuning at substantially lower resource cost: BitFit achieves the highest base NetScore in 18 of 24 model–task configurations due to its minimal trainable parameters, while LoRA\+ leads the energy\-focused NetScore\-E in 19 of 24 configurations by balancing accuracy with low training energy and is therefore the method selected by our energy\-first rule in 18 of 24 configurations; full fine\-tuning never ranks first on the base NetScore and is selected only once\. Second, TinyLlama\-1\.1B is the most efficient model overall, providing the best model–method combination under the base NetScore on all six benchmarks, largely because of its compact size and the maturity of the Transformer training stack\. The optimal PEFT method depends on the priority: BitFit for parameter efficiency, LoRA\+ for energy, and QLoRA for memory\. Although the Mamba models gain substantially from LoRA\+, they remain limited by longer training times due to less mature SSM libraries and toolkits\. These findings indicate that a practical path to sustainable on\-device NLP lies in compact SLMs combined with PEFT, rather than ever\-larger models\. Limitations include evaluation on a single GPU \(RTX 4090\), the unavailability of QLoRA for Mamba models, and the fact that the base NetScore does not directly penalize VRAM\. Future work should address these constraints and extend the study to edge hardware, hybrid Transformer–SSM architectures, and additional personalization benchmarks\.

## References

- \[1\]\(2024\)Parameter efficient fine tuning: a comprehensive analysis across applications\.External Links:2404\.13506,[Link](https://arxiv.org/abs/2404.13506)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p4.1.1)\.
- \[2\]P\. Belcak, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. C\. Lin, and P\. Molchanov\(2025\)Small language models are the future of agentic ai\.External Links:2506\.02153,[Link](https://arxiv.org/abs/2506.02153)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p3.1.1)\.
- \[3\]E\. Ben\-Zaken, S\. Ravfogel, and Y\. Goldberg\(2022\)BitFit: simple parameter\-efficient fine\-tuning for transformer\-based masked language\-models\.External Links:2106\.10199,[Link](https://arxiv.org/abs/2106.10199)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p1.4),[§IV\-D](https://arxiv.org/html/2608.04488#S4.SS4.p1.8)\.
- \[4\]H\. Chiang, C\. Chang, N\. Frumkin, K\. Wu, and D\. Marculescu\(2024\)Quamba: a post\-training quantization recipe for selective state space models\.External Links:2410\.13229,[Link](https://arxiv.org/abs/2410.13229)Cited by:[§V\-A1](https://arxiv.org/html/2608.04488#S5.SS1.SSS1.p6.1),[§V\-B1](https://arxiv.org/html/2608.04488#S5.SS2.SSS1.Px2.p2.4)\.
- \[5\]C\. Clarke, Y\. Heng, L\. Tang, and J\. Mars\(2024\)PEFT\-u: parameter\-efficient fine\-tuning for user personalization\.External Links:2407\.18078,[Link](https://arxiv.org/abs/2407.18078)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p4.1.1),[§I](https://arxiv.org/html/2608.04488#S1.p5.1.1)\.
- \[6\]T\. Dao and A\. Gu\(2024\)Transformers are ssms: generalized models and efficient algorithms through structured state space duality\.External Links:2405\.21060,[Link](https://arxiv.org/abs/2405.21060)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p4.1),[§IV\-B](https://arxiv.org/html/2608.04488#S4.SS2.p1.1),[§V\-B1](https://arxiv.org/html/2608.04488#S5.SS2.SSS1.Px2.p2.4)\.
- \[7\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized llms\.External Links:2305\.14314,[Link](https://arxiv.org/abs/2305.14314)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p1.4),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p4.1.1.1),[§IV\-D](https://arxiv.org/html/2608.04488#S4.SS4.p1.8)\.
- \[8\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.External Links:1810\.04805,[Link](https://arxiv.org/abs/1810.04805)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p1.1.1),[§IV\-E](https://arxiv.org/html/2608.04488#S4.SS5.p1.1)\.
- \[9\]K\. Galim, W\. Kang, Y\. Zeng, H\. I\. Koo, and K\. Lee\(2025\)Parameter\-efficient fine\-tuning of state space models\.External Links:2410\.09016,[Link](https://arxiv.org/abs/2410.09016)Cited by:[§IV\-D](https://arxiv.org/html/2608.04488#S4.SS4.p1.8),[§V\-A2](https://arxiv.org/html/2608.04488#S5.SS1.SSS2.p4.1)\.
- \[10\]A\. Gu and T\. Dao\(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.External Links:2312\.00752,[Link](https://arxiv.org/abs/2312.00752)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p4.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p3.1.1.1),[§IV\-B](https://arxiv.org/html/2608.04488#S4.SS2.p1.1),[§V\-B1](https://arxiv.org/html/2608.04488#S5.SS2.SSS1.Px2.p2.4)\.
- \[11\]J\. T\. Halloran, M\. Gulati, and P\. F\. Roysdon\(2025\)Mamba state\-space models are lyapunov\-stable learners\.External Links:2406\.00209,[Link](https://arxiv.org/abs/2406.00209)Cited by:[§V\-A2](https://arxiv.org/html/2608.04488#S5.SS1.SSS2.p3.1),[§V\-A2](https://arxiv.org/html/2608.04488#S5.SS1.SSS2.p4.1)\.
- \[12\]S\. Hayou, N\. Ghosh, and B\. Yu\(2024\)LoRA\+: efficient low rank adaptation of large models\.External Links:2402\.12354,[Link](https://arxiv.org/abs/2402.12354)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p1.4),[§IV\-D](https://arxiv.org/html/2608.04488#S4.SS4.p1.8),[§V\-A1](https://arxiv.org/html/2608.04488#S5.SS1.SSS1.p2.9)\.
- \[13\]J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre\(2022\)Training compute\-optimal large language models\.External Links:2203\.15556,[Link](https://arxiv.org/abs/2203.15556)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p1.1.1)\.
- \[14\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2021\)LoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p1.4),[§IV\-D](https://arxiv.org/html/2608.04488#S4.SS4.p1.8)\.
- \[15\]C\. Irugalbandara, A\. Mahendra, R\. Daynauth, T\. K\. Arachchige, J\. Dantanarayana, K\. Flautner, L\. Tang, Y\. Kang, and J\. Mars\(2024\)Scaling down to scale up: a cost\-benefit analysis of replacing openai’s llm with open source slms in production\.External Links:2312\.14972,[Link](https://arxiv.org/abs/2312.14972)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p2.1.1)\.
- \[16\]G\. Izacard, M\. Caron, L\. Hosseini, S\. Riedel, P\. Bojanowski, A\. Joulin, and E\. Grave\(2022\)Unsupervised dense information retrieval with contrastive learning\.External Links:2112\.09118,[Link](https://arxiv.org/abs/2112.09118)Cited by:[§IV\-C2](https://arxiv.org/html/2608.04488#S4.SS3.SSS2.p1.1)\.
- \[17\]Z\. Liu, C\. Zhao, F\. Iandola, C\. Lai, Y\. Tian, I\. Fedorov, Y\. Xiong, E\. Chang, Y\. Shi, R\. Krishnamoorthi, L\. Lai, and V\. Chandra\(2024\)MobileLLM: optimizing sub\-billion parameter language models for on\-device use cases\.External Links:2402\.14905,[Link](https://arxiv.org/abs/2402.14905)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p2.1)\.
- \[18\]Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, W\. Liu, J\. Luan, X\. Zhang, N\. Lane, and M\. Xu\(2025\-01\)Demystifying small language models for edge deployment\.pp\. 14747–14764\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.718)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p2.1)\.
- \[19\]Z\. Lu, X\. Li, D\. Cai, R\. Yi, F\. Liu, X\. Zhang, N\. D\. Lane, and M\. Xu\(2025\)Small language models: survey, measurements, and insights\.External Links:2409\.15790,[Link](https://arxiv.org/abs/2409.15790)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p3.1.1)\.
- \[20\]A\. Mirzal, S\. Yoshii, and M\. Furukawa\(2012\)PID parameters optimization by using genetic algorithm\.External Links:1204\.0885,[Link](https://arxiv.org/abs/1204.0885)Cited by:[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p5.2.2.2)\.
- \[21\]C\. V\. Nguyen, X\. Shen, R\. Aponte, Y\. Xia, S\. Basu, Z\. Hu, J\. Chen, M\. Parmar, S\. Kunapuli, J\. Barrow, J\. Wu, A\. Singh, Y\. Wang, J\. Gu, N\. K\. Ahmed, N\. Lipka, R\. Zhang, X\. Chen, T\. Yu, S\. Kim, H\. Deilamsalehy, N\. Park, M\. Rimer, Z\. Zhang, H\. Yang, P\. Mathur, G\. Wu, F\. Dernoncourt, R\. A\. Rossi, and T\. H\. Nguyen\(2025\-09\)A survey on small language models\.InProceedings of the 15th International Conference on Recent Advances in Natural Language Processing \- Natural Language Processing in the Generative AI Era,G\. Angelova, M\. Kunilovskaya, M\. Escribe, and R\. Mitkov \(Eds\.\),Varna, Bulgaria,pp\. 807–821\.External Links:[Link](https://aclanthology.org/2025.ranlp-1.93/)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p3.1.1)\.
- \[22\]G\. Pan, V\. Chodnekar, A\. Roy, and H\. Wang\(2025\)A cost\-benefit analysis of on\-premise large language model deployment: breaking even with commercial llm services\.External Links:2509\.18101,[Link](https://arxiv.org/abs/2509.18101)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p2.1.1)\.
- \[23\]G\. Pu, A\. Jain, J\. Yin, and R\. Kaplan\(2023\)Empirical analysis of the strengths and weaknesses of peft techniques for llms\.External Links:2304\.14999,[Link](https://arxiv.org/abs/2304.14999)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p4.1.1),[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p1.4)\.
- \[24\]P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang\(2016\)SQuAD: 100,000\+ questions for machine comprehension of text\.External Links:1606\.05250,[Link](https://arxiv.org/abs/1606.05250)Cited by:[§IV\-C1](https://arxiv.org/html/2608.04488#S4.SS3.SSS1.p1.1)\.
- \[25\]A\. Salemi, S\. Mysore, M\. Bendersky, and H\. Zamani\(2024\)LaMP: when large language models meet personalization\.External Links:2304\.11406,[Link](https://arxiv.org/abs/2304.11406)Cited by:[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p2.1.1.1),[§IV\-C2](https://arxiv.org/html/2608.04488#S4.SS3.SSS2.p1.1),[§IV\-E](https://arxiv.org/html/2608.04488#S4.SS5.p2.1)\.
- \[26\]T\. Schick and H\. Schütze\(2021\-06\)It’s not just size that matters: small language models are also few\-shot learners\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 2339–2352\.External Links:[Link](https://aclanthology.org/2021.naacl-main.185/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.185)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p3.1.1)\.
- \[27\]S\. Subramanian, V\. Elango, and M\. Gungor\(2025\)Small language models \(slms\) can still pack a punch: a survey\.External Links:2501\.05465,[Link](https://arxiv.org/abs/2501.05465)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p3.1.1)\.
- \[28\]A\. Toktassyn and J\. Park\(2026\)Efficient peft methods with adaptive checkpointing for vision models and vlms on resource constrained consumer\-gpus\.arXiv preprint arXiv:2607\.02158\.Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p6.1.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p7.1.1),[TABLE I](https://arxiv.org/html/2608.04488#S3.T1.11.8.1),[TABLE I](https://arxiv.org/html/2608.04488#S3.T1.11.9.1)\.
- \[29\]T\. Trinci, S\. Magistri, R\. Verdecchia, and A\. D\. Bagdanov\(2024\)How green is continual learning, really? analyzing the energy consumption in continual training of vision foundation models\.External Links:2409\.18664,[Link](https://arxiv.org/abs/2409.18664)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p6.1.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p7.1.1.2),[TABLE I](https://arxiv.org/html/2608.04488#S3.T1.11.7.1)\.
- \[30\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\(2023\)Attention is all you need\.External Links:1706\.03762,[Link](https://arxiv.org/abs/1706.03762)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p1.1.1)\.
- \[31\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2019\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.External Links:1804\.07461,[Link](https://arxiv.org/abs/1804.07461)Cited by:[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p2.1.1.1),[§IV\-C1](https://arxiv.org/html/2608.04488#S4.SS3.SSS1.p1.1)\.
- \[32\]F\. Wang, Z\. Zhang, X\. Zhang, Z\. Wu, T\. Mo, Q\. Lu, W\. Wang, R\. Li, J\. Xu, X\. Tang, Q\. He, Y\. Ma, M\. Huang, and S\. Wang\(2024\)A comprehensive survey of small language models in the era of large language models: techniques, enhancements, applications, collaboration with llms, and trustworthiness\.External Links:2411\.03350,[Link](https://arxiv.org/abs/2411.03350)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p2.1)\.
- \[33\]Y\. Wang, F\. Gunnarsson, and R\. Hai\(2026\)An attention\-based feature memory design for energy\-efficient continual learning\.External Links:2510\.04660,[Link](https://arxiv.org/abs/2510.04660)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p6.1.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p7.1.1.2),[TABLE I](https://arxiv.org/html/2608.04488#S3.T1.11.7.1)\.
- \[34\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. Rush\(2020\)HuggingFace’s transformers: state\-of\-the\-art natural language processing\.External Links:1910\.03771,[Link](https://arxiv.org/abs/1910.03771)Cited by:[§IV\-E](https://arxiv.org/html/2608.04488#S4.SS5.p1.1)\.
- \[35\]A\. Wong, Z\. Q\. Lin, and B\. Chwyl\(2019\)AttoNets: compact and efficient deep neural networks for the edge via human\-machine collaborative design\.External Links:1903\.07209,[Link](https://arxiv.org/abs/1903.07209)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p6.1.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p6.14.14),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p6.15.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p7.1.1),[TABLE I](https://arxiv.org/html/2608.04488#S3.T1.11.6.1)\.
- \[36\]A\. Wong\(2018\)NetScore: towards universal metrics for large\-scale performance analysis of deep neural networks for practical on\-device edge usage\.External Links:1806\.05512,[Link](https://arxiv.org/abs/1806.05512)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p6.1.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p6.14.14),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p6.15.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p7.1.1),[TABLE I](https://arxiv.org/html/2608.04488#S3.T1.11.6.1)\.
- \[37\]Y\. Xia, J\. Kim, Y\. Chen, H\. Ye, S\. Kundu, C\. Hao, and N\. Talati\(2024\)Understanding the performance and estimating the cost of llm fine\-tuning\.External Links:2408\.04693,[Link](https://arxiv.org/abs/2408.04693)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p2.1.1)\.
- \[38\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p2.1),[§III\-A](https://arxiv.org/html/2608.04488#S3.SS1.p3.1.1.1),[§IV\-B](https://arxiv.org/html/2608.04488#S4.SS2.p1.1)\.
- \[39\]G\. Yenduri, R\. M, C\. S\. G, S\. Y, G\. Srivastava, P\. K\. R\. Maddikunta, D\. R\. G, R\. H\. Jhaveri, P\. B, W\. Wang, A\. V\. Vasilakos, and T\. R\. Gadekallu\(2023\)Generative pre\-trained transformer: a comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions\.External Links:2305\.10435,[Link](https://arxiv.org/abs/2305.10435)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p1.1.1)\.
- \[40\]M\. Yoshimura, T\. Hayashi, and Y\. Maeda\(2025\)MambaPEFT: exploring parameter\-efficient fine\-tuning for mamba\.External Links:2411\.03855,[Link](https://arxiv.org/abs/2411.03855)Cited by:[§V\-A2](https://arxiv.org/html/2608.04488#S5.SS1.SSS2.p3.1),[§V\-A2](https://arxiv.org/html/2608.04488#S5.SS1.SSS2.p4.1)\.
- \[41\]D\. Zhang, T\. Feng, L\. Xue, Y\. Wang, Y\. Dong, and J\. Tang\(2025\)Parameter\-efficient fine\-tuning for foundation models\.External Links:2501\.13787,[Link](https://arxiv.org/abs/2501.13787)Cited by:[§I](https://arxiv.org/html/2608.04488#S1.p4.1.1)\.
- \[42\]P\. Zhang, G\. Zeng, T\. Wang, and W\. Lu\(2024\)TinyLlama: an open\-source small language model\.External Links:2401\.02385,[Link](https://arxiv.org/abs/2401.02385)Cited by:[§II\-C](https://arxiv.org/html/2608.04488#S2.SS3.p2.1),[§IV\-B](https://arxiv.org/html/2608.04488#S4.SS2.p1.1)\.