@0xSero: Highly recommended educational content. LoRA is one of the coolest things to dabble in, lets anyone fine tune models re…

Highly recommended educational content. LoRA is one of the coolest things to dabble in, lets anyone fine tune models relatively cheap.

https://t.co/SLU3GF9R3z

TL;DR: LoRA drastically reduces trainable parameters by decomposing weight updates into two low-rank matrices; QLoRA uses 4-bit quantization of the frozen model, enabling fine-tuning of a 70B-parameter model on a single 48GB GPU; VeRA shares random directions and only trains scaling vectors, with even fewer parameters; DoRA decouples direction and magnitude, behaving more like full fine-tuning.

Background: The Cost of Full Fine-Tuning

Modern AI models are enormous—from large language models to image generation and protein structure prediction, often hundreds of billions of parameters. While pre-trained models are general-purpose, they frequently need adaptation to specific domains like law, finance, or medicine. Full fine-tuning—updating every weight—is extremely expensive. For a 70B-parameter model, the weights alone require about 140 GB at 16-bit precision. Saving a full copy for each task quickly becomes unmanageable storage-wise.

Parameter-Efficient Fine-Tuning (PEFT) emerged to address this. The core idea is to freeze the large model and train only a small set of additional parameters. LoRA is one of the most popular methods in this family.

LoRA: Low-Rank Adaptation

Why Focus on Linear Layers?

In Transformers, the bulk of computation lies in linear layers: the Query, Key, Value projections in attention, and the expansion/contraction in feed-forward networks. All these layers perform Y = W * X, where X is the input, W is the weight matrix, and Y is the output.

Instead of fine-tuning all individual weights directly, freeze the original weight matrix W0 and add a trainable new matrix ΔW as a task-specific correction. After fine-tuning, ΔW can be merged with W0 (W0 + ΔW), so inference runs at the same speed as the original model. But this still doesn’t solve the core problem—ΔW is still huge.

The Low-Rank Hypothesis

The key insight: a task may only need the model to shift in a few important directions, meaning ΔW has a low-rank structure.
For example, a 6×6 matrix (36 numbers) might have redundant column vectors: only the first two columns C1, C2 are independent, and the rest are linear combinations of them. This matrix can be decomposed into two smaller matrices of size 6×2 and 2×6, whose product equals the original. Here, “2” is the rank.

Applied to weight updates: instead of learning a large ΔW directly, learn two smaller matrices B (tall and skinny) and A (short and flat), such that ΔW = B × A. This is called Low-Rank Adaptation (LoRA). The number of trainable parameters drops dramatically. In practice, the update strength is scaled by α / rank so that changing the rank keeps the update size roughly stable, avoiding the need to re-tune the learning rate.

Intuitive Explanation: Local Associative Memory

Decompose B and A into row and column vectors; then ΔW can be written as a sum of outer products. The row vectors A1, A2 act as “keys,” and the column vectors B1, B2 act as “values.” The dot product of input X with a key measures alignment; if activated, the corresponding value is added. For example, when fine-tuning a general model on medical literature, A1 might detect heart disease, B1 adds cardiology knowledge; A2 detects diabetes, B2 adds diabetes knowledge. If the input is unrelated to the domain, the match is low and the correction near zero. Thus LoRA behaves like a local associative memory—the base model retains general knowledge, and LoRA adds compact task-specific memories. Different tasks can share the same frozen backbone, swapping only adapters, or even combine multiple adapters with adjustable contributions.

Initialization Strategy

At the start of training, the adapter should not change the model’s output, otherwise it would interfere with pre-trained knowledge. If both B and A are randomly initialized, ΔY is usually non-zero. Ideally, initial ΔY = 0.
If both B and A are set to 0, gradients vanish. So: one matrix random, the other zero. LoRA defaults to random A, zero B.

• Random A provides diverse initial feature directions, zero B ensures zero initial correction.
• In the first step, B has no gradient, A starts learning; after that both update normally.
  Another scheme (random B, zero A) also works, but random A provides richer features earlier.

LoRA+: Different Learning Rates

Matrix A maps the wide input to a few dimensions (projecting down), while matrix B maps those few dimensions back to the full output space (projecting up). Their scales differ greatly; using the same learning rate leaves B undertrained. LoRA+ assigns a smaller learning rate to A and a larger one to B, allowing both matrices to learn at matching speeds, leading to faster convergence and better performance, with no extra parameters.

Memory Challenges and QLoRA

Even though LoRA has few trainable parameters, the base model itself consumes a lot of memory. For example, a 70B-parameter model at 16-bit precision weighs 140 GB, while a rank-64 adapter (adapting all linear layers) has only about 587M parameters (less than 1%), requiring just 1.17 GB at 16-bit. But the entire frozen model won’t fit on a single GPU.

Quantization is the solution: store the frozen pre-trained weights at low precision, dequantize them to higher precision during computation for matrix multiplication, and keep only the adapters at full precision for training. This fits the entire fine-tuning process onto a single 48 GB GPU. This is Quantized LoRA (QLoRA).

4-bit Quantization: NF4

With 4 bits, there are only 16 encodings. Neural network weights are usually symmetric around zero, and zero must be represented exactly. Using a symmetric signed integer (-8 to 7) lets you drop -8 for perfect symmetry (-7 to 7), where zero is exactly encoded. But that wastes one slot.

A better approach uses quantile quantization: place levels to fit a zero-mean normal distribution so that each bucket contains an equal proportion of weights. Buckets near zero, where weights are dense, are narrow; tail buckets are wide. All 16 non-uniform levels are used. Since 16 is even, no level falls exactly at zero, so it’s built asymmetrically: 8 levels between -1 and 0, 9 levels between 0 and 1—both halves include zero; remove the duplicate zero to get exactly 16 levels. This is called NormalFloat 4 (NF4). NF4 spends the 4-bit budget where it matters most, reducing quantization error.

Block-wise Quantization and Double Quantization

If a single scaling factor C is used for the entire matrix, one outlier can dominate C, compressing other weights into a tiny range. So weights are divided into blocks, each scaled independently (block-wise quantization). But each block needs a scaling factor (32 bits); with a block size of 64, that’s an extra 0.5 bits per parameter. QLoRA further quantizes the scaling factors to 8 bits—called double quantization.

Combining block-wise NF4 and double quantization, QLoRA makes it feasible to fine-tune a 70B-parameter model on a single 48 GB GPU.

VeRA: Even Greater Parameter Efficiency

LoRA still learns a pair of B and A for each adapted layer. VeRA goes further: it uses a pair of randomly initialized matrices, frozen and shared across all adapted layers. The model no longer learns the entries in B and A, but instead learns trainable scaling vectors that adjust per layer. Formula: ΔW = B · D · diag(d) · A, where D and d are diagonal scaling matrices. B starts from zero, D starts from a small tunable constant.
For example, with input dimension 4096, rank 64, and 80 adapted layers: standard LoRA has ~42M trainable parameters; VeRA trains only the scaling vectors, about 333k, 126 times fewer. VeRA doesn’t learn new directions—it learns how to use a set of shared, fixed random directions.

DoRA: Separating Direction and Magnitude

LoRA changes both the direction and length of weight vectors simultaneously, but the two are coupled. DoRA (Weight-Decomposed Low-Rank Adaptation) decomposes each row into direction and magnitude: the weight matrix is written as the product of a diagonal matrix (magnitudes) and a unit-direction matrix. The LoRA-style update focuses on direction, while magnitude parameters are learned independently. This decoupling makes the adapter behave more like full fine-tuning, with very few extra parameters.

Source: YouTube video (https://youtu.be/U80tjcThl9Q?is=jIB7vxGs5GooEtJ7)