@techNmak: Everyone is fine-tuning LLMs. Almost nobody understands what is actually being updated inside the model. Here are 5 tec…

Everyone is fine-tuning LLMs. Almost nobody understands what is actually being updated inside the model. Here are 5 techniques that change how you think about model adaptation, and what each one is actually doing to the weights: 1./ LoRA - Learn the update, not the weights The pretrained weight W is frozen. Completely untouched. Instead of updating W directly, two small matrices are trained => A ∈ ℝʳˣᵈ and B ∈ ℝᵈˣʳ, where r ≪ d The weight update is: ΔW = BA Effective weight: W' = W + BA The entire adaptation happens in a tiny low-rank space. W never changes. 2./ LoRA-FA - What if we freeze even more? Same structure as LoRA. One change. A is frozen alongside W. Only B is trained. Effective weight: W' = W + BA (A is fixed) Half the trainable matrices of LoRA. Same core idea. Fewer parameters. 3./ VeRA - What if the matrices don't need to be learned at all? This is where it gets interesting. A and B are both frozen, and randomly initialized. What gets trained are just two tiny scaling vectors => b ∈ ℝʳ and d ∈ ℝʳ Instead of learning the low-rank matrices themselves, VeRA keeps them frozen and learns small scaling vectors that modulate their contribution. Initialization => b = 0, d = 1 You're not learning matrices. You're learning how to scale them. One of the most parameter-efficient techniques on this list. 4./ Delta-LoRA - What if W itself learns from the low-rank updates? This one is fundamentally different. Unlike standard LoRA, the base weight W is not fully frozen. It is updated through low-rank delta propagation at every step => W^(t+1) = W^t + c(B_(t+1)A_(t+1) − B_t A_t) Where c is a scaling factor. A and B are trainable. W evolves, but guided entirely by low-rank changes. 5./ LoRA+ - Same structure. Smarter learning rates. Identical to LoRA, freeze W, train A and B. One change => B is assigned a larger learning rate than A. η_B > η_A A ← A − η_A · ∂J/∂A B ← B − η_B · ∂J/∂B A small optimization change that can make LoRA training more effective. The core idea running through all five: You do not always need full fine-tuning to adapt a model. LoRA updates two matrices. LoRA-FA updates one. LoRA+ updates two at different speeds. Delta-LoRA lets W evolve - guided by low-rank deltas. VeRA updates two vectors. Same goal. Five different answers to the same question: => What is the minimum we actually need to learn? That is the core idea behind parameter-efficient fine-tuning. And now you know what is actually happening inside the model.