Understanding and Enforcing Weight Disentanglement in Task Arithmetic

Hugging Face Daily Papers Papers
model-merging task-arithmetic weight-disentanglement fine-tuning regularization

Summary

The paper introduces OrthoReg, a plug-and-play regularizer that enforces weight orthogonality during fine-tuning to improve task arithmetic and model merging without extra compute.

Task arithmetic provides an efficient, training-free way to edit pre-trained models, yet lacks a fundamental theoretical explanation for its success. The existing concept of ``weight disentanglement" describes the ideal outcome of non-interfering task composition but does not reveal its underlying cause. Crucially, what intrinsic properties of the pre-trained model (θ_0) or the task vectors (τ_t) enable this disentanglement remains underexplored. In this paper, we introduce Task-Feature Specialization (TFS), a model's ability to allocate distinct internal features to different tasks, as the fundamental principle. We first prove that TFS is a sufficient condition for weight disentanglement. More importantly, we find that TFS also gives rise to an observable geometric consequence: weight vector orthogonality. This positions TFS as the common cause for both the desired functional outcome (disentanglement) and a measurable geometric property (orthogonality). This relationship provides the key insight for our method: since the abstract TFS property is intractable to enforce directly, we can instead promote weight disentanglement by shaping its concrete geometric consequence, orthogonality. Therefore, we propose OrthoReg, a simple and effective regularization method that actively enforces an internal orthogonal structure on weight updates (ΔW) that constitute τ_t during fine-tuning. And we theoretically prove that OrthoReg promotes disentanglement. Extensive experiments demonstrate that OrthoReg consistently and significantly enhances the performance of various task arithmetic methods. Code is available at https://github.com/RL-MIND/OrthoReg{https://github.com/RL-MIND/OrthoReg}.
Original Article
View Cached Full Text

Cached at: 04/22/26, 10:35 AM

Paper page - Understanding and Enforcing Weight Disentanglement in Task Arithmetic

Source: https://huggingface.co/papers/2604.17078 Hi everyone! 🤗 We are thrilled to share our latest work accepted atCVPR 2026:“Understanding and Enforcing Weight Disentanglement in Task Arithmetic”.

If you are interested inModel MergingandTask Arithmetic, here is a quick overview of what we did and how you can use it!

💡The Core Problem: Why does Task Arithmetic work? While task arithmetic (adding/subtracting task vectors) is an elegant, training-free way to edit models, its underlying mechanism has remained unclear. We theoretically prove that**Task-Feature Specialization (TFS)**is a sufficient condition for weight disentanglement. More importantly, TFS naturally leads to an observable geometric property:Weight Vector Orthogonality.

🛠️Our Solution: OrthoReg Since enforcing abstract feature specialization is intractable, we proposeOrthoReg—a simple, plug-and-play regularization term added to the standard fine-tuning loss. It actively enforces column-wise orthogonality on the weight updates (\Delta W) during fine-tuning.

The loss is incredibly simple to implement: L=Ltask+λ⋅∑l∥(ΔW(l))⊤ΔW(l)−I∥F2\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \cdot \sum_l \left\|(\Delta W^{(l)})^\top \Delta W^{(l)} - I\right\|_F^2

📊Key Results

  • **Consistent Boosts:**Significantly improves the performance of various task arithmetic baselines (Non-linear FT, TTA, ATT-FT, LoRA) across ViT-B-32, ViT-B-16, and ViT-L-14.
  • **High Efficiency:**Achieves the disentanglement benefits of Tangent Task Arithmetic (TTA) but without the massive computational overhead of Jacobian calculations.
  • **Cleaner Forgetting:**Substantially improves task negation (making a model forget a specific task) with minimal side effects on general capabilities.

🔗Resources We believe in open science and have released all our artifacts:

We would love to hear your thoughts, feedback, or answer any questions you might have. Feel free to drop a comment below! 👇

Similar Articles

Task-Restricted Symmetries in Recurrent Weight Space

arXiv cs.LG

This paper studies functional redundancy in recurrent neural networks by using ordered real Schur coordinates to identify structured ablations that preserve task performance, finding that task-restricted symmetries vary across tasks and trained solutions.

Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

arXiv cs.LG

This paper investigates how weight decay acts as a control parameter for transitioning between memorization and generalization in transformers trained on modular arithmetic, and introduces two cheap online diagnostic metrics from attention activations that track these dynamics.

Weight-Space Geometry of Offline Reasoning Training

arXiv cs.LG

This paper investigates whether different offline reinforcement learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) for reasoning distillation produce mechanistically distinct weight updates in a small language model. Using identical math rollouts and a controlled setup with Qwen3-4B and attention-only LoRA, they find that SFT, RFT, and RIFT yield nearly colinear weight deltas, while DPO sits in a near-orthogonal subspace and achieves the highest accuracy.

LoRA and Weight Decay (2023)

Hacker News Top

This blog post explores how LoRA's interaction with weight decay leads to a different optimization objective than full fine-tuning, where weights are regularized towards the initial model rather than zero. It explains the implications for practitioners.