HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging

# HyperFix: Combinatorial Nonlinear Correction for Task Vector Merging
Source: [https://arxiv.org/html/2608.11499](https://arxiv.org/html/2608.11499)
###### Abstract

Task vectors enable model merging without joint retraining\. In practice, the subset of task vectors to be merged may vary, but many existing methods use scalar tuning for a particular subset, requiring repeated tuning across subsets and restricting task vector merging to linear rescaling\. We therefore formulate merging across varying task subsets as a*combinatorial correction*problem and introduce HyperFix, a lightweight hypernetwork that predicts subset\-conditioned nonlinear corrections in weight space\. Trained once on singleton, pair, and triple subsets from a task bank, HyperFix generalizes to larger subsets without per\-subset optimization\. Our local perturbation analysis bounds the residual correction beyond linear merging and motivates learning it from small task updates\. Experiments across diverse benchmarks show that HyperFix outperforms existing task vector merging methods while reducing tuning cost\.

## Introduction

Task vectors represent the parameter differences between pretrained and task\-specific fine\-tuned weights\. Task arithmetic\([Ilharco et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib7)\)linearly combines these task vectors to integrate task\-specific capabilities without expensive joint retraining\. This approach has demonstrated strong performance across vision\([Ortiz\-Jimenez, Favero, and Frossard 2023](https://arxiv.org/html/2608.11499#bib.bib14)\), language\([Zhang et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib30);[Zeng et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib29)\), and multimodal domains\([Huang et al\. 2024](https://arxiv.org/html/2608.11499#bib.bib6)\)\.

Despite these advances, many existing task vector merging methods are developed and evaluated for a single fixed task subset, with scalar tuning for that subset\. However, different downstream needs may require different subsets of task\-specific capabilities, highlighting the need for merging methods that can accommodate varying target subsets\. In particular, the tuned scalar cannot be reliably reused because it is specific to a particular subset, as illustrated in Fig\.[1](https://arxiv.org/html/2608.11499#Sx1.F1)\(a\)\. As diverse subsets need to be considered, repeated per\-subset tuning is required, resulting in a combinatorial tuning cost\. Moreover, these merging methods are restricted to linear scaling of the merged task vector, limiting their ability to capture nonlinear interactions among tasks\.

![Refer to caption](https://arxiv.org/html/2608.11499v1/Figures/Teaser.png)Figure 1:Comparison of the existing merging approach and HyperFix\. \(a\) Existing merging requires separate scalar tuning for each task subset and can only rescale the linearly merged task vector\. \(b\) HyperFix uses a hypernetwork to predict subset\-conditioned correction, avoiding per\-subset tuning and enabling nonlinear correction\.We therefore view task vector merging as a*combinatorial correction*problem, where the goal is to develop a shared mechanism that constructs a subset\-conditioned merged model for any selected subset of task vectors, rather than optimizing a model for a single fixed subset\. To address this problem, we introduce HyperFix, a framework that predicts subset\-conditioned nonlinear corrections, as illustrated in Fig\.[1](https://arxiv.org/html/2608.11499#Sx1.F1)\(b\)\. Specifically, HyperFix uses a lightweight hypernetwork, trained on small subsets and applied to larger subsets, to generate the corrections\.

Our theoretical analysis explains why corrections learned from small subsets can generalize to larger ones\. Under mild local smoothness conditions, nonlinear interaction effects can be approximated using low\-order subsets\. Consistent with this analysis, our experimental results demonstrate that HyperFix consistently outperforms merging baselines across all subset sizes on eight image classification benchmarks and three CLIP\([Radford et al\. 2021](https://arxiv.org/html/2608.11499#bib.bib17)\)architectures\. By avoiding repeated per\-subset scalar tuning, HyperFix reduces tuning cost by up to 82\.6%\.

Our main contributions are as follows:

- •We view task vector merging across varying subsets as a*combinatorial correction*problem and introduce HyperFix, a hypernetwork\-based framework that learns a shared mapping from task\-subset embeddings to weight corrections that capture nonlinear task interactions\.
- •We theoretically show that nonlinear interactions can be approximated using low\-order subsets, explaining generalization from small to larger subsets\.
- •Across diverse benchmarks, HyperFix outperforms merging baselines across all subset sizes while reducing tuning cost dramatically\.

## Related Work

### Model Merging

Model merging aims to combine multiple models fine\-tuned from a shared foundation model into a single model that preserves their task\-specific capabilities\. A fundamental approach is Task Arithmetic\([Ilharco et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib7)\), which represents task\-specific adaptations as task vectors and merges them through linear addition\. Subsequent methods improve task vector merging through masking and magnitude\-based selection\([Yadav et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib26);[Yu et al\. 2024](https://arxiv.org/html/2608.11499#bib.bib28);[Kim, Han, and Choe 2025](https://arxiv.org/html/2608.11499#bib.bib8)\), geometry\-aware constraints\([Sun et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib21);[Porrello et al\. 2026](https://arxiv.org/html/2608.11499#bib.bib16)\), variational formulations\([Zhang et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib30)\), and low\-rank decomposition of task matrices\([Gargiulo et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib3)\)\. Adaptive coefficient\-based methods such as AdaMerging\([Yang et al\. 2024](https://arxiv.org/html/2608.11499#bib.bib27)\)optimize linear merging coefficients for a given set of target tasks\. In PEFT settings, alignment\-based merging of LoRA adapters has also been explored\([Panariello et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib15)\)\. Despite these advances, most existing methods construct a merged model separately for each target task subset and do not learn a shared merging function across subsets\. In contrast, HyperFix learns a shared merging function that predicts subset\-conditioned nonlinear corrections\.

### Hypernetwork\-based Model Editing

Hypernetworks\([Ha, Dai, and Le 2017](https://arxiv.org/html/2608.11499#bib.bib4);[Krueger et al\. 2017](https://arxiv.org/html/2608.11499#bib.bib10);[Ratzlaff and Fuxin 2019](https://arxiv.org/html/2608.11499#bib.bib18)\)are secondary neural networks that generate or modulate the parameters of a target network\. In model editing, hypernetworks produce parameter updates conditioned on editing requests\. MeG\([Wan et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib23)\)uses a diffusion\-based hypernetwork to generate dynamic weight neurons, enabling large\-scale knowledge updates without directly modifying internal model weights, unlike approaches such as MEMIT\([Meng et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib12)\)and MALMEN\([Tan, Zhang, and Fu 2024](https://arxiv.org/html/2608.11499#bib.bib22)\)\. LoRA\.rar\([Shenaj et al\. 2025](https://arxiv.org/html/2608.11499#bib.bib19)\)uses a hypernetwork to predict merging coefficients for combining subject and style LoRAs in real time, reducing computation compared with optimization\-based methods\. Unlike these hypernetwork\-based approaches, HyperFix neither generates full model weights nor predicts scalar merging coefficients\. Instead, it predicts a structured low\-rank nonlinear correction on top of the linearly merged task vector, conditioned on a permutation\-invariant representation of the selected task subset\.

## Method

![Refer to caption](https://arxiv.org/html/2608.11499v1/Figures/HyperFix.png)Figure 2:Overview of HyperFix\. Given a target task subsetSS, HyperFix constructs the merged task vectorτS\\tau\_\{S\}and uses the subset embeddingzSz\_\{S\}to predict a subset\-conditioned correctionΔS\\Delta\_\{S\}, yieldingθS=θ0\+τS\+ΔS\\theta\_\{S\}=\\theta\_\{0\}\+\\tau\_\{S\}\+\\Delta\_\{S\}\. Left: HyperFix is trained on task subsets with\|S\|≤3\|S\|\\leq 3\(singleton, pair, triple\) using KL\-based distillation from the corresponding single\-task teachers\. Right: HyperFix generalizes to larger task subsets \(\|S\|\>3\|S\|\>3\), where the learned correction is applied without additional optimization\.In this section, we present HyperFix, a hypernetwork\-based correction framework for task vector merging \(Figure[2](https://arxiv.org/html/2608.11499#Sx3.F2)\)\. We view merging across varying task subsets as a*combinatorial correction*problem, where the goal is to learn a shared correction function\. HyperFix augments linear merging with a subset\-conditioned correction generated by a nonlinear hypernetwork\. Learned from small task subsets, the shared correction function is applied without additional optimization to larger subsets from the same task bank\. We first motivate the need for subset\-conditioned corrections and then describe the subset embedding, hypernetwork parameterization, training objective, and inference procedure\.

### Motivation

Linear merging assumes that linearly combining task\-specific parameter updates in weight space is sufficient to retain the capabilities of the selected tasks in a single model\. Letθ0\\theta\_\{0\}denote the pretrained model parameters andθi\\theta\_\{i\}the fine\-tuned parameters for taskii, and define the task vector asτi=θi−θ0\\tau\_\{i\}=\\theta\_\{i\}\-\\theta\_\{0\}\. For a subset of tasksSS, linear merging produces a merged task vectorτS\\tau\_\{S\}

τS=1\|S\|​∑i∈Sτi\.\\tau\_\{S\}=\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}\\tau\_\{i\}\.\(1\)This yields the merged parametersθ0\+τS\\theta\_\{0\}\+\\tau\_\{S\}\. HyperFix operates on top of this merged task vector and can also be applied to other merging rules, such as TIES\([Yadav et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib26)\)\.

However, this additive assumption ignores potential nonlinear interactions between task vectors\. In practice, merging multiple task vectors can introduce non\-additive interference, leading to performance degradation as the subset size increases\. To capture interactions beyond linear merging, we introduce a subset\-conditioned correction term

θS=θ0\+τS\+ΔS,\\theta\_\{S\}=\\theta\_\{0\}\+\\tau\_\{S\}\+\\Delta\_\{S\},\(2\)whereΔS\\Delta\_\{S\}captures structured interaction effects that depend on the specific combination of tasks inSS\. Importantly, these effects are not universal offsets but vary with the alignment and conflict structure among task vectors in the subset\. The construction ofΔS\\Delta\_\{S\}is detailed in the following subsection\.

### HyperFix: Combinatorial Nonlinear Correction

To model interaction effects beyond linear merging, we require a compact representation of a task subset\. Conditioning directly on raw task vectors would entail operating in the full encoder parameter space, which contains tens of millions of dimensions \(e\.g\., CLIP ViT\-B/32\), making learning computationally expensive and difficult to optimize\. Instead, we construct a low\-dimensional task\-level representation that captures relationships between task vectors\.

Subset embedding\.We summarize pairwise relationships between task vectors using a task\-level Gram matrix

Gi​j=⟨τi,τj⟩,G\_\{ij\}=\\langle\\tau\_\{i\},\\tau\_\{j\}\\rangle,\(3\)where the inner product is taken over all encoder parameters\. The Gram matrix is computed once from the task vectors, mean\-centered across tasks, and kept fixed during HyperFix training\.

LetNNbe the number of tasks andG∈ℝN×NG\\in\\mathbb\{R\}^\{N\\times N\}the corresponding Gram matrix\. We define the task embeddingzi∈ℝNz\_\{i\}\\in\\mathbb\{R\}^\{N\}as theii\-th row ofGG

For a subsetSS, we define the subset embedding

zS=1\|S\|​∑i∈Szi\.z\_\{S\}=\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}z\_\{i\}\.\(5\)The Gram matrix provides pairwise relationship features among task vectors in weight space\. Averaging the corresponding task\-level embeddings yields a permutation\-invariant representation of the interaction structure within a subset\. This embedding, therefore, encodes interaction statistics rather than raw parameters, allowing the hypernetwork to focus on modeling residual nonlinear effects\.

Structured low\-rank correction via hypernetwork\.Given the subset embeddingzSz\_\{S\}, HyperFix uses a hypernetworkhϕh\_\{\\phi\}to predict a structured low\-rank correction

ΔS=hϕ​\(zS\)\.\\Delta\_\{S\}=h\_\{\\phi\}\(z\_\{S\}\)\.\(6\)For each encoder weight matrixWℓ∈ℝdℓ×mℓW\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times m\_\{\\ell\}\}, HyperFix predicts low\-rank factorsUℓ∈ℝdℓ×rU\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\\times r\}andVℓ∈ℝmℓ×rV\_\{\\ell\}\\in\\mathbb\{R\}^\{m\_\{\\ell\}\\times r\}, forming a LoRA\-style updateΔ​Wℓ=Uℓ​Vℓ⊤\\Delta W\_\{\\ell\}=U\_\{\\ell\}V\_\{\\ell\}^\{\\top\}, where the rank is set tor=4r=4in all experiments\. The low\-rank parameterization improves structural efficiency while allowing expressive corrections distributed across layers\. The hypernetwork is implemented as a two\-layer MLP with a hidden dimension of 512 and GELU activation\. The MLP outputs a single concatenated vector that is partitioned and reshaped into the per\-layer factors\(Uℓ,Vℓ\)\(U\_\{\\ell\},V\_\{\\ell\}\), thereby producing all encoder updates in a single forward pass\. The resulting correction augments the linear merge in Eq\. \([2](https://arxiv.org/html/2608.11499#Sx3.E2)\)\.

Training objective\.Our primary training objective is to optimize the hypernetwork parameters by aligning the predictive distributions of the merged modelθS\\theta\_\{S\}with those of the corresponding single\-task models\. Specifically, we employ KL\-based knowledge distillation\. During training, the image encoder and all task\-specific classification heads are frozen, and only the HyperFix parameters are optimized\. We train HyperFix on subsets of size one to three \(singleton, pair, and triple\) to learn correction patterns while keeping training computationally tractable\.

For each taski∈Si\\in S, we use the corresponding single\-task modelθi\\theta\_\{i\}as a reference and compare its predictions with those of the merged modelθS\\theta\_\{S\}\. Letpi\(⋅∣x\)p\_\{i\}\(\\cdot\\mid x\)andqi\(⋅∣x\)q\_\{i\}\(\\cdot\\mid x\)denote the predictive distributions induced byθi\\theta\_\{i\}andθS\\theta\_\{S\}, respectively, for taskiiand inputxx\. We minimize the average KL divergence

LS=1\|S\|∑i∈SKL\(pi\(⋅∣x\)∥qi\(⋅∣x\)\)\.L\_\{S\}=\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}\\mathrm\{KL\}\\\!\\left\(p\_\{i\}\(\\cdot\\mid x\)\\;\\\|\\;q\_\{i\}\(\\cdot\\mid x\)\\right\)\.\(7\)For eachi∈Si\\in S, the KL divergence is computed on mini\-batches sampled from taskii’s dataset, and the loss is averaged across tasks inSS\.

Inference\.At inference time, no additional optimization is required\. Given a subsetSS, we computezSz\_\{S\}, predictΔS\\Delta\_\{S\}via Eq\. \([6](https://arxiv.org/html/2608.11499#Sx3.E6)\), and constructθS\\theta\_\{S\}using Eq\. \([2](https://arxiv.org/html/2608.11499#Sx3.E2)\)\. The merged model is then directly evaluated\.

## Theoretical Analysis

In this section, we provide a theoretical analysis of HyperFix, explaining why training on task subsets of size at most three is sufficient for generalization to larger subsets not used during training\.111Our theoretical results are local and perturbative in nature\. We analyze behavior in a neighborhood of the pretrained modelθ0\\theta\_\{0\}under small task updates, where empirical loss landscapes are often locally smooth and well\-conditioned in practice\. The assumptions below are not intended to globally characterize deep networks, but to isolate the mechanisms governing nonlinear interaction effects nearθ0\\theta\_\{0\}\.Detailed proofs are provided in the Supplementary\.

### Preliminaries and Notation

We adopt the notation introduced in the Method section\. Recall that linear merging produces parametersθ0\+τS\\theta\_\{0\}\+\\tau\_\{S\}\(Eq\.[1](https://arxiv.org/html/2608.11499#Sx3.E1)\), and HyperFix augments this with a correction term as in Eq\.[2](https://arxiv.org/html/2608.11499#Sx3.E2)\. For analysis, define the subset\-averaged loss

LS​\(θ\):=1\|S\|​∑i∈SLi​\(θ\)\.L\_\{S\}\(\\theta\):=\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}L\_\{i\}\(\\theta\)\.\(8\)LetθS⋆\\theta\_\{S\}^\{\\star\}denote a local stationary point ofLSL\_\{S\}nearθ0\\theta\_\{0\}\(i\.e\.,∇LS​\(θS⋆\)=0\\nabla L\_\{S\}\(\\theta\_\{S\}^\{\\star\}\)=0\)\. The*ideal residual correction*beyond linear merging is

ΔS⋆:=θS⋆−\(θ0\+τS\)\.\\Delta^\{\\star\}\_\{S\}:=\\theta\_\{S\}^\{\\star\}\-\(\\theta\_\{0\}\+\\tau\_\{S\}\)\.\(9\)Finally, let

z¯:=1N​∑i=1Nzi,σz2:=1N​∑i=1N\|zi−z¯\|2\\bar\{z\}:=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}z\_\{i\},\\qquad\\sigma\_\{z\}^\{2\}:=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\|z\_\{i\}\-\\bar\{z\}\|^\{2\}\(10\)denote the mean and variance of the task embeddings, respectively\.

### Assumptions

We introduce the assumptions used in our analysis\.

A1 \(Local smoothness\)\.Each task lossLi​\(θ\)L\_\{i\}\(\\theta\)is three\-times continuously differentiable in a neighborhood𝒩\\mathcal\{N\}ofθ0\\theta\_\{0\}, and

‖∇2Li​\(θ\)‖op≤H,‖∇3Li​\(θ\)‖op≤M,\\\|\\nabla^\{2\}L\_\{i\}\(\\theta\)\\\|\_\{\\mathrm\{op\}\}\\leq H,\\quad\\\|\\nabla^\{3\}L\_\{i\}\(\\theta\)\\\|\_\{\\mathrm\{op\}\}\\leq M,\(11\)for allθ∈𝒩\\theta\\in\\mathcal\{N\}\.

A2 \(Local conditioning\)\.For any subsetSS, we assume thatLSL\_\{S\}admits a locally well\-conditioned Hessian in a neighborhood ofθ0\\theta\_\{0\}, i\.e\.,

∇2LS​\(θ\)⪰μ​I,\\nabla^\{2\}L\_\{S\}\(\\theta\)\\succeq\\mu I,\(12\)for someμ\>0\\mu\>0in𝒩\\mathcal\{N\}\.

A3 \(Small task updates\)\.Task vectors satisfy‖τi‖≤ρ\\\|\\tau\_\{i\}\\\|\\leq\\rhofor allii, whereρ\\rhois sufficiently small so thatθ0\+τS∈𝒩\\theta\_\{0\}\+\\tau\_\{S\}\\in\\mathcal\{N\}\.

A4 \(Low\-order representability\)\.The dominant component of the ideal correctionΔS⋆\\Delta^\{\\star\}\_\{S\}admits a smooth representation in a permutation\-invariant subset embedding space

ΔS⋆=g⁡\(zS\)\+η⁡\(S\),‖η⁡\(S\)‖≤ε,\\Delta^\{\\star\}\_\{S\}=g\(z\_\{S\}\)\+\\eta\(S\),\\quad\\\|\\eta\(S\)\\\|\\leq\\varepsilon,\(13\)whereggisLgL\_\{g\}\-Lipschitz\.

### Low\-Order Interaction Generalization

###### Theorem 1\(Nonlinear remainder\)\.

Under Assumptions A1–A3,

‖ΔS⋆‖≤1μ​\(‖∇LS​\(θ0\)‖\+H​ρ\),\\\|\\Delta^\{\\star\}\_\{S\}\\\|\\leq\\frac\{1\}\{\\mu\}\\bigl\(\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+H\\rho\\bigr\),\(14\)and the nonlinearity beyond the first\-order Hessian term satisfies

∇LS​\(θ0\+τS\)=∇LS​\(θ0\)\+∇2LS​\(θ0\)​τS\+RS,\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)=\\nabla L\_\{S\}\(\\theta\_\{0\}\)\+\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\)\\tau\_\{S\}\+R\_\{S\},\(15\)where‖RS‖≤M2​ρ2\\\|R\_\{S\}\\\|\\leq\\frac\{M\}\{2\}\\rho^\{2\}\. Hence the parameter effect induced by this nonlinear remainder is bounded by

‖\(∫01∇2LS​\(θ0\+τS\+t​ΔS⋆\)​𝑑t\)−1​RS‖≤M2​μ​ρ2\.\\bigl\\\|\\big\(\\int\_\{0\}^\{1\}\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\+t\\Delta^\{\\star\}\_\{S\}\)\\,dt\\big\)^\{\-1\}R\_\{S\}\\bigr\\\|\\leq\\frac\{M\}\{2\\mu\}\\rho^\{2\}\.\(16\)

#### Interpretation\.

The first inequality bounds the*overall*correction needed beyond linear merging\. It is controlled by \(i\) local conditioning \(1/μ1/\\mu\), \(ii\) the task\-update scaleρ\\rho, and \(iii\) the gradient bias atθ0\\theta\_\{0\}\. The gradient term represents baseline mismatch between the pretrained model and the task mixture\. It does not scale with task\-update magnitude nor reflect higher\-order interactions\. The second inequality isolates the*nonlinear interaction*component: under bounded third derivatives, the part not captured by the first\-order Hessian term \(instead captured by the curvature\-driven remainder\) shrinks quadratically withρ\\rho\(O⁡\(ρ2\)O\(\\rho^\{2\}\)\)\. Thus, when task vectors are small, the curvature\-induced component of the correction remains controlled\. This result motivates learning a compact residual correction\.

Theorem 1 establishes that the residual correction beyond linear merging is controlled in magnitude and dominated by low\-order interaction terms under small task updates\. In particular, the nonlinear remainder scales quadratically with the update size\. This suggests that the dominant interaction structure is smooth and locally well\-behaved in the subset embedding space\. We now formalize how learning such low\-order interactions from small task subsets supports generalization to larger subsets not used during training\.

###### Theorem 2\(Low\-order interaction generalization\)\.

Lethϕh\_\{\\phi\}be a hypernetwork trained on task subsets with\|S\|≤3\|S\|\\leq 3to minimize𝔼\|S\|≤3​‖hϕ​\(zS\)−ΔS⋆‖2\\mathbb\{E\}\_\{\|S\|\\leq 3\}\\\|h\_\{\\phi\}\(z\_\{S\}\)\-\\Delta^\{\\star\}\_\{S\}\\\|^\{2\}\. Then for any subset sizem≥2m\\geq 2,

𝔼\|S\|=m​‖hϕ​\(zS\)−ΔS⋆‖≤ε\+Lg​σzm,\\mathbb\{E\}\_\{\|S\|=m\}\\\|h\_\{\\phi\}\(z\_\{S\}\)\-\\Delta^\{\\star\}\_\{S\}\\\|\\leq\\varepsilon\+\\frac\{L\_\{g\}\\sigma\_\{z\}\}\{\\sqrt\{m\}\},\(17\)whereσz2\\sigma\_\{z\}^\{2\}bounds the variance of the task embeddingsziz\_\{i\}across tasks\. The1m\\frac\{1\}\{\\sqrt\{m\}\}term arises from the concentration of the empirical subset embeddingzSz\_\{S\}around its expectation as the subset size increases\.

#### Interpretation\.

HyperFix generalizes from small task subsets because task interactions are locally smooth and dominated by low\-order effects\. If each task updateτi\\tau\_\{i\}is small, and the loss landscape around the pretrained modelθ0\\theta\_\{0\}is smooth, then the deviation from linear merging arises primarily from pairwise and triple interactions\. Higher\-order interactions decay rapidly with the magnitude of task updates\. Furthermore, the subset embeddingzSz\_\{S\}is an empirical average of task interaction statistics\. As the subset size increases, this average becomes more stable due to the concentration of measure\. Since HyperFix learns a smooth mapping fromzSz\_\{S\}to correction parameters, it can extrapolate from small task subsets \(\|S\|≤3\|S\|\\leq 3\) to larger task subsets\. This explains why training up to triples is sufficient\.

## Experiments

In this section, we evaluate HyperFix under the combinatorial correction setting through three key questions\. First, can a correction function trained only on small task subsets generalize to larger subsets while maintaining performance? Second, do the predicted corrections meaningfully adapt to the selected task subset, rather than acting as a fixed global adjustment? Third, can HyperFix replace repeated per\-subset tuning with a shared training procedure, thereby reducing computational cost while maintaining competitive or superior performance?

### Setup

#### Combinatorial correction setting\.

We evaluate the*combinatorial correction*problem on eight image classification benchmarks: Cars\([Krause et al\. 2013](https://arxiv.org/html/2608.11499#bib.bib9)\), DTD\([Cimpoi et al\. 2014](https://arxiv.org/html/2608.11499#bib.bib2)\), EuroSAT\([Helber et al\. 2019](https://arxiv.org/html/2608.11499#bib.bib5)\), GTSRB\([Stallkamp et al\. 2011](https://arxiv.org/html/2608.11499#bib.bib20)\), MNIST\([LeCun 1998](https://arxiv.org/html/2608.11499#bib.bib11)\), RESISC45\([Cheng, Han, and Lu 2017](https://arxiv.org/html/2608.11499#bib.bib1)\), SUN397\([Xiao et al\. 2016](https://arxiv.org/html/2608.11499#bib.bib25)\), and SVHN\([Netzer et al\. 2011](https://arxiv.org/html/2608.11499#bib.bib13)\)\. Given task vectors\{τi\}i=1N\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}, the goal is to learn a shared correction function such that, for any subsetS⊆\{1,…,N\}S\\subseteq\\\{1,\\dots,N\\\}, the merged model performs well on all tasks inSS\. HyperFix is trained only on subsets with\|S\|≤3\|S\|\\leq 3\(singleton, pair, and triple\) and evaluated on both subset sizes used during training \(\|S\|=2,3\|S\|=2,3\) and larger subset sizes not used during training \(\|S\|=4,…,8\|S\|=4,\\dots,8\)\. For each subset size\|S\|\|S\|, we evaluate all\(8\|S\|\)\\binom\{8\}\{\|S\|\}possible subsets and report their average performance\.

#### Baselines\.

We evaluate both weight\-space\([Ilharco et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib7)\)and tangent\-space\([Ortiz\-Jimenez, Favero, and Frossard 2023](https://arxiv.org/html/2608.11499#bib.bib14)\)task vectors with the same merging strategies\.*Mean*\([Wortsman et al\. 2022](https://arxiv.org/html/2608.11499#bib.bib24)\)performs uniform averaging of task vectors without additional scaling\.*Sum\+\+Scalar*\([Ilharco et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib7)\)sums task vectors and selects a scalar coefficient separately for each subset from 21 evenly spaced values in\[0,1\]\[0,1\]to maximize the average normalized validation accuracy\.*TIES\+\+Scalar*\([Yadav et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib26)\)first resolves sign conflicts using TIES merging and then applies the same scalar search\. In contrast, HyperFix augments linear merging with a subset\-conditioned nonlinear correction and requires no per\-subset tuning\. We evaluate*Mean\+\+HyperFix*and*TIES\+\+HyperFix*, where the learned correction is applied on top of the corresponding linear merge rule\.

#### Training details\.

We conduct experiments using pretrained CLIP models with ViT\-B/32, ViT\-B/16, and ViT\-L/14 backbones\([Radford et al\. 2021](https://arxiv.org/html/2608.11499#bib.bib17)\)\. Main results are reported on ViT\-B/32, with additional results for ViT\-B/16 and ViT\-L/14 provided in the Supplementary\. During training, all encoder and task\-specific classification head parameters are frozen, and only the hypernetwork parameters are optimized\. Training is performed for 10 epochs over all singleton, pair, and triple task subsets \(\|S\|∈\{1,2,3\}\|S\|\\in\\\{1,2,3\\\}\), with random shuffling at each epoch and a batch size of 128 for each task\. For each subset, the hypernetwork is optimized by averaging the logit\-level KL distillation losses from the corresponding single\-task models, using temperatureT=2\.0T=2\.0\. We optimize the hypernetwork using AdamW with learning rate1×10−41\\times 10^\{\-4\}and weight decay0\.10\.1\. Gradients are clipped to 1\.0\. The hypernetwork is a two\-layer MLP with hidden dimension 512 and predicts low\-rank updates with rankr=4r=4for each encoder weight matrix\. Task vectors are computed from independently fine\-tuned models on the training split\. HyperFix is trained on the validation split, which is also used to tune the scalar coefficients for the scalar\-based baselines, and all reported results are evaluated on the held\-out test split\. All experiments are conducted on a single NVIDIA GH200 GPU\.

#### Metrics\.

We report normalized accuracy following prior work\([Ilharco et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib7)\), defined as the task accuracy divided by the accuracy of the corresponding single\-task fine\-tuned model\. For each subset, normalized accuracy is averaged over all tasks in the subset, and we report the mean and standard deviation across all subsets of the same size\. We additionally report absolute accuracies in the Supplementary, where we observe consistent method rankings and performance trends across subset sizes\.

### Generalization Across Task Subsets

Table 1:Generalization performance of HyperFix on task subsets\. We evaluate merging performance across subset sizes\|S\|=2,…,8\|S\|=2,\\dots,8after training only on small task subsets \(singleton, pair, and triple\)\. Results are averaged over all\(8\|S\|\)\\binom\{8\}\{\|S\|\}subsets for each\|S\|\|S\|and reported as normalized accuracy \(%\)\. We compare methods using standard fine\-tuning\([Ilharco et al\. 2023](https://arxiv.org/html/2608.11499#bib.bib7)\)\(top\) and tangent\-space fine\-tuning \(bottom\)\([Ortiz\-Jimenez, Favero, and Frossard 2023](https://arxiv.org/html/2608.11499#bib.bib14)\)\. Avg\. reports the average normalized accuracy across all subset sizes\.#### Training on small task subsets enables generalization to larger subsets\.

Table[1](https://arxiv.org/html/2608.11499#Sx5.T1)reports normalized accuracy across all subset sizes\. While Mean performs competitively for small task subsets \(\|S\|=2\|S\|=2\), its performance degrades sharply as more tasks are merged, dropping from 95\.7 % to 72\.9 % under standard fine\-tuning\. Although Scalar tuning mitigates this degradation, it still exhibits a substantial performance drop as\|S\|\|S\|increases\. In contrast, trained only on task subsets of size at most three, HyperFix maintains high performance on\|S\|=4,…,8\|S\|=4,\\dots,8without additional tuning\. Under standard fine\-tuning, Mean\+\+HyperFix improves the\|S\|=8\|S\|=8performance from 72\.9 % \(Mean\) to 92\.0 %, surpassing both Sum\+\+Scalar \(77\.0 %\) and TIES\+\+Scalar \(80\.9 %\), despite these methods performing per\-subset scalar tuning \(with TIES additionally resolving sign conflicts\)\. TIES\+\+HyperFix further reaches 92\.9 %\. Notably, the performance gap between linear merging baselines and HyperFix widens as\|S\|\|S\|increases, suggesting that non\-additive task interactions become increasingly important for larger task subsets\. The same trend holds under tangent\-space fine\-tuning\. Mean\+\+HyperFix consistently outperforms all linear merging baselines across every subset size\. This consistency across both fine\-tuning regimes indicates that the gains of HyperFix arise from modeling structured task interactions rather than from a specific task\-vector construction\. We confirm that the performance gains persist across backbone scales, including ViT\-B/16 and ViT\-L/14 \(see Supplementary\)\. The reported standard deviations reflect variation across all\(8\|S\|\)\\binom\{8\}\{\|S\|\}task subsets\. When\|S\|=8\|S\|=8, there is only a single subset, resulting in zero standard deviation by definition\.

Table 2:Effectiveness of training objective for combinatorial correction\. We compare cross\-entropy \(CE\) training with logit\-level KL distillation for Mean\+\+HyperFix\. We report normalized accuracy \(%\) averaged over all task subsets for each subset size\. Avg\. further averages results across\|S\|=2,…,8\|S\|=2,\\dots,8\. KL uses logit\-level distillation from the corresponding single\-task fine\-tuned models, while CE uses ground\-truth labels\. All other hyperparameters are identical\. KL consistently improves performance, particularly for larger task subsets not used during training \(\|S\|≥4\|S\|\\geq 4\)\.
#### KL distillation improves generalization across task subsets\.

We compare two training objectives for Mean\+\+HyperFix under identical data usage and optimization settings: \(i\) KL distillation from the corresponding single\-task fine\-tuned models, and \(ii\) supervised CE on ground\-truth labels from the same training task subsets\. Table[2](https://arxiv.org/html/2608.11499#Sx5.T2)shows that KL consistently yields stronger performance across all subset sizes, with increasingly larger gains as\|S\|\|S\|grows\. For example, at\|S\|=8\|S\|=8, KL achieves 92\.0% compared to 87\.8% for CE \(\+4\.2 percentage points\)\. Overall, KL improves the average normalized accuracy from 91\.1% to 94\.4% \(\+3\.3 percentage points\)\. These results indicate that distillation provides a smoother and more structured learning signal for predicting parameter corrections than direct supervision with ground\-truth labels alone\.

Table 3:Ablation on maximum training subset size\. HyperFix is trained only on subsets with size\|S\|≤\|S\|max\|S\|\\leq\|S\|\_\{\\max\}, where\|S\|max\|S\|\_\{\\max\}denotes the maximum subset size used during training\. Reported values are averaged across subset sizes\|S\|=2,…,8\|S\|=2,\\ldots,8, after averaging over all task subsets at each size\. Performance improves as\|S\|max\|S\|\_\{\\max\}increases from 1 to 3, while additional gains beyond\|S\|max≥4\|S\|\_\{\\max\}\\geq 4are limited, consistent with our theory that dominant interaction effects are captured by low\-order subsets\.
#### Ablation on maximum training subset size\.

Table[3](https://arxiv.org/html/2608.11499#Sx5.T3)analyzes how the maximum subset size\|S\|max\|S\|\_\{\\max\}\(i\.e\., the largest subset size included during HyperFix training\) affects merging performance for Mean\+\+HyperFix\. Training only on single\-task subsets \(\|S\|max=1\|S\|\_\{\\max\}=1\) yields 82\.0%\. Including pairwise subsets \(\|S\|max=2\|S\|\_\{\\max\}=2\) improves performance to 90\.4% \(\+8\.4 points\), indicating that including pair subsets provides substantial additional supervision\. Extending training to triple subsets \(\|S\|max=3\|S\|\_\{\\max\}=3\) further increases performance to 94\.4% \(\+4\.0 points\), already achieving strong generalization to unseen larger subsets up to\|S\|=8\|S\|=8\. Beyond this point, additional exposure to higher\-order subsets yields only marginal gains, increasing from 94\.4% to at most 95\.9% \(\+1\.5 points overall\)\. This saturation suggests that training on low\-order subsets captures most of the correction patterns needed for larger\-subset generalization\. These empirical findings are consistent with Theorem[1](https://arxiv.org/html/2608.11499#Thmtheorem1)and Theorem[2](https://arxiv.org/html/2608.11499#Thmtheorem2), which together indicate that nonlinear remainders are controlled in magnitude and that once dominant low\-order interactions are learned, higher\-order effects contribute diminishing additional benefit\.

### Subset\-Specificity of Corrections

![Refer to caption](https://arxiv.org/html/2608.11499v1/Figures/Magnitude_of_corrections.png)Figure 3:Magnitude of corrections across subset sizes\. We show the distribution of theℓ2\\ell\_\{2\}norm of the correctionΔS\\Delta\_\{S\}for each subset size, computed over all\(8\|S\|\)\\binom\{8\}\{\|S\|\}task subsets\. Boxes indicate interquartile ranges, center lines denote medians, and whiskers show the 5th–95th percentile range\. Across all subset sizes, the median remains stable\.Table 4:Dependence on the subset embedding\. We report normalized accuracy \(%\) for\|S\|=2\|S\|=2under different inputs\. ReplacingzSz\_\{S\}with shuffled or sign\-flipped embeddings degrades performance, indicating that the predicted correction meaningfully depends on the subset representation\.#### Magnitude of corrections\.

To better understand the behavior of the hypernetwork, we analyze theℓ2\\ell\_\{2\}norm of the predicted correctionΔS\\Delta\_\{S\}across subset sizes\|S\|\|S\|for Mean\+\+HyperFix\. For each\|S\|∈\{2,…,8\}\|S\|\\in\\\{2,\\dots,8\\\}, we compute‖ΔS‖2\\\|\\Delta\_\{S\}\\\|\_\{2\}over all\(8\|S\|\)\\binom\{8\}\{\|S\|\}task combinations and examine its distribution\. As shown in Figure[3](https://arxiv.org/html/2608.11499#Sx5.F3), the distribution of‖ΔS‖2\\\|\\Delta\_\{S\}\\\|\_\{2\}remains well\-controlled across subset sizes\. The median remains at a comparable scale across subset sizes\. This empirical behavior is consistent with the residual bound discussed in the Theoretical Analysis section, which predicts that the magnitude of the nonlinear correction is controlled by the local update scale under smoothness assumptions\.

#### Dependence on the subset embedding\.

We analyze how the predicted correction depends on the subset embeddingzSz\_\{S\}under the Mean\+\+HyperFix setting in Table[4](https://arxiv.org/html/2608.11499#Sx5.T4)\. Given the subset embeddingzSz\_\{S\}for a subsetSS, we consider two perturbations while keeping the base mergeτS\\tau\_\{S\}fixed: \(i\)*shuffled*: replacezSz\_\{S\}withzS~z\_\{\\tilde\{S\}\}from a randomly sampledS~≠S\\tilde\{S\}\\neq S, and \(ii\)*sign\-flipped*: replacezSz\_\{S\}with−zS\-z\_\{S\}\. Using the correct embedding yields the highest accuracy \(97\.7%\), improving over the Mean \(95\.7%\) by \+2\.0 points\. In contrast, perturbing the input embedding degrades performance, sign\-flipping reduces accuracy to 93\.3%, and shuffling yields 95\.4%, both close to or below the Mean\. These results indicate that the predicted correction meaningfully depends on the subset embedding rather than acting as a fixed offset independent of the selected task subset\.

### Training Cost and Computational Efficiency

Although scalar\-based merging requires no additional training, it relies on validation time optimization\. For each task subsetSS, scalar tuning evaluatesKKcandidate coefficients, each requiring evaluation over all tasks inSS\. Aggregated over all task subsets up to a maximum size\|S\|max\|S\|\_\{\\max\}, the cumulative validation cost becomes∑j=2\|S\|max\(Nj\)​j​K\\sum\_\{j=2\}^\{\|S\|\_\{\\max\}\}\\binom\{N\}\{j\}jK\. In our eight task setting withK=21K=21, this results in21,33621\{,\}336validation model evaluations before testing\. In contrast, HyperFix shifts this optimization to a single amortized training phase\. We train the hypernetwork once, using only task subsets with\|S\|≤3\|S\|\\leq 3, corresponding to∑j=13\(Nj\)\\sum\_\{j=1\}^\{3\}\\binom\{N\}\{j\}training subsets, after which the learned correction function is fixed and applied to all subsets without any per\-subset optimization\. For a fair comparison, HyperFix is trained on the validation split, using the same data employed for scalar coefficient search\. As the number of tasks increases, the number of possible task subsets grows combinatorially, whereas the training cost of HyperFix scales only with subsets up to size three\. To quantify the efficiency gain, we measure the wall\-clock time required to construct and evaluate the merged model under identical hardware and data settings\. All experiments are conducted on a single NVIDIA GH200 \(120GB HBM3\) GPU with CUDA 12\.4\. For the full task subset with\|S\|=8\|S\|=8, scalar tuning requires1634\.26seconds due to validation search overK=21K=21candidate coefficients\. In contrast, HyperFix requires only284\.16seconds without any per\-subset search, reducing the total execution time by82\.6%\. This empirical gap directly reflects the elimination of repeated validation\-time optimization in HyperFix\.

## Conclusion

We introduced*combinatorial correction*as the problem of learning a shared correction function over the combinatorial space of task subsets\. We showed that linear task vector merging faces both representational and scalability limitations as the subset size increases\. Although trained only on small task subsets, HyperFix generalizes to larger subsets not used during training without per\-subset optimization, consistently improving performance across diverse benchmarks\. These results demonstrate the importance of modeling nonlinear task interactions for scalable and generalizable task vector merging, opening a path toward more principled and general model merging frameworks\.

## References

- Cheng, Han, and Lu \(2017\)Cheng, G\.; Han, J\.; and Lu, X\. 2017\.Remote sensing image scene classification: Benchmark and state of the art\.*Proceedings of the IEEE*, 105\(10\): 1865–1883\.
- Cimpoi et al\. \(2014\)Cimpoi, M\.; Maji, S\.; Kokkinos, I\.; Mohamed, S\.; and Vedaldi, A\. 2014\.Describing Textures in the Wild\.In*CVPR*\.
- Gargiulo et al\. \(2025\)Gargiulo, A\. A\.; Crisostomi, D\.; Bucarelli, M\. S\.; Scardapane, S\.; Silvestri, F\.; and Rodola, E\. 2025\.Task singular vectors: Reducing task interference in model merging\.In*CVPR*\.
- Ha, Dai, and Le \(2017\)Ha, D\.; Dai, A\. M\.; and Le, Q\. V\. 2017\.HyperNetworks\.In*ICLR*\.
- Helber et al\. \(2019\)Helber, P\.; Bischke, B\.; Dengel, A\.; and Borth, D\. 2019\.Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification\.*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12\(7\): 2217–2226\.
- Huang et al\. \(2024\)Huang, B\.; Mitra, C\.; Arbelle, A\.; Karlinsky, L\.; Darrell, T\.; and Herzig, R\. 2024\.Multimodal task vectors enable many\-shot multimodal in\-context learning\.In*NeurIPS*\.
- Ilharco et al\. \(2023\)Ilharco, G\.; Ribeiro, M\. T\.; Wortsman, M\.; Schmidt, L\.; Hajishirzi, H\.; and Farhadi, A\. 2023\.Editing models with task arithmetic\.In*ICLR*\.
- Kim, Han, and Choe \(2025\)Kim, H\. S\.; Han, D\.; and Choe, J\. 2025\.NegMerge: Sign\-Consensual Weight Merging for Machine Unlearning\.In*ICML*\.
- Krause et al\. \(2013\)Krause, J\.; Stark, M\.; Deng, J\.; and Fei\-Fei, L\. 2013\.3D Object Representations for Fine\-Grained Categorization\.In*ICCVW*, 554–561\.
- Krueger et al\. \(2017\)Krueger, D\.; Huang, C\.\-W\.; Islam, R\.; Turner, R\.; Lacoste, A\.; and Courville, A\. 2017\.Bayesian hypernetworks\.
- LeCun \(1998\)LeCun, Y\. 1998\.The MNIST database of handwritten digits\.http://yann\.lecun\.com/exdb/mnist/\.
- Meng et al\. \(2023\)Meng, K\.; Sharma, A\. S\.; Andonian, A\. J\.; Belinkov, Y\.; and Bau, D\. 2023\.Mass\-Editing Memory in a Transformer\.In*ICLR*\.
- Netzer et al\. \(2011\)Netzer, Y\.; Wang, T\.; Coates, A\.; Bissacco, A\.; Wu, B\.; and Ng, A\. Y\. 2011\.Reading Digits in Natural Images with Unsupervised Feature Learning\.In*NeurIPSW*\.
- Ortiz\-Jimenez, Favero, and Frossard \(2023\)Ortiz\-Jimenez, G\.; Favero, A\.; and Frossard, P\. 2023\.Task arithmetic in the tangent space: Improved editing of pre\-trained models\.In*NeurIPS*, 66727–66754\.
- Panariello et al\. \(2025\)Panariello, A\.; Marczak, D\.; Magistri, S\.; Porrello, A\.; Twardowski, B\.; Bagdanov, A\. D\.; Calderara, S\.; and van de Weijer, J\. 2025\.Accurate and Efficient Low\-Rank Model Merging in Core Space\.In*NeurIPS*\.
- Porrello et al\. \(2026\)Porrello, A\.; Buzzega, P\.; Dangel, F\.; Sommariva, T\.; Salami, R\.; Bonicelli, L\.; and Calderara, S\. 2026\.Dataless Weight Disentanglement in Task Arithmetic via Kronecker\-Factored Approximate Curvature\.In*ICLR*\.
- Radford et al\. \(2021\)Radford, A\.; Kim, J\. W\.; Hallacy, C\.; Ramesh, A\.; Goh, G\.; Agarwal, S\.; Sastry, G\.; Askell, A\.; Mishkin, P\.; Clark, J\.; Krueger, G\.; and Sutskever, I\. 2021\.Learning Transferable Visual Models From Natural Language Supervision\.In*ICML*\.
- Ratzlaff and Fuxin \(2019\)Ratzlaff, N\.; and Fuxin, L\. 2019\.HyperGAN: A Generative Model for Diverse, Performant Neural Networks\.In*ICML*\.
- Shenaj et al\. \(2025\)Shenaj, D\.; Bohdal, O\.; Ozay, M\.; Zanuttigh, P\.; and Michieli, U\. 2025\.Lora\.rar: Learning to merge loras via hypernetworks for subject\-style conditioned image generation\.In*ICCV*\.
- Stallkamp et al\. \(2011\)Stallkamp, J\.; Schlipsing, M\.; Salmen, J\.; and Igel, C\. 2011\.The German traffic sign recognition benchmark: a multi\-class classification competition\.In*IJCNN*, 1453–1460\. IEEE\.
- Sun et al\. \(2025\)Sun, W\.; Li, Q\.; Li, B\.; Wang, W\.; and Geng, Y\. 2025\.Task Arithmetic in Trust Region: A Training\-Free Model Merging Approach to Navigate Knowledge Conflicts\.In*ICLR*\.
- Tan, Zhang, and Fu \(2024\)Tan, C\.; Zhang, G\.; and Fu, J\. 2024\.Massive Editing for Large Language Models via Meta Learning\.In*ICLR*\.
- Wan et al\. \(2025\)Wan, W\.; Lao, Q\.; Xie, Z\.; Wu, H\.; Lin, R\.; Lin, L\.; and Wang, K\. 2025\.Massive Editing for Large Language Models Based on Dynamic Weight Generation\.
- Wortsman et al\. \(2022\)Wortsman, M\.; Ilharco, G\.; Gadre, S\. Y\.; Roelofs, R\.; Gontijo\-Lopes, R\.; Morcos, A\. S\.; Namkoong, H\.; Farhadi, A\.; Carmon, Y\.; Kornblith, S\.; et al\. 2022\.Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.In*ICML*\.
- Xiao et al\. \(2016\)Xiao, J\.; Ehinger, K\. A\.; Hays, J\.; Torralba, A\.; and Oliva, A\. 2016\.Sun database: Exploring a large collection of scene categories\.*IJCV*, 119: 3–22\.
- Yadav et al\. \(2023\)Yadav, P\.; Tam, D\.; Choshen, L\.; Raffel, C\. A\.; and Bansal, M\. 2023\.Ties\-merging: Resolving interference when merging models\.In*NeurIPS*, 7093–7115\.
- Yang et al\. \(2024\)Yang, E\.; Wang, Z\.; Shen, L\.; Liu, S\.; Guo, G\.; Wang, X\.; and Tao, D\. 2024\.AdaMerging: Adaptive Model Merging for Multi\-Task Learning\.In*ICLR*\.
- Yu et al\. \(2024\)Yu, L\.; Yu, B\.; Yu, H\.; Huang, F\.; and Li, Y\. 2024\.Language models are super mario: Absorbing abilities from homologous models as a free lunch\.In*ICML*\.
- Zeng et al\. \(2025\)Zeng, F\.; Guo, H\.; Zhu, F\.; Shen, L\.; and Tang, H\. 2025\.Robustmerge: Parameter\-efficient model merging for mllms with direction robustness\.In*NeurIPS*\.
- Zhang et al\. \(2025\)Zhang, B\.; Du, Y\.; Zhen, X\.; and Shao, L\. 2025\.Variational Task Vector Composition\.In*NeurIPS*\.

## Supplementary Material

This supplementary material provides further experimental results and theoretical proof details\.

### Further Experimental Results

#### Results on Additional Backbones\.

To verify that the benefits of HyperFix are not specific to a single backbone scale, we additionally evaluate on CLIP ViT\-B/16 and ViT\-L/14\. Table[5](https://arxiv.org/html/2608.11499#Sx7.T5)shows that the overall trend observed on ViT\-B/32 remains consistent across larger backbones\. Linear merging methods degrade as the subset size increases, whereas Mean \+ HyperFix maintains much stronger and more stable performance across all subset sizes\. For example, on ViT\-B/16, the performance of Mean drops from 96\.4% at\|S\|=2\|S\|=2to 77\.5% at\|S\|=8\|S\|=8, while Mean \+ HyperFix retains 92\.7% at\|S\|=8\|S\|=8and improves the overall average from 85\.0% to 95\.1%\. Similar behavior is observed on ViT\-L/14, where HyperFix again achieves the best average performance and remains stable as more tasks are merged\. Each configuration is trained once\. Reported means and standard deviations are computed across all\(8\|S\|\)\\binom\{8\}\{\|S\|\}task subsets\.

Table 5:Generalization performance across additional backbones\. Results are averaged over all\(8\|S\|\)\\binom\{8\}\{\|S\|\}subsets and reported as normalized accuracy \(%\)\. Values after±\\pmdenote the standard deviation across subsets\. Avg\. reports the average normalized accuracy across subset sizes\|S\|=2,…,8\|S\|=2,\\ldots,8\.Table 6:Absolute accuracy across task subsets\. Results are averaged over all\(8\|S\|\)\\binom\{8\}\{\|S\|\}subsets and reported as absolute accuracy \(%\)\. Values after±\\pmdenote the standard deviation across subsets\. Avg\. reports the average absolute accuracy across subset sizes\|S\|=2,…,8\|S\|=2,\\ldots,8\.
#### Absolute Accuracy on Task Subsets\.

While the main paper reports normalized accuracy, we additionally report absolute accuracy\. Table[6](https://arxiv.org/html/2608.11499#Sx7.T6)summarizes the results across all task subsets\. The overall trends are consistent with the normalized accuracy results\. Linear merging methods degrade substantially as the subset size increases, whereas HyperFix maintains much stronger performance across all subset sizes\. For example, under standard fine\-tuning on ViT\-B/32, Mean decreases from 86\.6% at\|S\|=2\|S\|=2to 65\.4% at\|S\|=8\|S\|=8, while Mean \+ HyperFix retains 83\.5%\. TIES \+ HyperFix further improves this to 84\.3%, consistently outperforming scalar\-based baselines\. A similar pattern holds under tangent\-space fine\-tuning, where HyperFix variants achieve the strongest performance across all subset sizes\. The larger standard deviations in absolute accuracy mainly arise from differences in the performance of the single\-task fine\-tuned models across datasets\. Since normalized accuracy is divided by the corresponding single\-task performance, it reduces this cross\-dataset variation and exhibits smaller variance\. Overall, these results confirm that the improvements of HyperFix persist when evaluated using absolute accuracy\.

Table 7:Full results for maximum training subset size\. HyperFix is trained on subsets with size\|S\|≤\|S\|max\|S\|\\leq\|S\|\_\{\\max\}\. Reported values are the average normalized accuracy \(%\) across all\(8\|S\|\)\\binom\{8\}\{\|S\|\}task subsets\.
#### Full Results for the Training Subset Size Ablation\.

In the main paper, we report the average normalized accuracy across all subset sizes to analyze the effect of the maximum training subset size\|S\|max\|S\|\_\{\\max\}\. For completeness, Table[7](https://arxiv.org/html/2608.11499#Sx7.T7)provides the full performance across subset sizes\. Consistent with the trends in the main paper, increasing the training subset size from\|S\|max=1\|S\|\_\{\\max\}=1to\|S\|max=3\|S\|\_\{\\max\}=3leads to substantial improvements across most subset sizes\. For example, the average accuracy increases from 82\.0% to 94\.4%\. This suggests that learning pairwise and triple task interactions plays an important role in modeling nonlinear task vector merging\. Beyond\|S\|max≥4\|S\|\_\{\\max\}\\geq 4, additional gains become relatively small\. This saturation indicates that a large part of the interaction effects can already be captured by pairwise and triple task interactions\.

### Theory Proof Details

#### Proof of Theorem 1: A bound on the residual correction magnitude

###### Lemma 1\(Residual correction is controlled by local geometry\)\.

Under the assumptions in the main paper, the ideal correctionΔS⋆\\Delta^\{\\star\}\_\{S\}satisfies

‖ΔS⋆‖≤1μ​‖∇LS​\(θ0\+τS\)‖\.\\\|\\Delta^\{\\star\}\_\{S\}\\\|\\;\\leq\\;\\frac\{1\}\{\\mu\}\\,\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)\\\|\.\(18\)Moreover, local smoothness gives

‖∇LS​\(θ0\+τS\)‖≤‖∇LS​\(θ0\)‖\+H​‖τS‖\.\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)\\\|\\leq\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+H\\\|\\tau\_\{S\}\\\|\.\(19\)Since‖τS‖≤ρ\\\|\\tau\_\{S\}\\\|\\leq\\rho, we further have

‖∇LS​\(θ0\)‖\+H​‖τS‖≤‖∇LS​\(θ0\)‖\+H​ρ\.\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+H\\\|\\tau\_\{S\}\\\|\\leq\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+H\\rho\.\(20\)In the common case where‖∇LS​\(θ0\)‖\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|is small \(e\.g\.,θ0\\theta\_\{0\}is a good shared initializer\), this yields‖ΔS⋆‖=O⁡\(ρ\)\\\|\\Delta^\{\\star\}\_\{S\}\\\|=O\(\\rho\)\.

###### Proof\.

By definition,∇LS​\(θS⋆\)=0\\nabla L\_\{S\}\(\\theta\_\{S\}^\{\\star\}\)=0andθS⋆=θ0\+τS\+ΔS⋆\\theta\_\{S\}^\{\\star\}=\\theta\_\{0\}\+\\tau\_\{S\}\+\\Delta\_\{S\}^\{\\star\}\. Define the averaged Hessian along the segment fromθ0\+τS\\theta\_\{0\}\+\\tau\_\{S\}toθS⋆\\theta\_\{S\}^\{\\star\}by

AS:=∫01∇2LS​\(θ0\+τS\+t​ΔS⋆\)​𝑑t\.A\_\{S\}:=\\int\_\{0\}^\{1\}\\nabla^\{2\}L\_\{S\}\\bigl\(\\theta\_\{0\}\+\\tau\_\{S\}\+t\\Delta\_\{S\}^\{\\star\}\\bigr\)\\,dt\.\(21\)The mean\-value form of Taylor’s theorem then gives

∇LS​\(θS⋆\)=∇LS​\(θ0\+τS\)\+AS​ΔS⋆\.\\nabla L\_\{S\}\(\\theta\_\{S\}^\{\\star\}\)=\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)\+A\_\{S\}\\Delta\_\{S\}^\{\\star\}\.\(22\)Since∇LS​\(θS⋆\)=0\\nabla L\_\{S\}\(\\theta\_\{S\}^\{\\star\}\)=0, it follows that

AS​ΔS⋆=−∇LS​\(θ0\+τS\)\.A\_\{S\}\\Delta\_\{S\}^\{\\star\}=\-\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)\.\(23\)By local stability, each Hessian inside the integral has a minimum eigenvalue at leastμ\\mu, hence the averaged Hessian is also⪰μ​I\\succeq\\mu Iand is invertible with operator norm at most1/μ1/\\mu\. Therefore,

‖ΔS⋆‖≤1μ​‖∇LS​\(θ0\+τS\)‖\.\\\|\\Delta^\{\\star\}\_\{S\}\\\|\\leq\\frac\{1\}\{\\mu\}\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)\\\|\.\(24\)For the second inequality, apply Taylor’s theorem to∇LS\\nabla L\_\{S\}atθ0\\theta\_\{0\}:

∇LS​\(θ0\+τS\)\\displaystyle\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)=∇LS​\(θ0\)\\displaystyle=\\nabla L\_\{S\}\(\\theta\_\{0\}\)\(25\)\+\(∫01∇2LS​\(θ0\+t​τS\)​dt\)​τS\.\\displaystyle\+\\left\(\\int\_\{0\}^\{1\}\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\+t\\tau\_\{S\}\)\\,dt\\right\)\\tau\_\{S\}\.Taking norms and using‖∇2LS​\(⋅\)‖op≤H\\\|\\nabla^\{2\}L\_\{S\}\(\\cdot\)\\\|\_\{\\mathrm\{op\}\}\\leq H\(since it averages task Hessians bounded byHH\),

‖∇LS​\(θ0\+τS\)‖≤‖∇LS​\(θ0\)‖\+H​‖τS‖\.\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\)\\\|\\leq\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+H\\\|\\tau\_\{S\}\\\|\.\(26\)Finally,‖τS‖≤1\|S\|​∑i∈S‖τi‖≤ρ\\\|\\tau\_\{S\}\\\|\\leq\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}\\\|\\tau\_\{i\}\\\|\\leq\\rho\. ∎

#### Proof of Theorem 1: Why higher\-order interactions decay asO⁡\(ρ2\)O\(\\rho^\{2\}\)

###### Lemma 2\(Third\-order Taylor remainder bound\)\.

Under the smoothness assumptions, for anyuuwith‖u‖≤2​ρ\\\|u\\\|\\leq 2\\rho,

∇LS​\(θ0\+u\)=∇LS​\(θ0\)\+∇2LS​\(θ0\)​u\+RS​\(u\),\\nabla L\_\{S\}\(\\theta\_\{0\}\+u\)=\\nabla L\_\{S\}\(\\theta\_\{0\}\)\+\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\)\\,u\+R\_\{S\}\(u\),\(27\)where the remainder satisfies

‖RS​\(u\)‖≤M2​‖u‖2\.\\\|R\_\{S\}\(u\)\\\|\\leq\\frac\{M\}\{2\}\\\|u\\\|^\{2\}\.\(28\)

###### Proof\.

This is the standard integral remainder form for the gradient:

∇LS​\(θ0\+u\)\\displaystyle\\nabla L\_\{S\}\(\\theta\_\{0\}\+u\)=∇LS​\(θ0\)\+∇2LS​\(θ0\)​u\\displaystyle=\\nabla L\_\{S\}\(\\theta\_\{0\}\)\+\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\)u\(29\)\+∫01\[∇2LS\(θ0\+tu\)−∇2LS\(θ0\)\]udt\.\\displaystyle\+\\int\_\{0\}^\{1\}\\Bigl\[\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\+tu\)\-\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\)\\Bigr\]u\\,dt\.Using the mean value form for Hessians with bounded third derivative,‖∇2LS​\(θ0\+t​u\)−∇2LS​\(θ0\)‖op≤M​t​‖u‖\\\|\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\+tu\)\-\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\)\\\|\_\{\\mathrm\{op\}\}\\leq M\\,t\\\|u\\\|, we obtain

‖RS​\(u\)‖≤∫01M​t​‖u‖⋅‖u‖​𝑑t=M2​‖u‖2\.\\\|R\_\{S\}\(u\)\\\|\\leq\\int\_\{0\}^\{1\}M\\,t\\\|u\\\|\\cdot\\\|u\\\|\\,dt=\\frac\{M\}\{2\}\\\|u\\\|^\{2\}\.\(30\)∎

###### Lemma 3\(Residual contains only low\-order effects up toO⁡\(ρ2\)O\(\\rho^\{2\}\)\)\.

Assume additionally that‖∇LS​\(θ0\)‖\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|is small \(or treat it as a constant bias term\)\. Letu=τSu=\\tau\_\{S\}\. Then

‖∇LS​\(θ0\+u\)‖≤‖∇LS​\(θ0\)‖\+‖∇2LS​\(θ0\)‖op​‖u‖\+M2​‖u‖2≤‖∇LS​\(θ0\)‖\+H​ρ\+M2​ρ2\.\\begin\{gathered\}\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\+u\)\\\|\\\\ \\leq\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+\\\|\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\)\\\|\_\{\\mathrm\{op\}\}\\\|u\\\|\+\\frac\{M\}\{2\}\\\|u\\\|^\{2\}\\\\ \\leq\\\|\\nabla L\_\{S\}\(\\theta\_\{0\}\)\\\|\+H\\rho\+\\frac\{M\}\{2\}\\rho^\{2\}\.\\end\{gathered\}\(31\)Consequently, by Lemma[1](https://arxiv.org/html/2608.11499#Thmlemma1),

‖\(∫01∇2LS​\(θ0\+τS\+t​ΔS⋆\)​𝑑t\)−1​RS‖≤M2​μ​ρ2\.\\bigl\\\|\\big\(\\int\_\{0\}^\{1\}\\nabla^\{2\}L\_\{S\}\(\\theta\_\{0\}\+\\tau\_\{S\}\+t\\Delta^\{\\star\}\_\{S\}\)\\,dt\\big\)^\{\-1\}R\_\{S\}\\bigr\\\|\\leq\\frac\{M\}\{2\\mu\}\\rho^\{2\}\.\(32\)

###### Proof\.

Apply Lemma[2](https://arxiv.org/html/2608.11499#Thmlemma2)withu=τSu=\\tau\_\{S\}and‖u‖≤ρ\\\|u\\\|\\leq\\rho\. Then combine with Lemma[1](https://arxiv.org/html/2608.11499#Thmlemma1)\. ∎

##### Interpretation\.

The above shows \(conservatively\) that when task vectors are small, and the loss is smooth, the needed correction is*dominated by low\-order Taylor terms*\. Any interaction effects that require higher\-order derivatives become rapidly smaller withρ\\rho\.

#### Proof of Theorem 2: Why a hypernetwork conditioned onzSz\_\{S\}can generalize

HyperFix predictsΔShyper=hϕ​\(zS\)\\Delta^\{\\mathrm\{hyper\}\}\_\{S\}=h\_\{\\phi\}\(z\_\{S\}\)withzS=1\|S\|​∑i∈Sziz\_\{S\}=\\frac\{1\}\{\|S\|\}\\sum\_\{i\\in S\}z\_\{i\}\. The key property is thatzSz\_\{S\}is*size\-invariant*and*averages*per\-task interaction descriptors\.

###### Lemma 4\(Stability of the set embedding under increasing subset size\)\.

Assume the vectorsziz\_\{i\}have bounded second moment under the \(empirical\) task distribution:𝔼​‖zi−𝔼​z‖2≤σz2\\mathbb\{E\}\\\|z\_\{i\}\-\\mathbb\{E\}z\\\|^\{2\}\\leq\\sigma\_\{z\}^\{2\}\. If a subsetSSof sizemmis sampled by drawing tasks approximately i\.i\.d\. from this distribution, then

𝔼​‖zS−𝔼​z‖≤σzm\.\\mathbb\{E\}\\\|z\_\{S\}\-\\mathbb\{E\}z\\\|\\leq\\frac\{\\sigma\_\{z\}\}\{\\sqrt\{m\}\}\.\(33\)

###### Proof\.

SincezSz\_\{S\}is the average ofmmsamples, its variance scales as1/m1/m:𝔼​‖zS−𝔼​z‖2≤σz2/m\\mathbb\{E\}\\\|z\_\{S\}\-\\mathbb\{E\}z\\\|^\{2\}\\leq\\sigma\_\{z\}^\{2\}/m\. Jensen’s inequality gives𝔼​‖zS−𝔼​z‖≤𝔼​‖zS−𝔼​z‖2≤σz/m\\mathbb\{E\}\\\|z\_\{S\}\-\\mathbb\{E\}z\\\|\\leq\\sqrt\{\\mathbb\{E\}\\\|z\_\{S\}\-\\mathbb\{E\}z\\\|^\{2\}\}\\leq\\sigma\_\{z\}/\\sqrt\{m\}\. ∎

###### Lemma 5\(Why larger subsets are not harder in embedding space\)\.

Under Assumption A4, ifggisLgL\_\{g\}\-Lipschitz, then

𝔼​‖ΔS⋆−g⁡\(𝔼​z\)‖≤ε\+Lg​σzm\.\\mathbb\{E\}\\\|\\Delta^\{\\star\}\_\{S\}\-g\(\\mathbb\{E\}z\)\\\|\\leq\\varepsilon\+\\frac\{L\_\{g\}\\sigma\_\{z\}\}\{\\sqrt\{m\}\}\.\(34\)

###### Proof\.

By the triangle inequality and Assumption A4,

‖ΔS⋆−g⁡\(𝔼​z\)‖≤‖g⁡\(zS\)−g⁡\(𝔼​z\)‖\+‖η⁡\(S\)‖≤Lg​‖zS−𝔼​z‖\+ε\.\\begin\{gathered\}\\\|\\Delta^\{\\star\}\_\{S\}\-g\(\\mathbb\{E\}z\)\\\|\\\\\[1\.0pt\] \{\}\\leq\\\|g\(z\_\{S\}\)\-g\(\\mathbb\{E\}z\)\\\|\+\\\|\\eta\(S\)\\\|\\\\\[1\.0pt\] \{\}\\leq L\_\{g\}\\\|z\_\{S\}\-\\mathbb\{E\}z\\\|\+\\varepsilon\.\\end\{gathered\}\(35\)Taking expectations and applying Lemma[4](https://arxiv.org/html/2608.11499#Thmlemma4)yields the result\. ∎