Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

# Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
Source: [https://arxiv.org/html/2608.13596](https://arxiv.org/html/2608.13596)
###### Abstract

Heterogeneous model fusion combines models that differ in tasks, initializations, architectures, or scales\. We study an underexplored cross\-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch\. We ask whether useful capabilities can be transferred without explicit neuron\-wise semantic alignment\. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can improve the recipient, we propose Activation\-Prune\-Merge \(APM\), an activation\-guided framework for cross\-scale fusion\. APM constructs task\-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient\. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation\. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55\.5% to 60\.6% over the original 3B recipient\. RTE accuracy increases from 64\.3% to 82\.3%, QNLI from 52\.3% to 65\.7%, and BoolQ from 70\.8% to 79\.2%\. Analyses of injection ratios and sequential multi\-stage fusion further show that activation\-guided extraction improves the quality of the transferable donor slice while preserving the small\-ratio fusion regime\. These results show that cross\-scale heterogeneous fusion succeeds without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected\.

![Refer to caption](https://arxiv.org/html/2608.13596v1/Figures_im_vs_apm_fusion_comparison.png)Figure 1:Intersection\-Merge \(IM\) versus activation\-guided cross\-scale fusion\.## Introduction

Model merging combines the capabilities of multiple trained models within a single model and has proved effective for consolidating fine\-tuned checkpoints, composing task\-specific skills, and building multi\-task or domain\-specialized models\(Wortsman et al\.[2022](https://arxiv.org/html/2608.13596#bib.bib47); Ilharco et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib23); Yadav et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib48); Yu et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib51)\)\.

Classical homogeneous merging methods combine checkpoints or task vectors through parameter averaging, vector arithmetic, conflict resolution, or sparsification\(Wortsman et al\.[2022](https://arxiv.org/html/2608.13596#bib.bib47); Matena and Raffel[2022](https://arxiv.org/html/2608.13596#bib.bib29); Ilharco et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib23); Yadav et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib48); Yu et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib51)\)\. However, they require the models to share the same architecture and compatible parameterization, which substantially restricts the set of models that can be merged directly\. Approaches to heterogeneous transfer include knowledge distillation and adapter\- or projection\-based transformations, while alignment\-based methods establish feature or semantic correspondences before fusion\(Hinton, Vinyals, and Dean[2015](https://arxiv.org/html/2608.13596#bib.bib21); Pfeiffer et al\.[2021](https://arxiv.org/html/2608.13596#bib.bib31); Gu et al\.[2025](https://arxiv.org/html/2608.13596#bib.bib15); Stoica et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib37)\)\. Yet many of these methods require additional training, learned transformations, optimization, or explicit cross\-model alignment, making heterogeneous fusion considerably more involved\.

Prior work introduced Intersection\-Merge \(IM\), which directly truncates a large model to a target architecture by retaining the leading parameter indices and injects the resulting donor slice into a smaller model with a tiny mixing weight\(Fan et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib12)\)\. Large language models at different scales exhibit a surprising form of transferability: a stronger donor can improve a smaller recipient without neuron\-level semantic alignment\. Under this view, IM supplies a coarsely selected donor slice\. Building on this baseline, we ask a sharper question: can we extract a more task\-relevant donor slice so that the same micro\-injection budget produces larger gains?

We propose Activation\-Prune\-Merge \(APM\), a training\-free framework that extracts task\-relevant structure from a strong donor without explicit neuron\-wise semantic alignment \(Figure[1](https://arxiv.org/html/2608.13596#S0.F1)\)\. Given task data, APM first runs the donor model and collects activation statistics over a specified token range\. It then uses the resulting activation maps to select salient layers, hidden dimensions, attention heads, and MLP neurons\. These components are pruned and arranged into the exact architecture of the recipient, after which the resulting donor slice is injected into the original recipient with a micro interpolation coefficient\. APM concentrates the donor contribution before fusion while preserving the recipient as the dominant component of the merged model\.

Experiments across reasoning, mathematics, code generation, instruction following, and classification tasks show that APM improves both the original 3B recipient and IM\. The gains are especially pronounced on natural language inference and reasoning benchmarks\. At small injection ratios, cross\-scale heterogeneous fusion behaves less like neuron\-wise transplantation, which would require strict semantic alignment, and more like adding a concentrated donor extract to the recipient\. Additional analyses of injection ratios and sequential multi\-stage fusion indicate that activation\-guided extraction remains effective under small merge coefficients and that an APM\-fused model can continue to benefit from a subsequent donor injection\. Together, these findings support concentration transfer as a useful perspective on cross\-scale heterogeneous fusion\.

Our contributions are threefold:

1. 1\.APM framework\.We introduce APM, an activation\-guided heterogeneous fusion pipeline that combines donor activation mapping, target\-shape pruning, and micro\-injection into a smaller recipient model\. Under the same micro\-injection rule, activation\-guided pruning improves cross\-scale transfer beyond IM, indicating that APM extracts a more effective and task\-relevant donor slice\.
2. 2\.Transfer view\.Our results support a concentration\-transfer view: small\-ratio heterogeneous fusion depends on concentrated task\-relevant donor information rather than strict neuron\-wise semantic correspondence\. This explains why activation\-guided selection can outperform coarse truncation under the same micro\-injection budget\.
3. 3\.Broad evaluation\.We evaluate APM across benchmark families, injection ratios, and sequential fusion settings, covering mathematical reasoning, code generation, instruction following, language understanding, commonsense knowledge, and multi\-task reasoning\. The results show gains beyond a single benchmark or fusion configuration, including under repeated small\-ratio injection\.

## Related Work

### Homogeneous Model Merging

Homogeneous model merging combines parameters or task updates from models with a shared architecture\(Lu et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib28); Song and Zheng[2026](https://arxiv.org/html/2608.13596#bib.bib35); Yang et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib50)\), supported geometrically by low\-loss paths and linear mode connectivity between compatible solutions\(Garipov et al\.[2018](https://arxiv.org/html/2608.13596#bib.bib14); Frankle et al\.[2020](https://arxiv.org/html/2608.13596#bib.bib13)\)\. Checkpoint averaging and Fisher\-weighted merging combine parameters directly, whereas Task Arithmetic, TIES\-Merging, DARE, Model Breadcrumbs, and task localization operate on task vectors through arithmetic, trimming, sign resolution, sparsification, or selective updates\(Wortsman et al\.[2022](https://arxiv.org/html/2608.13596#bib.bib47); Matena and Raffel[2022](https://arxiv.org/html/2608.13596#bib.bib29); Ilharco et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib23); Yadav et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib48); Yu et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib51); Davari and Belilovsky[2024](https://arxiv.org/html/2608.13596#bib.bib9); He et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib17)\)\. Complementary work studies dataless knowledge fusion, task\-subspace matching, and task\-information localization\(Jin et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib24); Tam, Bansal, and Raffel[2024](https://arxiv.org/html/2608.13596#bib.bib40); Wang et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib44)\), while benchmarks and scaling analyses examine specialized LLM merging, model selection, and performance as merge count or model scale grows\(He et al\.[2025](https://arxiv.org/html/2608.13596#bib.bib18); Yadav et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib49); Wang et al\.[2025](https://arxiv.org/html/2608.13596#bib.bib45); Hitit, Girrbach, and Akata[2026](https://arxiv.org/html/2608.13596#bib.bib22)\)\. Training\-free alignment methods such as Git Re\-Basin and REPAIR establish parameter correspondences or correct activation statistics before merging structurally compatible models\(Ainsworth, Hayase, and Srinivasa[2023](https://arxiv.org/html/2608.13596#bib.bib1); Jordan et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib25)\)\.

### Heterogeneous and Cross\-Scale Model Fusion

Heterogeneous fusion spans models with different architectures or scales\(Chen et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib4); Soro et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib36)\); HM3 treats label\-space heterogeneity by merging same\-architecture classifiers with distinct outputs\(Hackmann[2024](https://arxiv.org/html/2608.13596#bib.bib16)\)\. Knowledge distillation transfers behavior by training a recipient on a donor’s output distribution without combining mismatched parameters\(Hinton, Vinyals, and Dean[2015](https://arxiv.org/html/2608.13596#bib.bib21)\), whereas adapter\-based methods learn task\-specific intermediates and compose them through an additional fusion mechanism\(Pfeiffer et al\.[2021](https://arxiv.org/html/2608.13596#bib.bib31)\)\. Alignment\-based methods establish semantic, feature, or structural correspondences before fusion\(Gu et al\.[2025](https://arxiv.org/html/2608.13596#bib.bib15)\)\. ZipIt\! extends training\-free fusion across tasks by matching and merging intermediate features while accommodating limited architectural differences\(Stoica et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib37)\)\. Other methods package knowledge into modular skills, fuse chat\-model capabilities, or optimize merge coefficients for heterogeneous multimodal models\(Du et al\.[2025a](https://arxiv.org/html/2608.13596#bib.bib10); Wan et al\.[2024a](https://arxiv.org/html/2608.13596#bib.bib41),[b](https://arxiv.org/html/2608.13596#bib.bib42); Du et al\.[2025b](https://arxiv.org/html/2608.13596#bib.bib11)\)\. Many such approaches require training, optimization, learned intermediates, or explicit correspondence; APM instead uses activations to rank donor\-internal components, without training or cross\-model feature or neuron\-wise semantic alignment\.

### Intersection\-Merge

Fan et al\. \([2026](https://arxiv.org/html/2608.13596#bib.bib12)\)proposed Intersection\-Merge \(IM\), which retains leading donor layers and parameter indices until the donor matches the recipient shape, then applies a very small linear interpolation coefficient\. IM showed that a target\-shaped donor slice can transfer capability without training or explicit semantic alignment\. Its static front\-aligned extraction, however, is not conditioned on the evaluated task\. APM retains the same cross\-scale injection regime while replacing this rule with task\-conditioned donor selection\.

### Activation\-Guided Structured Pruning

Activation\-based importance estimation is widely used for structured pruning\. Minitron scores layers, heads, hidden dimensions, and MLP neurons from calibration activations, while SliceGPT and Wanda provide complementary representation\- or activation\-aware pruning strategies\(Muralidharan et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib30); Ashkboos et al\.[2024](https://arxiv.org/html/2608.13596#bib.bib2); Sun et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib38)\)\. These methods optimize a standalone compressed model\. APM instead profiles a task\-conditioned donor, extracts a recipient\-shaped slice, and micro\-injects it into an independently trained recipient\. Its novelty lies in using activation saliency for transfer\-oriented donor selection within training\-free cross\-scale fusion\.

## Method

![Refer to caption](https://arxiv.org/html/2608.13596v1/Figures_activation_profiling_and_layer_selection.png)Figure 2:Overview of task\-conditioned activation profiling and layer selection in APM\. Task examples are divided into generative and multiple\-choice settings\. The activation strategy specifies which prompt, generation, or answer\-choice tokens contribute to the activation statistics\. The resulting scores support three target\-depth pruning strategies: first\-nn, uniform, and activation\-guided layer selection\.### Preliminaries

#### Problem Formulation

LetMs=\(𝒜s,θs\)M\_\{s\}=\(\\mathcal\{A\}\_\{s\},\\theta\_\{s\}\)be a small recipient model andMl=\(𝒜l,θl\)M\_\{l\}=\(\\mathcal\{A\}\_\{l\},\\theta\_\{l\}\)a larger donor from the same model family\. Their depths, widths, and attention structures differ\. Given task data𝒟\\mathcal\{D\}, we seek a fused model that retains the architecture and inference cost ofMsM\_\{s\}while absorbing task\-relevant donor components\. Formally, we seek a training\-free extraction operator𝒫𝒟:θl↦θ~l\\mathcal\{P\}\_\{\\mathcal\{D\}\}:\\theta\_\{l\}\\mapsto\\tilde\{\\theta\}\_\{l\}such thatθ~l\\tilde\{\\theta\}\_\{l\}has architecture𝒜s\\mathcal\{A\}\_\{s\}and can be fused directly withθs\\theta\_\{s\}\. APM instantiates this operator through task\-conditioned activation profiling and target\-shape pruning, followed by micro\-injection\. Figure[2](https://arxiv.org/html/2608.13596#Sx3.F2)illustrates the extraction stage\.

#### Intersection\-Merge

Intersection\-Merge \(IM\) is the direct starting point of APM\(Fan et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib12)\)\. Letℐfront​\(𝒜l,𝒜s\)\\mathcal\{I\}^\{\\mathrm\{front\}\}\(\\mathcal\{A\}\_\{l\},\\mathcal\{A\}\_\{s\}\)contain the leading donor layers and the leading indices of every tensor dimension required by the recipient\. IM constructs a target\-shaped donor by

θlIM=𝒫front​\(θl;ℐfront​\(𝒜l,𝒜s\)\),shape⁡\(θlIM\)=shape⁡\(θs\)\.\\begin\{gathered\}\\theta\_\{l\}^\{\\mathrm\{IM\}\}=\\mathcal\{P\}\_\{\\mathrm\{front\}\}\\\!\\left\(\\theta\_\{l\};\\mathcal\{I\}^\{\\mathrm\{front\}\}\(\\mathcal\{A\}\_\{l\},\\mathcal\{A\}\_\{s\}\)\\right\),\\\\ \\operatorname\{shape\}\(\\theta\_\{l\}^\{\\mathrm\{IM\}\}\)=\\operatorname\{shape\}\(\\theta\_\{s\}\)\.\\end\{gathered\}\(1\)and injects it with a small interpolation coefficient:

θIM=\(1−μ\)​θs\+μ​θlIM,0<μ≪1\.\\theta\_\{\\mathrm\{IM\}\}=\(1\-\\mu\)\\theta\_\{s\}\+\\mu\\theta\_\{l\}^\{\\mathrm\{IM\}\},\\qquad 0<\\mu\\ll 1\.\(2\)Thus, IM resolves cross\-scale tensor mismatch without training or semantic alignment, but its donor selection is fixed and task\-agnostic\. APM retains Equation[2](https://arxiv.org/html/2608.13596#Sx3.E2)as the controlled fusion rule and replaces the fixed front\-aligned operator𝒫front\\mathcal\{P\}\_\{\\mathrm\{front\}\}with the task\-conditioned operator𝒫𝒟\\mathcal\{P\}\_\{\\mathcal\{D\}\}\. This isolates the effect of selecting a more informative donor slice\.

### Task\-Conditioned Activation Mapping

We run the donor on𝒟\\mathcal\{D\}and accumulate activation magnitudes for four structural units: transformer layers, hidden channels, attention heads, and MLP neurons\. Only task\-relevant token positions contribute to these statistics\. For generative tasks, the profiling scope can cover prompt tokens, generation tokens, or both; for multiple\-choice tasks, it can cover prompt tokens, answer choices, or both\. LetΩ​\(x\)\\Omega\(x\)be the profiled token positions for inputxx, and letZ=∑x∈𝒟\|Ω​\(x\)\|Z=\\sum\_\{x\\in\\mathcal\{D\}\}\|\\Omega\(x\)\|\. For structural unituuwith activationgu​\(x,t\)g\_\{u\}\(x,t\), its importance score is

su=1Z​∑x∈𝒟∑t∈Ω​\(x\)ϕu​\(gu​\(x,t\)\),s\_\{u\}=\\frac\{1\}\{Z\}\\sum\_\{x\\in\\mathcal\{D\}\}\\sum\_\{t\\in\\Omega\(x\)\}\\phi\_\{u\}\\\!\\left\(g\_\{u\}\(x,t\)\\right\),\(3\)whereϕu​\(⋅\)\\phi\_\{u\}\(\\cdot\)is theℓ2\\ell\_\{2\}norm for layer and attention\-head vectors and the absolute value for hidden\-channel and MLP\-neuron scalars\. The resulting scores form a task\-conditioned concentration map used to rank donor components\. They are accumulated online without storing full activation tensors; the supplementary material gives the unit\-specific expansions of Equation[3](https://arxiv.org/html/2608.13596#Sx3.E3)\.

### Activation\-Guided Target\-Shape Pruning

APM converts the activation map into a dense donor slice with exactly the recipient’s depth, residual width, attention configuration, and MLP width\. Let these target cardinalities beLsL\_\{s\},dsd\_\{s\},\(Hs,HsKV\)\(H\_\{s\},H\_\{s\}^\{\\mathrm\{KV\}\}\), andmsm\_\{s\}, respectively\. For depth, the implementation supports three controlled strategies:

ℐL=\{\{0,…,Ls−1\},first,Uniform⁡\(Ll,Ls\),uniform,TopK⁡\(Slayer,Ls\),activation\.\\mathcal\{I\}\_\{L\}=\\begin\{cases\}\\\{0,\\ldots,L\_\{s\}\-1\\\},&\\text\{first\},\\\\ \\operatorname\{Uniform\}\(L\_\{l\},L\_\{s\}\),&\\text\{uniform\},\\\\ \\operatorname\{TopK\}\(S\_\{\\mathrm\{layer\}\},L\_\{s\}\),&\\text\{activation\}\.\\end\{cases\}\(4\)All selected layer indices are sorted to preserve donor\-depth order\. The other dimensions are selected by activation score:

ℐd=TopK⁡\(Shidden,ds\),ℐm\(ℓ\)=TopK⁡\(Smlp\(ℓ\),ms\)\.\\mathcal\{I\}\_\{d\}=\\operatorname\{TopK\}\(S\_\{\\mathrm\{hidden\}\},d\_\{s\}\),\\qquad\\mathcal\{I\}\_\{m\}^\{\(\\ell\)\}=\\operatorname\{TopK\}\(S\_\{\\mathrm\{mlp\}\}^\{\(\\ell\)\},m\_\{s\}\)\.\(5\)The hidden\-channel set is global, whereas MLP neurons are selected separately for every retained layerℓ\\ell\. For grouped\-query attention, query and key–value \(KV\) heads must be treated separately\. We select

ℐQ\(ℓ\)=TopK⁡\(Shead\(ℓ\),Hs\),SKV,k\(ℓ\)=∑h∈𝒢​\(k\)Shead,h\(ℓ\),ℐKV\(ℓ\)=TopK⁡\(SKV\(ℓ\),HsKV\),\\begin\{gathered\}\\mathcal\{I\}\_\{Q\}^\{\(\\ell\)\}=\\operatorname\{TopK\}\(S\_\{\\mathrm\{head\}\}^\{\(\\ell\)\},H\_\{s\}\),\\\\ S\_\{\\mathrm\{KV\},k\}^\{\(\\ell\)\}=\\sum\_\{h\\in\\mathcal\{G\}\(k\)\}S\_\{\\mathrm\{head\},h\}^\{\(\\ell\)\},\\\\ \\mathcal\{I\}\_\{\\mathrm\{KV\}\}^\{\(\\ell\)\}=\\operatorname\{TopK\}\(S\_\{\\mathrm\{KV\}\}^\{\(\\ell\)\},H\_\{s\}^\{\\mathrm\{KV\}\}\),\\end\{gathered\}\(6\)where𝒢​\(k\)\\mathcal\{G\}\(k\)is the group of donor query heads sharing KV headkk\. This mirrors the code’s grouped\-query structure rather than applying one head index set to all attention projections\.

The index collection𝒮𝒟=\{ℐL,ℐd,ℐQ\(ℓ\),ℐKV\(ℓ\),ℐm\(ℓ\)\}ℓ∈ℐL\\mathcal\{S\}\_\{\\mathcal\{D\}\}=\\\{\\mathcal\{I\}\_\{L\},\\mathcal\{I\}\_\{d\},\\mathcal\{I\}\_\{Q\}^\{\(\\ell\)\},\\mathcal\{I\}\_\{\\mathrm\{KV\}\}^\{\(\\ell\)\},\\mathcal\{I\}\_\{m\}^\{\(\\ell\)\}\\\}\_\{\\ell\\in\\mathcal\{I\}\_\{L\}\}defines the task\-conditioned extraction operator:

θlAPM=𝒫𝒟​\(θl;𝒮𝒟,𝒜s\)\.\\theta\_\{l\}^\{\\mathrm\{APM\}\}=\\mathcal\{P\}\_\{\\mathcal\{D\}\}\(\\theta\_\{l\};\\mathcal\{S\}\_\{\\mathcal\{D\}\},\\mathcal\{A\}\_\{s\}\)\.\(7\)Each index set is applied consistently to coupled tensors: hidden indices to embeddings, normalization parameters, and projection input/output axes; query heads to Q and output projections; KV heads to K/V projections; and MLP indices to gate, up, and down projections\. The result is a dense, recipient\-shaped donor slice that preserves tensor compatibility and is constructed without optimization\.

### Micro\-Injection Merge

LetθlAPM\\theta\_\{l\}^\{\\mathrm\{APM\}\}denote the extracted donor slice\. Because it has the same tensor structure asθs\\theta\_\{s\}, APM uses a single linear micro\-injection rule:

θAPM=\(1−μ\)​θs\+μ​θlAPM,μ∈\(0,0\.10\)\.\\theta\_\{\\mathrm\{APM\}\}=\(1\-\\mu\)\\theta\_\{s\}\+\\mu\\theta\_\{l\}^\{\\mathrm\{APM\}\},\\mu\\in\(0,0\.10\)\.\(8\)The recipient remains the dominant parameter source and retains its original inference architecture\. The small coefficient keeps the update in a local neighborhood of the recipient while allowing the selected donor slice to bias task\-relevant computations\. Because the extraction step has already matched every tensor shape, the merge does not require adapters, projections, or further optimization\. Multiple extracted donors can be composed by assigning each a small coefficient; the corresponding formula is given in the supplementary material\. Algorithm[1](https://arxiv.org/html/2608.13596#alg1)summarizes the complete framework\.

Algorithm 1Activation\-Prune\-Merge0:Donor

MlM\_\{l\}, recipient

MsM\_\{s\}, activation dataset

𝒟\\mathcal\{D\}, profiling token rule

Ω\\Omega, injection ratio

μ\\mu
0:Fused model

MAPMM\_\{\\mathrm\{APM\}\}
1:Run

MlM\_\{l\}on

𝒟\\mathcal\{D\}and collect activation statistics over

Ω\\Omega
2:Compute layer, hidden\-channel, attention\-head, and MLP\-neuron scores

3:Select depth by the specified strategy and activation\-ranked hidden, Q/KV\-head, and MLP indices to match

𝒜s\\mathcal\{A\}\_\{s\}
4:

θlAPM←𝒫𝒟​\(θl;𝒮𝒟,𝒜s\)\\theta\_\{l\}^\{\\mathrm\{APM\}\}\\leftarrow\\mathcal\{P\}\_\{\\mathcal\{D\}\}\(\\theta\_\{l\};\\mathcal\{S\}\_\{\\mathcal\{D\}\},\\mathcal\{A\}\_\{s\}\)
5:

θAPM←\(1−μ\)​θs\+μ​θlAPM\\theta\_\{\\mathrm\{APM\}\}\\leftarrow\(1\-\\mu\)\\theta\_\{s\}\+\\mu\\theta\_\{l\}^\{\\mathrm\{APM\}\}
6:return

MAPM=\(𝒜s,θAPM\)M\_\{\\mathrm\{APM\}\}=\(\\mathcal\{A\}\_\{s\},\\theta\_\{\\mathrm\{APM\}\}\)

## Experiments

Table 1:Benchmark families used in our experiments\.Table 2:Model roles in our experiments\. Qwen2\.5\-32B is the donor used for the primary benchmark results; Qwen2\.5\-14B is used in the sequential multi\-stage fusion analysis\.Table 3:Main results on 16 benchmarks using Qwen2\.5\-3B as the recipient and Qwen2\.5\-32B as the donor\. APM denotes Activation\-Prune\-Merge, and IM denotes Intersection\-Merge, the baseline introduced byFan et al\. \([2026](https://arxiv.org/html/2608.13596#bib.bib12)\)\. Scores are percentages, and differences are reported in percentage points \(pp\)\. Higher is better for all metrics\.### Benchmark Datasets

We evaluate APM on 16 benchmarks across six capability families \(Table[1](https://arxiv.org/html/2608.13596#Sx4.T1)\)\. GSM8K and the MATH dataset cover mathematical reasoning, while the MATH\-500 evaluation subset follows the representative MATH\-test subset introduced for process\-supervised verification\(Cobbe et al\.[2021](https://arxiv.org/html/2608.13596#bib.bib7); Hendrycks et al\.[2021b](https://arxiv.org/html/2608.13596#bib.bib20); Lightman et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib26)\); HumanEvalPlus and MBPPPlus assess code generation\(Liu et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib27)\); and IFEval measures instruction following\(Zhou et al\.[2023](https://arxiv.org/html/2608.13596#bib.bib53)\)\. The QA and NLU suite contains MNLI, RTE, QNLI, PIQA, WinoGrande, COPA, and BoolQ\(Williams, Nangia, and Bowman[2018](https://arxiv.org/html/2608.13596#bib.bib46); Dagan, Glickman, and Magnini[2006](https://arxiv.org/html/2608.13596#bib.bib8); Wang et al\.[2018](https://arxiv.org/html/2608.13596#bib.bib43); Bisk et al\.[2020](https://arxiv.org/html/2608.13596#bib.bib3); Sakaguchi et al\.[2020](https://arxiv.org/html/2608.13596#bib.bib34); Roemmele, Bejan, and Gordon[2011](https://arxiv.org/html/2608.13596#bib.bib33); Clark et al\.[2019](https://arxiv.org/html/2608.13596#bib.bib5)\)\. We further use ARC, HellaSwag, and MMLU for commonsense and broad\-domain knowledge, and BBH for challenging multi\-task reasoning\(Clark et al\.[2018](https://arxiv.org/html/2608.13596#bib.bib6); Zellers et al\.[2019](https://arxiv.org/html/2608.13596#bib.bib52); Hendrycks et al\.[2021a](https://arxiv.org/html/2608.13596#bib.bib19); Suzgun et al\.[2022](https://arxiv.org/html/2608.13596#bib.bib39)\)\.

### Models

We use Qwen2\.5\-3B as the recipient and Qwen2\.5\-14B and Qwen2\.5\-32B as cross\-scale donors, all from the Qwen2\.5 model family described in the technical report\(Qwen Team[2024](https://arxiv.org/html/2608.13596#bib.bib32)\)\. Table[2](https://arxiv.org/html/2608.13596#Sx4.T2)summarizes these roles\. We consistently use the terms*recipient*and*donor*to denote model roles\. APM performs direct, training\-free parameter transfer\. Each donor is activation\-profiled, pruned to the Qwen2\.5\-3B architecture, and then micro\-injected, allowing us to test transfer and composition across donor scales\.

### Experimental Setup

As summarized in Figure[2](https://arxiv.org/html/2608.13596#Sx3.F2), we construct a task\-specific activation set from each benchmark and profile the corresponding donor before extraction\. For generative tasks, we consider three profiling scopes:*prompt\-only*,*generation\-only*, and*prompt–generation*\. Generated continuations are capped at a task\-appropriate length\. For multiple\-choice tasks, we analogously use*prompt\-only*,*choice\-only*, or*prompt–choice*profiling\. These alternatives control which token positions contribute to the activation statistics while preserving the original model parameters during profiling\. The activation examples used for donor profiling are kept disjoint from the held\-out examples used for evaluation\.

After activation profiling, we compare three target\-depth layer\-selection rules\.*First\-nn*retains the firstnndonor layers,*uniform*selectsnnapproximately evenly spaced layers over the donor depth, and*activation\-guided*retains thennlayers with the highest activation scores while preserving their original order\. In every case,nnis fixed by the recipient depth\. The selected donor is subsequently reduced to the full recipient shape and merged using the same micro\-injection rule\.

### Baselines

We compare APM with the original recipient and IM\(Fan et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib12)\)\. IM performs deterministic front\-aligned extraction: it retains the firstLsL\_\{s\}donor layers and, for each weight tensor, the leading indices along every dimension required by the recipient shape\. The resulting recipient\-shaped donor slice is injected into the recipient with the same mixing coefficient used by APM\. This controlled comparison isolates the effect of activation\-guided donor selection from the linear merge rule itself\.

##### Overall results\.

Table[3](https://arxiv.org/html/2608.13596#Sx4.T3)provides a complete overview across all 16 benchmarks\. APM increases the average score of the original Qwen2\.5\-3B recipient from 55\.5% to 60\.6%\. Under the same cross\-scale fusion setting, APM exceeds or matches IM on every benchmark and raises its average from 57\.8% to 60\.6%\. The improvements span mathematics, code generation, instruction following, natural language understanding, and comprehensive reasoning, with particularly large gains on RTE \(\+18\.0\+18\.0pp over 3B and\+7\.9\+7\.9pp over IM\), QNLI \(\+13\.4\+13\.4and\+10\.4\+10\.4pp\), and BoolQ \(\+8\.4\+8\.4and\+6\.5\+6\.5pp\)\. Additional significance analysis in the supplementary material is consistent with these gains\. On the eight selected BBH subtasks, the task\-matched APM slices improve over IM on every task \(Table[4](https://arxiv.org/html/2608.13596#Sx5.T4)\(a\)\)\. The largest gains occur for T2 \(\+30\.0\+30\.0pp\), T1 \(\+17\.6\+17\.6pp\), and T8 \(\+12\.8\+12\.8pp\)\. The exact permutation tests in Table[4](https://arxiv.org/html/2608.13596#Sx5.T4)\(b\) further support a positive matched\-task advantage in both raw and standardized analyses\. These broad and consistent gains indicate that activation\-guided extraction produces a more effective donor slice than front\-aligned intersection under the same micro\-injection framework\.

## Analysis

\(a\) Cross\-task accuracy matrix \(b\) Exact permutation tests \(c\) Transfer to the complete BBH\-27 suite

Table 4:Fine\-grained BBH transfer and permutation analysis\. Panel \(a\) reports BBH\-8 cross\-task accuracies; Panel \(b\) reports exact tests of diagonal advantage; Panel \(c\) evaluates transfer of slices T1–T8 on the complete BBH\-27 suite\.\(a\) IM and APM across injection ratios \(b\) Recipient and donor reference scores

Table 5:Accuracy across the low\-to\-moderate injection region\. Panel \(a\) reports IM and task\-activated APM under the same injection ratio; bold entries are the best APM scores within the displayed interval\. Panel \(b\) gives the Qwen2\.5\-3B recipient and 32B donor references\. Scores are percentages\.![Refer to caption](https://arxiv.org/html/2608.13596v1/Figures_injection_ratio_sensitivity.png)Figure 3:Accuracy as a function of the injection ratio on four benchmarks\.\(a\) Task\-wise accuracy across fusion stages \(b\) Average gain at each fusion hop

Table 6:Sequential two\-stage fusion across seven QA and NLU benchmarks\. Panel \(a\) reports task\-wise accuracy, and Panel \(b\) summarizes the average gain at each fusion hop\. Stage 1 \(S1\) injects the 32B donor slice into the 3B recipient, and Stage 2 \(S2\) injects the 14B slice into the S1 model\. Each stage usesμ=0\.02\\mu=0\.02\. Accuracies are percentages, and gains are percentage points \(pp\)\.Table 7:Mean QNLI accuracy and normalized variance grouped by activation\-data selection strategy and activation\-set size\. Accuracy scores are percentages\.### Task\-Matched Transfer

We examine eight BBH subtasks with substantial transfer headroom: the Qwen2\.5\-3B student averages 27\.3%, whereas the Qwen2\.5\-32B teacher reaches 89\.6% \(Table[4](https://arxiv.org/html/2608.13596#Sx5.T4)\(a\)\)\. At the matched injection ratioμ=0\.02\\mu=0\.02, IM raises the average to 32\.1%, providing a controlled reference that isolates donor\-slice selection from the injected parameter mass\. All eight APM slices average 39\.9–42\.1%, exceeding both baselines\. Matched activation improves over IM on every subtask, by 0\.8–30\.0 pp\. Several off\-diagonal entries are also column maxima, indicating that activation\-guided selection captures donor components with cross\-task value rather than only task\-specific structure\.

This transfer persists across the complete BBH\-27 suite \(Table[4](https://arxiv.org/html/2608.13596#Sx5.T4)\(c\)\)\. Panel \(c\) reports average accuracies in percentages, gains over the 3B recipient and IM in percentage points, and the number of BBH tasks on which each APM model exceeds the corresponding baseline; T1–T8 follow the activation\-source mapping in Panel \(a\)\. The supplementary material reports the corresponding per\-task accuracies for all 27 BBH subtasks, together with the number of evaluation examples in each subtask\. The eight APM models average 48\.3–49\.9%, gaining 4\.7–6\.3 pp over the student and 3\.0–4\.6 pp over IM\. Each APM variant also outperforms the student on 17–21 of the 27 tasks and IM on 18–21 tasks\. Thus, the gains are not confined to the subtask used as the activation source: a donor slice extracted from one activation group can improve accuracy across many other BBH tasks\. This broad transfer indicates that activation\-guided pruning captures donor components with general cross\-task value, rather than only information specialized to the activation\-source task\.

Panel \(b\) reports raw accuracies and column\-normalized APM gains over IM; Diag\., Off\-diag\.,Δ\\Delta, andppdenote the matched mean, unmatched mean, their difference, and the one\-sided positive\-advantage probability\. Both the raw\-score and normalized\-gain tests show a significant matched\-task advantage, indicating that matched activation profiling usually yields the strongest donor slice, while several off\-diagonal maxima still show effective cross\-task transfer \(Table[4](https://arxiv.org/html/2608.13596#Sx5.T4)\(b\)\)\.

### Injection\-Ratio Sensitivity

Following the IM evaluation over ratios from0\.010\.01to0\.050\.05\(Fan et al\.[2026](https://arxiv.org/html/2608.13596#bib.bib12)\), we examine a finer range on BoolQ, MNLI, QNLI, and RTE to test robustness to the injected donor mass \(Table[5](https://arxiv.org/html/2608.13596#Sx5.T5)and Figure[3](https://arxiv.org/html/2608.13596#Sx5.F3)\)\. APM and IM show similar ratio\-dependent trends, but APM generally performs better throughout the effective region\(0,0\.10\)\(0,0\.10\)under the same interpolation rule\. Notably, APM reaches 82\.3% on RTE atμ=0\.05\\mu=0\.05, surpassing the 32B donor’s 80\.9%, and peaks at 73\.9% on QNLI atμ=0\.08\\mu=0\.08\. The optimal ratio varies by task, suggesting that task\-aware ratio selection can further improve activation\-guided extraction\. Results throughμ=0\.15\\mu=0\.15are reported in Supplementary Table 1\.

### Sequential Multi\-Stage Fusion

We test sequential fusion by injecting 32B and 14B donor slices into the 3B recipient in two stages, each atμ=0\.02\\mu=0\.02, using the same order and coefficients for IM \(Table[6](https://arxiv.org/html/2608.13596#Sx5.T6)\)\. APM raises the seven\-task average from 64\.3% to 68\.2% and then 69\.9%, whereas IM reaches 66\.4% and 66\.7%\. The second\-stage gain is therefore 1\.7 pp for APM versus 0\.3 pp for IM\. APM\-S2 achieves the highest scores on MNLI, RTE, QNLI, BoolQ, and COPA, while APM\-S1 leads on PIQA\. These results indicate that activation\-guided donor slices remain compatible with repeated small\-ratio fusion\.

### Sensitivity to Activation\-Data Selection

On QNLI atμ=0\.05\\mu=0\.05, donor\-correct subsets average 58\.5%, compared with 56\.9% for random selection across nine activation–pruning variants \(Table[7](https://arxiv.org/html/2608.13596#Sx5.T7)\)\. Thus, donor correctness is more informative than recipient difficulty alone, and recipient\-solved examples remain useful when the donor is also correct\. The 25\-example set performs best at 59\.6%, versus 58\.6% for 200 examples and 57\.3% for 1,000, so more activation data is not necessarily better\. Complete results are provided in the supplementary material\.

## Conclusion

We introduced Activation\-Prune\-Merge \(APM\), a training\-free extension of cross\-scale micro\-injection that replaces fixed donor truncation with task\-conditioned activation profiling and structured component selection\. Across diverse benchmark families, APM consistently strengthens the original recipient and improves over IM under the same interpolation rule\. Its fine\-grained BBH transfer, broad effectiveness across small injection ratios, and continued gains under sequential fusion further demonstrate the quality and composability of the extracted donor slices\.

The strength of APM lies in its simplicity\. By adding lightweight activation statistics and target\-shape component selection to the IM pipeline, APM obtains substantial improvements while preserving the recipient architecture, inference cost, and training\-free workflow\. The large gains produced by this lightweight extension demonstrate the effectiveness of activation\-guided donor slice selection for cross\-scale transfer\. More broadly, our results reveal substantial untapped potential for directly fusing heterogeneous models without explicit neuron\-wise semantic alignment, opening a practical path for strong models to transfer useful capabilities into compact recipients\.

Taken together, these results position APM as a lightweight complement to model scaling\. Its separation of target\-shape reduction and activation\-guided selection keeps the recipient architecture and micro\-injection rule unchanged, facilitating adaptation across benchmark families\.

## References

- Ainsworth, Hayase, and Srinivasa \(2023\)Ainsworth, S\. K\.; Hayase, J\.; and Srinivasa, S\. 2023\.Git Re\-Basin: Merging Models Modulo Permutation Symmetries\.In*International Conference on Learning Representations*\.
- Ashkboos et al\. \(2024\)Ashkboos, S\.; Croci, M\. L\.; do Nascimento, M\. G\.; Hoefler, T\.; and Hensman, J\. 2024\.SliceGPT: Compress Large Language Models by Deleting Rows and Columns\.arXiv:2401\.15024\.
- Bisk et al\. \(2020\)Bisk, Y\.; Zellers, R\.; Le Bras, R\.; Gao, J\.; and Choi, Y\. 2020\.PIQA: Reasoning about Physical Commonsense in Natural Language\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, 7432–7439\.
- Chen et al\. \(2026\)Chen, S\.; Zhou, J\.; Chen, Q\.; Wu, W\.; Li, X\.; Feng, Q\.; and He, L\. 2026\.Can Heterogeneous Language Models Be Fused?arXiv:2604\.01674\.
- Clark et al\. \(2019\)Clark, C\.; Lee, K\.; Chang, M\.\-W\.; Kwiatkowski, T\.; Collins, M\.; and Toutanova, K\. 2019\.BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 2924–2936\.
- Clark et al\. \(2018\)Clark, P\.; Cowhey, I\.; Etzioni, O\.; Khot, T\.; Sabharwal, A\.; Schoenick, C\.; and Tafjord, O\. 2018\.Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge\.arXiv:1803\.05457\.
- Cobbe et al\. \(2021\)Cobbe, K\.; Kosaraju, V\.; Bavarian, M\.; Chen, M\.; Jun, H\.; Kaiser, L\.; Plappert, M\.; Tworek, J\.; Hilton, J\.; Nakano, R\.; Hesse, C\.; and Schulman, J\. 2021\.Training Verifiers to Solve Math Word Problems\.arXiv:2110\.14168\.
- Dagan, Glickman, and Magnini \(2006\)Dagan, I\.; Glickman, O\.; and Magnini, B\. 2006\.The PASCAL Recognising Textual Entailment Challenge\.In*Machine Learning Challenges\. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment*, 177–190\. Springer\.
- Davari and Belilovsky \(2024\)Davari, M\.; and Belilovsky, E\. 2024\.Model Breadcrumbs: Scaling Multi\-Task Model Merging with Sparse Masks\.In*European Conference on Computer Vision*\.
- Du et al\. \(2025a\)Du, G\.; Li, Z\.; Zhou, X\.; Li, J\.; Shi, Z\.; Lin, W\.; Tang, H\.\-K\.; Li, X\.; Liu, F\.; Wang, W\.; Zhang, M\.; and Li, J\. 2025a\.Knowledge Fusion of Large Language Models via Modular SkillPacks\.arXiv:2505\.18502\.
- Du et al\. \(2025b\)Du, Y\.; Wang, X\.; Chen, C\.; Ye, J\.; Wang, Y\.; Li, P\.; Yan, M\.; Zhang, J\.; Huang, F\.; Sui, Z\.; Sun, M\.; and Liu, Y\. 2025b\.AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization\.arXiv:2503\.23733\.
- Fan et al\. \(2026\)Fan, J\.; Hou, Y\.; Chen, S\.; Zhang, A\.; Xie, H\.; and Lian, D\. 2026\.Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective\.arXiv:2607\.18026\.
- Frankle et al\. \(2020\)Frankle, J\.; Dziugaite, G\. K\.; Roy, D\. M\.; and Carbin, M\. 2020\.Linear Mode Connectivity and the Lottery Ticket Hypothesis\.In*Proceedings of the 37th International Conference on Machine Learning*, volume 119, 3259–3269\.
- Garipov et al\. \(2018\)Garipov, T\.; Izmailov, P\.; Podoprikhin, D\.; Vetrov, D\. P\.; and Wilson, A\. G\. 2018\.Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs\.In*Advances in Neural Information Processing Systems*, volume 31\.
- Gu et al\. \(2025\)Gu, J\.; Aleti, A\.; Chen, C\.; and Zhang, H\. 2025\.SeMe: Training\-Free Language Model Merging via Semantic Alignment\.arXiv:2505\.20144\.
- Hackmann \(2024\)Hackmann, S\. 2024\.HM3: Heterogeneous Multi\-Class Model Merging\.arXiv:2409\.19173\.
- He et al\. \(2024\)He, Y\.; Hu, Y\.; Lin, Y\.; Zhang, T\.; and Zhao, H\. 2024\.Localize\-and\-Stitch: Efficient Model Merging via Sparse Task Arithmetic\.*Transactions on Machine Learning Research*\.
- He et al\. \(2025\)He, Y\.; Zeng, S\.; Hu, Y\.; Yang, R\.; Zhang, T\.; and Zhao, H\. 2025\.MergeBench: A Benchmark for Merging Domain\-Specialized LLMs\.In*Advances in Neural Information Processing Systems*\.
- Hendrycks et al\. \(2021a\)Hendrycks, D\.; Burns, C\.; Basart, S\.; Zou, A\.; Mazeika, M\.; Song, D\.; and Steinhardt, J\. 2021a\.Measuring Massive Multitask Language Understanding\.In*International Conference on Learning Representations*\.
- Hendrycks et al\. \(2021b\)Hendrycks, D\.; Burns, C\.; Kadavath, S\.; Arora, A\.; Basart, S\.; Tang, E\.; Song, D\.; and Steinhardt, J\. 2021b\.Measuring Mathematical Problem Solving With the MATH Dataset\.In*Advances in Neural Information Processing Systems*\.
- Hinton, Vinyals, and Dean \(2015\)Hinton, G\.; Vinyals, O\.; and Dean, J\. 2015\.Distilling the Knowledge in a Neural Network\.arXiv:1503\.02531\.
- Hitit, Girrbach, and Akata \(2026\)Hitit, O\. K\.; Girrbach, L\.; and Akata, Z\. 2026\.A Systematic Study of In\-the\-Wild Model Merging for Large Language Models\.*Transactions on Machine Learning Research*\.
- Ilharco et al\. \(2023\)Ilharco, G\.; Ribeiro, M\. T\.; Wortsman, M\.; Gururangan, S\.; Schmidt, L\.; Hajishirzi, H\.; and Farhadi, A\. 2023\.Editing Models with Task Arithmetic\.In*International Conference on Learning Representations*\.
- Jin et al\. \(2023\)Jin, X\.; Ren, X\.; Preotiuc\-Pietro, D\.; and Cheng, P\. 2023\.Dataless Knowledge Fusion by Merging Weights of Language Models\.In*International Conference on Learning Representations*\.
- Jordan et al\. \(2023\)Jordan, K\.; Sedghi, H\.; Saukh, O\.; Entezari, R\.; and Neyshabur, B\. 2023\.REPAIR: Renormalizing Permuted Activations for Interpolation Repair\.In*International Conference on Learning Representations*\.
- Lightman et al\. \(2023\)Lightman, H\.; Kosaraju, V\.; Burda, Y\.; Edwards, H\.; Baker, B\.; Lee, T\.; Leike, J\.; Schulman, J\.; Sutskever, I\.; and Cobbe, K\. 2023\.Let’s Verify Step by Step\.arXiv:2305\.20050\.
- Liu et al\. \(2023\)Liu, J\.; Xia, C\. S\.; Wang, Y\.; and Zhang, L\. 2023\.Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation\.arXiv:2305\.01210\.
- Lu et al\. \(2024\)Lu, J\.; Pang, Z\.; Xiao, M\.; Zhu, Y\.; Xia, R\.; and Zhang, J\. 2024\.Merge, Ensemble, and Cooperate\! A Survey on Collaborative Strategies in the Era of Large Language Models\.arXiv:2407\.06089\.
- Matena and Raffel \(2022\)Matena, M\.; and Raffel, C\. 2022\.Merging Models with Fisher\-Weighted Averaging\.In*Advances in Neural Information Processing Systems*, volume 35\.
- Muralidharan et al\. \(2024\)Muralidharan, S\.; Sreenivas, S\. T\.; Joshi, R\.; Chochowski, M\.; Patwary, M\.; Shoeybi, M\.; Catanzaro, B\.; Kautz, J\.; and Molchanov, P\. 2024\.Compact Language Models via Pruning and Knowledge Distillation\.arXiv:2407\.14679\.
- Pfeiffer et al\. \(2021\)Pfeiffer, J\.; Kamath, A\.; Rücklé, A\.; Cho, K\.; and Gurevych, I\. 2021\.AdapterFusion: Non\-Destructive Task Composition for Transfer Learning\.In*Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics*, 487–503\.
- Qwen Team \(2024\)Qwen Team\. 2024\.Qwen2\.5 Technical Report\.Technical report, arXiv\.
- Roemmele, Bejan, and Gordon \(2011\)Roemmele, M\.; Bejan, C\. A\.; and Gordon, A\. S\. 2011\.Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning\.In*AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning*\.
- Sakaguchi et al\. \(2020\)Sakaguchi, K\.; Le Bras, R\.; Bhagavatula, C\.; and Choi, Y\. 2020\.WinoGrande: An Adversarial Winograd Schema Challenge at Scale\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, 8732–8740\.
- Song and Zheng \(2026\)Song, M\.; and Zheng, M\. 2026\.Model Merging in the Era of Large Language Models: Methods, Applications, and Future Directions\.arXiv:2603\.09938\.
- Soro et al\. \(2026\)Soro, B\.; Zhang, A\. S\.; Andreis, B\.; Jo, J\.; Chong, S\.; and Hwang, S\. J\. 2026\.LS\-Merge: Merging Language Models in Latent Space\.In*International Conference on Learning Representations*\.
- Stoica et al\. \(2024\)Stoica, G\.; Bolya, D\.; Bjorner, J\.; Ramesh, P\.; Hearn, T\.; and Hoffman, J\. 2024\.ZipIt\! Merging Models from Different Tasks without Training\.In*International Conference on Learning Representations*\.
- Sun et al\. \(2023\)Sun, M\.; Liu, Z\.; Bair, A\.; and Kolter, J\. Z\. 2023\.A Simple and Effective Pruning Approach for Large Language Models\.arXiv:2306\.11695\.
- Suzgun et al\. \(2022\)Suzgun, M\.; Scales, N\.; Schärli, N\.; Gehrmann, S\.; Tay, Y\.; Chung, H\. W\.; Chowdhery, A\.; Le, Q\. V\.; Chi, E\. H\.; Zhou, D\.; and Wei, J\. 2022\.Challenging BIG\-Bench Tasks and Whether Chain\-of\-Thought Can Solve Them\.arXiv:2210\.09261\.
- Tam, Bansal, and Raffel \(2024\)Tam, D\.; Bansal, M\.; and Raffel, C\. 2024\.Merging by Matching Models in Task Parameter Subspaces\.*Transactions on Machine Learning Research*\.
- Wan et al\. \(2024a\)Wan, F\.; Huang, X\.; Cai, D\.; Quan, X\.; Bi, W\.; and Shi, S\. 2024a\.Knowledge Fusion of Large Language Models\.arXiv:2401\.10491\.
- Wan et al\. \(2024b\)Wan, F\.; Zhong, L\.; Yang, Z\.; Chen, R\.; and Quan, X\. 2024b\.FuseChat: Knowledge Fusion of Chat Models\.arXiv:2408\.07990\.
- Wang et al\. \(2018\)Wang, A\.; Singh, A\.; Michael, J\.; Hill, F\.; Levy, O\.; and Bowman, S\. R\. 2018\.GLUE: A Multi\-Task Benchmark and Analysis Platform for Natural Language Understanding\.In*Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP*, 353–355\.
- Wang et al\. \(2024\)Wang, K\.; Dimitriadis, N\.; Ortiz\-Jimenez, G\.; Fleuret, F\.; and Frossard, P\. 2024\.Localizing Task Information for Improved Model Merging and Compression\.In*Proceedings of the 41st International Conference on Machine Learning*\.
- Wang et al\. \(2025\)Wang, Y\.; Gu, Y\.; Zhang, Y\.; Zhou, Q\.; Yan, Z\.; Xie, C\.; Wang, X\.; Yuan, J\.; and Yang, H\. 2025\.Model Merging Scaling Laws in Large Language Models\.arXiv:2509\.24244\.
- Williams, Nangia, and Bowman \(2018\)Williams, A\.; Nangia, N\.; and Bowman, S\. R\. 2018\.A Broad\-Coverage Challenge Corpus for Sentence Understanding through Inference\.In*Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 1112–1122\.
- Wortsman et al\. \(2022\)Wortsman, M\.; Ilharco, G\.; Gadre, S\. Y\.; Roelofs, R\.; Gontijo\-Lopes, R\.; Morcos, A\. S\.; Namkoong, H\.; Farhadi, A\.; Carmon, Y\.; Kornblith, S\.; and Schmidt, L\. 2022\.Model Soups: Averaging Weights of Multiple Fine\-Tuned Models Improves Accuracy Without Extra Training\.In*Proceedings of the 39th International Conference on Machine Learning*\.
- Yadav et al\. \(2023\)Yadav, P\.; Tam, D\.; Choshen, L\.; Raffel, C\.; and Bansal, M\. 2023\.TIES\-Merging: Resolving Interference When Merging Models\.In*Advances in Neural Information Processing Systems*\.
- Yadav et al\. \(2024\)Yadav, P\.; Vu, T\.; Lai, J\.; Chronopoulou, A\.; Faruqui, M\.; Bansal, M\.; and Munkhdalai, T\. 2024\.What Matters for Model Merging at Scale?arXiv:2410\.03617\.
- Yang et al\. \(2026\)Yang, E\.; Shen, L\.; Guo, G\.; Wang, X\.; Cao, X\.; Zhang, J\.; and Tao, D\. 2026\.Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications, and Opportunities\.*ACM Computing Surveys*, 58\(8\)\.
- Yu et al\. \(2024\)Yu, L\.; Yu, B\.; Yu, H\.; Huang, F\.; and Li, Y\. 2024\.Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch\.In*Proceedings of the 41st International Conference on Machine Learning*\.
- Zellers et al\. \(2019\)Zellers, R\.; Holtzman, A\.; Bisk, Y\.; Farhadi, A\.; and Choi, Y\. 2019\.HellaSwag: Can a Machine Really Finish Your Sentence?In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, 4791–4800\.
- Zhou et al\. \(2023\)Zhou, J\.; Lu, T\.; Mishra, S\.; Brahma, S\.; Basu, S\.; Luan, Y\.; Zhou, D\.; and Hou, L\. 2023\.Instruction\-Following Evaluation for Large Language Models\.arXiv:2311\.07911\.

## Supplementary Material

## Appendix AComplete BBH\-27 Transfer Results

Table 3:Complete BBH\-27 per\-task accuracy\.NNis the number of evaluation examples in each subtask; the total is 6,511\. The 3B and IM columns are the original Qwen2\.5\-3B recipient and Intersection\-Merge baseline\. T1–T8 are the eight APM slices defined in the main\-paper Table 4\(a\)\. The final row reports the unweighted macro\-average across the 27 tasks and reproduces the averages summarized in Table 4\(c\)\. Scores are percentages\. All results use an injection ratio ofμ=0\.02\\mu=0\.02\.
## Appendix BMATH\-500 Significance Test

\(a\) AccuracyModelAccuracyLower CI3B0\.38400\.3420IM0\.39400\.3520APM0\.42800\.3840\(b\) Pairwise differencesComparisonDifferenceLower CIppIM−\-3B\+0\.0100\-0\.02200\.5723APM−\-3B\+0\.04400\.01200\.0073APM−\-IM\+0\.03400\.00600\.0195
Table 4:MATH\-500 significance test undermath\_verify\. The upper panel reports accuracy and the lower confidence\-interval bound\. The lower panel reports pairwise accuracy differences, lower confidence\-interval bounds, andpp\-values\. Bold entries indicate significant APM improvements atp<0\.05p<0\.05\.

## Appendix CComplete Injection\-Ratio Results

Table 1:Complete injection\-ratio results on four benchmarks\. IM and task\-activated APM are evaluated at the same ratios\. Rows above the internal rule correspond to the effective interval reported in the main paper; the final two rows show the higher\-ratio region in which the benefits weaken or disappear\. Bold entries are the best APM scores for each benchmark across all evaluated ratios\. Scores are percentages\.
## Appendix DDetailed Activation\-Data Selection Results

\(a\) Maximum QNLI accuracy across nine variants per condition

\(b\) Mean QNLI accuracy / normalized variance across nine variants per condition

Table 2:Complete QNLI results by activation\-data selection strategy and activation\-set size\. Panel \(a\) reports the maximum accuracy among the nine activation–pruning variants in each condition\. Panel \(b\) reports the corresponding mean accuracy and normalized variance\. Accuracy scores are percentages; activation\-set sizes are numbers of examples\. All results use an injection ratio ofμ=0\.05\\mu=0\.05\.
## Appendix EDetailed APM Formulation

For each inputx∈𝒟x\\in\\mathcal\{D\}, letΩ​\(x\)\\Omega\(x\)denote the token positions used for profiling and letZ=∑x∈𝒟\|Ω​\(x\)\|Z=\\sum\_\{x\\in\\mathcal\{D\}\}\|\\Omega\(x\)\|\. For donor layerrr, hidden channelii, attention headkk, and MLP neuronjj, APM uses the following activation scores:

srlayer\\displaystyle s\_\{r\}^\{\\mathrm\{layer\}\}=1Z​∑x∈𝒟∑t∈Ω​\(x\)∥hr​\(x,t\)∥2,\\displaystyle=\\frac\{1\}\{Z\}\\sum\_\{x\\in\\mathcal\{D\}\}\\sum\_\{t\\in\\Omega\(x\)\}\\lVert h\_\{r\}\(x,t\)\\rVert\_\{2\},\(9\)sihidden\\displaystyle s\_\{i\}^\{\\mathrm\{hidden\}\}=1Z​∑x∈𝒟∑t∈Ω​\(x\)\|hi​\(x,t\)\|,\\displaystyle=\\frac\{1\}\{Z\}\\sum\_\{x\\in\\mathcal\{D\}\}\\sum\_\{t\\in\\Omega\(x\)\}\|h\_\{i\}\(x,t\)\|,\(10\)sr,khead\\displaystyle s\_\{r,k\}^\{\\mathrm\{head\}\}=1Z​∑x∈𝒟∑t∈Ω​\(x\)∥or,k​\(x,t\)∥2,\\displaystyle=\\frac\{1\}\{Z\}\\sum\_\{x\\in\\mathcal\{D\}\}\\sum\_\{t\\in\\Omega\(x\)\}\\lVert o\_\{r,k\}\(x,t\)\\rVert\_\{2\},\(11\)sr,jmlp\\displaystyle s\_\{r,j\}^\{\\mathrm\{mlp\}\}=1Z​∑x∈𝒟∑t∈Ω​\(x\)\|ar,j​\(x,t\)\|\.\\displaystyle=\\frac\{1\}\{Z\}\\sum\_\{x\\in\\mathcal\{D\}\}\\sum\_\{t\\in\\Omega\(x\)\}\|a\_\{r,j\}\(x,t\)\|\.\(12\)Here,hrh\_\{r\}is the layer output,hih\_\{i\}is a hidden\-channel activation,or,ko\_\{r,k\}is the output of attention headkk, andar,ja\_\{r,j\}is the post\-activation value of MLP neuronjj\. If the recipient requiresLsL\_\{s\}layers, hidden widthdsd\_\{s\}, attention configurationHsH\_\{s\}, and MLP widthmsm\_\{s\}, the selected index sets are

ℐL∗\\displaystyle\\mathcal\{I\}\_\{L\}^\{\*\}=TopK⁡\(Slayer,Ls\),\\displaystyle=\\operatorname\{TopK\}\(S\_\{\\mathrm\{layer\}\},L\_\{s\}\),ℐd∗\\displaystyle\\mathcal\{I\}\_\{d\}^\{\*\}=TopK⁡\(Shidden,ds\),\\displaystyle=\\operatorname\{TopK\}\(S\_\{\\mathrm\{hidden\}\},d\_\{s\}\),\(13\)ℐH∗\\displaystyle\\mathcal\{I\}\_\{H\}^\{\*\}=TopK⁡\(Shead,Hs\),\\displaystyle=\\operatorname\{TopK\}\(S\_\{\\mathrm\{head\}\},H\_\{s\}\),ℐm∗\\displaystyle\\mathcal\{I\}\_\{m\}^\{\*\}=TopK⁡\(Smlp,ms\)\.\\displaystyle=\\operatorname\{TopK\}\(S\_\{\\mathrm\{mlp\}\},m\_\{s\}\)\.Writing their collection as𝒮∗\\mathcal\{S\}^\{\*\}, structured extraction is

θlAPM=𝒫​\(θl;𝒮∗,𝒜s\)\.\\theta\_\{l\}^\{\\mathrm\{APM\}\}=\\mathcal\{P\}\(\\theta\_\{l\};\\mathcal\{S\}^\{\*\},\\mathcal\{A\}\_\{s\}\)\.\(14\)
ForKKdonor slices with coefficients\{μq\}q=1K\\\{\\mu\_\{q\}\\\}\_\{q=1\}^\{K\}satisfying∑qμq<1\\sum\_\{q\}\\mu\_\{q\}<1, multi\-donor micro\-injection is

θmulti=\(1−∑q=1Kμq\)​θs\+∑q=1Kμq​θlqAPM\.\\theta\_\{\\mathrm\{multi\}\}=\\left\(1\-\\sum\_\{q=1\}^\{K\}\\mu\_\{q\}\\right\)\\theta\_\{s\}\+\\sum\_\{q=1\}^\{K\}\\mu\_\{q\}\\theta\_\{l\_\{q\}\}^\{\\mathrm\{APM\}\}\.\(15\)

## Appendix FStatistical Analysis of Diagonal Effects

LetS=\(Si​j\)i,j=1KS=\(S\_\{ij\}\)\_\{i,j=1\}^\{K\}be a cross\-task accuracy matrix whose rows index activation sources and whose columns index evaluation tasks in the same order\. We quantify matched\-task performance by the diagonal mean

D​\(S\)=1K​∑i=1KSi​i,D\(S\)=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}S\_\{ii\},\(16\)and unmatched\-task performance by the mean of all off\-diagonal entries

O​\(S\)=1K​\(K−1\)​∑i=1K∑j≠iSi​j\.O\(S\)=\\frac\{1\}\{K\(K\-1\)\}\\sum\_\{i=1\}^\{K\}\\sum\_\{j\\neq i\}S\_\{ij\}\.\(17\)The observed diagonal advantage is

Tobs=D​\(S\)−O​\(S\)\.T\_\{\\mathrm\{obs\}\}=D\(S\)\-O\(S\)\.\(18\)
##### Exact permutation test\.

Under the null hypothesis that activation\-source labels are exchangeable with respect to evaluation\-task labels, we enumerate every permutationπ∈𝒮K\\pi\\in\\mathcal\{S\}\_\{K\}\. For each permutation, the entries treated as matched are\{Sπ​\(j\),j\}j=1K\\\{S\_\{\\pi\(j\),j\}\\\}\_\{j=1\}^\{K\}\. We compute

Dπ​\(S\)\\displaystyle D\_\{\\pi\}\(S\)=1K​∑j=1KSπ​\(j\),j,\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}S\_\{\\pi\(j\),j\},\(19\)Oπ​\(S\)\\displaystyle O\_\{\\pi\}\(S\)=∑i,jSi​j−∑jSπ​\(j\),jK​\(K−1\),\\displaystyle=\\frac\{\\sum\_\{i,j\}S\_\{ij\}\-\\sum\_\{j\}S\_\{\\pi\(j\),j\}\}\{K\(K\-1\)\},\(20\)Tπ\\displaystyle T\_\{\\pi\}=Dπ​\(S\)−Oπ​\(S\)\.\\displaystyle=D\_\{\\pi\}\(S\)\-O\_\{\\pi\}\(S\)\.\(21\)The one\-sided exactpp\-value for a positive diagonal advantage is

p\+=1K\!​∑π∈𝒮K𝟏​\[Tπ≥Tobs\]\.p\_\{\+\}=\\frac\{1\}\{K\!\}\\sum\_\{\\pi\\in\\mathcal\{S\}\_\{K\}\}\\mathbf\{1\}\\\!\\left\[T\_\{\\pi\}\\geq T\_\{\\mathrm\{obs\}\}\\right\]\.\(22\)Because the permutation spaces are small, we use exhaustive enumeration rather than Monte Carlo sampling:8\!=40,3208\!=40\{,\}320permutations for BBH\-8 and7\!=5,0407\!=5\{,\}040for QA\-7\.

##### IM\-relative gains and column standardization\.

Letbjb\_\{j\}be the IM score on evaluation taskjj\. We first express each APM result as its gain over IM,

Δi​j=Si​j−bj\.\\Delta\_\{ij\}=S\_\{ij\}\-b\_\{j\}\.\(23\)Subtracting a common baseline within each column improves interpretability but does not change the unstandardized test statistic: both the diagonal and off\-diagonal means decrease byK−1​∑jbjK^\{\-1\}\\sum\_\{j\}b\_\{j\}, soT​\(Δ\)=T​\(S\)T\(\\Delta\)=T\(S\)\. To give equal scale to tasks with different across\-source variation, we additionally compute column\-wise z\-scores

Zi​j=Δi​j−Δ¯⋅jσj,Δ¯⋅j=1K​∑i=1KΔi​j,Z\_\{ij\}=\\frac\{\\Delta\_\{ij\}\-\\bar\{\\Delta\}\_\{\\cdot j\}\}\{\\sigma\_\{j\}\},\\qquad\\bar\{\\Delta\}\_\{\\cdot j\}=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\Delta\_\{ij\},\(24\)where

σj=1K​∑i=1K\(Δi​j−Δ¯⋅j\)2\.\\sigma\_\{j\}=\\sqrt\{\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\left\(\\Delta\_\{ij\}\-\\bar\{\\Delta\}\_\{\\cdot j\}\\right\)^\{2\}\}\.\(25\)We then apply Equations[18](https://arxiv.org/html/2608.13596#A6.E18)–[22](https://arxiv.org/html/2608.13596#A6.E22)toZZ\. This standardized test asks whether matched activation is advantageous relative to the typical variation among activation sources for each evaluation task, rather than being driven by tasks with larger absolute score ranges\.