NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs

# NeuPAT: Neuron-aware Plasticity Allocation Tuning for Language-Preserving MLLMs
Source: [https://arxiv.org/html/2608.08107](https://arxiv.org/html/2608.08107)
Jiayue Jin1,2,Jingwei Zhang2,5,Chen Wang2,6,Jing Liu2,3,4,Longteng Guo2,3,4\*

1

College of Intelligent Robotics and Advanced Manufacturing, Fudan University
2Zhongguancun Academy,3Institute of Automation, Chinese Academy of Sciences
4School of Artificial Intelligence, University of Chinese Academy of Sciences
5Tianjin University,6Nankai University
Correspondence:[longteng\.guo@nlpr\.ia\.ac\.cn](https://arxiv.org/html/2608.08107v1/mailto:[email protected])###### Abstract

Multimodal expansion of large language models \(LLMs\) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining\. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge\. Based on this insight, we propose NeuPAT \(Neuron\-aware Plasticity Allocation Tuning\), a lightweight and architecture\-agnostic framework that allocates neuron\-wise update constraints during multimodal instruction tuning\. NeuPAT uses a small\-scale probing stage to estimate neuron adaptation patterns and selectively protects language\-sensitive neurons while promoting multimodal adaptation through more plastic neurons\. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94\.5% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability\-preserving multimodal expansion\.

NeuPAT: Neuron\-aware Plasticity Allocation Tuning for Language\-Preserving MLLMs

Jiayue Jin1,2, Jingwei Zhang2,5, Chen Wang2,6, Jing Liu2,3,4, Longteng Guo2,3,4\*1College of Intelligent Robotics and Advanced Manufacturing, Fudan University2Zhongguancun Academy,3Institute of Automation, Chinese Academy of Sciences4School of Artificial Intelligence, University of Chinese Academy of Sciences5Tianjin University,6Nankai UniversityCorrespondence:[longteng\.guo@nlpr\.ia\.ac\.cn](https://arxiv.org/html/2608.08107v1/mailto:[email protected])

![Refer to caption](https://arxiv.org/html/2608.08107v1/motivation_.png)Figure 1:Pure\-text capability degradation and recovery after visual instruction tuning\.\(a\) Expanding an LLM into an MLLM with multimodal instruction data can induce language forgetting\. \(b\) Vanilla tuning degrades performance on representative pure\-text benchmarks, whereas NeuPAT recovers most of the lost capability\. Scores are normalized by the original LLM performance on each benchmark\.## 1Introduction

Large language models \(LLMs\) demonstrate strong capabilities in language understanding, reasoning, knowledge acquisition, and code generation through large\-scale text pretrainingBrownet al\.\([2020](https://arxiv.org/html/2608.08107#bib.bib14)\); Chowdheryet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib15)\); Achiamet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib16)\); Touvronet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib17)\)\. Extending these models beyond text has driven rapid progress in multimodal large language models \(MLLMs\)Liuet al\.\([2023b](https://arxiv.org/html/2608.08107#bib.bib1)\); Daiet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib2)\); Wanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib3)\); Baiet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib4)\); Liet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib5)\); Teamet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib8)\); Wanget al\.\([2025b](https://arxiv.org/html/2608.08107#bib.bib9)\)\. A dominant paradigm preserves a pretrained LLM as the reasoning backbone, connects it to modality\-specific encoders via projection modules, and performs multimodal instruction tuning to align new modalities with the language spaceLiuet al\.\([2023b](https://arxiv.org/html/2608.08107#bib.bib1)\); Daiet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib2)\)\. This paradigm enables strong performance on multimodal tasks such as visual question answering and reasoningLiuet al\.\([2024c](https://arxiv.org/html/2608.08107#bib.bib61)\); Yueet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib62)\); Chenet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib63)\)\.

Ideally, multimodal expansion should represent capability evolution: the model acquires new multimodal understanding while preserving the language intelligence accumulated during pretraining\. This preservation is fundamental because modern MLLMs remain language\-grounded systems, where language capabilities provide the foundation for reasoning, knowledge organization, and multimodal generalization\.

However, this ideal preservation is not always achieved\. As shown in Figure[1](https://arxiv.org/html/2608.08107#S0.F1), multimodal instruction tuning often causes substantial degradation on language reasoning benchmarks, with an average performance drop of approximately 39\.8% across key language evaluations\. This reveals that multimodal expansion is not purely additive; adapting the pretrained backbone toward multimodal distributions can interfere with existing language representations and overwrite capabilities essential for language\-based reasoning\. Therefore, an important question arises:How can we expand the capability boundary of LLMs with new modalities while preserving the language intelligence that forms the foundation of multimodal intelligence?

Recent studies have explored strategies to alleviate language capability degradation during multimodal expansion\. Existing solutions mainly rely on external interventions, including text\-only data replayLuet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib12)\); Baiet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib4)\), architectural modificationsZhanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib10)\); Wanget al\.\([2025a](https://arxiv.org/html/2608.08107#bib.bib22)\); Luet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib23)\), and post\-training model mergingRatzlaffet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib11)\); Yu and Ananiadou \([2025](https://arxiv.org/html/2608.08107#bib.bib24)\); Wanget al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib25)\); Liet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib26)\)\. Text replay retains pretrained abilities through additional language supervision but requires extra data and careful objective balancing\. Architecture\-based methods isolate multimodal adaptation from the language backbone at the cost of customized designs and added complexity\. Model merging combines the original LLM and adapted MLLM after training, yet may introduce trade\-offs between language preservation and multimodal adaptation\. More importantly, these methods rely on auxiliary interventions rather than directly regulating backbone adaptation during multimodal learning\.

In this work, we revisit multimodal expansion from the perspective of internal adaptation dynamics\. We investigate whether the pretrained LLM contains heterogeneous adaptation capacities that can be selectively regulated during multimodal learning\. Our key insight is that neurons within the pretrained backbone exhibit distinct adaptation patterns: some are more critical for preserving language intelligence, while others provide greater flexibility for absorbing new multimodal knowledge\. Therefore, uniformly updating all neurons during multimodal tuning may unnecessarily disrupt language\-critical representations while underutilizing the model’s intrinsic plasticity\.

Based on this insight, we proposeNeuPAT\(Neuron\-aware Plasticity Allocation Tuning\), a neuron\-level capability\-preserving tuning framework for multimodal expansion\. NeuPAT introduces a lightweight probing stage to estimate neuron\-wise adaptation characteristics using a small set of diagnostic samples, without introducing additional training data or modifying the model architecture\. Based on these measurements, NeuPAT dynamically allocates update constraints during multimodal instruction tuning: neurons sensitive to language capability preservation are protected from excessive adaptation, while neurons with greater multimodal plasticity are encouraged to acquire new knowledge\. By regulating internal adaptation dynamics, NeuPAT enables capability\-preserving expansion from LLMs to MLLMs\.

As shown in Figure[1](https://arxiv.org/html/2608.08107#S0.F1), NeuPAT substantially reduces language capability degradation during multimodal expansion, recovering 90\.0% of lost performance across 4 language reasoning benchmarks\. Experiments across 6 LLMs, 11 language benchmarks, and 5 multimodal benchmarks demonstrate that NeuPAT consistently generalizes across model families and scales, providing an efficient and architecture\-agnostic solution for preserving language intelligence during multimodal expansion\.

Our contributions are summarized as follows:

- •We reveal heterogeneous adaptation dynamics within LLM backbones during multimodal expansion, showing that different neurons exhibit distinct adaptation patterns associated with language preservation and multimodal learning\.
- •We propose NeuPAT, a neuron\-aware plasticity allocation framework, enabling efficient and architecture\-agnostic multimodal expansion while preserving language intelligence\.
- •Extensive experiments demonstrate that NeuPAT consistently preserves language intelligence across diverse LLM families and model scales while maintaining comparable multimodal performance, enabling scalable capability expansion from LLMs to MLLMs\.

![Refer to caption](https://arxiv.org/html/2608.08107v1/x1.png)Figure 2:Neuron modality preference and importance during multimodal expansion of LLMs\.\(a\) Raw text\- and vision\-associated importance scores, showing heterogeneous modality associations across neurons\. \(b\) Layer\-wise modality preferencePl,u=vl,u−tl,uP\_\{l,u\}=v\_\{l,u\}\-t\_\{l,u\}, with neurons sorted by preference within each layer\. \(c\) Overall importanceIl,u=\(vl,u\+tl,u\)/2I\_\{l,u\}=\(v\_\{l,u\}\+t\_\{l,u\}\)/2under the same ordering, showing diverse importance levels among neurons with similar preferences\. These results reveal heterogeneous plasticity patterns in the pretrained backbone, motivating neuron\-aware plasticity allocation during multimodal adaptation\.
## 2Related Work

### 2\.1Multimodal Large Language Models

MLLMs typically adopt an encoder\-connector\-LLM architecture, mapping visual features into the language space via cross\-attention, Q\-Former, or lightweight projection modulesAlayracet al\.\([2022](https://arxiv.org/html/2608.08107#bib.bib27)\); Liet al\.\([2023a](https://arxiv.org/html/2608.08107#bib.bib28)\); Daiet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib2)\); Zhuet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib29)\); Liuet al\.\([2023b](https://arxiv.org/html/2608.08107#bib.bib1),[2024a](https://arxiv.org/html/2608.08107#bib.bib30)\)\. Recent models, including the Qwen\-VL series, DeepSeek\-VL, InternVL3, and LLaVA\-OneVision, improve multimodal perception and reasoning with stronger encoders, larger corpora, and advanced post\-trainingBaiet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib31)\); Wanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib3)\); Baiet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib4)\); Luet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib12)\); Wanget al\.\([2025b](https://arxiv.org/html/2608.08107#bib.bib9)\); Zhuet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib7)\); Liet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib5)\)\. Despite this progress, most MLLMs still adapt pretrained LLM backbones through image\-text alignment and visual instruction tuning, potentially degrading their language capabilities and motivating our study\.

### 2\.2Methods for Mitigating Forgetting

A common solution is to mix text\-only and multimodal dataLuet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib12)\); Baiet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib31)\); Wanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib3)\); Baiet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib4)\), which increases training cost and requires careful ratio tuning\. Existing alternatives apply general parameter\-efficient or continual\-learning methodsHuet al\.\([2022](https://arxiv.org/html/2608.08107#bib.bib13)\); Kirkpatricket al\.\([2017](https://arxiv.org/html/2608.08107#bib.bib21)\), modify the architectureZhanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib10)\); Wanget al\.\([2025a](https://arxiv.org/html/2608.08107#bib.bib22)\); Luet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib23)\), or merge the adapted model with the original LLMRatzlaffet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib11)\); Yu and Ananiadou \([2025](https://arxiv.org/html/2608.08107#bib.bib24)\); Wanget al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib25)\); Liet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib26)\)\. In contrast, NeuPAT leverages neuron modality preferences to preserve language capabilities during multimodal learning without text replay, architectural changes, or post\-hoc merging\.

![Refer to caption](https://arxiv.org/html/2608.08107v1/main_8.png)Figure 3:Overview of NeuPAT\. NeuPAT identifies neuron adaptation roles through lightweight modality probing and assigns role\-specific update constraints during multimodal tuning, preserving language intelligence while acquiring multimodal capabilities\.

## 3Neuron\-Level Modality Preference Analysis

To enable capability\-preserving multimodal expansion, we first analyze how different neurons within the pretrained LLM backbone respond to text and visual inputs\. Our goal is to investigate whether the backbone contains heterogeneous plasticity patterns during multimodal adaptation\. Without such knowledge, uniform update strategies, such as globally freezing or regularizing the backbone, may preserve language capabilities but unnecessarily restrict the model’s ability to acquire new multimodal knowledge\.

### 3\.1Modality\-Associated Neuron Importance Estimation

For a Transformer layerll, we useuuto index a neuron, corresponding to an intermediate hidden dimension together with its associated input\- and output\-side parameter slices\. Lethl,u​\(xt\)h\_\{l,u\}\(x\_\{t\}\)denote the activation of neuronuuat token positiontt, and letWl,uoutW^\{\\mathrm\{out\}\}\_\{l,u\}denote its output\-side parameter slice that writes the neuron response back to the residual stream\.

We construct lightweight text\-only and visual probing sets,𝒟T\\mathcal\{D\}\_\{T\}and𝒟V\\mathcal\{D\}\_\{V\}, respectively\. Inspired by activation\-aware importance estimationSunet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib36)\), we define the modality\-associated importance score of neuronuuunder probing set𝒟\\mathcal\{D\}as

sl,u​\(𝒟\)=‖Wl,uout‖F⋅RMSx∈𝒟,t∈𝒱​\(x\)​\(‖hl,u​\(xt\)‖2\),\\begin\{split\}s\_\{l,u\}\(\\mathcal\{D\}\)=\{\}&\\left\\\|W^\{\\mathrm\{out\}\}\_\{l,u\}\\right\\\|\_\{F\}\\\\ &\\cdot\\mathrm\{RMS\}\_\{x\\in\\mathcal\{D\},\\,t\\in\\mathcal\{V\}\(x\)\}\\left\(\\left\\\|h\_\{l,u\}\(x\_\{t\}\)\\right\\\|\_\{2\}\\right\),\\end\{split\}\(1\)where𝒱​\(x\)\\mathcal\{V\}\(x\)denotes the valid non\-padding token positions\. This score jointly considers neuron activation strength and its output contribution, providing a proxy for neuron importance under a specific input modality\.

We compute the visual\- and language\-associated importance scores as

sl,uV=sl,u​\(𝒟V\),sl,uT=sl,u​\(𝒟T\)\.s^\{V\}\_\{l,u\}=s\_\{l,u\}\(\\mathcal\{D\}\_\{V\}\),\\qquad s^\{T\}\_\{l,u\}=s\_\{l,u\}\(\\mathcal\{D\}\_\{T\}\)\.\(2\)
For visualization and comparison within each layer, the scores are normalized as

vl,u=Norm​\(sl,uV\),tl,u=Norm​\(sl,uT\),v\_\{l,u\}=\\mathrm\{Norm\}\(s^\{V\}\_\{l,u\}\),\\qquad t\_\{l,u\}=\\mathrm\{Norm\}\(s^\{T\}\_\{l,u\}\),\(3\)whereNorm​\(⋅\)\\mathrm\{Norm\}\(\\cdot\)denotes layer\-wise normalization\.

Based on the normalized importance scores, we define two complementary properties:

Pl,u=vl,u−tl,u,Il,u=vl,u\+tl,u2,P\_\{l,u\}=v\_\{l,u\}\-t\_\{l,u\},\\qquad I\_\{l,u\}=\\frac\{v\_\{l,u\}\+t\_\{l,u\}\}\{2\},\(4\)wherePl,uP\_\{l,u\}measures the relative modality preference of a neuron, whileIl,uI\_\{l,u\}measures its overall importance across both modalities\. A positivePl,uP\_\{l,u\}indicates stronger association with visual inputs, whereas a negative value indicates stronger association with language inputs\. Values close to zero indicate balanced importance across modalities\.

### 3\.2Empirical Observations

Figure[2](https://arxiv.org/html/2608.08107#S1.F2)illustrates neuron importance patterns from three perspectives\. Figure[2](https://arxiv.org/html/2608.08107#S1.F2)\(a\) compares visual\- and language\-associated importance scores, revealing heterogeneous modality associations among neurons\. Figures[2](https://arxiv.org/html/2608.08107#S1.F2)\(b\) and[2](https://arxiv.org/html/2608.08107#S1.F2)\(c\) further visualize the layer\-wise modality preferencePl,uP\_\{l,u\}and overall importanceIl,uI\_\{l,u\}, respectively, where neurons are sorted according to their preference within each layer\.

Based on these analyses, we obtain three observations\.

1\) Heterogeneous modality\-associated plasticity\.Neurons exhibit diverse preferences toward language and visual inputs, indicating that the pretrained backbone does not participate uniformly in multimodal adaptation\. Consequently, applying identical updates to all neurons may unnecessarily perturb language\-critical representations while limiting the utilization of neurons better suited for multimodal adaptation\.

2\) Different neurons require different update constraints\.Neurons with balanced modality preferences are not homogeneous: some exhibit high importance under both input types, suggesting shared functionality across modalities, while others show relatively low importance and may provide additional flexibility for acquiring new multimodal knowledge\. This indicates that different neurons require different levels of update constraint rather than uniform protection or adaptation\.

3\) Plasticity patterns vary across layers\.The distributions of modality preference and importance vary substantially across Transformer layers, indicating that different layers exhibit distinct adaptation patterns\. Therefore, fixed global allocation strategies may fail to capture layer\-specific characteristics\.

Together, these observations reveal heterogeneous adaptation requirements within the pretrained backbone: some neurons require protection to preserve language intelligence, while others provide flexibility for multimodal adaptation\. This motivates neuron\-aware allocation of update flexibility during multimodal expansion\.

## 4Methods

NeuPAT performs neuron\-aware update allocation during multimodal instruction tuning\. It estimates modality\-associated importance, assigns neurons with different adaptation roles, and applies role\-specific update constraints to balance language preservation and multimodal adaptation\. Figure[3](https://arxiv.org/html/2608.08107#S2.F3)illustrates the overall pipeline\.

### 4\.1Modality\-Associated Importance Estimation

Following the analysis in Section[3](https://arxiv.org/html/2608.08107#S3), NeuPAT first estimates modality\-associated importance for each neuron using lightweight probing sets\. Specifically, we construct visual and text\-only probing sets and compute the visual\- and language\-associated importance scores,sl,uVs^\{V\}\_\{l,u\}andsl,uTs^\{T\}\_\{l,u\}, using Eq\.[1](https://arxiv.org/html/2608.08107#S3.E1)\. These scores measure the contribution of individual neurons under different input modalities and provide the basis for subsequent adaptation role allocation\.

For each layerll, the importance scores are normalized using layer\-wise normalization using Eq\.[2](https://arxiv.org/html/2608.08107#S3.E2)\. The normalized scores are used to characterize the modality\-associated importance distribution of neurons within each layer\.

### 4\.2Neuron Adaptation Role Allocation

NeuPAT identifies neurons with different adaptation roles according to their modality\-associated importance distributions\. For each layerll, letslV∈ℝNls^\{V\}\_\{l\}\\in\\mathbb\{R\}^\{N\_\{l\}\}andslT∈ℝNls^\{T\}\_\{l\}\\in\\mathbb\{R\}^\{N\_\{l\}\}denote the visual\- and language\-associated importance scores of allNlN\_\{l\}neurons\.

For each modalitya∈\{V,T\}a\\in\\\{V,T\\\}, we select the smallest neuron subset whose cumulative importance accounts for a predefined coverage ratioτa\\tau\_\{a\}:

Kla=min⁡\{K:∑r=1Ksl,πla​\(r\)a∑r=1Nlsl,πla​\(r\)a\+ϵ≥τa\},K^\{a\}\_\{l\}=\\min\\left\\\{K:\\frac\{\\sum\_\{r=1\}^\{K\}s^\{a\}\_\{l,\\pi^\{a\}\_\{l\}\(r\)\}\}\{\\sum\_\{r=1\}^\{N\_\{l\}\}s^\{a\}\_\{l,\\pi^\{a\}\_\{l\}\(r\)\}\+\\epsilon\}\\geq\\tau\_\{a\}\\right\\\},\(5\)whereπla​\(⋅\)\\pi\_\{l\}^\{a\}\(\\cdot\)denotes the neuron indices ranked by modality\-associated importance\. The selected neuron sets for visual and language inputs are denoted as𝒱l\\mathcal\{V\}\_\{l\}and𝒯l\\mathcal\{T\}\_\{l\}, respectively\.

Based on the relationship between𝒱l\\mathcal\{V\}\_\{l\}and𝒯l\\mathcal\{T\}\_\{l\}, we derive four neuron adaptation roles:

𝒞ll​a​n​g\\displaystyle\\mathcal\{C\}^\{lang\}\_\{l\}=𝒯l∖𝒱l,\\displaystyle=\\mathcal\{T\}\_\{l\}\\setminus\\mathcal\{V\}\_\{l\},𝒞lm​u​l​t​i\\displaystyle\\mathcal\{C\}^\{multi\}\_\{l\}=𝒱l∖𝒯l,\\displaystyle=\\mathcal\{V\}\_\{l\}\\setminus\\mathcal\{T\}\_\{l\},\(6\)𝒞ls​h​a​r​e​d\\displaystyle\\mathcal\{C\}^\{shared\}\_\{l\}=𝒯l∩𝒱l,\\displaystyle=\\mathcal\{T\}\_\{l\}\\cap\\mathcal\{V\}\_\{l\},𝒞lr​e​s​e​r​v​e\\displaystyle\\mathcal\{C\}^\{reserve\}\_\{l\}=𝒯l∪𝒱l¯\.\\displaystyle=\\overline\{\\mathcal\{T\}\_\{l\}\\cup\\mathcal\{V\}\_\{l\}\}\.where the complement in𝒞lr​e​s​e​r​v​e\\mathcal\{C\}^\{reserve\}\_\{l\}is taken with respect to all neurons in layerll\. Here,𝒞ll​a​n​g\\mathcal\{C\}^\{lang\}\_\{l\}contains neurons primarily associated with language inputs and therefore requires protection during multimodal adaptation\.𝒞lm​u​l​t​i\\mathcal\{C\}^\{multi\}\_\{l\}contains neurons associated with visual inputs and provides adaptive capacity for acquiring multimodal knowledge\.𝒞ls​h​a​r​e​d\\mathcal\{C\}^\{shared\}\_\{l\}contains neurons important to both modalities and requires constrained updates, while𝒞lr​e​s​e​r​v​e\\mathcal\{C\}^\{reserve\}\_\{l\}provides additional flexibility for multimodal learning\.

Table 1:Comparison on 11 language benchmarks grouped into Language and Logical Reasoning Tasks, Math and Code Reasoning Tasks, and General Tasks\.Δ​\(2\)−\(1\)\\Delta\(2\)\\\!\-\\\!\(1\)denotes Vanilla Tuning−\-LLM, whileΔ​\(9\)−\(2\)\\Delta\(9\)\\\!\-\\\!\(2\)denotes NeuPAT−\-Vanilla Tuning\. Bold values indicate the best results among multimodally adapted models\.MethodLanguage and Logical Reasoning TasksMath and Code Reasoning TasksGeneral TasksOverallBBHbAbIDROPLogiQA2\\columncolorgray\!5Avg\.MATH\-500MBPPGSM8K\\columncolorgray\!5Avg\.SocialIQACoQAGPQAARC\-C\\columncolorgray\!5Avg\.\\columncolorgray\!8Avg\.\(1\)LLM47\.8313\.2015\.3035\.94\\columncolorgray\!528\.0764\.6065\.8086\.20\\columncolorgray\!572\.2050\.1067\.5336\.3855\.97\\columncolorgray\!552\.50\\columncolorgray\!848\.99\(2\)\+ Vanilla Tuning37\.122\.689\.9032\.95\\columncolorgray\!520\.6650\.4062\.6083\.47\\columncolorgray\!565\.4946\.7867\.3734\.1555\.72\\columncolorgray\!551\.01\\columncolorgray\!843\.92𝚫​\(𝟐\)−\(𝟏\)\\boldsymbol\{\\Delta\(2\)\\\!\-\\\!\(1\)\}\-10\.71\-10\.52\-5\.40\-2\.99\\columncolorgray\!5\-7\.41\-14\.20\-3\.20\-2\.73\\columncolorgray\!5\-6\.71\-3\.32\-0\.16\-2\.23\-0\.25\\columncolorgray\!5\-1\.49\\columncolorgray\!8\-5\.07\(3\)\+ LoRA49\.721\.5711\.4233\.97\\columncolorgray\!524\.1746\.2065\.4081\.96\\columncolorgray\!564\.5251\.3870\.5836\.1655\.63\\columncolorgray\!553\.44\\columncolorgray\!845\.82\(4\)\+ EWC42\.502\.7412\.9035\.88\\columncolorgray\!523\.5152\.8066\.0085\.52\\columncolorgray\!568\.1147\.6569\.7735\.7157\.51\\columncolorgray\!552\.66\\columncolorgray\!846\.27\(5\)\+ WINGS51\.762\.7011\.0334\.16\\columncolorgray\!524\.9153\.2063\.2079\.91\\columncolorgray\!565\.4450\.0070\.5035\.4956\.48\\columncolorgray\!553\.12\\columncolorgray\!846\.22\(6\)\+ TIES38\.692\.6714\.7035\.62\\columncolorgray\!522\.9251\.4063\.2084\.15\\columncolorgray\!566\.2546\.9367\.7234\.6056\.06\\columncolorgray\!551\.33\\columncolorgray\!845\.07\(7\)\+ L2M41\.382\.6714\.4735\.81\\columncolorgray\!523\.5850\.0063\.8084\.00\\columncolorgray\!565\.9346\.5268\.5234\.3855\.97\\columncolorgray\!551\.35\\columncolorgray\!845\.23\(8\)\+ PlaM36\.372\.6714\.6135\.69\\columncolorgray\!522\.3450\.8062\.8083\.32\\columncolorgray\!565\.6446\.4767\.4534\.6055\.89\\columncolorgray\!551\.10\\columncolorgray\!844\.61\\rowcolorgray\!10\(9\)\+ NeuPAT53\.8810\.6715\.7436\.39\\columncolorgray\!529\.1756\.4066\.6085\.97\\columncolorgray\!569\.6650\.2669\.5337\.2756\.91\\columncolorgray\!553\.49\\columncolorgray\!849\.06\\rowcolorgray\!10𝚫​\(𝟗\)−\(𝟐\)\\boldsymbol\{\\Delta\(9\)\\\!\-\\\!\(2\)\}\+16\.76\+7\.99\+5\.84\\columncolorgray\!5\+3\.44\+8\.51\+6\.00\+4\.00\\columncolorgray\!5\+2\.50\+4\.17\+3\.48\+2\.16\+3\.12\\columncolorgray\!5\+1\.19\\columncolorgray\!8\+2\.49\+5\.14

### 4\.3Role\-Aware Multimodal Tuning

Based on the assigned adaptation roles, NeuPAT regulates neuron\-wise update flexibility during multimodal instruction tuning\.

Language\-associated neurons are frozen to preserve pretrained language representations\. Multimodal\-adaptive neurons receive full updates to absorb new multimodal knowledge\. Reserve neurons are also fully optimized as additional adaptation capacity\. Shared neurons, which participate in both language and multimodal processing, receive constrained updates to balance preservation and adaptation\.

For shared neurons, we separate the corresponding parameters into input\- and output\-side components, denoted byWl,uinW^\{\\mathrm\{in\}\}\_\{l,u\}andWl,uoutW^\{\\mathrm\{out\}\}\_\{l,u\}\. Their deviation from pretrained parameters is constrained by:

ℛshared=∑l∑u∈𝒞lshared\[λin∥Wl,uin−Wl,uin,0∥F2\\displaystyle\\mathcal\{R\}\_\{\\mathrm\{shared\}\}=\\sum\_\{l\}\\sum\_\{u\\in\\mathcal\{C\}^\{\\mathrm\{shared\}\}\_\{l\}\}\\Bigg\[\\lambda\_\{\\mathrm\{in\}\}\\left\\\|W^\{\\mathrm\{in\}\}\_\{l,u\}\-W^\{\\mathrm\{in\},0\}\_\{l,u\}\\right\\\|\_\{F\}^\{2\}\(7\)\+λout\(1−cos\(Wl,uout,Wl,uout,0\)\)\]\.\\displaystyle\\qquad\+\\lambda\_\{\\mathrm\{out\}\}\\left\(1\-\\cos\(W^\{\\mathrm\{out\}\}\_\{l,u\},W^\{\\mathrm\{out\},0\}\_\{l,u\}\)\\right\)\\Bigg\]\.The input\-side constraint limits changes to neuron activation behavior, while the output\-side cosine constraint preserves the direction of neuron contributions to the residual stream\. During optimization, gradients of language\-associated neurons are masked, while other neuron roles follow their assigned update constraints\.

The final training objective isℒ=ℒori\+ℛshared,\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{ori\}\}\+\\mathcal\{R\}\_\{\\mathrm\{shared\}\},whereℒori\\mathcal\{L\}\_\{\\mathrm\{ori\}\}denotes the original autoregressive language\-modeling objective\.

Table 2:Comparison on multimodal benchmarks\.Δ\\Deltadenotes NeuPAT−\-Vanilla Tuning\.MethodMMBRWQAMMMUPOPEMMS\\columncolorgray\!5Avg\.Vanilla Tuning72\.0856\.6043\.1187\.6945\.40\\columncolorgray\!560\.98LoRA68\.2153\.5939\.4486\.5742\.56\\columncolorgray\!558\.07EWC67\.5350\.5943\.7884\.9940\.67\\columncolorgray\!557\.51WINGS73\.8857\.7843\.4487\.8647\.91\\columncolorgray\!562\.17TIES72\.4255\.5643\.0087\.7445\.81\\columncolorgray\!560\.91L2M72\.9455\.1642\.8987\.6746\.24\\columncolorgray\!560\.98PlaM72\.7756\.4743\.0087\.7745\.10\\columncolorgray\!561\.02\\rowcolorgray\!10NeuPAT72\.4856\.9942\.2288\.7244\.85\\columncolorgray\!561\.05\\rowcolorgray\!10𝚫\\boldsymbol\{\\Delta\}\+0\.40\+0\.39\-0\.89\\columncolorgray\!5\+1\.03\-0\.55\+0\.07

## 5Experiments

### 5\.1Experimental Settings

#### 5\.1\.1Implementation Details

We use RICE\-ViT\-Large\-Patch14\-560 as the vision towerXieet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib37)\)and Qwen3\-4B\-Instruct\-2507 as the default language backboneYanget al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib39)\)\. Generalization is evaluated on five additional backbones: Qwen3\-0\.6B, Phi\-4\-Mini\-Instruct, Qwen2\.5\-7B\-Instruct, Llama3\.1\-8B\-Instruct, and Qwen2\.5\-14B\-InstructYanget al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib39)\); Aboueleninet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib38)\); Yanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib40)\); Grattafioriet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib41)\)\. Training consists of image\-text alignment on LLaVA\-558KLiuet al\.\([2024a](https://arxiv.org/html/2608.08107#bib.bib30)\)with only the multimodal adapter updated, followed by NeuPAT\-based visual instruction tuning on LLaVA\-NeXT\-780KLiuet al\.\([2024b](https://arxiv.org/html/2608.08107#bib.bib42)\)\. We setτa=0\.8\\tau\_\{a\}=0\.8andλin=λout=0\.1\\lambda\_\{\\mathrm\{in\}\}=\\lambda\_\{\\mathrm\{out\}\}=0\.1\. All models are trained with Adam on 8 NVIDIA A100 GPUs\.

#### 5\.1\.2Baselines

We compare NeuPAT with three baseline groups:\(1\) reference models,including the original LLM and Vanilla Tuning;\(2\) general adaptation methods,including LoRAHuet al\.\([2022](https://arxiv.org/html/2608.08107#bib.bib13)\)and EWCKirkpatricket al\.\([2017](https://arxiv.org/html/2608.08107#bib.bib21)\); and\(3\) MLLM\-specific preservation methods,including the architecture\-based WINGSZhanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib10)\)and post\-hoc merging approaches such as TIESRatzlaffet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib11)\), Locate\-then\-MergeYu and Ananiadou \([2025](https://arxiv.org/html/2608.08107#bib.bib24)\), and PlaMWanget al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib25)\)\. The original LLM provides a reference for language capability, while Vanilla Tuning reflects the forgetting caused by standard visual instruction tuning\.

#### 5\.1\.3Evaluation Protocol

We evaluate language and multimodal capabilities usinglm\-evaluation\-harnessGaoet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib43)\)andlmms\-evalZhanget al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib44)\), respectively\. The language suite contains 11 benchmarks covering knowledge, mathematics, logic, coding, and reading comprehension: SocialIQASapet al\.\([2019](https://arxiv.org/html/2608.08107#bib.bib45)\), ARC\-ChallengeClarket al\.\([2018](https://arxiv.org/html/2608.08107#bib.bib53)\), GPQAReinet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib48)\), MATH\-500Hendryckset al\.\([2021](https://arxiv.org/html/2608.08107#bib.bib57)\), GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2608.08107#bib.bib47)\), LogiQA2Liuet al\.\([2023a](https://arxiv.org/html/2608.08107#bib.bib55)\), BBHSuzgunet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib51)\), bAbIWestonet al\.\([2015](https://arxiv.org/html/2608.08107#bib.bib52)\), MBPPAustinet al\.\([2021](https://arxiv.org/html/2608.08107#bib.bib56)\), DROPDuaet al\.\([2019](https://arxiv.org/html/2608.08107#bib.bib49)\)and CoQAReddyet al\.\([2019](https://arxiv.org/html/2608.08107#bib.bib50)\)\. Multimodal performance is evaluated on MMBench\-ENLiuet al\.\([2024c](https://arxiv.org/html/2608.08107#bib.bib61)\), RealWorldQAxAI \([2024](https://arxiv.org/html/2608.08107#bib.bib65)\), POPELiet al\.\([2023c](https://arxiv.org/html/2608.08107#bib.bib64)\), MMMUYueet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib62)\)and MMStarChenet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib63)\)\.

Table 3:Generalization across different LLM backbones\. For each backbone, we report the original LLM, Vanilla Tuning, and our method\. We show representative language benchmarks and multimodal benchmarks\.Δ\\Deltadenotes NeuPAT−\-Vanilla Tuning\.BackboneMethodLanguage BenchmarksMultimodal BenchmarksSIQAGSM8KBBHMBPPCoQAAvg\.MMBRWQAMMMUPOPEMMStarAvg\.LLM40\.3840\.5633\.3927\.4057\.5739\.86––––––\+ Vanilla Tuning39\.0035\.1828\.6922\.6055\.9336\.2852\.5845\.1030\.6785\.4237\.2350\.20\\rowcolorgray\!9\\cellcolorwhite\+ NeuPAT40\.3340\.1133\.4428\.2059\.4240\.3052\.5244\.7532\.6784\.2637\.0750\.25\\rowcolorgray\!9\\cellcolorwhiteQwen3\-0\.6B𝚫\\boldsymbol\{\\Delta\}\+1\.33\+4\.93\+4\.75\+5\.60\+3\.49\+4\.02\-0\.06\-0\.35\+2\.00\-1\.16\-0\.16\+0\.05LLM49\.5983\.6252\.8255\.4078\.4063\.97––––––\+ Vanilla Tuning46\.1173\.9240\.0447\.6065\.2354\.5850\.5245\.1036\.7877\.1630\.5848\.03\\rowcolorgray\!9\\cellcolorwhite\+ NeuPAT50\.0681\.4551\.2953\.8080\.0463\.3350\.8944\.3136\.2277\.8231\.2648\.10\\rowcolorgray\!9\\cellcolorwhitePhi\-4\-Mini\-Instruct𝚫\\boldsymbol\{\\Delta\}\+3\.95\+7\.53\+11\.25\+6\.20\+14\.81\+8\.75\+0\.37\-0\.79\-0\.56\+0\.66\+0\.68\+0\.07LLM51\.5976\.5045\.8147\.6078\.7460\.05––––––\+ Vanilla Tuning50\.2074\.8341\.9840\.2073\.6356\.1764\.7852\.0341\.0087\.1240\.5657\.10\\rowcolorgray\!9\\cellcolorwhite\+ NeuPAT55\.8978\.7047\.5342\.8078\.8060\.7464\.6452\.0142\.5686\.5440\.3757\.22\\rowcolorgray\!9\\cellcolorwhiteQwen2\.5\-7B\-Instruct𝚫\\boldsymbol\{\\Delta\}\+5\.69\+3\.87\+5\.55\+2\.60\+5\.17\+4\.57\-0\.14\-0\.02\+1\.56\-0\.58\-0\.19\+0\.12LLM49\.8578\.1744\.5958\.4077\.8361\.77––––––\+ Vanilla Tuning49\.7467\.5534\.9154\.2077\.5056\.7860\.1426\.6735\.8982\.2336\.5448\.29\\rowcolorgray\!9\\cellcolorwhite\+ NeuPAT49\.9574\.5541\.6856\.8080\.4860\.6961\.9444\.9736\.2280\.9033\.9451\.59\\rowcolorgray\!9\\cellcolorwhiteLlama3\.1\-8B\-Instruct𝚫\\boldsymbol\{\\Delta\}\+0\.21\+7\.00\+6\.77\+2\.60\+2\.98\+3\.91\+1\.80\+18\.30\+0\.33\-1\.33\-2\.60\+3\.30LLM54\.0479\.8352\.5466\.8078\.2066\.28––––––\+ Vanilla Tuning51\.2379\.3039\.8765\.6074\.4462\.0975\.1756\.3447\.6787\.2750\.2663\.34\\rowcolorgray\!9\\cellcolorwhite\+ NeuPAT55\.2285\.4452\.5966\.8077\.5167\.5174\.3155\.6448\.1187\.6750\.1463\.17\\rowcolorgray\!9\\cellcolorwhiteQwen2\.5\-14B\-Instruct𝚫\\boldsymbol\{\\Delta\}\+3\.99\+6\.14\+12\.72\+1\.20\+3\.07\+5\.42\-0\.86\-0\.70\+0\.44\+0\.40\-0\.12\-0\.17

### 5\.2Experimental Results

#### 5\.2\.1Main Results

Tables[4\.2](https://arxiv.org/html/2608.08107#S4.SS2)and[4\.3](https://arxiv.org/html/2608.08107#S4.SS3)report results on 11 language and 5 multimodal benchmarks\. Vanilla Tuning lowers the language average from48\.9948\.99to43\.9243\.92, with drops exceeding 10 points on BBH, bAbI, and MATH\-500\. NeuPAT recovers5\.145\.14points, reaching49\.0649\.06and slightly surpassing the original LLM, with particularly strong gains on language and logical reasoning tasks\. Meanwhile, it maintains comparable multimodal performance, improving the average from60\.9860\.98to61\.0561\.05\. Although WINGS achieves the highest multimodal average, NeuPAT provides the best overall language performance and a stronger balance between language preservation and multimodal adaptation\.

#### 5\.2\.2Generalization across LLM Backbones

To evaluate cross\-backbone generalization, we replace the default LLM with Qwen3\-0\.6B, Phi\-4\-Mini\-Instruct, Qwen2\.5\-7B/14B\-Instruct, and Llama3\.1\-8B\-Instruct\. Due to space limitations, Table[3](https://arxiv.org/html/2608.08107#S5.T3)reports five representative language benchmarks, while the complete results are provided in the appendix\. Vanilla Tuning consistently degrades language performance across all tested backbones, whereas NeuPAT improves the reported text average over Vanilla Tuning by4\.024\.02,8\.758\.75,4\.574\.57,3\.913\.91, and5\.425\.42points, respectively, while maintaining comparable multimodal performance\. For Qwen3\-0\.6B and both Qwen2\.5 backbones, NeuPAT even surpasses the original LLM average\. These consistent improvements across different model families and scales demonstrate that NeuPAT is not tied to a specific language backbone\.

Table 4:Ablation of neuron\-wise adaptation role allocation\. ✓: full update,×\\times: freeze, Reg\.: regularized update\.VariantsNeuron Adaptation RoleLanguage BenchmarksMultimodal BenchmarksLang\.Multi\.SharedReserveGSM8KMBPPbAbIAvg\.MMBRWQAPOPEAvg\.w/o Language Freeze✓✓Reg\.✓83\.9064\.805\.8751\.5272\.1157\.3087\.2072\.20w/o Multimodal Update×\\times×\\timesReg\.✓84\.4467\.1011\.8554\.4669\.6955\.1686\.6170\.49w/o Reserve Update×\\times✓Reg\.×\\times84\.7666\.8011\.6454\.4071\.0556\.3486\.9471\.44Shared Full Update×\\times✓✓✓83\.6164\.009\.9452\.5272\.3457\.2587\.2772\.29Shared Freeze×\\times✓×\\times✓85\.0667\.2015\.1655\.8169\.9355\.0387\.3070\.75\\rowcolorgray\!10Ours×\\times✓Reg\.✓85\.9766\.6010\.6754\.4172\.4856\.9988\.7272\.73Table 5:Ablation of global update schemes and role\-aware update allocation\.UpdateLanguage BenchmarksMultimodal BenchmarksstrategyGSM8KMBPPbAbIAvg\.MMBRWQAPOPEAvg\.Freeze86\.0567\.1013\.6555\.6068\.6453\.9986\.0369\.55Update83\.4762\.602\.6849\.5872\.0856\.6087\.6972\.12Reg\.84\.9965\.509\.3353\.2772\.0854\.2586\.0370\.79\\rowcolorgray\!10Ours85\.9766\.6010\.6754\.4172\.4856\.9988\.7272\.73Table 6:Ablation of neuron role allocation strategies\.AllocationLanguage BenchmarksMultimodal BenchmarksstrategyGSM8KMBPPbAbIAvg\.MMBRWQAPOPEAvg\.Random83\.6965\.401\.9450\.3469\.7654\.5187\.5270\.60Fixed\-ratio84\.9066\.2010\.2453\.7872\.1657\.2587\.2872\.23\\rowcolorgray\!10Ours85\.9766\.6010\.6754\.4172\.4856\.9988\.7272\.73
#### 5\.2\.3Ablation Studies

##### Global vs\. Neuron\-Aware Update\.

We compare NeuPAT with three global update schemes\.Freezefreezes all neurons,Updateuniformly updates the backbone, andReg\.applies the shared\-neuron regularization globally\. As shown in Table[5](https://arxiv.org/html/2608.08107#S5.T5), global strategies suffer from either limited multimodal adaptation or insufficient language preservation\. NeuPAT achieves the best balance, obtaining language and multimodal averages of54\.4154\.41and72\.7372\.73, demonstrating the effectiveness of neuron\-wise update allocation\.

##### Neuron Allocation Strategy\.

We compare importance\-guided role allocation withRandomandFixed\-ratiobaselines\.Randompreserves category sizes but randomly assigns neuron roles, whileFixed\-ratioapplies the same selection ratio across layers\. As shown in Table[6](https://arxiv.org/html/2608.08107#S5.T6), NeuPAT consistently outperforms both baselines, validating the importance of reliable neuron identification and layer\-adaptive allocation\.

##### Neuron\-Wise Update Constraint\.

We further evaluate each role\-specific update strategy by modifying one constraint at a time\. As shown in Table[4](https://arxiv.org/html/2608.08107#S5.T4), updating language\-associated neurons decreases text performance, while freezing vision\-associated or reserve neurons limits multimodal adaptation\. For shared neurons, full updating harms language preservation and full freezing sacrifices multimodal performance\. These results support the proposed strategy of freezing language neurons, updating multimodal and reserve neurons, and regularizing shared neurons\.

#### 5\.2\.4Neuron Distribution Visualization

We visualize the assigned neuron roles and their layer\-wise distributions in Figure[4](https://arxiv.org/html/2608.08107#S5.F4)\. As shown in Figure[4](https://arxiv.org/html/2608.08107#S5.F4)\(a\), the resulting neuron map is closely consistent with the modality\-associated response patterns observed in Figure[2](https://arxiv.org/html/2608.08107#S1.F2)\. Language\-associated neurons are concentrated in text\-preferred regions, whereas multimodal\-adaptive neurons primarily occupy vision\-preferred regions\. In contrast, shared and reserve neurons are located mainly in balanced\-response regions, but differ markedly in their overall response strength: shared neurons respond strongly to both input types, while reserve neurons remain weakly engaged and may provide underutilized capacity for multimodal adaptation\. Figure[4](https://arxiv.org/html/2608.08107#S5.F4)\(b\) further shows that shared neurons constitute the largest group in most layers, accounting for approximately45%45\\%, while language\-associated and multimodal\-adaptive neurons each represent around20%20\\%, and reserve neurons account for roughly15%15\\%\. These consistent yet layer\-dependent distributions reveal substantial heterogeneity in neuron functionality and adaptation capacity, supporting the need for layer\-adaptive neuron allocation and differentiated plasticity control within the LLM backbone\.

![Refer to caption](https://arxiv.org/html/2608.08107v1/x2.png)Figure 4:Layer\-wise adaptation role distribution\.\(a\) Adaptation roles identified by importance\-guided allocation\. \(b\) Layer\-wise proportions of neuron roles\.## 6Conclusion

Multimodal expansion enables LLMs to acquire capabilities beyond language, but may compromise the language intelligence inherited from pretraining\. We show that this degradation stems from heterogeneous adaptation behaviors within the pretrained LLM backbone and introduce NeuPAT, a neuron\-aware update allocation framework for capability\-preserving multimodal tuning\. By selectively regulating neuron update flexibility, NeuPAT balances language preservation and multimodal adaptation without additional data, architectural changes, or post\-training merging\. Experiments across diverse LLM backbones demonstrate that NeuPAT is an efficient, architecture\-agnostic solution for reliable LLM\-to\-MLLM expansion\.

## 7Limitations

NeuPAT has several limitations\. First, although we evaluate it across diverse LLM families and model scales, its effectiveness on substantially larger backbones remains unverified\. Larger models may exhibit more distributed and complex adaptation patterns, and the current neuron\-wise role allocation mechanism may require further investigation at greater scale\. Second, our experiments focus on vision\-language expansion\. Extending NeuPAT to other modalities, such as audio, video, embodied interaction, or unified multimodal systems, may introduce different alignment dynamics and modality interactions, potentially requiring modality\-specific importance estimation and update strategies\. Finally, this work considers a single\-stage multimodal expansion process\. In continual or sequential learning scenarios, newly introduced modalities and tasks may interact with both pretrained language capabilities and previously acquired multimodal knowledge\. Developing mechanisms for stable long\-term capability accumulation without progressive interference remains an important direction for future research\.

## 8Ethical Considerations

All training data and evaluation experiments in this work are based on publicly available datasets and benchmarks\. And we emphasize that the proposed neuron allocation strategies are response\-based functional approximations rather than causal explanations of model behavior\. Models trained with NeuPAT should therefore undergo standard safety, fairness, and robustness evaluations before deployment, especially in high\-stakes applications\. AI tools were used only for language polishing\. All research ideas, experiments, analyses, and manuscript organization were completed by the authors\.

## References

- A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen,et al\.\(2025\)Phi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-loras\.arXiv preprint arXiv:2503\.01743\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1)\.
- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.arXiv preprint arXiv:2204\.14198\.Cited by:[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1)\.
- S\. Chaudhary \(2023\)Code alpaca: an instruction\-following llama model for code generation\.GitHub\.Note:[https://github\.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca)Cited by:[§C\.1](https://arxiv.org/html/2608.08107#A3.SS1.p1.2)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin,et al\.\(2024\)Are we on the right way for evaluating large vision\-language models?\.Advances in Neural Information Processing Systems37,pp\. 27056–27087\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.\(2023\)Palm: scaling language modeling with pathways\.Journal of machine learning research24\(240\),pp\. 1–113\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- M\. Conover, M\. Hayes, A\. Mathur, J\. Xie, J\. Wan, S\. Shah, A\. Ghodsi, P\. Wendell, M\. Zaharia, and R\. Xin \(2023\)Free dolly: introducing the world’s first truly open instruction\-tuned llm\.External Links:[Link](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm)Cited by:[§C\.1](https://arxiv.org/html/2608.08107#A3.SS1.p1.2)\.
- W\. Dai, J\. Li, D\. Li, A\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. N\. Fung, and S\. Hoi \(2023\)Instructblip: towards general\-purpose vision\-language models with instruction tuning\.Advances in neural information processing systems36,pp\. 49250–49267\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- D\. Dua, Y\. Wang, P\. Dasigi, G\. Stanovsky, S\. Singh, and M\. Gardner \(2019\)DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2368–2378\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- T\. Fu, W\. Wang, F\. Li, H\. Zhang, J\. Zhang, D\. Li, Y\. Dong, R\. Liu, and W\. Zhou \(2026\)Correct when paired, wrong when split: decoupling and editing modality\-specific neurons in mllms\.arXiv preprint arXiv:2606\.17057\.Cited by:[§B\.1](https://arxiv.org/html/2608.08107#A2.SS1.p1.1)\.
- L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou \(2024\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§C\.3](https://arxiv.org/html/2608.08107#A3.SS3.SSS0.Px1.p1.3),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2608.08107#S5.SS1.SSS2.p1.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the national academy of sciences114\(13\),pp\. 3521–3526\.Cited by:[§C\.3](https://arxiv.org/html/2608.08107#A3.SS3.SSS0.Px2.p1.3),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2608.08107#S5.SS1.SSS2.p1.1)\.
- B\. Li, N\. Deng, T\. Dong, S\. Wang, S\. Zhu, and L\. Wen \(2026\)MNAFT: modality neuron\-aware fine\-tuning of multimodal large language models for image translation\.Science China Information Sciences69\(5\)\.Cited by:[§B\.1](https://arxiv.org/html/2608.08107#A2.SS1.p1.1)\.
- B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, P\. Zhang, Y\. Li, Z\. Liu,et al\.\(2024\)Llava\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023a\)Blip\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InInternational conference on machine learning,pp\. 19730–19742\.Cited by:[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- J\. Li, X\. Cheng, X\. Zhao, J\. Nie, and J\. Wen \(2023b\)Halueval: a large\-scale hallucination evaluation benchmark for large language models\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§C\.1](https://arxiv.org/html/2608.08107#A3.SS1.p1.2)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, X\. Zhao, and J\. Wen \(2023c\)Evaluating object hallucination in large vision\-language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 292–305\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- Z\. Li, H\. Wen, W\. Hu, Z\. Luo, L\. Chen, S\. Wang, and S\. Wu \(2025\)DPIMerge: an efficient dynamic parameter interpolation framework for alleviating pure text forgetting in multimodal large models\.InCCF International Conference on Natural Language Processing and Chinese Computing,pp\. 3–15\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- H\. Liu, J\. Liu, L\. Cui, Z\. Teng, N\. Duan, M\. Zhou, and Y\. Zhang \(2023a\)LogiQA 2\.0—an improved dataset for logical reasoning in natural language understanding\.IEEE/ACM Transactions on Audio, Speech, and Language Processing31\(\),pp\. 2947–2962\.External Links:[Document](https://dx.doi.org/10.1109/TASLP.2023.3293046)Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024a\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 26296–26306\.Cited by:[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1),[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024b\)LLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§C\.1](https://arxiv.org/html/2608.08107#A3.SS1.p1.2),[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023b\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024c\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- Z\. Liu, G\. Dou, X\. Yuan, C\. Zhang, Z\. Tan, and M\. Jiang \(2025\)Modality\-aware neuron pruning for unlearning in multimodal large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 5913–5933\.Cited by:[§B\.1](https://arxiv.org/html/2608.08107#A2.SS1.p1.1)\.
- H\. Lu, W\. Liu, B\. Zhang, B\. Wang, K\. Dong, B\. Liu, J\. Sun, T\. Ren, Z\. Li, H\. Yang,et al\.\(2024\)Deepseek\-vl: towards real\-world vision\-language understanding\.arXiv preprint arXiv:2403\.05525\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- X\. Lu, Y\. Chen, R\. Wu, H\. Gao, X\. Chen, X\. Yang, X\. Zhao, A\. Zhou, F\. Li, Y\. Wen,et al\.\(2025\)GenieBlue: integrating both linguistic and multimodal capabilities for large language models on mobile devices\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 4198–4210\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- Z\. Qin, Q\. Yu, K\. Lyu, Z\. Fan, and Y\. Sun \(2025\)The achilles’ heel of llms: how altering a handful of neurons can cripple language abilities\.arXiv preprint arXiv:2510\.10238\.Cited by:[§B\.1](https://arxiv.org/html/2608.08107#A2.SS1.p1.1)\.
- N\. Ratzlaff, M\. Luo, X\. Su, V\. Lal, and P\. Howard \(2024\)Training\-free mitigation of language reasoning degradation after multimodal instruction tuning\.arXiv preprint arXiv:2412\.03467\.Cited by:[§C\.3](https://arxiv.org/html/2608.08107#A3.SS3.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2608.08107#S5.SS1.SSS2.p1.1)\.
- S\. Reddy, D\. Chen, and C\. D\. Manning \(2019\)Coqa: a conversational question answering challenge\.Transactions of the Association for Computational Linguistics7,pp\. 249–266\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2023\)Gpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi \(2019\)Social iqa: commonsense reasoning about social interactions\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 4463–4473\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- M\. Sun, Z\. Liu, A\. Bair, and Z\. Kolter \(2024\)A simple and effective pruning approach for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 4942–4964\.Cited by:[§3\.1](https://arxiv.org/html/2608.08107#S3.SS1.p2.4)\.
- M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. H\. Chi, D\. Zhou,et al\.\(2023\)Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- K\. Team, A\. Du, B\. Yin, B\. Xing, B\. Qu, B\. Wang, C\. Chen, C\. Zhang, C\. Du, C\. Wei,et al\.\(2025\)Kimi\-vl technical report\.arXiv preprint arXiv:2504\.07491\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1)\.
- B\. Wang, C\. Xie, D\. Leng, and Y\. Yin \(2025a\)Iaa: inner\-adaptor architecture empowers frozen large language model with multimodal capabilities\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 21035–21043\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1)\.
- W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.\(2025b\)Internvl3\. 5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.arXiv preprint arXiv:2508\.18265\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- Z\. Wang, Y\. Liu, M\. Wang, E\. Nie, D\. Chen, Z\. Zhao, S\. Feng, D\. Wang, X\. Yang, Y\. Zhang,et al\.\(2026\)PlaM: training\-free plateau\-guided model merging for better visual grounding in mllms\.arXiv preprint arXiv:2601\.07645\.Cited by:[§C\.3](https://arxiv.org/html/2608.08107#A3.SS3.SSS0.Px6.p1.1),[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2608.08107#S5.SS1.SSS2.p1.1)\.
- J\. Weston, A\. Bordes, S\. Chopra, A\. M\. Rush, B\. Van Merriënboer, A\. Joulin, and T\. Mikolov \(2015\)Towards ai\-complete question answering: a set of prerequisite toy tasks\.arXiv preprint arXiv:1502\.05698\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- xAI \(2024\)Grok\-1\.5 vision preview\.Note:[https://x\.ai/news/grok\-1\.5v](https://x.ai/news/grok-1.5v)Introduces the RealWorldQA benchmarkCited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- Y\. Xie, K\. Yang, X\. An, K\. Wu, Y\. Zhao, W\. Deng, Z\. Ran, Y\. Wang, Z\. Feng, R\. Miles,et al\.\(2025\)Region\-based cluster discrimination for visual representation learning\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 1793–1803\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu,et al\.\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.08107#S5.SS1.SSS1.p1.2)\.
- L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. Liu \(2023\)Metamath: bootstrap your own mathematical questions for large language models\.arXiv preprint arXiv:2309\.12284\.Cited by:[§C\.1](https://arxiv.org/html/2608.08107#A3.SS1.p1.2)\.
- Z\. Yu and S\. Ananiadou \(2025\)Locate\-then\-merge: neuron\-level parameter fusion for mitigating catastrophic forgetting in multimodal llms\.arXiv preprint arXiv:2505\.16703\.Cited by:[§C\.3](https://arxiv.org/html/2608.08107#A3.SS3.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2608.08107#S5.SS1.SSS2.p1.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.\(2024\)Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9556–9567\.Cited by:[§1](https://arxiv.org/html/2608.08107#S1.p1.1),[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- K\. Zhang, B\. Li, P\. Zhang, F\. Pu, J\. A\. Cahyono, K\. Hu, S\. Liu, Y\. Zhang, J\. Yang, C\. Li,et al\.\(2025\)Lmms\-eval: reality check on the evaluation of large multimodal models\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 881–916\.Cited by:[§5\.1\.3](https://arxiv.org/html/2608.08107#S5.SS1.SSS3.p1.1)\.
- Y\. Zhang, S\. Lu, Y\. Li, Y\. Ma, Q\. Chen, Z\. Xu, W\. Luo, K\. Zhang, D\. Zhan, and H\. Ye \(2024\)Wings: learning multimodal llms without text\-only forgetting\.Advances in Neural Information Processing Systems37,pp\. 31828–31853\.Cited by:[§C\.3](https://arxiv.org/html/2608.08107#A3.SS3.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.08107#S1.p4.1),[§2\.2](https://arxiv.org/html/2608.08107#S2.SS2.p1.1),[§5\.1\.2](https://arxiv.org/html/2608.08107#S5.SS1.SSS2.p1.1)\.
- X\. Zhao, B\. Schuller, and B\. Sisman \(2026\)Discovering and causally validating emotion\-sensitive neurons in large audio\-language models\.arXiv preprint arXiv:2601\.03115\.Cited by:[§B\.1](https://arxiv.org/html/2608.08107#A2.SS1.p1.1)\.
- D\. Zhu, X\. Shen, X\. Li, M\. Elhoseiny,et al\.\(2024\)Minigpt\-4: enhancing vision\-language understanding with advanced large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 18378–18394\.Cited by:[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.
- J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao,et al\.\(2025\)Internvl3: exploring advanced training and test\-time recipes for open\-source multimodal models\.arXiv preprint arXiv:2504\.10479\.Cited by:[§2\.1](https://arxiv.org/html/2608.08107#S2.SS1.p1.1)\.

## Appendix AMethod Details

### A\.1Overall Algorithm

The complete pseudocode of our NeuPAT method is shown in Algorithm[1](https://arxiv.org/html/2608.08107#alg1)\.

Algorithm 1NeuPAT Training Procedure1:Aligned model

Θ0\\Theta^\{0\}, probing sets

𝒟T,𝒟V\\mathcal\{D\}\_\{T\},\\mathcal\{D\}\_\{V\}, training set

𝒟ori\\mathcal\{D\}\_\{\\mathrm\{ori\}\}, thresholds

τT,τV\\tau\_\{T\},\\tau\_\{V\}
2:Tuned model

Θ\\Theta
3:Compute text and vision response scores

\{sl,uT,sl,uV\}\\\{s^\{T\}\_\{l,u\},s^\{V\}\_\{l,u\}\\\}using

𝒟T\\mathcal\{D\}\_\{T\}and

𝒟V\\mathcal\{D\}\_\{V\}
4:foreach layer

lldo

5:Select important sets

𝒯l\\mathcal\{T\}\_\{l\}and

𝒱l\\mathcal\{V\}\_\{l\}using cumulative importance thresholds

τT\\tau\_\{T\}and

τV\\tau\_\{V\}
6:

𝒞lt​e​x​t←𝒯l∖𝒱l\\mathcal\{C\}^\{text\}\_\{l\}\\leftarrow\\mathcal\{T\}\_\{l\}\\setminus\\mathcal\{V\}\_\{l\}
7:

𝒞lv​i​s​i​o​n←𝒱l∖𝒯l\\mathcal\{C\}^\{vision\}\_\{l\}\\leftarrow\\mathcal\{V\}\_\{l\}\\setminus\\mathcal\{T\}\_\{l\}
8:

𝒞lh​i​g​h←𝒯l∩𝒱l\\mathcal\{C\}^\{high\}\_\{l\}\\leftarrow\\mathcal\{T\}\_\{l\}\\cap\\mathcal\{V\}\_\{l\}
9:

𝒞ll​o​w←𝒯l∪𝒱l¯\\mathcal\{C\}^\{low\}\_\{l\}\\leftarrow\\overline\{\\mathcal\{T\}\_\{l\}\\cup\\mathcal\{V\}\_\{l\}\}
10:endfor

11:

Θ←Θ0\\Theta\\leftarrow\\Theta^\{0\}
12:foreach minibatch

ℬ⊂𝒟ori\\mathcal\{B\}\\subset\\mathcal\{D\}\_\{\\mathrm\{ori\}\}do

13:Compute

ℒori\\mathcal\{L\}\_\{\\mathrm\{ori\}\}and

ℒ=ℒori\+ℛhigh\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{ori\}\}\+\\mathcal\{R\}\_\{\\mathrm\{high\}\}
14:Freeze text neurons, update vision and low response neurons with

ℒori\\mathcal\{L\}\_\{\\mathrm\{ori\}\}, and update high response neurons with

ℒ\\mathcal\{L\}
15:endfor

16:return

Θ\\Theta

Table 7:Training configurations for the two\-stage pipeline\.ConfigurationStage 1:AlignmentStage 2:Instruction TuningDatasetLLaVA\-558KLLaVA\-NeXT\-780KTraining steps2,5003,500Trainable modulesAdapter onlyLLM, adapter,and vision encoderGlobal batch size8224Micro\-batch size11Gradient accumulation128Number of GPUs88Peak learning rate1×10−41\\times 10^\{\-4\}1×10−51\\times 10^\{\-5\}Minimum learning rate1×10−61\\times 10^\{\-6\}1×10−61\\times 10^\{\-6\}Warmup ratio0\.0020\.002OptimizerAdam \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99\\beta\_\{2\}=0\.99,ϵ=10−5\\epsilon=10^\{\-5\}\)LR schedulerCosine decayWeight decay0Gradient clipping1\.0PrecisionBF16Sequence length32,768Tensor / pipeline parallelism1 / 1Image resolutionDefault1,000Offline packingEnabledDisabled

Table 8:Sensitivity analysis of the target importance massτa\\tau\_\{a\}on language and multimodal benchmarks\. The selected thresholdτa=0\.8\\tau\_\{a\}=0\.8is highlighted\.𝝉𝒂\\boldsymbol\{\\tau\_\{a\}\}Language BenchmarksMultimodal BenchmarksSIQAARC\-CGPQAMATHGSM8KLogiQA2BBHbAbIMBPPDROPCoQAAvg\.MMBRWQAMMMUPOPEMMStarAvg\.0\.649\.5456\.5735\.2754\.4084\.8436\.7753\.465\.8765\.4014\.5769\.1247\.8073\.2857\.6543\.2287\.5946\.2061\.590\.750\.5156\.4036\.1655\.6085\.1435\.3753\.1711\.8866\.6014\.3769\.8548\.6472\.5157\.5243\.6787\.0445\.5961\.27\\rowcolorgray\!10 0\.850\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.050\.949\.6955\.9735\.7155\.4086\.4335\.6953\.5113\.9967\.8015\.1170\.6349\.0871\.3954\.7743\.1186\.7343\.4759\.89

Table 9:Sensitivity analysis of the size of probing set on language and multimodal benchmarks\. The selected size20482048is highlighted\.𝑵𝒂\\boldsymbol\{N\_\{a\}\}Language BenchmarksMultimodal BenchmarksSIQAARC\-CGPQAMATHGSM8KLogiQA2BBHbAbIMBPPDROPCoQAAvg\.MMBRWQAMMMUPOPEMMStarAvg\.51250\.0557\.0835\.9453\.4085\.9736\.2052\.8510\.4167\.0015\.1870\.4548\.5972\.3957\.2541\.3387\.0046\.1760\.83102450\.3656\.8335\.9452\.0084\.7636\.0754\.1214\.8967\.0014\.1770\.3248\.7771\.9956\.7343\.1187\.2844\.4760\.72\\rowcolorgray\!10 204850\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.05409650\.5656\.4035\.9452\.2285\.4437\.1553\.3911\.1667\.2013\.8970\.6248\.5470\.8856\.9942\.5687\.1345\.8960\.69

Table 10:Sensitivity analysis of the high response neuron regularization coefficientsλin\\lambda\_\{\\mathrm\{in\}\}andλout\\lambda\_\{\\mathrm\{out\}\}\. The selected settingλin=λout=0\.1\\lambda\_\{\\mathrm\{in\}\}=\\lambda\_\{\\mathrm\{out\}\}=0\.1is highlighted\.𝝀𝐢𝐧=𝝀𝐨𝐮𝐭\\boldsymbol\{\\lambda\_\{\\mathrm\{in\}\}=\\lambda\_\{\\mathrm\{out\}\}\}Language BenchmarksMultimodal BenchmarksSIQAARC\-CGPQAMATHGSM8KLogiQA2BBHbAbIMBPPDROPCoQAAvg\.MMBRWQAMMMUPOPEMMStarAvg\.0\.0550\.2657\.4236\.6152\.6085\.9734\.4152\.0210\.0967\.0015\.5869\.7048\.3372\.6857\.9142\.1187\.1946\.2961\.24\\rowcolorgray\!10 0\.150\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.050\.550\.3156\.5737\.2854\.2084\.0836\.2652\.7913\.8668\.0014\.7570\.7848\.9971\.6556\.2143\.0087\.4145\.8260\.821\.050\.4156\.1437\.0555\.6085\.9036\.2652\.3013\.3866\.6013\.7170\.8548\.9371\.6555\.9543\.2287\.4144\.8760\.62

## Appendix BAdditional Related Work

### B\.1Neuron\-Level Analysis

Recent studies have investigated the functional specialization of individual neurons in language and multimodal models\. Fu et al\.Fuet al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib32)\)reveal that knowledge in MLLMs can be distributed across decoupled modality\-specific neuron pathways and localize these neurons to improve knowledge editing under different modality inputs\. Zhao et al\.Zhaoet al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib33)\)identify emotion\-sensitive neurons in large audio\-language models and causally validate their roles through neuron suppression and activation steering\. For multimodal unlearning, Liu et al\.Liuet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib34)\)locate neurons according to their relative importance across modalities and selectively prune them to remove targeted knowledge while preserving general model utility\. MNAFTLiet al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib6)\)identifies modality\-relevant neurons using activation and gradient information and selectively fine\-tunes them for image translation to reduce interference across languages and modalities\. Qin et al\.Qinet al\.\([2025](https://arxiv.org/html/2608.08107#bib.bib35)\)further show that LLMs contain ultra\-sparse critical neurons whose perturbation can severely impair overall language ability\.

These studies associate neuron\-level structures with modality\-specific knowledge, task behaviors, and fundamental model capabilities, but primarily focus on knowledge editing, emotion control, unlearning, task\-specific image translation, or vulnerability analysis\. In contrast, we study neuron\-wise adaptation dynamics during general multimodal learning\. NeuPAT uses lightweight text and vision probing to estimate heterogeneous neuron responses, followed by neuron adaptation role allocation and role\-aware multimodal tuning\. By protecting language\-sensitive neurons, promoting adaptation through vision\-responsive and underutilized neurons, and regularizing neurons important to both text and vision, NeuPAT preserves pretrained language capabilities while supporting multimodal learning\.

## Appendix CDetailed Experimental Setup

### C\.1Probing Sets

We construct separate text\-only and vision probing sets to estimate modality\-specific neuron responses\. The vision probing set containsNV=2048N\_\{V\}=2048samples randomly drawn from LLaVA\-NeXT\-780KLiuet al\.\([2024b](https://arxiv.org/html/2608.08107#bib.bib42)\)\. The text probing set containsNT=2048N\_\{T\}=2048prompts sampled from publicly available general\-domain text datasets that are disjoint from all evaluation benchmarks\. To improve coverage, we include prompts from instruction\-following, factuality, and general reasoning domains, including CodeAlpaca\-20kChaudhary \([2023](https://arxiv.org/html/2608.08107#bib.bib66)\), MetaMathQAYuet al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib67)\), databricks\-dolly\-15kConoveret al\.\([2023](https://arxiv.org/html/2608.08107#bib.bib68)\)and HaluEvalLiet al\.\([2023b](https://arxiv.org/html/2608.08107#bib.bib69)\)\. We allocate samples evenly across data sources\. When a source contains fewer samples than its assigned quota, the remaining quota is redistributed among the other sources\.

Both probing sets are used solely for forward\-pass activation statistics\. They do not contribute to the training objective or parameter updates\. We use the same processor and chat\-template pipeline for both sets, while providing images only for the vision probing samples\.

### C\.2Training Details

As shown in Table[7](https://arxiv.org/html/2608.08107#A1.T7), we follow a two\-stage training pipeline\. In Stage 1, we perform image\-text alignment on LLaVA\-558K for 2,500 steps, where only the multimodal adapter is optimized\. We use a global batch size of 8 on 8 A100 GPUs, with a micro\-batch size of 1\. The peak learning rate is1×10−41\\times 10^\{\-4\}and is decayed to1×10−61\\times 10^\{\-6\}using a cosine schedule\. The warmup ratio is set to0\.0020\.002, corresponding to approximately 5 warmup steps\.

In Stage 2, we conduct visual instruction tuning on LLaVA\-NeXT\-780K for 3,500 steps\. The language model, multimodal adapter, and vision encoder are included in optimization, while NeuPAT applies plasticity\-guided multimodal tuning strategies to the neurons in the language backbone\. Training is performed on 8 A100 GPUs with a global batch size of 224, a micro\-batch size of 1, and 28 gradient\-accumulation steps\. The peak learning rate is1×10−51\\times 10^\{\-5\}, with the same minimum learning rate of1×10−61\\times 10^\{\-6\}and a warmup ratio of0\.0020\.002\. Both stages use Adam withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99\\beta\_\{2\}=0\.99, andϵ=10−5\\epsilon=10^\{\-5\}, together with cosine learning\-rate decay, zero weight decay, and gradient clipping at 1\.0\.

### C\.3Baseline Implementations

##### LoRA\.

LoRAHuet al\.\([2022](https://arxiv.org/html/2608.08107#bib.bib13)\)inserts trainable low\-rank adapters into selected linear layers while freezing the original parameters\. We set the rank tor=32r=32, the scaling factor toα=64\\alpha=64, and the learning rate to1×10−41\\times 10^\{\-4\}\. The model is trained on the same multimodal instruction data as Vanilla Tuning\.

##### EWC\.

EWCKirkpatricket al\.\([2017](https://arxiv.org/html/2608.08107#bib.bib21)\)estimates parameter importance using the diagonal Fisher information and penalizes changes to important parameters\. We estimate the Fisher information at the shared Stage\-1 checkpoint using 2,048 text\-only instruction samples that are disjoint from all evaluation benchmarks\. The Fisher statistics are averaged over samples, and the EWC penalty is applied only to the language\-model parameters\. We searchλEWC\\lambda\_\{\\mathrm\{EWC\}\}over\{0\.1,1,10,100\}\\\{0\.1,1,10,100\\\}and selectλEWC=10\\lambda\_\{\\mathrm\{EWC\}\}=\\text\{10\}according to the text\-multimodal performance\. All remaining training settings are identical to Vanilla Tuning\.

##### WINGS\.

WINGSZhanget al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib10)\)introduces parallel visual and textual learners into attention layers to reduce over\-reliance on visual tokens\. Their outputs are fused with the original attention output through a learned router\. We reproduce WINGS using the architecture and hyperparameter settings recommended in the original paper\.

##### TIES\.

TIESRatzlaffet al\.\([2024](https://arxiv.org/html/2608.08107#bib.bib11)\)is a training\-free merging method that sparsifies the task vector between the visually tuned model and the original LLM\. We follow the original paper for the task\-vector density, merge coefficient, and merging procedure\.

##### Locate\-then\-Merge\.

Locate\-then\-MergeYu and Ananiadou \([2025](https://arxiv.org/html/2608.08107#bib.bib24)\)identifies high\-impact neurons from parameter changes, suppresses low\-impact updates, and restores selected neurons through replacement or rescaling\. We follow the original paper for the neuron\-retention ratio, parameter\-sparsification ratio, restoration strategy, and associated coefficients\. The hyperparameters are selected from the recommended ranges reported by the authors\.

##### PlaM\.

PlaMWanget al\.\([2026](https://arxiv.org/html/2608.08107#bib.bib25)\)locates a plateau layer through layer\-wise vision\-token masking and merges subsequent layers with the original LLM\. We follow the original paper to determine the plateau layer and search the merge coefficients within its recommended range\. Earlier visual\-alignment layers remain unchanged, while the selected later layers are linearly merged with the original language backbone\.

Table 11:Complete cross\-backbone results on 11 language and 5 multimodal benchmarks\. “LLM” denotes the original language backbone before multimodal training\.TaskQwen3\-0\.6BPhi\-4\-Mini\-InstructQwen2\.5\-7B\-InstructLlama3\.1\-8B\-InstructQwen2\.5\-14B\-InstructLLMVanilla TuningNeuPATLLMVanilla TuningNeuPATLLMVanilla TuningNeuPATLLMVanilla TuningNeuPATLLMVanilla TuningNeuPATLanguage BenchmarksMATH\-50014\.0013\.8013\.4039\.2032\.0034\.0044\.4028\.0035\.2036\.8028\.8035\.8043\.4025\.6039\.00BBH33\.3928\.6933\.4452\.8240\.0451\.2945\.8141\.9847\.5344\.5934\.9141\.6852\.5439\.8752\.59bAbI2\.501\.932\.212\.651\.062\.782\.710\.602\.560\.000\.140\.811\.170\.031\.48MBPP27\.4022\.6028\.2055\.4047\.6053\.8047\.6040\.2042\.8058\.4054\.2056\.8066\.8065\.6066\.80LogiQA230\.3428\.7530\.7936\.0730\.7935\.0840\.2035\.1840\.5238\.3631\.8737\.4542\.8837\.4740\.14GSM8K40\.5635\.1840\.1183\.6273\.9281\.4576\.5074\.8378\.7078\.1767\.5574\.5579\.8379\.3085\.44DROP12\.326\.129\.8016\.0113\.3916\.9416\.227\.1311\.6912\.057\.1511\.5320\.6020\.6026\.41CoQA57\.5755\.9359\.4278\.4065\.2380\.0478\.7473\.6378\.8077\.8377\.5080\.4878\.2074\.4477\.51SocialIQA40\.3839\.0040\.3349\.5946\.1150\.0651\.5950\.2055\.8949\.8549\.7449\.9554\.0451\.2355\.22GPQA29\.6927\.9029\.1230\.5830\.1331\.7034\.3832\.3734\.3834\.6031\.0333\.8136\.8335\.0436\.16ARC\-C34\.3934\.0436\.0958\.4558\.3658\.7055\.2952\.6554\.1053\.4153\.2453\.3360\.4160\.4161\.26Text Avg\.29\.3226\.7229\.3645\.7139\.8845\.0844\.8639\.7143\.8344\.0139\.6543\.2948\.7944\.5149\.27Multimodal BenchmarksMMBench\-EN–52\.5852\.52–50\.5250\.89–64\.7864\.64–60\.1461\.94–75\.1774\.31RealWorldQA–45\.1044\.75–45\.1044\.31–52\.0352\.01–26\.6744\.97–56\.3455\.64MMMU–30\.6732\.67–36\.7836\.22–41\.0042\.56–35\.8936\.22–47\.6748\.11POPE–85\.4284\.26–77\.1677\.82–87\.1286\.54–82\.2380\.90–87\.2787\.67MMStar–37\.2337\.07–30\.5831\.26–40\.5640\.37–36\.5433\.94–50\.2650\.14MM Avg\.–50\.2050\.25–48\.0348\.10–57\.1057\.22–48\.2951\.59–63\.3463\.17

## Appendix DAdditional Experimental Results

### D\.1Sensitivity Analysis

All sensitivity analyses are conducted after fixing the default configuration used in the main experiments\. We vary one hyperparameter at a time while keeping all other settings unchanged\. These experiments are intended to evaluate robustness\.

##### Target Importance Massτa\\tau\_\{a\}\.

We analyze the sensitivity to the target importance massτa\\tau\_\{a\}used in neuron adaptation role allocation\. As shown in Table[8](https://arxiv.org/html/2608.08107#A1.T8), language performance generally improves asτa\\tau\_\{a\}increases\. A largerτa\\tau\_\{a\}retains more cumulative response mass for each modality, thereby expanding the selected neuron sets and reducing the proportion of low response neurons\. This tends to assign more neurons to text\-related or high response neurons, strengthening language preservation but leaving less unconstrained capacity for multimodal adaptation\. Consequently, multimodal performance gradually declines at larger values ofτa\\tau\_\{a\}\. The results support our default choice ofτa=0\.8\\tau\_\{a\}=0\.8, which achieves a favorable balance between language preservation and multimodal adaptation\.

##### Probing Set Size\.

We further examine the sensitivity of NeuPAT to the number of samples in each probing set\. As shown in Table[9](https://arxiv.org/html/2608.08107#A1.T9), the overall performance is relatively stable across different sizes, with variations of only0\.520\.52and0\.360\.36points in the text and multimodal averages, respectively\. IncreasingNaN\_\{a\}from512512to20482048generally improves performance, andNa=2048N\_\{a\}=2048achieves the best averages on both language and multimodal benchmarks\. Further increasing the size to40964096provides no additional benefit, indicating diminishing returns from additional probing samples\. The results supportNa=2048N\_\{a\}=2048as a reasonable default while showing that NeuPAT is relatively insensitive to the probing\-set size\.

##### High Response Neuron Regularization Coefficients\.

We study the sensitivity to the high response neuron regularization coefficients by settingλin=λout\\lambda\_\{\\mathrm\{in\}\}=\\lambda\_\{\\mathrm\{out\}\}\. As shown in Table[10](https://arxiv.org/html/2608.08107#A1.T10), a weak constraint of0\.050\.05achieves the highest multimodal average but lower language performance\. Increasing the coefficients beyond0\.10\.1provides no further text improvement and gradually reduces multimodal performance\. These results support our default setting ofλin=λout=0\.1\\lambda\_\{\\mathrm\{in\}\}=\\lambda\_\{\\mathrm\{out\}\}=0\.1, which provides a favorable text\-multimodal trade\-off\.

Table 12:Complete ablation results on 11 language and 5 multimodal benchmarks\. Variants are grouped by global update strategy, neuron allocation, and neuron\-wise plasticity allocation strategies\. The complete NeuPAT configuration is repeated and highlighted in each group for comparison\.VariantLanguage BenchmarksMultimodal BenchmarksSIQAARC\-CGPQAMATHGSM8KLogiQA2BBHbAbIMBPPDROPCoQAAvg\.MMBRWQAMMMUPOPEMMStarAvg\.\(a\) Global vs\. Neuron\-Aware UpdateGlobal Freeze49\.3356\.0637\.7261\.2086\.0534\.2953\.5713\.6567\.1014\.9070\.3749\.4868\.6453\.9941\.3386\.0343\.0858\.61Uniform Update \(Vanilla Tuning\)46\.7855\.7234\.1550\.4083\.4732\.9537\.122\.6862\.609\.9067\.3743\.9272\.0856\.6043\.1187\.6945\.4060\.98Global Reg\.49\.3356\.1436\.8354\.4084\.9934\.7355\.239\.3365\.509\.6870\.1547\.8572\.0854\.2542\.5686\.0343\.9159\.77\\rowcolorgray\!10NeuPAT50\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.05\(b\) Neuron Allocation StrategyRandom Partition49\.3355\.5536\.1655\.4083\.6934\.4154\.321\.9465\.409\.6769\.4246\.8469\.7654\.5143\.3387\.5244\.4759\.92Fixed\-ratio Partition50\.2656\.2335\.4953\.2084\.9035\.9452\.8510\.2466\.2010\.1169\.8547\.7572\.1657\.2541\.3387\.2843\.2660\.26\\rowcolorgray\!10NeuPAT50\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.05\(c\) Neuron\-Wise Update Constraintw/o Text Freeze49\.8055\.2935\.7152\.6083\.9036\.6452\.965\.8764\.8010\.3970\.2347\.1172\.1157\.3042\.3387\.2045\.6860\.92w/o Vision Update50\.4155\.6336\.1653\.2084\.4436\.4553\.9911\.8567\.1012\.7770\.2048\.3869\.6955\.1642\.4486\.6144\.0459\.59w/o Low Response Update49\.7456\.3137\.2853\.0084\.7635\.8853\.8911\.6466\.809\.9670\.1348\.1371\.0556\.3442\.6786\.9445\.7260\.54High Response Full Update48\.4155\.4634\.1553\.6083\.6137\.4752\.409\.9464\.0011\.9967\.8247\.1772\.3457\.2542\.7887\.2744\.7260\.87High Response Freeze49\.9056\.6637\.0555\.8085\.0635\.3753\.2915\.1667\.2013\.7070\.4849\.0669\.9355\.0342\.6787\.3045\.1060\.01\\rowcolorgray\!10NeuPAT50\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.05\(d\) High Response Neuron Regularization Designl2\-l250\.0055\.5737\.9555\.2085\.2236\.1352\.9711\.4166\.2013\.0669\.4548\.4772\.2057\.6542\.0087\.8446\.0761\.15cos\-cos50\.0556\.6637\.2854\.2084\.9936\.5152\.348\.4467\.4014\.5169\.6248\.3672\.1656\.9942\.7887\.3744\.5060\.76cos\-l250\.0056\.9136\.3854\.6085\.2236\.0753\.2615\.6067\.0013\.4069\.7548\.9372\.5155\.2943\.4487\.1445\.2660\.73\\rowcolorgray\!10NeuPAT \(l2\-cos\)50\.2656\.9137\.2756\.4085\.9736\.3953\.8810\.6766\.6015\.7469\.5349\.0672\.4856\.9942\.2288\.7244\.8561\.05

![Refer to caption](https://arxiv.org/html/2608.08107v1/x3.png)Figure 5:Layer\-wise neuron distributions across LLM backbones\.High response neurons form the largest group in most layers, while the others vary across models and depths\.

### D\.2Complete Cross\-Backbone Results

Due to space limitations, the main text reports results on five representative language benchmarks\. Table[11](https://arxiv.org/html/2608.08107#A3.T11)provides the complete results on all 11 language and 5 multimodal benchmarks\. Vanilla Tuning consistently degrades the average language performance across all tested language backbones\. In contrast, NeuPAT improves over Vanilla Tuning by2\.642\.64,5\.205\.20,4\.124\.12,3\.643\.64, and4\.764\.76points on Qwen3\-0\.6B, Phi\-4\-Mini\-Instruct, Qwen2\.5\-7B\-Instruct, Llama3\.1\-8B\-Instruct, and Qwen2\.5\-14B\-Instruct, respectively\. It also recovers the original LLM average within approximately one point for all backbones and slightly surpasses it on Qwen3\-0\.6B and Qwen2\.5\-14B\-Instruct\. Meanwhile, multimodal performance remains comparable to Vanilla Tuning, with particularly notable gains on Llama3\.1\-8B\-Instruct\. Although the improvements vary across individual tasks, the overall results confirm that NeuPAT generalizes across different model families and scales\.

### D\.3Complete Ablation Results

##### Complete Results for the Main\-Text Ablations\.

Table[12](https://arxiv.org/html/2608.08107#A4.T12)reports the complete results for the three ablation groups presented in the main text\. Global update strategies reveal the trade\-off between language preservation and multimodal adaptation\. Alternative partitioning strategies verify the importance of neuron adaptation role allocation, while the update\-strategy ablations demonstrate the role of each neuron set\. The complete results are consistent with the conclusions drawn from the representative benchmarks in the main text\.

##### High Response Neuron Regularization Design\.

We further compare different input\- and output\-side regularization combinations for high response neurons\. As shown in Table[12](https://arxiv.org/html/2608.08107#A4.T12)\(d\), l2\-l2 achieves slightly higher multimodal performance but lower text performance, while cos\-cos and cos\-l2 yield weaker overall trade\-offs\. The proposed l2\-cos design obtains the highest text average while maintaining near\-best multimodal performance, providing the most balanced result\. This suggests that input\-side parameters benefit from magnitude constraints, whereas output\-side parameters are better regularized by preserving their transformation directions\.

### D\.4Additional Neuron Visualizations

#### D\.4\.1Neuron Distribution across LLM Backbones

Figure[5](https://arxiv.org/html/2608.08107#A4.F5)compares the layer\-wise distributions of the neuron sets identified by importance\-guided allocation across different LLM backbones\. Although their proportions vary across model families, scales, and layers, text\-critical, vision\-critical, high response, and low response neurons consistently coexist throughout the backbone, with high response neurons generally forming the largest set\. Some models exhibit stronger fluctuations in early layers, while later layers tend to show more stable distributions\. These consistent yet heterogeneous patterns suggest that neuron\-wise adaptation dynamics generalize across backbones, supporting the architecture\-agnostic applicability of NeuPAT\.