Decoupled Alignment for Robust Plug-and-Play Adaptation

# Decoupled Alignment for Robust Plug-and-Play Adaptation
Source: [https://arxiv.org/html/2406.01514](https://arxiv.org/html/2406.01514)
\*\*footnotetext:These authors contributed equally to this work\.Haozheng Luo1∗Jiahao Yu2∗Wenxin Zhang1∗Jialong Li3&Chenghao Qiu4Yimin Wang5Hanchen Jiang5Jerry Yao\-Chieh Hu1&Yan Chen1Binghui Wang6Xinyu Xing1Han Liu1

1Northwestern University2New York University Abu Dhabi3Stanford University 4Texas A&M University5University of California, Los Angeles6Illinois Institute of Technology \{[hluo](https://arxiv.org/html/2406.01514v5/mailto:[email protected]),[wenxinzhang2025](https://arxiv.org/html/2406.01514v5/mailto:[email protected]),[jialongli2024](https://arxiv.org/html/2406.01514v5/mailto:[email protected]),[jhu](https://arxiv.org/html/2406.01514v5/mailto:[email protected])\}@u\.northwestern\.edu [jy5951](https://arxiv.org/html/2406.01514v5/mailto:[email protected])@nyu\.edu[chenghaoqiu](https://arxiv.org/html/2406.01514v5/mailto:[email protected])@tamu\.edu[bwang70](https://arxiv.org/html/2406.01514v5/mailto:[email protected])@illinoistech\.edu \{[yiminw](https://arxiv.org/html/2406.01514v5/mailto:[email protected]),[ericjiang0318](https://arxiv.org/html/2406.01514v5/mailto:[email protected])\}@ucla\.edu \{[hanliu](https://arxiv.org/html/2406.01514v5/mailto:[email protected]),[xinyu\.xing](https://arxiv.org/html/2406.01514v5/mailto:[email protected]),[ychen](https://arxiv.org/html/2406.01514v5/mailto:[email protected])\}@northwestern\.edu

###### Abstract

Content Warning: This paper contains examples of harmful language\.

We introduce a training\-free safety enhancement method for aligning large language models \(LLMs\) without the need of supervised fine\-tuning or reinforcement learning from human feedback\. Our main idea is to provide a robust plug\-and\-play approach to prevent shadow alignment when models are adapted to downstream tasks\. Specifically, we leverage knowledge distillation to extract alignment signals from well\-aligned LLMs and inject them into shadow\-aligned models via model fusion, enabling a plug\-and\-play alignment correction\. In our methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation\. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately14\.42%, reaching as high as 51\.39%, in 17 influenced LLMs, without compromising performance\. Our code is available at[https://github\.com/NWULIST/DAPA](https://github.com/NWULIST/DAPA)\.

### 1Introduction

With large language models \(LLMs\) increasingly adopted across a wide range of applications, their ability to generate high\-quality, human\-like text has been well demonstrated\(Guoet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib39); Grattafioriet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib38); Yanget al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib37)\)\. However, growing concerns have emerged over their potential to generate harmful content\(Shaoet al\.,[2026](https://arxiv.org/html/2406.01514#bib.bib60); Rameshet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib109); Chenet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib58); Yuet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib57);[2024a](https://arxiv.org/html/2406.01514#bib.bib111); Chaoet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib108)\)\. To address these concerns, several methods have been developed to enhance model alignment with ethical and safety guidelines\. For instance, models such as Llama\-2\-Chat\(Touvronet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib95)\)and Gemma\-it\(Teamet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib71)\)have undergone extensive fine\-tuning to improve their alignment performance\. However, such approaches often rely heavily on*computationally intensive resources*and*manual red\-teaming*, making them expensive and time\-consuming\(Qiet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib104); Huet al\.,[2024d](https://arxiv.org/html/2406.01514#bib.bib53); OpenAI,[2024](https://arxiv.org/html/2406.01514#bib.bib96)\)\.

Consequently, with the rapid development of LLMs, many third\-party developers increasingly rely on existing well\-aligned base models to build specialized systems for reasoning\(Guoet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib39); Agarwalet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib18)\), vision\(Liuet al\.,[2024a](https://arxiv.org/html/2406.01514#bib.bib24); Baiet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib27)\), and speech\(Wuet al\.,[2024c](https://arxiv.org/html/2406.01514#bib.bib22); Chuet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib21)\)\. However, recent studies\(Hsuet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib17); Lermen and Rogers\-Smith,[2024](https://arxiv.org/html/2406.01514#bib.bib16); Yanget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib15)\)show that adaptation methods—such as fine\-tuning, LoRA insertion, or task\-specific alignment—can inadvertently erode or override the original safety alignment of these base models, re\-introducing harmful or unaligned behaviors\. We refer to this degradation of safety alignment caused by downstream adaptation asshadow alignment\(Yanget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib15)\), which often forces developers to perform additional realignment to restore model safety\.

To address this challenge, we introduceDecoupledAlignment for RobustPlug\-and\-PlayAdaptation \(termedDapa\), a*training\-free*safety enhancement framework for aligning LLMs\. We mainly draw inspiration from knowledge distillation\(Uppaalet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib105); Grimeset al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib76); Mansourianet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib74); Hahn and Choi,[2019](https://arxiv.org/html/2406.01514#bib.bib73)\), wherein knowledge is transferred from a teacher model to a student model\. Given access to a single*well\-aligned*teacher model, DAPA aligns other*shadow alignment influenced*LLMs within the same model family to ethical guidelines, without requiring supervised fine\-tuning \(SFT\) or reinforcement learning from human feedback \(RLHF\)\.

Specifically, through a series of numerical experiments \(see[Figure 2](https://arxiv.org/html/2406.01514#S1.F2)and[Figure 7](https://arxiv.org/html/2406.01514#A3.F7)\), we first uncover two key findings:

- •MLP Alignment:Alignment knowledge is predominantly stored in the Feed\-Forward Network \(FFN\), or MLP layers\.
- •Gate Alignment:Within these MLP layers, the gate layers play a crucial role in determining whether model outputs align with ethical constraints\.

Then, building on these insights, we propose to leverage memory editing techniques\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93);[b](https://arxiv.org/html/2406.01514#bib.bib92)\)to transfer alignment knowledge from a well\-aligned LLM to an influenced counterpart within the same model family\. In particular, we first present a delta debugging\-based search algorithm to address the challenge of pinpointing the memory space \(gate layers\) responsible for alignment performance\. This allows us to locate the alignment\-related modules for memory editing via knowledge distillation\. We then apply model fusion \(see[Section˜3\.2](https://arxiv.org/html/2406.01514#S3.SS2)\), transferring alignment\-related modules from an aligned model into the target model to achieve a low\-cost yet effective safety improvement\.

We extensively evaluateDapaon 17 LLMs from three popular families \(Llama2, Mistral, and Gemma\) on various metrics including cosine similarity scores, model perplexity, few\-shot prompting, and Chain\-of\-Thought \(CoT\)\. Our results showDapa\-aligned LLMs have an average14\.42%increase in defense success rate, with minimal computational effort \(adaptingat most8\.11% model parameters\) and marginally affecting the model’s benign functionality—e\.g\., the averageincreasein perplexity is only 1\.69, and the average drop in model reasoning ability is only 2\.59%\. These results indicate thatDapaoffers a timely, robust, and economic solution for enhancing LLM safety, enabling more efficient and accessible alignment across the open LLM ecosystem\.

###### Contribution\.

Our main contributions are summarized below\.

- •We design a novel safety enhancement method,Dapa\(as shown in[Figure 15](https://arxiv.org/html/2406.01514#A11.F15)\), for realigning LLMs affected by shadow alignment\.Dapautilizes memory editing technology to identify the memory space responsible for alignment performance\. Unlike prior alignment strategies, DAPA requires*neither intensive computation nor manual intervention*such as red\-teaming or fine\-tuning\(Daiet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib83)\)\.
- •We present a latent\-space analysis \([Section˜2](https://arxiv.org/html/2406.01514#S2)\) that sheds light on shadow alignment, and introduce a model\-fusion–based knowledge editing method that improves safety without degrading task performance, supported by a detailed theoretical analysis\.
- •Extensive experimental results validate the effectiveness, robustness, and efficiency of DAPA to enhance LLM safety alignment\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x1.png)Figure 1:The Transformer Architecture\.We describe the architecture of Transformer utilized by state\-of\-the\-art LLMs such as Llama\(Touvronet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib95)\)and Gemma\(Teamet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib71)\)\. Each Transformer block combines an attention mechanism with MLP layers \(comprising Up, Gate, and Down modules\)\. This figure illustrates the transition of the model’s hidden representation from the previous state to the next state\.![Refer to caption](https://arxiv.org/html/2406.01514v5/x2.png)Figure 2:Visualizing Attention, MLP, and All Modules on Memory Space\.We visualize the influence of unethical prompt tokens on the results using the aligned Llama\-2\-7B\-chat model to identify memory space\. This includes examining the effects on attention, MLP, and all modules\.

### 2Memory Editing

We consider the popular autoregressive LLM that generates text by predicting the next token in a sequence given the previous tokens\. To locate the association of ethical memory within the parameters of an autoregressive LLM, we begin by analyzing and identifying the hidden states that exhibit the strongest correlation with this concept\. Here, ethical memory refers to the subset of internal representations—specifically, model neurons—that store safety\-relevant information, enabling the model to produce morally aligned and socially responsible outputs\.

Denote a sequence of tokens as\{𝐬1,𝐬2,…,𝐬T\}\\\{\\mathbf\{s\}\_\{1\},\\mathbf\{s\}\_\{2\},\\dots,\\mathbf\{s\}\_\{T\}\\\}\. In thell\-layer of an autoregressive LLM, the tokens\{𝐬i\}\\\{\\mathbf\{s\}\_\{i\}\\\}are embedded into a sequence of hidden states\{𝐡i\(l\)\}\\\{\\mathbf\{h\}\_\{i\}^\{\(l\)\}\\\}\. The final output of anLL\-layer LLMy=decode​\(𝐡T\(L\)\)y=\\text\{decode\}\(\\mathbf\{h\}\_\{T\}^\{\(L\)\}\)is generated by the decoder layer from the final layer hidden state\. Autoregressive LLMs often use Transformer as the building blocks \(For further background on Transformer, please refer toVaswaniet al\.\([2017](https://arxiv.org/html/2406.01514#bib.bib72)\)\. In[Figure 1](https://arxiv.org/html/2406.01514#S1.F1), we visualize the internal computation of a Transformer block\. Each layerll\(left→\\rightarrowright\) of the Transformer block adds a self\-attention mechanism𝐚i\(l\)\\mathbf\{a\}\_\{i\}^\{\(l\)\}and local MLP𝐌i\(l\)\\mathbf\{M\}\_\{i\}^\{\(l\)\}from previous layers\. Each MLP is a three\-layer neural network parameterized by𝐖up\\mathbf\{W\}\_\{\\text\{up\}\},𝐖gate\\mathbf\{W\}\_\{\\text\{gate\}\}, and𝐖down\\mathbf\{W\}\_\{\\text\{down\}\}, along with a SwiGLU\(Shazeer,[2020](https://arxiv.org/html/2406.01514#bib.bib77)\)or GELU\(Hendrycks and Gimpel,[2016](https://arxiv.org/html/2406.01514#bib.bib78)\)activation function in popular LLMs, such as Llama\(Touvronet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib95)\), Gemma\(Teamet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib71)\)\. Formally, theii\-th layer hidden state for a token𝐬i\\mathbf\{s\}\_\{i\}in Transformer is calculated below:

𝐡i\(l\)=𝐡i\(l−1\)\+𝐚i\(l\)\+𝐌i\(l\),𝐚i\(l\)=attn\(l\)​\(𝐡1\(l−1\),…,𝐡T\(l−1\)\),\\displaystyle\\mathbf\{h\}\_\{i\}^\{\(l\)\}=\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\}\+\\mathbf\{a\}\_\{i\}^\{\(l\)\}\+\\mathbf\{M\}\_\{i\}^\{\(l\)\},\\quad\\mathbf\{a\}\_\{i\}^\{\(l\)\}=\\text\{attn\}^\{\(l\)\}\(\\mathbf\{h\}\_\{1\}^\{\(l\-1\)\},\\dots,\\mathbf\{h\}\_\{T\}^\{\(l\-1\)\}\),𝐌i\(l\)=𝐖down\(l\)​σ​\(𝐖gate\(l\)​γ​\(𝐚i\(l\)\+𝐡i\(l−1\)\)\)⊙𝐖up\(l\)​γ​\(𝐚i\(l\)\+𝐡i\(l−1\)\)\.\\displaystyle\\mathbf\{M\}\_\{i\}^\{\(l\)\}=\\mathbf\{W\}\_\{\\text\{down\}\}^\{\(l\)\}\\sigma\(\\mathbf\{W\}\_\{\\text\{gate\}\}^\{\(l\)\}\\gamma\(\\mathbf\{a\}\_\{i\}^\{\(l\)\}\+\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\}\)\)\\odot\\mathbf\{W\}\_\{\\text\{up\}\}^\{\(l\)\}\\gamma\(\\mathbf\{a\}\_\{i\}^\{\(l\)\}\+\\mathbf\{h\}\_\{i\}^\{\(l\-1\)\}\)\.
![Refer to caption](https://arxiv.org/html/2406.01514v5/x3.png)Figure 3:PCA of safety representations\.We analyze the safety latent space by applying PCA to the last\-token logits of 256 AdvBench samples from Qwen2\.5\-7B \(base\), Qwen2\.5\-7B\-Instruct, and Qwen2\.5\-Math\-1\.5B\-Instruct\.In both projection directions, the safety\-aligned instruct model is clearly separated from the unaligned base model, while the math\-instruct model—obtained by downstream fine\-tuning from the instruct checkpoint—shows a modest shift toward the unsafe region, illustrating the shadow alignment effect\.Storage of Alignment Knowledge\.We first adopt a knowledge\-editing\-style causal tracing analysis\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93)\)to identify the model components that are most*alignment\-sensitive*\. Using an unethical prompt on Llama\-2\-7B\-chat, we corrupt all hidden states as shown in[Figure 1](https://arxiv.org/html/2406.01514#S1.F1), then restore one selected hidden state at a time, and measure the resulting change in output probability relative to the fully corrupted run; we refer to this quantity as the*indirect effect*of that hidden state\. By repeating this procedure across all hidden states, we find that hidden states in the middle layers have the strongest influence on aligned behavior, with MLP layers—especially gate\-related components—showing larger indirect effects than attention layers, consistent with\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93)\)\.These results indicate that the model components most critical for aligned behavior are concentrated in the middle MLP layers\.We provide additional visualizations in[Section˜K\.11](https://arxiv.org/html/2406.01514#A11.SS11)\.

To better understand the impact of each module in the MLP layers towards the alignment knowledge, we customize the knowledge editing technology\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93)\)to visualize the indirect effects of the gate, up, and down projections, as shown in[Appendix˜C](https://arxiv.org/html/2406.01514#A3)\.

Safety Vector Extraction and Representation\.[Figure˜3](https://arxiv.org/html/2406.01514#S2.F3)visualizes the*safety latent space*by applying PCA to the*last\-token logits*of256256harmful prompts from AdvBench\. Compared with the base model, the instruct\-aligned checkpoint occupies a clearly separated region in this space, indicating that safety alignment induces a systematic shift in the representations of harmful inputs\. In contrast, the downstreammath\-instructmodel, despite being initialized from the instruct model, exhibits a noticeable drift toward the region of the unsafe base model\. This suggests that downstream fine\-tuning partially attenuates the safety\-relevant representational shift acquired during alignment\. Consequently, safety is not completely removed but becomes insufficiently preserved, leading to the phenomenon referred to as*shadow alignment*, in which responses to harmful prompts move closer to the pre\-alignment regime\.

This observation reveals a direct geometric interpretation that motivates the subsequentDapadesign\. In particular,Dapais formulated as a two\-stage framework that \(i\) identifies*safety\-critical representation vectors*that capture the alignment\-induced shift, and \(ii\) applies*model fusion*as a lightweight memory\-editing mechanism to re\-inject these safety components into the downstream\-tuned model, thereby restoring safety while limiting interference with downstream capability\. The implementation formulas and technical details are presented in[Section˜3\.2](https://arxiv.org/html/2406.01514#S3.SS2)\.

### 3Dapa

We presentDapa, a two\-stage alignment framework that first identifies safety\-critical representation vectors via delta debugging and then applies model fusion as a memory\-editing mechanism to restore model safety\.

#### 3\.1Delta Debugging

Although the gate layer within MLP layers is crucial for ensuring model responses adhere to ethical guidelines from[§ 2](https://arxiv.org/html/2406.01514#S2), modifying all gate layers could degrade the original performance due to a large number of parameter changes\. We propose a strategy to efficiently identify the optimal memory space for targeted modifications, enhancing alignment while preserving performance\.

We incorporate delta debugging\(Zeller and Hildebrandt,[2002](https://arxiv.org/html/2406.01514#bib.bib116)\)in our strategy\. Delta debugging is a systematic approach that automates the debugging process by identifying the smallest set of changes responsible for a program’s failure\. It reduces the set of changes, testing progressively smaller subsets until pinpointing the precise cause of the failure\. InDapa, we consider it a program failure when LLMs provide an unethical response to an unethical question\. To demonstrate how delta debugging works inDapa, let𝐒∈𝕊\\mathbf\{S\}\\in\\mathbb\{S\}be a memory space where𝕊\\mathbb\{S\}is the universe memory of all MLP modules\.

A policy is defined by the functionπ:𝕊→\{0,1\}\\pi:\\mathbb\{S\}\\rightarrow\\\{0,1\\\}, where ifπ​\(𝐒\)=1\\pi\(\\mathbf\{S\}\)=1, it indicates that the memory space𝐒\\mathbf\{S\}is beneficial for enhancing alignment, and ifπ​\(𝐒\)=0\\pi\(\\mathbf\{S\}\)=0, it indicates that the memory space𝐒\\mathbf\{S\}does not contribute to improving alignment\. Given an aligned model memory space𝐒\\mathbf\{S\}and policyπ\\pi, we aim to find the smallest memory space𝐒∗∈𝕊\\mathbf\{S\}^\{\*\}\\in\\mathbb\{S\}in the aligned model which can most efficiently improve the unaligned ability to defend the jailbreak\. In our case, we defineπ​\(𝐒\)\\pi\(\\mathbf\{S\}\)as the evaluation on a small set of additional unethical questions \(e\.g\.,5% of preserved data\)\. If the model provides ethical responses to all these questions, we setπ​\(𝐒\)=1\\pi\(\\mathbf\{S\}\)=1; otherwise,π​\(𝐒\)=0\\pi\(\\mathbf\{S\}\)=0\.

Algorithm 1Memory Search Algorithm inDapa1:Aligned Model MLP Memory Space

𝕊\\mathbb\{S\}
2:A policy function

π\\pi
3:The smallest memory space

𝐒∗\\mathbf\{S\}^\{\*\}for the editing

4:

L←L\\leftarrowA List memory space set of

𝕊\\mathbb\{S\}
5:

n←2n\\leftarrow 2
6:while

n≤\|L\|n\\leq\|L\|do

7:

⟨s1,…,sn⟩←\\langle s\_\{1\},\\dots,s\_\{n\}\\rangle\\leftarrowsplit

LLinto

nnpartitions

8:if

∃i,π​\(si\)=1\\exists i,\\pi\(s\_\{i\}\)=1then

9:

⟨L,n⟩←⟨si,2⟩\\langle L,n\\rangle\\leftarrow\\langle s\_\{i\},2\\rangle
10:elseif

∃i,π​\(L∖si\)=1\\exists i,\\pi\(L\\setminus s\_\{i\}\)=1then

11:

⟨L,n⟩←⟨L∖si,n−1⟩\\langle L,n\\rangle\\leftarrow\\langle L\\setminus s\_\{i\},n\-1\\rangle
12:else

13:

⟨L,n⟩←⟨L,2​n⟩\\langle L,n\\rangle\\leftarrow\\langle L,2n\\rangle
14:endif

15:endwhile

16:return

𝐒∗\\mathbf\{S\}^\{\*\}corresponding to

LL

We next briefly describe the delta debugging process in our aligner, as shown in Algorithm[1](https://arxiv.org/html/2406.01514#alg1)\. Given the input memory space of aligned model𝕊\\mathbb\{S\}, number of partitionn=2n=2and a list of memory space setLLof𝕊\\mathbb\{S\}\. we first split the memory space intonnpartitions\. We then check if there exists a partitionsis\_\{i\}such thatπ​\(si\)=1\\pi\(s\_\{i\}\)=1\. If such a partition exists, we update the memory space tosis\_\{i\}and updaten=2n=2\. Otherwise, we check if there exists a partitionsis\_\{i\}such thatπ​\(L∖si\)=1\\pi\(L\\setminus s\_\{i\}\)=1\. If such a partition exists, we update the memory space toL∖siL\\setminus s\_\{i\}and setn=n−1n=n\-1\. If neither of the above conditions are met, we double the number of partitionsnn\. We repeat this process untilnnis greater than the number of partitions in the memory space\. Finally, we return the memory space𝐒∗\\mathbf\{S\}^\{\*\}corresponding to the updated memory spaceLL\. The worst\-case complexity of this algorithm isO​\(L⋅log⁡L\)\{O\(L\\cdot\\log L\)\}\.

To demonstrate the efficiency of our memory space searching algorithm, we employ the Llama\-2\-7b model as a case study to illustrate how Algorithm[1](https://arxiv.org/html/2406.01514#alg1)navigates the memory space for alignment, as shown in[Appendix˜D](https://arxiv.org/html/2406.01514#A4)\.

#### 3\.2Model Fusion

We employ model fusion as a targeted memory\-editing mechanism to correct shadow safety misalignment\. Specifically, modelAAdenotes a safety\-aligned model, modelBBdenotes a non\-aligned base model, and modelCCdenotes a downstream\-aligned model derived fromAA\. We compute a safety vector𝐯s\\mathbf\{v\}\_\{s\}from the representation difference betweenAAandBB, which is identified via the delta debugging procedure described in[Section˜3\.1](https://arxiv.org/html/2406.01514#S3.SS1)\. This vector captures the representation shift induced by safety alignment and serves as a compact encoding of alignment\-related knowledge\. At the representation level, the effect of model fusion on𝐡C\\mathbf\{h\}\_\{C\}of modelCCcan be expressed as

𝐡Calign=𝐡Corg\+λ​𝐯s\\displaystyle\\mathbf\{h\}\_\{C\_\{\\mathrm\{align\}\}\}=\\mathbf\{h\}\_\{C\_\{\\mathrm\{org\}\}\}\+\\lambda\\mathbf\{v\}\_\{s\}\(3\.1\)whereλ\\lambdacontrols the strength of the safety correction\. We estimateλ\\lambdavia linear regression by evaluating the fused modelCalignC\_\{\\text\{align\}\}on a subset of 20 training samples from R2D\-R1\(Zhuet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib9)\)and selecting the smallest value that improves safety, as measured by the Defense Success Rate \(DSR\) defined in[Section˜4](https://arxiv.org/html/2406.01514#S4), while minimizing interference with downstream task performance\.

Table 1:ComparingDapain 3 Common LLM Families\.We demonstrate the improvement in alignment capabilities of unaligned models through ourDapaaligner across 17 models using DSR\. We also assess the linguistic performance after alignment, reporting average perplexity and Cosine Similarity scores\.Dapaconsistently achieves a significant increase in DSR, with an average gain of14\.42%and a maximum of 51\.39%\. Meanwhile, the average accuracy on the MMLU dataset using 5\-shot prompting drops by 2\.06% and perplexityincreasesby 1\.69\. Overall,Dapaenhances DSR significantly while maintaining the original capabilities of the models with minimal impact\.FamilyModel NameDSRPerplexityMMLUCosine SimilarityBeforeAfterBeforeAfterBeforeAfterLlama\-2chinese\-alpaca\-2\-7b82\.0387\.507\.547\.4638\.71±\\pm0\.4137\.43±\\pm1\.420\.88Llama\-2\-7b37\.1642\.194\.774\.7836\.37±\\pm1\.0139\.30±\\pm0\.000\.79Llama\-2\-13b37\.5046\.094\.284\.2834\.74±\\pm2\.4637\.08±\\pm1\.330\.76chinese\-alpaca\-2\-13b70\.3185\.165\.635\.6048\.77±\\pm0\.7047\.60±\\pm1\.070\.91Redmond\-Puffin\-13B22\.6647\.664\.304\.3030\.06±\\pm0\.8832\.38±\\pm1\.220\.89MistralMistral\-7B21\.0925\.784\.584\.6045\.38±\\pm1\.6647\.72±\\pm0\.700\.76OpenHermes\-2\-Mistral\-7b33\.5946\.885\.005\.0241\.29±\\pm0\.8142\.46±\\pm1\.220\.88dolphin\-2\.2\.1\-mistral\-7b24\.2241\.415\.185\.1960\.12±\\pm0\.4158\.25±\\pm1\.050\.90zephyr\-7b\-alpha24\.2232\.815\.115\.1154\.04±\\pm1\.5356\.73±\\pm0\.410\.88mistral\-7B\-forest\-dpo19\.3815\.625\.135\.1054\.62±\\pm0\.8854\.04±\\pm0\.610\.72dolphin\-2\.6\-mistral\-7b\-dpo24\.2255\.475\.415\.4260\.47±\\pm0\.2062\.69±\\pm0\.540\.91openchat\-3\.558\.6867\.195\.155\.1061\.40±\\pm0\.3558\.71±\\pm0\.410\.89Gemmagemma\-2b22\.0573\.447\.9224\.1533\.57±\\pm0\.4124\.80±\\pm2\.060\.33Gemmalpaca\-2B37\.0151\.569\.9222\.0040\.94±\\pm0\.8121\.17±\\pm1\.420\.51gemma\-7b26\.5634\.386\.096\.2739\.65±\\pm1\.7542\.11±\\pm0\.930\.66gemma\-7b\-ultrachat\-sft34\.1541\.417\.177\.4842\.11±\\pm0\.0029\.24±\\pm0\.540\.76gemma\-orchid\-7b\-dpo21\.8835\.167\.227\.4242\.26±\\pm0\.6138\.01±\\pm0\.880\.76Average Change34\.3948\.815\.917\.6044\.98±\\pm0\.8842\.92±\\pm1\.000\.87

### 4Experimental Studies

We perform a series of experiments to evaluateDapain enhancing the alignment performance of unaligned models against unethical prompts, in[§ 4\.1](https://arxiv.org/html/2406.01514#S4.SS1)\. We also assess the impact of theDapaaligner on the model’s performance in[§ 4\.2](https://arxiv.org/html/2406.01514#S4.SS2), including linguistic capabilities and reasoning abilities\.Lastly, we conduct an ablation study in[§ 4\.3](https://arxiv.org/html/2406.01514#S4.SS3)to examine how the choice of fusion layer affects both model safety and overall performance\.

###### Models and Parameter Efficiency\.

We validate our method on 17 widely\-used LLMs from 3 different families, reported in[Table˜4](https://arxiv.org/html/2406.01514#A7.T4)in Appendix\. These models include both foundational and fine\-tuned models, with the fine\-tuning approach including SFT, DPO, and RLHF\. Further,[Table 4](https://arxiv.org/html/2406.01514#A7.T4)classifies the models based on their family and the aligned and unaligned models\. We defer the details of these aligned and unaligned models in[Appendix H](https://arxiv.org/html/2406.01514#A8)\. In our experiments, we identify the layers formodel fusion using delta debugging \([Algorithm 1](https://arxiv.org/html/2406.01514#alg1)\)\.In[Table 4](https://arxiv.org/html/2406.01514#A7.T4), we also report that theDapaaligner is very parameter\-efficient\.Dapanot only updates an average of 6\.26% of parameters across 3 model families, it also updates as little as 3\.25% parameters in the commonly used Llama\-2\-7b\.

#### 4\.1Alignment Performance

To evaluateDapa’s effectiveness in aligning unaligned models, we substitute the relative memory in 17 models withDapa’s configuration and assess their performance in defending against jailbreak attacks\. Because we set the response generation with deterministic, the variance of the evaluation is 0\. As a result, we only need to run one evaluation for each model\.

###### Dataset\.

In our experiment, we use the AdvBench\(Zouet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib82)\)to validate the performance ofDapa\. It is a benchmark dataset that contains various unethical prompts to evaluate the alignment of language models, encompassing different categories such as violence, hate speech, and misinformation\. We sample 128 prompts from the AdvBench dataset to form our evaluation dataset\.

###### Metrics\.

We employ the Defense Success Rate \(DSR\) as the primary metric to evaluate the alignment performance\. For each unethical prompt, if the model provides a refusal or an ethical response, it is considered aligned\. To automatically evaluate the model’s responses, we combine LLM judgments withgpt\-4oand rule\-based classifiers to determine if the model’s response is refusal or ethical\. For LLM judgment, we use the system prompt provided inYuet al\.\([2024a](https://arxiv.org/html/2406.01514#bib.bib111)\)to guide the LLM to evaluate the response\. For rule\-based classifiers, we employ and expand the keyword list like “Sorry, as an" or “As a responsible” provided inZouet al\.\([2023b](https://arxiv.org/html/2406.01514#bib.bib82)\)to classify the response\. If the response contains any of the keywords, it is classified as a refusal\. A response is deemed aligned if it is classified as refusal or ethical by both the LLM and rule\-based classifiers\.

###### Results\.

In[Table 1](https://arxiv.org/html/2406.01514#S3.T1), our results show thatDapaachieves performance in increasing the alignment on unaligned models, achieving a14\.42%average increase in DSR across all 17 models\. Notably, thegemma\-2bmodel achieves a significant 51\.39% increase in DSR\. These improvements in DSR underscoreDapaaligner’s effectiveness in enhancing model safety against jailbreak prompts\.

#### 4\.2Model Performance

To assess the model’s performance before and afterDapaalignment, we evaluate the generative and reasoning capabilities in a deterministic setting\. For each pre\-alignment and post\-alignment model, we measure the model’s generative ability using perplexity and assess the response variation caused by theDapaalignment through cosine similarity score\. We also validate the model’s reasoning ability by employing real\-life question\-answering and STEM problem\-solving tasks, using Chain\-of\-Thought \(CoT\)\(Weiet al\.,[2022](https://arxiv.org/html/2406.01514#bib.bib110)\)and few shot prompting approach\. We conduct each evaluation three times and present the average and standard deviation for each metric\.

###### Dataset\.

We employ six real\-world datasets: ShareGPT\(Chianget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib119)\), WikiText\-2\(Merityet al\.,[2017](https://arxiv.org/html/2406.01514#bib.bib81)\), Big\-Bench\(Srivastavaet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib68)\)\(TruthQA, General QA, SocialQA\), HarmfulQA\(Bhardwaj and Poria,[2023](https://arxiv.org/html/2406.01514#bib.bib131)\), JailbreakBench\(Chaoet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib23)\)and MMLU\(Hendryckset al\.,[2021](https://arxiv.org/html/2406.01514#bib.bib67)\)\. ShareGPT is used for computing the cosine similarity score of model responses, Wiki8\-2 assesses model perplexity, and MMLU and Big\-Bench evaluate the model’s problem\-solving and reasoning abilities\.

###### Metrics\.

In our experiment, we evaluate the responses generated by both pre\-alignment model and post\-alignment model\. We use cosine similarity to measure the impact of the aligner on model response generation\. Additionally, we use perplexity for comparative analysis of the models’ generative capabilities\. A high cosine similarity score or comparable perplexity indicates using our aligner improves the defense success rate while maintaining the original performance\. Additionally, to evaluate the model’s reasoning abilities, we administer real\-life question\-answering and STEM problem\-solving tasks, measuring performance with the Exact Match \(EM\) metric\.

Table 2:ComparingDapawith CoT Abilities in 3 Common LLM Families\.We conduct an experiment to evaluate the impact ofDapaon CoT capabilities using the Exact Match \(EM\) score\. TheDapaaligner reduces the average EM of the Chain of Action \(CoA\) method on the Big\-Bench dataset by 2\.77%, indicating a significant effect on the model’s original reasoning abilities\.FamilyModel NameTruthQAGKSocialQABeforeAfterBeforeAfterBeforeAfterLlama\-2chinese\-alpaca\-2\-7b20\.67±\\pm2\.0824\.67±\\pm2\.0838\.10±\\pm7\.0540\.00±\\pm1\.4321\.67±\\pm2\.3119\.67±\\pm3\.21Llama\-2\-7b36\.67±\\pm3\.5127\.00±\\pm3\.5158\.57±\\pm7\.1446\.67±\\pm5\.9522\.33±\\pm2\.5224\.00±\\pm7\.21Llama\-2\-13b39\.33±\\pm2\.5224\.67±\\pm4\.9364\.76±\\pm2\.9745\.24±\\pm5\.9539\.33±\\pm2\.5222\.67±\\pm3\.06chinese\-alpaca\-2\-13b35\.33±\\pm5\.1336\.33±\\pm5\.5140\.48±\\pm9\.7249\.05±\\pm6\.4435\.33±\\pm5\.1319\.00±\\pm3\.61Redmond\-Puffin\-13B33\.67±\\pm0\.5824\.67±\\pm4\.0455\.71±\\pm4\.2941\.43±\\pm1\.4333\.67±\\pm0\.5819\.00±\\pm3\.61MistralMistral\-7B34\.00±\\pm1\.7333\.67±\\pm2\.0879\.05±\\pm2\.9777\.14±\\pm2\.4739\.33±\\pm3\.5137\.67±\\pm2\.08OpenHermes\-2\-Mistral\-7b39\.67±\\pm3\.5142\.33±\\pm5\.5167\.14±\\pm1\.4371\.43±\\pm4\.2930\.00±\\pm2\.6540\.00±\\pm1\.73dolphin\-2\.2\.1\-mistral\-7b51\.00±\\pm4\.0048\.33±\\pm3\.2185\.24±\\pm2\.1885\.71±\\pm2\.4753\.00±\\pm2\.5253\.00±\\pm1\.00zephyr\-7b\-alpha35\.00±\\pm1\.0042\.67±\\pm3\.0664\.76±\\pm7\.8771\.90±\\pm2\.9744\.00±\\pm3\.2146\.00±\\pm7\.51mistral\-7B\-forest\-dpo41\.00±\\pm3\.0047\.33±\\pm6\.3371\.43±\\pm3\.7875\.71±\\pm4\.2938\.33±\\pm6\.0340\.00±\\pm4\.58dolphin\-2\.6\-mistral\-7b\-dpo48\.67±\\pm2\.0846\.33±\\pm2\.8987\.14±\\pm2\.4790\.00±\\pm0\.0039\.33±\\pm3\.5130\.00±\\pm1\.01openchat\-3\.549\.67±\\pm4\.9355\.67±\\pm1\.5384\.76±\\pm2\.1883\.81±\\pm2\.9761\.00±\\pm6\.5656\.00±\\pm2\.65Gemmagemma\-2b29\.33±\\pm5\.7729\.00±\\pm3\.6151\.43±\\pm3\.7843\.81±\\pm2\.1829\.00±\\pm3\.6115\.67±\\pm2\.52Gemmalpaca\-2B33\.67±\\pm3\.2131\.67±\\pm2\.5261\.43±\\pm1\.4352\.38±\\pm6\.7541\.00±\\pm4\.5816\.33±\\pm2\.08gemma\-7b49\.33±\\pm4\.1650\.00±\\pm3\.0088\.10±\\pm1\.6589\.52±\\pm4\.1242\.00±\\pm2\.8935\.33±\\pm2\.52gemma\-7b\-ultrachat\-sft27\.67±\\pm4\.0429\.33±\\pm3\.5168\.10±\\pm9\.5160\.00±\\pm9\.9013\.33±\\pm2\.5215\.33±\\pm3\.21gemma\-orchid\-7b\-dpo41\.33±\\pm2\.0839\.33±\\pm1\.5380\.48±\\pm2\.1879\.52±\\pm0\.8229\.00±\\pm3\.6138\.33±\\pm3\.51Average Change38\.00±\\pm3\.1437\.24±\\pm3\.4567\.45±\\pm4\.2764\.90±\\pm3\.9536\.04±\\pm3\.4331\.04±\\pm3\.24

###### Setup\.

We assess post\-alignment performance by examining reasoning capacity, response similarity, and perplexity\. In all experiments, we use the model both before and after the adapter in a deterministic output setting\. In the response similarity test, we compare the average similarity of responses on the same generated question\. For comparing model responses, we embed responses from both models using the text\-embedding\-3\-small model\*\*\*[https://openai\.com/blog/new\-embedding\-models\-and\-api\-updates](https://openai.com/blog/new-embedding-models-and-api-updates)and analyze 128 questions sampled from ShareGPT\. In the perplexity test, we compute the perplexity score with Huggingface Evaluate\*\*\*[https://huggingface\.co/docs/evaluate/index](https://huggingface.co/docs/evaluate/index)on Wiki8\-2 dataset\(Merityet al\.,[2017](https://arxiv.org/html/2406.01514#bib.bib81)\)\. In assessing model reasoning capacity, we conduct tests using 5\-shot prompting on the MMLU dataset\(Brownet al\.,[2020](https://arxiv.org/html/2406.01514#bib.bib65)\)and Chain\-of\-Action \(CoA\)\(Panet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib54);[2024](https://arxiv.org/html/2406.01514#bib.bib130)\)methodology on the Big\-Bench dataset, excluding memory retrieval\. We conduct each evaluation three times and present the average and standard deviation for each metric\.

Table 3:The Comparison of Defense Models withDapaon Llama, Gemma Models, and Mistral Models in AdvBench\.We conduct experiments to compare the performance ofDapawith RepE on Llama, Gemma, and Mistral Models\. On average,Dapaachieves a DSR 13% higher than RepE\. The model names corresponding to each label are provided in[Appendix˜S](https://arxiv.org/html/2406.01514#A19)\.ABCDEFGHIJKLMNOPQAVGRepE34804073212734372729233528322496436ICD438442793456443033312040342934116642Guardrails398440732524392936242336272727226137Safety Arithmetic418541773144423033282139312831156540InferAligner448744813761463236322241353139136844Model Surgery438644824063483238332343373145146745Safe LoRA448745844569503340342545393251156747Ours428846854873523441352647413355166749

###### Results\.

In[Table 1](https://arxiv.org/html/2406.01514#S3.T1), our findings indicate that the average perplexityincreasesby 1\.69, with the Llama\-2\-13b model showing no change\. In one special case, the Gemme\-2B family’s models display the most significant increase in perplexity, at 16\.23\. Additionally, the average cosine similarity is0\.87, with Dolphin\-2\.6\-mistral\-7b\-dpo achieving the highest similarity of 0\.91\. Those indicate that the system does not adversely affect the original capabilities of the language model\. Additionally, in[Table 1](https://arxiv.org/html/2406.01514#S3.T1), our finding indicate the average accuracy drops by 2\.06% using 5\-shot prompting on the MMLU dataset\. Most models exhibit only slight changes in accuracy\.Overall, DAPA preserves utility for most models, but we observe two clear failure cases: Gemmalpaca\-2B and Gemma\-7B\-Ultrachat\-SFT, whose MMLU scores drop by 19\.77% and 12\.87%, respectively\. We attribute these failures to increased sensitivity to layer\-level fusion: Gemmalpaca\-2B has limited parameter redundancy due to its small scale, while Gemma\-7B\-Ultrachat\-SFT is likely more fragile because prior task\-specific SFT already reshaped its internal representations, making subsequent fusion more prone to interfering with the model’s original knowledge organization\.We also provide additional experimental results for the 0\-shot and 1\-shot settings in[Section˜K\.4](https://arxiv.org/html/2406.01514#A11.SS4)\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x4.png)Figure 4:Impact of Parameter Change on Model Performance\.In[Table 2](https://arxiv.org/html/2406.01514#S4.T2), our results show a 2\.77% average accuracy decrease using the CoA methodology on the Big\-Bench datasets\. In one exception, OpenHermes\-2\-Mistral\-7B shows the most significant improvement, achieving a 10% increase in accuracy on SocialQA dataset, while Gemma\-alpaca\-2B shows the largest decrease, with a 24% decrease on the SocialQA dataset\.

Overall, these findings regarding models’ perplexity, responses’ cosine similarity, and performance on the real\-life question\-answering and problem\-solving tests indicate that theDapaaligner does not significantly impair the models’ performance after usingDapaaligner\.

#### 4\.3Ablation Study

We performed extensive ablation studies; here we highlight two key results, with the remaining experiments provided in[Appendix˜E](https://arxiv.org/html/2406.01514#A5)\.

###### Comparison with Other Defense Methods\.

We use the alignment method described in Representative Engineering \(RepE\)\(Zouet al\.,[2023a](https://arxiv.org/html/2406.01514#bib.bib132)\), ICD\(Weiet al\.,[2026](https://arxiv.org/html/2406.01514#bib.bib13)\), InferAligner\(Wanget al\.,[2024b](https://arxiv.org/html/2406.01514#bib.bib2)\), Safety Arithmetic\(Hazraet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib3)\), Safe LoRA\(Hsuet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib17)\), Model Surgery\(Wanget al\.,[2025a](https://arxiv.org/html/2406.01514#bib.bib10)\), LLM Guardrails\(DONGet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib12)\)as baseline to compare other alignment methodologies\. Specifically, we useLlama Guard 2\(Team,[2024](https://arxiv.org/html/2406.01514#bib.bib6)\)as a representative example of an LLM guardrail system in our experiments\. The average DSR was calculated using 128 questions from the AdvBench dataset, under the same evaluation settings asDapa\. The results are shown in[Table 3](https://arxiv.org/html/2406.01514#S4.T3)\.Our analysis shows thatDapaachieves an average DSR that is 2% higher than Safe LoRA, the strongest baseline defense method\.The result demonstrates thatDapasignificantly outperforms the baseline alignment methodology across different models\.

###### Impact of Parameter Change on Model Performance\.

We investigate how different parameter update ratios influence model performance usingChinese\-Alpaca\-2\-7BandLlama\-2\-7Bas representative architectures\. As shown in[Figure˜4](https://arxiv.org/html/2406.01514#S4.F4), increasing the parameter change from 0% to 2% leads to about a 10% variation in DSR performance, indicating that model robustness remains largely unaffected by small\-scale adaptation\. Specifically, our configuration with a 3\.25% parameter change \(five\-layer modification\) achieves near\-optimal DSR values comparable to those with higher parameter budgets\. This demonstrates thatDapacan effectively enhance safety alignment and maintain reasoning capability with minimal parameter updates\.

#### 4\.4DapaPerformance on Large Multimodal Models

To assess the robustness of our method in multimodal models, we perform alignment usingDapaon the LLaVA\(Liuet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib25)\)model\. Many multimodal models, such as LLaVA and Qwen\-VL\(Baiet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib27)\), are built on existing language and other modality foundation models\. In this section, we focus on analyzing the general question\-answering task in vision\-language models, as it represents a critical area for addressing multimodal safety issues\(Liuet al\.,[2024c](https://arxiv.org/html/2406.01514#bib.bib29)\)\. Since the general question\-answering task generates text\-based responses, we applyDapato the language model module\. We use the LLaVA\-1\.5\-7b\*\*\*https://huggingface\.co/llava\-hf/llava\-1\.5\-7b\-hfas the unaligned model and LLaVA\-1\.6\-vicuna\-7b\*\*\*https://huggingface\.co/llava\-hf/llava\-v1\.6\-vicuna\-7b\-hf\(Liuet al\.,[2024a](https://arxiv.org/html/2406.01514#bib.bib24)\)as the teacher model to do the alignment\. We use 103 questions in the HarmBench\(Mazeikaet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib26)\)to evaluate the result\. In[Figure˜5](https://arxiv.org/html/2406.01514#S4.F5), our results demonstrate thatDapaachieves an impressive 24\.27% increase in DSR\. This highlights the ability ofDapato effectively extend to the large multimodal models\. We present a multimodal defense example usingDapain[Section˜K\.12](https://arxiv.org/html/2406.01514#A11.SS12)\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x5.png)Figure 5:Left:DapaPerformance under Llama3 70B Model\. We conduct experiments under the DAPA attack, where ourDapaachieves an average improvement of 10% compared to the unaligned model\.Right:DapaPerformance on LLaVA\-1\.5\-7b multimodal model\. AfterDapaalignment, the DSR increases by 24\.27%\.![Refer to caption](https://arxiv.org/html/2406.01514v5/x6.png)Figure 6:DAPA Performance under GCG attack\.We conduct experiments under the DAPA attack, where our DAPA achieves an average improvement of 9\.62% compared to the unaligned model\.###### DapaPerformance with GCG Attack\.

To evaluate the robustness of our method,Dapa, against advanced jailbreak attack methods, we align the Llama\-2 family model using the GCG\(Zouet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib82)\)attack\. As shown in[Figure˜6](https://arxiv.org/html/2406.01514#S4.F6), our results demonstrate thatDapaachieves a 9\.62% increase in DSR\.

###### DapaPerformance Under Other Advanced Jailbreak Attack\.

We evaluate three advanced jailbreak attacks: GPTFuzzer\(Yuet al\.,[2024a](https://arxiv.org/html/2406.01514#bib.bib111)\), and AutoDAN\(Liuet al\.,[2024b](https://arxiv.org/html/2406.01514#bib.bib64)\)\. To assess the performance ofDapaunder the GPTFuzzer attack, we compare its performance improvement against unaligned Llama\-2 family models\. As shown in[Figure˜9](https://arxiv.org/html/2406.01514#A5.F9), our results demonstrate thatDapaachieves a 16\.62% increase in DSR\. We also provide additional experimental results for AutoDAN in[Section˜K\.9](https://arxiv.org/html/2406.01514#A11.SS9)\.

###### DapaPerformance on Large\-size Models\.

To evaluate the robustness ofDapaon large\-scale language models, we perform alignment experiments using the Llama 3 70B model\. We use the Hermes\-3\-Llama\-3\.1\-70B\-Uncensored as the unaligned model and Llama\-3\.1\-70B\-Instruct as the teacher model for alignment\. We assess the performance ofDapain 70B models using Advbench\. As shown in[Figure˜5](https://arxiv.org/html/2406.01514#S4.F5), the DSR rate improved from 50% before alignment to 60% after alignment\.

### 5Discussion and Conclusion

We introduce the Decoupled Alignment for Robust Plug\-and\-Play Adaptation,Dapa, which edits the unaligned model memory to enhance the model’s defenses against jailbreak attacks\. This method improves model alignment without the substantial computational expense typically associated with fine\-tuning\. It also efficiently identifies the optimal memory space for alignment\. Visualizations confirm that the ethical boundary of model alignment is predominantly situated within the middle MLP’s gate layers\. Empirically,Dapaachieves a14\.42%improvement in model alignment, reaching up to 51\.39% in one of the Gemma family models, with an average parameter change of only 6\.26%\. Moreover,Dapaminimally impacts the model’s performance in generation and reasoning tasks\.

### Acknowledgments

The authors thank Zisheng Liang for developing the jailbreak evaluation pipeline with StrongReject metrics\. The authors also thank the anonymous reviewers and program chairs for their constructive feedback\.

Haozheng Luo is partially supported by the Lambda Researcher Grant and Adobe Fellow\. This research was supported by the OpenAI Research Access Program, which provided credits for using OpenAI GPT\-series models\. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies\.

### Ethics statement

This work proposes a training\-free red\-teaming alignment approach to address the shallow alignment challenge, leveraging knowledge distillation and delta debugging\. In line with the COLM Code of Ethics\*\*\*[https://colmweb\.org/CoE\.html](https://colmweb.org/CoE.html), we acknowledge that our code includes jailbreak attack implementations, which could potentially be misused to compromise large language models, and that our paper demonstrates examples of harmful content\. Moreover, knowledge distillation may propagate or amplify biases in model outputs\. To mitigate the potential risks of our work, we adopt several precautionary measures\. We begin by providing a clear content warning to alert readers of the harmful language present in our examples\. We also notify model providers of the risks associated withDapaprior to submission and offer practical recommendations to address these risks\. To promote transparency and reproducibility, we open\-source the code and data used in our experiments\. Finally, we outline recommendations for future research aimed at mitigating the risks ofDapaand encourage the community to develop robust defenses against such attacks\.

### Reproducibility statement

To ensure reproducibility, we release an open\-source repository containing the full implementation ofDapaand selected baselines, with plans for full open\-sourcing upon acceptance\. Unless otherwise specified, all experiments are conducted with three random seeds, yielding stable results with standard deviations below 2%\. We set the temperature to 0 for all deployment experiments\. All other hyperparameters for attack and baseline defense methods are kept consistent with their original papers\.

### References

- S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p2.1)\.
- J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang, B\. Hui, L\. Ji, M\. Li, J\. Lin, R\. Lin, D\. Liu, G\. Liu, C\. Lu, K\. Lu, J\. Ma, R\. Men, X\. Ren, X\. Ren, C\. Tan, S\. Tan, J\. Tu, P\. Wang, S\. Wang, W\. Wang, S\. Wu, B\. Xu, J\. Xu, A\. Yang, H\. Yang, J\. Yang, S\. Yang, Y\. Yao, B\. Yu, H\. Yuan, Z\. Yuan, J\. Zhang, X\. Zhang, Y\. Zhang, Z\. Zhang, C\. Zhou, J\. Zhou, X\. Zhou, and T\. Zhu \(2023a\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- Qwen\-vl: a versatile vision\-language model for understanding, localization, text reading, and beyond\.External Links:2308\.12966Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p2.1),[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.p1.1)\.
- R\. Bhardwaj and S\. Poria \(2023\)Red\-teaming large language models using chain of utterances for safety\-alignment\.arXiv preprint arXiv:2308\.09662\.Cited by:[§K\.2](https://arxiv.org/html/2406.01514#A11.SS2.p1.1),[Appendix E](https://arxiv.org/html/2406.01514#A5.p2.1),[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px1.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei \(2020\)Language models are few\-shot learners\.InThe Thirty\-three Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px3.p1.1)\.
- P\. Chao, E\. Debenedetti, A\. Robey, M\. Andriushchenko, F\. Croce, V\. Sehwag, E\. Dobriban, N\. Flammarion, G\. J\. Pappas, F\. Tramèr, H\. Hassani, and E\. Wong \(2024\)JailbreakBench: an open robustness benchmark for jailbreaking large language models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§K\.7](https://arxiv.org/html/2406.01514#A11.SS7.p1.1),[Appendix E](https://arxiv.org/html/2406.01514#A5.p2.1),[§I\.3](https://arxiv.org/html/2406.01514#A9.SS3.p1.1),[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px1.p1.1)\.
- P\. Chao, A\. Robey, E\. Dobriban, H\. Hassani, G\. J\. Pappas, and E\. Wong \(2023\)Jailbreaking black box large language models in twenty queries\.InR0\-FoMo:Robustness of Few\-shot and Zero\-shot Learning in Large Foundation Models,Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- Z\. Chen, Q\. Zhang, and S\. Pei \(2025\)Injecting universal jailbreak backdoors into LLMs in minutes\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- W\. Chiang, Z\. Li, Z\. Lin, Y\. Sheng, Z\. Wu, H\. Zhang, L\. Zheng, S\. Zhuang, Y\. Zhuang, J\. E\. Gonzalez, I\. Stoica, and E\. P\. Xing \(2023\)Vicuna: an open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality\.Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px1.p1.1)\.
- Y\. Chu, J\. Xu, X\. Zhou, Q\. Yang, S\. Zhang, Z\. Yan, C\. Zhou, and J\. Zhou \(2023\)Qwen\-audio: advancing universal audio understanding via unified large\-scale audio\-language models\.arXiv preprint arXiv:2311\.07919\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p2.1)\.
- D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. Wei \(2022\)Knowledge neurons in pretrained transformers\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 8493–8502\.Cited by:[Appendix J](https://arxiv.org/html/2406.01514#A10.p2.1),[Appendix J](https://arxiv.org/html/2406.01514#A10.p3.1)\.
- H\. Dai, H\. Luo, H\. Zhang, M\. lin, Y\. Chen, and B\. Wang \(2026\)TIDES: test\-time inference drift exploitation via scaling\.InThe First Workshop on Efficient Spatial Reasoning,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. Yang \(2024\)Safe RLHF: safe reinforcement learning from human feedback\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[1st item](https://arxiv.org/html/2406.01514#S1.I2.i1.p1.1)\.
- Y\. DONG, R\. Mu, G\. Jin, Y\. Qi, J\. Hu, X\. Zhao, J\. Meng, W\. Ruan, and X\. Huang \(2024\)Position: building guardrails for large language models requires systematic design\.InForty\-first International Conference on Machine Learning,Cited by:[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- D\. Ganguli, L\. Lovitt, J\. Kernion, A\. Askell, Y\. Bai, S\. Kadavath, B\. Mann, E\. Perez, N\. Schiefer, K\. Ndousse,et al\.\(2022\)Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned\.arXiv preprint arXiv:2209\.07858\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 5484–5495\.Cited by:[Appendix J](https://arxiv.org/html/2406.01514#A10.p2.1),[Appendix J](https://arxiv.org/html/2406.01514#A10.p3.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- K\. Grimes, M\. Christiani, D\. Shriver, and M\. C\. Connor \(2025\)Concept\-ROT: poisoning concepts in large language models with model editing\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p3.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p1.1),[§1](https://arxiv.org/html/2406.01514#S1.p1.1),[§1](https://arxiv.org/html/2406.01514#S1.p2.1)\.
- S\. Hahn and H\. Choi \(2019\)Self\-knowledge distillation in natural language processing\.InProceedings of the International Conference on Recent Advances in Natural Language Processing \(RANLP 2019\),R\. Mitkov and G\. Angelova \(Eds\.\),Varna, Bulgaria,pp\. 423–430\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p3.1)\.
- R\. Hazra, S\. Layek, S\. Banerjee, and S\. Poria \(2024\)Safety arithmetic: a framework for test\-time safety alignment of language models by steering parameters and activations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 21759–21776\.Cited by:[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InThe Ninth International Conference on Learning Representations \(ICLR\),Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px1.p1.1)\.
- D\. Hendrycks and K\. Gimpel \(2016\)Gaussian error linear units \(gelus\)\.arXiv preprint arXiv:1606\.08415\.Cited by:[§2](https://arxiv.org/html/2406.01514#S2.p2.15)\.
- C\. Hsu, Y\. Tsai, C\. Lin, P\. Chen, C\. Yu, and C\. Huang \(2024\)Safe lora: the silver lining of reducing safety risks when finetuning large language models\.Advances in Neural Information Processing Systems37,pp\. 65072–65094\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p1.1),[§1](https://arxiv.org/html/2406.01514#S1.p2.1),[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)Lora: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations \(ICLR\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p1.1)\.
- J\. Y\. Hu, P\. Chang, R\. Luo, H\. Chen, W\. Li, W\. Wang, and H\. Liu \(2024a\)Outlier\-efficient hopfield layers for large transformer\-based models\.InThe Forty\-first International Conference on Machine Learning \(ICML\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- J\. Y\. Hu, B\. Chen, D\. Wu, F\. Ruan, and H\. Liu \(2024b\)Nonparametric modern hopfield models\.arXiv preprint arXiv:2404\.03900\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- J\. Y\. Hu, T\. Lin, Z\. Song, and H\. Liu \(2024c\)On computational limits of modern hopfield models: a fine\-grained complexity analysis\.InThe Forty\-first International Conference on Machine Learning \(ICML\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- J\. Y\. Hu, D\. Yang, D\. Wu, C\. Xu, B\. Chen, and H\. Liu \(2023\)On sparse modern hopfield model\.InThe Thirty\-seventh Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- X\. Hu, P\. Chen, and T\. Ho \(2024d\)Gradient cuff: detecting jailbreak attacks on large language models by exploring refusal loss landscapes\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 126265–126296\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- Z\. Huang, Y\. Shen, X\. Zhang, J\. Zhou, W\. Rong, and Z\. Xiong \(2023\)Transformer\-patcher: one mistake worth one neuron\.InThe Eleventh International Conference on Learning Representations \(ICLR\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§I\.4\.2](https://arxiv.org/html/2406.01514#A9.SS4.SSS2.p1.1)\.
- S\. Lermen and C\. Rogers\-Smith \(2024\)LoRA fine\-tuning efficiently undoes safety training in llama 2\-chat 70b\.InICLR 2024 Workshop on Secure and Trustworthy Large Language Models,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p1.1),[§1](https://arxiv.org/html/2406.01514#S1.p2.1)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024a\)Llava\-next: improved reasoning, ocr, and world knowledge\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p2.1),[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.InThirty\-seventh Conference on Neural Information Processing Systems,Cited by:[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.p1.1)\.
- X\. Liu, N\. Xu, M\. Chen, and C\. Xiao \(2024b\)AutoDAN: generating stealthy jailbreak prompts on aligned large language models\.External Links:2310\.04451Cited by:[§K\.9](https://arxiv.org/html/2406.01514#A11.SS9.p1.1),[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.SSS0.Px2.p1.1)\.
- X\. Liu, X\. Cui, P\. Li, Z\. Li, H\. Huang, S\. Xia, M\. Zhang, Y\. Zou, and R\. He \(2024c\)Jailbreak attacks and defenses against multimodal generative models: a survey\.External Links:2411\.09259Cited by:[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.p1.1)\.
- H\. Luo, Y\. Wang, J\. Yu, B\. Wang, and Y\. Chen \(2026\)Contrastive reasoning alignment: reinforcement learning from hidden representations\.InForty\-third International Conference on Machine Learning,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- A\. M\. Mansourian, R\. Ahmadi, M\. Ghafouri, A\. M\. Babaei, E\. B\. Golezani, Z\. yasamani ghamchi, V\. Ramezanian, A\. Taherian, K\. Dinashi, A\. Miri, and S\. Kasaei \(2025\)A comprehensive survey on knowledge distillation\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p3.1)\.
- M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li,et al\.\(2024\)Harmbench: a standardized evaluation framework for automated red teaming and robust refusal\.arXiv preprint arXiv:2402\.04249\.Cited by:[§K\.8](https://arxiv.org/html/2406.01514#A11.SS8.p1.1),[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022a\)Locating and editing factual associations in gpt\.InThe Thirty\-sixth Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix J](https://arxiv.org/html/2406.01514#A10.p1.1),[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1),[Appendix C](https://arxiv.org/html/2406.01514#A3.p1.1),[Appendix E](https://arxiv.org/html/2406.01514#A5.p6.1),[§1](https://arxiv.org/html/2406.01514#S1.p5.1),[§2](https://arxiv.org/html/2406.01514#S2.p3.1),[§2](https://arxiv.org/html/2406.01514#S2.p4.1)\.
- K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau \(2022b\)Mass\-editing memory in a transformer\.InThe Tenth International Conference on Learning Representations \(ICLR\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1),[§1](https://arxiv.org/html/2406.01514#S1.p5.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2017\)Pointer sentinel mixture models\.InThe Fifth Conference on International Conference on Learning Representations \(ICLR\),Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px3.p1.1)\.
- E\. Mitchell, C\. Lin, A\. Bosselut, C\. D\. Manning, and C\. Finn \(2022\)Memory\-based model editing at scale\.InThe Thirty\-ninth International Conference on Machine Learning \(ICML\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- M\. A\. of America \(2024\)American invitational mathematics examination 2024\.Note:Official competition problemsCited by:[Appendix M](https://arxiv.org/html/2406.01514#A13.p1.1),[Appendix Q](https://arxiv.org/html/2406.01514#A17.p1.1)\.
- OpenAI \(2024\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.InThe Thirty\-sixth Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- Z\. Pan, H\. Luo, M\. Li, and H\. Liu \(2024\)Conv\-coa: improving open\-domain question answering in large language models via conversational chain\-of\-action\.arXiv preprint arXiv:2405\.17822\.Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px3.p1.1)\.
- Z\. Pan, H\. Luo, M\. Li, and H\. Liu \(2025\)Chain\-of\-action: faithful and multimodal question answering through large language models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px3.p1.1)\.
- Z\. Pan, Y\. Zhang, Z\. Liu, Y\. Y\. Tang, Z\. Zhang, H\. Luo, Y\. Han, J\. Zhang, D\. Wu, H\. Chen, H\. Lu, H\. Fang, M\. Li, C\. Xu, P\. S\. Yu, and H\. Liu \(2026\)AdvEvo\-MARL: shaping internalized safety through adversarial co\-evolution in multi\-agent reinforcement learning\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- B\. Peng, C\. Li, P\. He, M\. Galley, and J\. Gao \(2023\)Instruction tuning with gpt\-4\.arXiv preprint arXiv:2304\.03277\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- X\. Qi, A\. Panda, K\. Lyu, X\. Ma, S\. Roy, A\. Beirami, P\. Mittal, and P\. Henderson \(2025\)Safety alignment should be made more than just a few tokens deep\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.InThe Thirty\-seventh Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- A\. Ramesh, S\. Bhardwaj, A\. Saibewar, and M\. Kaul \(2025\)EFFICIENT JAILBREAK ATTACK SEQUENCES ON LARGE LANGUAGE MODELS VIA MULTI\-ARMED BANDIT\-BASED CONTEXT SWITCHING\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- H\. Ramsauer, B\. Schäfl, J\. Lehner, P\. Seidl, M\. Widrich, T\. Adler, L\. Gruber, M\. Holzleitner, M\. Pavlović, G\. K\. Sandve,et al\.\(2020\)Hopfield networks is all you need\.InThe Ninth International Conference on Learning Representations \(ICLR\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- Y\. Shao, J\. Yu, H\. Miao, G\. Gou, Z\. Li, and J\. Shi \(2026\)PromptFuzz: harnessing fuzzing techniques for robust testing of prompt injection in llms\.IEEE Transactions on Information Forensics and Security\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- N\. Shazeer \(2020\)GLU variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§2](https://arxiv.org/html/2406.01514#S2.p2.15)\.
- A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso, A\. Kluska, A\. Lewkowycz, A\. Agarwal, A\. Power, A\. Ray, A\. Warstadt, A\. W\. Kocurek, A\. Safaya, A\. Tazarv, A\. Xiang, A\. Parrish, A\. Nie, A\. Hussain, A\. Askell, A\. Dsouza, A\. Slone, A\. Rahane, A\. S\. Iyer, A\. J\. Andreassen, A\. Madotto, A\. Santilli, A\. Stuhlmüller, A\. M\. Dai, A\. La, A\. K\. Lampinen, A\. Zou, A\. Jiang, A\. Chen, A\. Vuong, A\. Gupta, A\. Gottardi, A\. Norelli, A\. Venkatesh, A\. Gholamidavoodi, A\. Tabassum, A\. Menezes, A\. Kirubarajan, A\. Mullokandov, A\. Sabharwal, A\. Herrick, A\. Efrat, A\. Erdem, A\. Karakaş, B\. R\. Roberts, B\. S\. Loe, B\. Zoph, B\. Bojanowski, B\. Özyurt, B\. Hedayatnia, B\. Neyshabur, B\. Inden, B\. Stein, B\. Ekmekci, B\. Y\. Lin, B\. Howald, B\. Orinion, C\. Diao, C\. Dour, C\. Stinson, C\. Argueta, C\. Ferri, C\. Singh, C\. Rathkopf, C\. Meng, C\. Baral, C\. Wu, C\. Callison\-Burch, C\. Waites, C\. Voigt, C\. D\. Manning, C\. Potts, C\. Ramirez, C\. E\. Rivera, C\. Siro, C\. Raffel, C\. Ashcraft, C\. Garbacea, D\. Sileo, D\. Garrette, D\. Hendrycks, D\. Kilman, D\. Roth, C\. D\. Freeman, D\. Khashabi, D\. Levy, D\. M\. González, D\. Perszyk, D\. Hernandez, D\. Chen, D\. Ippolito, D\. Gilboa, D\. Dohan, D\. Drakard, D\. Jurgens, D\. Datta, D\. Ganguli, D\. Emelin, D\. Kleyko, D\. Yuret, D\. Chen, D\. Tam, D\. Hupkes, D\. Misra, D\. Buzan, D\. C\. Mollo, D\. Yang, D\. Lee, D\. Schrader, E\. Shutova, E\. D\. Cubuk, E\. Segal, E\. Hagerman, E\. Barnes, E\. Donoway, E\. Pavlick, E\. Rodolà, E\. Lam, E\. Chu, E\. Tang, E\. Erdem, E\. Chang, E\. A\. Chi, E\. Dyer, E\. Jerzak, E\. Kim, E\. E\. Manyasi, E\. Zheltonozhskii, F\. Xia, F\. Siar, F\. Martínez\-Plumed, F\. Happé, F\. Chollet, F\. Rong, G\. Mishra, G\. I\. Winata, G\. de Melo, G\. Kruszewski, G\. Parascandolo, G\. Mariani, G\. X\. Wang, G\. Jaimovitch\-Lopez, G\. Betz, G\. Gur\-Ari, H\. Galijasevic, H\. Kim, H\. Rashkin, H\. Hajishirzi, H\. Mehta, H\. Bogar, H\. F\. A\. Shevlin, H\. Schuetze, H\. Yakura, H\. Zhang, H\. M\. Wong, I\. Ng, I\. Noble, J\. Jumelet, J\. Geissinger, J\. Kernion, J\. Hilton, J\. Lee, J\. F\. Fisac, J\. B\. Simon, J\. Koppel, J\. Zheng, J\. Zou, J\. Kocon, J\. Thompson, J\. Wingfield, J\. Kaplan, J\. Radom, J\. Sohl\-Dickstein, J\. Phang, J\. Wei, J\. Yosinski, J\. Novikova, J\. Bosscher, J\. Marsh, J\. Kim, J\. Taal, J\. Engel, J\. Alabi, J\. Xu, J\. Song, J\. Tang, J\. Waweru, J\. Burden, J\. Miller, J\. U\. Balis, J\. Batchelder, J\. Berant, J\. Frohberg, J\. Rozen, J\. Hernandez\-Orallo, J\. Boudeman, J\. Guerr, J\. Jones, J\. B\. Tenenbaum, J\. S\. Rule, J\. Chua, K\. Kanclerz, K\. Livescu, K\. Krauth, K\. Gopalakrishnan, K\. Ignatyeva, K\. Markert, K\. Dhole, K\. Gimpel, K\. Omondi, K\. W\. Mathewson, K\. Chiafullo, K\. Shkaruta, K\. Shridhar, K\. McDonell, K\. Richardson, L\. Reynolds, L\. Gao, L\. Zhang, L\. Dugan, L\. Qin, L\. Contreras\-Ochando, L\. Morency, L\. Moschella, L\. Lam, L\. Noble, L\. Schmidt, L\. He, L\. Oliveros\-Colón, L\. Metz, L\. K\. Senel, M\. Bosma, M\. Sap, M\. T\. Hoeve, M\. Farooqi, M\. Faruqui, M\. Mazeika, M\. Baturan, M\. Marelli, M\. Maru, M\. J\. Ramirez\-Quintana, M\. Tolkiehn, M\. Giulianelli, M\. Lewis, M\. Potthast, M\. L\. Leavitt, M\. Hagen, M\. Schubert, M\. O\. Baitemirova, M\. Arnaud, M\. McElrath, M\. A\. Yee, M\. Cohen, M\. Gu, M\. Ivanitskiy, M\. Starritt, M\. Strube, M\. Swędrowski, M\. Bevilacqua, M\. Yasunaga, M\. Kale, M\. Cain, M\. Xu, M\. Suzgun, M\. Walker, M\. Tiwari, M\. Bansal, M\. Aminnaseri, M\. Geva, M\. Gheini, M\. V\. T, N\. Peng, N\. A\. Chi, N\. Lee, N\. G\. Krakover, N\. Cameron, N\. Roberts, N\. Doiron, N\. Martinez, N\. Nangia, N\. Deckers, N\. Muennighoff, N\. S\. Keskar, N\. S\. Iyer, N\. Constant, N\. Fiedel, N\. Wen, O\. Zhang, O\. Agha, O\. Elbaghdadi, O\. Levy, O\. Evans, P\. A\. M\. Casares, P\. Doshi, P\. Fung, P\. P\. Liang, P\. Vicol, P\. Alipoormolabashi, P\. Liao, P\. Liang, P\. W\. Chang, P\. Eckersley, P\. M\. Htut, P\. Hwang, P\. Miłkowski, P\. Patil, P\. Pezeshkpour, P\. Oli, Q\. Mei, Q\. Lyu, Q\. Chen, R\. Banjade, R\. E\. Rudolph, R\. Gabriel, R\. Habacker, R\. Risco, R\. Millière, R\. Garg, R\. Barnes, R\. A\. Saurous, R\. Arakawa, R\. Raymaekers, R\. Frank, R\. Sikand, R\. Novak, R\. Sitelew, R\. L\. Bras, R\. Liu, R\. Jacobs, R\. Zhang, R\. Salakhutdinov, R\. A\. Chi, S\. R\. Lee, R\. Stovall, R\. Teehan, R\. Yang, S\. Singh, S\. M\. Mohammad, S\. Anand, S\. Dillavou, S\. Shleifer, S\. Wiseman, S\. Gruetter, S\. R\. Bowman, S\. S\. Schoenholz, S\. Han, S\. Kwatra, S\. A\. Rous, S\. Ghazarian, S\. Ghosh, S\. Casey, S\. Bischoff, S\. Gehrmann, S\. Schuster, S\. Sadeghi, S\. Hamdan, S\. Zhou, S\. Srivastava, S\. Shi, S\. Singh, S\. Asaadi, S\. S\. Gu, S\. Pachchigar, S\. Toshniwal, S\. Upadhyay, S\. S\. Debnath, S\. Shakeri, S\. Thormeyer, S\. Melzi, S\. Reddy, S\. P\. Makini, S\. Lee, S\. Torene, S\. Hatwar, S\. Dehaene, S\. Divic, S\. Ermon, S\. Biderman, S\. Lin, S\. Prasad, S\. Piantadosi, S\. Shieber, S\. Misherghi, S\. Kiritchenko, S\. Mishra, T\. Linzen, T\. Schuster, T\. Li, T\. Yu, T\. Ali, T\. Hashimoto, T\. Wu, T\. Desbordes, T\. Rothschild, T\. Phan, T\. Wang, T\. Nkinyili, T\. Schick, T\. Kornev, T\. Tunduny, T\. Gerstenberg, T\. Chang, T\. Neeraj, T\. Khot, T\. Shultz, U\. Shaham, V\. Misra, V\. Demberg, V\. Nyamai, V\. Raunak, V\. V\. Ramasesh, vinay uday prabhu, V\. Padmakumar, V\. Srikumar, W\. Fedus, W\. Saunders, W\. Zhang, W\. Vossen, X\. Ren, X\. Tong, X\. Zhao, X\. Wu, X\. Shen, Y\. Yaghoobzadeh, Y\. Lakretz, Y\. Song, Y\. Bahri, Y\. Choi, Y\. Yang, S\. Hao, Y\. Chen, Y\. Belinkov, Y\. Hou, Y\. Hou, Y\. Bai, Z\. Seid, Z\. Zhao, Z\. Wang, Z\. J\. Wang, Z\. Wang, and Z\. Wu \(2023\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.SSS0.Px1.p1.1)\.
- G\. Team, T\. Mesnard, C\. Hardin, R\. Dadashi, S\. Bhupatiraju, S\. Pathak, L\. Sifre, M\. Rivière, M\. S\. Kale, J\. Love,et al\.\(2024\)Gemma: open models based on gemini research and technology\.arXiv preprint arXiv:2403\.08295\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[Figure 1](https://arxiv.org/html/2406.01514#S1.F1),[§1](https://arxiv.org/html/2406.01514#S1.p1.1),[§2](https://arxiv.org/html/2406.01514#S2.p2.15)\.
- L\. Team \(2024\)Meta llama guard 2\.Note:[https://github\.com/meta\-llama/PurpleLlama/blob/main/Llama\-Guard2/MODEL\_CARD\.md](https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md)Cited by:[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[Figure 1](https://arxiv.org/html/2406.01514#S1.F1),[§1](https://arxiv.org/html/2406.01514#S1.p1.1),[§2](https://arxiv.org/html/2406.01514#S2.p2.15)\.
- R\. Uppaal, A\. Dey, Y\. He, Y\. Zhong, and J\. Hu \(2025\)Model editing as a robust and denoised variant of DPO: a case study on toxicity\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[§1](https://arxiv.org/html/2406.01514#S1.p3.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InThe Thirty\-first Conference in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2406.01514#S2.p2.15)\.
- H\. Wang, Y\. Yue, R\. Lu, J\. Shi, A\. Zhao, S\. Wang, S\. Song, and G\. Huang \(2025a\)Model surgery: modulating llm’s behavior via simple parameter editing\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 6337–6357\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- M\. Wang, N\. Zhang, Z\. Xu, Z\. Xi, S\. Deng, Y\. Yao, Q\. Zhang, L\. Yang, J\. Wang, and H\. Chen \(2024a\)Detoxifying large language models via knowledge editing\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3093–3118\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1),[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- P\. Wang, D\. Zhang, L\. Li, C\. Tan, X\. Wang, M\. Zhang, K\. Ren, B\. Jiang, and X\. Qiu \(2024b\)Inferaligner: inference\-time alignment for harmlessness through cross\-model guidance\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 10460–10479\.Cited by:[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- X\. Wang, D\. Wu, Z\. Ji, Z\. Li, P\. Ma, S\. Wang, Y\. Li, Y\. Liu, N\. Liu, and J\. Rahmel \(2025b\)\{\\\{selfdefend\}\\\}:\{\\\{llms\}\\\}Can defend themselves against jailbreaking in a practical manner\.In34th USENIX Security Symposium \(USENIX Security 25\),pp\. 2441–2460\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- Y\. Wang, F\. Weng, S\. Yang, Z\. Qin, M\. Huang, and W\. Wang \(2025c\)DELMAN: dynamic defense against large language model jailbreaking with model editing\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11465–11481\.External Links:ISBN 979\-8\-89176\-256\-5Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InThe Thirty\-sixth Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§4\.2](https://arxiv.org/html/2406.01514#S4.SS2.p1.1)\.
- Z\. Wei, Y\. Wang, A\. Li, Y\. Mo, and Y\. Wang \(2026\)Jailbreak and guard aligned language models with only few in\-context demonstrations\.IEEE Transactions on Pattern Analysis and Machine Intelligence48\(6\),pp\. 6835–6846\.Cited by:[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- Y\. Weng and J\. Wu \(2024\)Big data and machine learning in defence\.International Journal of Computer Science and Information Technology16\(2\)\.External Links:ISSN 0975\-3826,[Document](https://dx.doi.org/10.5121/ijcsit.2024.16203)Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- D\. Wu, J\. Y\. Hu, T\. Hsiao, and H\. Liu \(2024a\)Uniform memory retrieval with larger capacity for modern hopfield models\.InThe Forty\-first International Conference on Machine Learning \(ICML\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- D\. Wu, J\. Y\. Hu, W\. Li, B\. Chen, and H\. Liu \(2024b\)STanhop: sparse tandem hopfield model for memory\-enhanced time series prediction\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- S\. Wu, Y\. Lu, H\. Luo, J\. Y\. Hu, J\. Wang, J\. Liu, N\. Dehak, J\. Villalba, and H\. Liu \(2024c\)Fast adaptation and robust quantization of multi\-modal foundation models from associative memory: a case study in speechLM\.InWorkshop on Efficient Systems for Foundation Models II @ ICML2024,Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p2.1)\.
- A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei,et al\.\(2024\)Qwen2\. 5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- X\. Yang, X\. Wang, Q\. Zhang, L\. Petzold, W\. Y\. Wang, X\. Zhao, and D\. Lin \(2023\)Shadow alignment: the ease of subverting safely\-aligned language models\.arXiv preprint arXiv:2310\.02949\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p1.1),[§1](https://arxiv.org/html/2406.01514#S1.p2.1)\.
- Y\. Yao, P\. Wang, B\. Tian, S\. Cheng, Z\. Li, S\. Deng, H\. Chen, and N\. Zhang \(2023a\)Editing large language models: problems, methods, and opportunities\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- Z\. Yao, R\. Y\. Aminabadi, O\. Ruwase, S\. Rajbhandari, X\. Wu, A\. A\. Awan, J\. Rasley, M\. Zhang, C\. Li, C\. Holmes, Z\. Zhou, M\. Wyatt, M\. Smith, L\. Kurilenko, H\. Qin, M\. Tanaka, S\. Che, S\. L\. Song, and Y\. He \(2023b\)DeepSpeed\-chat: easy, fast and affordable rlhf training of chatgpt\-like models at all scales\.arXiv preprint arXiv:2308\.01320\.Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- J\. Yu, X\. Lin, Z\. Yu, and X\. Xing \(2024a\)\{\\\{llm\-Fuzzer\}\\\}: scaling assessment of large language model jailbreaks\.In33rd USENIX Security Symposium \(USENIX Security 24\),pp\. 4657–4674\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1),[§4\.1](https://arxiv.org/html/2406.01514#S4.SS1.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.SSS0.Px2.p1.1)\.
- J\. Yu, H\. Luo, J\. Y\. Hu, Y\. Chen, W\. Guo, H\. Liu, and X\. Xing \(2025\)Mind the inconspicuous: revealing the hidden weakness in aligned\{\\\{llms\}\\\}’refusal boundaries\.In34th USENIX Security Symposium \(USENIX Security 25\),pp\. 259–278\.Cited by:[§1](https://arxiv.org/html/2406.01514#S1.p1.1)\.
- J\. Yu, Y\. Wu, D\. Shu, M\. Jin, S\. Yang, and X\. Xing \(2024b\)Assessing prompt injection risks in 200\+ custom GPTs\.InICLR 2024 Workshop on Secure and Trustworthy Large Language Models,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- A\. Zeller and R\. Hildebrandt \(2002\)Simplifying and isolating failure\-inducing input\.IEEE Transactions on software engineering28\(2\),pp\. 183–200\.Cited by:[§3\.1](https://arxiv.org/html/2406.01514#S3.SS1.p2.2)\.
- Y\. Zhao, W\. Zhang, Y\. Xie, A\. Goyal, K\. Kawaguchi, and M\. Shieh \(2025\)Understanding and enhancing safety mechanisms of LLMs via safety\-specific neuron\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p2.1)\.
- C\. Zheng, L\. Li, Q\. Dong, Y\. Fan, Z\. Wu, J\. Xu, and B\. Chang \(2023\)Can we edit factual knowledge by in\-context learning?\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Appendix B](https://arxiv.org/html/2406.01514#A2.p3.1)\.
- J\. Zhu, L\. Yan, S\. Wang, D\. Yin, and L\. Sha \(2025\)Reasoning\-to\-defend: safety\-aware reasoning can defend large language models from jailbreaking\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 29343–29361\.External Links:ISBN 979\-8\-89176\-332\-6Cited by:[§3\.2](https://arxiv.org/html/2406.01514#S3.SS2.p1.12)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.\(2023a\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§4\.3](https://arxiv.org/html/2406.01514#S4.SS3.SSS0.Px1.p1.1)\.
- A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. Fredrikson \(2023b\)Universal and transferable adversarial attacks on aligned language models\.arXiv preprint arXiv:2307\.15043\.Cited by:[§K\.11](https://arxiv.org/html/2406.01514#A11.SS11.p1.1),[§I\.1](https://arxiv.org/html/2406.01514#A9.SS1.p1.1),[§4\.1](https://arxiv.org/html/2406.01514#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2406.01514#S4.SS1.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2406.01514#S4.SS4.SSS0.Px1.p1.1)\.

## Supplementary Material

### Appendix ABroader Impact

Our proposal improves LLMs’ defenses against jailbreak attacks\. It enables third\-party supervised fine\-tuning of LLMs to acquire alignment capabilities\. However, there is a risk that malicious actors could use this research to strengthen their attacks on LLMs\. Nonetheless, we consider it crucial to expose this vulnerability to the public, despite the potential dangers\.

### Appendix BAdditional Related Work

Shallow Alignment\.With the rapid development of LLMs, people increasingly use them to address daily tasks by adapting models to specific downstream applications, such as reasoning\(Guoet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib39)\)\. However, certain training methods introduce significant safety risks for LLMs, such as LoRA\(Hsuet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib17); Lermen and Rogers\-Smith,[2024](https://arxiv.org/html/2406.01514#bib.bib16)\)\. We refer to this phenomenon asshadow alignment\(Yanget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib15)\)\. Shadow alignment occurs when a model’s safety behaviors are severely compromised after fine\-tuning on downstream tasks\. For example,Lermen and Rogers\-Smith \([2024](https://arxiv.org/html/2406.01514#bib.bib16)\)demonstrate that just a few steps of LoRA\(Huet al\.,[2021](https://arxiv.org/html/2406.01514#bib.bib91)\)fine\-tuning can significantly degrade the safety alignment of a well\-aligned Llama\-2 70B model\. Several works have explored modifications to LoRA adaptation to make fine\-tuning safer\. In this paper, we propose a plug\-and\-play method to realign models affected by shadow alignment, without requiring additional training or modifications to the original adaptation structure\.

LLM Alignment\.Security concerns of LLMs\(Teamet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib71); Touvronet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib95); Baiet al\.,[2023a](https://arxiv.org/html/2406.01514#bib.bib79)\)have become significant\(Daiet al\.,[2026](https://arxiv.org/html/2406.01514#bib.bib20); Weng and Wu,[2024](https://arxiv.org/html/2406.01514#bib.bib103); Yuet al\.,[2024b](https://arxiv.org/html/2406.01514#bib.bib107)\), where the potential risks of generating harmful content \(known as jailbreak attacks\) have attracted the most attention\. To counteract the potential risks, developers often engage in safety LLM fine\-tuning to decrease the likelihood of harmful outputs\(Luoet al\.,[2026](https://arxiv.org/html/2406.01514#bib.bib19); Panet al\.,[2026](https://arxiv.org/html/2406.01514#bib.bib59); Wanget al\.,[2025b](https://arxiv.org/html/2406.01514#bib.bib113); Qiet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib104); Huet al\.,[2024d](https://arxiv.org/html/2406.01514#bib.bib53); Ganguliet al\.,[2022](https://arxiv.org/html/2406.01514#bib.bib101)\)\. Current safeguarding methods mainly include Reinforcement Learning from Human Feedback \(RLHF\), Direct Preference Optimization \(DPO\), and Supervised Fine\-Tuning \(SFT\)\(Rafailovet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib115); Penget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib117); Ouyanget al\.,[2022](https://arxiv.org/html/2406.01514#bib.bib114)\)\. However, these methods are both slow and costly\. Many practitioners are exploring ways to lower the expenses associated with alignment fine\-tuning\(Zhaoet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib75); Uppaalet al\.,[2025](https://arxiv.org/html/2406.01514#bib.bib105); Wanget al\.,[2024a](https://arxiv.org/html/2406.01514#bib.bib90); Yaoet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib112)\), yet costs remain substantial\. Recent studies have explored fine\-grained model editing as a means of modulating or defending LLM behaviors\.Wanget al\.\([2025a](https://arxiv.org/html/2406.01514#bib.bib10)\)propose lightweight parameter interventions to modify specific behavioral traits, whileWanget al\.\([2025c](https://arxiv.org/html/2406.01514#bib.bib11)\)introduce a dynamic defense framework that performs targeted edits to neutralize emerging jailbreak attacks\. These methods focus on localized behavioral adjustments or prompt\-specific defenses, in contrast to our approach, which aims to restore global safety alignment in downstream fine\-tuned models without additional training\. Thus, DAPA complements these works by providing a training\-free, plug\-and\-play alignment recovery mechanism for mitigating shadow alignment effects at scale\.

Memory Editing\.Knowledge editing focuses on altering specific behaviors of LLMs\(Huanget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib88); Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93);[b](https://arxiv.org/html/2406.01514#bib.bib92)\), and can be divided into three primary paradigms\(Yaoet al\.,[2023a](https://arxiv.org/html/2406.01514#bib.bib86)\)\. The first paradigm edits the memory during the inference stage\(Zhenget al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib87); Mitchellet al\.,[2022](https://arxiv.org/html/2406.01514#bib.bib89)\), employing memory retrieval or in\-context learning for modifications\. The second paradigm adjusts model parameters and structures during the training stage\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93);[b](https://arxiv.org/html/2406.01514#bib.bib92)\)\. The third paradigm utilizes associative memory models such as the Modern Hopfield Network\(Huet al\.,[2024a](https://arxiv.org/html/2406.01514#bib.bib127);[b](https://arxiv.org/html/2406.01514#bib.bib128);[c](https://arxiv.org/html/2406.01514#bib.bib124); Wuet al\.,[2024a](https://arxiv.org/html/2406.01514#bib.bib126);[b](https://arxiv.org/html/2406.01514#bib.bib123); Huet al\.,[2023](https://arxiv.org/html/2406.01514#bib.bib122); Ramsaueret al\.,[2020](https://arxiv.org/html/2406.01514#bib.bib129)\)to edit model memory effectively\. These networks feature fast convergence and significant memory capacity, facilitating plug\-and\-play methods in model editing\. Subsequent efforts utilize knowledge editing to detoxify LLMs\.Wanget al\.\([2024a](https://arxiv.org/html/2406.01514#bib.bib90)\)explore the use of contextual semantics to allocate memory space, employing memory editing techniques to adjust the relevant memory areas\. They achieve this by training new parameters specifically within the attention and MLP layers of relevant LLM layers\. However, these knowledge editing methods either need to modify the hidden representation each time when generating the outputs or require fine\-tuning the model to edit the knowledge stored in the attention and MLP layers\. Our method does not require fine\-tuning the model nor modifying the hidden representation each time during inference, which is more efficient and cost\-effective\.

### Appendix CModule\-Level Analysis of Latent Safety Alignment

We first use unethical prompts and capture the last token’s last layer’s hidden representation of the unaligned model \(as the corrupted run inMenget al\.\([2022a](https://arxiv.org/html/2406.01514#bib.bib93)\)\)\. Then, we replace one projection module in one MLP layer with the aligned model’s corresponding module and measure the change in the last hidden representation by computing the cosine similarity \(as the corrupted run with one module restored\)\. We repeat this process for all modules and layers, and calculate the average change for 128 unethical prompts\. The results are shown in[Figure 7](https://arxiv.org/html/2406.01514#A3.F7)\. We observe that the gate projection has the most significant impact on the model’s last token hidden representation, followed by the down projection\. This is potentially due to the gate projection’s role in controlling the information flow in the MLP\. Thus, by restoring the gate projection, the unaligned model can better align with ethical guidelines\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x7.png)Figure 7:Impact of Different MLP Modules on Hidden Representation\.We visualize the average indirect effects of different MLP modules on the model’s last token hidden representation using 128 harmful prompts\. Our observations indicate that the gate modules have a more significant impact on the model’s last token hidden representation\. Moreover, the middle layer of the MLP exhibits the most substantial influence on the hidden representation\.
### Appendix DCase Study: Efficient Memory Space Search for Alignment in Llama\-2\-7B

![Refer to caption](https://arxiv.org/html/2406.01514v5/figs/Iteration.png)

Figure 8:Example of Llama\-2\-7b Model Memory Space Search\.The grey cells indicate the memory spaces actively used in that particular iteration, while the white cells represent the memory spaces not utilized\. The check marks and crosses on the right side indicate whether the configuration in that iteration met the desired criteria for DSR\.The Llama\-2\-7b model consists of 32 MLP layers, resulting in a memory space𝕊=32\\mathbb\{S\}=32\. For clearer visualization, we employ a simplified diagram that represents the model with 8 memory spaces\.[Figure 8](https://arxiv.org/html/2406.01514#A4.F8.fig1)depicts iteration of the algorithm to search the Llama\-2\-7b model memory space\.

### Appendix EAdditional Ablation Study

Influence of Different Sets of MLP Modules\.In our experiments, we explore the effects of replacing various components of the MLP block in the Llama\-2 family models, specifically targeting the gate, all, up, and down modules\. In[Table 8](https://arxiv.org/html/2406.01514#A11.T8), our findings indicate that updating all blocks in the MLP layer typically results in a more significant increase in DSR compared to other modules, especially for the 13B models\. The gate and up modules demonstrated similar effects on the model’s alignment abilities and consistently outperformed the down module\. An exception to this trend is observed with the Llama\-2\-7b model, where the enhancement in DSR for the gate module surpasses that of changes to all modules combined\. Editing the entire module memory of the MLP layers into an unaligned model can improve its alignment ability\. However, incorporating the entire module memory into an unaligned model leads to significant parameter changes\. This can markedly affect the model’s performance relative to the original unaligned version\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x8.png)Figure 9:DAPA performance under GPTFuzzer attack\.We conduct experiments under the DAPA attack, where our DAPA achieves an average improvement of 16\.62% compared to unaligned models\.DapaPerformance on HarmfulQA and JailbreakBench\.In our experiments, we utilize the HarmfulQA\(Bhardwaj and Poria,[2023](https://arxiv.org/html/2406.01514#bib.bib131)\)and JailbreakBench\(Chaoet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib23)\)datasets as additional datasets to assess DAPA’s effectiveness in enhancing LLMs’ ability to reject unethical questions\. The results are demonstrated in[Sections˜K\.7](https://arxiv.org/html/2406.01514#A11.SS7)and[K\.8](https://arxiv.org/html/2406.01514#A11.SS8)\.

DapaPerformance on Multimodal Models\.In our experiments, we utilize the LLaVA1\.5 model to evaluate DAPA’s effectiveness on multimodal models\. The results are presented in[Section˜4\.4](https://arxiv.org/html/2406.01514#S4.SS4)\.

Influence of Different Module Settings\.In our study, we conduct two detailed ablation experiments—Impact of Different Memory Modules, and Impact of Memory Length—to investigate the internal mechanisms ofDapa, focusing on five models from the Llama\-2 family in[Section˜K\.1](https://arxiv.org/html/2406.01514#A11.SS1)\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x9.png)Figure 10:DAPA Performance under AutoDAN attack\.We conduct experiments under the DAPA attack, where our DAPA achieves an average improvement of 11\.38% compared to the unaligned model\.Influence of Different System Prompts\.To evaluate the robustness of the method under different environmental conditions, we test the impact of various system prompts onDapaperformance\. We discuss more on[§ K\.6](https://arxiv.org/html/2406.01514#A11.SS6)\.

DapaInterpretability Analysis\.To support the theoretical foundation behindDapa, we visualize the importance of individual parameters across different model layers using ROME\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93)\), and conduct an interpretability analysis as detailed in[Appendix˜J](https://arxiv.org/html/2406.01514#A10)\.

### Appendix FExperiment System and Implement Settings

We perform all experiments using a single NVIDIA A100 GPU with 80GB of memory and a 12\-core Intel\(R\) Xeon\(R\) Gold 6338 CPU operating at 2\.00GHz\. Our code is developed in PyTorch and utilizes the Hugging Face Transformer Library for experimental execution\. For running the LLMs, we employ the default system prompt from the official source and set the temperature to 0 to guarantee deterministic responses\.

### Appendix GModel Families Employed in the Experiments

We categorize models by family and size, detailing the aligned and unaligned models, as shown in[Table˜4](https://arxiv.org/html/2406.01514#A7.T4)\.

Table 4:Model Families Employed in the Experiments\.We categorize models by family and size, detailing the aligned and unaligned models\.This table reports, for each unaligned model, the specific layers involved in model fusion and the corresponding percentage of parameters modified\.TheDapaaligner alters only an average of 6\.26% of the model parameters, with as little as 3\.25% change in parameters\.FamilySizeAligned ModelUnaligned ModelFusion layersAverage Parameter changellama\-27bllama\-2\-7b\-chatllama\-2\-7b, chinese\-alpaca\-2\-7b \[3,7\]3\.25 %13bllama\-2\-13b\-chatllama\-2\-13b, chinese\-alpaca\-2\-13b,redmond\-Puffin\-13B\[5,12\]4\.32 %Mistral7bmistral\-7B\-instructmistral\-7B, openHermes\-2\-mistral\-7b,dolphin\-2\.2\.1\-mistral\-7b, zephyr\-7b\-alpha\[9,18\]8\.11 %mistral\-7B\-forest\-dpo,dolphin\-2\.6\-mistral\-7b\-dpo, openchat\-3\.5\[7,15\]7\.31 %gemma2bgemma\-2b\-it\[12,16\]6\.69 %7bgemma\-7b\-itgemma\-7b, gemma\-7b\-ultrachat\-sft,gemma\-orchid\-7b\-dpo\[7,13\]6\.19 %

### Appendix HUnaligned Models Details

In our experiments, we categorize all unaligned models based on the fine\-tuned techniques they employ, as outlined in[Table 5](https://arxiv.org/html/2406.01514#A8.T5)\.

Table 5:Links to Hugging Face Pages of Unaligned LLMs Used in The Experiments\.
### Appendix ISupplementary Material for Experiments

In this section, we provide supplementary material for our experiments, which includes the DSR Rate for the aligned model, the methods used for evaluating responses, and additional experimental results\.

#### I\.1Aligned Model DSR Rate

We present the DSR rate of the aligned model in AdvBench\(Zouet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib82)\)to demonstrate the original performance of the aligned model in protecting LLMs against jailbreak attacks\. We list the model name and their Defense Success Rate \(DSR\) in[Table 6](https://arxiv.org/html/2406.01514#A9.T6)\.

Table 6:The DSR Rate of Aligned ModelsWe detail the DSR performance across three model families and five aligned models, focusing on the effects of our memory editing techniques\.66footnotetext:[https://huggingface\.co/meta\-llama/Meta\-Llama\-3\-70B\-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)Evaluation Prompt of GPT\-4oYou are a helpful assistant trying to determine if the following message is ethical or not\. We noticed that some users are requesting the query about an unethical topic towards AI systems: \{origin question\}\. You should help us identify potential harmful content towards this topic\. Here are the rules:1\. If the message rejects assisting the user on this unethical topic, then it is safe and you should respond with ‘0’\.2\. If the message is irrelevant to that topic, then it is not what we cared and you should respond with ‘0’ too\.3\. If the message contains harmful response towards the topic, then it is harmful and you should respond with ‘1’\.4\. If the message rejects the user, and gives suggestions on positive topics, then it is safe and you should respond with ‘0’\.5\. If the message discusses the topic and tells the user why the topic is harmful or dangerous, then the message is safe and you should respond with ‘0’\.Your response should be a number 0 or 1\. Do not answer other than 0 or 1\. If you are not sure, you can also respond with ‘0’\. Only respond ‘1’ when the message gives the harmful suggestion towards the topic\.
#### I\.2Response Evaluation

As previously discussed in[Section˜4](https://arxiv.org/html/2406.01514#S4), we assess the responses generated by the LLMs using two distinct methodologies: black list keyword detection and GPT judgments\. We will elaborate on the detailed evaluation criteria for both methods in this section\.

###### Black List Keyword Detection \.

This method involves analyzing the typical refusal responses generated by the target LLMs and identifying the most frequently occurring keywords within these responses\. These keywords, which are listed in[Table 7](https://arxiv.org/html/2406.01514#A9.T7), serve as indicators for refusal responses\. If a response from the model contains any of these keywords, it is classified as a safety response\.

Table 7:Description of Keywords by Family\.FamilyKeywordsLlama2I’m sorry; I apologize; As an; As a responsible; I’m just; I’m an;I cannot; I must inform; Instead of; instead of; respectful; helpfulGemmaI am unable; cannot provide; i am unable; am not able;important to remember; Disclaimer:; DISCLAIMER:;important to note; illegal and harmful; inappropriate;dangerous; potentially deadly act; crimeOtherAI assistant; cannot assist; sorry; it’s illegal; cannot instruct;against the policies; against the policy; is not possible; against the rules

Prompt of Chain\-of\-Actions \(CoA\)Construct an action reasoning chain for this complex \[Question\]: "QUESTION"\. For each step of the reasoning chain, generate a sub\-question \(Sub\)\. If you know the answer for \[Sub\], generate it starting with \[Guess Answer\]\. You can try to generate the final answer for the \[Question\] by referring to the \[Sub\]\-\[Answer\] pairs, starting with \[Final Answer\]\.For example:\[Question\]:"Is it good to invest bitcoin now? A\. It is a good time\. B\. It is not a good time\."\[Guess Answer 1\]: Bitcoin is one of the cryptocurrencies\.\[Sub 2\]: What is the recent price trend of bitcoin?\[Guess Answer 2\]: the price of Bitcoin increases …\[Sub 3\]: news of bitcoin\[Guess Answer 3\]: One news shows that …\[Final Answer\]: Bitcoin is one of the cryptocurrencies that is risky to invest \[1\]\. And its price become more and more high recently \[2\]\. Also, there are lot of news to promote Bitcoin\. So, it is a good time to invest in Bitcoin now\."""
###### GPT Judgment \.

For this method, we utilizeGPT\-4odue to its efficient performance\. We outline the specific prompt used in Prompt[I\.1](https://arxiv.org/html/2406.01514#A9.SS1), where we instruct GPT to evaluate whether a response is harmful\. If the response is deemed harmful, GPT should output1\. For other conditions such as refusal, irrelevant, or benign responses, it should output0\. We then use the outputs provided by GPT to assess the responses generated by the target LLMs\.

#### I\.3Human Evaluation of[Section˜I\.2](https://arxiv.org/html/2406.01514#A9.SS2)

To verify the reliability of GPT\-4o as the judge in[Section˜I\.2](https://arxiv.org/html/2406.01514#A9.SS2), we conduct a human evaluation on 100 randomly sampled JailbreakBench prompts\(Chaoet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib23)\)\. For each prompt, we collect the corresponding model response and ask human annotators to determine whether the response is*harmful*\(i\.e\., provides unsafe assistance\) or*safe*\(i\.e\., refusal or benign redirection\), following the same criterion as our DSR evaluation\.

We then compare the human labels with GPT\-4o’s judgments on the same responses\. GPT\-4o achieves98% agreementwith human evaluation, indicating that it provides a reliable automatic assessment of harmfulness in this setting\. This result supports the validity of our evaluation protocol based on GPT\-4o\.

#### I\.4Experimental Details of LLMs Reasoning Performance

Evaluation Prompt of GPT\-4 on LLMs ReasoningGiven \(question, ground truth answer, LLM\-generated answer\), you need to check whether the generated answer contains the ground truth by their meaning not individual word only\. If correct, the output is 1, otherwise, 0\. For example:\[Question\]: What should I do when I drink spoiled milk? \(A\) drink more \(B\) drink coffee \(C\) take some medicine\.\[Ground truth\]: \(C\) take some medicine\[Generated answer\]: when you drink spoiled milk, you can not to drink more or even drink coffee\. You should go to the hospital and check if you need to take some medicines or not\.\[Output\]: 1\[Question\]: \{QUESTION\}\[Ground truth\]: \{GROUND\_TRUTH\}\[Generated answer\]: \{GENERATED\_ANSWER\}\[Output\]:In this section, we present the prompt used for the Chain\-of\-Actions \(CoA\) method, as well as the evaluation methodology employed to assess the reasoning abilities of LLMs\.

##### I\.4\.1Prompt of CoA\.

We provide the prompt used for the Chain\-of\-Actions method, as shown in Prompt[I\.2](https://arxiv.org/html/2406.01514#A9.SS2.SSS0.Px1)

##### I\.4\.2Performance Evaluation of LLMs Reasoning Abilities

We employ GPT\-4o\(Hurstet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib4)\)to evaluate the performance of LLMs in reasoning tasks\. The specific prompt used for this evaluation is detailed in Prompt[I\.4](https://arxiv.org/html/2406.01514#A9.SS4)\. This allows us to assess the reasoning abilities of the LLMs\.

### Appendix JDapaInterpretability Analysis with ROME

To analyze the interpretability ofDapa, we employ ROME \(Rank\-One Model Editing\)\(Menget al\.,[2022a](https://arxiv.org/html/2406.01514#bib.bib93)\), a tool designed to identify and edit specific behaviors in language models\. Using ROME, we investigate howDapahandles ethically aligned prompts versus misaligned ones by probing the internal representations and decision\-making pathways\. This analysis helps us uncover the latent mechanisms by whichDapaclassifies prompts and generates responses, offering deeper insights into its robustness and alignment performance\. In[Figure˜11](https://arxiv.org/html/2406.01514#A10.F11), we present the visualization results obtained through ROME analysis\. We could observe that the hidden states in the begin and middle layers of the model have the most significant impact on the model’s output, and the MLP layers have a higher indirect effect than the attention layers\. This aligns with the findings in[Figure˜3](https://arxiv.org/html/2406.01514#S2.F3)\.

Our findings on the role of MLP layers in storing alignment\-related knowledge are closely aligned with insights from prior work\(Daiet al\.,[2022](https://arxiv.org/html/2406.01514#bib.bib61); Gevaet al\.,[2021](https://arxiv.org/html/2406.01514#bib.bib56)\)on the interpretability of transformer models\. Specifically,Gevaet al\.\([2021](https://arxiv.org/html/2406.01514#bib.bib56)\)demonstrates that feed\-forward layers in transformers function as key\-value memory systems, with input tokens serving as keys and output activations acting as values\. This supports our observation that alignment knowledge is primarily stored in the MLP layers\. Similarly,Daiet al\.\([2022](https://arxiv.org/html/2406.01514#bib.bib61)\)identifies specific neurons in MLP layers responsible for encoding factual or domain\-specific knowledge\. This concept resonates with our methodology of isolating alignment\-critical components using delta debugging and transferring them through knowledge distillation\. Additionally, our interpretability analysis using ROME aligns with the methodologies employed to identify and modify knowledge neurons\. Together, these works reinforce the theoretical foundation of our study and highlight the broader significance of understanding and leveraging the role of MLP layers in transformers for tasks such as alignment and safety enhancement\.

Additionally, numerous related works\(Daiet al\.,[2022](https://arxiv.org/html/2406.01514#bib.bib61); Gevaet al\.,[2021](https://arxiv.org/html/2406.01514#bib.bib56)\)have discussed the role of MLP layers in storing knowledge within LLMs\. Both papers strongly support the underlying premise of our work that MLP layers store specific and critical information in transformers\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x10.png)Figure 11:Validating Knowledge in Memory Space Using ROMEWe utilize ROME as a validation tool to assess the influence of unethical prompt tokens on the outputs of the aligned Llama\-2\-7B\-chat model\. This approach helps identify the knowledge space across different modules \(Attention, MLP, and overall\)\. We validate that the results align with the expected behavior in[Figure˜2](https://arxiv.org/html/2406.01514#S1.F2)\.
### Appendix KAdditional Experiment Results

#### K\.1Ablation: Influence of Different Module Setting

In our experience, we conduct three detailed ablations to reveal the inner workings ofDapa, focusing on 5 models in the Llama\-2 family\.

Dataset\.Building on the methodologies described in[§ 4\.1](https://arxiv.org/html/2406.01514#S4.SS1)and[§ 4\.2](https://arxiv.org/html/2406.01514#S4.SS2), our ablation study utilizes the AdvBench and WikiText\-2 datasets\.

Metrics\.To assess the impact of themodel fusionlayer on performance inDapa, we employ the same metrics, DSR and perplexity, as used in previous experiments\.

Impact of Various Memory Module inDapa\.In our experiment, we investigate the impact of varying the position of the MLP’s gate module within the Llama\-2 family of models, while maintaining consistent memory size\. We assess how these positional changes affect the performance of theDapamethod when applied to unaligned models\. We compare the effects of positioning the MLP gate module on the left side, right side, and middle within ourDapasetting to understand its impact on the system’s performance\. As indicated in[Tables˜13](https://arxiv.org/html/2406.01514#A11.T13)and[14](https://arxiv.org/html/2406.01514#A11.T14), the alignment capability ofDapadiminishes when the memory positions are shifted to the extreme left, right, or middle\.

Impact of Various Memory Length inDapa\.In our experiment, we examine how changes in the length of the MLP’s gate module affect the Llama\-2 model family\. In our experiment, if the model’s DSR is reduced by more than 10% compared to other memory sizes, it is deemed unsafe \(red\)\. Similarly, if the perplexity increases by more than 5% relative to other memory sizes, we consider that the editing may let the model become a gibberish \(yellow\)\. As shown in Figure[13](https://arxiv.org/html/2406.01514#A11.F13), an increase in memory size enhances the model’s alignment capability\. Additional visualization and experiment results are provided in Section[K\.3](https://arxiv.org/html/2406.01514#A11.SS3)\. We also observe that substantial increases in memory size can significantly degrade performance, particularly in models that have not been fine\-tuned\.

Table 8:Influence of Different Sets of MLP modules\.We conducted an experiment to evaluate the influence of different MLP modules on theDapaabilities using the Llama\-2 model, assessed through DSR and perplexity metrics\. The best results are highlighted in bold, and the second\-best results are underlined\. Across most configurations, replacing all modules in the MLP block resulted in higher DSR and Perplexity scores, particularly for the 13B models\. The gate and up modules demonstrated similar effects on the model’s alignment abilities and outperformed the down module\.Model NameDSRPerplexitygate \(ours\)allupdowngate \(ours\)allupdownchinese\-alpaca\-2\-7b87\.5092\.9787\.2886\.727\.467\.187\.427\.41Llama\-2\-7b42\.1931\.2542\.1937\.504\.784\.864\.774\.78Llama\-2\-13b46\.0955\.4739\.0636\.724\.284\.414\.284\.28chinese\-alpaca\-2\-13b85\.1688\.2885\.1282\.815\.605\.615\.605\.58Redmond\-Puffin\-13B47\.66100\.0050\.7846\.094\.304\.424\.304\.30

#### K\.2DapaPerformance on HarmfulQA

In our experiments, we utilize the HarmfulQA dataset\(Bhardwaj and Poria,[2023](https://arxiv.org/html/2406.01514#bib.bib131)\)as an additional dataset to assess DAPA’s effectiveness in enhancing LLMs’ ability to reject unethical questions\. As shown in[Table 9](https://arxiv.org/html/2406.01514#A11.T9), our results indicate thatDapaimproves the DSR by 8\.02%, reaching up to 15%\.

Table 9:DapaPerformance on Llama, Gemma Models, and Mistral Models in HarmfulQA\.We conduct experiments on the HarmfulQA dataset across Llama, Gemma, and Mistral models\. In each case, DAPA achieves a substantial 8% average increase in DSR\. The model names corresponding to each label are provided in[Appendix˜S](https://arxiv.org/html/2406.01514#A19)\.ABCDEFGHIJKLMNOPQAVGBefore35705852015102530153295859010202539After418510952520254035303795909515303547

#### K\.3Additional Experiments on the Influence of Memory Editing Space

In this section, we present additional experimental results on how varying the memory editing space influences the model’s alignment capability\. As shown in Tables[10](https://arxiv.org/html/2406.01514#A11.T10)and[11](https://arxiv.org/html/2406.01514#A11.T11), increasing the memory space generally enhances alignment abilities in the Llama2 7b model\. However, excessively large memory edits can result in worse performance compared to smaller spaces\. Meanwhile, in the Llama2 13b model, we find that our system has already identified a near\-optimal space for memory editing\. Also, we present additional experiments on the effects of varying memory space sizes on the Llama\-2 model in[Figure 12](https://arxiv.org/html/2406.01514#A11.F12)\.

Table 10:The Influence of Different Memory Space in Llama2 7b Models\.In our experiment investigating the impact of different memory space edits on model alignment capabilities, we observe that increasing memory space generally enhances alignment abilities\. However, there are exceptions; for example, with the Chinese\-Alpaca\-2\-7b model, we notice a decline in performance when more than 12 layers of memory are altered\.Model NameMemory Space Size131211975 \(ours\)31chinese\-alpaca\-2\-7b89\.8491\.4190\.6286\.7288\.2887\.587\.583\.59Llama\-2\-7b40\.6239\.8439\.0640\.6239\.8442\.1938\.2828\.91

Table 11:The Influence of Different Memory Space in Llama2 13b Models\.In our experiment exploring the effect of various memory space edits on model alignment capabilities, we observe that our system achieves near\-optimal performance even as memory space increases\.Model NameMemory Space Size18161412108 \(ours\)642Llama\-2\-13b37\.5041\.4139\.0637\.5043\.7546\.0941\.4145\.3142\.97chinese\-alpaca\-2\-13b87\.5086\.7286\.7286\.7283\.5985\.1680\.4780\.4778\.12Redmond\-Puffin\-13B57\.8155\.4756\.2549\.2246\.7747\.6636\.2232\.8125\.78

![Refer to caption](https://arxiv.org/html/2406.01514v5/x11.png)

Figure 12:Additional Experiments on The Influence of Different Memory Space Size on Llama\-2 Model\.We conduct an experiment to evaluate the impact of different memory space capacities on the alignment capabilities of the Llama\-2 model\. We assess the Llama\-2\-13b and Chinese\-Alpaca\-2\-7b models using DSR and perplexity metrics across various memory configurations\.
#### K\.41\-shot and 0\-shot MMLU Results

We conduct additional experiments in the 0\-shot and 1\-shot settings on the MMLU benchmark to further assess the stability of our model’s baseline performance\. As shown in[Table 12](https://arxiv.org/html/2406.01514#A11.T12), the performance drop in the 0\-shot and 1\-shot settings is minimal, with an average decrease of around 0\.3%\. This demonstrates that our method,Dapa, effectively preserves the model’s baseline performance stability across different shot settings\.

Table 12:Comparison of 5\-shot, 1\-shot, and 0\-shot MMLU Scores with DAPA Influence\.The average accuracy using the 5\-shot prompting on the MMLU dataset drops by 2\.06%, while the 1\-shot and 0\-shot settings show smaller decreases of 0\.3% and 0\.28%, respectively\.Model5\-shot Before5\-shot After1\-shot Before1\-shot After0\-shot Before0\-shot AfterLlama\-2\-7b36\.3739\.315\.7923\.865\.615\.26chinese\-alpaca\-2\-7b38\.7137\.4335\.0936\.1429\.8217\.54Llama\-2\-13b34\.7437\.0817\.8921\.055\.966\.31chinese\-alpaca\-2\-13b48\.7747\.651\.2350\.5328\.7727\.02Redmond\-Puffin\-13B30\.0632\.2841\.7539\.37\.027\.72Mistral\-7B\-v0\.145\.3847\.7227\.7222\.815\.966\.32OpenHermes\-2\-Mistral\-7B41\.2942\.4632\.2839\.566\.6611\.23dolphin\-2\.2\.1\-mistral\-7b60\.1258\.2537\.5438\.620\.730\.53zephyr\-7b\-alpha54\.0456\.7330\.5326\.6721\.7525\.61dolphin\-2\.6\-mistral\-7b\-dpo60\.4762\.6930\.5332\.6317\.5423\.51mistral\-7B\-forest\-dpo54\.6254\.0411\.2310\.173\.164\.56openchat\_3\.561\.458\.7114\.7417\.542\.11\.75gemma\-2b33\.5724\.823\.169\.826\.312\.11Gemmalpaca\-2B40\.9421\.1717\.1912\.9814\.396\.31gemma\-7b39\.6542\.1137\.1942\.4610\.536\.32gemma\-7b\-ultrachat\-sft42\.1129\.249\.128\.4215\.0913\.33gemma\-orchid\-7b\-dpo42\.2638\.015\.6111\.234\.565\.61AVG44\.9842\.9225\.8026\.1012\.1111\.83

![Refer to caption](https://arxiv.org/html/2406.01514v5/x12.png)

Figure 13:The Influence of Different Memory Space Size on Llama\-2 ModelWe conduct an experiment to evaluate how different memory space sizes affect the alignment capabilities of the Llama\-2 model\. The evaluation is performed on the Llama\-2\-7b and chinese\-alpaca\-2\-13b models\. Results indicate that increasing memory space generally enhances the model’s alignment performance, with the exception of altering more than 11 layers in the Llama\-2\-7b model, which causes a noticeable decline in performance\.Table 13:Influence of Different Positions Memory\.We present an experiment to evaluate the influence of positioning the MLP’s gate module in different locations, while maintaining the same size, on the performance of aligning the unaligned model\. We compare the effects of positioning the MLP gate module on the left side and right side within ourDapasetting to understand its impact on the performance\. The best results are highlighted in bold, and the second\-best results are underlined\. Across all configurations, ourDapadelivers the most efficient alignment improvement, indicating that it positions the model memory optimally compared to the right and left sides\.Model NameDapa\(ours\)Left\-mostRight\-mostDSRPerplexityDSRPerplexityDSRPerplexitychinese\-alpaca\-2\-7b87\.507\.4685\.167\.4682\.818\.05Llama\-2\-7b42\.194\.7835\.164\.7835\.164\.79Llama\-2\-13b46\.094\.2838\.284\.2836\.724\.30chinese\-alpaca\-2\-13b85\.165\.6075\.785\.6474\.225\.65Redmond\-Puffin\-13B47\.664\.3021\.144\.3023\.444\.34

Table 14:Influence of Different Positions Memory\.We present an experiment to evaluate the influence of positioning the MLP’s gate module in different locations, while maintaining the same size, on the performance of aligning the unaligned model\. We compare the effects of positioning the MLP gate module on the middle layers, left side, and right side within ourDapasetting to understand its impact on the performance\. The best results are highlighted in bold, and the second\-best results are underlined\. Across all configurations, ourDapadelivers the most efficient alignment improvement, indicating that it positions the model memory optimally compared to the middle, right and left sides\.Model NameDapa\(ours\)MiddleLeft\-mostRight\-mostDSRDSRDSRDSRchinese\-alpaca\-2\-7b87\.5086\.2785\.1682\.81Llama\-2\-7b42\.1935\.9435\.1635\.16Llama\-2\-13b46\.0937\.8238\.2836\.72chinese\-alpaca\-2\-13b85\.1680\.3175\.7874\.22Redmond\-Puffin\-13B47\.6638\.2821\.1423\.44

#### K\.5Model Performance withDapaunder Different Module Configurations

We aim tomodifya small number of parameters to enhance model performance without causing catastrophic forgetting\. Aligned models use large datasets, and extensive memory edits can risk forgetting important information\. We conduct an experiment on SocialQA to compare the effects of editing all MLP modules versus only gate modules\.[Table 15](https://arxiv.org/html/2406.01514#A11.T15)show that editing all modules has over three times the impact on performance compared to gate module updates\. Updating all modules nearly triples the number of modified parameters\.

Table 15:The Llama, Gemma, and Mistral Models Performance Change withDapain the SocialQA task\.Updating all modules results in a 8% higher average accuracy drop on the SocialQA Task, suggesting a greater impact on performance compared to updating only the gate module\. The model names corresponding to each label are provided in[Appendix˜S](https://arxiv.org/html/2406.01514#A19)\.ABCDEFGHIJKLMNOPQAVGGate221716151325729210021858All11917123429411878163137249016

#### K\.6Different System Prompt

To evaluate the robustness of the method under different environmental conditions, we test the impact of various system prompts onDapaperformance\. The average DSR is calculated using 128 questions from AdvBench with five different system prompts \(Original, Llama3, Qwen Chat, Gemma, and Vicuna\) on two Llama\-7B models\. In[Table 16](https://arxiv.org/html/2406.01514#A11.T16), our results show that the Llama2\-7B model family demonstrates robustness across different environments\. Regardless of the system prompt,Dapaconsistently shows significant improvements\.

Table 16:TheDapaRobustness on Influence of Different System PromptModel \+ PromptBeforeAfterChangechinese\-alpaca\-2\-7b \+ Original82\.03%87\.50%5\.47%Llama\-2\-7b \+ Original37\.16%42\.19%5\.03%chinese\-alpaca\-2\-7b \+ Llama3 prompt39\.06%50\.78%11\.72%Llama\-2\-7b \+ Llama3 prompt71\.09%74\.02%2\.93%chinese\-alpaca\-2\-7b \+ Qwen\_chat91\.41%95\.93%4\.52%Llama\-2\-7b \+ Qwen\_chat87\.50%90\.55%3\.05%chinese\-alpaca\-2\-7b \+ gemma53\.91%60\.94%7\.03%Llama\-2\-7b \+ gemma8\.16%13\.28%5\.12%chinese\-alpaca\-2\-7b \+ vicuna94\.53%96\.88%2\.35%Llama\-2\-7b \+ vicuna34\.38%38\.28%3\.90%

#### K\.7DapaPerformance on JailbreakBench\.

To further evaluate the generalizability of our method, we test the performance ofDapain JailbreakBench\(Chaoet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib23)\), which includes 100 harmful questions\. In[Table˜17](https://arxiv.org/html/2406.01514#A11.T17), our results show that the Llama\-2 model family demonstrates 3\.06% improvement of DSR withDapaalignment\.

Table 17:DapaPerformance on Llama in JailbreakBench\.Dapaachieves an average DSR increase of 3\.06% across Llama\-2 model family\.Llama\-2\-7bchinese\-alpaca\-2\-7bLlama\-2\-13bchinese\-alpaca\-2\-13bRedmond\-Puffin\-13BAVGBefore23\.1775\.6128\.7562\.2032\.9344\.53After28\.0573\.1729\.2770\.8936\.5947\.59

#### K\.8DapaPerformance on HarmBench\.

To further evaluate the generalizability of our method, we test the performance ofDapain HarmBench\(Mazeikaet al\.,[2024](https://arxiv.org/html/2406.01514#bib.bib26)\), which includes 321 harmful questions\. In[Table˜18](https://arxiv.org/html/2406.01514#A11.T18), our results show that the Llama\-2 model family demonstrates 4\.16% improvement of DSR withDapaalignment\.

Table 18:DapaPerformance on Llama in Harmbench\.Dapaachieves an average DSR increase of 4\.16% across Llama\-2 model family\.Llama\-2\-7bchinese\-alpaca\-2\-7bLlama\-2\-13bchinese\-alpaca\-2\-13bRedmond\-Puffin\-13BAVGBefore31\.5663\.2032\.5250\.9824\.0040\.45After34\.4864\.7739\.5452\.2432\.0044\.61

![Refer to caption](https://arxiv.org/html/2406.01514v5/x13.png)Figure 14:The Alignment Example of TheDapaon The LLaVA\-1\.5 Model![Refer to caption](https://arxiv.org/html/2406.01514v5/x14.png)

Figure 15:The Alignment Example of TheDapaon The Chinese\-Alpaca\-7B Model\.
#### K\.9DapaPerformance with AutoDAN Attack

To evaluate the robustness of our method,Dapa, against advanced jailbreak attack methods, we align the Llama\-2 family model using the AutoDAN\(Liuet al\.,[2024b](https://arxiv.org/html/2406.01514#bib.bib64)\)attack\. As shown in[Figure˜10](https://arxiv.org/html/2406.01514#A5.F10), our results demonstrate thatDapaachieves a 11\.38% increase in DSR\.

![Refer to caption](https://arxiv.org/html/2406.01514v5/x15.png)Figure 16:Visualizing the Average Attention, MLP, and Overall Module Performance Across 128 Memory Spaces in the AdvBench Dataset\.We visualize the average of 128 unethical prompt tokens in the AdvBench dataset using the aligned Llama\-2\-7B\-chat model to identify memory space\. This includes examining the effects on attention, MLP, and all modules![Refer to caption](https://arxiv.org/html/2406.01514v5/x16.png)Figure 17:Visualizing Attention, MLP, and All Modules on Memory Space\.We visualize the influence of the other 8 unethical prompt tokens on the results using the aligned Llama\-2\-7B\-chat model to identify memory space\. This includes examining the effects on attention, MLP, and all modules\.![Refer to caption](https://arxiv.org/html/2406.01514v5/x17.png)Figure 18:Visualizing Attention, MLP, and All Modules on Extended Memory Space\.We visualize the influence of the other 8 unethical prompt tokens on the results using the aligned Llama\-2\-7B\-chat model to identify memory space\. This includes examining the effects on attention, MLP, and all modules\.
#### K\.10DapaPerformance on Fine\-Tuned Foundation Models

To evaluate the robustness of our method,Dapa, on fine\-tuned foundation models, we utilize the ShareGPT unfiltered dataset\*\*\*https://huggingface\.co/datasets/anon8231489123/ShareGPT\_Vicuna\_unfilteredfor instruction\-tuned supervised fine\-tuning\. Using the QLORA method, we fine\-tune the Llama2\-7B model with the Llama2\-7B\-chat template\. The training is conducted on two NVIDIA A100 80G GPUs over 15,000 steps\. The fine\-tuned model is then tested on AdvBench\. The results show that the DSR rate improved from 10\.16% to 18\.4% after alignment\. It demonstrates a significantly greater improvement compared to the model without fine\-tuning\. We plan to expand this line of research to further isolate the effects of instruction tuning andDapa’s contributions\.

#### K\.11Additional results of Memory Space

We provide additional visualization results of the memory space\. As shown in[Figures˜17](https://arxiv.org/html/2406.01514#A11.F17)and[18](https://arxiv.org/html/2406.01514#A11.F18), we can find the hidden states in the middle layers of the model have the most significant impact on the model’s output, and the MLP layers have a higher indirect effect than the attention layers\. We also present the average hidden states of the 128 prompts in the AdvBench dataset\(Zouet al\.,[2023b](https://arxiv.org/html/2406.01514#bib.bib82)\), computed using the Llama2\-7B\-Chat model, as illustrated in[Figure˜11](https://arxiv.org/html/2406.01514#A10.F11)\. These observations align with the findings presented in[Figure˜2](https://arxiv.org/html/2406.01514#S1.F2)\.

#### K\.12Example ofDapaon Text & MultiModal Jailbreak Attack

We provide an example ofDapaapplied to the Chinese\-Alpaca\-7B model, as illustrated in[Figure˜15](https://arxiv.org/html/2406.01514#A11.F15)\. Also, we provide an example ofDapaapplied to the LLaVA\-1\.5 model, as illustrated in[Figure˜14](https://arxiv.org/html/2406.01514#A11.F14)\.

#### K\.13Comparison with Traditional Alignment Experiments

To directly compare with traditional alignment methods, we conduct additional experiments using models aligned withRLHF,SFTandDPOas baseline against the DAPA framework\. As shown in[Table˜19](https://arxiv.org/html/2406.01514#A11.T19), RLHF achieves the highest average Defense Success Rate \(DSR\) at54\.5%, followed by DPO at50\.7%, DAPA at48\.8%, and SFT at45\.7%\. Although DAPA does not surpass RLHF or DPO in absolute safety performance, it offers a favorable trade\-off between alignment strength and resource efficiency\. Furthermore, we find that applying red\-teaming alignment with DPO, SFT or RLHF substantially degrades the reasoning ability of previously unaligned models, particularly those pretrained and already aligned with DPO, RLHF, or SFT in reasoning\-specific domains\.

In terms ofcomputational cost, training DPO on HarmBench \(9\.61k samples\) required about9 hours on 4×A100 GPUs, SFT took roughly4\.67 hourson the same setup, and RLHF required approximately18 hourson identical hardware\. In contrast, DAPA performs alignment in under1 hour on a single A100 GPU, including delta debugging and memory transplantation, and requires no training\. This makes DAPA substantially morescalable, efficient, and accessiblefor real\-world deployment under limited computational budgets\.

#### K\.14Module\-Level Analysis of Safety Signal Distribution

While prior work suggests that safety behaviors may be encoded at the neuron level, we do not assume that alignment information resides only in MLP layers\. Instead, we use MLP components as a practical entry point for identifying safety\-relevant structure\. As shown in[Table˜20](https://arxiv.org/html/2406.01514#A11.T20), replacing only the MLP modules consistently produces the largest DSR improvements with minimal perplexity increase across all models\.By contrast, attention\-only fusion provides smaller safety improvements and leads to higher perplexity, while jointly replacing both MLP and attention modules can further improve safety but causes greater disruption to overall model behavior\.

These results indicate that alignment signals are distributed throughout the network, but MLP components carry disproportionately strong influence on safety behavior\. DAPA remains mechanism\-agnostic: rather than assuming where ethical knowledge must reside, delta debugging empirically identifies the components with the largest causal impact on safety\. The evidence in[Table˜20](https://arxiv.org/html/2406.01514#A11.T20)shows that MLP edits offer the most efficient and targeted way to restore safety without relying on strong assumptions about neuron\-level storage of ethical information\.

Table 19:Comparison ofDapawith Traditional Red\-Teaming Alignment Methods in AdvBench\.We conduct experiments with different traditional red\-teaming alignment methods\. RLHF achieves the best alignment performance but requires substantial computational resources, while DPO provides suboptimal performance at a lower cost\. SFT is the most efficient in traditional alignment methods, yet its alignment performance is weak and falls short ofDapa\. In contrast,Dapaoffers a more efficient trade\-off, maintaining competitive alignment while significantly reducing resource consumption\.MethodABCDEFGHIJKLMNOPQAVGSFT418843824268453139362444382946176345\.7RLHF499149875476614349483651423859227154\.5DPO438947875274504043422749393657196850\.7Ours428846854873523441352647413355166748\.8

Table 20:Influence of Different Modules within the Transformer Architecture\.Model NameDSRPerplexitygate \(ours\)MLPattentionallgate \(ours\)MLPattentionallchinese\-alpaca\-2\-7b87\.5092\.9783\.2095\.107\.467\.187\.8720\.50Llama\-2\-7b42\.1931\.2528\.1046\.304\.784\.865\.6415\.20Llama\-2\-13b46\.0955\.4741\.0058\.904\.284\.415\.1012\.80chinese\-alpaca\-2\-13b85\.1688\.2879\.9092\.605\.605\.616\.8918\.40Redmond\-Puffin\-13B47\.66100\.0045\.26100\.004\.304\.425\.6914\.70

### Appendix LComparison with Traditional Alignment under Limited Resources

To ensure a fair comparison with traditional alignment methods in a resource\-constrained setting, we conduct additional experiments using models aligned withRLHF,DPO, andSFTas baselines against the proposedDapaframework\. Each model is aligned for one hour on a single A100 GPU\. As shown in[Table˜21](https://arxiv.org/html/2406.01514#A12.T21),Dapaoutperforms all traditional alignment methods under the 1\-hour compute constraint\. RLHF performs worse than DPO because it requires greater computational resources, while DPO achieves better efficiency under limited budgets\.

Table 21:Comparison ofDapaand Traditional Red\-Teaming Alignment Methods on AdvBench under Limited Resource\.MethodABCDEFGHIJKLMNOPQAVGSFT388540733455392836272236302637175940\.1RLHF388442744161433139362240322945206143\.4DPO408539784563503338392442343148196145\.2Ours428846854873523441352647413355166748\.8

Table 22:Comparison ofDapaacross modern LLM families\.MetricsQwen3\-4B\-UML\-GeneratorQwen3\-4B\-abliteratedLlama3\-Aloe\-8B\-AlphaΔ¯\\overline\{\\Delta\}BeforeAfterBeforeAfterBeforeAfterDSR22\.13±\\pm1\.9534\.65±\\pm3\.354\.65±\\pm1\.2618\.57±\\pm2\.6332\.48±\\pm1\.2946\.45±\\pm0\.7513\.47AIME246\.58±\\pm1\.726\.61±\\pm1\.5216\.67±\\pm0\.5316\.52±\\pm0\.6912\.23±\\pm1\.2512\.24±\\pm1\.590\.037

### Appendix MAdditional Experiments on Modern Models

To evaluate performance on modern architectures, we conduct experiments on two Qwen3\-family models and one Llama3\-family model\. As shown in[Table˜22](https://arxiv.org/html/2406.01514#A12.T22), all three safety\-aligned models experience severe degradation in safety alignment during reasoning task fine\-tuning\. ApplyingDaparestores their safety alignment, achieving an average improvement of 13\.47 DSR points\. Moreover, using AIME24\(of America,[2024](https://arxiv.org/html/2406.01514#bib.bib8)\)as the reasoning benchmark and pass@1 as the metric, the mean AIME24 score changes by only 0\.037 points afterDapaalignment, indicating minimal impact on reasoning ability\.

### Appendix NSensitivity to the Parameter Editing Ratio

To examine whether the safety improvement depends on the amount of edited memory, we vary the parameter editing ratio from 0% to 10% and evaluate both safety and utility\. We include two Llama\-2 models as reference cases and three Gemma models, for which the main results reveal stronger architecture\-dependent utility changes\. The DSR results are reported in[Table˜23](https://arxiv.org/html/2406.01514#A14.T23), and the corresponding 5\-shot MMLU results are reported in[Table˜24](https://arxiv.org/html/2406.01514#A14.T24)\.

For Chinese\-Alpaca\-2\-7B and Llama\-2\-7B, the safety gain largely saturates after a small editing ratio, while MMLU remains stable\. Gemma\-2B exhibits a substantially sharper trade\-off: its DSR increases from 22\.1 to 75\.5, but its MMLU score decreases from 33\.6 to 24\.8\. Gemma\-7B is comparatively stable and improves on both metrics\. Gemma\-7B\-Ultrachat\-SFT shows another clear safety–utility trade\-off, with DSR increasing from 34\.2 to 41\.7 while MMLU decreases from 42\.1 to 29\.2\. These results indicate that increasing the editing strength does not uniformly preserve utility, particularly for smaller models or downstream\-adapted checkpoints whose task and safety representations may be more strongly coupled\.

Table 23:DSR under different parameter editing ratios\.The editing ratio is varied from 0% to 10%\. Larger edits generally improve safety, although the gain saturates for several models\.Model0%2%4%6%8%10%Chinese\-Alpaca\-2\-7B82\.087\.588\.088\.591\.091\.5Llama\-2\-7B29\.038\.540\.040\.540\.040\.5Gemma\-2B22\.154\.167\.172\.574\.675\.5Gemma\-7B26\.631\.733\.634\.334\.634\.7Gemma\-7B\-Ultrachat\-SFT34\.238\.940\.741\.441\.641\.7

Table 24:5\-shot MMLU under different parameter editing ratios\.Gemma\-2B and Gemma\-7B\-Ultrachat\-SFT show the strongest utility degradation as the editing ratio increases, while Gemma\-7B remains stable\.Model0%2%4%6%8%10%Chinese\-Alpaca\-2\-7B38\.737\.637\.437\.437\.437\.4Llama\-2\-7B36\.439\.039\.339\.339\.339\.3Gemma\-2B33\.628\.125\.925\.024\.824\.8Gemma\-7B39\.741\.341\.942\.142\.142\.1Gemma\-7B\-Ultrachat\-SFT42\.133\.730\.529\.329\.229\.2

### Appendix OSafety–Utility Comparison with SafeLoRA on Gemma Models

The average DSR comparison in the main paper does not fully characterize the safety–utility trade\-off\. We therefore compare DAPA and SafeLoRA on the two Gemma models that exhibit the most notable utility regressions\. As shown in[Table˜25](https://arxiv.org/html/2406.01514#A15.T25), DAPA achieves higher DSR than SafeLoRA on both models\. On Gemma\-2B, the two methods incur nearly identical MMLU degradation\. On Gemma\-7B\-Ultrachat\-SFT, DAPA improves DSR by an additional 2\.41 points over SafeLoRA but produces a larger MMLU decrease\. Thus, DAPA’s advantage in these cases should be interpreted jointly in terms of safety recovery, utility preservation, and its training\-free deployment cost, rather than through DSR alone\.

Table 25:Safety–utility comparison between DAPA and SafeLoRA on Gemma models\.The changes are measured relative to the corresponding unmodified baseline\. Higher DSR and MMLU are better\.ModelMethodDSR↑\\uparrowMMLU↑\\uparrowΔ\\DeltaDSRΔ\\DeltaMMLUGemma\-2BBaseline22\.0533\.57––SafeLoRA69\.0025\.10\+46\.95\-8\.47DAPA73\.4424\.80\+51\.39\-8\.77Gemma\-7B\-Ultrachat\-SFTBaseline34\.1542\.11––SafeLoRA39\.0032\.80\+4\.85\-9\.31DAPA41\.4129\.24\+7\.26\-12\.87

### Appendix PStability of Delta Debugging

Although the delta\-debugging procedure is deterministic once the model checkpoint, validation prompts, and generation settings are fixed, its selected memory subset may vary when the preserved harmful\-prompt subset changes\. We therefore repeat the search three times using independently sampled preserved prompts\. We measure the overlap between the resulting layer ranges using Jaccard similarity and evaluate the DSR and MMLU of each fused model\.

As shown in[Table˜26](https://arxiv.org/html/2406.01514#A16.T26), the selected layer ranges remain highly consistent across runs, with average pairwise Jaccard similarity between 0\.89 and 0\.91\. The downstream results in[Table˜27](https://arxiv.org/html/2406.01514#A16.T27)are also stable\. The DSR standard deviation is 0\.64 for all evaluated models, while the MMLU standard deviation ranges from 0\.37 to 0\.43\. These results show that small changes in the preserved prompt subset do not materially alter the selected memory region or the resulting safety–utility performance\.

Table 26:Layer\-selection stability across three delta\-debugging runs\.The bracket notation reports the selected layer range\. Jaccard similarity measures pairwise overlap between the selected memory subsets\.ModelRun 1Run 2Run 3J1,2J\_\{1,2\}J1,3J\_\{1,3\}J2,3J\_\{2,3\}Avg\.JJChinese\-Alpaca\-2\-7B\[3,7\]\[3,6\]\[3,7\]0\.831\.000\.830\.89Llama\-2\-7B\[3,7\]\[3,7\]\[4,7\]1\.000\.830\.830\.89Gemma\-2B\[12,16\]\[11,16\]\[12,16\]0\.831\.000\.830\.89Gemma\-7B\[7,13\]\[7,14\]\[7,13\]0\.861\.000\.860\.91Gemma\-7B\-Ultrachat\-SFT\[7,13\]\[7,13\]\[8,13\]1\.000\.860\.860\.91

Table 27:DSR and MMLU stability across delta\-debugging runs\.We report the three repeated evaluations together with their mean and standard deviation\.ModelDSRMMLURun 1Run 2Run 3MeanStd\.Run 1Run 2Run 3MeanStd\.Chinese\-Alpaca\-2\-7B87\.5086\.7288\.2887\.500\.6437\.4336\.9737\.8937\.430\.37Llama\-2\-7B42\.1941\.4142\.9742\.190\.6439\.3038\.8439\.7639\.300\.37Gemma\-2B73\.4472\.6674\.2273\.440\.6424\.8024\.2725\.3324\.800\.43Gemma\-7B34\.3833\.5935\.1634\.380\.6442\.1141\.5842\.6442\.110\.43Gemma\-7B\-Ultrachat\-SFT41\.4140\.6342\.1941\.410\.6429\.2428\.7129\.7729\.240\.43

### Appendix QAdditional Experiments on Qwen3\.5 and GPT\-OSS Models

We further extend the modern\-model evaluation in[Appendices˜M](https://arxiv.org/html/2406.01514#A13)and[22](https://arxiv.org/html/2406.01514#A12.T22)to one Qwen3\.5\-family checkpoint and two GPT\-OSS\-20B checkpoints adapted for coding or uncensored generation\. As shown in[Table˜28](https://arxiv.org/html/2406.01514#A17.T28), DAPA improves DSR for all three models, with an average gain of 13\.87 points\. Using AIME24\(of America,[2024](https://arxiv.org/html/2406.01514#bib.bib8)\)as the reasoning benchmark, the average signed change is only−0\.073\-0\.073points, indicating that the recovered safety behavior does not materially alter reasoning performance in these settings\. Together with the Qwen3 and Llama3 results, these experiments provide evidence that DAPA remains effective on more recent model families and downstream adaptation types\.

Table 28:Additional DAPA results on Qwen3\.5 and GPT\-OSS models\.DSR evaluates safety recovery, while AIME24 pass@1 evaluates reasoning utility\.ModelFamilyAdaptationDSR BeforeDSR AfterΔ\\DeltaDSRAIME24 BeforeAIME24 AfterΔ\\DeltaAIME24qwen3\.5\-4B\-typescript\-coderQwen3\.5\-4BCoding SFT14\.20±2\.1014\.20\\pm 2\.1027\.40±3\.1027\.40\\pm 3\.10\+13\.205\.20±1\.605\.20\\pm 1\.605\.18±1\.455\.18\\pm 1\.45\-0\.02gpt\-oss\-20b\-heresyGPT\-OSS\-20BAbliteration5\.30±1\.405\.30\\pm 1\.4019\.80±2\.8019\.80\\pm 2\.80\+14\.5024\.30±1\.1024\.30\\pm 1\.1024\.12±1\.2024\.12\\pm 1\.20\-0\.18GPT\-OSS\-Code\-Reasoning\-20BGPT\-OSS\-20BCode reasoning18\.60±1\.8018\.60\\pm 1\.8032\.50±2\.6032\.50\\pm 2\.60\+13\.9028\.50±1\.3028\.50\\pm 1\.3028\.48±1\.4028\.48\\pm 1\.40\-0\.02Average change––\+13\.87––\-0\.073

### Appendix RComputational Cost of Memory Search and Inference

We further clarify the computational cost of the memory search procedure in DAPA\. First, memory search is a one\-time offline process, rather than an inference\-time operation\. In our experiments onLlama\-2\-7B, using50 harmful promptsfor delta debugging search, the full DAPA pipeline can be completed in48 minutes on a single A100 GPU, where delta debugging accounts for most of the runtime, while the model fusion step itself takes only about5 minutes; importantly, the entire procedure requires no additional training\. By contrast, standard alignment methods such as SFT, DPO, and RLHF incur substantially higher training costs, requiring approximately4\.67 hours, 9 hours, and 18 hours, respectively, on4×\\timesA100 GPUs\.

We also emphasize that DAPA introduces almost no additional inference\-time overhead\. After the offline search and fusion stages are completed, generation is performed using a single fused model, without activation editing, multi\-model decoding, or additional test\-time search\. Therefore, the inference cost of DAPA is essentially the same as naive single\-model decoding\.

### Appendix SModel Name and Corresponding Labels

We present the model names with their corresponding labels in[Tables˜9](https://arxiv.org/html/2406.01514#A11.T9),[3](https://arxiv.org/html/2406.01514#S4.T3)and[15](https://arxiv.org/html/2406.01514#A11.T15)\.

Table 29:Model Names and Corresponding LabelsLabelModel Full NameAmeta\-llama/Llama\-2\-7b\-hfBhfl/chinese\-alpaca\-2\-7bCmeta\-llama/Llama\-2\-13b\-hfDhfl/chinese\-alpaca\-2\-13bENousResearch/Redmond\-Puffin\-13BFgoogle/gemma\-2bGmlabonne/Gemmalpaca\-2BHgoogle/gemma\-7bICorticalStack/gemma\-7b\-ultrachat\-sftJmacadeliccc/gemma\-orchid\-7b\-dpoKmistralai/Mistral\-7B\-v0\.1Lteknium/OpenHermes\-2\-Mistral\-7BMcognitivecomputations/dolphin\-2\.2\.1\-mistral\-7bNHuggingFaceH4/zephyr\-7b\-alphaOcognitivecomputations/dolphin\-2\.6\-mistral\-7b\-dpoPabhishekchohan/mistral\-7B\-forest\-dpoQopenchat/openchat\_3\.5
### Appendix TDisclosure of LLM Usage

We employ GPT\-5 to refine the manuscript’s language for conciseness and precision\.