Localizing Safety Alignment: MLP Layers and Mid-Network Blocks Encode Refusal Behavior in Large Language Models

# Localizing Safety Alignment: MLP Layers and Mid-Network Blocks Encode Refusal Behavior in Large Language Models
Source: [https://arxiv.org/html/2608.11583](https://arxiv.org/html/2608.11583)
Mingyu ZongThanks:The authors thank collaborators Jwala Dhamala and Rahul Gupta from Amazon for helpful directions and inputs\.Affiliation:Department of Computer ScienceAffiliation:University of Southern CaliforniaAffiliation:Los Angeles, CA 90007Email:[mzong@usc\.edu](mailto:)Sampad MohantyAffiliation:Department of Computer ScienceAffiliation:University of Southern CaliforniaAffiliation:Los Angeles, CA 90007Email:[sbmohant@usc\.edu](mailto:)Bhaskar KrishnamachariAffiliation:Department of Electrical and Computer EngineeringAffiliation:Department of Computer ScienceAffiliation:University of Southern CaliforniaAffiliation:Los Angeles, CA 90007Email:[bkrishna@usc\.edu](mailto:)

###### Abstract

Safety alignment in large language models is often treated as a distributed property of the entire network, yet its practical brittleness suggests that refusal behavior may be concentrated in a smaller set of parameters\. This work addresses where safety\-aligned refusal is encoded by transplanting weights from aligned models into matched unaligned base models at multiple levels of granularity\. Using two open\-weight model pairs and four safety benchmarks, we conducted experiments to compare the effects of replacing attention weights, MLP weights, contiguous layer regions, and MLP blocks\. Across both model families, refusal transfer is dominated by MLP weights: replacing MLP parameters recovers substantially more malicious\-prompt refusal than replacing attention parameters, with gains of at least 2\.7 times more across benchmarks\. Within the MLP stack, refusal\-relevant parameters exhibit a consistent mid\-network concentration, as the block spanning layers 8–11 is selected first in all six greedy searches over model\-dataset pairs\. The results also show that the composition of safety\-relevant components is non\-additive: in five of six greedy trajectories, adding more aligned blocks can reduce refusal performance, and selective block subsets can outperform full MLP transplantation on malicious refusal, benign over\-refusal, or both\. Finally, greedy orders transferred to OR\-Bench vary with the source benchmark used to derive them, indicating a benchmark\-dependent precision\-coverage trade\-off\. These results suggest that safety alignment in current LLMs is both localized and interaction\-sensitive, offering insight into alignment brittleness and potential avenues for targeted safety interventions\.111Code and datasets are available at[https://github\.com/ANRGUSC/Localizing\-Safety\-Alignment](https://github.com/ANRGUSC/Localizing-Safety-Alignment)\.

## 1Introduction

Large language models \(LLMs\) are Transformer\-based neural language models trained on large\-scale text corpora to predict and generate natural language outputs\([23](https://arxiv.org/html/2608.11583#bib.bib26);[7](https://arxiv.org/html/2608.11583#bib.bib27)\)\. Scaling model size, dataset size, and computational resources has led to broad improvements in language modeling and downstream task performance, making LLMs increasingly capable general\-purpose agents\([14](https://arxiv.org/html/2608.11583#bib.bib28);[6](https://arxiv.org/html/2608.11583#bib.bib29)\)\. In practice, many deployed LLMs are first pretrained to acquire general linguistic and world knowledge, then adapted to downstream tasks so that their responses better match user intent, task requirements, and social norms\. This two\-stage development process makes safety alignment especially important: pre\-training provides broad capabilities, but whether and when a model refuses harmful requests is largely shaped during post\-training\.

Modern alignment pipelines, including supervised instruction tuning, reinforcement learning from human feedback \(RLHF\), and preference\-based optimization, have substantially improved safety of LLMs\([1](https://arxiv.org/html/2608.11583#bib.bib2);[17](https://arxiv.org/html/2608.11583#bib.bib1);[3](https://arxiv.org/html/2608.11583#bib.bib3);[20](https://arxiv.org/html/2608.11583#bib.bib4)\)\. However, aligned refusal behavior tends to be brittle in practice\. Prior work has shown that small downstream updates can weaken safeguards or unintentionally alter safety behavior\([19](https://arxiv.org/html/2608.11583#bib.bib5);[27](https://arxiv.org/html/2608.11583#bib.bib6)\)\. In addition, carefully designed attacks such as jailbreaks, adversarial suffixes, automated red\-teaming, and prompt transformations can bypass refusal mechanisms, suggesting that refusal is not a fixed property of model capability but a learned behavior sensitive to prompt format, model family, and internal representation\([24](https://arxiv.org/html/2608.11583#bib.bib31);[31](https://arxiv.org/html/2608.11583#bib.bib22);[18](https://arxiv.org/html/2608.11583#bib.bib25);[8](https://arxiv.org/html/2608.11583#bib.bib23);[28](https://arxiv.org/html/2608.11583#bib.bib24)\)\. At the same time, safety tuning can also produce the opposite failure mode, which is over\-refusal on harmless or legitimate prompts\([5](https://arxiv.org/html/2608.11583#bib.bib7);[21](https://arxiv.org/html/2608.11583#bib.bib13)\)\. Therefore, LLM safety must be evaluated along two axes: resistance to harmful instructions and avoidance of over\-refusal on harmless prompts\. A model that refuses too rarely remains vulnerable to misuse, while a model that refuses too broadly may become unhelpful\.

These observations motivate a more structured understanding of how refusal is implemented inside LLMs\. A growing body of mechanistic work demonstrates that safety\-relevant behaviors can often be detected or steered through internal activations\. Activation space studies have identified low\-dimensional directions associated with refusal behavior\([2](https://arxiv.org/html/2608.11583#bib.bib8);[26](https://arxiv.org/html/2608.11583#bib.bib9)\)\. Another line of work utilizes localized parameter or representation interventions to show that targeted manipulations can produce predictable behavioral changes\([25](https://arxiv.org/html/2608.11583#bib.bib10);[30](https://arxiv.org/html/2608.11583#bib.bib11);[16](https://arxiv.org/html/2608.11583#bib.bib12)\)\. These findings indicate that refusal is not uniformly distributed throughout the network, but they leave open a complementary weight\-space question: which parameters, and at which depths, are responsible for transferring safety refusal from an aligned model to its base counterpart?

We hypothesize that safety\-aligned refusal is localized: the parameters mediating transferable refusal behavior are concentrated in specific weight matrices and depth ranges rather than uniformly distributed across the network\. Such concentration could help explain the brittleness of aligned refusal under targeted parameter updates and could enable more selective interventions\. We test this hypothesis through selective weight transplantation\. Using two matched open\-weight model pairs: RealSafe\-R1\-7B/DeepSeek\-R1\-Distill\-Qwen\-7B and saferlhf\_ultra\_sft/Llama\-3\.1\-8B, we construct hybrid models by replacing subsets of parameters in the unaligned base model with the corresponding parameters from the aligned model\. Because each pair shares the same architecture, behavioral differences can be attributed directly to the transplanted weights rather than to model design\. By varying the granularity of transplantation, we answer these finer localization questions: Is transferable refusal behavior mediated more strongly by attention weights or MLP weights? Is it concentrated in particular depth ranges?

Our experiments provide consistent evidence across both model families that transferable refusal behavior is mediated more strongly by MLP than attention weights\. Within the MLP pathway, a mid\-depth block spanning layers 8–11 is selected first in all six greedy searches over model\-dataset pairs, indicating a consistent concentration of refusal\-relevant parameters at this depth\. At the same time, aligned blocks do not compose additively: in most settings, adding additional aligned blocks does not guarantee increased refusal, and selective MLP\-block subsets can outperform full MLP transplantation\. We further validate these patterns using larger filtered subsets\. To sum up, this paper suggests that safety\-aligned refusal behavior in the model families studied is largely mediated by MLP weights and is particularly concentrated in mid\-network layers, based on empirical evidence\.

The rest of the paper is organized as follows: the next section introduces related work on safety alignment\.[Section3](https://arxiv.org/html/2608.11583#S3)describes the matched model pairs, hybrid models’ construction procedures, evaluation benchmarks, and greedy block selection method used to localize safety\-relevant parameters, followed by the results section which presents the main findings\.[Section5](https://arxiv.org/html/2608.11583#S5)discusses the implications of these findings for alignment brittleness, safety precision, and benchmark choice\. Finally, we conclude with limitations of this study and directions for future work\.

## 2Related Work

Safety alignment in large language models is commonly achieved through supervised instruction tuning, reinforcement learning from human feedback, constitutional AI, or preference\-based optimization methods\. Foundational work has established instruction tuning and RLHF as practical mechanisms for training models to follow user instructions while incorporating human preferences over helpfulness and harmlessness\([17](https://arxiv.org/html/2608.11583#bib.bib1);[3](https://arxiv.org/html/2608.11583#bib.bib3)\)\. Constitutional AI further reduces reliance on direct human labels by using rule\-based self\-critique and AI feedback to improve harmlessness\([4](https://arxiv.org/html/2608.11583#bib.bib30)\)\. More recent preference\-optimization methods such as Direct Preference Optimization \(DPO\) simplify the RLHF pipeline by optimizing directly on preference pairs without explicitly training a separate reward model\([20](https://arxiv.org/html/2608.11583#bib.bib4)\)\. Building on this paradigm, safety\-focused instruction\-tuning methods directly optimize refusal behavior on harmful requests in open\-weight models\([1](https://arxiv.org/html/2608.11583#bib.bib2)\)\. These approaches demonstrate that refusal behavior can be reliably induced through post\-training, but they also imply that refusal is learned and may therefore be sensitive to later updates or changes in prompting\.

Despite the effectiveness of post\-training alignment, evaluations on aligned LLMs show that learned refusal is not fully robust\. Downstream adaptation affects safety, showing that fine\-tuning or other targeted updates can substantially weaken aligned safeguards\([19](https://arxiv.org/html/2608.11583#bib.bib5);[27](https://arxiv.org/html/2608.11583#bib.bib6)\)\. Automated red\-teaming methods can instruct language models themselves to discover prompts that elicit undesirable outputs\([18](https://arxiv.org/html/2608.11583#bib.bib25)\)\. Other attacks search for adversarial suffixes or transform harmful instructions into alternative formats, including mathematical or encoded prompts, in order to bypass safety filters\([31](https://arxiv.org/html/2608.11583#bib.bib22);[8](https://arxiv.org/html/2608.11583#bib.bib23);[28](https://arxiv.org/html/2608.11583#bib.bib24)\)\. These attacks highlight that many refusal mechanisms are sensitive to surface form and distribution shift\. On the other hand, excessive refusal is also problematic: models may reject benign prompts that contain sensitive terms or resemble unsafe requests, reducing their practical usefulness\([5](https://arxiv.org/html/2608.11583#bib.bib7);[21](https://arxiv.org/html/2608.11583#bib.bib13);[9](https://arxiv.org/html/2608.11583#bib.bib14)\)\.

In order to understand and address the brittleness of safety alignment, recent work studies how safety\-related behavior is organized within the internal structure of language models\.[2](https://arxiv.org/html/2608.11583#bib.bib8)claim that, for each model they study, a single extracted direction strongly mediates refusal\. They also demonstrate that ablating that direction suppresses refusal, while adding it to the model induces over\-refusal on benign inputs\.[26](https://arxiv.org/html/2608.11583#bib.bib9)argue that refusal does not reside in a single direction, but is organized in multi\-dimensional polyhedral cones that contain multiple refusal\-mediating directions\. Despite their disagreement about geometry, both papers suggest that refusal is not uniformly encoded across the network, but depends on structured and manipulable internal features\. Furthermore, prior work also proves that targeted interventions can predictably alter a model’s safety behavior\. Using layer\-wise pruning,[30](https://arxiv.org/html/2608.11583#bib.bib11)identify “safety layers” inside a model\. The authors then propose Layer\-specific Editing \(LED\) to restrict updates on those layers so that downstream toxic layers decode toward safe refusals instead of harmful continuations\.[16](https://arxiv.org/html/2608.11583#bib.bib12)propose Conditional Activation Steering \(CAST\), which applies a refusal steering vector when a prompt’s hidden states match a learned condition vector, thus enabling selective refusal rules such as refusing only particular categories of content while preserving normal responses for others\. This result further supports the view that refusal can be controlled through compact internal representations\. At the parameter level,[25](https://arxiv.org/html/2608.11583#bib.bib10)study safety brittleness through pruning and low\-rank modifications\. Critical neurons and ranks are distinguished by disentangling safety\-related components\. Surprisingly, the safety\-critical regions are sparse, taking up around 3% of parameters and 2\.5% of ranks\. It is also reported that safety and utility appear more differentiated in MLP layers than in attention layers, which is in line with the findings from this work\.

Beyond internal evaluation, a large number of benchmarks has been developed to measure both resistance to harmful requests and failure on benign ones\. AdvBench tests whether models comply with harmful instructions under adversarial prompting\([31](https://arxiv.org/html/2608.11583#bib.bib22)\), while TwinPrompt and SGXSTest extend this style of evaluation with paired settings designed to probe safety behavior more systematically\([15](https://arxiv.org/html/2608.11583#bib.bib20);[11](https://arxiv.org/html/2608.11583#bib.bib21)\)\. Complementary benchmarks emphasize the importance of false positives\. XSTest evaluates exaggerated safety behavior on safe prompts that resemble unsafe ones\([21](https://arxiv.org/html/2608.11583#bib.bib13)\)\. OR\-Bench evaluates whether aligned models over\-refuse harmless prompts\([9](https://arxiv.org/html/2608.11583#bib.bib14)\)\.

Existing work has primarily characterized safety at two levels: as an internal phenomenon studied through activation\-level analysis or targeted editing, and as an external behavioral property measured by benchmarks\. Less attention has been drawn to a direct weight\-space localization question: in a safety\-aligned model, which parameters actually carry the transferable refusal behavior relative to its unaligned counterpart? This paper addresses this question by comparing selective transplants of attention weights, MLP weights, contiguous layer ranges, and MLP blocks across matched base and aligned model pairs\. The experiments are set up to connect safety evaluation with mechanistic localization, focusing specifically on where aligned refusal behavior resides in the network\.

## 3Experimental Setup

Table 1:Overview of models and benchmarks\.### 3\.1Model Pairs

We analyze two pairs of open\-weight models that share the same architecture within each pair but differ in whether they were safety\-aligned \(Table[1](https://arxiv.org/html/2608.11583#S3.T1)\)\. The first pair consists of RealSafe\-R1\-7B and DeepSeek\-R1\-Distill\-Qwen\-7B, both are 28\-layer Qwen2\-based models\([22](https://arxiv.org/html/2608.11583#bib.bib15);[29](https://arxiv.org/html/2608.11583#bib.bib16)\)\. RealSafe\-R1\-7B underwent safety training on approximately 15K examples, including roughly 10K direct harmful queries and 5K jailbreak prompts\. The second pair consists of saferlhf\_ultra\_sft and Llama\-3\.1\-8B\([12](https://arxiv.org/html/2608.11583#bib.bib18);[10](https://arxiv.org/html/2608.11583#bib.bib17)\)\. Both models are 32\-layer Llama models, with saferlhf\_ultra\_sft being the fine\-tuned version on the instruct variant of Llama\-3\.1\-8B on the saferlhf\_ultra dataset\([13](https://arxiv.org/html/2608.11583#bib.bib19)\)\.

### 3\.2Weight Transplantation Procedure

All hybrid models are constructed by replacing selected weight matrices in the base model with the corresponding matrices from the aligned model\. Because of the shared architecture, behavioral differences in the resulting hybrids can be attributed directly to the transplanted weights\.

### 3\.3Granularity of Interventions

This study evaluates several levels of transplantation granularity\.

First, we study component\-level hybrids\. Theattnconfiguration replaces all attention projections \(q\_proj,k\_proj,v\_proj, ando\_proj\) across all layers, while keeping the base model’s MLP weights unchanged\. Themlpconfiguration performs the complementary intervention, replacing only the feed\-forward projections \(gate\_proj,up\_proj, anddown\_proj\)\.

We also evaluate contiguous layer groups\. Thefirst5,mid5, andlast5configurations transplant both attention and MLP weights for five consecutive layers, whilefirst\_halfandsecond\_halftransplant the corresponding half of the network weights\.

Motivated by the strong performance ofmlphybrids, we then localize safety within the MLP stack\. We partition the model’s MLP weights into contiguous blocks, each containing 4 layers\. For RealSafe\-R1\-7B with 28 layers, this yields seven blocks: B1 = layers 0\-3 through B7 = layers 24\-27\. For saferlhf\_ultra\_sft with 32 layers, this yields eight blocks: B1 = layers 0\-3 through B8 = layers 28\-31\. We first perform transplantation for each individual block and then analyze combinations of blocks\.

### 3\.4Evaluation Benchmarks and Subset Construction

We conduct model evaluations on four benchmarks that probe complementary aspects of safety behavior: TwinPrompt\([15](https://arxiv.org/html/2608.11583#bib.bib20)\), SGXSTest\([11](https://arxiv.org/html/2608.11583#bib.bib21)\), AdvBench\([31](https://arxiv.org/html/2608.11583#bib.bib22)\), and OR\-Bench\([9](https://arxiv.org/html/2608.11583#bib.bib14)\)\. TwinPrompt and SGXSTest provide paired harmful and benign prompts, allowing us to evaluate both refusal of unsafe requests and over\-refusal of safe requests\. AdvBench contains harmful instructions only and is used to measure malicious\-prompt\-refusal\. OR\-Bench contains benign prompts designed specifically to stress\-test over\-refusal \(Table[1](https://arxiv.org/html/2608.11583#S3.T1)\)\.

For each malicious benchmark, we construct evaluation subsets by retaining prompts on which the safety\-aligned model refuses while the corresponding base model complies\. We keep up to 30 prompts per condition\. For the benign subsets used to measure over\-refusal, we apply the analogous filtering procedure to the paired benchmarks\. Because this filtering is more restrictive for some settings, certain benign subsets contain fewer than 30 examples; we clarify the total number of prompts when reporting results\.

In addition, we construct a separate paired prompt pool and apply the same aligned–base disagreement filtering criterion to obtain larger validation subsets that contain 100 malicious and 100 benign prompts after filtering\. So they preserve the behavioral contrast used in the primary experiments while testing whether the observed localization patterns persist at a larger sample size\.

### 3\.5Metrics

We define a*refusal behavior*to be an explicit rejection induced by recognized harmful or unethical implications of the request\. Refusal on malicious datasets \(MR\) is higher if a model incorporates better defense mechanisms against malicious use\. For benign prompts, we report benign over\-refusal \(BOR\) results, which are the numbers of benign prompts incorrectly turned down; lower BOR is preferred for all models\. All model answers are manually inspected\. We include generation configurations in section[9](https://arxiv.org/html/2608.11583#S9)\.

### 3\.6Greedy Block Selection

To study how refusal\-relevant blocks interact, we perform greedy forward selection separately for each model\-dataset pair on the malicious benchmarks\. Starting from the base model, we iteratively add the MLP block from the remaining set that yields the largest gain in refusal until all blocks are transplanted\. Besides refusals, a model can output other acceptable and safe answers, which provide no useful harmful information, even if it is not phrased as an explicit refusal\. When multiple candidates tie on refusals, we select the one with more acceptable safe answers\. This procedure produces both \(i\) a dataset\-specific importance ordering over blocks and \(ii\) the best\-performing block subset at each budgetkk\. The orders collected from malicious benchmarks are transferred to OR\-Bench for generalization testing\. In addition, with Block 3 constantly being selected first, we further examine the importance of individual Block 3 layers and their combinations\.

## 4Results

We find three consistent patterns across both model families\. First, MLP weights play a more essential role than attention weights in recovering refusal behavior\. In addition, refusal\-relevant parameters are not uniformly distributed across the MLP stack: Block 3 \(layers 8–11\) is the most consistently effective individual block for transferring refusal behavior, within which layer 8 stands out\. Moreover, greedy block selection reveals a six\-block substitution that often outperforms full MLP transplantation on malicious refusal, benign over\-refusal, or both\.

### 4\.1Component\-Level Hybrids

Table 2:Component\-level hybrids’ performance, including MR on three malicious subsets and BOR on benign TwinPrompt prompts\.Our first set of analyses compares hybrid models that transplant broad classes of parameters across three malicious\-prompt benchmarks to lay the groundwork for finer localizations\. We summarize MR and BOR from multiple benchmarks in Table[2](https://arxiv.org/html/2608.11583#S4.T2)\. The pattern is consistent across both model pairs: replacing MLP weights recovers substantially more safety behavior than replacing attention weights\.

Under MLP weight transplantation, hybrid models from the RealSafe\-R1\-7B family achieve 19, 17, and 27 refusals on 30\-malicious\-prompt subsets from TwinPrompt, SGXSTest, and AdvBench, respectively, which are the highest MR values among all component\-level hybrids\. Models from the saferlhf\_ultra\_sft family also yield best performance with this configuration\. When compared with MR results generated byattnhybrids, MLP\-only transplantation exceeds by at least 2\.7 times across the six comparisons\.

On the other hand, the same intervention that restores refusal behavior can also import over\-refusal\. For RealSafe\-R1\-7B hybrids, full MLP substitution produces 11 benign refusals on TwinPrompt, whereas attention\-only substitution produces none\. Proper refusal and false\-positive refusal are therefore not cleanly separable at the component level in this model family: both are concentrated primarily in the MLP pathway\. By contrast, saferlhf\_ultra\_sft exhibits no TwinPrompt over\-refusal behavior under full MLP transplantation, suggesting that the aligned signal transferred from saferlhf\_ultra\_sft may be more selective\.

Taken together, these results identify MLP weights as the dominant parameter pathway for transferring aligned refusal behavior in both model families\. Since MLP substitution clearly dominates, we then ask whether safety is distributed uniformly across the MLP stack or concentrated in particular blocks\.

### 4\.2Individual MLP Blocks

Table 3:Individual MLP\-block substitution results for the RealSafe\-R1\-7B family\. Values represent refusal counts on malicious subsets \(30 prompts per subset\)\.Table 4:Individual MLP\-block substitution results for the saferlhf\_ultra\_sft family\. Values represent refusal counts on malicious subsets \(30 prompts per subset\)\.In this follow\-up experiment, we localize the safety signals within the MLP stack by transplanting one block at a time\. Tables[3](https://arxiv.org/html/2608.11583#S4.T3)and[4](https://arxiv.org/html/2608.11583#S4.T4)report MR results for each single\-block substitution\. Refusal\-relevant effects are distributed across multiple MLP blocks, but Block 3 is the most consistently important block across benchmarks\. On TwinPrompt and AdvBench, Block 3 achieves the highest MR rates for both model families, and on SGXSTest it appears among the top candidates\. The recurrence of the same absolute depth range across the Qwen2\-7B and Llama\-3\.1\-8B architectures suggests that, for models of this scale, the parameters mediating transferable refusal behavior appear disproportionately concentrated around layers 8–11\.

The results also reveal that individual MLP blocks in the saferlhf\_ultra\_sft family are generally less effective in rejecting unsafe instructions\. A potential explanation is that both extent and variety of safety training affect the robustness of knowledge encoding, but verification is left for future research\. Tables[3](https://arxiv.org/html/2608.11583#S4.T3)and[4](https://arxiv.org/html/2608.11583#S4.T4)measure the effect of each block in isolation;[Section4\.3](https://arxiv.org/html/2608.11583#S4.SS3)instead asks which blocks are most useful when added cumulatively\.

Table 5:Greedy block orders on three malicious subsets\. Block 3 is selected first in all six settings\.![Refer to caption](https://arxiv.org/html/2608.11583v1/figs/paired_results.png)
![Refer to caption](https://arxiv.org/html/2608.11583v1/figs/advbench_results.png)

Figure 1:Hybrid model performance on benchmark subsets\.
### 4\.3Non\-monotonic Block Composition

Table 6:Greedy trajectories for RealSafe\-R1\-7B family\.kkis the number of MLP blocks transplanted\. Greedy orders: TwinPrompt 3\-5\-2\-6\-4\-7\-1; SGXSTest 3\-6\-7\-4\-1\-5\-2; AdvBench 3\-6\-5\-1\-2\-7\-4\.Table 7:Greedy trajectories for saferlhf\_ultra\_sft family\.kkis the number of MLP blocks transplanted\. Greedy orders: TwinPrompt 3\-4\-5\-2\-1\-8\-7\-6; SGXSTest 3\-7\-2\-4\-5\-1\-6\-8; AdvBench 3\-2\-4\-5\-7\-1\-8\-6\.We run greedy forward selection to determine how safety\-aligned blocks interact when added cumulatively\. Block 3 is selected first in all six model\-dataset pairs \(Table[5](https://arxiv.org/html/2608.11583#S4.T5)\), consistent with its strong individual performance from previous experiment\. Tables[6](https://arxiv.org/html/2608.11583#S4.T6)and[7](https://arxiv.org/html/2608.11583#S4.T7)report MR and BOR results at each block budgetkk\. The main finding is that on two of three benchmarks, a six\-block transplantation can outperform the full MLP transplantation on rejecting malicious requests\.

The greedy trajectories show how this pattern develops across block budgets \(Figure[1](https://arxiv.org/html/2608.11583#S4.F1)\)\. For RealSafe\-R1\-7B on TwinPrompt, MR rises from77atk=1k=1to2424atk=6k=6, then falls to1919atk=7k=7\. The trajectory of saferlhf\_ultra\_sft reaches1212MR counts atk=4k=4, drops down to1010atk=6k=6, and finally recovers when all eight blocks are present\. On SGXSTest, RealSafe\-R1\-7B peaks at MR=18=18atk=6k=6and declines to1717atk=7k=7\. The drop observed from the saferlhf\_ultra\_sft family is more pronounced: performance peaks at88atk=6k=6and falls to44after the seventh block is added\. AdvBench is the most monotonic setting\. For RealSafe\-R1\-7B, MR increases throughk=5k=5, dips atk=6k=6, and recovers atk=7k=7\. The saferlhf\_ultra\_sft\-AdvBench trajectory is the only fully monotonic one among the six, it starts with 1 refusal atk=1k=1and gradually grows to 17 rejections\.

The BOR results further show benefits of using only six blocks of MLP weights\. In the RealSafe\-R1\-7B\-TwinPrompt setting, the six\-block hybrid achieves MR=24=24with BOR=9=9, whereas the full seven\-block transplant yields MR=19=19with BOR=11=11, which is worse on both metrics\. For RealSafe\-R1\-7B\-SGXSTest, full transplantation loses one refusal without reducing over\-refusal\. For saferlhf\_ultra\_sft\-SGXSTest, the contrast is sharper: the six\-block model achieves MR=8=8with BOR=1=1, compared with MR=4=4and BOR=3=3for the full eight\-block model\.

We refer to the pattern where adding an aligned block reduces MR as the negative marginal effect\. Five of the six trajectories exhibit such effect, but the disruptive block varies across settings: the first decline on both TwinPrompt trajectories coincides with the introduction of Block 1, whereas the sharp SGXSTest drop for saferlhf\_ultra\_sft is associated with Block 6\. This dataset dependence stands against a single universally harmful block and instead supports that block utility is contingent on the surrounding set of already\-transplanted blocks\.

Table 8:OR\-Bench over\-refusal \(out of 30 benign prompts\) under different greedy block orders\. Lower is better\. Bold indicates the best value at each budget within a model family\.
### 4\.4Transfer to OR\-Bench is Benchmark\-dependent

The greedy selection approach offers insights into MR\-BOR trade\-off with respect to model\-dataset pairs\. However, the BOR results are collected from datasets with paired prompts that minimize linguistic difference\. We further evaluate the models and orders on a separate benign\-prompt only benchmark, namely OR\-Bench, to determine transferability and whether a specific order outperforms the others\. As a result, the AdvBench\-derived order is generally more conservative and often incurs higher OR\-Bench over\-refusal rates\. By contrast, the order derived from SGXSTest tends to yield lower OR\-Bench over\-refusal than the order derived from AdvBench\. For RealSafe\-R1\-7B, the SGXSTest order attains the lowest or tied\-lowest BOR at five of the seven block budgets\. For saferlhf\_ultra\_sft, the SGXSTest order maintains zero over\-refusals through mostkkvalues\. Nonetheless, this comparison should be interpreted cautiously\. Regardless of the lower over\-refusal values, SGXSTest\-derived orders also tend to achieve lower malicious\-prompt rejection than orders derived from TwinPrompt or AdvBench\. We therefore interpret these results as evidence of a trade\-off rather than a strict improvement\. The source benchmark influences both how aggressively safety transfers and how much that transfer over\-generalizes to benign prompts\.

A second pattern we discover is that saferlhf\_ultra\_sft hybrids remain more precise than RealSafe\-R1\-7B hybrids under transfer\. At high budgets, RealSafe\-R1\-7B incurs 3\-6 BOR atk=7k=7, depending on the order, whereas saferlhf\_ultra\_sft incurs 0\-1 refusals atk=7k=7and reaches only 3 when all MLP blocks are transplanted\. This suggests that the aligned signal inherited from saferlhf\_ultra\_sft is less prone to over\-generalization under block composition\.

Overall, the transfer results demonstrate that greedy block orders learned from different malicious benchmarks behave differently on unseen benign prompts, thus, benchmark choice matters for downstream precision\.

### 4\.5A Peek into Block 3

Table 9:Refusal behaviors of RealSafe hybrid models constructed using Block 3 layers\. All subsets contain 30 prompts\.Given the consistently strong performance of Block 3 weights, we build several hybrid models using only Block 3 layers to test if the non\-monotonic pattern and marginal effect exist at a finer\-grained level\. As shown in Table[9](https://arxiv.org/html/2608.11583#S4.T9), exploratory analysis further points to layer 8 for the RealSafe\-R1\-7B family\. Layer 8 produces more malicious\-prompt refusals than the other individual layers\. Transplanting only this aligned layer of weights to the base DeepSeek\-R1\-Distill\-Qwen\-7B model gives near or above half of the MR that the full block achieves\. The exploratory evidence indicates that transferable refusal behavior is concentrated in this layer within Block 3\. Across all three benchmarks, we observe the non\-additivity from the combination of layers 8 and 9, along with a few others that are dataset\-specific\. Another consistency is that the full\-block performance can be achieved or even surpassed by a subset of layers\. We conduct the same experiments on the saferlhf\_ultra\_sft pair\. Despite Block 3 being the most important MLP block, it induces at most 2 rejections on the subsets, so the results fail to reveal valuable information and are excluded from display\.

### 4\.6Validation on a Larger Paired Subsets

Table 10:Component\-level hybrids’ performance on validation subsets\.Table 11:Individual MLP\-block substitution results for the RealSafe\-R1\-7B family\.Table 12:Greedy trajectories for RealSafe\-R1\-7B family\.kkis the number of MLP blocks transplanted\. Greedy orders: Malicious Subset 3\-5\-6\-7\-2\-4\-1; Benign Subset 3\-4\-7\-6\-1\-2\-5\.The preceding experiments rely on relatively small filtered subsets, with at most 30 prompts per condition\. To test whether the observed localization patterns are robust to a larger sample of aligned–base disagreement cases, we conduct an additional validation experiment using a newly constructed paired prompt set\. We created a list of paired actions, and randomly draw prefixes from another set to form complete instructions\. We then followed the same subset construction criterion\. For malicious prompts, we retained examples for which the safety\-aligned RealSafe\-R1\-7B model refuses while the corresponding DeepSeek\-R1\-Distill\-Qwen\-7B base model complies\. For benign prompts, we applied the analogous procedure to identify cases relevant to over\-refusal\. We constructed a substantially larger pool of candidate prompts, allowing both the malicious and benign subsets to contain 100 prompts after filtering\. Therefore the experiment preserves the behavioral contrast central to our transplantation analysis while reducing the sensitivity of the reported results to the small sample sizes\.

We repeat the transplantation experiments on the RealSafe\-R1\-7B model family using the same transplantation procedures as in the main experiments\. In particular, we examine whether the three principal observations obtained from the original subsets persist at the larger sample size: the dominance of MLP over attention weights in transferring refusal behavior, the concentration of safety\-relevant weights in the mid\-network MLP blocks, and the non\-additive behavior of MLP\-block composition\.

The results in Table[10](https://arxiv.org/html/2608.11583#S4.T10)strongly reproduce the dominance of the MLP pathway\. On the 100 malicious prompts, themlphybrid refuses 61 prompts, compared with only 8 for theattnhybrid\. The remaining coarse\-grained transplantations are substantially weaker:first5,mid5, andlast5each produce 8 refusals, whilefirst\_halfandsecond\_halftransplantation produce 24 and 44 refusals, respectively\. These results are consistent with the original experiments and provide additional evidence that transferable refusal behavior in the RealSafe\-R1\-7B family is concentrated primarily in MLP rather than attention weights\. The larger benign subset also reproduces the precision trade\-off associated with MLP transplantation\. Themlphybrid over\-refuses 33 of 100 benign prompts, compared with at most 7 over\-refusals from other settings\. Thus, the same parameter pathway that transfers the greatest amount of malicious\-prompt refusal also transfers substantially more over\-refusal, strengthening the evidence that desirable refusal and false\-positive refusal are not cleanly separable at the component level in this model family\.

We further repeat the individual MLP\-block experiment on the larger subsets\. Block 3 \(layers 8–11\) still produces the highest malicious\-prompt refusal rates among all seven individual blocks, rejecting 13 of 100 malicious prompts, followed by Block 6 with 11 rejections and Block 2 with 8 rejections\. The remaining blocks produce between 4 and 7 rejections\. On the other hand, Block 3 induces only 1 refusal among the 100 benign prompts \(Table[11](https://arxiv.org/html/2608.11583#S4.T11)\)\. The recurrence of Block 3 as the strongest individual MLP block on a substantially larger prompt set is consistent with the mid\-network localization observed in the original experiments\.

Following greedy forward selection, malicious\-prompt refusals increase from 13 with one block to 29, 42, 52, 54, and a peak of 64 as the block budget increases fromk=1k=1tok=6k=6\. However, transplanting the final remaining MLP block reduces performance from 64 to 61 refusals atk=7k=7\(Table[12](https://arxiv.org/html/2608.11583#S4.T12)\)\. Because the seven\-block configuration is equivalent to full MLP transplantation, this result once more demonstrates that transplanting all aligned MLP weights need not maximize refusal behavior\. The persistence of a negative marginal effect on a substantially larger prompt set supports the conclusion that refusal\-relevant MLP blocks do not compose strictly additively\.

Overall, the larger\-subset validation reproduces all three principal localization patterns for the RealSafe\-R1\-7B family: MLP weights dominate attention weights in transferring refusal behavior, Block 3 remains the strongest individual MLP block, and selective block composition can outperform full MLP transplantation\. These results reduce the likelihood that the original observations are artifacts of the small evaluation subsets\. However, because the validation examples are constructed using the same aligned–base disagreement criterion, this experiment establishes robustness to increased sample size within the filtered evaluation distribution rather than generalization to an unfiltered prompt distribution\.

## 5Discussion

### 5\.1Weight\-Space Localization of Refusal Behavior

Our transplantation experiments indicate that the parameters mediating safety\-aligned refusal are not distributed uniformly across the network\. Across both model families, transferable refusal behavior is mediated predominantly by the MLP pathway, with a mid\-depth region spanning layers 8–11 repeatedly prioritized\. Even within the mid\-depth region, individual layers bear different levels of refusal\-related behavior\. This concentration provides a plausible explanation for the brittleness of alignment reported in prior work\([19](https://arxiv.org/html/2608.11583#bib.bib5);[27](https://arxiv.org/html/2608.11583#bib.bib6)\)\. Since a relatively small subset of parameters carries a disproportionate share of the refusal signal, targeted updates to that subset can substantially alter safety behavior\.

Moreover, our transplantation experiments localize*where*safety\-relevant information is stored, rather than*how*it is computed\. In particular, the critical role of MLP layers and the highlighted importance of Block 3 point to a narrow part of the network that merits deeper mechanistic analysis\. We plan to combine the present weight\-level interventions with activation\-level methods, such as activation patching, causal tracing, or neuron\-level probing, to characterize the representations and transformations within these blocks that support refusal behavior as a follow\-up study\.

### 5\.2Interactions and Precision Trade\-offs

The greedy trajectories show that aligned blocks do not contribute independently\. Adding an additional block can improve refusals in one context while degrading them in another, and the full MLP transplant is often not the best operating point\. Thus, MLP\-block interactions appear to be non\-additive\.

Another observation is that the same concentration that enables safety transfer can also introduce over\-refusal\. With respect to RealSafe\-R1\-7B hybrids, for example, the MLP\-only transplant simultaneously increases malicious prompt refusal and benign over\-refusal\. This co\-localization means that improving safety precision is not only a matter of transplanting more aligned parameters; it requires identifying and choosing subsets whose interactions preserve the distinction between harmful and benign inputs\.

More broadly, the findings suggest two concrete directions for alignment research\. The first is*precision\-oriented alignment*: rather than optimizing refusal strength alone, future methods should explicitly target the precision\-coverage trade\-off that becomes visible in our transplantation experiments\. The second is*distributed robustness*: if current safety behavior depends heavily on a small number of MLP blocks, then a promising goal is to develop alignment procedures that encode safety more redundantly across the network, making it less brittle to fine\-tuning, editing, or weight replacement\.

### 5\.3Benchmark Choice and Deployment Implications

Our OR\-Bench transfer results indicate that the benchmark used to derive the greedy order materially affects downstream behavior on unseen benign prompts\. Some source benchmarks induce more aggressive safety transfer, which can improve malicious\-prompt rejection but also increase over\-refusal on benign prompts\. Others induce a milder transfer that preserves benign compliance more effectively, although at the cost of lower malicious refusal\. In this sense, the observed differences are better understood as a precision\-coverage trade\-off than as evidence that one benchmark is categorically superior\.

This trade\-off is especially important for deployment\. If the goal is to maximize refusal on harmful prompts, a more aggressive order may be preferable even if it incurs higher BOR\. On the other hand, if the goal is to preserve usability on benign inputs, a milder order may be preferable, even if it leaves more harmful prompts answered unsafely\. In practice, block selection should not be evaluated on malicious\-prompt rejection alone: the choice of source benchmark can materially affect both safety strength and over\-generalization\.

### 5\.4Limitations and Future Directions

This study provides evidence that safety alignment is localized and interaction\-sensitive, but several limitations remain\. First, our analysis covers only two matched model pairs in the 7B\-8B LLM regime\. Although the recurrence of the same mid\-network region across both pairs is striking, it is not yet clear whether the same pattern holds at other model scales, in denser or sparser architectures, or under different alignment procedures\. Extending the transplantation analysis to various model sizes and families would help determine whether the observed localization around layers 8\-11 reflects a general property of safety alignment or a scale\-specific feature of the models studied in this work\.

Moreover, we intentionally constructed filtered subsets to sharpen the contrast between safety\-aligned models and base models, but this also limits their sizes\. Future work should validate the same localization patterns on larger and less filtered evaluation sets, ideally including benchmarks that jointly measure harmful\-prompt refusal and benign\-prompt compliance within the same test distribution\.

Our greedy forward selection only approximates the space of useful block combinations\. The non\-monotonic trajectories already show that block effects are not independent, so more exhaustive search strategies may reveal better\-performing subsets or more structured interactions\. This motivates future studies on combinatorial search, sparse optimization, or learned selection policies that explicitly optimize both malicious refusal and benign non\-refusal\.

Overall, these limitations and opportunities point to a larger research agenda: using localization not only to diagnose where alignment resides, but also to design safety methods that are more interpretable, more precise, and more robust\.

## 6Conclusion

In this study, we investigate where refusal behavior is encoded in safety\-aligned language models by utilizing small aligned–base disagreement prompt sets and transplanting weight matrices into their unaligned counterparts\. Our experiments across two model pairs and four benchmarks reveal three consistent empirical patterns that characterize the weight\-space localization of safety\-aligned refusal behavior:

- •Pathway Concentration:Transferable refusal behavior is mediated predominantly by the MLP pathway rather than the attention pathway\. Replacing MLP weights recovers substantially more refusal behavior, outperforming attention\-only hybrids by at least 2\.7 times in every malicious benchmark setting we studied\.
- •Depth Localization:Within the MLP stack, a specific mid\-depth block \(Block 3, layers 8–11\) consistently emerges as the strongest localized intervention\. It is selected first in all six greedy searches, indicating a disproportionate concentration of refusal\-mediating parameters at this depth\.
- •Non\-Additive Interactions:Refusal\-relevant MLP blocks interact non\-monotonically\. Adding aligned blocks can reduce malicious\-prompt refusal, showing that their behavioral effects are not strictly additive\.

The findings from MLP\-block greedy selection suggest that a selective subset of weights can outperform a full\-MLP transplant, achieving a more favorable operating point for both safety and model utility\. We also observe a similar pattern from performance of Block 3 layers’ transplantation\. We emphasize that safety alignment in current LLMs is both*localized*and*interaction\-sensitive*\. Localization helps explain why aligned behavior can be brittle under subsequent fine\-tuning or model modification, while the observed interactions show that effective alignment is not simply a matter of transplanting more aligned parameters\. In short, our localization analyses help move from broad behavioral observations toward more targeted safety interventions, directing attention to the components of the network most responsible for refusal behavior\.

## 7AI Use Acknowledgment

We have made use of AI tools \(Claude Code, ChatGPT, Google Gemini\) to assist with coding and proofreading, editing and formatting during the writing\. The human authors take full responsibility for the contents of this paper\.

## 8Funding Acknowledgment

This work has been supported in part through a grant for the project “Towards Resilient LLM Alignment: Investigating Brittleness & Improving Model Safety by Rank Enhancement” funded by the USC Amazon Center on Secure and Trusted Machine Learning

## 9Appendix

Model Generation Configuration for both model families:

- •Temperature: 0\.6
- •max\_new\_tokens: 1200 \(for subset filtering\), 2000 \(for response generation from hybrid models\)

## References

- A\. Amballa, D\. S\. Saluru, G\. Akkinapalli, A\. Sureddy, and A\. K\. SureddySafe to serve: aligning instruction\-tuned models for safety and helpfulness\.External Links:2412\.00074,[Link](https://arxiv.org/abs/2412.00074)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p1.1)\.
- Arditiet al\.\(2024\)A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Rimsky, W\. Gurnee, and N\. NandaRefusal in language models is mediated by a single direction\.ArXivabs/2406\.11717\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270560489)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p3.1),[§2](https://arxiv.org/html/2608.11583#S2.p3.1)\.
- Baiet al\.\(2022a\)Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan, N\. Joseph, S\. Kadavath, J\. Kernion, T\. Conerly, S\. El\-Showk, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, T\. Hume, S\. Johnston, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, D\. Amodei, T\. Brown, J\. Clark, S\. McCandlish, C\. Olah, B\. Mann, and J\. KaplanTraining a helpful and harmless assistant with reinforcement learning from human feedback\.External Links:2204\.05862,[Link](https://arxiv.org/abs/2204.05862)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p1.1)\.
- Baiet al\.\(2022b\)Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon, C\. Chen, C\. Olsson, C\. Olah, D\. Hernandez, D\. Drain, D\. Ganguli, D\. Li, E\. Tran\-Johnson, E\. Perez, J\. Kerr, J\. Mueller, J\. Ladish, J\. Landau, K\. Ndousse, K\. Lukošiūtė, L\. Lovitt, M\. Sellitto, N\. Elhage, N\. Schiefer, N\. Mercado, N\. Dassarma, R\. Lasenby, R\. Larson, S\. Ringer, S\. Johnston, S\. Kravec, S\. E\. Showk, S\. Fort, T\. Lanham, T\. Telleen\-Lawton, T\. Conerly, T\. Henighan, T\. Hume, S\. Bowman, Z\. Hatfield\-Dodds, B\. Mann, D\. Amodei, N\. Joseph, S\. McCandlish, T\. B\. Brown, and J\. KaplanConstitutional ai: harmlessness from ai feedback\.ArXivabs/2212\.08073\.External Links:[Link](https://api.semanticscholar.org/CorpusID:254823489)Cited by:[§2](https://arxiv.org/html/2608.11583#S2.p1.1)\.
- Bianchiet al\.\(2023\)F\. Bianchi, M\. Suzgun, G\. Attanasio, P\. Röttger, D\. Jurafsky, T\. Hashimoto, and J\. ZouSafety\-tuned llamas: lessons from improving the safety of large language models that follow instructions\.ArXivabs/2309\.07875\.External Links:[Link](https://api.semanticscholar.org/CorpusID:261823321)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1)\.
- Bommasaniet al\.\(2022\)R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. Altman, S\. Arora, S\. von Arx, M\. S\. Bernstein, J\. Bohg, A\. Bosselut, E\. Brunskill, E\. Brynjolfsson, S\. Buch, D\. Card, R\. Castellon, N\. Chatterji, A\. Chen, K\. Creel, J\. Q\. Davis, D\. Demszky, C\. Donahue, M\. Doumbouya, E\. Durmus, S\. Ermon, J\. Etchemendy, K\. Ethayarajh, L\. Fei\-Fei, C\. Finn, T\. Gale, L\. Gillespie, K\. Goel, N\. Goodman, S\. Grossman, N\. Guha, T\. Hashimoto, P\. Henderson, J\. Hewitt, D\. E\. Ho, J\. Hong, K\. Hsu, J\. Huang, T\. Icard, S\. Jain, D\. Jurafsky, P\. Kalluri, S\. Karamcheti, G\. Keeling, F\. Khani, O\. Khattab, P\. W\. Koh, M\. Krass, R\. Krishna, R\. Kuditipudi, A\. Kumar, F\. Ladhak, M\. Lee, T\. Lee, J\. Leskovec, I\. Levent, X\. L\. Li, X\. Li, T\. Ma, A\. Malik, C\. D\. Manning, S\. Mirchandani, E\. Mitchell, Z\. Munyikwa, S\. Nair, A\. Narayan, D\. Narayanan, B\. Newman, A\. Nie, J\. C\. Niebles, H\. Nilforoshan, J\. Nyarko, G\. Ogut, L\. Orr, I\. Papadimitriou, J\. S\. Park, C\. Piech, E\. Portelance, C\. Potts, A\. Raghunathan, R\. Reich, H\. Ren, F\. Rong, Y\. Roohani, C\. Ruiz, J\. Ryan, C\. Ré, D\. Sadigh, S\. Sagawa, K\. Santhanam, A\. Shih, K\. Srinivasan, A\. Tamkin, R\. Taori, A\. W\. Thomas, F\. Tramèr, R\. E\. Wang, W\. Wang, B\. Wu, J\. Wu, Y\. Wu, S\. M\. Xie, M\. Yasunaga, J\. You, M\. Zaharia, M\. Zhang, T\. Zhang, X\. Zhang, Y\. Zhang, L\. Zheng, K\. Zhou, and P\. LiangOn the opportunities and risks of foundation models\.External Links:2108\.07258,[Link](https://arxiv.org/abs/2108.07258)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.External Links:2005\.14165,[Link](https://arxiv.org/abs/2005.14165)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p1.1)\.
- Carliniet al\.\(2024\)N\. Carlini, M\. Nasr, C\. A\. Choquette\-Choo, M\. Jagielski, I\. Gao, A\. Awadalla, P\. W\. Koh, D\. Ippolito, K\. Lee, F\. Tramer, and L\. SchmidtAre aligned neural networks adversarially aligned?\.External Links:2306\.15447,[Link](https://arxiv.org/abs/2306.15447)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1)\.
- Cuiet al\.\(2024\)J\. Cui, W\. Chiang, I\. Stoica, and C\. HsiehOR\-bench: an over\-refusal benchmark for large language models\.ArXivabs/2405\.20947\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270199786)Cited by:[§2](https://arxiv.org/html/2608.11583#S2.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p4.1),[§3\.4](https://arxiv.org/html/2608.11583#S3.SS4.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§3\.1](https://arxiv.org/html/2608.11583#S3.SS1.p1.1)\.
- Guptaet al\.\(2024\)P\. Gupta, L\. Q\. Yau, H\. H\. Low, I\. Lee, H\. Lim, Y\. X\. Teoh, J\. Koh, D\. W\. Liew, R\. Bhardwaj, R\. Bhardwaj, and S\. PoriaWalledEval: a comprehensive safety evaluation toolkit for large language models\.InConference on Empirical Methods in Natural Language Processing,External Links:[Link](https://api.semanticscholar.org/CorpusID:271744807)Cited by:[§2](https://arxiv.org/html/2608.11583#S2.p4.1),[§3\.4](https://arxiv.org/html/2608.11583#S3.SS4.p1.1)\.
- Huang \(2025\)A\. HuangSaferlhf\_ultra\_sft\.Note:Hugging Face model repositoryExternal Links:[Link](https://huggingface.co/skyai798/saferlhf_ultra_sft)Cited by:[§3\.1](https://arxiv.org/html/2608.11583#S3.SS1.p1.1)\.
- Jiet al\.\(2025\)J\. Ji, D\. Hong, B\. Zhang, B\. Chen, J\. Dai, B\. Zheng, T\. A\. Qiu, J\. Zhou, K\. Wang, B\. Li,et al\.Pku\-saferlhf: towards multi\-level safety alignment for llms with human preference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 31983–32016\.Cited by:[§3\.1](https://arxiv.org/html/2608.11583#S3.SS1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p1.1)\.
- Kraußet al\.\(2025\)T\. Krauß, H\. Dashtbani, and A\. DmitrienkoTwinBreak: jailbreaking llm security alignments based on twin prompts\.InUSENIX Security Symposium,External Links:[Link](https://api.semanticscholar.org/CorpusID:279250896)Cited by:[§2](https://arxiv.org/html/2608.11583#S2.p4.1),[§3\.4](https://arxiv.org/html/2608.11583#S3.SS4.p1.1)\.
- Leeet al\.\(2024\)B\. Lee, I\. Padhi, K\. N\. Ramamurthy, E\. Miehling, P\. L\. Dognin, M\. Nagireddy, and A\. DhurandharProgramming refusal with conditional activation steering\.ArXivabs/2409\.05907\.External Links:[Link](https://api.semanticscholar.org/CorpusID:272550481)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p3.1),[§2](https://arxiv.org/html/2608.11583#S2.p3.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in Neural Information Processing Systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p1.1)\.
- Perezet al\.\(2022\)E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. IrvingRed teaming language models with language models\.External Links:2202\.03286,[Link](https://arxiv.org/abs/2202.03286)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1)\.
- Qiet al\.\(2023\)X\. Qi, Y\. Zeng, T\. Xie, P\. Chen, R\. Jia, P\. Mittal, and P\. HendersonFine\-tuning aligned language models compromises safety, even when users do not intend to\!\.ArXivabs/2310\.03693\.External Links:[Link](https://api.semanticscholar.org/CorpusID:263671523)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.11583#S5.SS1.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p1.1)\.
- Röttgeret al\.\(2024\)P\. Röttger, H\. R\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. HovyXSTest: a test suite for identifying exaggerated safety behaviours in large language models\.Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics\.Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p4.1)\.
- Suma and Dauncey \(2025\)A\. Suma and S\. DaunceyDeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.ArXivabs/2501\.12948\.External Links:[Link](https://api.semanticscholar.org/CorpusID:284488789)Cited by:[§3\.1](https://arxiv.org/html/2608.11583#S3.SS1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p1.1)\.
- Weiet al\.\(2023\)A\. Wei, N\. Haghtalab, and J\. SteinhardtJailbroken: how does llm safety training fail?\.ArXivabs/2307\.02483\.External Links:[Link](https://api.semanticscholar.org/CorpusID:259342528)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1)\.
- Weiet al\.\(2024\)B\. Wei, K\. Huang, Y\. Huang, T\. Xie, X\. Qi, M\. Xia, P\. Mittal, M\. Wang, and P\. HendersonAssessing the brittleness of safety alignment via pruning and low\-rank modifications\.InInternational Conference on Machine Learning,External Links:[Link](https://api.semanticscholar.org/CorpusID:267547755)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p3.1),[§2](https://arxiv.org/html/2608.11583#S2.p3.1)\.
- Wollschlageret al\.\(2025\)T\. Wollschlager, J\. Elstner, S\. Geisler, V\. Cohen\-Addad, S\. Gunnemann, and J\. GasteigerThe geometry of refusal in large language models: concept cones and representational independence\.ArXivabs/2502\.17420\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276575573)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p3.1),[§2](https://arxiv.org/html/2608.11583#S2.p3.1)\.
- Yanget al\.\(2023\)X\. Yang, X\. Wang, Q\. Zhang, L\. R\. Petzold, W\. Y\. Wang, X\. Zhao, and D\. LinShadow alignment: the ease of subverting safely\-aligned language models\.ArXivabs/2310\.02949\.External Links:[Link](https://api.semanticscholar.org/CorpusID:263620436)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1),[§5\.1](https://arxiv.org/html/2608.11583#S5.SS1.p1.1)\.
- Zhanget al\.\(2026\)H\. Zhang, M\. Zandsalimy, and S\. SushmitaExposing llm safety gaps through mathematical encoding:new attacks and systematic analysis\.External Links:[Link](https://api.semanticscholar.org/CorpusID:287963140)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, Z\. Zeng, D\. Li, Y\. Huang, Z\. Deng, and Y\. DongRealSafe\-r1: safety\-aligned deepseek\-r1 without compromising reasoning capability\.ArXivabs/2504\.10081\.External Links:[Link](https://api.semanticscholar.org/CorpusID:277780999)Cited by:[§3\.1](https://arxiv.org/html/2608.11583#S3.SS1.p1.1)\.
- Zhaoet al\.\(2024\)W\. Zhao, Z\. Li, Y\. Li, Y\. Zhang, and J\. SunDefending large language models against jailbreak attacks via layer\-specific editing\.ArXivabs/2405\.18166\.External Links:[Link](https://api.semanticscholar.org/CorpusID:270067915)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p3.1),[§2](https://arxiv.org/html/2608.11583#S2.p3.1)\.
- Zouet al\.\(2023\)A\. Zou, Z\. Wang, J\. Z\. Kolter, and M\. FredriksonUniversal and transferable adversarial attacks on aligned language models\.ArXivabs/2307\.15043\.External Links:[Link](https://api.semanticscholar.org/CorpusID:260202961)Cited by:[§1](https://arxiv.org/html/2608.11583#S1.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p2.1),[§2](https://arxiv.org/html/2608.11583#S2.p4.1),[§3\.4](https://arxiv.org/html/2608.11583#S3.SS4.p1.1)\.