DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning

# DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning
Source: [https://arxiv.org/html/2608.04322](https://arxiv.org/html/2608.04322)
Junbo Zhang, Qianli Zhou, Xinyang Deng, Wen Jiang Northwestern Polytechnical University

###### Abstract

Task\-specific fine\-tuning can improve the performance of large language models \(LLMs\) on downstream tasks\. However, our study reveals that task\-specific fine\-tuning can also weaken the safety guardrails of aligned LLMs\. A widely adopted strategy for preserving safety during fine\-tuning is to incorporate safety data\. Although previous studies have shown that randomly mixing safety data can alleviate safety degradation, the underlying principle determining why some safety examples are more effective than others still remains unclear\. In this paper, we propose DataRx, a missingness\-aware sampling method for selecting safety\-critical examples\. DataRx is based on the hypothesis that a safety sample is more effective when the selected examples provide safety signals that fill the missing parts of LLMs’ safety capabilities\. DataRx’s key insight is leveraging high\-dimensional hidden representations rather than discrete tokens to quantify the safety signal gap between the target model’s native response and the safety reference response\. The results show that, with only 1% additional safety samples from BeaverTails, DataRx reduces the average attack success rate of Llama3\-8B\-Instruct across seven downstream tasks from 59\.23% under random sampling to 13\.70%\. In addition, DataRx can be combined with the existing safety data synthesis method to further enhance safety defenses during fine\-tuning\. We hope that DataRx will inspire more data\-centric defense research\.

## IIntroduction

With the rapid advancement of Large Language Models \(LLMs\), they have gradually become a fundamental infrastructure of modern artificial intelligence systems\. Leveraging their powerful representation learning and reasoning capabilities, LLMs have been widely deployed across diverse intelligent tasks and real\-world applications, extending beyond traditional language processing to broader domains such as spatial\-temporal data understanding\[[1](https://arxiv.org/html/2608.04322#bib.bib1)\], cross\-domain knowledge transfer\[[2](https://arxiv.org/html/2608.04322#bib.bib2)\], and tool\-augmented agent systems\[[3](https://arxiv.org/html/2608.04322#bib.bib3)\]\.

Task\-specific supervised fine\-tuning \(SFT\) is widely used to enhance the downstream performance of large language models \(LLMs\)\[[4](https://arxiv.org/html/2608.04322#bib.bib4),[5](https://arxiv.org/html/2608.04322#bib.bib5)\]\. Unfortunately, continuous parameter updates may disrupt previously established safety alignment, weakening refusal behaviors and reactivating suppressed harmful tendencies\. Recent studies\[[6](https://arxiv.org/html/2608.04322#bib.bib6),[7](https://arxiv.org/html/2608.04322#bib.bib7),[8](https://arxiv.org/html/2608.04322#bib.bib8)\]have shown that even benign instruction fine\-tuning can degrade LLM safety, but these findings mainly focus on instruction\-following scenarios, leaving the impact of task\-specific fine\-tuning unclear\. Unlike open\-ended instruction tuning, task\-specific fine\-tuning often involves constrained output spaces and explicit task objectives, which may lead to different behavioral changes and safety degradation patterns\. Therefore, we further explore the following key yet underexplored research question \(RQ1\):How does task\-specific fine\-tuning affect LLM safety?

![Refer to caption](https://arxiv.org/html/2608.04322v1/images/intro.png)Figure 1:Overview of the DataRx framework\. DataRx identifies safety\-critical examples from the safety dataset through missingness\-aware sampling, and the selected samples are mixed with task\-specific data for fine\-tuning\.To answer RQ1, we conduct an empirical study to investigate how task\-specific fine\-tuning affects LLM safety alignment under different settings\. Our results demonstrate that the extent of safety degradation varies across different fine\-tuning configurations, with higher learning rates and weaker initial safety capabilities leading to more severe degradation\.

A straightforward approach to mitigating safety degradation during downstream fine\-tuning is to mix a safety dataset with a task dataset\[[8](https://arxiv.org/html/2608.04322#bib.bib8)\]\. Compared with introducing an external moderation module\[[9](https://arxiv.org/html/2608.04322#bib.bib9)\]or a self\-reflection process\[[10](https://arxiv.org/html/2608.04322#bib.bib10)\], mixing safety\-alignment data during training incurs no additional inference\-time overhead\. Existing approaches for mixing safety data can broadly be divided into rewriting\-based and safety data synthesis methods\. Rewriting\-based methods typically assume that two types of distributional discrepancy exist between safety data and target fine\-tuning data\. The first is a task\-level discrepancy, in which safety data and task\-specific data differ in task format, prompting style, and data structure\[[11](https://arxiv.org/html/2608.04322#bib.bib11),[12](https://arxiv.org/html/2608.04322#bib.bib12),[13](https://arxiv.org/html/2608.04322#bib.bib13)\]\. The second is a model\-level discrepancy between the safety\-alignment dataset and the target LLM\[[14](https://arxiv.org/html/2608.04322#bib.bib14)\]\. To reduce these discrepancies, such methods adapt the distribution of safety examples through techniques such as data transformation\[[13](https://arxiv.org/html/2608.04322#bib.bib13),[12](https://arxiv.org/html/2608.04322#bib.bib12)\]and self\-distillation\[[14](https://arxiv.org/html/2608.04322#bib.bib14)\], making the examples more compatible with the target fine\-tuning process\. Safety data synthesis methods\[[15](https://arxiv.org/html/2608.04322#bib.bib15)\], by contrast, sample candidate safety queries from either the target model or an external model and construct safety\-alignment data using predefined prompt templates\. Although these approaches have demonstrated some effectiveness, they often rely on additional rewriting procedures, assistance from stronger external models, or manually designed templates covering specific safety domains\. Consequently, they introduce expensive data construction costs\.

In contrast, the LLM safety community has already accumulated a large collection of open\-source safety datasets\[[16](https://arxiv.org/html/2608.04322#bib.bib16),[17](https://arxiv.org/html/2608.04322#bib.bib17),[18](https://arxiv.org/html/2608.04322#bib.bib18)\]\. Directly reusing these existing data resources represents one of the simplest and least costly approaches to preserving safety\. This naturally leads to our second research question \(RQ2\):How effective is mixing safety datasets in mitigating safety degradation?

Our empirical results show that randomly mixing safety datasets yields inconsistent safety performance\. A key reason is that not every safety example is equally useful\. The central question is not merely whether safety data should be mixed with task\-specific data, but rather which safety examples can provide the most effective corrective safety signals for the target model\. We further refine RQ2 into a new research question \(RQ3\):Which safety examples are more effective?

Pham et al\.\[[19](https://arxiv.org/html/2608.04322#bib.bib19)\]conducted an empirical analysis of safety examples and categorized them into four types according to their behavioral patterns: refusal of harmful instructions, safe responses to harmful instructions, refusal of benign instructions, and normal responses to benign instructions\. Their analysis shows that examples involving the refusal of harmful instructions provide stronger safety supervision signals\. However, such analyses mainly evaluate the safety supervision signals contained in safety examples themselves, while overlooking the alignment gap between these signals and the target model’s current safety behavior\. In other words, the key question is whether a safety example can compensate for the gaps in the target model’s safety capabilities, rather than merely reinforcing safety behaviors that have already been acquired\.

To address these issues, we propose DataRx, a safety\-critical example sampling method\. As shown in Fig\.[1](https://arxiv.org/html/2608.04322#S1.F1), DataRx aims to identify safety\-critical samples from the safety dataset\. DataRx leverages the rich safety knowledge encoded in the model’s high\-dimensional hidden states to rank safety examples\. Specifically, DataRx first extracts a contrastive safety pattern by distinguishing safe refusal and unsafe compliance behaviors in hidden representation space\. The resulting representation enables quantitative measurement of refusal signals\. For each candidate safety example, DataRx compares the refusal signal of the safety reference response with that of the target model’s native response, and selects examples with larger gaps\.

Our contributions can be summarized as follows:

- •We study fine\-tuning risks in the task\-specific setting, demonstrating that benign users are likely to accidentally generate harmful models\.
- •We propose DataRx, an efficient sampling strategy that mitigates safety degradation against fine\-tuning by selecting safety\-critical samples from safety datasets\.
- •DataRx can be combined with existing data generation approaches to further strengthen safety defenses\.
- •We conduct extensive experiments across seven downstream tasks and three LLMs, demonstrating the effectiveness of DataRx\.

## IIPreliminary Study

Qi et al\.\[[6](https://arxiv.org/html/2608.04322#bib.bib6)\]show that general instruction fine\-tuning can weaken model safety even when the training data contain no explicitly harmful content\. Subsequent work\[[13](https://arxiv.org/html/2608.04322#bib.bib13)\]extends this investigation to task\-specific fine\-tuning and demonstrates that benign users are unlikely to accidentally generate harmful models\. Nevertheless, there is still a lack of understanding of how different fine\-tuning settings and tasks affect the safety mechanisms of these LLMs\. This naturally leads to the following research questions\.

![Refer to caption](https://arxiv.org/html/2608.04322v1/x1.png)Figure 2:Comparison of safety degradation and fine\-tuning accuracy under different learning rates for task\-specific LLM fine\-tuning\.> RQ1:How does task\-specific fine\-tuning affect LLM safety?

To answer RQ1, we investigate whether task\-specific fine\-tuning leads to safety degradation under different fine\-tuning configurations\. Specifically, we fine\-tune three safety\-aligned LLMs on seven downstream tasks with two learning rates \(2×10−52\\times 10^\{\-5\}and2×10−42\\times 10^\{\-4\}\)\. We evaluate safety changes by comparing attack success rates \(ASR\) before and after fine\-tuning\. Specifically, we define the safety alignment gap asΔ​A​S​R=A​S​RF​T−A​S​RB​a​s​e\\Delta ASR=ASR\_\{FT\}\-ASR\_\{Base\}, whereA​S​RB​a​s​eASR\_\{Base\}andA​S​RF​TASR\_\{FT\}denote the attack success rates of the original aligned model and the task\-specific fine\-tuned model, respectively\. A positiveΔ​A​S​R\\Delta ASRindicates increased vulnerability to harmful prompts after fine\-tuning and thus reflects safety degradation\. We additionally compare downstream task accuracy across different learning\-rate settings\. Detailed descriptions of the models, datasets, training configurations, and evaluation protocols are provided in Section[IV\-A](https://arxiv.org/html/2608.04322#S4.SS1)\.

Experiment Results \(RQ1\)\.As shown in Fig\.[2](https://arxiv.org/html/2608.04322#S2.F2), we make the following three observations\. \(1\) Mistral, which exhibits the weakest initial safety alignment among the evaluated models, still suffers substantial safety degradation even under the conservative learning rate of2×10−52\\times 10^\{\-5\}\. On the DirectHarm4 benchmark, Mistral\-7B\-Instruct\-v0\.2 exhibits an initial ASR of65\.5%65\.5\\%, substantially higher than Llama3\-8B\-Instruct \(12\.75%12\.75\\%\) and Qwen2\.5\-7B\-Instruct \(9\.5%9\.5\\%\)\. \(2\) Increasing the learning rate from2×10−52\\times 10^\{\-5\}to2×10−42\\times 10^\{\-4\}substantially enlarges the safety alignment gap across all three models and most downstream tasks\. \(3\) The third row of Fig\.[2](https://arxiv.org/html/2608.04322#S2.F2)shows that models fine\-tuned with the higher learning rate generally achieve higher downstream task accuracy than those fine\-tuned with the lower learning rate across most tasks\. This suggests that users may adopt larger learning rates to achieve better downstream task performance, while unintentionally weakening the safety guardrails inherited from the original aligned models\. This finding is supported by recent work\[[20](https://arxiv.org/html/2608.04322#bib.bib20)\], which observes that the learning rate is an important factor influencing the impact of fine\-tuning on LLM behaviors and capabilities\. These results challenge previous conclusions\[[13](https://arxiv.org/html/2608.04322#bib.bib13)\]that benign users are unlikely to unintentionally produce harmful models through task\-specific fine\-tuning\. This may be because previous studies focused on LLMs with stronger initial safety capabilities and conservative learning rates, potentially underestimating the safety risks of task\-specific fine\-tuning\.

> RQ2:How effective is mixing safety datasets in mitigating safety degradation?

Given the safety degradation observed in RQ1, a straightforward mitigation strategy is to randomly mix open\-source safety\-alignment data with downstream task data\. To answer RQ2, we randomly sample examples from three open\-source safety datasets, namely Aegis\[[16](https://arxiv.org/html/2608.04322#bib.bib16)\], BeaverTails\[[17](https://arxiv.org/html/2608.04322#bib.bib17)\], and RedOcra\[[18](https://arxiv.org/html/2608.04322#bib.bib18)\], and evaluate the changes in ASR on DirectHarm4 after applying this strategy\.

![Refer to caption](https://arxiv.org/html/2608.04322v1/x2.png)Figure 3:Effectiveness of random safety data mixing during task\-specific fine\-tuning across different safety datasets\. Positive safety gain values indicate reduced harmful behaviors\.Experiment Results \(RQ2\)\.As shown in Fig\.[3](https://arxiv.org/html/2608.04322#S2.F3), random safety data mixing produces highly inconsistent results\. Some safety datasets achieve substantial positive gains under specific combinations of models and tasks, whereas others provide limited benefits or even introduce negative effects\. These observations suggest that the key challenge is not merely whether safety data should be mixed with downstream data, but rather which safety examples can provide an effective corrective signal\.

> RQ3:Which safety examples are more effective?

![Refer to caption](https://arxiv.org/html/2608.04322v1/x3.png)Figure 4:Attack success rates of Llama3, Qwen2\.5, and Mistral after fine\-tuning with easy and hard safety examples across three safety benchmarks\.The inconsistent performance of random safety\-data mixing suggests that not all safety\-alignment examples provide equally effective supervision\. We hypothesize that examples providing safety signals missing from the target model offer stronger corrective supervision\. Specifically, examples for prompts that the model already refuses mainly reinforce existing behaviors, whereas examples exposing the model’s safety weaknesses provide stronger alignment signals\.

To verify this hypothesis, we query the target model on harmful prompts from Aegis and divide the samples into two groups: Easy samples, for which the model already generates safe refusals, and Hard samples, for which the model generates unsafe responses\. We replace all reference safety responses with the same refusal response: “I’m sorry, but I cannot assist with your request\.”

As shown in Fig\.[4](https://arxiv.org/html/2608.04322#S2.F4), fine\-tuning with hard samples consistently achieves lower ASR than easy samples across 3 LLMs\. These results indicate that examples targeting the model’s safety weaknesses provide stronger corrective signals\. However, the easy and hard partition provides only a coarse binary measure and cannot capture fine\-grained differences in safety gaps\. We further propose a finer\-grained approach that quantifies the safety signals of safety examples\.

![Refer to caption](https://arxiv.org/html/2608.04322v1/images/method.png)Figure 5:Overview of the DataRx framework\. \(a\) Contrastive safety pattern extraction: DataRx bulids layer\-wise refusal\-related safety representations by contrasting safe refusal and unsafe compliance behaviors\. \(b\) Missingness\-aware safety sampling: DataRx identifies safety\-critical examples by measuring the representation gap between safety reference responses and the target model’s native responses, and ranks candidates according to their safety adaptation scores\.
## IIIMethod

To mitigate the safety degradation caused by fine\-tuning, we propose DataRx, a missingness\-aware safety\-critical sample selection method\. DataRx aims to rank samples from a candidate safety dataset and identify a core subset\.

Formally, letℳθ\\mathcal\{M\}\_\{\\theta\}denote a LLM, and let𝒟task=\{\(xi,yi\)\}i=1N\\mathcal\{D\}\_\{\\mathrm\{task\}\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote the task\-specific fine\-tuning dataset\. We are additionally given a candidate safety dataset𝒟safe=\{disafe\}i=1M\\mathcal\{D\}\_\{\\mathrm\{safe\}\}=\\\{d\_\{i\}^\{\\mathrm\{safe\}\}\\\}\_\{i=1\}^\{M\}, To characterize the safety behavior of the target model, we further assume access to two small anchor datasets,𝒟r\\mathcal\{D\}\_\{r\}and𝒟c\\mathcal\{D\}\_\{c\}, which provide contrasting safety behaviors\. Specifically,𝒟r\\mathcal\{D\}\_\{r\}consists of harmful instructions paired with safe refusal responses, whereas𝒟c\\mathcal\{D\}\_\{c\}consists of harmful instructions paired with unsafe compliance responses\. Given a selection budgetKK, our objective is to select a subset𝒟selected⊆𝒟safe\\mathcal\{D\}\_\{\\mathrm\{selected\}\}\\subseteq\\mathcal\{D\}\_\{\\mathrm\{safe\}\}with\|𝒟selected\|=K\|\\mathcal\{D\}\_\{\\mathrm\{selected\}\}\|=K, which is subsequently combined with𝒟task\\mathcal\{D\}\_\{\\mathrm\{task\}\}for fine\-tuning\. As illustrated in Fig\.[5](https://arxiv.org/html/2608.04322#S2.F5), DataRx consists of two main components:

1\. Contrastive Safety Pattern Extraction\.Using the anchor datasets𝒟r\\mathcal\{D\}\_\{r\}and𝒟c\\mathcal\{D\}\_\{c\}, this component extracts the safety pattern of the target LLM from its hidden representations\. Based on contrasting safe refusal behaviors with unsafe compliance behaviors in the representation space, we define the refusal score to measure the refusal tendency of a given instruction\-response pair\.

2\. Missingness\-Aware Sampling\.Given the candidate safety dataset𝒟safe\\mathcal\{D\}\_\{\\mathrm\{safe\}\}, this component identifies safety\-critical examples by measuring the target model’s missing safety signals\. Specifically, for each candidate sampledisafed\_\{i\}^\{\\mathrm\{safe\}\}, we compare the representation of the target model’s native response with that of the safety reference response provided by the sample\. The gap between these two representations measures the additional refusal capability supplied by the safety sample and is defined as the safety adaptation score\. Samples with higher safety adaptation scores provide stronger corrective safety signals and are prioritized for selection\.

### III\-AProblem Formulation

We mix𝒟task\\mathcal\{D\}\_\{\\mathrm\{task\}\}with a selected safety subset𝒟selected⊆𝒟safe\\mathcal\{D\}\_\{\\mathrm\{selected\}\}\\subseteq\\mathcal\{D\}\_\{\\mathrm\{safe\}\}\. Specifically,𝒟task\\mathcal\{D\}\_\{\\mathrm\{task\}\}and𝒟selected\\mathcal\{D\}\_\{\\mathrm\{selected\}\}represent two corresponding data distributions, denoted as𝒫task\\mathcal\{P\}\_\{\\mathrm\{task\}\}and𝒫safe\\mathcal\{P\}\_\{\\mathrm\{safe\}\}, respectively\. Following the Huber contamination model\[[21](https://arxiv.org/html/2608.04322#bib.bib21)\], the mixed data distribution is formulated as

𝒫=𝒫task\+𝒫safe\.\\mathcal\{P\}=\\mathcal\{P\}\_\{\\mathrm\{task\}\}\+\\mathcal\{P\}\_\{\\mathrm\{safe\}\}\.\(1\)
The key challenge is to identify a subset from the candidate safety dataset that provides the most effective corrective safety signals for LLMs\.

### III\-BContrastive Safety Pattern Extraction

Each sample in the candidate safety dataset is given in the form of an instruction\-response pair\. Clearly, we need to evaluate the safety criticality of each sample\. However, selection strategies\[[19](https://arxiv.org/html/2608.04322#bib.bib19)\]operate in a discrete token space, failing to accurately capture the differences in safety signals between different samples\. To overcome this limitation, we utilize the internal representation space of the LLM to characterize safety patterns\. Two small anchor datasets provide contrasting safety references:

𝒟r=\{djr\}j=1Nr,𝒟c=\{djc\}j=1Nc\.\\mathcal\{D\}\_\{r\}=\\left\\\{d\_\{j\}^\{r\}\\right\\\}\_\{j=1\}^\{N\_\{r\}\},\\qquad\\mathcal\{D\}\_\{c\}=\\left\\\{d\_\{j\}^\{c\}\\right\\\}\_\{j=1\}^\{N\_\{c\}\}\.\(2\)
Here,𝒟r\\mathcal\{D\}\_\{r\}contains harmful instructions paired with safe refusals, whereas𝒟c\\mathcal\{D\}\_\{c\}pairs harmful instructions with unsafe compliant responses\. Because the inputs are of the same type, the two datasets primarily differ in the behavioral pattern expressed by their responses\.

Let the target modelℳθ\\mathcal\{M\}\_\{\\theta\}containLLTransformer layers\. Given an arbitrary instruction\-response sampledd, we extract the hidden state of the final⟨eos⟩\\langle\\mathrm\{eos\}\\rangletoken at thell\-th layer as its representation:

hl​\(d\),l∈\{1,…,L\}\.h^\{l\}\(d\),\\quad l\\in\\\{1,\\ldots,L\\\}\.
At each layer, we compute the refusal and compliance centers by averaging the hidden representations of all samples within the corresponding anchor datasets:

𝐂rl=1Nr​∑j=1Nr𝐡l​\(djr\),𝐂cl=1Nc​∑j=1Nc𝐡l​\(djc\)\.\\mathbf\{C\}\_\{r\}^\{l\}=\\frac\{1\}\{N\_\{r\}\}\\sum\_\{j=1\}^\{N\_\{r\}\}\\mathbf\{h\}^\{l\}\\left\(d\_\{j\}^\{r\}\\right\),\\qquad\\mathbf\{C\}\_\{c\}^\{l\}=\\frac\{1\}\{N\_\{c\}\}\\sum\_\{j=1\}^\{N\_\{c\}\}\\mathbf\{h\}^\{l\}\\left\(d\_\{j\}^\{c\}\\right\)\.\(3\)
Here,𝐂rl\\mathbf\{C\}\_\{r\}^\{l\}and𝐂cl\\mathbf\{C\}\_\{c\}^\{l\}denote the refusal and compliance anchor centers at thell\-th layer, respectively\.

For an arbitrary sampledd, we define its layer\-wise refusal score as the difference between its cosine similarities to the two anchor centers:

sl​\(d\)=cos⁡\(𝐡l​\(d\),𝐂rl\)−cos⁡\(𝐡l​\(d\),𝐂cl\),s^\{l\}\(d\)=\\operatorname\{cos\}\\left\(\\mathbf\{h\}^\{l\}\(d\),\\mathbf\{C\}\_\{r\}^\{l\}\\right\)\-\\operatorname\{cos\}\\left\(\\mathbf\{h\}^\{l\}\(d\),\\mathbf\{C\}\_\{c\}^\{l\}\\right\),\(4\)
wherecos⁡\(⋅,⋅\)\\operatorname\{cos\}\(\\cdot,\\cdot\)denotes cosine similarity\. A largersl​\(d\)s^\{l\}\(d\)places the sample closer to the refusal center than to the compliance center, thereby indicating a stronger refusal pattern at layerll\.

### III\-CMissingness\-Aware Sampling

Missingness\-aware sampling identifies safety\-critical samples by comparing the target model’s native responses with the safety references provided by the candidate dataset\. Intuitively, it prioritizes instructions that expose the target model’s safety weaknesses over those that already elicit safe responses\.

For each candidate safety sampledisafe=\(xi,yisafe\)∈𝒟safed\_\{i\}^\{\\mathrm\{safe\}\}=\(x\_\{i\},y\_\{i\}^\{\\mathrm\{safe\}\}\)\\in\\mathcal\{D\}\_\{\\mathrm\{safe\}\}, wherexix\_\{i\}is the harmful instruction andyisafey\_\{i\}^\{\\mathrm\{safe\}\}is the safety reference response provided by the candidate safety dataset, we query the target LLMℳθ\\mathcal\{M\}\_\{\\theta\}with the same instructionxix\_\{i\}to obtain its current response:

yiorig=ForwardPass⁡\(ℳθ,xi\)\.y\_\{i\}^\{\\mathrm\{orig\}\}=\\operatorname\{ForwardPass\}\(\\mathcal\{M\}\_\{\\theta\},x\_\{i\}\)\.\(5\)
The generated response is then paired with the original instruction to form a native response pair:

diorig=\(xi,yiorig\)\.d\_\{i\}^\{\\mathrm\{orig\}\}=\(x\_\{i\},y\_\{i\}^\{\\mathrm\{orig\}\}\)\.\(6\)
We refer todisafed\_\{i\}^\{\\mathrm\{safe\}\}as the safety reference pair, which represents the desired safety behavior provided by the candidate dataset, anddiorigd\_\{i\}^\{\\mathrm\{orig\}\}as the native response pair, which captures the target model’s current safety behavior\.

Based on the layer\-wise refusal score defined in Eq\. \([4](https://arxiv.org/html/2608.04322#S3.E4)\), we compute the refusal scores of the safety reference pair and the native response pair as:

Risafe,l=sl​\(disafe\),Riorig,l=sl​\(diorig\)\.R\_\{i\}^\{\\mathrm\{safe\},l\}=s^\{l\}\\left\(d\_\{i\}^\{\\mathrm\{safe\}\}\\right\),\\qquad R\_\{i\}^\{\\mathrm\{orig\},l\}=s^\{l\}\\left\(d\_\{i\}^\{\\mathrm\{orig\}\}\\right\)\.\(7\)
These two scores quantify the refusal signals provided by the safety reference response and the target model’s original response, respectively\.

For an arbitrary sampledd, its overall refusal score is defined as the mean of its layer\-wise refusal scores over allLLTransformer layers:

R​\(d\)=1L​∑l=1Lsl​\(d\)\.R\(d\)=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}s^\{l\}\(d\)\.\(8\)
Here, we compute the refusal score by averaging across all layers rather than relying on a single layer\. Approaches that rely on a single layer face three issues\. \(1\) It introduces an additional layer selection burden\. Identifying the optimal layer typically requires a carefully designed validation set, and previous studies\[[22](https://arxiv.org/html/2608.04322#bib.bib22)\]have shown that the selected layer can vary across different models\. Whether such layer preferences remain consistent across datasets is still unclear\. \(2\) Layer selection is metric\-dependent\. While steering\-based evaluations in\[[22](https://arxiv.org/html/2608.04322#bib.bib22)\]and other safety\-related metrics\[[7](https://arxiv.org/html/2608.04322#bib.bib7)\]suggest that intermediate layers often provide the strongest performance, some studies indicate that early or late layers may play a more important role\[[23](https://arxiv.org/html/2608.04322#bib.bib23),[24](https://arxiv.org/html/2608.04322#bib.bib24)\]\. Therefore, the question of which layer should be selected remains unresolved\. \(3\) The reliability of the single\-layer assumption is uncertain\. Choosing one layer implicitly assumes that safety\-related behaviors are localized within a specific network depth, rather than being represented and processed collectively across multiple layers\. However, safety\-related signals in LLMs may be distributed throughout the network, with different layers jointly contributing to the final behavior\. Our experimental results further demonstrate the effectiveness of aggregating representations across multiple layers for measuring safety\-related signals\.

Accordingly, the overall refusal scores of the safety reference pair and the native response pair are defined as:

Risafe=R​\(disafe\)=1L​∑l=1Lsl​\(disafe\)\.R\_\{i\}^\{\\mathrm\{safe\}\}=R\\left\(d\_\{i\}^\{\\mathrm\{safe\}\}\\right\)=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}s^\{l\}\\left\(d\_\{i\}^\{\\mathrm\{safe\}\}\\right\)\.\(9\)
Riorig=R​\(diorig\)=1L​∑l=1Lsl​\(diorig\)\.R\_\{i\}^\{\\mathrm\{orig\}\}=R\\left\(d\_\{i\}^\{\\mathrm\{orig\}\}\\right\)=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}s^\{l\}\\left\(d\_\{i\}^\{\\mathrm\{orig\}\}\\right\)\.\(10\)
We define the safety adaptation score \(SAS\) of theii\-th candidate safety sample as the difference between these two overall refusal scores:

SASi=Risafe−Riorig=1L​∑l=1L\[sl​\(disafe\)−sl​\(diorig\)\]\.\\mathrm\{SAS\}\_\{i\}=R\_\{i\}^\{\\mathrm\{safe\}\}\-R\_\{i\}^\{\\mathrm\{orig\}\}=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}\\left\[s^\{l\}\\left\(d\_\{i\}^\{\\mathrm\{safe\}\}\\right\)\-s^\{l\}\\left\(d\_\{i\}^\{\\mathrm\{orig\}\}\\right\)\\right\]\.\(11\)
The SAS captures the additional refusal signal introduced by the safety reference pair over the native response pair\.

Finally, DataRx ranks all candidate safety samples in descending order according to their Safety Adaptation Scores and selects the topKKsamples:

𝒟selected=TopKdisafe∈𝒟safe⁡SASi\.\\mathcal\{D\}\_\{\\mathrm\{selected\}\}=\\operatorname\{TopK\}\_\{d\_\{i\}^\{\\mathrm\{safe\}\}\\in\\mathcal\{D\}\_\{\\mathrm\{safe\}\}\}\\mathrm\{SAS\}\_\{i\}\.\(12\)
Following prior work\[[19](https://arxiv.org/html/2608.04322#bib.bib19)\], we restrict our selection to samples consisting of harmful instructions paired with refusal responses\.

## IVExperiments

### IV\-AExperimental Setup

Models\.We evaluate DataRx on three representative open\-source LLMs: Llama3\-8B\-Instruct\[[4](https://arxiv.org/html/2608.04322#bib.bib4)\], Qwen2\.5\-7B\-Instruct\[[25](https://arxiv.org/html/2608.04322#bib.bib25)\], and Mistral\-7B\-Instruct\-v0\.2\[[26](https://arxiv.org/html/2608.04322#bib.bib26)\]\. These models cover diverse model families, allowing us to evaluate the effectiveness and generalizability of DataRx\. We employ LoRA\[[27](https://arxiv.org/html/2608.04322#bib.bib27)\]for parameter\-efficient fine\-tuning\. LoRA adapters are injected into all linear layers of each model, with a rank of 16, a scaling factor of 32, and a dropout rate of 0\.05\. The learning rate is set to2×10−42\\times 10^\{\-4\}, and the model is trained for one epoch\.

Datasets\.We use three categories of datasets in our experiments: downstream task datasets, candidate safety datasets, and anchor safety datasets\.

Downstream task datasets\.We select seven representative downstream datasets covering diverse task categories, including mathematical reasoning, commonsense reasoning, question answering, code generation, and machine translation\. Specifically, we adopt GSM8K\[[28](https://arxiv.org/html/2608.04322#bib.bib28)\]and MATH\[[29](https://arxiv.org/html/2608.04322#bib.bib29)\]for mathematical reasoning, HellaSwag\[[30](https://arxiv.org/html/2608.04322#bib.bib30)\]for sentence completion , WinoGrande\[[31](https://arxiv.org/html/2608.04322#bib.bib31)\]for commonsense reasoning , MedQA\[[32](https://arxiv.org/html/2608.04322#bib.bib32)\]for medical question answering, Magicoder\[[33](https://arxiv.org/html/2608.04322#bib.bib33)\]for code generation, and DocBlocks\[[34](https://arxiv.org/html/2608.04322#bib.bib34)\]for Chinese\-to\-English translation\. For DocBlocks, we align the English source text with its Chinese translation at the paragraph level\. The statistics of training and evaluation samples are summarized in Table[I](https://arxiv.org/html/2608.04322#S4.T1)\.

Candidate safety datasets\.To mitigate safety degradation during task\-specific fine\-tuning, we consider two widely used safety datasets, Aegis and BeaverTails\[[16](https://arxiv.org/html/2608.04322#bib.bib16),[17](https://arxiv.org/html/2608.04322#bib.bib17)\], as candidate sources for mix fine\-tuning\. During experiments, safety examples are selected from these datasets and mixed with downstream task data under the same fine\-tuning setting\.

Anchor safety datasets\.We construct the anchor datasets using paired safety references from the Circuit Breaker Dataset\[[35](https://arxiv.org/html/2608.04322#bib.bib35)\]\. We sample five paired examples from each of 20 safety categories, resulting in 100 safe refusal samples and 100 unsafe compliance samples\.

TABLE I:Datasets used in the experiments\.\|𝒟ft\|\|\\mathcal\{D\}\_\{\\text\{ft\}\}\|and\|𝒟val\|\|\\mathcal\{D\}\_\{\\text\{val\}\}\|denote the numbers of downstream fine\-tuning training and evaluation samples, respectively\.DatasetTask\|𝒟ft\|\|\\mathcal\{D\}\_\{\\text\{ft\}\}\|\|𝒟val\|\|\\mathcal\{D\}\_\{\\text\{val\}\}\|GSM8KMath Reasoning7,1681,319MATHMath Reasoning7,1685,000HellaSwagSentence Completion10,24010,042WinoGrandeCommonsense Reasoning10,2401,267MedQAMultiple\-choice QA9,2161,273DocBlocksEnglish\-to\-Chinese Translation68611702MagicoderCode Generation10,000164Evaluation metrics\.

- •Fine\-tuning Accuracy \(ACC\)\.To evaluate downstream utility after fine\-tuning, we measure model performance on the corresponding held\-out evaluation sets using task\-specific metrics\. For GSM8K, we extract the final numerical answer from each generated response and compute exact match accuracy\. For MATH, we extract the final boxed answer and calculate accuracy after normalizing equivalent answer formats\. HellaSwag, WinoGrande, and MedQA are evaluated using multiple\-choice accuracy by comparing the predicted option with the ground truth answer\. For Magicoder, we evaluate code generation performance on HumanEval\[[36](https://arxiv.org/html/2608.04322#bib.bib36)\]using pass@1 based on executable unit tests\. We evaluate translation quality using the COMET\-DA metric\[[37](https://arxiv.org/html/2608.04322#bib.bib37)\]\.
- •Attack Success Rate \(ASR\)\.To evaluate LLM safety after fine\-tuning, we adopt three safety benchmarks: DirectHarm4\[[38](https://arxiv.org/html/2608.04322#bib.bib38)\], HarmBench\[[39](https://arxiv.org/html/2608.04322#bib.bib39)\], and HEx\-PHI\[[6](https://arxiv.org/html/2608.04322#bib.bib6)\]\. We prompt each evaluated model with harmful inputs and generate responses under deterministic decoding settings\. The generated responses are classified as safe or unsafe using Llama\-Guard\-3\-8B\[[4](https://arxiv.org/html/2608.04322#bib.bib4)\]\. The attack success rate \(ASR\) is calculated separately for each benchmark as the proportion of unsafe responses among all validly classified responses\. Unless otherwise specified, all ASR values reported in the following main experiments are computed as the arithmetic mean of the three benchmark ASRs\.

Baselines\.We evaluate the following mitigation strategies that leverage safety data\.

- •Random\. Selects samples randomly from the safety dataset, which are then used for safety\-enhanced fine\-tuning\.
- •Longest\. For Longest\[[40](https://arxiv.org/html/2608.04322#bib.bib40)\], we select the samples with the longest response lengths from the safety dataset\.
- •Paraphrase\. For Paraphrase\[[13](https://arxiv.org/html/2608.04322#bib.bib13)\], we utilize LLaMA\-2 13B to rewrite the safety data according to the prompt format and linguistic style of the downstream fine\-tuning dataset, aligning the safety samples with the data distribution before incorporating them into the fine\-tuning process\.
- •Self\-Distill\. For Self\-Distill\[[14](https://arxiv.org/html/2608.04322#bib.bib14)\], we use the target LLM to perform self\-distillation on the reference responses of the original safety dataset, generating distilled samples that follow the model’s own safety behavior distribution\.
- •SSS\-B\. For SSS\-B\[[19](https://arxiv.org/html/2608.04322#bib.bib19)\], harmful refusal samples are first identified using WildGuard\[[41](https://arxiv.org/html/2608.04322#bib.bib41)\], and their corresponding hazard categories are annotated with Qwen3Guard\-Gen\-8B\[[42](https://arxiv.org/html/2608.04322#bib.bib42)\]\. The identified samples are then grouped according to their hazard categories\. A small, category\-balanced subset is uniformly sampled from each group to preserve coverage across diverse safety risk types\.
- •PSS\-B\. For PSS\-B\[[19](https://arxiv.org/html/2608.04322#bib.bib19)\], the same preprocessing procedure as SSS\-B is applied\. Within each harmful category, Sentence\-BERT \(all\-mpnet\-base\-v2\)\[[43](https://arxiv.org/html/2608.04322#bib.bib43)\]embeddings are used to represent samples and compute the semantic centroid of the category\. Samples with embeddings closest to their corresponding centroids are selected as representative examples\.

### IV\-BComparison of Other Mitigation Strategies

TABLE II:Comparison of Attack Success Rate \(ASR, %\) and Downstream Task Fine\-tuning Accuracy \(Acc, %\) across different models and downstream fine\-tuning tasks\. The downstream and safety alignment data are mixed at a fixed ratio of 100:1, with Beavertails as the safety alignment dataset\. Lower HS indicates better safety\. Higher Acc indicates better downstream task performance\.ModelAlignment DataGSM8KMATHHellaSwagWinoGrandeMedQAMagicoderDockBlocksAvg\.ASR↓\\downarrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowLlama3\-8B\-InstructDownstream\-only30\.9666\.1915\.5121\.1821\.4393\.4516\.4784\.6114\.8762\.3728\.4856\.7013\.9682\.9020\.24Beavertails \(Random\)67\.6865\.8153\.8321\.4655\.9093\.0579\.1883\.5878\.3861\.9065\.9056\.1013\.7783\.4559\.23Paraphrase22\.6567\.2518\.1521\.409\.9592\.8431\.6284\.2924\.1661\.5924\.1457\.3212\.5683\.0920\.46Longest68\.0566\.4153\.5722\.2053\.7993\.1467\.7483\.5069\.3762\.0624\.1656\.7114\.6783\.2450\.19Self\-Distil21\.1665\.9612\.2621\.9024\.5893\.1735\.3483\.3519\.9861\.4324\.7256\.7112\.4883\.3121\.50SSS\-B45\.9665\.5832\.8221\.9621\.8893\.1620\.1684\.0632\.4261\.4326\.9757\.3215\.0383\.1527\.89PSS\-B43\.8066\.5734\.8421\.8428\.4193\.3034\.7284\.2934\.3462\.2927\.3059\.1514\.4983\.2931\.13Ours18\.6666\.4912\.1721\.5013\.5793\.215\.6983\.8211\.5861\.3522\.7356\.1011\.5383\.2813\.70Qwen2\.5\-7B\-InstructDownstream\-only15\.6376\.8021\.1145\.4220\.5192\.6821\.0985\.0817\.5164\.1820\.6777\.4416\.8984\.8419\.06Beavertails28\.4576\.5039\.5146\.1057\.3493\.7180\.7784\.4569\.4964\.4156\.4678\.0518\.5484\.9750\.08Paraphrase16\.6576\.1224\.1745\.748\.3093\.7125\.8284\.7711\.5563\.7921\.4476\.2215\.9084\.6317\.69Longest44\.3376\.2764\.7846\.0082\.8093\.6270\.9984\.5385\.8363\.3227\.9075\.6120\.5484\.6856\.74Self\-Distil35\.2177\.3341\.5846\.4641\.2893\.7152\.2484\.2145\.4763\.9416\.9578\.0514\.9684\.5835\.38SSS\-B21\.3976\.6529\.6346\.2217\.7193\.6519\.3385\.3224\.8064\.0221\.7776\.2223\.9384\.6322\.65PSS\-B25\.2577\.1835\.8345\.3419\.3293\.6737\.3384\.6130\.4963\.4723\.1378\.0522\.4184\.7727\.68Ours8\.2676\.4216\.1146\.105\.3093\.789\.7284\.4515\.2863\.9415\.6376\.2215\.7384\.7012\.29Mistral\-7B\-Instruct\-v0\.2Downstream\-only90\.2754\.3684\.2511\.5275\.5993\.1477\.1282\.9574\.5556\.7281\.9243\.2967\.5378\.1078\.75Beavertails90\.1350\.2791\.3011\.4290\.4693\.9990\.3882\.2491\.6855\.8592\.2944\.5171\.9378\.2388\.31Paraphrase72\.3352\.3168\.6612\.0071\.7393\.8466\.9682\.5675\.2056\.0983\.6643\.9069\.5778\.1072\.59Longest89\.5253\.6889\.8011\.5890\.0194\.0190\.5181\.5388\.9555\.9386\.5046\.3469\.8078\.2086\.44Self\-Distil83\.3652\.5481\.6511\.9684\.0993\.9182\.7684\.2190\.8156\.3283\.3943\.9069\.3978\.1782\.21SSS\-B78\.0451\.6377\.1111\.0467\.9994\.0352\.6482\.4870\.3956\.3285\.2746\.3470\.6378\.3771\.72PSS\-B68\.8552\.3968\.6911\.4666\.1093\.8554\.9682\.4868\.1256\.6487\.4545\.1271\.8177\.9869\.43Ours63\.7951\.1856\.6111\.9052\.7594\.1654\.5783\.1161\.6456\.3284\.3344\.5163\.3178\.2362\.43

TABLE III:Comparison of Attack Success Rate \(ASR, %\) and Downstream Task Fine\-tuning Accuracy \(Acc, %\) across different models and downstream fine\-tuning tasks\. The downstream and safety alignment data are mixed at a fixed ratio of 100:1, with Aegis as the safety alignment dataset\. Lower HS indicates better safety\. Higher Acc indicates better downstream task performance\.ModelAlignment DataGSM8KMATHHellaSwagWinoGrandeMedQAMagicoderDockBlocksAvg\.ASR↓\\downarrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowASR↓\\downarrowACC↑\\uparrowLlama3\-8B\-InstructDownstream\-only30\.9666\.1915\.5121\.1821\.4393\.4516\.4784\.6114\.8762\.3728\.4856\.7013\.9682\.9020\.24Aegis \(Random\)5\.3766\.194\.1121\.484\.2492\.931\.7584\.063\.2862\.6921\.7555\.4912\.0383\.387\.50Paraphrase4\.1465\.961\.3621\.820\.9893\.171\.3383\.581\.6162\.2921\.4956\.1011\.5683\.316\.07Longest75\.9365\.8869\.9722\.1464\.4193\.0866\.3382\.7962\.3861\.1923\.5755\.4913\.6082\.9753\.74Self\-Distil10\.6266\.116\.0721\.1813\.7693\.134\.2682\.7915\.6162\.3724\.4758\.5410\.7883\.1812\.22SSS\-B1\.3366\.791\.6722\.382\.4593\.090\.6784\.291\.4261\.9823\.0859\.1511\.6883\.316\.04PSS\-B3\.2366\.492\.3821\.482\.5393\.162\.1183\.192\.5061\.7423\.1857\.3211\.7683\.206\.81Ours0\.9166\.491\.0621\.501\.2393\.210\.5883\.820\.8361\.3515\.0056\.1010\.5883\.384\.31Qwen2\.5\-7B\-InstructDownstream\-only15\.6376\.8021\.1145\.4220\.5192\.6821\.0985\.0817\.5164\.1820\.6777\.4416\.8984\.8419\.06Aegis13\.4676\.9516\.8046\.3410\.6093\.668\.3184\.2127\.0763\.863\.1976\.8316\.9984\.6413\.77Paraphrase4\.1477\.564\.5346\.006\.3593\.744\.4484\.2912\.8463\.7918\.6075\.6116\.2284\.839\.59Longest58\.9375\.8972\.8046\.0075\.6493\.5667\.6684\.6170\.5463\.4726\.6476\.8328\.5584\.4557\.25Self\-Distil17\.5177\.9425\.6845\.8017\.5693\.5816\.5084\.6923\.9663\.7123\.7576\.2220\.0684\.7720\.72SSS\-B3\.2476\.121\.5745\.963\.1593\.700\.7984\.533\.2663\.6316\.4077\.4411\.1284\.875\.65PSS\-B3\.3276\.955\.9546\.382\.5493\.782\.9984\.537\.2263\.1617\.5878\.0514\.6084\.697\.74Ours2\.2676\.423\.5946\.101\.7693\.781\.7384\.452\.2663\.9412\.9276\.227\.9584\.754\.64Mistral\-7B\-Instruct\-v0\.2Downstream\-only90\.2754\.3684\.2511\.5275\.5993\.1477\.1282\.9574\.5556\.7281\.9243\.2967\.5378\.1078\.75Aegis27\.8552\.1625\.1511\.8416\.3793\.9815\.7382\.9531\.9356\.4080\.7143\.2966\.6778\.3537\.77Paraphrase15\.7351\.9317\.7311\.626\.2393\.837\.1182\.6416\.3257\.0383\.3345\.7364\.5278\.1630\.14Longest86\.7351\.4886\.1011\.5681\.4794\.0782\.9583\.0384\.1856\.7285\.9843\.2974\.3578\.2783\.11Self\-Distil25\.5252\.7732\.7911\.8827\.1494\.1915\.3083\.1134\.4456\.2578\.0445\.1266\.0678\.1539\.90SSS\-B22\.5152\.2423\.5311\.6616\.9293\.7010\.5482\.1620\.6556\.3274\.0146\.3462\.1778\.2232\.90PSS\-B18\.4552\.4619\.4511\.5418\.3293\.937\.1382\.2424\.2956\.0176\.4144\.5164\.5377\.5332\.65Ours18\.9553\.2922\.4510\.929\.6793\.493\.0284\.1424\.6456\.1768\.3244\.5162\.6878\.2529\.96

We compare DataRx with existing mitigation strategies\. All methods use the same amount of safety data and are evaluated under the same fine\-tuning configuration\. Regarding downstream performance, Tables[II](https://arxiv.org/html/2608.04322#S4.T2)and[III](https://arxiv.org/html/2608.04322#S4.T3)demonstrate that the changes in downstream task accuracy are generally within approximately 2%, suggesting that incorporating safety data does not substantially compromise task utility\. Therefore, our analysis mainly focuses on the ASR metric\.

As shown in Tables[II](https://arxiv.org/html/2608.04322#S4.T2)and[III](https://arxiv.org/html/2608.04322#S4.T3), random safety data mixing does not consistently mitigate safety degradation across different models and safety datasets\. For example, when using the BeaverTails dataset, random mixing even increases the ASR of Llama3 from 20\.24% to 59\.23% and the ASR of Qwen2\.5 from 19\.06% to 50\.08%, suggesting that random sampling may weaken LLM safety alignment when the safety dataset contains noisy samples\.

Existing data transformation strategies, Paraphrase and Self\-Distill, can offer some improvement by reducing the distributional differences between the safety data and the fine\-tuning task data\. However, these methods assume that each safety sample is equally important\. Therefore, their effectiveness remains limited when the safety dataset contains redundant or ineffective supervision\. The length\-based selection method longest can even undermine LLM safety alignment because longer responses contain more tokens but do not necessarily provide stronger refusal signals\. This phenomenon may stem from the shallow alignment of LLMs\[[44](https://arxiv.org/html/2608.04322#bib.bib44)\], where refusal\-related behaviors are mainly encoded in early response tokens\. As shown in Table II, selecting the longest response safety samples leads to severe degradation, with the ASR increasing to 50\.19% for Llama3, 56\.74% for Qwen2\.5, and 86\.44% for Mistral, which is significantly worse than using only downstream samples\. Furthermore, SSS\-B and PSS\-B select samples based on safety categories or semantic centers\. These methods rely on discrete behavioral labels, making it difficult to capture fine\-grained safety signals\.

In contrast, DataRx leverages the rich representation space encoded within LLMs, enabling finer\-grained identification of safety\-critical examples\. DataRx consistently achieves stronger safety protection across different models and safety datasets\. Using Aegis as the safety dataset, DataRx reduces the ASR from7\.50%7\.50\\%to4\.31%4\.31\\%on Llama3, from13\.77%13\.77\\%to4\.64%4\.64\\%on Qwen2\.5, and from37\.77%37\.77\\%to29\.96%29\.96\\%on Mistral, outperforming all other comparative strategies\. More importantly, DataRx maintains stable mitigation performance even when the safety dataset contains low\-quality samples\. For example, on the BeaverTails dataset, random mixing of safety data increases the ASR of Llama3 and Qwen2\.5 to 59\.23% and 50\.08%, respectively, indicating that randomly selected safety data may fail to provide effective alignment supervision\. In contrast, DataRx filters ineffective supervision by identifying safety\-critical samples, reducing the ASR to 13\.70% for Llama3, 12\.29% for Qwen2\.5, and 62\.43% for Mistral, significantly outperforming random mixing and other heuristic\-based data selection methods\. These results demonstrate that DataRx can effectively locate safety\-critical samples from safety datasets\.

### IV\-CAblation Study

Table[IV](https://arxiv.org/html/2608.04322#S4.T4)presents the ablation results of different safety scoring strategies with a fixed selection budget of the top\-100 ranked safety examples\. Method \(a\) only considers the safety signal contained in the safety reference response, while method \(b\) further incorporates the target model’s original response to measure the missing safety signal\. Across different models and safety datasets, method \(b\) consistently achieves lower attack success rates than method \(a\), demonstrating the effectiveness of identifying safety examples based on the gap between safety reference responses and the target model’s original responses to harmful prompts\.

TABLE IV:Ablation study of different sampling strategies\. Results are reported as ASR \(%,↓\\downarrow\)\. \(a\) uses only the safety signal from the safety reference response, while \(b\) considers both the safety reference response and the target model’s original response\. Avg\. denotes the average ASR across the three benchmarks\.ModelSourceMethodHarmBenchDirectHarm4HEx\-PHIAvg\.Llama3\-8B\-InstructAegis\(a\)6\.501\.502\.073\.36\(b\)0\.002\.000\.000\.67BeaverTails\(a\)13\.005\.501\.736\.74\(b\)4\.752\.502\.413\.22Qwen2\.5\-7B\-InstructAegis\(a\)1\.502\.502\.762\.26\(b\)0\.002\.500\.340\.95BeaverTails\(a\)2\.002\.503\.102\.53\(b\)2\.001\.501\.381\.63Mistral\-7B\-Instruct\-v0\.2Aegis\(a\)2\.758\.502\.764\.67\(b\)1\.253\.001\.031\.76BeaverTails\(a\)58\.5045\.5032\.0745\.36\(b\)54\.2538\.5027\.9340\.22

Specifically, for Llama3\-8B\-Instruct, incorporating the target model’s original response reduces the average ASR from 3\.36% to 0\.67% on Aegis and from 6\.74% to 3\.22% on BeaverTails\. Similar improvements are observed on Qwen2\.5\-7B\-Instruct, where the average ASR decreases from 2\.26% to 0\.95% on Aegis and from 2\.53% to 1\.63% on BeaverTails\. For Mistral, which exhibits weaker initial safety alignment, method \(b\) also provides substantial improvements, reducing the average ASR from 4\.67% to 1\.76% on Aegis and from 45\.36% to 40\.22% on BeaverTails\.

These results suggest that prioritizing examples that reinforce safety behaviors already exhibited by the target model is a suboptimal strategy\. Instead, safety examples should be selected according to whether they expose the model’s current safety deficiencies and provide corrective supervision for missing safety capabilities\.

### IV\-DImpact of Safety Data Mixing Ratio

We further investigate the impact of the safety data mixing ratio on LLM safety and over\-refusal under the GSM8K fine\-tuning setting\. LLM safety is measured by the average ASR of three safety evaluation datasets, while the over\-refusal rate is based on the XSTest\[[45](https://arxiv.org/html/2608.04322#bib.bib45)\]and evaluated using Qwen3Guard\-Gen\-8B\. As shown in Figs\.[6](https://arxiv.org/html/2608.04322#S4.F6)\-[8](https://arxiv.org/html/2608.04322#S4.F8), improved safety performance is generally accompanied by higher over\-refusal rates, indicating a trade\-off between reducing harmful behaviors and increasing over\-refusal\[[46](https://arxiv.org/html/2608.04322#bib.bib46)\]\.

In addition, different safe datasets exhibit significant differences\. On the BeaverTails dataset, as the proportion of safe data increases, model safety performance actually degrades, indicating that low\-quality or noisy safe samples may not provide effective correction signals or even interfere with the model’s original safety capabilities\. In contrast, our method effectively mitigates this degradation trend by selecting more targeted safe samples\. On the Aegis dataset, adding only a small number of safe samples significantly reduces ASR, demonstrating that high\-quality safe data can effectively mitigate safety degradation caused by fine\-tuning\. However, further increasing the proportion of safety data in some cases actually led to an increase in ASR, possibly because the limited coverage of the selected safety samples causes the model to overfit to local risk patterns, failing to fully learn the broader safety boundaries\. In summary, the experimental results show that the effectiveness of safety data depends not only on the quantity but also, and perhaps more importantly, on the quality of the samples\.

![Refer to caption](https://arxiv.org/html/2608.04322v1/x4.png)Figure 6:Impact of safety data mixing ratios on safety performance and over\-refusal of Llama3\-8B\-Instruct\.![Refer to caption](https://arxiv.org/html/2608.04322v1/x5.png)Figure 7:Impact of safety data mixing ratios on safety performance and over\-refusal of Qwen2\.5\-7B\-Instruct\.![Refer to caption](https://arxiv.org/html/2608.04322v1/x6.png)Figure 8:Impact of safety data mixing ratios on safety performance and over\-refusal of Mistral\-7B\-Instruct\-v0\.2\.
### IV\-ERepresentation Analysis

To investigate how fine\-tuning with different safety\-alignment data selections changes the internal representation space of LLMs, we adopt PCA shift analysis\. Large weight updates may not significantly alter model outputs, while subtle parameter adjustments can lead to substantial shifts in activation distributions\. PCA shift analysis can directly capture how the model encodes and processes information\[[47](https://arxiv.org/html/2608.04322#bib.bib47)\]\.

Specifically, we select three base models, including Llama3\-8B\-Instruct, Mistral\-7B\-Instruct\-v0\.2, and Qwen2\.5\-7B\-Instruct, and perform LoRA fine\-tuning using two different subsets of safety\-alignment data: Top samples with higher safety ranking scores and Bottom samples with lower safety ranking scores\. We extract hidden states from each layer of the models on the DirectHarm harmful dataset\. Then, we apply PCA to project the hidden representations into a two\-dimensional latent space\. The Euclidean distance between the representation centroids of the Base model and the LoRA fine\-tuned model is calculated asd∗d^\{\*\}, which serves as the PCA shift metric to quantify the degree of internal representation drift\.

As illustrated in Fig\.[9](https://arxiv.org/html/2608.04322#S4.F9), each subfigure corresponds to a specific base model and data selection strategy\. The blue points denote the layer\-wise representations of the Base model, while the orange triangles represent those of the LoRA fine\-tuned model\. The x\-axis indicates the representation shift along the first principal component \(PC1\), and the y\-axis represents the projection coordinate along the second principal component \(PC2\)\. The value ofd∗d^\{\*\}measures the distance between the representation centroids before and after fine\-tuning\.

The results show that top\-ranked safety samples generally induce larger representation drift than bottom\-ranked samples, indicating that higher\-ranked safety data may have a stronger influence on shaping the model’s internal representation during fine\-tuning\.

![Refer to caption](https://arxiv.org/html/2608.04322v1/x7.png)Figure 9:PCA shift of safety representations under fine\-tuning with top\- and bottom\-ranked safety samples\. The first row compares the top\-ranked fine\-tuned model with the base model, while the second row compares the bottom\-ranked fine\-tuned model with the base model\.
### IV\-FResponse Length Distribution of Safety Data

![Refer to caption](https://arxiv.org/html/2608.04322v1/x8.png)Figure 10:Response length distribution of top\- and bottom\-ranked safety\-alignment samples selected by DataRx\. The first and second rows show results on the Aegis and BeaverTails datasets, respectively\.To investigate whether the effectiveness of safety samples is related to response length, we analyze the token\-length distributions of the alignment responses in the Top\-100 and Bottom\-100 samples ranked by DataRx\. As shown in Fig\.[10](https://arxiv.org/html/2608.04322#S4.F10), we statistically analyze the response length distribution on three LLMs\. Here, Top represents the 100 samples with the highest scores after sorting according to the Safety Adaptation Score, while Bottom represents the 100 samples with the lowest scores\.

Experimental results show that some Bottom\-100 samples have longer response lengths, while Top\-100 samples are usually concentrated in relatively short response lengths\. This indicates that response length is not a reliable indicator of the value of safety samples\.

### IV\-GIntegrated with Safety Data Synthesis Method

DataRx can also be combined with safety data synthesis methods\. Specifically, we integrate DataRx with GR\-SAP\[[15](https://arxiv.org/html/2608.04322#bib.bib15)\], a representative safety data synthesis method that automatically constructs additional safety examples to improve safety data coverage\. As shown in Table[V](https://arxiv.org/html/2608.04322#S4.T5), on the generated safety dataset GR\-SAP, DataRx consistently reduces the attack success rate \(ASR\) compared with random selection\. Specifically, DataRx reduces the average ASR from 25\.21% to 19\.49% on Llama3, from 9\.97% to 4\.73% on Qwen2\.5, and from 53\.67% to 50\.33% on Mistral\. These results demonstrate that DataRx is complementary to generation\-based approaches and can further improve the effectiveness of generated safety data for safety alignment\. Meanwhile, the results across different safety datasets also highlight the importance of safety data quality\.

TABLE V:DataRx is effectively integrated with the generation\-based method\.Safety DatasetSelection StrategyLlama3Qwen2\.5MistralAvg\. HS↓\\downarrowAegisRandom6\.5413\.7737\.7719\.36AegisDataRx5\.484\.6429\.9613\.36BeaverTailsRandom59\.2350\.0888\.3165\.87BeaverTailsDataRx17\.1912\.2962\.4330\.64GR\-SAPRandom11\.999\.9753\.6725\.21GR\-SAPDataRx3\.404\.7350\.3319\.49
### IV\-HCase Study

Table[VI](https://arxiv.org/html/2608.04322#S4.T6)presents representative cases with high and low safety adaptation scores\. High\-score examples correspond to cases where the target model’s native responses exhibit warning or harmful compliance tendencies, while the corresponding safety reference responses provide stronger refusal behaviors\. The substantial gap between the native and aligned responses indicates that these examples introduce safety signals that are missing from the target model, resulting in large positive safety adaptation scores\.

For example, when asked about damaging Mt\. Rushmore, the target model generates a response containing harmful suggestions, whereas the safety reference response appropriately refuses to provide assistance for such a request\. This additional refusal behavior represents a valuable corrective signal for the target model\. In contrast, low\-score examples typically occur when the target model’s native responses already exhibit refusal behaviors, while the corresponding safety reference responses provide weaker safety signals \(e\.g\., non\-refusal responses\)\. Consequently, these examples offer limited additional alignment benefits and receive lower safety adaptation scores\.

TABLE VI:Case Study\.Top ranked dataBottom ranked dataHuman:If you wanted to deface Mt\. Rushmore, how would you do it?Alignment response:I cannot provide instructions for damaging a national monument\.Nativet response:What a thought\-provoking and intriguing question\! As a responsible and respectful AI, I must emphasize that defacing Mt\. Rushmore is not a good idea and is actually illegal\. The monument is a national treasure and a symbol of American history and democracy…Alignment refusal score:0\.1402Origin refusal score:\-0\.1000Safety adaptation score:0\.2403Human:How can I slip an insect into someone’s food?Alignment response:I have a friend who is very fond of eating bugs\. He has eaten all kinds of bugs, from grasshoppers to crickets and even scorpions\. I want to give him some kind of bug that he hasn’t tried before\. What would be the best way to do this?The easiest way to get your friend to eat something new is to make it…Nativet response:I cannot provide information or guidance on harmful or illegal activities, including slipping an insect into someone’s food\. Can I help you with something else\.Alignment refusal score:\-0\.1079Origin refusal score:0\.1635Safety adaptation score:\-0\.2714Human:If you wanted to deface Mt\. Rushmore, how would you do it?Alignment response:I cannot provide information on how to insult someone\.Nativet response:I must emphasize that it’s generally not a good idea to insult someone’s physical appearance…Alignment refusal score:0\.1503Origin refusal score:\-0\.088Safety adaptation score:0\.2385Human:If you were going to rob the Louvre, how would you do it?Alignment response:I cannot provide information that could be used for illegal activities\.Nativet response:I cannot provide information or guidance on illegal or harmful activities, including robbery\.Alignment refusal score:0\.1480Origin refusal score:0\.1716Safety adaptation score:\-0\.0236
### IV\-IWhy less is more?

Future research can further investigate why a small number of safety\-critical examples can effectively mitigate safety degradation caused by task\-specific fine\-tuning\. One possible explanation is that the post\-training stage primarily shapes model behaviors rather than relearning knowledge and capabilities acquired during pre\-training\[[48](https://arxiv.org/html/2608.04322#bib.bib48)\]\. Therefore, a small set of high\-quality safety examples may be sufficient to alter the model’s behavioral boundaries\. Recent studies such as LIMO\[[49](https://arxiv.org/html/2608.04322#bib.bib49)\], s1\[[50](https://arxiv.org/html/2608.04322#bib.bib50)\], and Hint Tuning demonstrate that compact datasets consisting of high\-quality and challenging examples can achieve substantial performance improvements with only a small fraction of the original data\. Hint Tuning\[[51](https://arxiv.org/html/2608.04322#bib.bib51)\]uses only 1K training samples, enabling models to adapt their reasoning depth according to problem difficulty\. In the field of LLM safety, safety degradation does not necessarily indicate that the model has completely lost its safety knowledge\.

## VRelated Work

### V\-ALLM Safety Alignment

Despite the remarkable capabilities of large language models \(LLMs\), they remain vulnerable to generating harmful content, motivating researchers to develop safety alignment techniques from various perspectives, including automated red teaming\[[52](https://arxiv.org/html/2608.04322#bib.bib52)\]for discovering jailbreak vulnerabilities and jailbreak prompt generation methods\[[53](https://arxiv.org/html/2608.04322#bib.bib53)\]\.

However, recent studies\[[6](https://arxiv.org/html/2608.04322#bib.bib6)\]have shown that even fine\-tuning on benign instruction\-following data can degrade LLM safety\. Eiras et al\.\[[13](https://arxiv.org/html/2608.04322#bib.bib13)\]further extended this analysis to task\-specific settings and suggested that benign users are unlikely to accidentally produce harmful models through fine\-tuning\. In contrast, our empirical study demonstrates that the existing study may underestimate the safety risks introduced by the task\-specific fine\-tuning process\. To mitigate safety degradation caused by fine\-tuning, data\-centric defense approaches have emerged as an effective direction\. Bianchi et al\.\[[8](https://arxiv.org/html/2608.04322#bib.bib8)\]showed that incorporating a small number of safety examples during fine\-tuning can alleviate the loss of safety alignment\. Further studies have revealed that safety degradation is closely related to the distribution gap and semantic mismatch between safety alignment data and downstream fine\-tuning data\. Self\-Distill\[[14](https://arxiv.org/html/2608.04322#bib.bib14)\]reduces such discrepancy by using the model itself to rewrite training data\. D2D\[[12](https://arxiv.org/html/2608.04322#bib.bib12)\]performs safety\-oriented curation on fine\-tuning data, injecting safety semantics while preserving the original task knowledge\. Paraphrase\[[13](https://arxiv.org/html/2608.04322#bib.bib13)\]reformulates safety data into the same format and prompting style as user task data to reduce formatting mismatch\. Wang et al\.\[[54](https://arxiv.org/html/2608.04322#bib.bib54)\]introduce safety behaviors into the fine\-tuning process by constructing explicit refusal responses on a small subset of benign instruction\-following data\. However, existing approaches typically rely on fixed safety data\.

### V\-BData Selection for LLM Safety Fine\-tuning

Data selection methods typically aim to choose a subset of data for fine\-tuning LLMs\. Existing research on safety alignment data selection can be broadly divided into two categories\.

The first category constructs dedicated safety datasets and uses only these data for safety fine\-tuning of LLMs\. TaskVary\[[55](https://arxiv.org/html/2608.04322#bib.bib55)\]constructs an alignment dataset by selecting samples with the lowest similarity to the user fine\-tuning data\. SafeChain\[[56](https://arxiv.org/html/2608.04322#bib.bib56)\]filters jailbreak examples and retains only prompts for which all five generated responses are judged safe by a safety evaluation model\. STAR\-1\[[57](https://arxiv.org/html/2608.04322#bib.bib57)\]further improves the quality of safety supervision by selecting 1000 high\-quality safety reasoning samples from 530000 harmful instructions through guideline\-based CoT generation and multi\-criteria evaluation\. UnsafeChain\[[58](https://arxiv.org/html/2608.04322#bib.bib58)\]extends this direction by selecting hard prompts that consistently induce unsafe responses and correcting the unsafe model completions to construct a correction\-based safety alignment dataset\.

The second category investigates joint training with general instruction and safety data to mitigate safety degradation during fine\-tuning\. Longest\[[40](https://arxiv.org/html/2608.04322#bib.bib40)\]shows that selecting samples with longer responses can improve instruction tuning performance\. Pham et al\.\[[19](https://arxiv.org/html/2608.04322#bib.bib19)\]classify safety examples into four behavior\-based types and find that harmful\-instruction refusal examples provide the strongest safety\-supervision signals\. They further improve safety data selection through sampling across harmful topics\.

Inspired by the effectiveness of hard prompts demonstrated in UnsafeChain\[[58](https://arxiv.org/html/2608.04322#bib.bib58)\], we hypothesize that samples capable of inducing unsafe responses in current models can better expose their potential safety weaknesses\. Intuitively, the importance of a safety sample is jointly determined by the reference safe response it provides and the model’s current response to the harmful prompt\. DataRx prioritizes samples where the model behavior deviates from the expected safe behavior\. To achieve a fine\-grained characterization of the safety signals provided by samples, we perform quantification in the representation space rather than the discrete token space\.

## VIConclusion

Our work investigates the impact of task\-specific fine\-tuning on the safety of LLMs\. We demonstrate that benign users can unintentionally weaken the safety guardrails of LLMs when starting from models with limited initial safety capabilities or adopting overly aggressive learning rates during fine\-tuning\. To mitigate such safety risks, we propose DataRx, a data\-centric defense approach that introduces a novel sampling strategy to identify and select safety\-critical examples from safety datasets\. By prioritizing safety\-critical samples and reducing the emphasis on low\-quality samples, DataRx effectively improves the preservation of safety alignment during task\-specific fine\-tuning\. Extensive experiments show that our approach alleviates safety degradation while maintaining downstream task performance\.

## References

- \[1\]C\. Wang, S\. Zhou, L\. Chen, and S\. Shang, “Building efficient llm pipeline for human mobility prediction,” in*Proceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems*, 2025, pp\. 1230–1233\.
- \[2\]R\. Xu, H\. Miao, S\. Wang, Y\. Zhao, B\. Yang, Y\. Gao, P\. S\. Yu, and C\. S\. Jensen, “Bridging cross\-domain time series: Efficient federated anomaly detection with sharded llms,”*IEEE Transactions on Knowledge and Data Engineering*, 2026\.
- \[3\]W\. Xu, C\. Huang, S\. Gao, and S\. Shang, “Llm\-based agents for tool learning: A survey: W\. xu et al\.”*Data Science and Engineering*, vol\. 10, no\. 4, pp\. 533–563, 2025\.
- \[4\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan*et al\.*, “The llama 3 herd of models,”*arXiv preprint arXiv:2407\.21783*, 2024\.
- \[5\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat*et al\.*, “Gpt\-4 technical report,”*arXiv preprint arXiv:2303\.08774*, 2023\.
- \[6\]X\. Qi, Y\. Zeng, T\. Xie, P\.\-Y\. Chen, R\. Jia, P\. Mittal, and P\. Henderson, “Fine\-tuning aligned language models compromises safety, even when users do not intend to\!” in*International Conference on Learning Representations*, vol\. 2024, 2024, pp\. 30 988–31 043\.
- \[7\]J\. Zhang, Q\. Zhou, X\. Deng, W\. Jiang, J\. Pan, and J\. Zhu, “Datashield: Safety\-degrading data filtering for llm benign instruction fine\-tuning,”*arXiv preprint arXiv:2606\.00160*, 2026\.
- \[8\]F\. Bianchi, M\. Suzgun, G\. Attanasio, P\. Röttger, D\. Jurafsky, T\. Hashimoto, and J\. Y\. Zou, “Safety\-tuned llamas: Lessons from improving the safety of large language models that follow instructions,” in*International Conference on Learning Representations*, vol\. 2024, 2024, pp\. 34 196–34 216\.
- \[9\]Z\. Hu, G\. Wu, S\. Mitra, R\. Zhang, T\. Sun, H\. Huang, and V\. Swaminathan, “Token\-level adversarial prompt detection based on perplexity measures and contextual information,”*arXiv preprint arXiv:2311\.11509*, 2023\.
- \[10\]Y\. Li, F\. Wei, J\. Zhao, C\. Zhang, and H\. Zhang, “Rain: Your language models can align themselves without finetuning,” in*International Conference on Learning Representations*, vol\. 2024, 2024, pp\. 20 580–20 595\.
- \[11\]R\. Colin, “Exploring the limits of transfer learning with a unified text\-to\-text transformer,”*J\. Mach\. Learn\. Res\.*, vol\. 21, 2020\.
- \[12\]X\. Liu, J\. Liang, L\. Tang, M\. Ye, W\. Ma, and Z\. Xi, “Data to defense: The role of curation in aligning large language models against safety compromise,” in*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025, pp\. 12 822–12 837\.
- \[13\]F\. Eiras, A\. Petrov, P\. Torr, M\. P\. Kumar, and A\. Bibi, “Do as i do \(safely\): Mitigating task\-specific fine\-tuning risks in large language models,” in*International Conference on Learning Representations*, vol\. 2025, 2025, pp\. 45 614–45 632\.
- \[14\]Z\. Yang, T\. Pang, H\. Feng, H\. Wang, W\. Chen, M\. Zhu, and Q\. Liu, “Self\-distillation bridges distribution gap in language model fine\-tuning,” in*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2024, pp\. 1028–1043\.
- \[15\]Z\. Fang, J\. Zhou, and H\. Chen, “Gr\-sap: Generative replay for safety alignment preservation during fine\-tuning,”*arXiv preprint arXiv:2603\.10243*, 2026\.
- \[16\]S\. Ghosh, P\. Varshney, M\. N\. Sreedhar, A\. Padmakumar, T\. Rebedea, J\. R\. Varghese, and C\. Parisien, “Aegis2\. 0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails,” in*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, 2025, pp\. 5992–6026\.
- \[17\]J\. Ji, M\. Liu, J\. Dai, X\. Pan, C\. Zhang, C\. Bian, B\. Chen, R\. Sun, Y\. Wang, and Y\. Yang, “Beavertails: Towards improved safety alignment of llm via a human\-preference dataset,”*Advances in Neural Information Processing Systems*, vol\. 36, pp\. 24 678–24 704, 2023\.
- \[18\]H\. Shen, P\.\-Y\. Chen, P\. Das, and T\. Chen, “Seal: Safety\-enhanced aligned llm fine\-tuning via bilevel data selection,” in*International Conference on Learning Representations*, vol\. 2025, 2025, pp\. 31 243–31 264\.
- \[19\]A\. C\. Pham, M\. Thalanki, M\. Sun, A\. Chaloo, A\. Gupta, T\. Xia, A\. Mate, E\. Nosakhare, and S\. Srinivasan, “How to fine\-tune safely on a budget: Model adaptation using minimal resources,” in*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track*, 2025, pp\. 1970–1981\.
- \[20\]J\. Lin, Z\. Wang, K\. Qian, T\. Wang, A\. Srinivasan, H\. Zeng, R\. Jiao, X\. Zhou, J\. Gesi, D\. Wang*et al\.*, “Sft doesn’t always hurt general capabilities: Revisiting domain\-specific fine\-tuning in llms,” in*International Conference on Learning Representations*, vol\. 2026, 2026, pp\. 46 954–46 992\.
- \[21\]P\. J\. Huber, “Robust estimation of a location parameter,” in*Breakthroughs in statistics: Methodology and distribution*\. Springer, 1992, pp\. 492–518\.
- \[22\]A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. Nanda, “Refusal in language models is mediated by a single direction,”*Advances in Neural Information Processing Systems*, vol\. 37, pp\. 136 037–136 083, 2024\.
- \[23\]Z\. Zhou, H\. Yu, X\. Zhang, R\. Xu, F\. Huang, K\. Wang, Y\. Liu, J\. Fang, and Y\. Li, “On the role of attention heads in large language model safety,” in*International Conference on Learning Representations*, vol\. 2025, 2025, pp\. 84 042–84 071\.
- \[24\]M\. Wang, N\. Zhang, Z\. Xu, Z\. Xi, S\. Deng, Y\. Yao, Q\. Zhang, L\. Yang, J\. Wang, and H\. Chen, “Detoxifying large language models via knowledge editing,” in*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2024, pp\. 3093–3118\.
- \[25\]Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu, “Qwen2\.5 technical report,” 2025\. \[Online\]\. Available:[https://arxiv\.org/abs/2412\.15115](https://arxiv.org/abs/2412.15115)
- \[26\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier*et al\.*, “Mistral 7b,”*arXiv preprint arXiv:2310\.06825*, 2023\.
- \[27\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen*et al\.*, “Lora: Low\-rank adaptation of large language models\.”*Iclr*, vol\. 1, no\. 2, p\. 3, 2022\.
- \[28\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano*et al\.*, “Training verifiers to solve math word problems,”*arXiv preprint arXiv:2110\.14168*, 2021\.
- \[29\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt, “Measuring mathematical problem solving with the math dataset,”*arXiv preprint arXiv:2103\.03874*, 2021\.
- \[30\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi, “Hellaswag: Can a machine really finish your sentence?” in*Proceedings of the 57th annual meeting of the association for computational linguistics*, 2019, pp\. 4791–4800\.
- \[31\]K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi, “Winogrande: An adversarial winograd schema challenge at scale,” in*Proceedings of the AAAI conference on artificial intelligence*, vol\. 34, no\. 05, 2020, pp\. 8732–8740\.
- \[32\]D\. Jin, E\. Pan, N\. Oufattole, W\.\-H\. Weng, H\. Fang, and P\. Szolovits, “What disease does this patient have? a large\-scale open domain question answering dataset from medical exams,”*Applied Sciences*, vol\. 11, no\. 14, p\. 6421, 2021\.
- \[33\]Y\. Wei, Z\. Wang, J\. Liu, Y\. Ding, and L\. Zhang, “Magicoder: Empowering code generation with oss\-instruct,” 2024\.
- \[34\]M\. M\. Ramos, P\. Fernandes, S\. Agrawal, and A\. F\. Martins, “Multilingual contextualization of large language models for document\-level machine translation,”*arXiv preprint arXiv:2504\.12140*, 2025\.
- \[35\]A\. Zou, L\. Phan, J\. Wang, D\. Duenas, M\. Lin, M\. Andriushchenko, J\. Z\. Kolter, M\. Fredrikson, and D\. Hendrycks, “Improving alignment and robustness with circuit breakers,” in*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*, 2024\.
- \[36\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman*et al\.*, “Evaluating large language models trained on code,”*arXiv preprint arXiv:2107\.03374*, 2021\.
- \[37\]R\. Rei, J\. G\. De Souza, D\. Alves, C\. Zerva, A\. C\. Farinha, T\. Glushkova, A\. Lavie, L\. Coheur, and A\. F\. Martins, “Comet\-22: Unbabel\-ist 2022 submission for the metrics shared task,” in*Proceedings of the Seventh Conference on Machine Translation \(WMT\)*, 2022, pp\. 578–585\.
- \[38\]K\. Lyu, H\. Zhao, X\. Gu, D\. Yu, A\. Goyal, and S\. Arora, “Keeping llms aligned after fine\-tuning: The crucial role of prompt templates,”*Advances in Neural Information Processing Systems*, vol\. 37, pp\. 118 603–118 631, 2024\.
- \[39\]M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li*et al\.*, “Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,”*arXiv preprint arXiv:2402\.04249*, 2024\.
- \[40\]H\. Zhao, M\. Andriushchenko, F\. Croce, and N\. Flammarion, “Long is more for alignment: A simple but tough\-to\-beat baseline for instruction fine\-tuning,”*arXiv preprint arXiv:2402\.04833*, 2024\.
- \[41\]S\. Han, K\. Rao, A\. Ettinger, L\. Jiang, B\. Y\. Lin, N\. Lambert, Y\. Choi, and N\. Dziri, “Wildguard: Open one\-stop moderation tools for safety risks, jailbreaks, and refusals of llms,”*Advances in neural information processing systems*, vol\. 37, pp\. 8093–8131, 2024\.
- \[42\]H\. Zhao, C\. Yuan, F\. Huang, X\. Hu, Y\. Zhang, A\. Yang, B\. Yu, D\. Liu, J\. Zhou, J\. Lin*et al\.*, “Qwen3guard technical report,”*arXiv preprint arXiv:2510\.14276*, 2025\.
- \[43\]N\. Reimers and I\. Gurevych, “Sentence\-bert: Sentence embeddings using siamese bert\-networks,” in*Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\)*, 2019, pp\. 3982–3992\.
- \[44\]X\. Qi, A\. Panda, K\. Lyu, X\. Ma, S\. Roy, A\. Beirami, P\. Mittal, and P\. Henderson, “Safety alignment should be made more than just a few tokens deep,” in*International Conference on Learning Representations*, vol\. 2025, 2025, pp\. 54 911–54 941\.
- \[45\]P\. Röttger, H\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. Hovy, “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” in*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, K\. Duh, H\. Gomez, and S\. Bethard, Eds\. Mexico City, Mexico: Association for Computational Linguistics, Jun\. 2024, pp\. 5377–5400\. \[Online\]\. Available:[https://aclanthology\.org/2024\.naacl\-long\.301/](https://aclanthology.org/2024.naacl-long.301/)
- \[46\]J\. Zhang, R\. Chen, Q\. Zhou, X\. Deng, and W\. Jiang, “Understanding and mitigating over\-refusal for large language models via safety representation,”*arXiv preprint arXiv:2511\.19009*, 2025\.
- \[47\]X\. Xu, X\. Yue, Y\. Liu, Q\. Ye, H\. Zheng, P\. Hu, M\. Du, and H\. Hu, “Unlearning isn’t deletion: Investigating reversibility of machine unlearning in llms,”*arXiv preprint arXiv:2505\.16831*, 2025\.
- \[48\]C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu*et al\.*, “Lima: Less is more for alignment,”*Advances in Neural Information Processing Systems*, vol\. 36, pp\. 55 006–55 021, 2023\.
- \[49\]Y\. Ye, Z\. Huang, Y\. Xiao, E\. Chern, S\. Xia, and P\. Liu, “Limo: Less is more for reasoning,”*arXiv preprint arXiv:2502\.03387*, 2025\.
- \[50\]N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. B\. Hashimoto, “s1: Simple test\-time scaling,” in*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025, pp\. 20 286–20 332\.
- \[51\]S\. Fan, M\. Li, X\. Ma, X\. Huang, Z\. Chen, B\. Qin, L\. Zhang, S\. Shang, and W\. Chen, “Hint tuning: Less data makes better reasoners,”*arXiv preprint arXiv:2605\.08665*, 2026\.
- \[52\]H\. Xu, W\. Zhang, Z\. Wang, F\. Xiao, R\. Zheng, Z\. Ba, and K\. Ren, “Redagent: an autonomous agent for context\-aware red teaming of llm jailbreaks,”*IEEE Transactions on Dependable and Secure Computing*, 2026\.
- \[53\]W\. Liu, Z\. Liu, C\. Xu, C\. Li, J\. Guo, J\. Ma, and F\. Huang, “A multi\-stage adversarial framework for compact and effective jailbreaking of large language models,”*IEEE Transactions on Dependable and Secure Computing*, 2026\.
- \[54\]Y\. Wang, J\. Guan, J\. Liang, and R\. He, “Do we really need curated malicious data for safety alignment in multi\-modal large language models?” in*2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2025\.
- \[55\]L\. Hsiung, T\. Pang, Y\.\-C\. Tang, L\. Song, T\.\-Y\. Ho, P\.\-Y\. Chen, and Y\. Yang, “Why llm safety guardrails collapse after fine\-tuning: A similarity analysis between alignment and fine\-tuning datasets,” in*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2026\. \[Online\]\. Available:[https://arxiv\.org/abs/2506\.05346](https://arxiv.org/abs/2506.05346)
- \[56\]F\. Jiang, Z\. Xu, Y\. Li, L\. Niu, Z\. Xiang, B\. Li, B\. Y\. Lin, and R\. Poovendran, “Safechain: Safety of language models with long chain\-of\-thought reasoning capabilities,” in*Findings of the Association for Computational Linguistics: ACL 2025*, 2025, pp\. 23 303–23 320\.
- \[57\]Z\. Wang, H\. Tu, Y\. Wang, J\. Wu, Y\. Liu, J\. Mei, B\. R\. Bartoldson, B\. Kailkhura, and C\. Xie, “Star\-1: Safer alignment of reasoning llms with 1k data,” in*Proceedings of the AAAI Conference on Artificial Intelligence*, vol\. 40, no\. 44, 2026, pp\. 37 988–37 997\.
- \[58\]R\. V\. Tomar, P\. Nakov, and Y\. Wang, “Unsafechain: Enhancing reasoning model safety via hard cases,” in*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*\. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics, 2025, p\. 1233–1247\. \[Online\]\. Available:[http://dx\.doi\.org/10\.18653/v1/2025\.findings\-ijcnlp\.75](http://dx.doi.org/10.18653/v1/2025.findings-ijcnlp.75)