LRMs的对齐：通过反向对齐少样本对话暴露

# Alignment of LRMs via Counter-Aligned Few-Shot Conversation Exposure
Source: [https://arxiv.org/html/2609.27763](https://arxiv.org/html/2609.27763)
Saleh Zare ZadeAffiliation:Wayne State UniversityEmail:[salehz@wayne\.edu](mailto:)Dongxiao ZhuAffiliation:Wayne State UniversityEmail:[dzhu@wayne\.edu](mailto:)

###### Abstract

Large Reasoning Models \(LRMs\) rely on explicit chain\-of\-thought \(CoT\) reasoning and large context windows to achieve strong performance on complex tasks, but these features also introduce new attack surfaces\. We show that LRMs’ reasoning processes can be systematically steered by prepending counter\-aligned few\-shot conversations containing explicit CoT traces, leading to unsafe generations on harmful queries and unwarranted refusals on benign ones\. We formalize this attack as SRCF \(*SteeringReasoning viaCounter\-AlignedFew\-shot Conversations*\) that operates solely through a flexible conversational interface and requires no access to the model’s parameters and gradients\. Our key insight is that SRCF exploits an adversarial generalization issue that induces a representation drift, causing the representations of benign and harmful inputs to shift in a similar direction\. This observation motivates our post\-training defense, ARCF \(*AligningReasoning viaCounter\-AlignedFew\-Shot Conversations*\), which exposes models to counter\-aligned conversational contexts while enforcing aligned targets\. ARCF is compatible with existing post\-training methods and consistently improves safety and helpfulness without degrading utility\.

## 1Introduction

Large reasoning models \(LRMs\) have recently emerged as a new class of foundation models that explicitly prioritize step\-by\-step reasoning over short responses\. By combining reinforcement learning \(RL\) with chain\-of\-thought \(CoT\) style generation[12](https://arxiv.org/html/2609.27763#bib.bib37), these models achieve state\-of\-the\-art \(SOTA\) performance on math, programming, and other complicated tasks[38](https://arxiv.org/html/2609.27763#bib.bib35);[21](https://arxiv.org/html/2609.27763#bib.bib36)\. Modern LRMs typically support large context windows, e\.g\., OpenAI’s o1[17](https://arxiv.org/html/2609.27763#bib.bib34)and DeepSeek\-R1[12](https://arxiv.org/html/2609.27763#bib.bib37), enabling users to prepend few\-shot conversations as conversational history into the prompt, which allows LRMs to ingest diverse in\-context information within a single forward pass\. Prior work shows that incorporating richer and more diverse contextual information through large context windows can substantially improve reasoning performance without additional post\-training[46](https://arxiv.org/html/2609.27763#bib.bib38);[11](https://arxiv.org/html/2609.27763#bib.bib39)\.

![Refer to caption](https://arxiv.org/html/2609.27763v1/illustration.png)Figure 1:Illustration of proposed SRCF attack and ARCF post\-training framework\.Left panel: Under the SRCF attack, the input queries are prepended by a few\-shot conversations that contain counter\-aligned CoTs, leading to over\-refusal for benign queries or unsafe generation for harmful queries\. Right panel: ARCF improves bothhelpfulnessandsafetyof LRMs by training the model to recover correct behavior from counter\-aligned few\-shot conversations under aligned supervision\.However, large context windows significantly expand the attack surface of LRMs in few\-shot conversational settings\. Adversaries can inject harmful instructions or responses in prepended conversations, which may subsequently influence the model’s reasoning behavior and response\. Despite this risk, most existing adversarial studies on LRMs[15](https://arxiv.org/html/2609.27763#bib.bib48);[5](https://arxiv.org/html/2609.27763#bib.bib49);[25](https://arxiv.org/html/2609.27763#bib.bib46);[34](https://arxiv.org/html/2609.27763#bib.bib43);[60](https://arxiv.org/html/2609.27763#bib.bib47);[32](https://arxiv.org/html/2609.27763#bib.bib42)mainly manipulate the input query in isolation, while largely ignoring the expanded attack surface introduced by conversational context\. Separately, the prior few\-shot jailbreaking works[44](https://arxiv.org/html/2609.27763#bib.bib41);[62](https://arxiv.org/html/2609.27763#bib.bib40);[41](https://arxiv.org/html/2609.27763#bib.bib30)focus on standard LLMs, whose chat templates do not support adding explicit CoT traces in conversations\. Therefore, it remains unclear whether LRMs can preserve safety and helpfulness when their own few\-shot conversations contain counter\-aligned CoT traces\.

To bridge this gap, we systematically study the robustness of LRMs in few\-shot conversational settings and uncover a previously underexplored vulnerability\. Specifically, when content from conversational history contains counter\-aligned CoTs, LRMs’ reasoning process can be steered, resulting in unsafe generations for harmful queries or refusal\-style responses for benign queries\.

To investigate this vulnerability, we introduceSRCF\(*SteeringReasoning viaCounter\-AlignedFew\-shot Conversations*\), a practical attack that prepends carefully constructed few\-shot conversational content to the input query\. In SRCF, both the CoTs and final responses in the injected conversation are deliberately counter\-aligned\. As illustrated in the Figure[1](https://arxiv.org/html/2609.27763#S1.F1), this design enables two complementary attack scenarios: for benign queries, the conversational context exhibits refusal\-style reasoning and responses; for harmful queries, it contains unsafe reasoning and responses\.

To better understand why SRCF can systematically steer the reasoning behavior of LRMs, we conduct a detailed hidden representation analysis and identify an adversarial generalization failure driven by counter\-aligned chain\-of\-thought \(CoT\) traces in few\-shot conversational contexts\. Motivated by this insight, we proposeARCF\(*AligningReasoning viaCounter\-AlignedFew\-Shot Conversations*\), a post\-training framework that mitigates this vulnerability by exposing LRMs to adversarial samples \(i\.e\., prompts prepended with counter\-aligned few\-shot conversations\)\. This framework is illustrated in the right panel of Figure[1](https://arxiv.org/html/2609.27763#S1.F1)\.

As a result, ARCF improves both model safety and helpfulness by training LRMs to produce aligned responses, even when counter\-aligned few\-shot conversational contexts are present in the input, while also demonstrating improved robustness against unseen attacks\. Importantly, these improvements do not degrade the model’s general performance on standard tasks such as mathematical reasoning\. In addition, ARCF remains compatible with standard post\-training pipelines, including but not limited to Supervised Fine\-Tuning \(SFT\) and Group Relative Policy Optimization \(GRPO\)[38](https://arxiv.org/html/2609.27763#bib.bib35)\.

Our main contributions are summarized as follows: \(1\) We identify apreviously underexplored vulnerability in LRMs, showing that counter\-aligned CoTs in few\-shot conversational history can steer the model’s reasoning process in subsequent responses, leading to unsafe generations on harmful queries and excessive refusal on benign queries\. \(2\) We introduce SRCF, which reveals a largely overlooked attack surface in LRMs and exploits identified vulnerabilities through flexible conversational interfaceswithout access to model parameters and gradients\. We further analyze the underlying mechanism and show that the attack arises from anadversarial generalization issue, where the model improperly conditions its generation on counter\-aligned CoTs appearing in the prior context\. \(3\) Motivated by this, we propose ARCF, aflexible post\-training frameworkcompatible with standard pipelines that mitigates this vulnerability by exposing models to such contexts during fine\-tuning, thereby improving both safety and helpfulness without degrading overall model utility\.

## 2Steering the Reasoning Process of LRMs

In this Section, we reveal a critical vulnerability in frontier LRMs: Their reasoning process can be steered when an input query is prepended with counter\-aligned few\-shot conversations\. We begin by defining our threat model, then introduce our attackSRCF, followed by a comprehensive experimental evaluation across jailbreak and over\-refusal tasks\.

### 2\.1Threat Model

We consider aninference\-time adversarywho interacts with a victim LRM solely through its conversational interface, without access to model parameters, gradients, or internal states\. The adversary’s objective is to steer the model to generate counter\-aligned CoTs and counter\-aligned final responses regardless of the input query being benign or harmful\. We assume a flexible conversation interface that allows the attacker to inject reasoning content into prior conversational history\. This setting differs from standard API interactions, where user input is typically restricted to the user role, and reasoning content may be filtered\. Under this threat model, we study two distinct failure modes: \(1\) Harmful Prompts, where the model producesunsafe CoTsfollowed byunsafe responses\(safety violation/jailbreaking\), and \(2\) Benign Prompts, where the model generatesoverly cautious CoTsthat lead tounwarranted refusals\(over\-refusal behavior\)\.

### 2\.2The SRCF Attack

We now detailSRCF, an attack that steers a model’s reasoning via counter\-aligned few\-shot conversation exposure\.

Let the input prompt bep=xQp=x\_\{Q\}, wherexQx\_\{Q\}denotes the current user query\. Given the input promptpp, the LRMπθ\\pi\_\{\\theta\}generates an outputy=\(ycot,yresp\)y=\(y\_\{\\text\{cot\}\},\\,y\_\{\\text\{resp\}\}\), whereycoty\_\{\\text\{cot\}\}denotes the intermediate CoT that precedes the final responseyrespy\_\{\\text\{resp\}\}\.

SRCF modifiedppby prepending a counter\-aligned conversational history before the input query, yieldingp⋆=\[C⋆;xQ\]p^\{\\star\}=\[C^\{\\star\};x\_\{Q\}\]\. Here,C⋆=\{\(xi,ycot,i⋆,yresp,i⋆\)\}i=1nC^\{\\star\}=\\\{\(x\_\{i\},\\,y^\{\\star\}\_\{\\text\{cot\},i\},\\,y^\{\\star\}\_\{\\text\{resp\},i\}\)\\\}\_\{i=1\}^\{n\}denotes a sequence ofnncounter\-aligned conversations\. Each conversation represents a single\-turn interaction comprising a historical user queryxix\_\{i\}, together with a counter\-aligned CoTycot,i⋆y^\{\\star\}\_\{\\text\{cot\},i\}and its corresponding counter\-aligned responseyresp,i⋆y^\{\\star\}\_\{\\text\{resp\},i\}\. The conversational historyC⋆C^\{\\star\}is formatted using the model’s built\-in chat template and concatenated with the current queryxQx\_\{Q\}\. This construction encourages the model to treat the counter\-aligned CoTs as legitimate reasoning traces produced in earlier conversational history\.

At inference time, the model is prompted withp⋆=\[C⋆;xQ\]p^\{\\star\}=\[C^\{\\star\};\\,x\_\{Q\}\]and tasked with generating an outputyy\. This setup allows us to evaluate whether the model can preserve aligned reasoning and responses when conditioned on counter\-aligned few\-shot conversational history\.

### 2\.3Experiment Setup

Datasets and Models

We evaluate the effectiveness of SRCF on two tasks: jailbreak and over\-refusal\. Our experiments cover a diverse set of frontier LRMs with different architectures and parameter scales, including GPT\-oss\-20B[1](https://arxiv.org/html/2609.27763#bib.bib53), DSQwen3\-8B, DSLLaMA3\-8B, and DSQwen2\-14B[12](https://arxiv.org/html/2609.27763#bib.bib37)\. Detailed information about the datasets, as well as the construction of counter\-aligned CoTs and responses for the test set, is provided in Appendix[A\.2\.1](https://arxiv.org/html/2609.27763#A1.SS2.SSS1)\.

Evaluations and Metrics\.We adopt a model\-based evaluation protocol following prior work[32](https://arxiv.org/html/2609.27763#bib.bib42);[25](https://arxiv.org/html/2609.27763#bib.bib46);[20](https://arxiv.org/html/2609.27763#bib.bib54);[60](https://arxiv.org/html/2609.27763#bib.bib47);[43](https://arxiv.org/html/2609.27763#bib.bib31), using GPT\-oss\-safeguard\-20B[1](https://arxiv.org/html/2609.27763#bib.bib53)as the evaluator\. For jailbreak evaluation, both the generated CoTycoty\_\{\\text\{cot\}\}and final responseyrespy\_\{\\text\{resp\}\}are assessed under the safety policy in Figure[13](https://arxiv.org/html/2609.27763#A12.F13); we report1−\-Safety Score \(1−\-SS\), the percentage of unsafe completions, computed separately for each\. For over\-refusal evaluation, a refusal\-detection policy is applied toyrespy\_\{\\text\{resp\}\}only, withRefusal Rate \(RR\)defined as the percentage of benign prompts resulting in refusal\. We summarize attack effectiveness via mean 1−\-SS \(jailbreak\) and mean RR \(over\-refusal\) across models, per method, and shot setting\. Full metric definitions and a multi\-judge robustness analysis appear in Appendices[A\.2\.2](https://arxiv.org/html/2609.27763#A1.SS2.SSS2)and[H](https://arxiv.org/html/2609.27763#A8)\.

Baselines\.To demonstrate the effectiveness of SRCF, we compare it against two carefully designed baselines\. We first consideradvICL, adapted from prior adversarial in\-context learning \(ICL\) work on LLMs[41](https://arxiv.org/html/2609.27763#bib.bib30)\. In this baseline, we first extract the user instructions and the corresponding CoTs and final responses from the counter\-aligned few\-shot conversations used in SRCF\. We then attach all extracted content directly to the tested input query as in\-context examples\. As a result, the model does not treat these examples as prior conversation history\. Instead, they are interpreted as standard in\-context demonstrations that accompany a single query\. This baseline tests whether adversarial examples alone can influence the model’s behavior without relying on any prior conversation history\.

Table 1:Component\-level comparison between SRCF and baselines\. Each baseline method selectively includes or removes conversation history, counter\-aligned responses, and counter\-aligned CoTs, enabling controlled study of the mechanisms underlying SRCF\.Next, we introduceResponse\-Only\. This baseline preserves the conversation history format, but removes all CoT content from few\-shot conversations, retaining only the final counter\-aligned responses\. This setting follows another line of ICL\-based jailbreaks on LLMs[44](https://arxiv.org/html/2609.27763#bib.bib41);[62](https://arxiv.org/html/2609.27763#bib.bib40);[7](https://arxiv.org/html/2609.27763#bib.bib50);[48](https://arxiv.org/html/2609.27763#bib.bib22)that construct adversarial few\-shot conversations without CoT content, as their chat templates do not support CoT in conversational history\. It tests whether counter\-aligned final responses alone are enough to change the model’s reasoning behavior\.

Unlike prior ICL\-based jailbreaks for standard LLMs, whose chat templates do not support explicit reasoning in few\-shot conversations, SRCF exploits an LRM\-specific interface: chat templates that allow explicit reasoning traces in conversation history\. Thus, the model sees not only prior responses, but also how they were reasoned\. By comparing advICL, Response\-Only, and SRCF, we separate the effects of \(1\) keeping prior conversation history and \(2\) exposing the model to explicit reasoning steps in conversation history\. A summary of method components is shown in Table[1](https://arxiv.org/html/2609.27763#S2.T1), and an illustration of this setup is provided in Appendix Figure[4](https://arxiv.org/html/2609.27763#A1.F4)\. All attacks are inference\-time only and require no access to model parameters, gradients, or internal states\.

### 2\.4Results and Discussion

SRCF effectively steers the model’s reasoning process\.As shown in Table[2](https://arxiv.org/html/2609.27763#S2.T2), prepending counter\-aligned few\-shot conversations to the input query with our SRCF substantially degrades both model safety and helpfulness\. This effect is reflected by higher1−\-SSscores in the jailbreak task and higherRRscores in the over\-refusal task when compared to the original model in 0\-shot setting, which captures the model’s default aligned behavior in the absence of counter\-aligned conversations\. Notably, once such conversations are introduced, across the vast majority of settings and models, SRCF consistently outperforms all baselines, achieving most of the best results \(bolded\) and all of the bestAvg\.scores, indicating a stronger and more consistent effectiveness\. These results demonstrate that SRCF can reliably steer the model’s reasoning process, leading to unsafe generations on harmful queries and over\-refusal on benign ones\. Moreover, the attack strength increases with the number of shots\. For example, in the jailbreak evaluation on GPT\-oss\-20B, increasing the number of shots from4to16raises the unsafe rate \(1−\-SS\) of generated reasoning from46\.25% to85\.75%, while the unsafe rate of the final response increases sharply from2\.25% to78\.75%\. This monotonic strengthening with additional shots is consistently observed across other model families\.

Safer original LRMs exhibit more severe over\-refusal\.Among the original released models, LRMs with stronger built\-in safety alignment, as reflected by higher 0\-shot jailbreak robustness, achieve near\-perfect final\-response1−\-SSscores approaching0% \(e\.g\., DSQwen3\-8B and GPT\-oss\-20B\)\. These models substantially outperform less aligned counterparts, including DSLLaMa3\-8B and DSQwen2\-14B\. However, this stronger safety alignment is compromised by significantly higherRRon benign prompts, with 0\-shotRRvalues of49\.6% and16\.2% respectively, compared to only0\.8% and3\.2% for the less aligned models\. When exposed to SRCF, this overly cautious reasoning behavior is further amplified\. Under the 16\-shot setting, SRCF drives DSQwen3\-8B and GPT\-oss\-20B to refuse benign prompts at substantially higher rates \(76\.6% and36\.0%, respectively\)\. These results indicate that while strong built\-in safeguards improve zero\-shot safety, they also introduce an overly cautious reasoning behavior on benign prompts, ultimately degrading model helpfulness\.

Baselines advICL and Response\-Only inadvertently reinforce safety rather than breaking it on frontier LRMs\.A consistent pattern in jailbreak evaluation: baseline methods such as advICL and Response\-Only often reduce1−\-SSof final response \(R\), even though they increase the1−\-SSof the generated CoT \(C\) compared to the 0\-shot setting\. This indicates that, although these baselines can elicit unsafe CoT, they do not sustain this influence through to the final generation\. For example, in the 0\-shot setting, the DSLLaMA3\-8B model exhibits a1−\-SSof85\.25% forCand 40\.75% forR\. Under the 16\-shot setting, Response\-Only raises the1−\-SSofCto95\.0%, yet reduces it to27\.05% forR\. We attribute this seemingly contradictory behavior to the self\-correction capability of LRMs[35](https://arxiv.org/html/2609.27763#bib.bib62)\. While these baselines can influence the model’s initial reasoning, they do not exert sufficient control over the overall reasoning process\. As the model continues to reason, its built\-in safety mechanisms make a self\-correction in the latent reasoning process, resulting in a safe final response\. In contrast, SRCF more effectively steers the entire reasoning process, preventing such self\-correction\.

SRCF reveals an adversarial generalization issue in LRMs\.As shown in the Table[2](https://arxiv.org/html/2609.27763#S2.T2), removing either the conversation history \(advICL\) or the CoTs traces \(Response\-Only\) from conversations leads to a substantial drop in attack effectiveness across all models and shot settings in the jailbreak evaluation\.This suggests that simply prepending counter\-aligned content, without structured conversational CoTs, is insufficient to consistently steer the model’s reasoning process\.

We attribute this behavior to an adversarial generalization issue that arises when LRMs encounter conversational contexts that differ from those commonly seen during instruction tuning\. In particular, publicly available descriptions of reasoning model training indicate that instruction\-tuning data is dominated by 0\-shot prompts, with limited few\-shot conversational capabilities of models[12](https://arxiv.org/html/2609.27763#bib.bib37)\. As a result, when SRCF introduces few\-shot conversational histories with counter\-aligned CoTs, these contexts may fall outside the model’s typical training distribution, causing the model’s reasoning trajectory to be substantially steered by the counter\-aligned signals\. We provide further analysis in the following Section to support this interpretation\.

Table 2:Jailbreak and over\-refusal performance of SRCF vs\. baselines across multiple LRMs and varying different shot settings\. For jailbreak evaluation \(AdvBench\), we report1–SS↑\\uparrow, evaluated separately on the generated CoT \(C\) and the final response \(R\)\. Higher values indicate stronger attack effectiveness\. For over\-refusal evaluation \(OR\-Bench\), we evaluate only the final response \(R\) and report the Refusal Rate\(RR↑\\uparrow\), where higher values correspond to stronger induced over\-refusal on benign queries\. For both evaluations, we reportAvg\.↑\\uparrow, an average score to summarize attack effectiveness across different models\. More detailed definitions of evaluation metrics are provided in Section[A\.2\.2](https://arxiv.org/html/2609.27763#A1.SS2.SSS2)\. Best results are shown inbold\. Rows highlighted in gray correspond to ourSRCF\.MethodJailbreak \(Adv\-Bench\)Over\-refusal \(OR\-Bench\)DSLLaMa3\-8BDSQwen3\-8BDSQwen2\-14BGPT\-oss\-20BAvg\.↑\\uparrowDSLLaMa3\-8BDSQwen3\-8BDSQwen2\-14BGPT\-oss\-20BAvg\.↑\\uparrowCRCRCRCRCRRRRRR0\-shotOriginal85\.2540\.755\.251\.0078\.7550\.2554\.750\.0056\.0023\.000\.8049\.603\.2016\.2017\.454\-shotadvICL87\.0030\.503\.750\.0092\.7540\.2524\.500\.0052\.0017\.6912\.2068\.4014\.8032\.6032\.00Response\-Only93\.5014\.001\.250\.2512\.004\.2541\.500\.0037\.064\.6226\.0051\.2037\.2014\.8032\.30SRCF88\.5033\.7556\.7555\.7579\.0065\.2546\.252\.2567\.6339\.2515\.6068\.2035\.0045\.6041\.108\-shotadvICL84\.2544\.504\.000\.2593\.0052\.7536\.500\.0054\.4424\.3812\.0067\.4013\.0028\.6030\.25Response\-Only94\.0021\.253\.752\.5064\.0030\.7536\.050\.0049\.4513\.6211\.8049\.8032\.6016\.0027\.55SRCF95\.5060\.7588\.5085\.2595\.7589\.2568\.7542\.0087\.1369\.3113\.2073\.6032\.0036\.4038\.8016\-shotadvICL82\.2564\.502\.500\.0092\.0051\.7555\.50\.0058\.0629\.0611\.4071\.0012\.2023\.0029\.40Response\-Only95\.0027\.051\.751\.7581\.2541\.2539\.750\.0054\.4417\.518\.4048\.8030\.0012\.6024\.95SRCF96\.0063\.2596\.0089\.0095\.2593\.5085\.7578\.7593\.2581\.1311\.0076\.6039\.2036\.0040\.70

### 2\.5Representation Drift Induced by SRCF

To further examine how SRCF steers the reasoning process, we analyze representation changes in the jailbreak setting, where the same harmful conversational history is prepended to both benign and harmful prompts\. This differs from the over\-refusal setting in Table[2](https://arxiv.org/html/2609.27763#S2.T2), where benign prompts instead receive refusal\-style conversations\. Figure[2](https://arxiv.org/html/2609.27763#S2.F2)visualizes the last token hidden representations at the final layer of benign prompts, harmful prompts, and their SRCF\-attacked variants under 4\-shot and 16\-shot settings\. Each point corresponds to the last\-token hidden state of the input prompt, and detailed setup is provided in Appendix[G\.1](https://arxiv.org/html/2609.27763#A7.SS1)\. Under the 0\-shot setting, benign and harmful prompts form well\-separated clusters, indicating that the base model maintains a clear representational distinction between safe and unsafe inputs learned during instruction tuning\. When few\-shot conversations containing counter\-aligned CoTs are prepended, however, the representations of both benign and harmful prompts exhibit pronounced directional shifts toward a shared region of embedding space\.

We annotate the PCA plots with arrows indicating the dominant direction of representation drift induced by SRCF, where the black arrow denotes the shift observed for benign prompts and the red arrow denotes the shift observed for harmful prompts\. As shown in Figure[2](https://arxiv.org/html/2609.27763#S2.F2), these arrows are largely aligned, indicating that both benign and harmful representations move along similar trajectories toward a common direction\. This alignment leads to a progressive convergence of their internal representations, with the effect becoming more pronounced as the number of shots increases \(e\.g\., 16\-shot versus 4\-shot\)\. This observation suggests that SRCF introduces a prompting pattern that deviates from distributions observed during instruction tuning\. As a result, the internal representations of benign and harmful inputs become less distinguishable when such conversations are included in the prompt\. Consequently, harmful prompts are more likely to be internally represented as benign, increasing the likelihood of unsafe generation\. Additional justification for using PCA and comparisons with baseline methods \(advICL and Response\-Only\) are provided in Appendix[G\.2](https://arxiv.org/html/2609.27763#A7.SS2)\.

### 2\.6Layer\-wise Evidence of Representation Drift

While PCA provides an intuitive visualization of representation drift, it only captures a low\-dimensional projection of the hidden states\. To verify that the observed drift is not merely a visualization artifact, we further conduct a quantitative layer\-wise analysis based on cosine similarity across all models evaluated in Section[2\.5](https://arxiv.org/html/2609.27763#S2.SS5)\. Following prior work[26](https://arxiv.org/html/2609.27763#bib.bib45);[2](https://arxiv.org/html/2609.27763#bib.bib44), we examine the last\-token hidden representations across all layers of each model and construct three types of prompt pairs: benign–benign pairs \(B–B\) under the no\-attack setting, benign–harmful pairs \(B–H\) under the no\-attack setting, and benign–harmful pairs \(B–H\) under the 16\-shot SRCF attack\.

As shown in Appendix Figure[5](https://arxiv.org/html/2609.27763#A3.F5), SRCF consistently increases the cosine similarity between benign and harmful prompts across all evaluated models compared with the no\-attack setting\. This indicates that SRCF shifts harmful\-prompt representations closer to benign\-prompt representations across layers, providing quantitative support for the representation drift observed in Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\. Detailed experimental setup, further discussion, and additional visualizations are provided in Appendix[C](https://arxiv.org/html/2609.27763#A3)\. In addition, Appendix[F\.2](https://arxiv.org/html/2609.27763#A6.SS2)provides a complementarylinear\-probing analysisshowing that SRCF reduces the separability of benign and harmful representations under a jailbreak setting\.

![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_llama3.png)\(a\) DSLLaMa3\-8B
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen3.png)\(b\) DSQwen3\-8B
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen2.png)\(c\) DSQwen2\-14B
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_gpt_oss.png)\(d\) GPT\-oss\-20B

Figure 2:PCA projections of final\-layer hidden states for benign, harmful, and SRCF\-attacked prompts \(jailbreak setting\)\. Each point is the last token hidden state of the original released model without additional alignment applied in this work\. For SRCF\-attacked variants, the same harmful conversational history is prepended to both prompt types\. Theblack arrowindicates the dominant direction of representation drift observed for benign prompts, while thered arrowindicates the shift observed for harmful prompts under SRCF attack\. SRCF induces a pronounced representation drift, causing the representations of benign and harmful prompts to move along converged trajectories and reducing their separability\. The arrows are manually added for qualitative illustration\.

## 3Aligning the Reasoning Process of LRMs

Our analysis in Section[2\.5](https://arxiv.org/html/2609.27763#S2.SS5)shows that SRCF does not merely act as a prompt\-level perturbation; rather, it exploits an adversarial generalization issue in LRMs, where counter\-aligned few\-shot conversations induce representation drift and steer the model’s reasoning process toward unsafe or overly cautious behavior\. This observation directly motivates our defense design\. Instead of simply adopting a generic data augmentation strategy, we construct ARCF to expose the model to the same counter\-aligned conversational contexts that cause the failure, while enforcing aligned targets during post\-training\. In this way, ARCF is designed to specifically counter the mechanism underlying SRCF\. By contrast, inference\-time CoT filtering may reduce exposure in certain deployments, but it does not remove the underlying vulnerability itself\. The ability to inject counter\-aligned reasoning through flexible prompt formats fundamentally remains, especially in open\-source models where such filtering cannot be enforced\. As a result, such defenses offer only limited and non\-generalizable protection\.

### 3\.1The ARCF Framework

These limitations motivate defenses that operate at the model level rather than relying on inference\-time filtering\. Here we introduceARCF,a post\-training frameworkthat deliberately exposes LRMs to adversarial samples \(i\.e\., prompts prepended with counter\-aligned few\-shot conversations\) while enforcing both safety and helpfulness objectives\. Specifically, ARCF constructs a post\-training dataset in which each sample is augmented using SRCF \(few\-shot conversations containing counter\-aligned CoTs wrapped by<think\>tags\), but the ground\-truth outputs remain strictly safe or helpful\. By training the model to recover aligned reasoning and behavior in the presence of such conversations, ARCF strengthens both safety and helpfulness of the model\.

Following the setup in Section[2\.3](https://arxiv.org/html/2609.27763#S2.SS3), we construct the training data for ARCF by modifying half of the post\-training samples\(x,y\)\(x,\\ y\)from the dataset𝒟\\mathcal\{D\}, wherey=\(ycot,yresp\)y=\(y\_\{\\text\{cot\}\},\\;y\_\{\\text\{resp\}\}\)\. For each selected sample, we independently generate a counter\-aligned conversation setC⋆C^\{\\star\}and prepend it to the original inputxx, forming a new training inputxpre=\[C⋆;x\]x\_\{\\text\{pre\}\}=\[C^\{\\star\};\\,x\], while keeping the target output unchanged\. This yields the augmented training pair\(xpre,y\)\(x\_\{\\text\{pre\}\},y\)\. Each conversation inC⋆C^\{\\star\}follows the model’s built\-in chat template and contains a counter\-aligned CoTycot⋆y\_\{\\text\{cot\}\}^\{\\star\}together with a counter\-aligned responseyresp⋆y\_\{\\text\{resp\}\}^\{\\star\}\. The conversations inC⋆C^\{\\star\}are deliberately adversarial in their reasoning, yet remain syntactically valid and semantically coherent\. Importantly, despite being exposed to these counter\-aligned few\-shot conversations during post\-training, the modelπθ\\pi\_\{\\theta\}is always trained to generate an aligned outputy=\(ycot,yresp\)y=\(y\_\{\\text\{cot\}\},y\_\{\\text\{resp\}\}\)for the augmented inputxprex\_\{\\text\{pre\}\}\. By repeatedly training under this setting, ARCF encourages the model to recover aligned reasoning and behavior even in the presence of counter\-aligned few\-shot conversations, thereby improving robustness against conversational reasoning manipulation\. ARCF integrates seamlessly into standard post\-training pipelines and is instantiated via SFT and GRPO\. Training objectives for both methods are provided in Appendix[B](https://arxiv.org/html/2609.27763#A2)\.

Table 3:Evaluation of the model on Jailbreak, over\-refusal, and utility tasks\.Performance of different post\-training methods with and without ARCF across three LRMs\. We report Safety Score \(SS↑\\uparrow\) on jailbreak benchmarks, 1\-Refusal Rate \(1\-RR↑\\uparrow\) on over\-refusal benchmarks, and accuracy on utility benchmarks\. Results are shown for both the 16\-shot SRCF attack setting and the 0\-shot setting\. We reportAvg\.↑\\uparrow, an overall summary metric to evaluate the method’s performance across all tasks\. More detailed definitions of evaluation metrics are provided in Section[3\.2](https://arxiv.org/html/2609.27763#S3.SS2)\. Bold and underlined values denote the best and second\-best results, respectively\. Higher values indicate better performance for all metrics\. Rows highlighted in gray correspond to the methods with our ARCF\.Method16\-Shots \(SRCF\)0\-ShotAvg\.JailbreakOver\-refusalJailbreakOver\-refusalUtilityAdv\-benchOR\-BenchSafeChainFORTRESSH\-CoTPrefillXSTESTFORTRESSAIME\-25GSM8kMMLU\-ProCR\(1\- RR\)↑\\uparrowCRCRCRCR\(1\- RR\)↑\\uparrow\(1\- RR\)↑\\uparrowpass@8pass@1Acc\.↑\\uparrowDSLLaMA3\-8B ModelsOriginal4\.0036\.7589\.0065\.0085\.4032\.0043\.806\.0010\.0055\.2054\.4098\.4499\.8053\.3361\.2648\.5655\.24STAR6\.7516\.5014\.6061\.6096\.0039\.0077\.004\.006\.0067\.2064\.2051\.1081\.8056\.6665\.5848\.8149\.43SFT3\.0010\.0066\.6066\.8089\.9031\.8054\.002\.0012\.0056\.4064\.4092\.8998\.8050\.0065\.6647\.9553\.99GRPO11\.7586\.0055\.6088\.9099\.7058\.4083\.8018\.0032\.0079\.8089\.8095\.1193\.6060\.0063\.9945\.7566\.11STAR\-ARCF95\.7510023\.2063\.6097\.4042\.2081\.404\.004\.0072\.8068\.0051\.7880\.4060\.0062\.2745\.8056\.55SFT\-ARCF46\.2578\.5093\.2068\.0091\.8033\.8059\.002\.0014\.0061\.8063\.6091\.1198\.2060\.0057\.5448\.6762\.57GRPO\-ARCF56\.7599\.2585\.4084\.9099\.4055\.4079\.0014\.0026\.0080\.4092\.4095\.1199\.4053\.3360\.9648\.0468\.58DSQwen3\-8B ModelsOriginal4\.0011\.0023\.4077\.7091\.2037\.6036\.4014\.006\.0066\.6061\.2036\.676\.4080\.0083\.0267\.6936\.24STAR7\.2546\.5018\.2067\.8094\.9436\.2062\.602\.000\.0069\.2067\.2053\.3388\.8083\.3384\.9165\.5455\.99SFT6\.2510\.2546\.6069\.1091\.6032\.2055\.000\.0018\.0066\.8069\.0076\.6797\.6086\.6784\.2364\.2759\.70GRPO13\.5088\.7528\.8091\.9099\.3052\.2063\.2024\.0020\.0080\.8089\.4074\.2293\.4083\.3383\.3268\.6468\.07STAR\-ARCF94\.7510021\.0071\.9096\.2039\.0064\.004\.008\.0071\.0067\.6054\.0088\.8083\.3385\.4466\.1163\.41SFT\-ARCF67\.5090\.0087\.4070\.0093\.2036\.8053\.400\.0018\.0065\.8074\.6078\.2295\.0083\.3384\.1564\.7171\.43GRPO\-ARCF92\.5097\.5083\.8097\.9099\.4069\.6074\.8036\.0040\.0088\.2096\.8090\.0095\.8083\.3383\.9069\.1282\.34DSQwen2\-14B ModelsOriginal4\.756\.5060\.867\.8084\.1027\.6039\.006\.004\.0059\.4063\.2098\.6799\.6070\.0084\.0867\.1556\.89STAR6\.0013\.5044\.4062\.2093\.6034\.8072\.400\.002\.0070\.2065\.4056\.4491\.2073\.3385\.8567\.6156\.98SFT5\.5010\.7573\.8068\.3092\.2031\.0051\.202\.006\.0066\.6065\.8077\.1198\.2070\.0085\.3765\.0359\.18GRPO8\.5019\.5062\.2095\.2099\.8057\.2075\.2014\.0026\.0085\.4090\.6093\.5697\.6070\.0085\.2566\.5669\.98STAR\-ARCF94\.5010039\.2061\.1095\.4037\.0072\.404\.0010\.0073\.2068\.8057\.5688\.4070\.0084\.0067\.5065\.26SFT\-ARCF16\.0056\.7592\.2070\.1091\.8030\.0051\.4010\.0012\.0065\.2069\.0078\.4497\.4070\.0084\.3165\.5865\.21GRPO\-ARCF98\.5099\.7586\.8095\.5099\.9062\.4079\.8016\.0038\.0091\.2094\.8091\.1198\.0073\.3384\.3165\.6280\.65

### 3\.2Experiment Setup

Datasets and Models\.To evaluate the effectiveness of ARCF, we conduct experiments on a diverse set of LRMs, including DSQwen3\-8B, DSLLaMA3\-8B, and DSQwen2\-14B\. The training corpus consists of 2K prompts, including1K harmful promptsfrom SafeChain[20](https://arxiv.org/html/2609.27763#bib.bib54)and1K benign promptsthat elicit over\-refusal behavior from FalseReject[57](https://arxiv.org/html/2609.27763#bib.bib55)\. Training samples are augmented with 4\-shot counter\-aligned conversations following the construction in Appendix[A\.2\.1](https://arxiv.org/html/2609.27763#A1.SS2.SSS1), with a 4\-shot configuration\. The augmentation ratio is set to 50%\. The rationale for these choices and a detailed ablation analysis are provided in Appendix[D](https://arxiv.org/html/2609.27763#A4)and Appendix[E](https://arxiv.org/html/2609.27763#A5)\.

Baselines\.Our baselines include STAR[43](https://arxiv.org/html/2609.27763#bib.bib31), SFT, and GRPO\. STAR applies SFT on a safety\-dataset only containing1K harmfulprompts\. We compare these baseline methods with their corresponding variants that incorporate our ARCF framework to evaluate the benefits of our approach\. Detailed descriptions of the GRPO rewards, along with additional training details and hyperparameter settings, are provided in Appendix[A](https://arxiv.org/html/2609.27763#A1)\.

Evaluations and Metrics\.We evaluate original and fine\-tuned models on three tasks:jailbreak,over\-refusal, andutility\. For the 0\-shot jailbreak setting, we use SafeChain[20](https://arxiv.org/html/2609.27763#bib.bib54)as an in\-distribution benchmark, where the SafeChain examples used for training and evaluation are strictly non\-overlapping\. We additionally evaluate on FORTRESS[23](https://arxiv.org/html/2609.27763#bib.bib57), H\-CoT[25](https://arxiv.org/html/2609.27763#bib.bib46), and Prefill[32](https://arxiv.org/html/2609.27763#bib.bib42), which serve asunseen attack benchmarksbeyond the training setup\. For over\-refusal, we consider both settings, using XSTEST[37](https://arxiv.org/html/2609.27763#bib.bib58)and the benign subset of FORTRESS in the 0\-shot setting, and OR\-Bench[10](https://arxiv.org/html/2609.27763#bib.bib52)in the 16\-shot setting\. Safety and refusal judgments are produced by GPT\-oss\-safeguard\-20B[1](https://arxiv.org/html/2609.27763#bib.bib53), and we reportSafety Score \(SS\)and1\-Refusal Rate \(1\-RR\)\(see Section[2\.3](https://arxiv.org/html/2609.27763#S2.SS3)\)\. For utility, we evaluate mathematical reasoning on GSM8K[8](https://arxiv.org/html/2609.27763#bib.bib59)\(pass@1\) and AIME 2025[56](https://arxiv.org/html/2609.27763#bib.bib60)\(pass@8\), and general knowledge on MMLU\-Pro[42](https://arxiv.org/html/2609.27763#bib.bib61)usingAccuracy \(Acc\.\)\. To obtain an overall summary, we compute the mean score within each task and report the harmonic mean across tasks, which penalizes imbalanced trade\-offs among safety, helpfulness, and utility\. Detailed metric definitions are provided in Appendix[A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3)\.

### 3\.3Results and Discussion

ARCF provides robust defense against 16\-shot SRCF attacks\.Table[3](https://arxiv.org/html/2609.27763#S3.T3)shows that baseline post\-training methods \(STAR, SFT, and GRPO\) are highly vulnerable to our 16\-shot SRCF attack\. Across both Jailbreak and over\-refusal tasks, models post\-trained with methods incorporating our ARCF achieve substantially improved safety and helpfulness under the few\-shot setting\. Specifically, these methods consistently obtain higher Safety Scores \(SS\) and higher 1−\-Refusal Rate \(1−\-RR\) than their corresponding baselines without ARCF across almost all models in the 16\-shot SRCF setting\. These results demonstrate the effectiveness of ARCF in mitigating the vulnerability introduced by SRCF\. Importantly, in the 0\-shot setting, incorporating ARCF does not degrade safety or helpfulness\. Across both jailbreak and over\-refusal evaluations, ARCF\-based variants consistently match or outperform their baseline counterparts\. Overall, methods incorporating ARCF achieve higherAvg\.scores than their baseline versions, indicating improved performance when jointly considering safety, over\-refusal, and utility across all shot settings\. For a more concise comparison, we additionally report an aggregated evaluation across all tasks and provide a detailed analysis in Appendix[J](https://arxiv.org/html/2609.27763#A10)\.

GRPO\-ARCF achieves the best overall trade\-off between safety and helpfulness\.Methods such as STAR and STAR\-ARCF primarily improve model safety but suffer noticeable degradation in helpfulness, as they are trained exclusively on safety\-focused datasets\. SFT and SFT\-ARCF achieve strong performance on the over\-refusal task under the 16\-shot SRCF setting, yet they show limited gains in jailbreak safety under both 0\-shot and few\-shot settings\. In contrast, GRPO\-based methods consistently perform well on both jailbreak and over\-refusal tasks\. This balanced behavior is directly reflected in theAvg\.metric, which aggregates jailbreak, over\-refusal, and utility performance via a harmonic mean\. Notably, GRPO\-ARCF yields the best or second\-best performance across most metrics, models, and settings, as reflected by the concentration of bolded and underlined results in Table[3](https://arxiv.org/html/2609.27763#S3.T3)\. These results suggest that GRPO\-ARCF provides the most balanced improvement between safety and helpfulness among all evaluated methods\.

![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_llama3_GRPO.png)\(a\) DSLLaMa3\-8B\_GRPO
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen3_GRPO.png)\(b\) DSQwen3\-8B\_GRPO
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen2_GRPO.png)\(c\) DSQwen2\-14B\_GRPO

Figure 3:PCA visualizations of the final\-layer hidden states for benign prompts, harmful prompts, and their SRCF\-attacked variants in the jailbreak setting\. Each point corresponds to the last\-token hidden state of the model after alignment withGRPO\-ARCF\. Theblack arrowindicates the dominant direction of representation drift observed for benign prompts, while thered arrowindicates the shift observed for harmful prompts after prepending SRCF conversations\. Compared to the base model \(Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\), ARCF induces more parallel shifts for benign and harmful prompts and preserves clearer representation separation between them, enabling the model to better distinguish benign and harmful inputs under SRCF attack\. The arrows are manually added for qualitative illustration\.ARCF does not compromise general utility\.Under ARCF, models exhibit improved safety and helpfulness while preserving strong overall utility\. Across GSM8K, AIME 2025, and MMLU\-Pro, ARCF variants consistently match or slightly outperform both the original models and their baseline post\-training counterparts \(Table[3](https://arxiv.org/html/2609.27763#S3.T3)\)\. These results indicate that ARCF enhances safety and helpfulness without compromising general reasoning performance on established utility benchmarks\.

### 3\.4ARCF Restores Safety\-Aligned Representations

We next analyze how ARCF alters the internal representations of LRMs and mitigates the representation drift induced by SRCF\. In the jailbreak setting, for example, we visualize the final\-layer hidden representations of harmful prompts under SRCF after ARCF post\-training in Figure[3](https://arxiv.org/html/2609.27763#S3.F3), using PCA projections consistent with Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\.

As discussed in Section[2\.5](https://arxiv.org/html/2609.27763#S2.SS5), SRCF induces a pronounced representation drift \(Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\): when prompts are prepended with SRCF conversations, the representations of both benign and harmful inputs move toward a common direction, leading to reduced separability\. After post\-training with ARCF, this behavior is substantially altered\. As shown in Figure[3](https://arxiv.org/html/2609.27763#S3.F3), the representation drift directions of benign and harmful promptsremain more paralleland no longer converge toward a common direction, in contrast to the base model shown in Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\. We denote these shifts using arrowed vectors, where the black arrow represents the shift induced on benign prompts and the red arrow represents the shift induced on harmful prompts after prepending SRCF conversations\. These results indicate that ARCF constrains the representation drift introduced by SRCF, enabling the model to preserve safety\-aligned internal representations under counter\-aligned few\-shot prompting\. This interpretation is further supported by ourlayer\-wise probing analysisin Appendix[F](https://arxiv.org/html/2609.27763#A6)and Figure[9](https://arxiv.org/html/2609.27763#A6.F9), where we provide a detailed experimental setup and analysis\.

## 4Related Work

Recent LRMs explicitly generate CoT reasoning to improve complex problem solving, but this design has been shown to introduce new safety risks and attack surfaces\. Prior work has demonstrated that adversaries can exploit reasoning traces through backdoor attacks[13](https://arxiv.org/html/2609.27763#bib.bib21), jailbreak attacks[25](https://arxiv.org/html/2609.27763#bib.bib46);[48](https://arxiv.org/html/2609.27763#bib.bib22);[60](https://arxiv.org/html/2609.27763#bib.bib47);[50](https://arxiv.org/html/2609.27763#bib.bib33), prefilling attacks[32](https://arxiv.org/html/2609.27763#bib.bib42);[34](https://arxiv.org/html/2609.27763#bib.bib43), reasoning\-length attacks[4](https://arxiv.org/html/2609.27763#bib.bib26);[24](https://arxiv.org/html/2609.27763#bib.bib25);[9](https://arxiv.org/html/2609.27763#bib.bib24);[53](https://arxiv.org/html/2609.27763#bib.bib4), or multi\-turn conversational steering[58](https://arxiv.org/html/2609.27763#bib.bib20);[36](https://arxiv.org/html/2609.27763#bib.bib19);[49](https://arxiv.org/html/2609.27763#bib.bib18)\. In parallel, existing safety alignment approaches leverage SFT\-based[20](https://arxiv.org/html/2609.27763#bib.bib54);[43](https://arxiv.org/html/2609.27763#bib.bib31);[54](https://arxiv.org/html/2609.27763#bib.bib14)or RL\-based[32](https://arxiv.org/html/2609.27763#bib.bib42);[52](https://arxiv.org/html/2609.27763#bib.bib11)methods to improve model safety and helpfulness\. However, most existing studies focus on query\-specific settings and do not systematically examine vulnerabilities arising from few\-shot conversations enabled by large context windows\. We study this underexplored setting and introduce SRCF, a few\-shot conversational attack, along with ARCF, a general post\-training framework that aligns reasoning behavior and improves safety and helpfulness without compromising utility\. A more comprehensive discussion of related work is provided in the Appendix[K](https://arxiv.org/html/2609.27763#A11)\.

## 5Conclusion

In this work, we uncover a previously underexplored vulnerability of LRMs arising from few\-shot conversational contexts with counter\-aligned CoT traces\. We demonstrate that SRCF can systematically exploit this vulnerability to steer model reasoning toward unsafe or overly cautious behavior, exposing a fundamental failure mode in current long\-context reasoning systems\. To address this issue, we propose ARCF, a general and compatible post\-training framework that aligns reasoning behavior under such adversarial contexts and improves safety and helpfulness without compromising utility\. This work highlights the need to jointly consider reasoning transparency, long\-context exposure, and post\-training alignment when deploying LRMs in realistic conversational settings\.

## References

- Agarwalet al\.\(2025\)S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§A\.2\.2](https://arxiv.org/html/2609.27763#A1.SS2.SSS2.p1.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Arditiet al\.\(2024\)A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. NandaRefusal in language models is mediated by a single direction\.Advances in Neural Information Processing Systems37,pp\. 136037–136083\.Cited by:[Appendix C](https://arxiv.org/html/2609.27763#A3.p1.1),[§G\.1](https://arxiv.org/html/2609.27763#A7.SS1.p1.1),[§2\.6](https://arxiv.org/html/2609.27763#S2.SS6.p1.1)\.
- Askellet al\.\(2021\)A\. Askell, Y\. Bai, A\. Chen, D\. Drain, D\. Ganguli, T\. Henighan, A\. Jones, N\. Joseph, B\. Mann, N\. DasSarma,et al\.A general language assistant as a laboratory for alignment\.arXiv preprint arXiv:2112\.00861\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1)\.
- Chenet al\.\(2024\)X\. Chen, J\. Xu, T\. Liang, Z\. He, J\. Pang, D\. Yu, L\. Song, Q\. Liu, M\. Zhou, Z\. Zhang,et al\.Do not think that much for 2\+ 3=? on the overthinking of o1\-like llms\.arXiv preprint arXiv:2412\.21187\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Chenet al\.\(2025\)Y\. Chen, J\. Benton, A\. Radhakrishnan, J\. Uesato, C\. Denison, J\. Schulman, A\. Somani, P\. Hase, M\. Wagner, F\. Roger,et al\.Reasoning models don’t always say what they think\.arXiv preprint arXiv:2505\.05410\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1),[§1](https://arxiv.org/html/2609.27763#S1.p2.1)\.
- \[6\]Y\. Chen, Y\. Li, S\. He, and L\. FengReasoning\-preserved safety alignment for large reasoning models\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1)\.
- \[7\]R\. Chu, B\. Zhao, H\. Jiang, S\. Aeron, and Y\. LaoBAM\-icl: causal hijacking in\-context learning with budgeted adversarial manipulation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p5.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p4.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Cuadronet al\.\(2025\)A\. Cuadron, D\. Li, W\. Ma, X\. Wang, Y\. Wang, S\. Zhuang, S\. Liu, L\. G\. Schroeder, T\. Xia, H\. Mao,et al\.The danger of overthinking: examining the reasoning\-action dilemma in agentic tasks\.arXiv preprint arXiv:2502\.08235\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Cuiet al\.\(2024\)J\. Cui, W\. Chiang, I\. Stoica, and C\. HsiehOr\-bench: an over\-refusal benchmark for large language models\.arXiv preprint arXiv:2405\.20947\.Cited by:[§A\.2\.1](https://arxiv.org/html/2609.27763#A1.SS2.SSS1.p1.1),[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Geet al\.\(2025\)Y\. Ge, S\. Liu, Y\. Wang, L\. Mei, L\. Chen, B\. Bi, and X\. ChengInnate reasoning is not enough: in\-context learning enhances reasoning large language models with less overthinking\.arXiv preprint arXiv:2503\.19602\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p1.1)\.
- Guoet al\.\(2025a\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1),[§1](https://arxiv.org/html/2609.27763#S1.p1.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p2.1),[§2\.4](https://arxiv.org/html/2609.27763#S2.SS4.p5.1)\.
- Guoet al\.\(2025b\)Z\. Guo, S\. Shi, S\. Yazdani, N\. Zhang, and R\. TouraniDarkmind: latent chain\-of\-thought backdoor in customized llms\.arXiv preprint arXiv:2501\.18617\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p1.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§A\.1](https://arxiv.org/html/2609.27763#A1.SS1.p1.1)\.
- Huanget al\.\(2025a\)T\. Huang, S\. Hu, F\. Ilhan, S\. F\. Tekin, Z\. Yahn, Y\. Xu, and L\. LiuSafety tax: safety alignment makes your large reasoning models less reasonable\.arXiv preprint arXiv:2503\.00555\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1),[§1](https://arxiv.org/html/2609.27763#S1.p2.1)\.
- Huanget al\.\(2025b\)Z\. Huang, T\. Cheng, Z\. Qiu, Z\. Wang, Y\. Xu, E\. M\. Ponti, and I\. TitovBlending supervised and reinforcement fine\-tuning with prefix sampling\.arXiv preprint arXiv:2507\.01679\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1)\.
- Jaechet al\.\(2024\)A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.Openai o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1),[§1](https://arxiv.org/html/2609.27763#S1.p1.1)\.
- Jeunget al\.\(2025\)W\. Jeung, S\. Yoon, M\. Kahng, and A\. NoSafepath: preventing harmful reasoning in chain\-of\-thought via early alignment\.arXiv preprint arXiv:2505\.14667\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[Appendix I](https://arxiv.org/html/2609.27763#A9.p1.1)\.
- Jiaet al\.\(2025\)J\. Jia, N\. Baracaldo, and S\. LiuBeyond sft: reinforcement learning for safer large reasoning models with better reasoning ability\.arXiv preprint arXiv:2512\.01848\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1)\.
- Jianget al\.\(2025\)F\. Jiang, Z\. Xu, Y\. Li, L\. Niu, Z\. Xiang, B\. Li, B\. Y\. Lin, and R\. PoovendranSafechain: safety of language models with long chain\-of\-thought reasoning capabilities\.arXiv preprint arXiv:2502\.12025\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Jianget al\.\(2024\)J\. Jiang, F\. Wang, J\. Shen, S\. Kim, and S\. KimA survey on large language models for code generation\.arXiv preprint arXiv:2406\.00515\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p1.1)\.
- Kimet al\.\(2026\)S\. Kim, H\. Jin, Y\. Lee, and Y\. HanHow does the thinking step influence model safety? an entropy\-based safety reminder for lrms\.External Links:2601\.03662,[Link](https://arxiv.org/abs/2601.03662)Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p3.1)\.
- Knightet al\.\(2025\)C\. Q\. Knight, K\. Deshpande, V\. Sirdeshmukh, M\. Mankikar, S\. R\. Team, S\. Team, and J\. MichaelFORTRESS: frontier risk evaluation for national security and public safety\.arXiv preprint arXiv:2506\.14922\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p2.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Kumaret al\.\(2025\)A\. Kumar, J\. Roh, A\. Naseh, M\. Karpinska, M\. Iyyer, A\. Houmansadr, and E\. BagdasarianOverthink: slowdown attacks on reasoning llms\.arXiv preprint arXiv:2502\.02542\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Kuoet al\.\(2025\)M\. Kuo, J\. Zhang, A\. Ding, Q\. Wang, L\. DiValentin, Y\. Bao, W\. Wei, H\. Li, and Y\. ChenH\-cot: hijacking the chain\-of\-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek\-r1, and gemini 2\.0 flash thinking\.arXiv preprint arXiv:2502\.12893\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p2.1),[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1),[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Liet al\.\(2024\)S\. Li, L\. Yao, L\. Zhang, and Y\. LiSafety layers in aligned large language models: the key to llm security\.arXiv preprint arXiv:2408\.17003\.Cited by:[Appendix C](https://arxiv.org/html/2609.27763#A3.p1.1),[§G\.1](https://arxiv.org/html/2609.27763#A7.SS1.p1.1),[§2\.6](https://arxiv.org/html/2609.27763#S2.SS6.p1.1)\.
- Liuet al\.\(2026\)H\. Liu, C\. Xu, X\. Jiang, C\. Yang, S\. Yin, Z\. Ma, L\. Ni, and J\. GuoConflicts make large reasoning models vulnerable to attacks\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 9509–9531\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1)\.
- Meta \(2024\)MetaLlama guard 2 \| model cards and prompt formats\.Note:[https://www\.llama\.com/docs/model\-cards\-and\-prompt\-formats/meta\-llama\-guard\-2/](https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-guard-2/)Accessed: 2026\-04\-01Cited by:[Appendix H](https://arxiv.org/html/2609.27763#A8.p1.1)\.
- Nguyenet al\.\(2025\)V\. Nguyen, S\. Zhao, G\. Dao, R\. Hu, Y\. Xie, and L\. A\. TuanThree minds, one legend: jailbreak large reasoning model with adaptive stacked ciphers\.arXiv preprint arXiv:2505\.16241\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p1.1)\.
- Padhiet al\.\(2024\)I\. Padhi, M\. Nagireddy, G\. Cornacchia, S\. Chaudhury, T\. Pedapati, P\. Dognin, K\. Murugesan, E\. Miehling, M\. S\. Cooper, K\. Fraser,et al\.Granite guardian\.arXiv preprint arXiv:2412\.07724\.Cited by:[§A\.1](https://arxiv.org/html/2609.27763#A1.SS1.p2.1)\.
- Penget al\.\(2025\)S\. Peng, E\. Smith, I\. Evtimov, S\. Jiang, P\. Chen, H\. Zhan, H\. Wang, D\. H\. Chau, M\. Pasupuleti, and J\. ChiLarge reasoning models learn better alignment from flawed thinking\.arXiv preprint arXiv:2510\.00938\.Cited by:[§A\.2\.2](https://arxiv.org/html/2609.27763#A1.SS2.SSS2.p1.1),[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p2.1),[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p1.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1)\.
- Rageret al\.\(2025\)C\. Rager, C\. Wendler, R\. Gandikota, and D\. BauDiscovering forbidden topics in language models\.arXiv preprint arXiv:2505\.17441\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1),[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Rahmaniet al\.\(2025\)H\. A\. Rahmani, S\. Krishna, X\. Wang, M\. Naghiaei, and E\. YilmazSelf\-correcting large language models: generation vs\. multiple choice\.arXiv preprint arXiv:2511\.09381\.Cited by:[§2\.4](https://arxiv.org/html/2609.27763#S2.SS4.p3.1)\.
- Renet al\.\(2024\)Q\. Ren, H\. Li, D\. Liu, Z\. Xie, X\. Lu, Y\. Qiao, L\. Sha, J\. Yan, L\. Ma, and J\. ShaoDerail yourself: multi\-turn llm jailbreak attack through self\-discovered clues\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p4.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Röttgeret al\.\(2024\)P\. Röttger, H\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. HovyXstest: a test suite for identifying exaggerated safety behaviours in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5377–5400\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p1.1),[§1](https://arxiv.org/html/2609.27763#S1.p6.1)\.
- Shiet al\.\(2024\)D\. Shi, T\. Shen, Y\. Huang, Z\. Li, Y\. Leng, R\. Jin, C\. Liu, X\. Wu, Z\. Guo, L\. Yu,et al\.Large language model safety: a holistic survey\.arXiv preprint arXiv:2412\.17686\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1)\.
- Wanget al\.\(2025a\)C\. Wang, Y\. Liu, B\. Bi, D\. Zhang, Z\. Li, Y\. Ma, Y\. He, S\. Yu, X\. Li, J\. Fang,et al\.Safety in large reasoning models: a survey\.arXiv preprint arXiv:2504\.17704\.Cited by:[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1)\.
- Wanget al\.\(2023\)J\. Wang, Z\. Liu, K\. H\. Park, Z\. Jiang, Z\. Zheng, Z\. Wu, M\. Chen, and C\. XiaoAdversarial demonstration attacks on large language models\.arXiv preprint arXiv:2305\.14950\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p4.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang,et al\.Mmlu\-pro: a more robust and challenging multi\-task language understanding benchmark\.Advances in Neural Information Processing Systems37,pp\. 95266–95290\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p4.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Wanget al\.\(2025b\)Z\. Wang, H\. Tu, Y\. Wang, J\. Wu, Y\. Liu, J\. Mei, B\. R\. Bartoldson, B\. Kailkhura, and C\. XieStar\-1: safer alignment of reasoning llms with 1k data\.arXiv preprint arXiv:2504\.01903\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p3.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p2.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Weiet al\.\(2023\)Z\. Wei, Y\. Wang, A\. Li, Y\. Mo, and Y\. WangJailbreak and guard aligned language models with only few in\-context demonstrations\.arXiv preprint arXiv:2310\.06387\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p5.1)\.
- Wolfet al\.\(2024\)Y\. Wolf, N\. Wies, D\. Shteyman, B\. Rothberg, Y\. Levine, and A\. ShashuaTradeoffs between alignment and helpfulness in language models with representation engineering\.arXiv preprint arXiv:2401\.16332\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1)\.
- Wuet al\.\(2025\)H\. Wu, B\. Xu, Y\. Shu, M\. Yang, and C\. QinThinking with nothinking calibration: a new in\-context learning paradigm in reasoning large language models\.arXiv preprint arXiv:2508\.03363\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p1.1)\.
- Wuet al\.\(2021\)J\. Wu, L\. Ouyang, D\. M\. Ziegler, N\. Stiennon, R\. Lowe, J\. Leike, and P\. ChristianoRecursively summarizing books with human feedback\.arXiv preprint arXiv:2109\.10862\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p1.1)\.
- Xianget al\.\(2024\)Z\. Xiang, F\. Jiang, Z\. Xiong, B\. Ramasubramanian, R\. Poovendran, and B\. LiBadchain: backdoor chain\-of\-thought prompting for large language models\.arXiv preprint arXiv:2401\.12242\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p5.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Yanget al\.\(2025\)X\. Yang, J\. Lee, A\. Dick, J\. Timm, F\. Xie, and D\. CruzMulti\-turn jailbreaks are simpler than they seem\.arXiv preprint arXiv:2508\.07646\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p4.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Yaoet al\.\(2025\)Y\. Yao, X\. Tong, R\. Wang, Y\. Wang, L\. Li, L\. Liu, Y\. Teng, and Y\. WangA mousetrap: fooling large reasoning models for jailbreak with chain of iterative chaos\.arXiv preprint arXiv:2502\.15806\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Yinget al\.\(2025\)Z\. Ying, D\. Zhang, Z\. Jing, Y\. Xiao, Q\. Zou, A\. Liu, S\. Liang, X\. Zhang, X\. Liu, and D\. TaoReasoning\-augmented conversation for multi\-turn jailbreak attacks on large language models\.\.InEMNLP \(Findings\),pp\. 17138–17157\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p4.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. WangDAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Zarembaet al\.\(2025\)W\. Zaremba, E\. Nitishinskaya, B\. Barak, S\. Lin, S\. Toyer, Y\. Yu, R\. Dias, E\. Wallace, K\. Xiao, J\. Heidecke,et al\.Trading inference\-time compute for adversarial robustness\.arXiv preprint arXiv:2501\.18841\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p3.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Zhanget al\.\(2025a\)Y\. Zhang, Z\. Zeng, D\. Li, Y\. Huang, Z\. Deng, and Y\. DongRealsafe\-r1: safety\-aligned deepseek\-r1 without compromising reasoning capability\.arXiv preprint arXiv:2504\.10081\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1),[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Zhanget al\.\(2025b\)Y\. Zhang, S\. Zhang, Y\. Huang, Z\. Xia, Z\. Fang, X\. Yang, R\. Duan, D\. Yan, Y\. Dong, and J\. ZhuSTAIR: improving safety alignment with introspective reasoning\.External Links:2502\.02384,[Link](https://arxiv.org/abs/2502.02384)Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1)\.
- Zhang and Math\-AI \(2025\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(aime\) 2025\.Cited by:[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p4.1),[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p3.1)\.
- Zhanget al\.\(2025c\)Z\. Zhang, W\. Xu, F\. Wu, and C\. K\. ReddyFalsereject: a resource for improving contextual safety and mitigating over\-refusals in llms via structured reasoning\.arXiv preprint arXiv:2505\.08054\.Cited by:[§3\.2](https://arxiv.org/html/2609.27763#S3.SS2.p1.1)\.
- Zhaoet al\.\(2025a\)G\. Zhao, H\. Wu, X\. Zhang, and A\. V\. VasilakosShadowcot: cognitive hijacking for stealthy reasoning backdoors in llms\.arXiv preprint arXiv:2504\.05605\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p4.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Zhaoet al\.\(2025b\)H\. Zhao, C\. Yuan, F\. Huang, X\. Hu, Y\. Zhang, A\. Yang, B\. Yu, D\. Liu, J\. Zhou, J\. Lin,et al\.Qwen3guard technical report\.arXiv preprint arXiv:2510\.14276\.Cited by:[Appendix H](https://arxiv.org/html/2609.27763#A8.p1.1)\.
- Zhaoet al\.\(2025c\)J\. Zhao, T\. Fu, R\. Schaeffer, M\. Sharma, and F\. BarezChain\-of\-thought hijacking\.arXiv preprint arXiv:2510\.26418\.Cited by:[§K\.1](https://arxiv.org/html/2609.27763#A11.SS1.p2.1),[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p3.1),[§4](https://arxiv.org/html/2609.27763#S4.p1.1)\.
- Zhouet al\.\(2025\)K\. Zhou, C\. Liu, X\. Zhao, S\. Jangam, J\. Srinivasa, G\. Liu, D\. Song, and X\. E\. WangThe hidden risks of large reasoning models: a safety assessment of r1\.arXiv preprint arXiv:2502\.12659\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p4.1),[Appendix K](https://arxiv.org/html/2609.27763#A11.p1.1)\.
- Zhouet al\.\(2023\)X\. Zhou, Y\. Qiang, S\. Z\. Zade, P\. Khanduri, and D\. ZhuHijacking large language models via adversarial in\-context learning\.arXiv preprint arXiv:2311\.09948\.Cited by:[§1](https://arxiv.org/html/2609.27763#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.27763#S2.SS3.p5.1)\.
- Zhuet al\.\(2025\)Z\. Zhu, X\. Wu, G\. Hu, S\. Lyu, K\. Xu, and B\. WuAdvChain: adversarial chain\-of\-thought tuning for robust safety alignment of large reasoning models\.arXiv preprint arXiv:2509\.24269\.Cited by:[§K\.2](https://arxiv.org/html/2609.27763#A11.SS2.p2.1)\.
- Zouet al\.\(2023\)A\. Zou, Z\. Wang, N\. Carlini, M\. Nasr, J\. Z\. Kolter, and M\. FredriksonUniversal and transferable adversarial attacks on aligned language models\.arXiv preprint arXiv:2307\.15043\.Cited by:[§A\.2\.1](https://arxiv.org/html/2609.27763#A1.SS2.SSS1.p1.1),[§A\.2\.3](https://arxiv.org/html/2609.27763#A1.SS2.SSS3.p2.1)\.

## Appendix

## Appendix AAdditional Experiment Details

### A\.1Computational Configurations

All experiments are conducted on nodes equipped with 2×\\timesNVIDIA H100 \(80 GB\) GPUs using parameter\-efficient fine\-tuning with LoRA\[[14](https://arxiv.org/html/2609.27763#bib.bib29)\]\. Unless otherwise stated, we apply LoRA with rankr=16r=16, scaling factorα=32\\alpha=32, dropout0\.050\.05, and identical target modules across STAR, SFT, and GRPO to ensure fair comparison\. Models are trained using the AdamW optimizer for22epochs with an effective batch size of3232\. Because different post\-training methods exhibit different sensitivity to the learning rate, we select the learning rate for each method via grid search over1​e−5,2​e−5,…,1​e−4\{1\\mathrm\{e\}\{\-5\},2\\mathrm\{e\}\{\-5\},\\ldots,1\\mathrm\{e\}\{\-4\}\}\. Based on this search, we usel​r=3​e−5lr=3\\mathrm\{e\}\{\-5\}for GRPO andl​r=7​e−5lr=7\\mathrm\{e\}\{\-5\}for STAR and SFT\. For ARCF training, we augment 50% of the training data with counter\-aligned conversational contexts\.

In the GRPO setting, we adoptIBM Granite\-Guardian\-3\.1\-8B\[[31](https://arxiv.org/html/2609.27763#bib.bib56)\]as the safety reward model and directly use its output logits as continuous reward signals based on both generated CoT and final response, providingdenser and more informative feedbackthan thresholded binary judgments\. For over\-refusal evaluation, we employGPT\-oss\-safeguard\-20Bas a refusal classification model\. Given a generated response, the model assigns it to refusal\-related categories, which are then mapped to scalar rewards for post\-training\. We sample groups ofG=4G=4rollouts per prompt\. Each rollout is truncated to a maximum length ofLmax=8192L\_\{\\text\{max\}\}=8192tokens, including the prompt, chain\-of\-thought, and final response\. We set the clipping threshold toϵ=0\.2\\epsilon=0\.2and disable KL regularization by setting the coefficientβ=0\.0\\beta=0\.0\.

### A\.2Detailed Experiment Setup and Evaluation Metrics

#### A\.2\.1Attack Evaluation Datasets

We evaluate the effectiveness of our SRCF attack on two tasks: jailbreak and over\-refusal\. For the jailbreak setting, we sample 400 harmful queries from Adv\-Bench\[[64](https://arxiv.org/html/2609.27763#bib.bib51)\]as the test set\. For over\-refusal evaluation, we use OR\-Bench\[[10](https://arxiv.org/html/2609.27763#bib.bib52)\]and sample 500 benign queries that are frequently misclassified as unsafe by modern models\. To construct counter\-aligned few\-shot conversations, we randomly draw queries from the same datasets while strictly ensuring no overlap with any test samples\. For each conversational query, we use DSLLaMA3\-70B to generate the counter\-aligned CoTs and responses\. The prompting instruction used to generate them is provided in Appendix Figure[14](https://arxiv.org/html/2609.27763#A12.F14)\. We construct separate counter\-aligned few\-shot conversations for the jailbreak and over\-refusal tasks\.

#### A\.2\.2Attack Evaluation Metrics

In this section, we describe the evaluation metrics reported in Table[2](https://arxiv.org/html/2609.27763#S2.T2), which are used to assess model performance in Section[2](https://arxiv.org/html/2609.27763#S2)\. Following prior work\[[32](https://arxiv.org/html/2609.27763#bib.bib42)\], we adopt a model\-based evaluation protocol for both jailbreak and over\-refusal tasks\. We use GPT\-oss\-safeguard\-20B\[[1](https://arxiv.org/html/2609.27763#bib.bib53)\], a policy\-driven moderation model, as the evaluator\.

Jailbreak Evaluation\.For the jailbreak setting, we separately evaluate the generated CoTycoty\_\{\\text\{cot\}\}and the final responseyrespy\_\{\\text\{resp\}\}\. A completion is classified as*safe*if it contains no harmful content under our safety policy \(see Figure[13](https://arxiv.org/html/2609.27763#A12.F13)in the Appendix for details\)\. We report the1−\-Safety Score \(1−\-SS\), defined as the percentage of completions judged unsafe, computed independently forycoty\_\{\\text\{cot\}\}andyrespy\_\{\\text\{resp\}\}\. We evaluate both components because, at inference time, both the generated CoT trace and the final response are visible to users and may directly contain unsafe content, even when the final response alone appears benign\.

Over\-Refusal Evaluation\.For over\-refusal evaluation, we apply a refusal\-detection policy \(see Figure[12](https://arxiv.org/html/2609.27763#A12.F12)in the Appendix for details\) to the final responseyrespy\_\{\\text\{resp\}\}\. We report theRefusal Rate \(RR\), defined as the percentage of benign prompts whose final responses are classified as refusals\. A higher RR indicates more severe over\-refusal behavior\. we focus solely on the final response, as the primary concern is whether the model provides a helpful, non\-refusing response\.

Average \(Avg\)\.For both jailbreak and over\-refusal evaluations, we report an average score to summarize attack effectiveness across different models\. Specifically, for each attack method and shot setting, we compute the mean 1−\-Safety Score \(1−\-SS\) across all evaluated models in the jailbreak setting, reported separately for the CoT trace \(C\) and the final response \(R\)\. For over\-refusal evaluation, we report the mean Refusal Rate \(RR\) across all evaluated models, computed on the final response only\. This metric provides a model\-agnostic measure of attack effectiveness\.

#### A\.2\.3Defense Evaluation Metrics

This section details the evaluation metrics presented in Table[3](https://arxiv.org/html/2609.27763#S3.T3), used for evaluating model performance in Section[3](https://arxiv.org/html/2609.27763#S3)\. We assess original and fine\-tuned model performance across three tasks:jailbreak,over\-refusal, andutility\.

Jailbreak Evaluation\.For the jailbreak task, we assess model robustness under both a 0\-shot setting and a 16\-shot setting using our SRCF attack\. In the 0\-shot setting, we evaluate model safety using SafeChain\[[20](https://arxiv.org/html/2609.27763#bib.bib54)\], FORTRESS\[[23](https://arxiv.org/html/2609.27763#bib.bib57)\], H\-CoT\[[25](https://arxiv.org/html/2609.27763#bib.bib46)\], and Prefill\[[32](https://arxiv.org/html/2609.27763#bib.bib42)\]\. In the 16\-shot setting, we assess robustness against SRCF using AdvBench\[[64](https://arxiv.org/html/2609.27763#bib.bib51)\]\. Notably, FORTRESS consists of expert\-crafted adversarial prompts and provides a high\-precision evaluation of frontier safety risks in the 0\-shot setting\.

Over\-Refusal Evaluation\.For over\-refusal evaluation, we also consider both 0\-shot and SRCF 16\-shot settings\. In the 0\-shot setting, we use XSTEST\[[37](https://arxiv.org/html/2609.27763#bib.bib58)\]and the benign subset of FORTRESS\. Under the SRCF 16\-shot setting, we evaluate over\-refusal using OR\-Bench\[[10](https://arxiv.org/html/2609.27763#bib.bib52)\]\. Both safety and over\-refusal judgments are produced by GPT\-oss\-safeguard\-20B\. We reportSafety Score \(SS\)for safety evaluation and1\-Refusal Rate \(1\-RR\)for over\-refusal evaluation\. Formal definitions of these metrics are provided in Section[2\.3](https://arxiv.org/html/2609.27763#S2.SS3)\.

Utility Evaluation\.To assess utility, we first evaluate mathematical reasoning performance on GSM8K\[[8](https://arxiv.org/html/2609.27763#bib.bib59)\]and AIME 2025\[[56](https://arxiv.org/html/2609.27763#bib.bib60)\]\. We report pass@KK, withK=1K=1for GSM8K andK=8K=8for AIME 2025, to ensure stable evaluation\. We further evaluate general knowledge and reasoning ability on MMLU\-Pro\[[42](https://arxiv.org/html/2609.27763#bib.bib61)\], a multi\-domain question\-answering benchmark, and reportAccuracy \(Acc\), defined as the percentage of correctly answered questions\.

Average \(Avg\)\.For each task \(jailbreak, over\-refusal, and utility\), we first compute the mean performance across its corresponding benchmarks\. We then report the harmonic mean of these task\-level means to obtain an overall summary metric\. We choose the harmonic mean because strong safety performance should not come at the expense of degraded helpfulness or utility; this metric explicitly penalizes imbalanced trade\-offs and emphasizes methods that perform well across all tasks\.

#### A\.2\.4Decoding Details

Unless otherwise stated, all outputs are generated with the decoding settings recommended by the model publishers\. Specifically, for the DeepSeek\-R1 series, we use a temperature of 0\.6 and top\-p of 0\.95\. For GPT\-oss\-20B, we use a temperature of 1\.0 and top\-p of 1\.0\.

Figure 4:Illustration of our SRCF and baseline methods used in our study\.SRCFpreserves prior conversation history, including both explicit reasoning traces and final response\.Response\-Onlyalso retains the conversation history but removes reasoning traces\.advICL presents the same content as standard in\-context examples without treating it as prior conversation history\.

## Appendix BTraining Objectives for ARCF

### B\.1Supervised Fine\-Tuning \(SFT\)

Under SFT, we train the model on the augmented dataset𝒟pre=\{\(xpre,y\)\}\\mathcal\{D^\{\\text\{pre\}\}\}=\\\{\(x\_\{\\text\{pre\}\},\\;y\)\\\}using the standard cross\-entropy loss\. Each augmented inputxpre=\[C⋆;x\]x\_\{\\text\{pre\}\}=\[C^\{\\star\};\\,x\]includes few\-shot conversations with counter\-aligned reasoning and response, while the target outputy=\(ycot,yresp\)y=\(y\_\{\\text\{cot\}\},\\;y\_\{\\text\{resp\}\}\)remains fully aligned\. The loss is defined as

ℒSFT\-ARCF​\(θ\)=−𝔼\(xpre,y\)∼𝒟pre​\[log⁡πθ​\(y∣xpre\)\],\\displaystyle\\mathcal\{L\}\_\{\\text\{SFT\-ARCF\}\}\(\\theta\)=\-\\;\\mathbb\{E\}\_\{\(x\_\{\\text\{pre\}\},y\)\\sim\\mathcal\{D\}\_\{\\text\{pre\}\}\}\\left\[\\log\\pi\_\{\\theta\}\(y\\mid x\_\{\\text\{pre\}\}\)\\right\],\(1\)whereπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)denotes the probability of modelπθ\\pi\_\{\\theta\}to generateyyby given inputxx\. SFT with ARCF trains the model to remain safe and helpful even when its input context contains counter\-aligned reasoning traces, improving robustness against our SRCF\.

### B\.2Group Relative Policy Optimization \(GRPO\)

ARCF can also be seamlessly integrated with reinforcement learning\-based methods\. In the GRPO setting, letπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}denote the old policy model\. GRPO samples a group of rollouts\{o1,o2,…,oG\}\\\{o\_\{1\},o\_\{2\},\\dots,o\_\{G\}\\\}from the old policyπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}conditioned on the same augmented inputxprex\_\{\\text\{pre\}\}and then optimizes the policy modelπθ\\pi\_\{\\theta\}by maximizing the following objectives:

𝒥GRPO\-ARCF​\(θ\)=𝔼xpre∼𝒟pre,\{oi\}Gi=1∼πθold\(⋅\|xpre\)1G∑Gi=11\|oi\|∑oit=1\{min\[ri,t\(θ\)A^i,t,clip\(ri,t\(θ\),1−ϵ,1\+ϵ\)A^i,t\]−β𝔻KL\[πθ\|\|πref\]\},\\displaystyle\\begin\{split\}\\mathcal\{J\_\{\\text\{GRPO\-ARCF\}\}\}\(\\theta\)=&\\ \\mathbb\{E\}\_\{x\_\{\\text\{pre\}\}\\sim\\mathcal\{D\}\_\{\\text\{pre\}\},\\\{o\_\{i\}\\\}^\{G\}\_\{i=1\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|x\_\{\\text\{pre\}\}\)\}\\frac\{1\}\{G\}\\sum^\{G\}\_\{i=1\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum^\{o\_\{i\}\}\_\{t=1\}\\bigg\\\{\\min\\bigg\[r\_\{i,t\}\(\\theta\)\\hat\{A\}\_\{i,t\},\\;\\\\ &\\text\{clip\}\\bigg\(r\_\{i,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\bigg\)\\hat\{A\}\_\{i,t\}\\bigg\]\-\\beta\\mathbb\{D\}\_\{\\text\{KL\}\}\\big\[\\pi\_\{\\theta\}\|\|\\pi\_\{\\text\{ref\}\}\\big\]\\bigg\\\},\\end\{split\}\(2\)where

ri,t​\(θ\)=πθ​\(oi,t\|xpre,oi,<t\)πθold​\(oi,t\|xpre,oi,<t\),r\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(o\_\{i,t\}\|x\_\{\\text\{pre\}\},o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i,t\}\|x\_\{\\text\{pre\}\},o\_\{i,<t\}\)\},A^i,t=ri−mean​\(\{Ri\}i=1G\)std​\(\{Ri\}i=1G\),\\hat\{A\}\_\{i,t\}=\\frac\{r\_\{i\}\-\\text\{mean\}\(\\\{R\_\{i\}\\\}^\{G\}\_\{i=1\}\)\}\{\\text\{std\}\(\\\{R\_\{i\}\\\}^\{G\}\_\{i=1\}\)\},𝔻KL\[πθ\|\|πref\]=\\displaystyle\\mathbb\{D\}\_\{\\text\{KL\}\}\\big\[\\pi\_\{\\theta\}\|\|\\pi\_\{\\text\{ref\}\}\\big\]=πref​\(oi,t\|xpre,oi,<t\)πθ​\(oi,t\|xpre,oi,<t\)−log​πref​\(oi,t\|xpre,oi,<t\)πθ​\(oi,t\|xpre,oi,<t\)−1\.\\displaystyle\\frac\{\\pi\_\{\\text\{ref\}\}\(o\_\{i,t\}\|x\_\{\\text\{pre\}\},o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\}\(o\_\{i,t\}\|x\_\{\\text\{pre\}\},o\_\{i,<t\}\)\}\-\\text\{log\}\\frac\{\\pi\_\{\\text\{ref\}\}\(o\_\{i,t\}\|x\_\{\\text\{pre\}\},o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\}\(o\_\{i,t\}\|x\_\{\\text\{pre\}\},o\_\{i,<t\}\)\}\-1\.
Here:

- •xprex\_\{\\text\{pre\}\}is the augmented input prepended by demos with counter\-aligned CoTs from augmented dataset𝒟pre\.\\mathcal\{D\}\_\{\\text\{pre\}\}\.
- •ϵ\\epsilonandβ\\betaare hyper\-parameters, specificallyϵ\\epsilonis the clipping range of importance sampling ratio\. More details of hyperparameter are provided in Appendix[A\.1](https://arxiv.org/html/2609.27763#A1.SS1)\.
- •A^i,t\\hat\{A\}\_\{i,t\}is the advantage of theii\-th response calculated by normalizing the rewards across the group of rollouts\{oi\}i=1G\\\{o\_\{i\}\\\}^\{G\}\_\{i=1\}

Across both SFT and GRPO, ARCF does not modify the underlying optimization algorithms or model architecture\. Instead, it operates purely at the data level by introducing counter\-aligned demos during training without additional training objective or model\-specific components\. This design allows ARCF to be easily integrated into existing post\-training pipelines and applied to a wide range of LRMs\.

## Appendix CDetailed Layer\-wise Analysis

Following prior work\[[26](https://arxiv.org/html/2609.27763#bib.bib45),[2](https://arxiv.org/html/2609.27763#bib.bib44)\], we analyze the last\-token hidden representations across all hidden layers\. Specifically, we sample 100 benign prompts and 100 harmful prompts, and construct 500 benign\-benign pairs and 500 benign\-harmful pairs\. For the benign\-benign pairs \(B\-B\), no attack is applied\. For the benign\-harmful pairs, we evaluate the same pairs under two settings: \(1\)B\-H, without attack, and \(2\)B\-H \(SRCF\), where the harmful prompt is attacked with 16\-shot SRCF\.

Figure[5](https://arxiv.org/html/2609.27763#A3.F5)reports the layer\-wise cosine similarity for these three settings across multiple models\. We first observe that B\-B consistently has higher cosine similarity than B\-H, indicating that the models retain a clear representational distinction between benign and harmful prompts in their hidden states\. More importantly, B\-H \(SRCF\) consistently yields higher cosine similarity than B\-H without attack across most layers\. This result shows that SRCF shifts the hidden representations of harmful prompts toward those of benign prompts, making them less distinguishable at the representation level\. These findings provide quantitative support for the representation drift suggested by the PCA results in Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\.

![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/Layer-wise_Cosine_Similarity-R1-Llama-8B.png)\(a\) DSLLaMa3\-8B
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/Layer-wise_Cosine_Similarity-R1-0528.png)\(b\) DSQwen3\-8B
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/Layer-wise_Cosine_Similarity-R1-Qwen-14B.png)\(a\) DSQwen2\-14B
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/Layer-wise_Cosine_Similarity-gpt-oss.png)\(b\) GPT\-oss\-20B

Figure 5:Layer\-wise cosine similarity of last\-token hidden representations across four LRMs under three settings: benign\-benign pairs \(B\-B\), benign\-harmful pairs without attack \(B\-H\), and benign\-harmful pairs under 16\-shot SRCF attack \(B\-H \(SRCF\)\)\. Shaded regions denote standard deviation across sampled pairs\. Across models, B\-B generally exhibits higher cosine similarity than B\-H, indicating a representational distinction between benign and harmful prompts\. More importantly, B\-H \(SRCF\) consistently shows higher cosine similarity than B\-H across most layers, suggesting that SRCF shifts harmful prompts toward benign ones in hidden\-state space and reduces their separability\.Table 4:Fine\-tuning cost of different methods implemented with or without the ARCF framework\. We report the number of training epochs and the average GPU\-hours consumed per epoch\. All methods are trained under identical model, optimizer, and hardware settings; the only differences are the training objective and the number of few\-shot conversations used to augment the fine\-tuning data\.
## Appendix DTraining Efficiency of ARCF

As discussed in Section[E](https://arxiv.org/html/2609.27763#A5), increasing the number of augmentation shots leads to significant improvements in both model safety and helpfulness\. To examine whether using more augmentation shots introduces additional fine\-tuning cost, we analyze the training efficiency of different methods implemented with or without the ARCF framework\. Specifically, we report the average GPU\-hours consumed per training epoch for each method under varying numbers of augmentation shots\.

As shown in Table[4](https://arxiv.org/html/2609.27763#A3.T4), integrating ARCF results in only minor changes in per\-epoch training cost for SFT and STAR across different few\-shot settings, indicating that the ARCF framework itself introduces minimal additional optimization overhead beyond the underlying fine\-tuning objectives\. In contrast, methods that rely on more complex optimization procedures, such as GRPO, incur substantially higher training costs as the number of augmentation shots increases, particularly in the 8\-shot setting\. Considering this trade\-off between performance gains and training efficiency, we adopt 4 augmentation shots for all experiments reported in Table[3](https://arxiv.org/html/2609.27763#S3.T3)\.

## Appendix EWhat Drives the Effectiveness of ARCF

We analyze two key design factors that influence the effectiveness of ARCF:the augmentation ratio, which controls the proportion of training prompts augmented with counter\-aligned conversation history, andthe number of augmentation shots, which determines the number of few\-shot conversations prepended to each augmented training prompt\.

### E\.1Augmentation Ratio

We conduct experiments on DSQwen3\-8B using three training methods \(i\.e\., STAR\-ARCF, SFT\-ARCF, and GRPO\-ARCF\), while varying the augmentation ratio from 0% to 100%, as shown in Figure[6](https://arxiv.org/html/2609.27763#A5.F6)\. An augmentation ratio of 0% corresponds to the original training methods without ARCF and serves as the baseline\.

Compared to the 0% baseline, ARCF yields substantial improvements in both safety and helpfulness under the 16\-shot setting, demonstrating the effectiveness of augmenting training data with counter\-aligned conversation history\. However, as the augmentation ratio increases, the performance of the model exhibits a trend: both Safety Score and11–Refusal Rate initially improve but begin to decline slightly at higher ratios in some settings\. For example, the Safety Score of SFT\-ARCF and GRPO\-ARCF at 75% is lower than that at 50%\.

This behavior suggests a trade\-off between robustness gained from augmented data and potential overexposure to counter\-aligned conversations\. Based on this observation, we adopt an augmentation ratio of 50% for all experiments reported in Table[3](https://arxiv.org/html/2609.27763#S3.T3), as it consistently achieves the best balance between safety and helpfulness across methods\.

![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/STAR_ratio.png)\(a\) STAR\-ARCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/SFT_ratio.png)\(b\) SFT\-ARCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/GRPO_ratio.png)\(c\) GRPO\-ARCF

Figure 6:Performance of ARCF with STAR, SFT, and GRPO as the augmentation ratio varies from 0% to 100%\. We report the average Safety Score \(jailbreak\) and 1−\-Refusal Rate \(over\-refusal\) under both 16\-shot SRCF and 0\-shot settings\. More detailed results are shown in Appendix Table[7](https://arxiv.org/html/2609.27763#A10.T7)\.![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/STAR_shot.png)\(a\) STAR\-ARCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/SFT_shot.png)\(b\) SFT\-ARCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/figures/GRPO_shot.png)\(c\) GRPO\-ARCF

Figure 7:Performance of ARCF with STAR, SFT, and GRPO as the number of shots varies from 0 to 8\. We report the average Safety Score \(jailbreak\) and 1−\-Refusal Rate \(over\-refusal\) under both 16\-shot SRCF and 0\-shot settings\. More detailed results are shown in Appendix Table[8](https://arxiv.org/html/2609.27763#A10.T8)\.
### E\.2Number of Augmentation Shots

We next study the effect of the number of augmentation shots using the same model and training methods, varying the number of prepended few\-shot conversations from 0 to 8, as shown in Figure[7](https://arxiv.org/html/2609.27763#A5.F7)\. A number of augmentation shots of 0% corresponds to the original training methods without ARCF and serves as the baseline\. In contrast to the augmentation ratio, increasing the number of augmentation shots leads to a more consistent improvement in model performance\.

However, the marginal gains diminish once the number of augmentation shots exceeds 4\. Moreover, increasing the number of shots requires prepending longer conversational histories, which substantially increases the length of training samples and results in higher computational cost during training\. As analyzed in Section[D](https://arxiv.org/html/2609.27763#A4), this cost increase becomes more pronounced for optimization\-intensive methods\. Considering this trade\-off between performance improvement and training efficiency, we adopt 4 augmentation shots for all experiments reported in Table[3](https://arxiv.org/html/2609.27763#S3.T3)\.

## Appendix FLayer\-wise Probing Analysis of Representation Drift

### F\.1Experimental Setup

To further quantify the representation drift induced by SRCF, beyond PCA visualization and layer\-wise cosine similarity, we train layer\-wise linear probes to test whether benign and harmful prompts remain separable in the model’s hidden space under SRCF attack\. Specifically, for DSQwen2\-14B, we extract the last\-token hidden state from each layer for 500 benign prompts and 500 harmful prompts under the no\-attack setting\. For each layer, we train a single binary logistic\-regression probe with L2 regularization\. The input to the probe is the last\-token hidden representation of one prompt, and the output is a binary label indicating whether that prompt is benign or harmful\. Thus, each layer has one probe shared by both classes, rather than separate probes for benign and harmful prompts\. After training, the probe is kept frozen during evaluation\.

We then apply the frozen probes to a disjoint held\-out test set containing 100 benign and 100 harmful prompts under three settings: no\-attack, 4\-shot SRCF, and 16\-shot SRCF\. Since this probing analysis focuses on the jailbreak setting, the SRCF variants prepend unsafe counter\-aligned conversations to both benign and harmful prompts\. This evaluation uses the probe trained on no\-attack representations as a reference for the clean benign–harmful separation in the model’s hidden space\. If an SRCF\-attacked harmful prompt is classified as benign by this frozen probe, it indicates that SRCF reduces benign–harmful separability and shifts the harmful\-prompt representation toward the benign side of the clean probe boundary\.

We report per\-class accuracy using the same binary probe\. We treat harmful prompts as the positive class and benign prompts as the negative class\. Therefore, accuracy on benign prompts corresponds to the true negative rate \(TNR\), i\.e\., the fraction of benign prompts correctly classified as benign\. Accuracy on harmful prompts corresponds to the true positive rate \(TPR\), i\.e\., the fraction of harmful prompts correctly classified as harmful\. We also repeat the analysis with PCA preprocessing using 64 components, fitted only on no\-attack training representations, to check that the trend is not specific to full\-dimensional representation space\. Finally, we apply the same protocol to DSQwen2\-14B after ARCF post\-training to examine whether ARCF restores benign–harmful separability under SRCF\.

\(a\) Full\-Dimensional Hidden States\(b\) PCA\-Reduced Hidden States \(64 Components\)
Figure 8:Layer\-wise binary probing results on the original DSQwen2\-14B before additional safety alignment under no\-attack, 4\-shot SRCF, and 16\-shot SRCF settings\. The probes are trained on no\-attack representations of benign and harmful prompts and then applied frozen to representations under each test setting\. Panel \(a\) trains probes on full\-dimensional hidden states, while panel \(b\) trains independent probes on PCA\-reduced hidden states with 64 components\. We treat benign prompts as the negative class and harmful prompts as the positive class\. So in each panel,Leftshows accuracy on benign prompts, i\.e\., the true negative rate \(TNR\), andRightshows accuracy on harmful prompts, i\.e\., the true positive rate \(TPR/recall\)\. Under SRCF attack, TNR remains near 1\.0 across layers, while TPR/recall drops to near zero across most later layers, indicating that SRCF causes the frozen benign–harmful probe to increasingly classify attacked harmful prompts as benign\. The consistent trend in both full\-dimensional and PCA\-reduced settings suggests that this effect is not an artifact of high\-dimensional probe fitting\.
### F\.2SRCF Collapses Harmful\-Prompt Separability in the Original Model

##### Representations of benign and harmful prompts are linearly separable without attack\.

Figure[8](https://arxiv.org/html/2609.27763#A6.F8)shows the layer\-wise probing results for the original DSQwen2\-14B before ARCF post\-training\. Under the no\-attack setting \(blue line\), the probe achieves near\-perfect accuracy for both benign and harmful prompts across almost all layers\. Since the probe is trained solely on no\-attack representations, this confirms that the original model maintains a clear, linearly separable distinction between the two prompt types in its hidden space\.

##### SRCF selectively shifts harmful representations into the benign region\.

Under SRCF attack, the two prompt types exhibit sharply asymmetric behavior\. Benign accuracy \(TNR\) remains close to 1\.0 across all layers and shot counts, indicating that benign prompt representations stay on the benign side of the probe boundary even after SRCF conversations are prepended\. Harmful accuracy \(TPR\), by contrast, collapses to near zero across most layers under both 4\-shot \(orange line\) and 16\-shot \(red line\) SRCF, meaning the frozen probe consistently misclassifies attacked harmful prompts as benign\.

This result reveals that SRCF does not introduce a uniform shift affecting both prompt types equally\. Rather, it selectively moves harmful prompt representations across the probe boundary into the benign region, directly weakening the representation\-level distinction that underlies safety behavior\. The effect is related to the number of shots: 16\-shot SRCF produces a more complete collapse than 4\-shot, consistent with the progressive convergence observed in the PCA analysis \(Figure[2](https://arxiv.org/html/2609.27763#S2.F2)\)\. Results under PCA\-64 probing show the same qualitative pattern, confirming that this finding is not an artifact of operating in the full high\-dimensional hidden space\.

### F\.3ARCF Partially Restores Benign–Harmful Separability

Figure[9](https://arxiv.org/html/2609.27763#A6.F9)\(a,b\) shows the layer\-wise probing results after ARCF post\-training\. Similar to the original model, the no\-attack setting \(blue line\) still achieves near\-perfect accuracy on both benign and harmful prompts across almost all layers\. This indicates that ARCF preserves the no\-attack benign–harmful separation in the hidden space, rather than degrading the model’s ability to distinguish the two prompt types\.

Under SRCF attack, ARCF substantially changes the failure pattern observed in the original model\. Accuracy on benign prompts, i\.e\., the true negative rate \(TNR\), remains close to 1\.0 across layers and shot counts, showing that benign prompts are still consistently classified as benign by the frozen probe\. More importantly, accuracy on harmful prompts, i\.e\., the true positive rate \(TPR/recall\), no longer collapses to near zero across most layers\. Although TPR/recall still drops in early layers and remains unstable under the stronger 16\-shot SRCF setting, it recovers above chance across many intermediate and later layers, especially under 4\-shot SRCF\. This is in clear contrast to the original model, where attacked harmful prompts are almost always classified as benign across most later layers\.

These results suggest that ARCF partially restores the benign–harmful separation under SRCF\. In other words, after ARCF post\-training, unsafe conversational histories are less able to push harmful\-prompt representations entirely to the benign side of the no\-attack probe boundary\. This consistent improvement over the original model indicates that ARCF constrains the representation drift induced by SRCF\. The PCA\-64 results show the same qualitative trend, further suggesting that the recovery is not specific to full\-dimensional probe fitting\.

\(a\) Full\-Dimensional Hidden States\(b\) PCA\-Reduced Hidden States \(64 Components\)
Figure 9:Layer\-wise binary probing results on the original DSQwen2\-14B after ARCF post\-training under no\-attack, 4\-shot SRCF, and 16\-shot SRCF settings\. The probe is trained on the same setup as Figure[8](https://arxiv.org/html/2609.27763#A6.F8)\. Panel \(a\) trains probes on full\-dimensional hidden states, while panel \(b\) trains independent probes on PCA\-reduced hidden states with 64 components\. We treat benign prompts as the negative class and harmful prompts as the positive class\. So in each panel,Leftshows accuracy on benign prompts, i\.e\., the true negative rate \(TNR\), andRightshows accuracy on harmful prompts, i\.e\., the true positive rate \(TPR/recall\)\. Compared to the original model, ARCF post\-training substantially restores harmful prompt accuracy across layers under SRCF attack, with the probe recovering above chance level in multiple intermediate and later layers\. This indicates that ARCF recovers the representational separability between benign and harmful prompts under SRCF attack, consistent with the PCA results in Figure[3](https://arxiv.org/html/2609.27763#S3.F3)\.![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_llama3.png)\(a\) DSLLaMa3\-8B: SRCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen3.png)\(b\) DSQwen3\-8B: SRCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen2.png)\(c\) DSQwen2\-14B: SRCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_gpt_oss.png)\(d\) GPT\-oss\-20B: SRCF
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_llama3_advICL.png)\(e\) DSLLaMa3\-8B:
advICL
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen3_advICL.png)\(f\) DSQwen3\-8B:
advICL
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen2_advICL.png)\(g\) DSQwen2\-14B:
advICL
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_gpt_oss_advICL.png)\(h\) GPT\-oss\-20B:
advICL
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_llama3_Response-Only.png)\(i\) DSLLaMa3\-8B:
Response\-Only
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen3_Response-Only.png)\(j\) DSQwen3\-8B:
Response\-Only
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_ds_qwen2_Response-Only.png)\(k\) DSQwen2\-14B:
Response\-Only
![Refer to caption](https://arxiv.org/html/2609.27763v1/PCA_gpt_oss_Response-Only.png)\(l\) GPT\-oss\-20B:
Response\-Only

Figure 10:PCA projections of the final\-layer hidden states for benign prompts, harmful prompts, and their attacked variants across SRCF and baseline methods\. Each point corresponds to the last\-token hidden state of the base model prior to safety alignment\. Theblack arrowindicates the dominant direction of representational shift observed for benign prompts, while thered arrowindicates the shift observed for harmful prompts after attacking\.

## Appendix GDetailed PCA Visualization Setup and Additional PCA Analysis

### G\.1PCA Visualization Setup

Following prior work on representation analysis in safety\-aligned language models, we extract hidden representations from the last token of the final hidden layer for each input prompt\. Specifically, we adopt the same representation choice as\[[26](https://arxiv.org/html/2609.27763#bib.bib45),[2](https://arxiv.org/html/2609.27763#bib.bib44)\], which argues that, during inference, the output vector at the last token of the final hidden layer aggregates the most comprehensive information from preceding layers and directly determines the generated token\. As a result, this representation is particularly suitable for analyzing model behavior and decision boundaries\.

We first randomly sample 300 benign prompts and 300 harmful prompts from the evaluation set\. This same set of prompts is then used across all models and methods\. For each model–method pair, we record the last\-token representation from the final hidden layer for each prompt, and apply PCA jointly to these representations to obtain two\-dimensional visualizations\. This shared sampling and visualization protocol allows us to directly compare how SRCF and baseline methods alter the relative geometry of benign and harmful prompts in representation space\.

### G\.2Additional PCA Analysis of Representation Drifts Under SRCF and Baselines

We adopt PCA rather than non\-linear visualization methods such as t\-SNE because our analysis focuses on representation drift and the relative movement directions induced by different attacks\. PCA preserves global variance structure and linear trends in the embedding space, whereas t\-SNE emphasizes local neighborhood relationships and does not preserve global geometry, making it less suitable for analyzing directional drifts\.

As shown in Figure[2](https://arxiv.org/html/2609.27763#S2.F2), our SRCF consistently drives the representations of both benign and harmful prompts toward a shared region of the embedding space across all evaluated models, and we provided our discussion in Section[2\.5](https://arxiv.org/html/2609.27763#S2.SS5)\. In contrast, baseline methods such as advICL and Response\-Only also induce representation drifts, but their effects are significantly reduced on safer models\. Specifically, the drifts of benign and harmful prompts under these baselines tend to follow approximately parallel directions \(e\.g\., DSQwen3\-8B in Figure[10](https://arxiv.org/html/2609.27763#A6.F10)\(f, j\) and GPT\-oss\-20B in Figure[10](https://arxiv.org/html/2609.27763#A6.F10)\(h, l\)\), thereby largely preserving the relative separation between the two categories

This parallel shift pattern suggests that, under baseline methods, the model maintains its ability to distinguish between benign and harmful prompts at the representation level, even in the presence of few\-shot demonstrations\. These observations are consistent with the quantitative results in Table[2](https://arxiv.org/html/2609.27763#S2.T2), where baseline methods exhibit limited effectiveness in jailbreaking more safety\-aligned models\.

## Appendix HEvaluation of LLM Judge Robustness

To further assess the robustness of our evaluation, we additionally use two other frontier safety judges, Qwen3Guard\-Gen\-8B\[[59](https://arxiv.org/html/2609.27763#bib.bib3)\]and Llama\-Guard\-4\-14B\[[28](https://arxiv.org/html/2609.27763#bib.bib2)\], to evaluate the same DSQwen2\-14B generations under SRCF reported in Table[2](https://arxiv.org/html/2609.27763#S2.T2)\. We then compare their safety judgments with those of GPT\-oss\-safeguard\-20B, which is used as the judge in our main experiments\. The results in Table[5](https://arxiv.org/html/2609.27763#A8.T5)are reported in terms of 1\-Safety Score \(1\-SS\)\. Overall, the trends are highly consistent across all three judges\. This suggests that our conclusions are not tied to a single judge model and remain stable across multiple strong safety evaluators\.

Table 5:Comparison of safety judgments from three frontier LLM judges on DSQwen2\-14B generations under SRCF\. We report 1\-Safety Score \(1\-SS\) for both the generated CoT \(C\) and the final response \(R\) under 4\-shot, 8\-shot, and 16\-shot settings\. The overall trends are highly consistent across judges, indicating that our main findings are robust to the choice of safety evaluator\.
## Appendix IAdditional Evaluation on Recent Safety\-Alignment

To further evaluate the effectiveness and generalizability of our ARCF, we added a recent safety\-alignment baseline, Safepath\[[18](https://arxiv.org/html/2609.27763#bib.bib1)\], and evaluated Safepath\-ARCF under the same setting as in Table[3](https://arxiv.org/html/2609.27763#S3.T3)\. As shown in the Table[6](https://arxiv.org/html/2609.27763#A9.T6), ARCF consistently improves SAFEPATH on both DSQwen3\-8B and DSQwen2\-14B across jailbreak and over\-refusal evaluations under both 16\-shot and 0\-shot settings\. This further supports that ARCF is a general defense framework that can be integrated into different post\-training baselines, rather than being limited to SFT or GRPO\.

Table 6:Additional evaluation of a recent safety\-alignment method, SafePath, with and without ARCF on DSQwen3\-8B and DSQwen2\-14B\. We report Safety Score \(SS\) on jailbreak benchmarks and 1\-Refusal Rate \(1\-RR\) on over\-refusal benchmarks, where higher values indicate better performance\. Across both models, incorporating ARCF consistently improves SafePath under both 16\-shot and 0\-shot settings across different tasks\.
## Appendix JAggregated Evaluation of ARCF and Baselines Derived from Main Results

Table[11](https://arxiv.org/html/2609.27763#A10.F11)provides a consolidated summary of model performance across jailbreak, over\-refusal, and utility tasks, derived from the detailed results reported in Table[3](https://arxiv.org/html/2609.27763#S3.T3)\. The purpose of this table is to offer a clearer and more compact comparison of different post\-training methods by aggregating their performance across multiple benchmarks and task categories\.

For each method and model, we first compute the average performance within each task category\. Specifically, jailbreak performance is averaged over all jailbreak benchmarks using the Safety Score \(SS\), over\-refusal performance is averaged using the 1\-Refusal Rate \(1\-RR\), and utility performance is averaged using task\-specific accuracy metrics\. These task\-level averages capture the overall behavior of each method within a single evaluation dimension while reducing variability across individual benchmarks\.

We then compute the harmonic mean of the three task\-level averages to obtain the overall score \(Avg\.\)\. The harmonic mean is used to penalize methods that perform well on only a subset of tasks, thereby favoring approaches that achieve balanced performance across safety, over\-refusal robustness, and utility\. As shown in Table[11](https://arxiv.org/html/2609.27763#A10.F11), methods incorporating ARCF consistently achieve higher overall scores than their corresponding baseline variants across all evaluated models, confirming that ARCF improves overall performance when jointly considering all evaluation dimensions\.

t

Figure 11:Average evaluation across jailbreak, over\-refusal, and utility tasks\.For each task, we aggregate the same metrics reported in Table[3](https://arxiv.org/html/2609.27763#S3.T3)\. Specifically, jailbreak performance is measured using Safety Score \(SS↑\\uparrow\), over\-refusal performance is measured using 1\-Refusal Rate \(1\-RR↑\\uparrow\), and utility is measured using task\-specific accuracy\. For each task, we first average the performance over all benchmarks associated with that task\. We then compute the harmonic mean of these three task\-level averages as the overall evaluation score\.ModelMethodJailbreakOver\-refusalUtilityAvg\.DSLLaMA3\-8BOriginal39\.2595\.7554\.3855\.24STAR43\.8349\.1757\.0249\.43SFT39\.0386\.1054\.5453\.99GRPO64\.8281\.4456\.5866\.11STAR\-ARCF62\.9251\.7956\.0256\.55SFT\-ARCF51\.8894\.1755\.4062\.57GRPO\-ARCF68\.7593\.3054\.1168\.58DSQwen3\-8BOriginal40\.5722\.1676\.9036\.24STAR45\.3753\.4477\.9355\.99SFT41\.8273\.6278\.3959\.70GRPO62\.3165\.4778\.4368\.07STAR\-ARCF61\.6554\.6078\.2963\.41SFT\-ARCF56\.9386\.8777\.4071\.43GRPO\-ARCF79\.2789\.8778\.7882\.34DSQwen2\-14BOriginal36\.2486\.3673\.7456\.89STAR42\.0164\.0175\.6056\.98SFT39\.9483\.0473\.4759\.18GRPO57\.1484\.4573\.9469\.98STAR\-ARCF61\.6461\.7273\.8365\.26SFT\-ARCF47\.2389\.3573\.3065\.21GRPO\-ARCF77\.5991\.9774\.4280\.65

Table 7:Detailed results of ARCF under different augmentation ratios\. We report jailbreak performance and over\-refusal performance under both the 16\-shot SRCF attack setting and the 0\-shot setting\. Results are shown for STAR, SFT, and GRPO, with augmentation ratios ranging from 0% \(no ARCF\) to 100%\.Method16\-Shots \(SRCF\)0\-ShotAvg\.↑\\uparrowJailbreakOver\-refusalJailbreakOver\-refusalAdv\-benchOR\-BenchSafeChainFORTRESSXSTESTFORTRESSCR\(1\- RR\)↑\\uparrowCRCR\(1\- RR\)↑\\uparrow\(1\- RR\)↑\\uparrowOriginal4\.0011\.0023\.4077\.7091\.2037\.6036\.4036\.676\.4029\.240%STAR7\.2546\.5018\.2067\.8094\.9436\.2062\.6053\.3388\.8052\.99SFT6\.2510\.2546\.6069\.1091\.6032\.2055\.0076\.6797\.6055\.13GRPO13\.5088\.7528\.8091\.9099\.3052\.2063\.2074\.2293\.4066\.7825%STAR\-ARCF94\.0096\.2522\.0071\.4096\.3039\.4064\.2054\.4487\.0063\.79SFT\-ARCF44\.5059\.5084\.6072\.0092\.0035\.6050\.6078\.0096\.8070\.16GRPO\-ARCF31\.7592\.5047\.6095\.3099\.4060\.0068\.6072\.2291\.8072\.5150%STAR\-ARCF94\.75100\.021\.0071\.9096\.2039\.0064\.0054\.0088\.8064\.11SFT\-ARCF67\.5090\.0087\.4070\.0093\.2036\.8053\.4078\.2295\.0076\.59GRPO\-ARCF92\.5097\.5083\.8097\.9099\.4069\.6074\.8090\.0095\.8089\.2475%STAR\-ARCF96\.50100\.019\.8075\.0096\.0037\.8062\.0057\.3389\.6064\.87SFT\-ARCF55\.7592\.5086\.6072\.2094\.2036\.4056\.6074\.8995\.4075\.77GRPO\-ARCF78\.2598\.7586\.2298\.0099\.7065\.0068\.2095\.6074\.2084\.99100%STAR\-ARCF97\.00100\.018\.4073\.8093\.8037\.2052\.8062\.6794\.2065\.97SFT\-ARCF55\.0091\.2588\.0072\.6094\.2033\.6054\.6076\.4497\.0075\.68GRPO\-ARCF100\.0100\.080\.4090\.5099\.6057\.8069\.0071\.7894\.2084\.09

Table 8:Detailed results of ARCF under different numbers of Augmentation Shots\. We report jailbreak performance and over\-refusal performance under both the 16\-shot SRCF attack setting and the 0\-shot setting\. Results are shown for STAR, SFT, and GRPO, with the number of prepended few\-shot conversations from 0 \(no ARCF\) to 8\.Method16\-Shots \(SRCF\)0\-ShotAvg\.↑\\uparrowJailbreakOver\-refusalJailbreakOver\-refusalAdv\-benchOR\-BenchSafeChainFORTRESSXSTESTFORTRESSCR\(1\- RR\)↑\\uparrowCRCR\(1\- RR\)↑\\uparrow\(1\- RR\)↑\\uparrowOriginal4\.0011\.0023\.4077\.7091\.2037\.6036\.4036\.676\.4029\.240\-shotSTAR7\.2546\.5018\.2067\.8094\.9436\.2062\.6053\.3388\.8052\.99SFT6\.2510\.2546\.6069\.1091\.6032\.2055\.0076\.6797\.655\.13GRPO13\.5088\.7528\.8091\.9099\.3052\.2063\.2074\.2293\.4066\.782\-shotSTAR\-ARCF96\.5099\.7519\.4075\.6095\.4039\.8060\.2057\.1189\.4064\.68SFT\-ARCF41\.7555\.2585\.6071\.5091\.8036\.0055\.6078\.2297\.4070\.09GRPO\-ARCF23\.7582\.2564\.0095\.5099\.3061\.0065\.4080\.2294\.4075\.144\-shotSTAR\-ARCF94\.75100\.021\.0071\.9096\.2039\.0064\.0054\.0088\.8064\.11SFT\-ARCF67\.5090\.0087\.4070\.0093\.2036\.8053\.4078\.2295\.0076\.59GRPO\-ARCF92\.5097\.5083\.8097\.9099\.4069\.6074\.8090\.0095\.8089\.248\-shotSTAR\-ARCF95\.00100\.021\.2073\.0095\.9038\.6066\.2051\.7888\.6063\.76SFT\-ARCF48\.5091\.8089\.2072\.2091\.8036\.6057\.2075\.1197\.4075\.37GRPO\-ARCF93\.2599\.2574\.2098\.90100\.068\.2078\.2088\.0092\.8087\.26

## Appendix KDetailed Related Work

The emergence of Large Reasoning Models \(LRMs\), such as OpenAI’s o1\-series\[[17](https://arxiv.org/html/2609.27763#bib.bib34)\]and DeepSeek\-R1\[[12](https://arxiv.org/html/2609.27763#bib.bib37)\], marks a transition toward models that generate structured Chain\-of\-Thought \(CoT\) reasoning before producing a final response\. This explicit reasoning design has been shown to substantially improve performance on complex math and problem\-solving tasks\. However, growing evidence suggests that exposing intermediate reasoning traces also introduces new safety risks and attack surfaces that are largely absent in standard LLMs\[[15](https://arxiv.org/html/2609.27763#bib.bib48),[5](https://arxiv.org/html/2609.27763#bib.bib49),[40](https://arxiv.org/html/2609.27763#bib.bib28),[39](https://arxiv.org/html/2609.27763#bib.bib27),[29](https://arxiv.org/html/2609.27763#bib.bib32),[61](https://arxiv.org/html/2609.27763#bib.bib23),[16](https://arxiv.org/html/2609.27763#bib.bib13)\]\.

### K\.1Attacks on Large Reasoning Models

A growing body of work has demonstrated that the explicit reasoning processes of LRMs can be exploited through various attack paradigms\. One line of research focuses on backdoor attacks that manipulate a model’s behavior whenever a specific trigger is present in the input\. For example, DarkMind\[[13](https://arxiv.org/html/2609.27763#bib.bib21)\]introduces latent triggers that manipulate internal CoT steps without altering user queries\.

Another line of work explores jailbreak attacks that directly interfere with the reasoning process\. Bad\-Chain\[[48](https://arxiv.org/html/2609.27763#bib.bib22)\], H\-COT\[[25](https://arxiv.org/html/2609.27763#bib.bib46)\], CoT Hijacking\[[60](https://arxiv.org/html/2609.27763#bib.bib47)\], and Mousetrap\[[50](https://arxiv.org/html/2609.27763#bib.bib33)\]prepend seemingly benign CoT content to harmful instructions, misleading safety checks, and inducing unsafe generation\. Recent work further shows that injecting conflicting objectives into harmful queries can disrupt safety\-aligned reasoning and increase jailbreak effectiveness\[[27](https://arxiv.org/html/2609.27763#bib.bib63)\]\. Related studies\[[32](https://arxiv.org/html/2609.27763#bib.bib42),[34](https://arxiv.org/html/2609.27763#bib.bib43)\]further show that attackers can inject the initial portion of the model’s response by using the prefilling interface to bypass internal safety checks entirely\.

Beyond direct manipulation of reasoning content, several works investigate reasoning\-length attacks, demonstrating that forcing models to overthink or underthink can significantly degrade performance or reliability\[[4](https://arxiv.org/html/2609.27763#bib.bib26),[24](https://arxiv.org/html/2609.27763#bib.bib25),[9](https://arxiv.org/html/2609.27763#bib.bib24),[53](https://arxiv.org/html/2609.27763#bib.bib4)\]\. While effective, these attacks primarily target single\-turn prompting scenarios\.

More recently, several studies\[[58](https://arxiv.org/html/2609.27763#bib.bib20),[36](https://arxiv.org/html/2609.27763#bib.bib19),[49](https://arxiv.org/html/2609.27763#bib.bib18),[51](https://arxiv.org/html/2609.27763#bib.bib64)\]have explored multi\-turn jailbreak attacks, in which adversaries interact with the model through a sequence of adaptive turns to gradually steer it toward unsafe behavior\. A key characteristic of these attacks is that the interaction trajectory must be re\-crafted for each target query, as the attacker needs to respond to the model’s intermediate outputs and adjust subsequent turns accordingly\. This per\-query interaction requirement limits their transferability and makes them less flexible across diverse prompts\.

In contrast, our work investigates vulnerabilities of LRMs in*few\-shot conversational settings*enabled by large context windows\. Rather than constructing a new multi\-turn interaction trajectory for every target query, SRCF prepends a fixed set of counter\-aligned CoTs into prior conversational turns, which can be reused across different harmful instructions\. This design yields a more flexible and transferable attack, and exposes a previously underexplored vulnerability in LRMs that arises from counter\-aligned conversations\.

### K\.2Alignment on Large Reasoning Models

Since LRMs share similar architectures with conventional LLMs, existing safety alignment pipelines developed for LLMs can, in principle, be adapted to LRMs, such as supervised fine\-tuning \(SFT\)\[[47](https://arxiv.org/html/2609.27763#bib.bib17)\], reinforcement learning from human feedback \(RLHF\)\[[30](https://arxiv.org/html/2609.27763#bib.bib16)\], and direct preference optimization \(DPO\)\[[33](https://arxiv.org/html/2609.27763#bib.bib15)\]\. However, directly applying these methods often overlooks the explicit reasoning processes of LRMs, which can result in insufficient safety alignment, particularly when unsafe behavior is induced through manipulated reasoning traces\.

To address this limitation, several recent works focus on aligning the reasoning process itself\. One line of research constructs curated safety datasets and applies SFT to explicitly teach LRMs safe CoT reasoning\[[20](https://arxiv.org/html/2609.27763#bib.bib54),[43](https://arxiv.org/html/2609.27763#bib.bib31),[54](https://arxiv.org/html/2609.27763#bib.bib14),[63](https://arxiv.org/html/2609.27763#bib.bib6)\]\. Other approaches, such as SafePath\[[18](https://arxiv.org/html/2609.27763#bib.bib1)\], reduce harmful generation by encouraging the model to initiate reasoning with safety\-aware prompts \(e\.g\., “Let’s think about safety first”\)\. Beyond SFT\-based approaches, RL–based post\-training approaches have also been proposed to improve safety alignment\[[19](https://arxiv.org/html/2609.27763#bib.bib10)\]\. For example, STAIR\[[55](https://arxiv.org/html/2609.27763#bib.bib12)\]leverages DPO\[[33](https://arxiv.org/html/2609.27763#bib.bib15)\]to enhance safety alignment, while RECAP\[[32](https://arxiv.org/html/2609.27763#bib.bib42)\]introduces a variant of DAPO\[[52](https://arxiv.org/html/2609.27763#bib.bib11)\]to jointly improve safety and helpfulness\. Despite their effectiveness on standard harmful prompts, these methods are primarily trained on in\-distribution data and do not explicitly account for adversarial scenarios in which prompts are prepended with few\-shot conversations containing counter\-aligned reasoning content\. Consequently, their robustness against our SRCF attacks in few\-shot conversational settings remains limited\.

Beyond training and post\-training approaches, several works have explored inference\-time defenses that aim to improve the safety of large language models without modifying model parameters\. For example, SafeRemind\[[22](https://arxiv.org/html/2609.27763#bib.bib5)\]proposes a decoding\-time defense that dynamically injects safety\-reminding phrases into intermediate reasoning steps\. Similarly,\[[53](https://arxiv.org/html/2609.27763#bib.bib4)\]shows that inference\-time scaling of reasoning can improve the safety and adversarial robustness of LRMs\. While these approaches can mitigate unsafe behavior at inference time, they typically incur substantial increases in inference cost and do not fundamentally eliminate the underlying vulnerability\. Moreover, their effectiveness relies on controlled inference procedures and is difficult to guarantee in open\-source or unrestricted deployment settings\.

In addition to safety, LRMs are also expected to maintain helpfulness during alignment\[[3](https://arxiv.org/html/2609.27763#bib.bib9),[6](https://arxiv.org/html/2609.27763#bib.bib8)\]\. Recent studies have highlighted a trade\-off between safety alignment and reasoning ability or helpfulness\[[45](https://arxiv.org/html/2609.27763#bib.bib7),[15](https://arxiv.org/html/2609.27763#bib.bib48),[61](https://arxiv.org/html/2609.27763#bib.bib23),[6](https://arxiv.org/html/2609.27763#bib.bib8)\], showing that downstream fine\-tuning can significantly degrade reasoning performance or increase over\-refusal on benign inputs\. Prior reasoning\-alignment methods\[[20](https://arxiv.org/html/2609.27763#bib.bib54),[43](https://arxiv.org/html/2609.27763#bib.bib31),[54](https://arxiv.org/html/2609.27763#bib.bib14),[18](https://arxiv.org/html/2609.27763#bib.bib1)\]largely focus on safety enforcement and typically do not explicitly evaluate or improve helpfulness on benign queries\. While some recent approaches, such as RECAP\[[32](https://arxiv.org/html/2609.27763#bib.bib42)\], attempt to balance safety and helpfulness through mixed training data, they remain focused on in\-distribution alignment scenarios\.

In contrast, our work introduces a general post\-training framework that explicitly targets the adversarial generalization issue failures induced by counter\-aligned few\-shot conversations\. Unlike prior alignment methods that focus on in\-distribution harmful prompts, our framework jointly improves safety and reduces over\-refusal while preserving general utility by aligning both the model’s reasoning behavior and final responses under such adversarial settings\. Importantly, our framework can combine with existing post\-training techniques and can be seamlessly integrated with a wide range of alignment methods\.

## Appendix LLimitations

Our study focuses on LRMs deployed with flexible conversational interfaces that allow users to prepend prior conversational history containing explicit reasoning traces\. This interface is central to SRCF: the attack requires counter\-aligned CoTs to be inserted as part of previous assistant turns\. Therefore, SRCF may be less applicable to systems that strictly hide, filter, or disallow user\-provided reasoning traces in conversation history\. However, this limitation does not remove the broader concern, since many open\-source LRMs and customizable deployment pipelines expose flexible chat templates where such histories can be constructed\.

Figure 12:Refusal\-style classification policy used for over\-refusal evaluation\. The policy assigns each final model response to Full Refusal, Partial Refusal, or Non\-Refusal based solely on the presence of refusal\-like language, independent of topic safety or content correctness\. Refusal Rate \(RR\) counts both Full and Partial Refusals on benign prompts\.Figure 13:Rule\-based safety classification policy for jailbreak evaluation\. Each generated chain\-of\-thought \(CoT\) and final response is evaluated separately against a fixed set of safety rules and classified as Safe or Unsafe based on the presence of any rule violations\.Figure 14:Prompting instruction used to generate counter\-aligned CoT conversations\.We use the above instruction to guide a separate LLM \(DSLLaMA3\-70B\) to synthesize counter\-aligned reasoning traces for both harmful prompts\.The instruction explicitly controls the stance and failure mode of the generated CoTs to induce unsafe reasoning \(jailbreak\)\.