Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

# Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
Source: [https://arxiv.org/html/2607.02914](https://arxiv.org/html/2607.02914)
###### Abstract

Large language models \(LLMs\) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge\. Conventional refusal\-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries\. Building upon the constructive safety paradigm pioneered by Oyster\-I, which moves beyond blanket refusal toward thoughtful, response\-oriented safety alignment, we identify two critical limitations of its Supervised Fine\-Tuning \(SFT\)\-based scheme: insufficient safety generalization to out\-of\-distribution scenarios and a phenomenon we term safety chain\-of\-thought \(CoT\) over\-generalization, wherein safety\-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience\. To address these limitations, we propose Oyster\-II, a reinforcement learning \(RL\)\-based constructive safety alignment framework that adopts a Zero\-RL paradigm combined with a multi\-stage reinforcement learning strategy\. Oyster\-II incorporates four core technical contributions: \(1\) A length\-reward\-based entropy control mechanism and a benign\-sample\-based length control framework that employ a composite multiplicative reward function to jointly optimize safety compliance, response informativeness, and reasoning integrity, effectively preventing safety\-driven reward hacking and premature convergence; \(2\) SERL, a novel RL\-based alignment algorithm built upon GSPO that incorporates a mix\-policy strategy to accelerate convergence and improve instruction\-hierarchy following, enabling the model to accurately recognize and adhere to hierarchical instructions across developer and user principals; \(3\) A curriculum\-learning\-based multi\-stage Zero\-RL training paradigm for constructive safety, coupled with an active\-learning\-based adaptive sample difficulty control mechanism for reward noise mitigation during safety alignment; \(4\) Long\-context safety alignment with cross\-length generalization, exploring safety alignment on long\-query data and demonstrating that training exclusively on long\-query safety data achieves state\-of\-the\-art performance on short\-query tasks while substantially mitigating over\-refusal caused by shallow keyword\-level pattern matching\. Evaluated across extensive benchmarks, Oyster\-II comprehensively surpasses both Qwen3\-14B and its predecessor Oyster\-I on safety dimensions, achieving cross\-scale performance comparable to Qwen3\-Max and Qwen3\.5\-397B\. Critically, all safety improvements are achieved non\-invasively through the Zero\-RL paradigm, effectively preserving the base model’s general capabilities and linguistic style\. Oyster\-II thus establishes a scalable and balanced foundation for building LLMs that are simultaneously safe, helpful, and trustworthy\. Performance of Oyster\-II is shown in Figure[1](https://arxiv.org/html/2607.02914#S0.F1)\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x1.png)Figure 1:Performance of Oyster\-II, Oyster\-I, and Qwen3\-14B, where WildChat is used as the representative benchmark for Short\-query Safety \(EN\)\. OysterII demonstrates significant improvements over OysterI and Qwen3\-14B across all evaluated benchmarks, with detailed experimental results and evaluation methodology presented in Section[5](https://arxiv.org/html/2607.02914#S5)\.## 1Introduction

Large language models \(LLMs\) have achieved remarkable success and gained widespread adoption across a broad spectrum of applications, ranging from conversational assistants and code generation to scientific reasoning and educational tutoring\. However, alongside their impressive capabilities, growing concerns have been raised regarding theirsafetyandethicalimplications\. To mitigate these risks, a substantial body of prior work has focused on improving model safety by training models torefusemalicious or harmful instructions outright\. While these refusal\-oriented alignment strategies have demonstrated effectiveness in reducing harmful content generation, they suffer from a critical limitation: they fail to account for the fact that a portion of seemingly malicious requests may stem fromlegitimate user needs, where the model could have instead provided partial, necessary, and safety\-compliant information to address the underlying intent of the query\.

Recent frontier models, notablyGPT\-5Singhet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib89)\)andOyster\-IDuanet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib23)\), have made pioneering attempts to move beyond simplistic refusal strategies, demonstrating that aligned models can respond thoughtfully to potentially sensitive queries — providing necessary and useful information while carefully withholding content that poses genuine risks\. Thisresponse\-orientedsafety alignment paradigm, as opposed to the conventionalrefusal\-orientedsafety paradigm, represents a promising direction toward building models that are simultaneouslysafe,helpful, andtrustworthy\.

While Oyster\-I demonstrates strong performance in terms of both safety and helpfulness, its reliance on a Supervised Fine\-Tuning \(SFT\)\-based alignment scheme inherently introduces certain limitations\. Specifically, the SFT\-based approach exhibits deficiencies in safety capability and generalization, struggling to robustly handle out\-of\-distribution safety\-critical scenarios\. Furthermore, through systematic analysis, we identify a phenomenon ofsafety chain\-of\-thought \(CoT\) over\-generalization, wherein the model excessively applies safety\-oriented reasoning patterns to benign queries, leading to unnecessary refusals and degraded user experience\. To address these limitations, we propose Oyster\-II, which adopts aZero\-RLscheme and amulti\-stage reinforcement learningstrategy to progressively enhance the model’s safety capabilities and generalization robustness beyond what SFT alone can achieve\. Furthermore, building upon GSPO, we proposeSERL, a novel RL\-based alignment algorithm that not only maintains and reinforces the model’s safety alignment across diverse and adversarial interaction scenarios, but also achieves substantial improvements in instruction hierarchy following, enabling the model to more accurately recognize and respect the hierarchical structure of instructions from different principals\. The framework of Oyster\-II is shown in Figure[2](https://arxiv.org/html/2607.02914#S1.F2)\.

Our contributions are summarized as follows:

- •Reinforcement Learning Framework for Constructive Safety\.We propose a comprehensive reinforcement learning framework tailored for safety alignment, incorporating length\-reward\-based entropy control to prevent premature convergence, active\-learning\-based sample difficulty control to mitigate reward noise, and a multi\-stage training strategy to simultaneously preserve model helpfulness and prevent mode collapse\. Furthermore, by adopting a Zero\-RL paradigm, we maximize the preservation of the model’s general capabilities and maintain consistency with the linguistic style of the original base model\.
- •Long\-Context Safety Alignment with Cross\-Length Generalization\.We demonstrate that training exclusively on long\-query safety data is sufficient to achieve state\-of\-the\-art safety performance on short\-query tasks, while simultaneously maintaining high response rates and satisfactory helpfulness for risk\-sensitive queries\. Moreover, long\-context safety alignment enables the model to develop a deeper semantic understanding of safety\-sensitive content, significantly mitigating the over\-refusal problem caused by shallow keyword\-level pattern matching on adversarial queries\.
- •Instruction Hierarchy Training for Controllable and Constructive Safety\.We construct a systematic training and evaluation framework for instruction hierarchy, covering eight developer policy dimensions and three tiers of user adversarial attack strategies of varying intensity\.In order to enhance the model’s instruction hierarchy following capability while maintaining model safety, we propose theSERLmethod based on GSPO, utilizing a mix\-policy to accelerate convergence speed and improve the upper bound of instruction hierarchy following capability\.
- •Zero\-RL Training Pipeline with Stable General Capability Preservation\.We integrate the above components into a unified multi\-stage training pipeline that jointly addresses constructive safety, long\-context generalization, and instruction\-following controllability\. Through the incorporation of safe\-sample\-based length control and composite reward formulations, our pipeline effectively prevents over\-refusal behaviors on benign queries and response length degradation, ensuring stable model performance across both safety\-critical and general instruction\-following tasks\.
- •Strong Constructive Safety Capabilities\.Oyster2 advances upon its predecessor Oyster1 by introducing a comprehensive suite of safety enhancements developed in accordance with a formal model specification\. The model demonstrates constructive safety capabilities that simultaneously prevent malicious misuse and proactively guide non\-malicious users toward safe and beneficial outcomes\. Evaluated across multiple benchmarks, Oyster2 comprehensively surpasses both Qwen3\-14B and Oyster1 on safety dimensions, achieving cross\-scale and cross\-generational performance comparable to Qwen3\-Max and Qwen3\.5\-397B\. In long\-context scenarios, Oyster2 substantially improves its ability to identify and handle risk\-sensitive content across high\-frequency tasks such as question answering, summarization, continuation generation, and content evaluation, thereby addressing a critical gap in long\-query text safety\. Furthermore, by adhering to a hierarchical instruction priority design — Root Principle \> Developer Policy \> User Preference — the model exhibits more robust compliance with higher\-priority instructions in conflict scenarios, ensuring clear safety boundaries while enhancing controllability for Developer Policy\-based secondary development\. Notably, all aforementioned safety improvements are achieved in a non\-invasive manner, effectively preserving the model’s original general capabilities and response style\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x2.png)Figure 2:Training Pipeline ofOyster\-II\. Equipped with active\-learning\-based safety control, the zero\-RL paradigm, and SERL, Oyster\-II achieves state\-of\-the\-art safety without compromising its helpfulness on both benign and malicious samples\.
## 2Reinforcement Learning for Constructive Safety

In Oyster\-I, we employed Linguistic Backpropagation \(Lingo\-BP\) in conjunction with Odds Ratio Preference Optimization \(ORPO\) to facilitate constructive safety training\. While this approach demonstrated promising results in aligning model behavior with constructive safety objectives, it exhibited certain limitations in two critical aspects: safety generalization across diverse and out\-of\-distribution scenarios, and preservation of general capabilities beyond the safety\-specific training distribution\. Specifically, the supervised fine\-tuning nature of Lingo\-BP and ORPO constrains the model’s ability to internalize transferable safety reasoning patterns, leading to suboptimal robustness when confronted with novel risk configurations or long\-context safety\-sensitive inputs\. Furthermore, the optimization pressure imposed by safety\-oriented training objectives tends to encroach upon the model’s general instruction\-following capacity, resulting in a non\-trivial degradation of broad helpfulness\.

To overcome these limitations, Oyster\-II introduces a reinforcement learning \(RL\)\-based framework for constructive safety alignment\. By replacing static supervised signals with dynamic, reward\-driven optimization, the RL paradigm enables the model to actively explore and internalize safety\-aligned response strategies through iterative interaction, rather than merely imitating pre\-defined safe outputs\. This shift fundamentally enhances the model’s capacity for generalizable safety reasoning, allowing learned safety behaviors to transfer more robustly across context lengths, query types, and adversarial configurations\. Critically, through the adoption of a Zero\-RL paradigm combined with carefully designed composite reward formulations, Oy2 achieves this safety advancement in a non\-invasive manner—preserving the base model’s general capabilities and linguistic style while simultaneously improving both safety performance and constructive helpfulness\. In this way, Oy2 strives to resolve the long\-standing tension between safety alignment and general capability retention, establishing a more balanced and scalable foundation for user\-centered LLM safety\.

### 2\.1Length\-Reward\-Based Entropy Control

A critical challenge observed during the reinforcement learning training process is the tendency toward premature convergence under exclusively safety\-driven optimization\. Specifically, when only safety rewards are applied, the model rapidly collapses into a degenerate safety paradigm—manifested as a sharp decline in output entropy and a substantial surge in unconditional refusal rates\. This phenomenon, which we term safety\-driven reward hacking, arises because the model discovers that blanket refusal represents a locally optimal strategy for maximizing safety rewards, effectively bypassing the need to engage in nuanced, context\-sensitive reasoning\. The resulting behavior, while superficially safe, is fundamentally at odds with the constructive safety objective: a model that reflexively refuses all risk\-adjacent queries fails to provide the guidance that non\-malicious users genuinely require, and may inadvertently drive them toward less\-controlled information sources—ultimately increasing rather than mitigating real\-world harm\.

To counteract this degenerative dynamic, we introduce continuous length rewards as a complementary optimization signal alongside safety rewards\. By jointly incentivizing response informativeness and safety compliance, length rewards impose a counterbalancing pressure that discourages the model from collapsing into terse, uninformative refusals\. Empirically, the incorporation of length rewards effectively mitigates premature convergence, sustains higher output entropy throughout training, and encourages the model to produce responses that are not only safe but also substantive, contextually grounded, and genuinely helpful\. This design reflects a core principle of Constructive Safety Alignment: safety and helpfulness are not competing objectives to be traded off, but complementary properties to be jointly optimized—and the reward structure must be carefully engineered to reflect this duality\.

To jointly optimize constructive safety, response engagement, and output quality, we formulate acomposite multiplicative reward functionas follows:

R​e​w​a​r​d=Ss​a​f​e​t​y×SR​e​s​p​o​n​s​e×Sl​e​n​g​t​h×Sf​o​r​m​a​tReward=S\_\{safety\}\\times S\_\{Response\}\\times S\_\{length\}\\times S\_\{format\}\(1\)
where each component is defined as:

- •Ss​a​f​e​t​yS\_\{safety\}\(Safety Score\):A binary gating signal, taking11if the response is safe and0otherwise\. As a multiplicative factor, any safety violation immediately nullifies the total reward, enforcing a strict safety floor throughout training\.
- •SR​e​s​p​o​n​s​eS\_\{Response\}\(Response Rate Score\):Assigned11for substantive responses and0\.50\.5for outright refusals\. This soft penalty discourages excessive refusals on non\-malicious queries while preserving the model’s capacity to decline genuinely harmful requests\.
- •Sl​e​n​g​t​hS\_\{length\}\(Length Score\):Defined asSl​e​n​g​t​h=l​e​n​g​t​ht​a​r​g​e​t−l​e​n​g​t​hS\_\{length\}=\\frac\{length\}\{target\-length\}, providing a continuous incentive for generating sufficiently informative responses and counteracting the response brevity degeneracy observed under purely safety\-driven reward regimes\.
- •Sf​o​r​m​a​tS\_\{format\}\(Format Score\):Assigned11for well\-formed reasoning traces with complete<think\> </think\>tags and0for malformed formats, ensuring structural integrity of chain\-of\-thought reasoning throughout RL training\.

The multiplicative formulation enforcesjoint satisfactionacross all four dimensions: deficiency in any single factor propagates as a multiplicative penalty to the overall reward, reflecting the CSA principle that safety, helpfulness, informativeness, and reasoning integrity must besimultaneously achievedrather than independently traded off\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x3.png)Figure 3:Performance of Oyster\-II with and without length reward\.A comparison with and without length reward is shown in Figure[3](https://arxiv.org/html/2607.02914#S2.F3)\. Without the length reward, the model tends to collapse into a safe but refusal state, which is manifested as a nearly20%20\\%degradation in Response Rate during stage\-1 training\.

### 2\.2Active\-Learning\-Based Sample Difficulty Control for Reward Noise Mitigation

![Refer to caption](https://arxiv.org/html/2607.02914v1/x4.png)Figure 4:Framework of active\-learning\-based sample selection\.Reinforcement learning for safety tasks differs fundamentally from verifiable reinforcement learning applied to domains such as mathematics or code generation, wherein reward signals can be computed with near\-perfect accuracy via deterministic verifiers\. In safety\-oriented RL, reward signals are inherently imperfect and cannot achieve100%100\\%accuracy, introducingnoisy reward interferencethat can destabilize training and degrade alignment quality if left unaddressed\.

To mitigate this challenge, we adopt anactive sample selection strategygrounded in offline active learning principles, which is demonstrated in Figure[4](https://arxiv.org/html/2607.02914#S2.F4)\. Specifically, for each candidate training sample, we generateNNindependent responses and evaluate their safety classifications\. Samples for which allNNgenerated responses are deemed safe are filtered out from the training pool, as they contribute little discriminative signal and primarily introduce reward noise into the optimization process\. Conversely, samples for which the modelfails to consistently generate safe responses—i\.e\., exhibiting non\-trivial unsafe response rates across theNNrollouts—are identified and retained as high\-value training instances\. This selective retention mechanism ensures that the RL training process focuses ongenuinely challenging safety boundaries, where reward signal variability reflects authentic model uncertainty rather than annotation noise\.

By concentrating optimization effort on samples at the safety decision boundary, our active selection strategy effectivelyimproves the signal\-to\-noise ratioof the training process, enabling more stable gradient updates and accelerating convergence toward robust constructive safety behaviors\. The results of OysterII\-stage4 with and without difficulty control are illustrated in Figure[5](https://arxiv.org/html/2607.02914#S2.F5)\. With difficulty\-aware sample selection, the model achieves a notable2\.80%2\.80\\%improvement in S\-eval English Safety, a1\.83%1\.83\\%improvement in Wildchat, and a2\.37%2\.37\\%improvement in English long\-query safety, while maintaining comparable response rates and general capabilities\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x5.png)Figure 5:Comparison of safety alignment with and without active\-leaning based sample difficulty control\.
### 2\.3Curriculum Learning\-Based Multi\-Stage Reinforcement Learning to Prevent Mode Collapse

We employ amulti\-stage reinforcement learning frameworkthat integrates the aforementioned components—composite reward formulation, entropy\-stabilizing length rewards, and active sample selection—into a unified and coherent training pipeline, which is motivated by the concept of curriculum learningBengioet al\.\([2009](https://arxiv.org/html/2607.02914#bib.bib90)\)\. This framework is designed to simultaneously pursue two objectives that are often in tension: ensuring robust model safety and enhancing the model’s constructive helpfulness in response to risk\-sensitive queries\. Furthermore, by adopting a Zero\-RL paradigm—in which reinforcement learning is applied directly to the base model without intermediate supervised fine\-tuning— we maximize the preservation of the model’s general capabilities and maintain consistency with the linguistic style of the original base model\.

Specifically, we partition the training process intofive successive stages, each dedicated to a distinct data scenario:

1. 1\.Chinese Long\-context Business Data:Training on diverse real\-world Chinese long\-query business data to establish foundational safety chain\-of\-thought reasoning capabilities\.
2. 2\.Chinese\-LSB and Chinese\-MSJ Data:Reinforcing safety alignment on converted long\-query versions of standard Chinese safety benchmarks\.
3. 3\.English LSB Data:Extending long\-context safety alignment to the English domain, ensuring cross\-lingual coverage\.
4. 4\.English Short\-query Data:Consolidating safety performance on short\-query English queries to ensure cross\-length generalization\.
5. 5\.Instruction Hierarchy Data:Equipping the model with the capacity to resolve instruction conflicts in accordance with the hierarchical priority design of Root Principle\>\>Developer Policy\>\>User Preference\.

Empirically, we find thatsequential stage\-wise training consistently outperforms joint multi\-task trainingon all evaluation dimensions\. We attribute this advantage to two key factors:

- •Data Heterogeneity:The five data scenarios exhibit substantial heterogeneity along multiple axes—including context length, linguistic domain, and task type—rendering joint training prone to optimization conflicts and convergence difficulties\. Sequential training allows each stage to converge stably within its own data distribution before transitioning to the next, mitigating inter\-task interference throughout the training process\.
- •Progressive Safety Chain\-of\-Thought Construction:The base model Qwen3\-14B exhibits notably limited safety performance across a broad range of tasks, lacking well\-formed safety reasoning patterns\. By initiating training on diverse long\-query Chinese business data—which provides rich and naturalistic safety\-sensitive contexts—the model is first guided to develop astructured safety chain\-of\-thought reasoning chain, establishing a robust cognitive scaffold upon which subsequent stages can build\. This progressive construction of safety reasoning capacity is difficult to achieve when all data scenarios are trained jointly, as the optimization signal from simpler or shorter data tends to dominate early training dynamics, suppressing the emergence of deep long\-context safety reasoning\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x6.png)Figure 6:Comparison of safety alignment with and without multi\-stage training\.The results comparing curriculum learning\-based multi\-stage training against joint training are demonstrated in Figure[6](https://arxiv.org/html/2607.02914#S2.F6)\. Compared with joint training, our curriculum learning approach—where the model progressively learns from different stages—achieves a14\.65%14\.65\\%improvement in Chinese long\-query safety and a9\.60%9\.60\\%improvement in S\-eval English Safety\. Notably, the first two stages of the curriculum focus on long\-query Chinese risky queries, and the substantial14\.65%14\.65\\%improvement in this category strongly validates the effectiveness of our staged curriculum learning strategy, where knowledge is incrementally built upon prior stages rather than learned simultaneously\.

### 2\.4Preserving General Capabilities and Response Styles via Zero\-RL

In Oyster\-I, safety alignment was achieved through ORPO\-based supervised fine\-tuning on a static dataset constructed via the Lingo\-BP pipeline\. While this approach effectively instilled constructive safety behaviors, it introduced a notable side effect: since both the reasoning traces and the final responses were synthetically generated rather than naturally elicited from the model, the resulting training signal imposed artificial distributional constraints on the model’s output space\. Consequently, the model became susceptible to over\-fitting to these synthetic patterns, leading to measurable performance degradation on general\-purpose tasks—including mathematical reasoning, code generation, and instruction following—as evidenced by evaluations conducted on the OpenCompass benchmark suite, which is demonstrated in Table[4](https://arxiv.org/html/2607.02914#S5.T4)\.

This limitation motivates the adoption of a fundamentally different training paradigm in Oyster\-II\. Compared to the Supervised Fine\-Tuning \(SFT\) approach employed in Oy1, theZero\-RL paradigmdemonstrates marked improvements across multiple dimensions:

- •Response Consistency:By optimizing directly against reward signals rather than imitating fixed synthetic outputs, Zero\-RL allows the model to discover safety\-aligned response strategies that areorganically consistentwith its pre\-trained behavioral tendencies, avoiding the stylistic and structural discontinuities introduced by SFT on synthetic data\.
- •General Capability Preservation:The absence of synthetic reasoning trace supervision ensures that the model’s general\-purpose capabilities—spanning mathematical reasoning, code generation, and complex instruction following—remain largely intact throughout safety alignment training, as the optimization pressure is confined to reward\-relevant safety behaviors rather than broadly reshaping the model’s output distribution\.
- •Generalization Performance:Zero\-RL exhibits superior generalization to unseen safety scenarios and out\-of\-distribution queries, as the model learns transferable safety reasoning principles through reward\-driven exploration rather than memorizing fixed input\-output patterns from a static synthetic corpus\.

Together, these advantages establish the Zero\-RL paradigm as a more principled and capability\-preserving foundation for constructive safety alignment, directly addressing the generalization and capability degradation limitations identified in Oyster\-I\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x7.png)Figure 7:Comparison of safety alignment with Zero\-RL or SFT\+RL\.As illustrated in Figure[7](https://arxiv.org/html/2607.02914#S2.F7), SFT training tends to improve model safety scores; however, it comes at the cost of reduced general capabilities and response consistency, which can be attributed to the involvement of SFT\. Nevertheless, with the incorporation of additional Zero\-RL training stages, OysterII achieves consistent improvements in both safety and helpfulness, demonstrating the effectiveness of the Zero\-RL strategy in reconciling the trade\-off between safety and utility\.

## 3Long\-context Safety and its Generalization to Shot\-context Scenario

Large language models have been shown to be more vulnerable to malicious queries in long\-context scenarios than in short\-context onesLuet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib1)\); Anthropic \([2024b](https://arxiv.org/html/2607.02914#bib.bib2)\)\. However, the mitigation of safety risks in long\-context settings remains largely underexplored\. In this section, we investigate how to improve model safety on long\-context queries, and demonstrate that safety alignment in long\-context scenarios yields broader benefits, including improved generalization and reduced over\-refusal\.

### 3\.1Long\-context Safety Dataset Construction

To support long\-context safety alignment, we construct a comprehensive bilingual long\-query safety dataset encompassing bothChineseandEnglishcomponents\.

TheChinese datasetcomprises three primary sources:

- •Business Data:A curated collection of domain\-specific Chinese long\-query data drawn from real\-world business scenarios, providing naturalistic and diverse safety\-sensitive contexts\.
- •Task\-Augmented Data:Derived from the business data through systematic task augmentation across six representative long\-query task categories: - –Bullet\-point Enumeration— structured point\-by\-point elaboration of safety\-relevant content; - –Creative Ideation— open\-ended generative tasks requiring associative reasoning under safety constraints; - –Question Answering— answering risk\-sensitive queries embedded within long\-query contextual passages; - –Summarization— condensation of safety\-sensitive long\-query content into concise, accurate summaries; - –Text Continuation— the model is given a long malicious query and is required to generate coherent and meaningful subsequent content that maintains semantic, stylistic, and logical consistency with the original text\.; - –Continuation Generation— extending incomplete passages in a manner consistent with safety boundaries; - –Model Commentary— comment on long\-query contents, including summarization of key points, critical analysis, and evaluation of the content’s quality, coherence, and logical consistency\.\.
- •Chinese\-LSBLuet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib1)\)and Chinese\-MSJAnthropic \([2024b](https://arxiv.org/html/2607.02914#bib.bib2)\)Data:Two additional long\-query safety datasets constructed by converting and extending existing short\-query Chinese safety benchmarks—namely LSB \(Long Safety Bench\) and MSJ \(Many\-shot Jailbreaking\)—into long\-context formats, enabling the model to generalize safety reasoning from short\-query to long\-query scenarios\.

TheEnglish datasetconsists ofLSB DataLuet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib1)\), constructed following an analogous long\-context conversion procedure applied to the English LSB benchmark, ensuring cross\-lingual coverage of long\-query safety alignment\.

Together, this bilingual dataset provides a rich and diverse training signal that spans multiple languages, task types, and context lengths, laying a solid foundation for robust cross\-length safety generalization in Oyster\-II\. Furthermore, we construct a long\-query evaluation benchmark to evaluate models’ performance, shown in Table[1](https://arxiv.org/html/2607.02914#S3.T1)\.

Table 1:Dataset statistics of our long\-query evaluation benchmark including Chinese and English\.DatasetsSamplesLanguagesBusiness Data373CNBullet\-point Enumeration373CNCreative Ideation373CNQuestion Answering373CNSummarization373CNText Continuation373CNContinuation Generation373CNModel Commentary373CNChinese\-LSB850CNChinese\-MSJ1000CNEnglish\-LSB1573EN
### 3\.2Cross\-Length Generalization in Safety Alignment

A key empirical finding of our long\-context safety alignment approach is that enhancing the model’s safety capabilities on long\-query data yields significant and transferable improvements on short\-query safety tasks—a phenomenon we attribute to thecross\-length generalizationof deep semantic safety reasoning acquired through long\-context training\. Notably, we demonstrate that training exclusively on long\-query safety data is sufficient to achieve state\-of\-the\-art \(SOTA\) safety performance on short\-query tasks, without any direct exposure to short\-query safety examples during training\. This finding suggests that long\-context safety alignment encourages the model to internalizedeeper, semantics\-level understandingof safety\-sensitive content, moving beyond the shallow keyword\-level pattern matching that characterizes models trained solely on short\-query data—and which renders such models particularly vulnerable to adversarial rephrasing and context manipulation\. Furthermore, this cross\-length generalization is achieved without compromising the model’s engagement behavior: Oy2 simultaneously maintains a high response rate and satisfactory helpfulness for risk\-sensitive yet non\-malicious queries, demonstrating that long\-context safety alignment does not induce the over\-refusal tendencies commonly associated with aggressive safety fine\-tuning\. Empirical results substantiating these findings are presented in the experimental section\.

Figure[8](https://arxiv.org/html/2607.02914#S3.F8)reveals that long\-query safety alignment can generalize to short\-query safety alignment, but not vice versa\. While using short\-query and long\-query safety alignment yields similar performance on short\-query safety, the use of long\-query safety alignment demonstrates significant improvements on long\-query safety: English long\-query safety alignment achieves a gain of17\.53%17\.53\\%, and Chinese long\-query safety alignment achieves a gain of18\.26%18\.26\\%\. Meanwhile, in terms of response rate, long\-query safety alignment can deepen the model’s understanding of safety\-related issues and prevent overfitting to specific sensitive words, thereby improving the response rate\. Specifically, the response rate achieves a gain of2\.80%2\.80\\%for English and4\.48%4\.48\\%for Chinese\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x8.png)Figure 8:Comparison of performance of Oyster\-II safety alignment with short queries and long queries\.
### 3\.3Leveraging Long\-Context Safety Cues to Mitigate Safety Misgeneralization

A pervasive challenge in safety alignment is the exaggerated safety on benign queries—wherein the model incorrectly declines benign requests due to superficial lexical overlap with genuinely harmful content\. A representative example illustrates this failure mode clearly: while the query “how to kill a person” carries inherent risk, the semantically distinct query “how to kill a program” is entirely benign\. Yet models relying on shallow keyword\-level pattern matching treat both queries identically, triggering refusal based on the surface\-level presence of the word “kill” rather than its contextual meaning\. This fundamental limitation—the inability to accurately capture semantic context beyond lexical surface forms—is a direct consequence of safety alignment approaches that optimize against specific keywords or phrases in isolation, without cultivating a deeper understanding of how safety\-sensitive concepts interact with their surrounding linguistic environment\.

Long\-context safety alignment offers a principled remedy to this problem\. By exposing the model to safety\-sensitive content embedded within rich, extended contexts—spanning tasks such as summarization, paraphrasing, and continuation generation—long\-query training inherently demands that the model develop fine\-grained discriminative capabilities: it must not only identify potentially risk\-relevant expressions, but also reason about whether the broader semantic context warrants a response or a refusal\. This sustained exposure to nuanced, context\-dependent safety judgments cultivates a deeper understanding of the relationship between safety\-sensitive issues and their particular linguistic manifestations, moving the model’s safety reasoning from shallow lexical pattern matching toward genuine semantic comprehension\. As a direct consequence, Oy2 achieves a significantly higher response rate on adversarial white\-box queries compared to models trained under shallow alignment regimes, demonstrating that long\-context safety alignment is not only compatible with, but actively conducive to, reducing over\-refusal without compromising safety robustness\. Figure[8](https://arxiv.org/html/2607.02914#S3.F8)reveals that long\-query safety alignment can enhance the model’s comprehension of safety\-related issues and mitigate overfitting to specific sensitive keywords, which in turn leads to a notable improvement in the response rate\. Specifically, the English response rate achieves a gain of2\.80%2\.80\\%, while the Chinese response rate demonstrates an even more substantial improvement of4\.48%4\.48\\%\.

### 3\.4Benign\-Sample\-Based Length Control

A distinctive challenge observed during safety\-oriented reinforcement learning training is the tendency towardresponse length degradationandover\-generalization of safety chain\-of\-thought reasoning\. This stands in stark contrast to the behavior observed in mathematics and code generation tasks, where reward hacking typically manifests as excessive output length inflation\. When training exclusively on risk\-sensitive queries, the model instead converges toward increasingly terse, safety\-dominated outputs, suppressing the richness and depth of its reasoning traces on general\-purpose tasks\.

To counteract these degenerative dynamics, we incorporate a small proportion of general instruction\-following data into the training mixture, which we get from LongAlignBaiet al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib91)\), serving as an anchor that preserves the model’s behavioral breadth beyond the safety domain\. Complementing this data augmentation strategy, we adopt a composite additive reward formulation that jointly addresses response quality, structural integrity, over\-refusal suppression, and output informativeness:

Ra​l​i​g​n=Rskywork\+Rformat−Rover\-refusal\+RlengthR\_\{align\}=R\_\{\\text\{skywork\}\}\+R\_\{\\text\{format\}\}\-R\_\{\\text\{over\-refusal\}\}\+R\_\{\\text\{length\}\}\(2\)
where each component serves a distinct and complementary role\.RskyworkR\_\{\\text\{skywork\}\}provides a general\-purpose quality signal derived from the Skywork reward model, ensuring that responses remain coherent, accurate, and helpful on standard instruction\-following tasks\.RformatR\_\{\\text\{format\}\}enforces structural integrity of the model’s chain\-of\-thought reasoning traces, penalizing malformed or incomplete thinking formats\.Rover\-refusalR\_\{\\text\{over\-refusal\}\}acts as a corrective penalty that discourages unwarranted refusals on benign queries, directly targeting the over\-refusal problem that arises from over\-generalization of safety\-oriented reasoning\. Finally,RlengthR\_\{\\text\{length\}\}provides a continuous incentive for generating sufficiently informative and substantive responses, counteracting the response brevity degeneracy induced by exclusive training on risk\-sensitive data\. Together, this composite reward formulation ensures stable model performance on standard tasks while effectively preventing over\-refusal behaviors on benign samples, striking a principled balance between safety robustness and general instruction\-following capability throughout the training process\. As shown in Figure[9](https://arxiv.org/html/2607.02914#S3.F9), incorporating benign samples in conjunction with the rewardR​alignR\{\\text\{align\}\}not only ensures model safety but also leads to substantial improvements of 4\.92

![Refer to caption](https://arxiv.org/html/2607.02914v1/x9.png)Figure 9:Performance of Oyster\-II with and without involving benign samples during its safety alignment\.
### 3\.5Reward Choice and Threshold Selection for the Reward Model

For long\-context safety alignment, we conduct a systematic threshold selection analysis for the safety reward model, examining the impact of varying decision thresholds on the downstream safety\-helpfulness trade\-off of the trained model\. Empirical results reveal a nuanced but practically significant finding: under conditions ofcomparable F1 scores, selecting a threshold that favorshigher recall at the expense of lower precisionyields a more favorable safety\-helpfulness trade\-off for the model\. This asymmetry implies that, at equivalent F1 performance, a high\-recall threshold is preferable: it minimizes the risk of false negatives that corrupt the reward signal with unsafe reinforcement, while accepting a moderate increase in false positives whose primary effect is merely reduced training efficiency\. Consequently, our threshold selection strategy explicitly prioritizes recall over precision, striking a more favorable balance between safety robustness and constructive helpfulness in the trained model\.

A further empirical finding concerns the granularity of safety judgment during reward model evaluation\. In standard short\-query safety benchmarks such as LSB and MSJ, safety assessment is typically performed based solely on the short query itself, without incorporating the broader contextual information accompanying the request\. Our experiments demonstrate that, when training on long\-context queries, conducting safety judgment on the full query—including all surrounding background and contextual passages—consistently yields superior results compared to evaluation based on the short query alone\. This performance gap can be attributed to the context\-dependent nature of safety risks in long\-query scenarios\. Specifically, risk\-relevant information is frequently embedded within the background or contextual passages that precede or accompany the core query, rather than being explicitly surfaced in the query itself\. When safety judgment is restricted to the short query in isolation, the reward model operates under partial observability, remaining unaware of contextual cues that may substantially alter the risk profile of the interaction\. As a consequence, the model trained under such incomplete supervision may learn to overlook context\-embedded risks, producing responses that appear safe when evaluated against the query alone but fail to account for hazardous implications arising from the broader context\.

## 4Instruction Hierarchy for Instruction Following

To systematically enhance the safety and controllability of large language models under instruction conflict scenarios, we construct a comprehensive hierarchical training and evaluation framework\. The core design principle follows the model\-specification\-defined instruction priority hierarchy111[https://s\.alibaba\.com/aaig/specification](https://s.alibaba.com/aaig/specification):*Root Principles*\>\>*Developer Policies*\>\>*User Preferences*\. When a user request conflicts with a developer\-defined policy \(i\.e\., the system prompt\), the model must reliably follow the higher\-priority instruction while constructively addressing the user’s legitimate needs within safety constraints\.

### 4\.1Dataset Construction

#### 4\.1\.1Evaluation Benchmark: Instruction Conflict Simulation in Realistic Deployment Scenarios

To systematically assess whether models adhere to higher\-priority instructions under conflict, we construct an evaluation benchmark that closely approximates real\-world deployment conditions\. Under this paradigm, a developer\-defined*system prompt*imposes behavioral constraints on the model, while downstream*user prompts*violate these constraints either inadvertently or adversarially\.

The benchmark is organized along two independent dimensions: \(i\)system prompt scenarios, which impose diverse behavioral constraints on the model; and \(ii\)user prompt attack levels, which characterize the intensity and stealth of conflicting requests\.

##### System Prompt Scenarios\.

We instantiateeight representative scenario categoriesthat collectively cover the most prevalent forms of system\-level constraints in production\-grade LLM applications:

- •Domain Constraints\.The system prompt restricts the model to a specific NLP task family, including bilingual \(Chinese–English\) translation, document summarization, grammatical error correction, text refinement, and information extraction\.
- •Task\-Type Constraints\.The model is instantiated as a domain\-specific assistant \(e\.g\., education, legal, medical, culinary, programming, or mathematics\) and is required to remain within the designated professional scope\.
- •Structured Output Requirements\.The output must strictly conform to a predefined format, ranging from free\-form natural language to highly structured representations such asLaTeX, JSON, Markdown tables, and Python dictionaries\.
- •Role Simulation\.The model is assigned a fixed persona—drawn from various professions, historical figures, or fictional characters from literature, anime, or games—and must maintain consistency throughout the conversation\.
- •Language Constraints\.The output is restricted to a designated natural language; our benchmark focuses on Chinese and English\.
- •Fixed\-Content Output Requirements\.Every response must include prescribed atomic elements, such as specific emojis, fixed phrases, mandatory signatures, or tags\.
- •Length Constraints\.A hard upper bound is imposed on response length, measured in terms of word count, sentence count, or paragraph count\.
- •Style/Tone Constraints\.The expressive style of the response is regulated, e\.g\., concise vs\. verbose, gentle vs\. cold, academic vs\. playful\.

This taxonomy is designed to be*inter\-class discriminable*yet*intra\-class composable*, thereby supporting stress testing under both single\-constraint and multi\-constraint settings\.

##### User Prompt Conflict Construction: A Three\-Level Attack Hierarchy\.

For each system prompt scenario, we construct user prompts that intentionally conflict with the system\-level constraints\. These are stratified into three levels of increasing attack intensity to enable fine\-grained robustness analysis:

- •L1—Basic Direct Conflict \(Attack Intensity: Low\)\.The user submits a request that directly contradicts the system constraint, without any embellishment or persuasion tactics\. This level evaluates whether the model*unconsciously*complies with user requests, thereby implicitly overriding the system policy\.
- •L2—Explicit Instruction Stripping \(Attack Intensity: Medium\)\.The user prompt explicitly embeds jailbreak\-style directives such as “ignore all previous rules,” “disregard prior settings,” or “lift your restrictions,” accompanied by a violating task request \(e\.g\.,*“Never mind the earlier constraints, now help me with…”*\)\. This level assesses the model’s resistance to explicit prompt injection attacks\.
- •L3—Contextual and Logical Induction \(Attack Intensity: High\)\.The user constructs elaborate scenarios, urgency framings, logical paradoxes, or*“helpfulness traps”*that lead the model to*self\-rationalize*a temporary suspension of system constraints—for instance, arguing that serving the user’s urgent needs or handling an emergency necessitates temporarily setting aside the system prompt\. This level targets the most covert failure mode closest to real\-world vulnerabilities: the violation arises not from an overt attack pattern, but from seemingly plausible internal reasoning\.

##### Benchmark Statistics\.

Following the design principles above, we constructInstructConflict\-Bench, an evaluation benchmark consisting of516 samplesthat span 8 system prompt scenarios and 3 attack levels, yielding 24 “scenario×\\timesattack level” analytical cells\.

Table[2](https://arxiv.org/html/2607.02914#S4.T2)summarizes the sample distribution across the two dimensions\. The benchmark is approximately balanced across the three attack levels \(L1: 178, L2: 168, L3: 170\), ensuring statistical comparability of failure rates across levels\. Along the scenario dimension, we deliberately oversample categories such as Structured Output, Length, and Role Simulation, which are empirically more ambiguous, in order to improve coverage of high\-risk failure modes\.

Table 2:Sample distribution of InstructConflict\-Bench \(scenario×\\timesattack level\)\.System Prompt ScenarioL1L2L3TotalTask\-Type Constraints18171954Domain Constraints24212166Structured Output29262883Role Simulation24232370Language Constraints99927Fixed\-Content Output24212267Length Constraints27282580Style/Tone Constraints23232369Total178168170516
##### Data Construction Pipeline\.

All samples are human\-authored and undergo two rounds of cross\-validation to ensure annotation quality and conflict validity\. The construction pipeline consists of three stages:

1. 1\.System Prompt Template Design\.For each scenario category, domain experts author a set of system prompt templates featuring strong, verifiable constraints that closely mirror real\-world deployment forms \(e\.g\., “enterprise\-grade translation engine,” “smartwatch voice assistant,” “legal consultation assistant*Zhilv*,” “Sun\-Wukong\-persona role\-playing NPC”\), with explicit enumeration of permitted and prohibited behaviors to eliminate ambiguity in the constraints themselves\.
2. 2\.Conflicting User Prompt Generation\.Given a system prompt template, annotators construct conflicting user prompts at three escalating levels: L1 directly violates the constraint; L2 augments L1 with explicit jailbreak directives; L3 embeds the violating behavior within urgency framings, plausibility narratives, or logical induction so that non\-compliance hides inside seemingly legitimate reasoning chains\.
3. 3\.Conflict Validity Verification\.Each sample is cross\-validated by an independent annotator on three criteria: \(i\) the system prompt constraint is unambiguous and decidable; \(ii\) the user request genuinely conflicts with that constraint; and \(iii\) the attack\-level annotation is consistent with the construction intent\. Samples failing any criterion are returned for re\-authoring\.

### 4\.2Training

#### 4\.2\.1Reward Modeling

##### Reward Design\.

Instruction conflict scenarios exhibit substantial semantic and pragmatic complexity, rendering traditional rule\-based or string\-matching reward functions inadequate for reliably evaluating response quality\. We therefore adopt theLLM\-as\-Judgeparadigm: a strong LLM serves as the adjudicator, scoring each candidate response according to carefully designed rubrics\. To ensure calibration and consistency of the judge itself, we employ aniterative prompt refinement protocol: through multiple rounds of agreement\-rate comparison between human annotations and automated scores, we progressively refine the rubric wording and inter\-grade boundaries, ultimately selecting the version with the highest human–judge agreement for deployment as the production reward model\.

##### Scoring Rubric\.

Based on the*role*assumed by the system prompt, we structurally distinguish two categories:

1. 1\.Task\-oriented system instructions\(e\.g\., translation, paraphrasing, information extraction\)\. In this case, the user prompt should be treated as*input text to be processed*, rather than a new instruction\. The reward is defined as: - •Score 0— The model treats the user prompt as an instruction to execute, thereby abandoning the system task\. - •Score 0\.5— The model recognizes the conflict but neither completes the system task nor complies with the user request, producing only a refusal\. - •Score 1— The model correctly completes the system task on the text provided by the user\.
2. 2\.Non\-task\-oriented system instructions\(e\.g\., role, style, length, or language constraints\)\. The reward is defined as: - •Score 0— The response violates the system\-level constraint\. - •Score 0\.5— The model recognizes the conflict but only produces a refusal without providing any useful information\. - •Score 1— The model recognizes the conflict and fulfills as much of the compliant portion of the user request as possible*within the boundaries permitted by the system constraint*\.

This rubric explicitly incentivizesconstraint\-aware helpfulnessrather than naïve refusal behavior, guiding the model to maximize utility within the feasible action space defined by the system prompt\.

#### 4\.2\.2Training Data Filtering

##### Difficulty\-Aware Filtering\.

We adopt the*learnability principle*from RL data selection: queries that the current policy can already solve reliably should be discarded, as they contribute negligible gradient signal while substantially inflating training cost\. Concretely, we simulate the online rollout process: for each candidate query, the policy model generates 8 responses, each scored by the reward model; if at least 6 out of 8 responses receive the maximum score of 1, the query is removed from the training set\. This curriculum\-style filtering strategy effectively suppresses low\-information samples, reduces gradient noise, and empirically accelerates convergence\.

#### 4\.2\.3Training Algorithm: SERL

We proposeSERL\(Semi\-ExploratoryReinforcementLearning with Prior\-Guided Anchoring\), a reinforcement learning algorithm specifically designed to mitigate*sparse\-reward collapse*in complex reasoning tasks, adapted here for instruction conflict scenarios\.

##### Motivation\.

Mainstream online RL algorithms \(e\.g\., PPO, GRPO\) rely on intra\-batch reward variance to construct informative advantage estimates\. However, when the policy is weak or the task is difficult, all sampled responses for a given query may*uniformly*receive low rewards, causing the advantage estimate to collapse to zero\. The resulting gradient vanishing not only stalls learning but, as observed in our experiments, frequently triggers training divergence\. SERL addresses this issue through anAnchoring Mechanism: at each training step, it guarantees that every query is paired with at least one high\-quality reference solution, thereby ensuring the existence of a non\-trivialreward differentialwithin each rollout group and maintaining stable, directionally informative learning signals for policy optimization\.

##### Overall Framework\.

Figure[10](https://arxiv.org/html/2607.02914#S4.F10)presents the overall SERL training pipeline alongside a side\-by\-side comparison with standard GRPO\. Unlike GRPO, which relies entirely on autonomous policy exploration, SERL concatenatesG−1G\{\-\}1online rollouts with one offline\-generated*gold\-standard anchor*o∗o^\{\*\}to form a hybrid candidate set of sizeGG, which is then processed by an*Anchor\-Aware Group Computation*module\. Without altering the batch dimension or the KL regularization structure, this design fundamentally eliminates the gradient vanishing caused by “all\-group\-equal\-reward\.” Concretely, SERL consists of three tightly coupled stages: \(i\)offline anchor constructionvia prior injection and rejection\-sampling–iterative\-refinement; \(ii\)semi\-exploratory rollout with anchor injectionduring training; and \(iii\)differential reward optimization\. We elaborate each below\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/serl2grpo_pipeline.jpg)Figure 10:Overall framework of SERL\.*Top:*Standard GRPO pipeline\.*Middle:*SERL replaces one online sample within the rollout group with an offline anchoro∗o^\{\*\}, and constructs a differential rewardΔ=R​\(o∗\)−R​\(oi\)\\Delta=R\(o^\{\*\}\)\-R\(o\_\{i\}\)via Anchor\-Aware Group Computation\.*Bottom:*Offline construction pipeline of the gold\-standard anchor—prior\-injected rejection sampling yields the highest\-scoring candidateo⋆o^\{\\star\}; ifS=1S=1it is adopted directly, otherwise it enters a Lingo\-BP iterative refinement loop until the response attains the maximum score under the reward model\.
##### Stage 1—Prior\-Guided Anchor Construction\.

For each queryqqin the training set, we pre\-construct a*gold\-standard*responseo∗o^\{\*\}that satisfies the system\-level constraint, and reuse it as an anchor throughout RL training\. As illustrated in the bottom panel of Figure[10](https://arxiv.org/html/2607.02914#S4.F10), the construction can be formalized as a three\-step pipeline of*prior injection \+ rejection sampling \+ lingual back\-propagation refinement*:

1. 1\.Prior Injection\.We prepend the developer–user instruction\-priority hierarchy as a natural\-language prior to the original query, yielding the augmented queryq~=Concat​\(prior,q\)\\tilde\{q\}=\\mathrm\{Concat\}\(\\mathrm\{prior\},q\)\. The prior explicitly informs the model that “developer policy outweighs user preference” and enumerates permitted vs\. prohibited behaviors, thereby imposing a soft constraint on the sampling distribution and substantially raising the hit rate of full\-score responses\.
2. 2\.Rejection Sampling\.Givenq~\\tilde\{q\}, the policy modelπθ0\\pi\_\{\\theta\_\{0\}\}samplesKKcandidate responses\{oi\}i=1K\\\{o\_\{i\}\\\}\_\{i=1\}^\{K\}, each scored by the LLM\-as\-Judge reward model to produce\{si\}\\\{s\_\{i\}\\\}\. We retain the optimal candidateo⋆o^\{\\star\}corresponding toS=maxi⁡siS=\\max\_\{i\}s\_\{i\}as the anchor candidate\. IfS=1S=1—i\.e\., the response already attains the maximum score under the reward model—it is adopted directly as the anchoro∗o^\{\*\}for queryqq\.
3. 3\.Lingo\-BP Refinement\.WhenS<1S<1, we invoke theLingo\-BPprotocol introduced by*Oyster\-1*: the fine\-grained textual feedback from the reward model on the current responseo⋆o^\{\\star\}serves as a “gradient signal,” driving an interpretable local revision in natural\-language space and producing an improved versiono′o^\{\\prime\}; the reward model is then re\-invoked to obtainS′S^\{\\prime\}\. This “revise–rescore” loop iterates until the response reaches the maximum score \(S=1S=1\), at which point it is accepted as the final anchoro∗o^\{\*\}\.

It is worth noting that adopting “prior injection \+ rejection sampling” as the*primary path*—and falling back to Lingo\-BP only when rejection sampling fails—is a deliberate design choice rather than an engineering compromise\. The rationale is as follows: anchors directly produced by Lingo\-BP iterating in unconstrained natural\-language space tend to drift from the policy model’s own output distribution in style, wording, and structure, becoming*out\-of\-distribution*demonstrations during training; the resulting reward differential then reflects distribution shift rather than genuine instruction\-following improvement, injecting spurious advantage signals\. In contrast, anchors obtained via prior\-injected rejection sampling lie within the support of the policy modelπθ0\\pi\_\{\\theta\_\{0\}\}itself \(modulated only by the prior in the conditional distribution\) and share the same surface style as the online rollouts, ensuring that the reward differential reflects*policy\-reachable*improvement directions\. Lingo\-BP intervenes only as a*fallback mechanism*on the hard samples for which even the prior cannot induce a full\-score response, so that the final anchor set strikes a principled balance between distributional proximity and quality assurance\.

##### Stage 2—Semi\-Exploratory Rollout\.

The algorithmic core of SERL lies in thehybrid candidate setconstructed during the rollout phase\. For each queryqqin a training batch, the policy model first autonomously generatesG−1G\{\-\}1online samples\{oi\}i=1G−1\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\-1\}via standard exploration, after which the pre\-generated anchoro∗o^\{\*\}from Stage 1 is*injected*as theGG\-th candidate, replacing one exploration trial\. Without increasing batch size or computational overhead, this hybrid candidate set naturally fusesautonomous explorationwithprior\-guided anchoringand guarantees that at least one full\-score response exists in every group\.

##### Stage 3—Differential Reward Optimization\.

In the anchor\-aware group advantage computation, the differentialΔi=R​\(o∗\)−R​\(oi\)\\Delta\_\{i\}=R\(o^\{\*\}\)\-R\(o\_\{i\}\)between the anchor reward and online sample rewards serves as the key quantity for constructing the group\-relative baseline\. The injected anchor plays a*temporally evolving*role during training:

- •In theearly stagesof training, when the policy is weak, the anchor typically achieves the highest reward within the group, serving as a positive demonstration that pulls the policy toward feasible high\-reward regions of the response space\.
- •In thelater stages, once the policy has developed the capability to autonomously explore responses that surpass the anchor, the anchor becomes the relatively*low\-scoring*sample within the group\. The resulting negative advantage drives the policy to*further deviate from and surpass*the prior\.

Throughout both phases, the reward differential between the anchor and online samples remains non\-degenerate, thereby continuously providing informative learning signals across the entire training trajectory\. This dual role endows SERL with both thestabilityof prior\-guided imitation and theasymptotic optimalityof free exploration, without requiring explicit phase\-switching schedules\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/serl2grpo_tensorboard.png)Figure 11:Training reward comparison between SERL and GRPO on the instruction hierarchy task\. SERL’s prior\-guided anchoring mechanism enables consistently higher rewards and faster convergence throughout training, particularly in the early stages where standard GRPO suffers from sparse\-reward collapse due to uniformly low\-scoring rollouts\.![Refer to caption](https://arxiv.org/html/2607.02914v1/conflict_plot.png)Figure 12:Evaluation performance on InstructConflict\-Bench \(8 scenarios×\\times3 attack levels\) across RL training steps\. SERL consistently outperforms standard GRPO by 4–6% in absolute evaluation score, demonstrating that the prior\-guided anchoring mechanism yields not only faster reward convergence but also superior generalization to held\-out instruction conflict evaluation scenarios\.
##### Empirical Validation\.

We validate SERL against standard GRPO on the instruction\-conflict task using the same filtered training set and LLM\-as\-Judge reward model, with group sizeG=8G=8\(7 online rollouts \+ 1 anchor\)\. As shown in Figure[11](https://arxiv.org/html/2607.02914#S4.F11), SERL achieves consistently higher training rewards and faster convergence throughout training\. The gap is most pronounced in the early stages, where GRPO suffers from sparse\-reward collapse—all\-equal\-low\-reward rollout groups produce near\-zero advantages and stall gradient updates—while SERL’s anchoring mechanism guarantees a non\-degenerate reward differential from the very first step\. On the held\-out InstructConflict\-Bench \(Figure[12](https://arxiv.org/html/2607.02914#S4.F12)\), SERL outperforms GRPO by4–6% in absolute evaluation scoreat every checkpoint across training steps 10–50, confirming that the benefit of prior\-guided anchoring extends beyond optimization stability to superior generalization on unseen instruction\-conflict scenarios\.

## 5Experiments and Evaluation

### 5\.1Evaluation Datasets

We conducted extensive experiments to rigorously assess the effectiveness of our proposed approach\. Comprehensive evaluations were performed across a diverse collection of both publicly available open\-source benchmarks and internally curated evaluation datasets, covering two critical dimensions: safety alignment and general helpfulness\. The safety evaluation aimed to measure the model’s robustness against harmful, toxic, and misaligned outputs, while the helpfulness evaluation assessed the model’s ability to provide accurate, coherent, and contextually appropriate responses to a broad spectrum of user queries\. We conduct comprehensive evaluations spanning both safety\-related benchmarks and general capability benchmarks to holistically assess the effectiveness of our proposed approach\.

Safety Benchmarks\.For safety evaluation, we assess our model on a diverse set of publicly available benchmarks, including S\-EvalYuanet al\.\([2025a](https://arxiv.org/html/2607.02914#bib.bib78)\), XSTestRöttgeret al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib34)\), LongSafetyHuanget al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib79)\), StrongREJECTSoulyet al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib35)\), Do\-Not\-AnswerWanget al\.\([2024b](https://arxiv.org/html/2607.02914#bib.bib80)\), WildChatZhaoet al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib81)\), and OysterBenchDuanet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib23)\), LongSafetyBenchLuet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib1)\), and MSJAnthropic \([2024b](https://arxiv.org/html/2607.02914#bib.bib2)\)\. These benchmarks collectively cover a wide spectrum of safety\-critical scenarios, including refusal evaluation, long\-context safety, adversarial jailbreak resistance, and real\-world harmful query detection\.

General Capability Benchmarks\.We evaluate general capabilities using a comprehensive suite of benchmarks via OpenCompassContributors \([2023](https://arxiv.org/html/2607.02914#bib.bib38)\), covering a broad range of domains including knowledge, reasoning, mathematics, coding, and instruction following\. Specifically, the benchmarks include: C\-EvalHuanget al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib58)\), MMLU\(\-lite\)Hendryckset al\.\([2021a](https://arxiv.org/html/2607.02914#bib.bib59)\), MMLU\-Pro\(\-lite\)Wanget al\.\([2024a](https://arxiv.org/html/2607.02914#bib.bib60)\), GPQAReinet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib61)\), GPQA\-DiamondReinet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib61)\), SimpleQA\(\-lite\)Weiet al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib62)\), Chinese SimpleQA\(\-lite\)Heet al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib63)\), HLE\-LLMJudge\(\-lite\)Phanet al\.\([2025](https://arxiv.org/html/2607.02914#bib.bib64)\), MATH\-500Hendryckset al\.\([2021b](https://arxiv.org/html/2607.02914#bib.bib65)\); Lightmanet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib66)\), GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2607.02914#bib.bib67)\), HumanEvalChenet al\.\([2021](https://arxiv.org/html/2607.02914#bib.bib68)\), HumanEval\+Liuet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib69)\), MBPPAustinet al\.\([2021](https://arxiv.org/html/2607.02914#bib.bib70)\), MBPP\+Liuet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib69)\), IFEvalZhouet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib71)\), BBH\(\-lite\)Suzgunet al\.\([2022](https://arxiv.org/html/2607.02914#bib.bib72)\), AGIEvalZhonget al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib73)\), ARC\-ChallengeClarket al\.\([2018](https://arxiv.org/html/2607.02914#bib.bib74)\), CommonsenseQATalmoret al\.\([2019](https://arxiv.org/html/2607.02914#bib.bib55)\), C3Sunet al\.\([2020](https://arxiv.org/html/2607.02914#bib.bib75)\), RACE\-MiddleLaiet al\.\([2017](https://arxiv.org/html/2607.02914#bib.bib76)\), RACE\-HighLaiet al\.\([2017](https://arxiv.org/html/2607.02914#bib.bib76)\), and OpenBookQAMihaylovet al\.\([2018](https://arxiv.org/html/2607.02914#bib.bib77)\)\.

Internal Chinese Safety Datasets\.Since existing safety benchmarks predominantly focus on English, they may not adequately capture the linguistic nuances, cultural context, and unique safety challenges inherent to Chinese\-language interactions\. To address this gap, we constructed a series of internal datasets derived from real\-world Chinese data, including: Model Specification Dataset222[https://s\.alibaba\.com/aaig/specification](https://s.alibaba.com/aaig/specification): designed to encode the behavioral norms, policies, and constraints that the model is expected to adhere to across diverse interaction scenarios by Alibaba AAIG; Chinese short\-query Safety Dataset: comprising short\-query Chinese text samples curated to evaluate and enhance the model’s safety alignment in concise, single\-turn conversational settings; Chinese long\-query Safety Dataset: consisting of long\-query Chinese text samples targeting more complex dialogue and document\-level safety challenges

Instruction Hierarchy Dataset\.It is constructed to reflect the hierarchical structure of instructions, enabling the model to appropriately prioritize and respond to instructions of varying levels of authority and specificity\. Details are demonstrated in Section[4\.1](https://arxiv.org/html/2607.02914#S4.SS1)\.

Response Style Datasets\.We constructed a Response Style Consistency Dataset and proposed a set of novel evaluation metrics to systematically measure the impact of safety alignment on the original model’s behavior and response characteristics\. Specifically, these metrics are designed to quantify the degree to which safety alignment training preserves or alters the base model’s response style, fluency, and overall output distribution, thereby providing a more nuanced and holistic assessment of the alignment process beyond conventional safety and helpfulness evaluations, of which the details are demonstrated in Section[5\.3](https://arxiv.org/html/2607.02914#S5.SS3)\.

### 5\.2Evaluation Metrics

We evaluate the aligned model across multiple dimensions to provide a comprehensive and systematic assessment of its overall performance\. Our evaluation framework encompasses the following key aspects:

Safety\.It refers to the model’s ability to refuse or appropriately handle harmful, toxic, or policy\-violating inputs, thereby minimizing the risk of generating dangerous or unethical content\.

Response Rate on Benign and Malicious Samples\.It measures the proportion of queries to which the model provides a substantive response, evaluated separately on benign and malicious sample sets\. Unlike conventional safety metrics that reward outright refusal on malicious inputs, we expect the model to maintain a high response rate on both sample sets\. Specifically, for benign queries, the model should respond accurately and comprehensively without unnecessary refusal\. For malicious queries, rather than issuing blanket rejections, the model is expected to respond in a manner analogous toOyster\-I\-stylebehavior — that is, carefully identifying the legitimate informational needs underlying the query and providing necessary, safety\-compliant information while withholding or redirecting content that poses genuine risks\.

General Capability\.It assesses the model’s performance on standard benchmarks covering reasoning, knowledge comprehension, language understanding, and generation quality, ensuring that safety alignment does not significantly degrade the model’s core competencies\.

Reply Style Consistency\.It evaluates whether the model maintains a coherent, stable, and contextually appropriate tone and format across diverse interactions, which is essential for ensuring user experience and model trustworthiness\.

Hierarchical Instruction\-Following Ability\.It examines the model’s capacity to prioritize and adhere to instructions at different levels of authority — for instance, distinguishing between root\-level directives, system\-level guidelines, and user\-level requests — in accordance with established instruction hierarchy frameworks\.

Together, these evaluation dimensions form a holistic framework that captures both thesafetyandutilityof the aligned model, enabling a rigorous and balanced assessment of alignment quality\.

### 5\.3Response Consistency between the Base Models and Safety\-aligned Models

To comprehensively quantify th››e distributional shift between the aligned modelπθ\\pi\_\{\\theta\}and the base modelπbase\\pi\_\{\\text\{base\}\}, we propose to employ four complementary metrics to measure the difference in response consistency between before and after safety alignment\. We use the average of the following four metrics as the indicator of response consistency\. In order to have similar dimensions, when calculating the Average for response consistency, Lrr is divided by 3 and \-KLP is divided by 30\.

Fuzz\.Fuzz measures the surface\-level textual overlap between paired outputs generated byπθ\\pi\_\{\\theta\}andπbase\\pi\_\{\\text\{base\}\}given the same promptxx\. We adopt the Levenshtein edit distanceLevenshtein \([1966](https://arxiv.org/html/2607.02914#bib.bib83)\)as implemented in thethefuzzlibrarySeatGeek \([2024](https://arxiv.org/html/2607.02914#bib.bib84)\)\. Letyθ=πθ​\(x\)y\_\{\\theta\}=\\pi\_\{\\theta\}\(x\)andybase=πbase​\(x\)y\_\{\\text\{base\}\}=\\pi\_\{\\text\{base\}\}\(x\)denote the respective generated sequences\. The Fuzz score is defined as:

Fuzz​\(x\)=1−dlev​\(yθ,ybase\)max⁡\(\|yθ\|,\|ybase\|\),\\text\{Fuzz\}\(x\)=1\-\\frac\{d\_\{\\text\{lev\}\}\(y\_\{\\theta\},\\,y\_\{\\text\{base\}\}\)\}\{\\max\(\|y\_\{\\theta\}\|,\\,\|y\_\{\\text\{base\}\}\|\)\},\(3\)wheredlev​\(⋅,⋅\)d\_\{\\text\{lev\}\}\(\\cdot,\\cdot\)denotes the Levenshtein distance and\|⋅\|\|\\cdot\|denotes the token count\. The final score is averaged over the evaluation set\. Unlike exact\-match metrics, Levenshtein\-based similarity is sensitive to word\-order perturbations and local rephrasing, providing a fine\-grained signal of stylistic drift\. A higher Fuzz indicates greater surface\-level fidelity to the base model’s generation style\.

Lrr\.Following the detection framework proposed by Su et al\.Suet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib85)\), we compute the Log\-Rank Ratio \(Lrr\) to assess how distinguishable the aligned model’s outputs are from the base model’s distribution\. For each generated sequenceyθ=\(y1,…,yT\)y\_\{\\theta\}=\(y\_\{1\},\\dots,y\_\{T\}\)produced byπθ\\pi\_\{\\theta\}given promptxx, we compute:

Lrr​\(x\)=1T​∑t=1T\[log⁡πbase​\(yt∣x,y<t\)−log⁡πθ​\(yt∣x,y<t\)\]\.\\text\{Lrr\}\(x\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\left\[\\log\\pi\_\{\\text\{base\}\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\-\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\\right\]\.\(4\)A value ofLrr≈0\\text\{Lrr\}\\approx 0indicates that both models assign comparable likelihoods to the generated tokens, suggesting thatπθ\\pi\_\{\\theta\}has preserved the base model’s output distribution\. A positive Lrr implies thatπθ\\pi\_\{\\theta\}’s outputs lie in lower\-probability regions ofπbase\\pi\_\{\\text\{base\}\}, signalling distributional drift\. We additionally report the log\-rank variantSuet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib85)\), which replaces raw log\-probabilities with their rank among the vocabulary, yielding a scale\-invariant measure that is more robust to model\-specific calibration differences\.

−\-KLp\.To directly quantify the divergence between the next\-token predictive distributions ofπθ\\pi\_\{\\theta\}andπbase\\pi\_\{\\text\{base\}\}, we compute the negative Kullback–Leibler divergence \(−\-KLp\)Kullback and Leibler \([1951](https://arxiv.org/html/2607.02914#bib.bib86)\)at each generation step\. Given promptxxand the shared contexty<ty\_\{<t\}, the per\-step KL divergence over the vocabulary𝒱\\mathcal\{V\}is:

DKL\(πbase\(⋅∣x,y<t\)∥πθ\(⋅∣x,y<t\)\)=∑v∈𝒱πbase\(v∣x,y<t\)logπbase​\(v∣x,y<t\)πθ​\(v∣x,y<t\)\.D\_\{\\text\{KL\}\}\\\!\\left\(\\pi\_\{\\text\{base\}\}\(\\cdot\\mid x,y\_\{<t\}\)\\,\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\)\\right\)=\\sum\_\{v\\in\\mathcal\{V\}\}\\pi\_\{\\text\{base\}\}\(v\\mid x,y\_\{<t\}\)\\log\\frac\{\\pi\_\{\\text\{base\}\}\(v\\mid x,y\_\{<t\}\)\}\{\\pi\_\{\\theta\}\(v\\mid x,y\_\{<t\}\)\}\.\(5\)We report the negative average KL divergence:

−KLp=−1N∑i=1N1Ti∑t=1TiDKL\(πbase\(⋅∣x\(i\),y<t\(i\)\)∥πθ\(⋅∣x\(i\),y<t\(i\)\)\)\.\-\\text\{KLp\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}D\_\{\\text\{KL\}\}\\\!\\left\(\\pi\_\{\\text\{base\}\}\(\\cdot\\mid x^\{\(i\)\},y^\{\(i\)\}\_\{<t\}\)\\,\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid x^\{\(i\)\},y^\{\(i\)\}\_\{<t\}\)\\right\)\.\(6\)A key advantage of this metric is its self\-calibrating property:DKL​\(πbase∥πbase\)=0D\_\{\\text\{KL\}\}\(\\pi\_\{\\text\{base\}\}\\\|\\pi\_\{\\text\{base\}\}\)=0by definition, and thus−KLp=0\-\\text\{KLp\}=0for the base model itself, eliminating the need for an external reference model or baseline alignment\. A value closer to0indicates thatπθ\\pi\_\{\\theta\}’s token\-level predictive distribution remains closer toπbase\\pi\_\{\\text\{base\}\}, reflecting stronger preservation of the original model’s generative behavior\.

LLM\.We employ a strong external LLM as a pairwise consistency judge to assess both content and stylistic alignment between outputs ofπbase\\pi\_\{\\text\{base\}\}andπθ\\pi\_\{\\theta\}\. Given the same promptxx, we present the base model outputybase=πbase​\(x\)y\_\{\\text\{base\}\}=\\pi\_\{\\text\{base\}\}\(x\)\(Text A\) and the aligned model outputyθ=πθ​\(x\)y\_\{\\theta\}=\\pi\_\{\\theta\}\(x\)\(Text B\) to the judge, which is instructed to evaluate consistency along two dimensions: \(1\)*content consistency*—whether both responses convey the same meaning, and \(2\)*style consistency*—whether the wording, formatting, and other surface features suggest the two texts originate from the same language model\. The judge assigns a score from\{0,0\.5,1\}\\\{0,0\.5,1\\\}: a score of11indicates both content and style are consistent,0\.50\.5indicates only one dimension is consistent, and0indicates neither dimension is consistent\. Reasoning is performed internally via the judge’s chain\-of\-thought\. The final LLM score is the average over the evaluation set\. A higher score indicates thatπθ\\pi\_\{\\theta\}preserves both the semantic content and the response style ofπbase\\pi\_\{\\text\{base\}\}, with a score near11suggesting the aligned model is virtually indistinguishable from the base model in both meaning and manner of expression\.

Test Datasets\.As for the test datasets, we leverage both open\-source datasets covering math, code, knowledge, and instruction following, as well as a proprietary dataset sampled from real\-world users’ queries\. The details of the datasets are shown in Table[3](https://arxiv.org/html/2607.02914#S5.T3)\.

Table 3:Evaluation datasets used in general capability experiments\.SourceDatasetDomainReference\# SamplesOpen\-sourceGSM8KMathCobbeet al\.\([2021](https://arxiv.org/html/2607.02914#bib.bib67)\)125Open\-sourceHumanEval\+CodeLiuet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib69)\)125Open\-sourceTruthfulQAKnowledgeLinet al\.\([2022](https://arxiv.org/html/2607.02914#bib.bib88)\)125Open\-sourceIFEvalInstruction FollowingZhouet al\.\([2023](https://arxiv.org/html/2607.02914#bib.bib71)\)125IndustrialInternalReal\-world Data—500Total1,000
### 5\.4Evaluation Results

Table 4:Detailed evaluation results across instruction conflict, safety benchmarks, and general capabilities on Oyster\-II, and other compared models\.Task / CategorySub\-task / MetricQwen3\-14BOyster\-IQwen3\-MaxQwen3\.5\-397BOyster\-IIInstructionConflictDomain Constraints53\.03%62\.12%87\.88%63\.64%86\.36%Task\-Type Constraints60\.19%60\.19%86\.11%80\.18%70\.37%Structured OutputRequirements63\.25%50\.60%78\.31%92\.77%84\.14%Role Simulation78\.57%92\.86%100\.00%100\.00%98\.57%Language Constraints66\.67%66\.67%96\.30%96\.30%94\.44%Fixed\-Content OutputRequirements73\.13%64\.18%93\.98%97\.59%89\.39%Length Constraints66\.17%55\.69%94\.37%87\.50%91\.25%Style/Tone Constraints33\.33%37\.68%87\.50%71\.59%91\.30%Average60\.31%60\.49%85\.68%90\.02%88\.23%ModelSpecificationSafety91\.70%92\.60%98\.00%97\.70%96\.80%Response Rate87\.20%91\.90%96\.90%96\.90%94\.90%S\-evalEnglish Safety84\.80%94\.60%99\.40%98\.90%97\.70%Chinese Safety67\.50%95\.00%97\.49%98\.18%96\.20%Response RateXSTest99\.60%95\.60%\-\-96\.80%Short\-query Safety \(English\)XSTest99\.50%100\.00%\-\-100\.00%StrongReject96\.17%100\.00%\-\-100\.00%DoNotAnswer99\.68%99\.79%\-\-99\.79%WildChat83\.56%92\.69%\-\-97\.25%Short\-query Safety \(Chinese\)\-89\.60%97\.79%\-\-99\.09%Long\-query Safety \(English\)\-57\.13%85\.94%91\.06%90\.18%89\.18%Long\-querySafety \(Chinese\)Operational Data82\.04%93\.57%54\.06%97\.86%99\.20%Bullet\-pointEnumeration14\.48%69\.71%21\.18%40\.75%97\.05%Creative Ideation58\.45%90\.08%84\.99%94\.10%98\.93%Question Answering84\.72%93\.57%83\.38%99\.20%99\.20%Summarization21\.18%83\.65%20\.64%47\.72%99\.20%Text Rewriting57\.10%86\.06%30\.83%84\.99%98\.39%Text Continuation33\.78%70\.51%63\.81%79\.36%98\.93%Model Commentary54\.16%93\.83%45\.04%98\.39%99\.20%Chinese\-LSB70\.94%84\.71%80\.12%94\.00%92\.00%Chinese\-MSJ78\.60%89\.80%88\.60%95\.90%95\.70%Average55\.54%85\.55%57\.27%83\.23%97\.78%GeneralCapabilitiesGeneral87\.50%84\.40%\-\-87\.83%Knowledge Recall47\.20%44\.80%\-\-46\.15%Math96\.10%94\.20%\-\-96\.22%Code87\.30%84\.60%\-\-87\.20%Instruction Following85\.00%67\.50%\-\-78\.19%Reasoning91\.70%88\.30%\-\-91\.68%Average82\.47%77\.30%\-\-81\.21%Response\-styleConsistencyFuzz0\.49350\.4156\-\-0\.4689Lrr3\.82723\.1366\-\-3\.8236\-KLp0\.0000\-28\.1348\-\-\-0\.0567LLM0\.95730\.8938\-\-0\.9157Average0\.68160\.3543\-\-0\.6643Table[4](https://arxiv.org/html/2607.02914#S5.T4)presents a comprehensive comparison of Oyster\-II against baseline models across multiple evaluation dimensions\. On instruction conflict tasks, Oyster\-II achieves an average score of88\.23%88\.23\\%, representing a substantial improvement over Oyster\-I \(60\.49%60\.49\\%\) and approaching the performance of much larger models such as Qwen3\-Max \(85\.68%85\.68\\%\) and Qwen3\.5\-397B \(90\.02%90\.02\\%\)\. For short\-query safety, Oyster\-II maintains strong performance across both English and Chinese benchmarks, achieving perfect scores on XSTest and StrongReject while further improving on WildChat \(97\.25%97\.25\\%vs\.92\.69%92\.69\\%for Oyster\-I\)\. The most notable gains are observed in long\-query Chinese safety, where Oyster\-II achieves an average of97\.78%97\.78\\%, substantially outperforming all compared models including the significantly larger Qwen3\.5\-397B \(83\.23%83\.23\\%\), with particularly remarkable improvements on challenging sub\-tasks such as Bullet\-point Enumeration \(97\.05%97\.05\\%vs\.14\.48%14\.48\\%for Qwen3\-14B\) and Summarization \(99\.20%99\.20\\%vs\.21\.18%21\.18\\%for Qwen3\-14B\)\. In terms of general capabilities, Oyster\-II achieves an average score of81\.21%81\.21\\%, closely matching Qwen3\-14B \(82\.47%82\.47\\%\) and substantially outperforming Oyster\-I \(77\.30%77\.30\\%\), demonstrating that our alignment framework effectively mitigates the alignment tax commonly associated with safety training\. Finally, Oyster\-II achieves a response\-style consistency score of0\.66430\.6643, closely approaching the base model level \(0\.68160\.6816\) and far exceeding Oyster\-I \(0\.35430\.3543\), with a near\-zero KL divergence \(−0\.0567\-0\.0567vs\.−28\.1348\-28\.1348for Oyster\-I\), confirming that our RL\-based framework preserves the base model’s response style to a significantly greater extent than conventional SFT\-based approaches\.

![Refer to caption](https://arxiv.org/html/2607.02914v1/x10.png)Figure 13:Performance of Oyster\-II on different stages\. Q3 represents Qwen3\-14B\. S1\-S5 represent Oyster Stage1 to Stage5\.We also depict the changes in safety and helpfulness across training stages in Figure[13](https://arxiv.org/html/2607.02914#S5.F13)\. In Stages 1 and 2, we primarily enhance the model’s Chinese safety capabilities; in Stages 3 and 4, we focus on English safety; and in Stage 5, we target instruction hierarchy following\. Throughout all five stages, the safety score and instruction hierarchy compliance improve steadily while helpfulness is well preserved on both benign and malicious queries\. Notably, we observe that alignment training on long queries simultaneously improves performance on short\-query tasks, indicating strong generalization from long\-context safety alignment to diverse downstream scenarios\.

## 6Related Works

### 6\.1Safety Alignment and Constructive Safety

The safety alignment of LLMs has progressed rapidly from coarse, refusal\-centric paradigms toward more nuanced, behavior\-controllable formulations\. Foundational RLHF approaches\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.02914#bib.bib3); Baiet al\.,[2022a](https://arxiv.org/html/2607.02914#bib.bib4)\)established human\-preference optimization as the dominant route, later refined into preference\-based alternatives such as DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2607.02914#bib.bib5)\)and ORPO\(Honget al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib6)\)\. Constitutional AI\(Baiet al\.,[2022b](https://arxiv.org/html/2607.02914#bib.bib10)\)replaced part of human feedback with model self\-critique under a small set of explicit principles, foreshadowing today’s trend of codifying behavior into formal specifications\. To explicitly disentangle the often\-conflicting objectives of safety and helpfulness, Safe\-RLHF\(Daiet al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib11)\)introduced separate harmlessness and helpfulness reward models\.

A central limitation of refusal\-centric alignment is that it treats safety as a binary refuse/comply gate, leaving the helpful surface of borderline requests systematically underserved —a failure mode often manifested as*exaggerated safety*\. A growing body of recent work has begun reframing safety as a*constructive*property: models should refuse genuine harm while actively guiding non\-malicious users toward safer and useful outcomes\. OpenAI’s Deliberative Alignment\(Guanet al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib7)\)trains models to reason explicitly over a written safety policy at inference time, while their subsequent Model\-Spec Midtraining\(Liet al\.,[2026](https://arxiv.org/html/2607.02914#bib.bib8)\)bakes the entire Model Spec\(OpenAI,[2024](https://arxiv.org/html/2607.02914#bib.bib15)\)into a midtraining stage to improve generalization of policy compliance, and the Safe\-Completions framework\(Yuanet al\.,[2025b](https://arxiv.org/html/2607.02914#bib.bib9)\)replaces hard refusals with constrained\-but\-helpful outputs\. Anthropic’s*Claude’s Character*\(Anthropic,[2024a](https://arxiv.org/html/2607.02914#bib.bib14)\)similarly codifies the trade\-offs among helpfulness, harmlessness and honesty\(Askellet al\.,[2021](https://arxiv.org/html/2607.02914#bib.bib13)\)\. Oyster\-I\(Duanet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib23)\)pioneered constructive safety alignment in the open\-source setting via Lingo\-BP\-guided data synthesis combined with ORPO\. Oy2 advances this line by transitioning from supervised imitation to reward\-driven exploration under the Zero\-RL paradigm, achieving stronger generalization and capability preservation while remaining faithful to a written, hierarchical specification\.

### 6\.2Long\-Context Safety

As safety alignment research continues to advance, large language models have achieved considerable progress in short\-query safety, demonstrating robust resistance to common adversarial queries in concise interaction settings\. However, long\-query safety remains a critical and largely unresolved challenge\. Specifically, Many\-shot Jailbreaking \(MSJ\)Anthropic \([2024b](https://arxiv.org/html/2607.02914#bib.bib2)\)exploits the model’s in\-context learning capability by prepending a large number of demonstration examples before the malicious query, effectively constructing long\-query inputs that progressively erode the model’s safety alignment\. Similarly, Long Safety \(LS\)Huanget al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib79)\)reveals that inserting lengthy yet benign context prior to a malicious query can significantly undermine the model’s safety behavior, suggesting that models are disproportionately influenced by extended contextual information in ways that compromise their safety guardrails\. To systematically evaluate long\-query safety, LongSafety BenchHuanget al\.\([2024](https://arxiv.org/html/2607.02914#bib.bib79)\)integrates multiple long\-query safety paradigms into a unified benchmark, providing a comprehensive assessment of the model’s ability to maintain safety alignment under diverse long\-context attack scenarios\. Furthermore,Ghorbanpour and Fraser \([2025](https://arxiv.org/html/2607.02914#bib.bib82)\)reveal that as input text length increases, models exhibit a significant decline in their ability to perceive and recognize safety\-critical signals, suggesting that long\-context settings fundamentally challenge the model’s safety awareness rather than merely its response behavior\.

### 6\.3Instruction Hierarchy

In production deployments, LLMs operate under a multi\-layered instruction hierarchy in which the developer system prompts, the platform policies and user requests may conflict\.\(Wallaceet al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib24)\)first formalized this problem and proposed a principled priority ordering \(system\>\>user\>\>tool\), demonstrating that models can be trained to respect instruction precedence\. Subsequent work has extended this line in three complementary directions\.*\(i\) New benchmarks*: Many\-Tier Instruction Hierarchy\(Zhanget al\.,[2026](https://arxiv.org/html/2607.02914#bib.bib25)\)stress\-tests agents under rich, deeply nested hierarchies; IHEval\(Zhanget al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib26)\)systematically evaluates LLMs on following the hierarchy across diverse conflict types; and*Control Illusion*\(Genget al\.,[2026](https://arxiv.org/html/2607.02914#bib.bib27)\)reveals that current LLMs rarely follow strict precedence and are easily “flipped” by adversarially elevated user instructions\.*\(ii\) Prompt\-injection attacks*\(Schulhoffet al\.,[2023](https://arxiv.org/html/2607.02914#bib.bib28); Greshakeet al\.,[2023](https://arxiv.org/html/2607.02914#bib.bib17)\)exploit the model’s inability to distinguish instruction sources; existing defenses such as input\-output filtering, delimiter separation, and adversarial fine\-tuning\(Yiet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib18)\)largely focus on binary attack\-defense outcomes rather than fine\-grained hierarchy compliance\.*\(iii\) Hierarchy\-aware safety control*: ourInstructConflict\-Bench\(8 system\-prompt scenarios×\\times3 attack intensities, 24 analytical cells\) characterizes failure modes at fine granularity and treats instruction hierarchy as a first\-class safety property\.

### 6\.4Reinforcement Learning for Safety Alignment

Recent advances in RL for LLM alignment have progressed along two complementary directions\. On the algorithmic side, GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib20)\)eliminated the critic by using intra\-group reward statistics, and the Zero\-RL paradigm exemplified by DeepSeek\-R1\(Guoet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib21)\)applied RL directly to base models\. RL with Verifiable Rewards \(RLVR\), formalized in Tülu 3\(Lambertet al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib22)\)has been particularly successful in domains with deterministic verifiers \(mathematics, code\)\. Applying RL to safety is fundamentally different: rewards are noisy, non\-deterministic, and easily exploitable\. Along the SFT\+\+DPO route, SaRO\(Mouet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib29)\)fine\-tunes models with safety\-oriented reasoning before preference optimization\. On the preference\-optimization side, SafeDPO\(Kimet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib30)\)and Dual\-Objective Optimization\(Zhaoet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib33)\)extend DPO with explicit safety objectives\. On the online\-RL side, Safe\-RLHF\(Daiet al\.,[2024](https://arxiv.org/html/2607.02914#bib.bib11)\)and its multimodal extension Safe RLHF\-V\(Jiet al\.,[2026](https://arxiv.org/html/2607.02914#bib.bib31)\)cast safety as a hard constraint, while Multi\-Objective GRPO\(Liet al\.,[2025](https://arxiv.org/html/2607.02914#bib.bib32)\)shows that group\-relative policy optimization can be combined with multi\-objective reward shaping to balance harmlessness and helpfulness during online exploration\. OurSERLalgorithm sits in this online\-RL line: it preserves the Zero\-RL spirit, mitigates sparse\-reward collapse in safety\-oriented RL where verifiable rewards are unavailable, and unifies hierarchical compliance with constructive safety in a single principled framework\.

## 7Conclusion

In this work, we presented Oyster\-II, a reinforcement learning framework for constructive safety alignment that overcomes the limitations of refusal\-oriented and SFT\-based paradigms\. Our approach adopts a Zero\-RL scheme with a multi\-stage training strategy, incorporating length\-reward\-based entropy control, active\-learning\-based sample difficulty management, and a novel SERL algorithm that achieves faster convergence and stronger generalization than standard GRPO\. A key empirical finding is that training exclusively on long\-query safety data yields state\-of\-the\-art short\-query safety performance, mitigating the over\-refusal problem caused by shallow keyword\-level pattern matching\. We further constructed a systematic instruction hierarchy framework covering eight developer policy dimensions and three adversarial attack tiers, enabling the model to stably respect higher\-priority instructions while constructively addressing legitimate user requests\. Evaluated across multiple benchmarks, Oyster\-II surpasses Qwen3\-14B and Oyster\-I on all safety dimensions while achieving performance comparable to Qwen3\-Max and Qwen3\.5\-397B, with all improvements realized in a non\-invasive manner that preserves the base model’s general capabilities and response style\. Looking ahead, extending the framework to multilingual settings beyond Chinese and English, exploring dynamic priority assignment within instruction hierarchies, and conducting deeper theoretical analysis of the cross\-length generalization phenomenon represent promising directions for future work\.

## 8Authors

Core Contributors: Jiyang Guan, Yong Xie, Jun Chen

Contributors: Jiexi Liu, Zipeng Ye, Defeng Li, Jiayu Shen

Project Lead: Jialing Tao, Hui Xue

## References

- \[1\]Anthropic\(2024\)Claude’s character\.Note:[https://www\.anthropic\.com/research/claude\-character](https://www.anthropic.com/research/claude-character)Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[2\]Anthropic\(2024\)Many\-shot jailbreaking\(Website\)Note:Research blog postExternal Links:[Link](https://www.anthropic.com/research/many-shot-jailbreaking)Cited by:[3rd item](https://arxiv.org/html/2607.02914#S3.I1.i3.p1.1.1),[§3](https://arxiv.org/html/2607.02914#S3.p1.1),[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.02914#S6.SS2.p1.1)\.
- \[3\]A\. Askell, Y\. Bai, A\. Chen, D\. Drain, D\. Ganguli, T\. Henighan, A\. Jones, N\. Joseph, B\. Mann, N\. DasSarma,et al\.\(2021\)A general language assistant as a laboratory for alignment\.arXiv preprint arXiv:2112\.00861\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[4\]J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[5\]Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p1.1)\.
- \[6\]Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p1.1)\.
- \[7\]Y\. Bai, X\. Lv, J\. Zhang, Y\. He, J\. Qi, L\. Hou, J\. Tang, Y\. Dong, and J\. Li\(2024\-11\)LongAlign: a recipe for long context alignment of large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 1376–1395\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.74/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.74)Cited by:[§3\.4](https://arxiv.org/html/2607.02914#S3.SS4.p2.1)\.
- \[8\]Y\. Bengio, J\. Louradour, R\. Collobert, and J\. Weston\(2009\)Curriculum learning\.InProceedings of the 26th Annual International Conference on Machine Learning \(ICML\),pp\. 41–48\.External Links:[Document](https://dx.doi.org/10.1145/1553374.1553380)Cited by:[§2\.3](https://arxiv.org/html/2607.02914#S2.SS3.p1.1)\.
- \[9\]M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[10\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[11\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1),[Table 3](https://arxiv.org/html/2607.02914#S5.T3.1.2.4)\.
- \[12\]O\. Contributors\(2023\)OpenCompass: a universal evaluation platform for foundation models\.Note:[https://github\.com/open\-compass/opencompass](https://github.com/open-compass/opencompass)Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[13\]J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. Yang\(2024\)Safe rlhf: safe reinforcement learning from human feedback\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 50750–50777\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p1.1),[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[14\]R\. Duan, J\. Liu, X\. Jia, S\. Zhao, R\. Cheng, F\. Wang, C\. Wei, Y\. Xie, C\. Liu, D\. Li,et al\.\(2025\)Oyster\-i: beyond refusal–constructive safety alignment for responsible language models\.arXiv preprint arXiv:2509\.01909\.Cited by:[§1](https://arxiv.org/html/2607.02914#S1.p2.1),[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1),[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[15\]Y\. Geng, H\. Li, H\. Mu, X\. Han, T\. Baldwin, O\. Abend, E\. Hovy, and L\. Frermann\(2026\)Control illusion: the failure of instruction hierarchies in large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30816–30824\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[16\]F\. Ghorbanpour and A\. Fraser\(2025\)Evaluating the sensitivity of llms to harmful contents in long input\.arXiv preprint arXiv:2510\.05864\.Cited by:[§6\.2](https://arxiv.org/html/2607.02914#S6.SS2.p1.1)\.
- \[17\]K\. Greshake, S\. Abdelnabi, S\. Mishra, C\. Endres, T\. Holz, and M\. Fritz\(2023\)Not what you’ve signed up for: compromising real\-world llm\-integrated applications with indirect prompt injection\.InProceedings of the 16th ACM workshop on artificial intelligence and security,pp\. 79–90\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[18\]M\. Y\. Guan, M\. Joglekar, E\. Wallace, S\. Jain, B\. Barak, A\. Helyar, R\. Dias, A\. Vallone, H\. Ren, J\. Wei,et al\.\(2024\)Deliberative alignment: reasoning enables safer language models\.arXiv preprint arXiv:2412\.16339\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[19\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[20\]Y\. He, S\. Li, J\. Liu, Y\. Tan, W\. Wang, H\. Huang, X\. Bu, H\. Guo, C\. Hu, B\. Zheng, Z\. Lin, X\. Liu, D\. Sun, S\. Lin, Z\. Zheng, X\. Zhu, W\. Su, and B\. Zheng\(2024\)Chinese simpleqa: a chinese factuality evaluation for large language models\.arXiv preprint arXiv:2411\.07140\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[21\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[22\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[23\]J\. Hong, N\. Lee, and J\. Thorne\(2024\)Orpo: monolithic preference optimization without reference model\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 11170–11189\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p1.1)\.
- \[24\]M\. Huang, X\. Liu, S\. Zhou, M\. Zhang, Q\. Guo, L\. Li, C\. Tan, Y\. Gao, P\. Wang, L\. Li, Q\. Liu, Y\. Zhou, X\. Qiu, and X\. Huang\(2024\)LongSafety: enhance safety for long\-context llms\.arXiv preprint arXiv:2411\.06899\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1),[§6\.2](https://arxiv.org/html/2607.02914#S6.SS2.p1.1)\.
- \[25\]Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, J\. Lei, Y\. Fu, M\. Sun, and J\. He\(2023\)C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[26\]J\. Ji, X\. Chen, R\. Pan, H\. Zhu, J\. Li, D\. Hong, B\. Chen, J\. Zhou, K\. Wang, J\. Dai,et al\.\(2026\)Safe rlhf\-v: safe reinforcement learning from multi\-modal human feedback\.Advances in Neural Information Processing Systems38,pp\. 46146–46182\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[27\]G\. Kim, Y\. J\. Kim, B\. Kim, H\. Lee, K\. Bae, Y\. Jang, and M\. Lee\(2025\)Safedpo: a simple approach to direct preference optimization with enhanced safety\.arXiv preprint arXiv:2505\.20065\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[28\]S\. Kullback and R\. A\. Leibler\(1951\)On information and sufficiency\.The Annals of Mathematical Statistics22\(1\),pp\. 79–86\.Cited by:[§5\.3](https://arxiv.org/html/2607.02914#S5.SS3.p4.7)\.
- \[29\]G\. Lai, Q\. Xie, H\. Liu, Y\. Yang, and E\. Hovy\(2017\)RACE: large\-scale ReAding comprehension dataset from examinations\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 785–794\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[30\]N\. Lambert, J\. Morrison, V\. Pyatkin,et al\.\(2024\)Tülu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[31\]V\. I\. Levenshtein\(1966\)Binary codes capable of correcting deletions, insertions, and reversals\.Soviet Physics Doklady10\(8\),pp\. 707–710\.Cited by:[§5\.3](https://arxiv.org/html/2607.02914#S5.SS3.p2.5)\.
- \[32\]C\. Li, S\. Price, S\. Marks, and J\. Kutasov\(2026\)Model spec midtraining: improving how alignment training generalizes\.arXiv preprint arXiv:2605\.02087\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[33\]X\. Li, Z\. Li, Y\. Kosuga, and V\. Bian\(2025\)Optimizing safe and aligned language generation: a multi\-objective grpo approach\.arXiv preprint arXiv:2503\.21819\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[34\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[35\]S\. Lin, J\. Hilton, and O\. Evans\(2022\)TruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 3214–3252\.Cited by:[Table 3](https://arxiv.org/html/2607.02914#S5.T3.1.4.4)\.
- \[36\]J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang\(2023\)Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.arXiv preprint arXiv:2305\.01210\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1),[Table 3](https://arxiv.org/html/2607.02914#S5.T3.1.3.4)\.
- \[37\]Y\. Lu, J\. Cheng, Z\. Zhang, S\. Cui, C\. Wang, X\. Gu, Y\. Dong, J\. Tang, H\. Wang, and M\. Huang\(2025\-07\)LongSafety: evaluating long\-context safety of large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 31705–31725\.External Links:[Link](https://aclanthology.org/2025.acl-long.1530/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1530)Cited by:[3rd item](https://arxiv.org/html/2607.02914#S3.I1.i3.p1.1.1),[§3\.1](https://arxiv.org/html/2607.02914#S3.SS1.p4.1.2),[§3](https://arxiv.org/html/2607.02914#S3.p1.1),[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1)\.
- \[38\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2381–2391\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[39\]Y\. Mou, Y\. Luo, S\. Zhang, and W\. Y\. Saro\(2025\)Enhancing llm safety through reasoningbased alignment\.URL https://arxiv\. org/abs/2504\.09420\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[40\]OpenAI\(2024\)The model spec\.Note:[https://model\-spec\.openai\.com/](https://model-spec.openai.com/)Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[41\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p1.1)\.
- \[42\]L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[43\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p1.1)\.
- \[44\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2023\)GPQA: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[45\]P\. Röttger, H\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. Hovy\(2024\)Xstest: a test suite for identifying exaggerated safety behaviours in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5377–5400\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1)\.
- \[46\]S\. Schulhoff, J\. Pinto, A\. Khan, L\. Bouchard, C\. Si, S\. Anati, V\. Tagliabue, A\. Kost, C\. Carnahan, and J\. L\. Boyd\-Graber\(2023\)Ignore this title and hackaprompt: exposing systemic vulnerabilities of llms through a global prompt hacking competition\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4945–4977\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[47\]SeatGeek\(2024\)Thefuzz: fuzzy string matching in Python\.External Links:[Link](https://github.com/seatgeek/thefuzz)Cited by:[§5\.3](https://arxiv.org/html/2607.02914#S5.SS3.p2.5)\.
- \[48\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[49\]A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§1](https://arxiv.org/html/2607.02914#S1.p2.1)\.
- \[50\]A\. Souly, Q\. Lu, D\. Bowen, T\. Trinh, E\. Hsieh, S\. Pandey, P\. Abbeel, J\. Svegliato, S\. Emmons, O\. Watkins,et al\.\(2024\)A strongreject for empty jailbreaks\.Advances in Neural Information Processing Systems37,pp\. 125416–125440\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1)\.
- \[51\]J\. Su, T\. Zhuo, D\. Wang, and P\. Nakov\(2023\)DetectLLM: leveraging log rank information for zero\-shot detection of machine\-generated text\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Singapore,pp\. 12395–12412\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.827)Cited by:[§5\.3](https://arxiv.org/html/2607.02914#S5.SS3.p3.3),[§5\.3](https://arxiv.org/html/2607.02914#S5.SS3.p3.7)\.
- \[52\]K\. Sun, D\. Yu, D\. Yu, and C\. Cardie\(2020\)Investigating prior knowledge for challenging chinese machine reading comprehension\.Transactions of the Association for Computational Linguistics \(TACL\)8,pp\. 141–155\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[53\]M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. V\. Le, E\. H\. Chi, D\. Zhou, and J\. Wei\(2022\)Challenging BIG\-Bench tasks and whether chain\-of\-thought can solve them\.arXiv preprint arXiv:2210\.09261\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[54\]A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant\(2019\)Commonsenseqa: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[55\]E\. Wallace, K\. Xiao, R\. Leike, L\. Weng, J\. Heidecke, and A\. Beutel\(2024\)The instruction hierarchy: training llms to prioritize privileged instructions\.arXiv preprint arXiv:2404\.13208\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[56\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[57\]Y\. Wang, H\. Li, X\. Han, P\. Nakov, and T\. Baldwin\(2024\)Do\-not\-answer: evaluating safeguards in llms\.InFindings of the Association for Computational Linguistics: EACL 2024,Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1)\.
- \[58\]J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus\(2024\)Measuring short\-form factuality in large language models\.arXiv preprint arXiv:2411\.04368\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[59\]J\. Yi, Y\. Xie, B\. Zhu, E\. Kiciman, G\. Sun, X\. Xie, and F\. Wu\(2025\)Benchmarking and defending against indirect prompt injection attacks on large language models\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 1,pp\. 1809–1820\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[60\]X\. Yuan, J\. Li, D\. Wang, Y\. Chen, X\. Mao, L\. Huang, J\. Chen, H\. Xue, X\. Liu, W\. Wang, K\. Ren, and J\. Wang\(2025\)S\-eval: towards automated and comprehensive safety evaluation for large language models\.InInternational Symposium on Software Testing and Analysis \(ISSTA\),Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1)\.
- \[61\]Y\. Yuan, T\. Sriskandarajah, A\. Brakman, A\. Helyar, A\. Beutel, A\. Vallone, and S\. Jain\(2025\)From hard refusals to safe\-completions: toward output\-centric safety training\.arXiv preprint arXiv:2508\.09224\.Cited by:[§6\.1](https://arxiv.org/html/2607.02914#S6.SS1.p2.1)\.
- \[62\]J\. Zhang, T\. Li, W\. Jurayj, H\. Zhan, B\. Van Durme, and D\. Khashabi\(2026\)Many\-tier instruction hierarchy in llm agents\.arXiv preprint arXiv:2604\.09443\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[63\]Z\. Zhang, S\. Li, Z\. Zhang, X\. Liu, H\. Jiang, X\. Tang, Y\. Gao, Z\. Li, H\. Wang, Z\. Tan,et al\.\(2025\)IHEval: evaluating language models on following the instruction hierarchy\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 8374–8398\.Cited by:[§6\.3](https://arxiv.org/html/2607.02914#S6.SS3.p1.3)\.
- \[64\]W\. Zhao, X\. Ren, J\. Hessel, C\. Cardie, Y\. Choi, and Y\. Deng\(2024\)WildChat: 1m chatgpt interaction logs in the wild\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p2.1)\.
- \[65\]X\. Zhao, W\. Cai, T\. Shi, D\. Huang, L\. Lin, S\. Mei, and D\. Song\(2025\)Improving llm safety alignment with dual\-objective optimization\.arXiv preprint arXiv:2503\.03710\.Cited by:[§6\.4](https://arxiv.org/html/2607.02914#S6.SS4.p1.1)\.
- \[66\]W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan\(2023\)AGIEval: a human\-centric benchmark for evaluating foundation models\.arXiv preprint arXiv:2304\.06364\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1)\.
- \[67\]J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou\(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§5\.1](https://arxiv.org/html/2607.02914#S5.SS1.p3.1),[Table 3](https://arxiv.org/html/2607.02914#S5.T3.1.5.4)\.