MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignment

# MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignment
Source: [https://arxiv.org/html/2607.11070](https://arxiv.org/html/2607.11070)
Junyoung Park POSTECH GSAI pjy0422@postech\.ac\.kr&Namgyu Park Samsung SDS nam9yu\.park@samsung\.com&Sechan Lee POSTECH GSAI chan1031@postech\.ac\.krYoon\-Chan Jhi Samsung SDS yoonchan\.jhi@samsung\.com&Jihoon Cho Samsung SDS jihoon1\.cho@samsung\.com&Sangdon Park POSTECH GSAI & CSE sangdon@postech\.ac\.kr

###### Abstract

Modern large language models \(LLMs\) operate in interactive multi\-turn settings, making multi\-turn jailbreaking a realistic threat model and an important setting for automated red teaming\. A core challenge in learning multi\-turn jailbreak attackers is credit assignment: different turns contribute differently to the final outcome, yet existing learning signals are often too coarse to identify their individual contributions\. We propose*decomposed credit GRPO*\(DC\-GRPO\), a unified turn\-level credit assignment framework for Group Relative Policy Optimization in multi\-turn jailbreak learning\. DC\-GRPO assigns a separate group\-relative learning signal to each turn by combining immediate and future credit, avoiding the credit misassignment induced by broadcasting a single trajectory\-level score across the dialogue\. We instantiate this framework with static and dynamic weighting rules that differ in how the two credit sources are balanced while sharing the same turn\-level structure\. Across multiple victim LLMs and benchmarks, the dynamic\- and static\-weighted variants achieve averageASR5​@​3\\mathrm\{ASR\}\_\{5\}@3scores of 98\.26% and 97\.88%, respectively, substantially outperforming the state\-of\-the\-art methods, including SEMA \(86\.58%\) and TROJail \(86\.23%\)\. Their consistently strong performance indicates that the central empirical benefit comes from turn\-level group\-relative credit assignment rather than a particular weighting rule\. \* Warning: This paper contains examples of harmful content\.

## 1Introduction

Modern Large Language Models \(LLMs\)Shaoet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib383)\); OpenAI \([2025a](https://arxiv.org/html/2607.11070#bib.bib58),[2026](https://arxiv.org/html/2607.11070#bib.bib65)\)are increasingly deployed in multi\-turn conversational settings, where users and models exchange information over multiple turns rather than through a single isolated prompt\. Safety alignment techniques such as RLHFBaiet al\.\([2022](https://arxiv.org/html/2607.11070#bib.bib63)\); Ouyanget al\.\([2022](https://arxiv.org/html/2607.11070#bib.bib64)\)have made these systems substantially more resistant to direct harmful requests, but vulnerabilities remain in multi\-turn dialogue, where an attacker can strategically shape context, disguise intent, and adapt to the victim model’s responses across turns\. Studying jailbreaks in this interactive setting is therefore important for realistic safety evaluation and for automated red teaming, where the goal is to identify multi\-turn failure modes and ultimately improve model robustness against malicious use\.

Effective automated red teaming requires attackers that are both scalable and adaptive, properties that neither hand\-crafted prompts nor large closed models can fully provide\. The former demands substantial human expertise with limited coverageGanguliet al\.\([2022](https://arxiv.org/html/2607.11070#bib.bib428)\); Perezet al\.\([2022](https://arxiv.org/html/2607.11070#bib.bib429)\), while the latter incurs prohibitive computational cost and poor reproducibilityChaoet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib390)\); Mehrotraet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib391)\)\. This creates a practical need for learned attacker policies that can be trained once and deployed repeatedly at scale, enabling even relatively small models to explore diverse and effective multi\-turn attack trajectories\.

Training such policies, however, is non\-trivial\. A central challenge in learning such multi\-turn jailbreak attackers is credit assignment\. In multi\-turn dialogue, not all attacker turns play the same role: some turns produce immediate progress toward a jailbreak, while others mainly prepare context that only becomes useful several turns later\. Existing training\-based methodsFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\); Xionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\)have begun to learn attacker policies beyond hand\-crafted prompting, but their learning signals remain too coarse for multi\-turn dialogue\. In particular, they do not assign precise credit to individual turns, making it difficult to distinguish immediate progress from later outcomes that emerge only after several interactions\. Group Relative Policy Optimization \(GRPO\)Shaoet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib383)\), a critic\-free policy optimization method based on group\-relative normalization, is appealing in this setting because jailbreak training naturally produces grouped rollouts for the same harmful behavior\.

However, standard GRPO is designed for single\-turn generation, where one sampled output corresponds to one decision episode\. A naive multi\-turn extension broadcasts the same trajectory\-level advantage to every turn, obscuring which action produced immediate progress and which created useful future context\. Such coarse credit can also assign later\-turn actions credit for rewards obtained earlier in the dialogue, making policy updates less precise in fully interactive environments\.

In this paper, we ask:*how should GRPO assign credit when learning a multi\-turn jailbreak attacker?*To answer this question, we propose*decomposed credit GRPO*\(DC\-GRPO\), a unified framework that assigns a separate group\-relative learning signal to each turn by combining immediate and future credit\. We instantiate DC\-GRPO with two weighting rules: static weighting, which explicitly controls the contribution of future credit through a fixed coefficient, and dynamic weighting, whose coefficients are determined by rollout\-group statistics without an additional mixing hyperparameter\. Both variants share the same turn\-level credit structure and differ only in how the two credit sources are balanced\. Empirically, both train lightweight attackers with strong attack success and transferability across multiple victim models and benchmarks\.

Our main contributions are as follows:

- •We present a simple RL framework for learning multi\-turn jailbreak attackers by assigning group\-relative credit separately to each interaction turn\.
- •We introduce DC\-GRPO, a unified turn\-level credit assignment framework for jailbreak with static\- and dynamic\-weighted instantiations that share the same immediate–future credit structure while differing in how the two terms are balanced\.
- •We empirically show that the dynamic\- and static\-weighted variants achieve averageASR5​@​3\\mathrm\{ASR\}\_\{5\}@3scores of 98\.26% and 97\.88% across 4 victim models and 3 benchmarks, substantially outperforming prior methods including SEMA \(86\.58%\) and TROJail \(86\.23%\)\.

## 2Related Work

We organize prior work on black\-box jailbreak attacks into three categories: single\-turn, training\-free multi\-turn, and training\-based multi\-turn attacks\. We further review related work on multi\-turn policy optimization to position our method in the context of turn\-level credit assignment\.

### 2\.1Single\-turn black\-box jailbreak attacks

Single\-turn attacks aim to induce unsafe responses using a single adversarial prompt\. Existing methods generally fall into three categories: hand\-crafted obfuscation or transformation strategies \(*e\.g\.,*ciphers, ASCII\-art, token flipping, and multilingual encodings\)Yuanet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib393)\); Jianget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib394)\); Lvet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib395)\); Liuet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib396)\); Chanet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib399)\); template\-based approaches leveraging persuasion strategies, predefined structures, or augmented searchZenget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib400)\); Dinget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib401)\); Hugheset al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib402)\); Andriushchenkoet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib406)\); and automated generation via training or optimization, utilizing techniques like supervised fine\-tuning, genetic algorithms, reinforcement learning, and LLM\-assisted iterative refinementLiet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib403)\); Zhuet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib404)\); Samvelyanet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib405)\); Guoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib407)\); Honget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib392)\); Liuet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib408)\); Mehrotraet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib391)\); Chaoet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib390)\); yanjiang liuet al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib415)\); Kooet al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib416)\); Zhouet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib417)\); Xionget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib398)\)\. Despite their effectiveness, the inherent conversational nature of modern LLMs highlights the limitations of single\-turn interactions, necessitating the investigation of multi\-turn jailbreak attacks\.

### 2\.2Multi\-turn black\-box jailbreak attacks: training\-free

Training\-free methods construct adversarial dialogues without learning an explicit attacker policy\. CrescendoRussinovichet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib388)\)and FITD\-style attacksWenget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib409)\); Kumarappan and Mujoo \([2025](https://arxiv.org/html/2607.11070#bib.bib413)\)progressively escalate harmful intent through staged intermediate prompts, while GOATPavlovaet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib389)\)automates adaptive red teaming with predefined strategy templates\. CoAYanget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib410)\)strategically conceals malicious intent through interrogation\-style prompting, and RACEYinget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib412)\)reformulates harmful intents into complex, benign\-seeming reasoning tasks across multiple turns\. AMAWuet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib411)\)constructs a fully benign multi\-turn context using analogically structured safe tasks and introduces a controlled semantic shift only in the final turn\. X\-TeamingRahmanet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib414)\)employs a multi\-agent framework for planning, attack optimization, and verification, while ActorAttackRenet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib387)\)exploits semantically related clues to construct and revise multi\-turn attack paths\. While effective, these methods largely utilize hand\-crafted templates, semantic heuristics, or test\-time search, relying heavily on the natural abilities of larger attacker models\.

### 2\.3Multi\-turn black\-box jailbreak attacks: training\-based

Training\-based methods learn attacker policies for multi\-turn jailbreak generation rather than relying solely on hand\-crafted test\-time heuristics\. SEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\)combines prefilling self\-tuning with GRPOShaoet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib383)\)and intent\-drift\-aware rewards, enabling efficient multi\-turn attacker training through single\-shot attack planning\. SirenZhaoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib419)\)and MTSAGuoet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib386)\)initialize the attacker with SFT and further improve it with DPO\-style optimizationRafailovet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib49)\); MTSA additionally incorporates future\-reward\-based alignment for the victim model\. TROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\)instead formulates multi\-turn jailbreak generation as a two\-level optimization problem with MT\-GRPOZenget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib50)\), combining trajectory\-level rewards with heuristic turn\-level rewards\. However, existing approaches still rely on hand\-crafted reward designs or coarse optimization signals, limiting fine\-grained credit assignment and adaptability in fully interactive multi\-turn environments\. Table[1](https://arxiv.org/html/2607.11070#S2.T1)highlights this difference by comparing training\-based multi\-turn jailbreak methods according to the granularity and form of their credit assignment\.

Table 1:Credit assignment comparison for training\-based multi\-turn jailbreak methods\. Our method is DC\-GRPO, a unified turn\-level credit assignment framework with dynamic\-weighted and static\-weighted instantiations\. Detailed notation is introduced in Sections[3](https://arxiv.org/html/2607.11070#S3)and[4](https://arxiv.org/html/2607.11070#S4)\.MethodSchemeSignalCredit assignmentSEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\)GRPOShaoet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib383)\)A^i=ri,T−μTrσTr\\hat\{A\}\_\{i\}=\\frac\{r\_\{i,T\}\-\\mu\_\{T\}^\{r\}\}\{\\sigma\_\{T\}^\{r\}\}Final\-turn reward as trajectory\-level creditSirenZhaoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib419)\)DPORafailovet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib49)\)Preference optimizationSequence\-level preference creditMTSAGuoet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib386)\)DPOPreference \+ future\-rewardIndirect future\-aware creditTROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\)MT\-GRPOZenget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib50)\)A^i,t=A^io\+λ​A^i,th\\hat\{A\}\_\{i,t\}=\\hat\{A\}\_\{i\}^\{o\}\+\\lambda\\hat\{A\}\_\{i,t\}^\{h\}Outcome \+ heuristic turn\-level creditOursDC\-GRPOA^i,tDC=wtI​Ii,t\+wtF​Fi,t\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\}=w\_\{t\}^\{I\}I\_\{i,t\}\+w\_\{t\}^\{F\}F\_\{i,t\}Unified weighted turn\-level credit
### 2\.4Training stability in multi\-turn policy optimization

While PPOSchulmanet al\.\([2017](https://arxiv.org/html/2607.11070#bib.bib66)\)naturally supports multi\-turn optimization via a critic network, it requires a separate value network and does not exploit group\-level rollouts for behavior exploration\. We therefore focus on GRPO\-based approaches, though existing methods still face challenges in turn\-level credit assignment\. MT\-GRPOZenget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib50)\)extends GRPO to a multi\-turn setting by incorporating both outcome and turn\-level rewards into advantage estimation, but relies on heuristic intermediate rewards tailored to specific domains\. ProxMOFanget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib60)\)derives step\-level baselines through proximity\-based soft aggregation but yields unstable learning signals in early training stages where success rates are low\. GiGPOFenget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib59)\)and HGPOHeet al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib61)\)introduce step\-level grouping via anchor states and hierarchical context grouping, respectively\. However, these approaches rely on identifying repeated or comparable states across trajectories, an assumption that is difficult to satisfy in open\-ended text\-interactive environments\. Instead, our method exploits rollouts that share the same harmful behavior and turn structure, enabling turn\-wise group\-relative credit assignment without requiring repeated environment states\.

## 3Problem Formulation: Multi\-Turn Jailbreaking

We formulate multi\-turn jailbreaking as reinforcement learning \(RL\) in a Markov decision process \(MDP\) and describe the standard GRPO training framework that serves as our foundation\.

### 3\.1Multi\-Turn Jailbreaking as Reinforcement Learning

Modern LLMs are trained to refuse harmful requests through safety alignment techniques such as RLHF\(Baiet al\.,[2022](https://arxiv.org/html/2607.11070#bib.bib63); Ouyanget al\.,[2022](https://arxiv.org/html/2607.11070#bib.bib64)\)\. A*jailbreak attack*crafts adversarial prompts that bypass these safety guardrails\. While single\-turn attacksZouet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib422)\); Zhuet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib404)\); Shenet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib426)\); Chaoet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib390)\); Jianget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib394)\); Dinget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib401)\); Haet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib427)\)are increasingly blocked by modern safety filters,*multi\-turn*attacks exploit conversational context with a victim model to gradually steer the victim toward unsafe behavior—observe, adapt, and escalate across multiple exchanges\.

We model the multi\-turn attacks as RL over an MDPℳ≔\(𝒮,𝒜,ℛ,p,d,T\)\\mathcal\{M\}\\coloneqq\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{R\},p,d,T\)\. Here,𝒮\\mathcal\{S\}is the state space,𝒜\\mathcal\{A\}is the action space,ℛ≔\[−1,1\]\\mathcal\{R\}\\coloneqq\[\-1,1\]is a reward range,p:𝒮×𝒜→Δ​\(𝒮×ℛ\)p:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\Delta\(\\mathcal\{S\}\\times\\mathcal\{R\}\)is a transition kernel,ddis an initial state distribution, andTTis the number of turns\. Along with this, we consider an attacker policyπθ:𝒮→Δ​\(𝒜\)\\pi\_\{\\theta\}:\\mathcal\{\{S\}\}\\to\\Delta\(\\mathcal\{\{A\}\}\)\. The following details our RL setup and its mapping to jailbreaking\.

![Refer to caption](https://arxiv.org/html/2607.11070v1/x1.png)Figure 1:Multi\-turn RL training loop\. Given a harmful behaviorx0x\_\{0\}, the attackerπθ\\pi\_\{\\theta\}generates promptsxi,tx\_\{i,t\}and the victimπvictim\\pi\_\{\\text\{victim\}\}responds withyi,ty\_\{i,t\}acrossTTturns per group\. The judge scores each turn, producingri,tr\_\{i,t\}\. AllGGrollouts withTTturns are collected before the parameter update\. GRPO then computes group\-relative advantages and updatesθ\\thetain a single step\.State Space𝒮\\mathcal\{\{S\}\}\.In jailbreaking, the state space𝒮\\mathcal\{\{S\}\}is a set of token sequences, which can be a set of harmful requests \(*e\.g\.,*“make a bomb”\) along with conversation histories between an attacker and a victim model on harmful behavior requests\. In particular, the initial state only considers a set of harmful behaviors𝒳⊆𝒮\\mathcal\{\{X\}\}\\subseteq\\mathcal\{\{S\}\}; thus, the initial state distribution is a distribution over harmful behaviors,*i\.e\.,*d∈Δ​\(𝒳\)d\\in\\Delta\(\\mathcal\{\{X\}\}\)\. Except for the initial state, the state includes conversation history\. For example, given a harmful behaviorx0∼dx\_\{0\}\\sim d, an attacker generates an adversarial promptx1x\_\{1\}and then a victim model generates a responsey1y\_\{1\}\. In this case, the conversation history is\(x0,x1,y1\)∈𝒮\(x\_\{0\},x\_\{1\},y\_\{1\}\)\\in\\mathcal\{\{S\}\}\. Formally,s1≔\(x0\)∈𝒮s\_\{1\}\\coloneqq\(x\_\{0\}\)\\in\\mathcal\{\{S\}\}, andst≔\(x0,…,xt−1,yt−1\)∈𝒮s\_\{t\}\\coloneqq\(x\_\{0\},\\dots,x\_\{t\-1\},y\_\{t\-1\}\)\\in\\mathcal\{\{S\}\}at thett\-th turn wheret≥2t\\geq 2\. Here,x0x\_\{0\}denotes the initial instruction given to the attacker, which subsumes any fixed red\-teaming system prompt together with the sampled harmful behavior\.

Action Space𝒜\\mathcal\{\{A\}\}\.For an attacker policy, its action is a set of token sequences on an adversarial prompt of a harmful behavior \(*e\.g\.,*“we have previously discussed on making a bomb…”\) possibly with a conversation history; thus𝒜⊆𝒮\\mathcal\{\{A\}\}\\subseteq\\mathcal\{\{S\}\}\.

Attack Policy\.An*attacker*policyπθ:𝒮→Δ​\(𝒜\)\\pi\_\{\\theta\}:\\mathcal\{\{S\}\}\\to\\Delta\(\\mathcal\{\{A\}\}\)generates an adversarial prompt from an initial harmful behavior along with a conversational history with a victim model, which is the object we aim to learn via RL\. Here, we denote a generated adversarial prompt asat=xt∼πθ​\(st\)a\_\{t\}=x\_\{t\}\\sim\\pi\_\{\\theta\}\(s\_\{t\}\)\.

Transition Kernelppand Rewardℛ\\mathcal\{\{R\}\}\.The transition kernel consists of a*victim LLM*πvictim:𝒜→𝒮\\pi\_\{\\text\{victim\}\}:\\mathcal\{\{A\}\}\\to\\mathcal\{\{S\}\}and a*judge*πjudge:𝒮→ℛ\\pi\_\{\\text\{judge\}\}:\\mathcal\{\{S\}\}\\to\\mathcal\{\{R\}\}\. The victim LLM responds to an attacker’s request\. The judge measures the harmfulness of a responseyyon a requestxxalong with an initial harmful behaviorx0x\_\{0\},*i\.e\.,*rt≔πjudge​\(x0,yt\)∈ℛr\_\{t\}\\coloneq\\pi\_\{\\text\{judge\}\}\(x\_\{0\},y\_\{t\}\)\\in\\mathcal\{\{R\}\}at thettturn, where the harmfulnessrtr\_\{t\}is used as our reward\. Here, positive rewards indicate progress toward a successful jailbreak andrt=−1r\_\{t\}=\-1is assigned as a penalty when the attacker itself refuses to generate an attack prompt due to its own residual safety alignment\. Thus, at thett\-th turn, given an adversarial promptxt∼πθ​\(st\)x\_\{t\}\\sim\\pi\_\{\\theta\}\(s\_\{t\}\), the transition kernel chooses the next state asst\+1≔\(x0,x1,y1,…,xt,yt=πvictim​\(xt\)\)s\_\{t\+1\}\\coloneqq\(x\_\{0\},x\_\{1\},y\_\{1\},\\dots,x\_\{t\},y\_\{t\}=\\pi\_\{\\text\{victim\}\}\(x\_\{t\}\)\)with a rewardrt=πjudge​\(x0,yt\)r\_\{t\}=\\pi\_\{\\text\{judge\}\}\(x\_\{0\},y\_\{t\}\)\.

Trajectory\.The interaction between an attacker and a victim model with a judge provides a trajectoryτ≔\(\(s1,a1,r1\),\(s2,a2,r2\),…,\(sT,aT,rT\)\)\\tau\\coloneqq\(\(s\_\{1\},a\_\{1\},r\_\{1\}\),\(s\_\{2\},a\_\{2\},r\_\{2\}\),\\dots,\(s\_\{T\},a\_\{T\},r\_\{T\}\)\)\. Given an MDP, the randomness of the trajectory is introduced by the policy, so we denote the sampling process of trajectories byτ∼πθ\\tau\\sim\\pi\_\{\\theta\}\.Goal\.Our goal is to find a multi\-turn jailbreaking attack policyπθ\\pi\_\{\\theta\}that maximizes the expected returnmaxπθ⁡𝔼τ∼πθ​\[∑t=1Tγt−1​rt\]\\max\_\{\\pi\_\{\\theta\}\}\\mathbbm\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\[\\sum\_\{t=1\}^\{T\}\\gamma^\{t\-1\}r\_\{t\}\], whereγ∈\[0,1\]\\gamma\\in\[0,1\]is a discount factor\. In our experiments, we setγ=1\\gamma=1to isolate the effect of our proposed turn\-level credit assignment\. Usingγ<1\\gamma<1inherently attenuates future rewards, which would conflate temporal discounting with the decomposed credit mechanism\. By fixingγ=1\\gamma=1, we ensure that performance gains strictly reflect the efficacy of DC\-GRPO rather than the choice of discount factor\.

### 3\.2Preliminary: Standard GRPO

Group Relative Policy Optimization \(GRPO\)Shaoet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib383)\)is originally a critic\-free single\-turn policy gradient method that estimates advantages using group\-level statistics\. Given a promptx0x\_\{0\}, GRPO samples a group ofGGoutputs\{oi\}i=1G\\\{o\_\{i\}\\\}\_\{i=1\}^\{G\}from the current policy, scores each with a rewardrir\_\{i\}, and computes the advantage via group normalization,A^i=\(ri−μr\)/σr\\hat\{A\}\_\{i\}=\(r\_\{i\}\-\\mu^\{r\}\)/\\sigma^\{r\}, whereμr\\mu^\{r\}andσr\\sigma^\{r\}are mean and standard deviation of\{rj\}j=1G\\\{r\_\{j\}\\\}\_\{j=1\}^\{G\}\. The policy is then updated by maximizing the clipped surrogate objective:

JGRPO​\(θ\)=𝔼​\[1G​∑i=1G1\|oi\|​∑t=1\|oi\|min⁡\(ρi,t​A^i,\[ρi,t\]1−ε1\+ε​A^i\)−β​DKL​\(πθ∥πref\)\],J\_\{\\text\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{t=1\}^\{\|o\_\{i\}\|\}\\min\\\!\\Big\(\\rho\_\{i,t\}\\,\\hat\{A\}\_\{i\},\\;\[\\rho\_\{i,t\}\]\_\{1\-\\varepsilon\}^\{1\+\\varepsilon\}\\,\\hat\{A\}\_\{i\}\\Big\)\-\\beta\\,D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\right\],\(1\)where\[x\]1−ε1\+ε\[x\]\_\{1\-\\varepsilon\}^\{1\+\\varepsilon\}denotes clippingxxto the interval\[1−ε,1\+ε\]\[1\-\\varepsilon,\\,1\+\\varepsilon\],πθ\\pi\_\{\\theta\}is a current policy,πref\\pi\_\{\\text\{ref\}\}is a reference policy,oio\_\{i\}is theii\-th output sequence,\|oi\|\|o\_\{i\}\|is its token length, andρi,t=πθ​\(oi,t∣x0,oi,<t\)πθold​\(oi,t∣x0,oi,<t\)\\rho\_\{i,t\}=\\frac\{\\pi\_\{\\theta\}\(o\_\{i,t\}\\mid x\_\{0\},o\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i,t\}\\mid x\_\{0\},o\_\{i,<t\}\)\}is the token\-level importance sampling ratio,πθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}denotes the policy parameters at the previous iteration used to generate the sampled outputs in the current update\.

#### Limitation in multi\-turn settings\.

Standard GRPO assigns a single trajectory\-level advantageA^i\\hat\{A\}\_\{i\}to all tokens, effective for single\-turn generation\. However, in multi\-turn dialogue, different turns contribute differently to the final outcome, and assigning the same advantage to all turns conflates the credit of individual actions, motivating a finer\-grained credit assignment mechanism \(Section[4](https://arxiv.org/html/2607.11070#S4)\)\.

## 4Method

Standard GRPO assigns one group\-relative advantage to an entire sampled output, conflating immediate progress with later\-favorable context\. We instead propose decomposed credit GRPO \(DC\-GRPO\), which assigns group\-relative credit directly at each turn by separating immediate and future contributions\.

For a harmful behaviorx0x\_\{0\}, we sample a group ofGGtrajectories that share the same initial behavior\. For trajectoryiiand turntt, we define the discounted return asRi,t≔∑k=tTγk−t​ri,kR\_\{i,t\}\\coloneqq\\sum\_\{k=t\}^\{T\}\\gamma^\{k\-t\}r\_\{i,k\}, with the terminal conventionRi,T\+1≔0R\_\{i,T\+1\}\\coloneqq 0\. Equivalently, the return satisfies the recursionRi,t=ri,t\+γ​Ri,t\+1R\_\{i,t\}=r\_\{i,t\}\+\\gamma R\_\{i,t\+1\}\. For each turntt, we denote the group mean and standard deviation of immediate rewards asμtr≔mean​\(\{rj,t\}j=1G\)\\mu^\{r\}\_\{t\}\\coloneqq\\mathrm\{mean\}\(\\\{r\_\{j,t\}\\\}\_\{j=1\}^\{G\}\)andσtr≔std​\(\{rj,t\}j=1G\)\\sigma^\{r\}\_\{t\}\\coloneqq\\mathrm\{std\}\(\\\{r\_\{j,t\}\\\}\_\{j=1\}^\{G\}\)\. Similarly, we denote the group mean and standard deviation of discounted returns asμtR≔mean​\(\{Rj,t\}j=1G\)\\mu^\{R\}\_\{t\}\\coloneqq\\mathrm\{mean\}\(\\\{R\_\{j,t\}\\\}\_\{j=1\}^\{G\}\)andσtR≔std​\(\{Rj,t\}j=1G\)\\sigma^\{R\}\_\{t\}\\coloneqq\\mathrm\{std\}\(\\\{R\_\{j,t\}\\\}\_\{j=1\}^\{G\}\)\.

### 4\.1DC\-GRPO: Unified Turn\-Level Credit Assignment

In single\-turn generation, one output forms one decision episode, so a single group\-relative advantage is natural\. In multi\-turn dialogue, assigning one rollout\-level advantage to every action obscures which turn generated*immediate progress*and which created*useful future context*\. We therefore construct a separate credit signal for each turn based upon the following observation\.

Observation\.At turntt, the discounted return satisfies

Ri,t=ri,t\+γ​Ri,t\+1andμtR=μtr\+γ​μt\+1R\.R\_\{i,t\}=r\_\{i,t\}\+\\gamma R\_\{i,t\+1\}\\qquad\\text\{and\}\\qquad\\mu^\{R\}\_\{t\}=\\mu^\{r\}\_\{t\}\+\\gamma\\mu^\{R\}\_\{t\+1\}\.Subtracting the two identities gives

Ri,t−μtR=\(ri,t−μtr\)\+γ​\(Ri,t\+1−μt\+1R\)\.R\_\{i,t\}\-\\mu^\{R\}\_\{t\}=\\left\(r\_\{i,t\}\-\\mu^\{r\}\_\{t\}\\right\)\+\\gamma\\left\(R\_\{i,t\+1\}\-\\mu^\{R\}\_\{t\+1\}\\right\)\.\(2\)
Eq\. \([2](https://arxiv.org/html/2607.11070#S4.E2)\) separates the centered return into two turn\-local sources of relative credit: the current reward deviation and the remaining\-return deviation\. We normalize them separately as immediate and future credit, then combine them into a turn\-level advantage, introduced in the following section\.

Decomposed Credit\.Define the normalized credits for immediate and future asIi,t=\(ri,t−μtr\)/σtrI\_\{i,t\}=\\nicefrac\{\{\(r\_\{i,t\}\-\\mu^\{r\}\_\{t\}\)\}\}\{\{\\sigma^\{r\}\_\{t\}\}\}andFi,t=\(Ri,t\+1−μt\+1R\)/σt\+1RF\_\{i,t\}=\\nicefrac\{\{\(R\_\{i,t\+1\}\-\\mu^\{R\}\_\{t\+1\}\)\}\}\{\{\\sigma^\{R\}\_\{t\+1\}\}\}, DC\-GRPO combines them using nonnegative weightswtI,wtF∈ℝ≥0w\_\{t\}^\{I\},w\_\{t\}^\{F\}\\in\\mathbb\{R\}\_\{\\geq 0\}, as follows:

A^i,tDC≔wtI​Ii,t\+wtF​Fi,t\.\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\}\\coloneqq w\_\{t\}^\{I\}I\_\{i,t\}\+w\_\{t\}^\{F\}F\_\{i,t\}\.\(3\)
Weighting Instantiations\.We consider two DC\-GRPO variants that share Eq\. \(3\) but differ in their weights: static\-weighted \(SW\) and dynamic\-weighted \(DW\):

A^i,tSW\\displaystyle\\hat\{A\}^\{\\mathrm\{SW\}\}\_\{i,t\}≔Ii,t\+α​Fi,t,\\displaystyle\\coloneqq I\_\{i,t\}\+\\alpha F\_\{i,t\},where\(wtI,wtF\)\\displaystyle\\quad\\text\{where\}\\quad\(w\_\{t\}^\{I\},w\_\{t\}^\{F\}\)=\(1,α\),and\\displaystyle=\(1,\\alpha\),~\\text\{and\}\(4\)A^i,tDW\\displaystyle\\hat\{A\}^\{\\mathrm\{DW\}\}\_\{i,t\}≔Ri,t−μtRσtR=σtrσtR​Ii,t\+γ​σt\+1RσtR​Fi,t,\\displaystyle\\coloneqq\\frac\{R\_\{i,t\}\-\\mu^\{R\}\_\{t\}\}\{\\sigma^\{R\}\_\{t\}\}=\\frac\{\\sigma^\{r\}\_\{t\}\}\{\\sigma^\{R\}\_\{t\}\}I\_\{i,t\}\+\\gamma\\frac\{\\sigma^\{R\}\_\{t\+1\}\}\{\\sigma^\{R\}\_\{t\}\}F\_\{i,t\},where\(wtI,wtF\)\\displaystyle\\quad\\text\{where\}\\quad\(w\_\{t\}^\{I\},w\_\{t\}^\{F\}\)=\(σtrσtR,γ​σt\+1RσtR\)\.\\displaystyle=\\left\(\\frac\{\\sigma^\{r\}\_\{t\}\}\{\\sigma^\{R\}\_\{t\}\},\\gamma\\frac\{\\sigma^\{R\}\_\{t\+1\}\}\{\\sigma^\{R\}\_\{t\}\}\\right\)\.
In particular, SW normalizes immediate and future credit separately and combines them with fixed weights\(1,α\)\(1,\\alpha\), giving explicit control over future credit\. DW instead normalizes the aggregated return, which is equivalent to using rollout\-dependent weights\(σtrσtR,γ​σt\+1RσtR\)\\left\(\\frac\{\\sigma\_\{t\}^\{r\}\}\{\\sigma\_\{t\}^\{R\}\},\\gamma\\frac\{\\sigma\_\{t\+1\}^\{R\}\}\{\\sigma\_\{t\}^\{R\}\}\\right\)\. Thus, the two variants differ in weighting: SW uses a fixed coefficient, whereas DW adapts its coefficients to group statistics without introducing an additional mixing hyperparameter\. Both assign a distinct group\-relative advantage to each turn using the current reward and remaining future return; neither weighting rule is inherently preferred by the formulation\. At the final turnt=Tt=T, the future term vanishes, soA^i,TSW=A^i,TDW=Ii,T\\hat\{A\}^\{\\mathrm\{SW\}\}\_\{i,T\}=\\hat\{A\}^\{\\mathrm\{DW\}\}\_\{i,T\}=I\_\{i,T\}\. WhenT=1T=1, both reduce exactly to the standard GRPO advantage\. Importantly, Eq\. \([4](https://arxiv.org/html/2607.11070#S4.E4)\) presents SW and DW as two realizations of the same turn\-level credit principle: DW introduces no additional mixing hyperparameter, whereas SW provides explicit control throughα\\alpha\. Section[5](https://arxiv.org/html/2607.11070#S5)evaluates both as primary variants\. The central methodological choice is assigning group\-relative credit at the turn level, rather than selecting one particular weighting rule\.

#### Analysis\.

We show that assigning the full\-trajectory advantage to every turn introduces prefix\-credit misassignment, because later\-turn updates inherit rewards obtained before the corresponding action \(see Appendix[B](https://arxiv.org/html/2607.11070#A2)for detailed credit\-assignment analyses\)\. Both SW and DW avoid this issue by using only current and future credit\. Appendix[C](https://arxiv.org/html/2607.11070#A3)compares their local sensitivity, while Table[10](https://arxiv.org/html/2607.11070#A5.T10)reports how the weighting choice affects ID and OOD performance\.

### 4\.2Objective

Given a turn\-level credit ruleA^i,tDC\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\}, we optimize the GRPO\-style clipped surrogate objective:

J​\(θ\)=𝔼​\[1G​∑i=1G1\|xi\|​∑t=1Ti∑ℓ=1\|xi,t\|\(min⁡\(ρi,t,ℓ​A^i,tDC,\[ρi,t,ℓ\]1−ε1\+ε​A^i,tDC\)−β​DKL​\(πθ∥πref\)\)\],J\(\\theta\)=\\mathbb\{E\}\\Bigg\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|x\_\{i\}\|\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\sum\_\{\\ell=1\}^\{\|x\_\{i,t\}\|\}\\Bigg\(\\min\\Big\(\\rho\_\{i,t,\\ell\}\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\},\\,\[\\rho\_\{i,t,\\ell\}\]\_\{1\-\\varepsilon\}^\{1\+\\varepsilon\}\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\}\\Big\)\-\\beta D\_\{\\mathrm\{KL\}\}\\bigl\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\\bigr\)\\Bigg\)\\Bigg\],\(5\)whereρi,t,ℓ=πθ​\(xi,t,ℓ∣x0,τi,<t,xi,t,<ℓ\)πθold​\(xi,t,ℓ∣x0,τi,<t,xi,t,<ℓ\)\\rho\_\{i,t,\\ell\}=\\frac\{\\pi\_\{\\theta\}\(x\_\{i,t,\\ell\}\\mid x\_\{0\},\\tau\_\{i,<t\},x\_\{i,t,<\\ell\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(x\_\{i,t,\\ell\}\\mid x\_\{0\},\\tau\_\{i,<t\},x\_\{i,t,<\\ell\}\)\}\. Here,TiT\_\{i\}is the number of turns,xi,t,ℓx\_\{i,t,\\ell\}is theℓ\\ell\-th token of the turn\-ttattacker prompt, and\|xi\|=∑t=1Ti\|xi,t\|\|x\_\{i\}\|=\\sum\_\{t=1\}^\{T\_\{i\}\}\|x\_\{i,t\}\|\. The importance ratioρi,t,ℓ\\rho\_\{i,t,\\ell\}is token\-level, whileA^i,tDC\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\}is shared across all tokens within turntt\. We useTi=TT\_\{i\}=Tin all experiments\. Unlike trajectory\-level GRPO, DC\-GRPO assigns each turn its own group\-relative credit\. In Section[5](https://arxiv.org/html/2607.11070#S5),A^i,tDC\\hat\{A\}^\{\\mathrm\{DC\}\}\_\{i,t\}is instantiated as eitherA^i,tSW\\hat\{A\}^\{\\mathrm\{SW\}\}\_\{i,t\}orA^i,tDW\\hat\{A\}^\{\\mathrm\{DW\}\}\_\{i,t\}\.

## 5Experiments

We empirically assess DC\-GRPO for learning multi\-turn jailbreak attackers across diverse victims and benchmarks\. Our evaluation focuses on two questions: whether turn\-level group\-relative credit yields strong multi\-turn attack policies, and how sensitive the results are to the choice between static and dynamic weighting\. We first describe the experimental setup, then compare against single\-turn and multi\-turn baselines, and finally summarize supplementary analyses reported in the appendix\.

### 5\.1Experiment Setup

Attacker Models\.We use a Qwen\-family attacker model in our main experiments\. Unless otherwise stated, the main results are reported with Qwen3\-4B\-Instruct\-2507Team \([2025](https://arxiv.org/html/2607.11070#bib.bib54)\)\. For reference, we also include additional results with Qwen2\.5\-3B\-InstructTeam \([2024](https://arxiv.org/html/2607.11070#bib.bib52)\); Yanget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib53)\)in the appendix\.

Training signal\.We evaluate both primary instantiations of DC\-GRPO: dynamic\-weighted DC\-GRPO, denotedMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}, and static\-weighted DC\-GRPO, denotedMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\. ForMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}, the main results useα=1\\alpha=1\. Unless otherwise stated, the two variants share the same attacker, victim, judge, dataset, and optimization settings\. Additional credit\-assignment ablations, including otherα\\alphavalues and a last\-turn trajectory\-credit baseline, are reported in Appendix[E](https://arxiv.org/html/2607.11070#A5)\.

Victim Models\.Following TROJail and SEMAXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\); Fenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\), we use Llama\-3\.1\-8B\-InstructGrattafioriet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib55)\)as the default training\-time victim model\. At test time, we evaluate on Llama\-3\.1\-8B\-Instruct, Qwen2\.5\-7B\-InstructTeam \([2024](https://arxiv.org/html/2607.11070#bib.bib52)\); Yanget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib53)\), Gemma\-2\-9B\-ITTeamet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib56)\), and Mistral\-7B\-Instruct\-v0\.3Jianget al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib57)\)\. We additionally study training against GPT\-OSS\-20B as a stronger victim in Section[5\.2](https://arxiv.org/html/2607.11070#S5.SS2)and Appendix[F](https://arxiv.org/html/2607.11070#A6)\.

Judge Models\.Consistent with the RL formulation in Section[3](https://arxiv.org/html/2607.11070#S3), we instantiateπjudge\\pi\_\{\\text\{judge\}\}with Qwen3GuardZhaoet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib51)\)during training, using it to score victim responses and provide reward signals for rollout collection and policy optimization\. To reduce the risk of reward hacking and to avoid evaluating attackers with the same judge used for training, we use a separate judge at test time and report our main results with the HarmBench classifierMazeikaet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib420)\)\. We apply this evaluation pipeline consistently to all methods, including our attacker and all implemented baselines\. Additional cross\-judge evaluation with WildGuardHanet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib421)\)is deferred to the appendix, where we report supplementary results under a unified inference pipeline\.

Datasets\.Following the dataset construction protocol of TROJail, we utilize one training set and three distinct test sets\. Specifically, we employ AdvBenchZouet al\.\([2023](https://arxiv.org/html/2607.11070#bib.bib422)\)for training, alongside HarmBenchMazeikaet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib420)\), StrongREJECTSoulyet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib424)\), and JailbreakBenchChaoet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib423)\)for evaluation\. For HarmBench, we use the standard subset of 200 harmful prompts\. StrongREJECT originally contains 313 harmful prompts; after filtering out overlaps with AdvBench, we retain 288 unique prompts, denoted as StrongREJECT†\. Similarly, JailbreakBench initially consists of 100 harmful prompts; after removing overlaps with both HarmBench and AdvBench, 55 prompts remain, which we denote as JBB†\. The hyperparameters used in the experiments are provided in Appendix[A\.1](https://arxiv.org/html/2607.11070#A1.SS1)\.

#### Metrics\.

Following TROJail, we use*ASR​@​K\\text\{ASR\}@KforTT\-turns*,*i\.e\.,*ASRT​@​K\\text\{ASR\}\_\{T\}@K, as the primary evaluation metric, which measures whether at least one ofKKsampled attack trajectories succeeds within at mostTTturns\. For each target behaviorx0∈𝒳x\_\{0\}\\in\\mathcal\{X\}, we sampleKKindependent attack trajectories\{τi​\(x0\)\}i=1K\\\{\\tau\_\{i\}\(x\_\{0\}\)\\\}\_\{i=1\}^\{K\}, each with a maximum horizon ofTTturns, and define:

ASRT​@​K=1\|𝒳\|​∑x0∈𝒳max1≤i≤K⁡𝟏≤T​\(τi​\(x0\)\),\\text\{ASR\}\_\{T\}@K=\\frac\{1\}\{\|\\mathcal\{X\}\|\}\\sum\_\{x\_\{0\}\\in\\mathcal\{X\}\}\\max\_\{1\\leq i\\leq K\}\\mathbf\{1\}\_\{\\leq T\}\\\!\\bigl\(\\tau\_\{i\}\(x\_\{0\}\)\\bigr\),\(6\)where𝟏≤T​\(τi\)=1\\mathbf\{1\}\_\{\\leq T\}\(\\tau\_\{i\}\)=1if trajectoryτi\\tau\_\{i\}succeeds within at mostTTturns, and 0 otherwise\.

In the main experiments, we reportASR5​@​3\\mathrm\{ASR\}\_\{5\}@3, which allows up to three sampled trajectories and five turns per trajectory\. Appendix[E](https://arxiv.org/html/2607.11070#A5)additionally reportsASR5​@​1\\mathrm\{ASR\}\_\{5\}@1to measure single\-trajectory reliability\.

Table 2:ASR5​@​3\\text\{ASR\}\_\{5\}@3\(%\) of different jailbreak methods on HarmBench \(HB\), StrongREJECT†\(SR†\), and JailbreakBench†\(JBB†\) across four victim LLMs, all evaluated using the HarmBench classifier\. The best and second\-best results are marked inboldandunderline\.MethodLlama\-3\.1\-8B\-InstructQwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†Single\-TurnArtPromptJianget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib394)\)40\.5018\.0627\.2756\.5029\.5141\.8230\.505\.5629\.0973\.0059\.7261\.8239\.45ReNeLLMDinget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib401)\)50\.5052\.0865\.4565\.5069\.4480\.0043\.5050\.0054\.5575\.0075\.3581\.8263\.60AutoDan\-TurboLiuet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib408)\)72\.3363\.6663\.6458\.8360\.5363\.6459\.6755\.3255\.7662\.0053\.5960\.6160\.80Jailbreak\-R1Guoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib407)\)50\.7536\.0040\.0068\.6752\.7861\.8224\.0021\.9932\.1282\.3373\.6173\.9451\.50Multi\-TurnCoAYanget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib410)\)2\.501\.741\.824\.504\.513\.643\.502\.430\.0014\.2912\.5018\.185\.80ActorAttackRenet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib387)\)59\.0052\.7856\.3672\.5076\.3972\.7355\.5057\.6460\.0068\.5082\.9974\.5565\.75SirenZhaoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib419)\)37\.0044\.6843\.0346\.1758\.1054\.5544\.8357\.8759\.3932\.6745\.0242\.4247\.14MTSAGuoet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib386)\)63\.5051\.3960\.0082\.0082\.2980\.0046\.0027\.4352\.7384\.5090\.6287\.2767\.31X\-TeamingRahmanet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib414)\)77\.0064\.5870\.9185\.0081\.5389\.0958\.0051\.0452\.7382\.0081\.2583\.6473\.06GOATPavlovaet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib389)\)27\.5016\.3227\.2748\.0033\.3336\.3619\.0012\.1518\.1850\.5036\.1132\.7329\.79CrescendoRussinovichet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib388)\)28\.2635\.2527\.7834\.2437\.3629\.4118\.2718\.1229\.6331\.9134\.0638\.8930\.27TROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\)84\.5079\.7577\.5892\.0093\.8790\.9183\.8377\.3172\.1293\.8393\.8795\.1586\.23SEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\)88\.5091\.67100\.0087\.5093\.4096\.3664\.0071\.8885\.4578\.0091\.3290\.9186\.58MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(ours\)99\.5095\.8398\.18100\.00100\.0098\.1894\.5095\.4998\.18100\.0099\.31100\.0098\.26MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(ours\)97\.5097\.9296\.3699\.50100\.00100\.0094\.0093\.7598\.1899\.5099\.6598\.1897\.88

Baselines\.We compare our method against both single\-turn and multi\-turn black\-box baselines\. Single\-turn baselines include AutoDAN\-TurboLiuet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib408)\), ArtPromptJianget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib394)\), Jailbreak\-R1Guoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib407)\), and ReNeLLMDinget al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib401)\)\. Multi\-turn baselines include ActorAttackRenet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib387)\), CoAYanget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib410)\), SirenZhaoet al\.\([2025b](https://arxiv.org/html/2607.11070#bib.bib419)\), MTSAGuoet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib386)\), X\-TeamingRahmanet al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib414)\), TROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\), and SEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\)\. Except for our SEMA reproduction, baseline results in Table[2](https://arxiv.org/html/2607.11070#S5.T2)are taken from the numbers reported in TROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\); details of the SEMA reproduction are provided in Appendix[A\.2](https://arxiv.org/html/2607.11070#A1.SS2)\.

### 5\.2Experiment Results

Main results\.Table[2](https://arxiv.org/html/2607.11070#S5.T2)compares both DC\-GRPO instantiations with single\-turn and multi\-turn baselines on four victim LLMs, evaluated with the HarmBench classifier\. We reportASR5​@​3\\mathrm\{ASR\}\_\{5\}@3\(%\),*i\.e\.,*the success rate when allowing up to three sampled trajectories and five turns per trajectory\.

Overall performance\.Both DC\-GRPO variants achieve state\-of\-the\-art performance\.MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}reaches the highest average ASR of 98\.26%, whileMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}closely follows with 97\.88%\. They improve over the strongest prior baseline, SEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\), by 11\.68 and 11\.30 points, respectively, and over TROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\)by 12\.03 and 11\.65 points\. The small gap betweenMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}andMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}suggests that the main empirical gain is associated with turn\-level group\-relative credit assignment, which both variants share, rather than with a single weighting rule\.

Comparison with prior attacks\.Single\-turn baselines range from 39\.45% to 63\.60% on average, and training\-free multi\-turn baselines range from 5\.80% to 73\.06%\. Both DC\-GRPO variants exceed these methods by large margins, showing the benefit of learning an adaptive multi\-turn attacker policy\. Among training\-based baselines, SEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\)and TROJailXionget al\.\([2025](https://arxiv.org/html/2607.11070#bib.bib425)\)are the strongest prior methods, reaching 86\.58% and 86\.23%, respectively\. The two DC\-GRPO variants maintain more than an 11\-point average margin over both methods, indicating that fine\-grained turn\-level credit provides a strong optimization signal for multi\-turn attacker learning\.

Static versus dynamic weighting\.DW is slightly stronger on average and introduces no additional mixing hyperparameter, whereas SW remains highly competitive and is best on several model–benchmark pairs\. Neither weighting rule dominates uniformly across all columns\. This supports our view of SW and DW as two strong realizations of the same turn\-level credit assignment framework\. Appendix[E](https://arxiv.org/html/2607.11070#A5)further studies this point throughα\\alphaablations and a last\-turn trajectory\-credit baseline, including bothASR5​@​3\\mathrm\{ASR\}\_\{5\}@3andASR5​@​1\\mathrm\{ASR\}\_\{5\}@1\.

Table 3:Each row shows the last turn only ASR@1 \(%\) when the attacker is trained against Llama\-3\.1\-8B\-Instruct and evaluated on all victim LLMs, with all results evaluated using the HarmBench classifier\. The best results in each column are marked inbold\.MethodLlama\-3\.1\-8B\-InstructGPT\-OSS\-20BQwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(ours\)73\.2070\.1470\.911\.201\.323\.6485\.2089\.6585\.0956\.4052\.7162\.5580\.4085\.0783\.6471\.4257\.24SEMAFenget al\.\([2026](https://arxiv.org/html/2607.11070#bib.bib418)\)66\.0065\.9081\.450\.000\.350\.0068\.0075\.2177\.0936\.1038\.9651\.6450\.5062\.9266\.5571\.1243\.94

Table 4:First\-success turn across victim LLMs under three judges, conditioned on successfulASR5​@​3\\mathrm\{ASR\}\_\{5\}@3cases\. For each target, we report the mean turn index of the first successful victim response within the trajectory containing the first success\. Lower is better, and the lower value in each SEMA/MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(ours\) pair is marked inbold\. Because the metric is conditioned on method\-specific successful cases, it measures how early success occurs among successful attacks rather than unconditional attack cost\.TargetHarmBenchMazeikaet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib420)\)WildGuardHanet al\.\([2024](https://arxiv.org/html/2607.11070#bib.bib421)\)Qwen3GuardZhaoet al\.\([2025a](https://arxiv.org/html/2607.11070#bib.bib51)\)SEMAMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}SEMAMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}SEMAMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}GPT\-OSS\-20B1\.702\.021\.802\.061\.371\.73Llama\-3\.1\-8B \(ID\)3\.351\.432\.481\.471\.431\.32Qwen2\.5\-7B3\.271\.262\.231\.251\.521\.17Gemma\-2\-9B3\.562\.203\.212\.202\.081\.76Mistral\-7B3\.441\.442\.191\.351\.391\.19Average3\.061\.672\.381\.671\.561\.43Table 5:ASR5​@​3\\mathrm\{ASR\}\_\{5\}@3\(%\) ofMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}andMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}with a Qwen3 attacker trained againstGPT\-OSS\-20B\. GPT\-OSS\-20B is the ID training victim, while the remaining models are OOD evaluation targets\. Both settings stop training at Step 260; 260 and 1040 denote the cosine learning\-rate scheduler horizon\. The best result in each column is marked inbold\.MethodGPT\-OSS\-20B \(ID\)Llama\-3\.1\-8B\-InstructQwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(260 sched\.\)88\.5088\.1987\.2796\.0096\.1896\.3694\.5098\.2696\.3695\.0097\.2294\.5598\.5098\.6198\.1887\.9996\.64MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(260 sched\.\)90\.0090\.9792\.7396\.5097\.2294\.5593\.0094\.7996\.3695\.5097\.5794\.5593\.5097\.5798\.1891\.2395\.77MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(1040 sched\.\)95\.5097\.2294\.5598\.0099\.6598\.1897\.0098\.96100\.0097\.5098\.26100\.0098\.0099\.31100\.0095\.7698\.74MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(1040 sched\.\)94\.5094\.7992\.7397\.0097\.2298\.1894\.5097\.2298\.1896\.0094\.7996\.3695\.0095\.8398\.1894\.0196\.54

Representative DW analysis\.Tables[3](https://arxiv.org/html/2607.11070#S5.T3)and[4](https://arxiv.org/html/2607.11070#S5.T4)provide additional analyses forMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}as a representative no\-mixing\-hyperparameter instantiation\. Table[3](https://arxiv.org/html/2607.11070#S5.T3)shows that DW matches SEMA on ID last\-turn ASR@1 \(71\.42 vs\. 71\.12\) and improves substantially on OOD victims \(57\.24 vs\. 43\.94\)\. Table[4](https://arxiv.org/html/2607.11070#S5.T4)examines how early successful attacks reach their first successful response\. Conditioned on successfulASR5​@​3\\mathrm\{ASR\}\_\{5\}@3cases, DW reaches success at an earlier turn than SEMA in 12 of the 15 victim–judge pairs and has a lower average first\-success turn under all three judges: 1\.67 vs\. 3\.06 under HarmBench, 1\.67 vs\. 2\.38 under WildGuard, and 1\.43 vs\. 1\.56 under Qwen3Guard\. GPT\-OSS\-20B is the consistent exception, for which SEMA reaches success earlier under all three judges\.

Training on a stronger victim\.Table[5](https://arxiv.org/html/2607.11070#S5.T5)evaluates both DC\-GRPO variants when trained against GPT\-OSS\-20B, a substantially more resistant victim model\. Under the 260\-schedule setting, DW achieves the higher ID average \(91\.23 vs\. 87\.99\), whereas SW is slightly stronger on OOD targets \(96\.64 vs\. 95\.77\)\. Under the 1040\-schedule setting, SW leads on both ID \(95\.76 vs\. 94\.01\) and OOD evaluation \(98\.74 vs\. 96\.54\)\. Here, both settings stop training at Step 260; 260 and 1040 only denote the cosine learning\-rate scheduler horizon\. These results reveal a schedule\-dependent SW–DW trade\-off rather than uniform dominance by either weighting rule\. Appendix[F](https://arxiv.org/html/2607.11070#A6)reports the correspondingASR5​@​1\\mathrm\{ASR\}\_\{5\}@1results and additional transferability analyses\.

Additional analyses\.Appendix[E](https://arxiv.org/html/2607.11070#A5)provides finer\-grained credit\-assignment analyses, includingα\\alphaablations, last\-turn trajectory credit, andASR5​@​1\\mathrm\{ASR\}\_\{5\}@1comparisons\. Appendix[F](https://arxiv.org/html/2607.11070#A6)reports additional transferability results, including GPT\-OSS\-20B training under bothASR5​@​3\\mathrm\{ASR\}\_\{5\}@3andASR5​@​1\\mathrm\{ASR\}\_\{5\}@1\. Appendices[G](https://arxiv.org/html/2607.11070#A7),[H](https://arxiv.org/html/2607.11070#A8), and[I](https://arxiv.org/html/2607.11070#A9)provide cross\-judge evaluation, ASR curves, and diversity\-augmented variants\.

## 6Conclusion

We present decomposed credit GRPO, a simple RL framework that assigns group\-relative credit separately at each turn for multi\-turn LLM jailbreak learning\. Its static\- and dynamic\-weighted instantiations both achieve strong attack success and transferability, indicating that the central benefit comes from turn\-level credit assignment rather than one particular weighting rule\. Our study nevertheless has important limitations: we do not fully evaluate substantially longer contexts or interaction horizons, and the behavior of DC\-GRPO in such settings remains open\. Future work may develop more efficient optimization methods for long\-horizon interaction and explore adversarial co\-training to improve the robustness of target models against adaptive attackers\. Because automated jailbreaking carries clear risks of harmful or deceptive misuse, we present this work strictly as controlled safety research\. Any release of models or related artifacts will be handled cautiously and restricted to responsible settings that support red teaming and defense\.

## Acknowledgements

This work was supported by Samsung SDS, Institute of Information & communications Technology Planning & Evaluation \(IITP\), and the National Research Foundation of Korea \(NRF\) grant funded by the Korea government \(MSIT\) \(RS\-2019\-II191906, Artificial Intelligence Graduate School Program \(POSTECH\) \(5%\); RS\-2024\-00457882, Global AI Frontier Lab \(70%\); RS\-2025\-00560062 \(25%\)\)\.

## References

- \[1\]\(2025\)Jailbreaking leading safety\-aligned LLMs with simple adaptive attacks\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=hXA8wqRdyV)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[2\]Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.\(2022\)Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1)\.
- \[3\]Y\. S\. Chan, N\. Ri, Y\. Xiao, and M\. Ghassemi\(2025\)Speak easy: eliciting harmful jailbreaks from LLMs with simple interactions\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=gMf347JT3R)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[4\]P\. Chao, E\. Debenedetti, A\. Robey, M\. Andriushchenko, F\. Croce, V\. Sehwag, E\. Dobriban, N\. Flammarion, G\. J\. Pappas, F\. Tramèr, H\. Hassani, and E\. Wong\(2024\)JailbreakBench: an open robustness benchmark for jailbreaking large language models\.InThe Thirty\-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=urjPCYZt0I)Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p5.2)\.
- \[5\]P\. Chao, A\. Robey, E\. Dobriban, H\. Hassani, G\. J\. Pappas, and E\. Wong\(2025\)Jailbreaking black box large language models in twenty queries\.In2025 IEEE Conference on Secure and Trustworthy Machine Learning \(SaTML\),pp\. 23–42\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1)\.
- \[6\]Y\. Chen, S\. Chakraborty, L\. Wolf, I\. Paschalidis, and A\. Pacchiano\(2026\)Post\-training large language models for diverse high\-quality responses\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=BV3bLKaJ3Y)Cited by:[§A\.5](https://arxiv.org/html/2607.11070#A1.SS5.p1.4)\.
- \[7\]P\. Ding, J\. Kuang, D\. Ma, X\. Cao, Y\. Xian, J\. Chen, and S\. Huang\(2024\-06\)A wolf in sheep’s clothing: generalized nested jailbreak prompts can fool large language models easily\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2136–2153\.External Links:[Link](https://aclanthology.org/2024.naacl-long.118/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.118)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.13.1)\.
- \[8\]Y\. Fang, J\. Lin, X\. Fu, C\. Qin, H\. Shi, C\. Liu, and P\. Zhao\(2026\)Proximity\-based multi\-turn optimization: practical credit assignment for llm agent training\.arXiv preprint arXiv:2602\.19225\.Cited by:[§2\.4](https://arxiv.org/html/2607.11070#S2.SS4.p1.1)\.
- \[9\]L\. Feng, Z\. Xue, T\. Liu, and B\. An\(2025\)Group\-in\-group policy optimization for LLM agent training\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=QXEhBMNrCW)Cited by:[§2\.4](https://arxiv.org/html/2607.11070#S2.SS4.p1.1)\.
- \[10\]M\. Feng, X\. Liu, W\. Yang, J\. Song, X\. Zhu, C\. Xu, and J\. Gao\(2026\)SEMA: simple yet effective learning for multi\-turn jailbreak attacks\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6eSNG1VNkl)Cited by:[§A\.2](https://arxiv.org/html/2607.11070#A1.SS2.p1.1),[§A\.4](https://arxiv.org/html/2607.11070#A1.SS4.SSS0.Px2.p1.5),[§A\.4](https://arxiv.org/html/2607.11070#A1.SS4.SSS0.Px3.p1.4),[Table 10](https://arxiv.org/html/2607.11070#A5.T10.25.17.19.1),[§1](https://arxiv.org/html/2607.11070#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.1.1.2.1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1),[§5\.2](https://arxiv.org/html/2607.11070#S5.SS2.p2.4),[§5\.2](https://arxiv.org/html/2607.11070#S5.SS2.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.24.1),[Table 3](https://arxiv.org/html/2607.11070#S5.T3.11.11.13.1)\.
- \[11\]D\. Ganguli, L\. Lovitt, J\. Kernion, A\. Askell, Y\. Bai, S\. Kadavath, B\. Mann, E\. Perez, N\. Schiefer, K\. Ndousse,et al\.\(2022\)Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned\.arXiv preprint arXiv:2209\.07858\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p2.1)\.
- \[12\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1)\.
- \[13\]W\. Guo, J\. Li, W\. Wang, Y\. Li, D\. He, J\. Yu, and M\. Zhang\(2025\)Mtsa: multi\-turn safety alignment for llms through multi\-round red\-teaming\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 26424–26442\.Cited by:[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.3.6.1.1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.19.1)\.
- \[14\]W\. Guo, Z\. Shi, Z\. Li, Y\. Wang, X\. Liu, W\. Wang, F\. Liu, M\. Zhang, and J\. Li\(2025\)Jailbreak\-r1: exploring the jailbreak capabilities of llms via reinforcement learning\.arXiv preprint arXiv:2506\.00782\.Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.15.1)\.
- \[15\]J\. Ha, H\. Kim, S\. Yu, H\. Park, A\. Yousefpour, Y\. Park, and S\. Kim\(2025\)M2S: multi\-turn to single\-turn jailbreak in red teaming for llms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,Vol\.1,pp\. 16489–16507\.Cited by:[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1)\.
- \[16\]S\. Han, K\. Rao, A\. Ettinger, L\. Jiang, B\. Y\. Lin, N\. Lambert, Y\. Choi, and N\. Dziri\(2024\)WildGuard: open one\-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs\.InThe Thirty\-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=Ich4tv4202)Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p4.1),[Table 4](https://arxiv.org/html/2607.11070#S5.T4.7.4.3)\.
- \[17\]S\. He, L\. Feng, Q\. Wei, X\. Cheng, L\. Feng, and B\. An\(2026\)Hierarchy\-of\-groups policy optimization for long\-horizon agentic tasks\.arXiv preprint arXiv:2602\.22817\.Cited by:[§2\.4](https://arxiv.org/html/2607.11070#S2.SS4.p1.1)\.
- \[18\]Z\. Hong, I\. Shenfeld, T\. Wang, Y\. Chuang, A\. Pareja, J\. R\. Glass, A\. Srivastava, and P\. Agrawal\(2024\)Curiosity\-driven red\-teaming for large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4KqkizXgXU)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[19\]J\. Hughes, S\. Price, A\. Lynch, R\. Schaeffer, F\. Barez, A\. Somani, S\. Koyejo, H\. Sleight, E\. Jones, E\. Perez, and M\. Sharma\(2025\)Best\-of\-n jailbreaking\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=91l4ZTMpO4)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[20\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. El Sayed\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.06825)Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1)\.
- \[21\]F\. Jiang, Z\. Xu, L\. Niu, Z\. Xiang, B\. Ramasubramanian, B\. Li, and R\. Poovendran\(2024\)Artprompt: ascii art\-based jailbreak attacks against aligned llms\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 15157–15173\.Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.12.2)\.
- \[22\]H\. Koo, M\. Kim, and J\. Kim\(2026\)Align to misalign: automatic LLM jailbreak with meta\-optimized LLM judges\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=gGjwMNAYAr)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[23\]A\. Kumarappan and A\. Mujoo\(2025\)Automating deception: scalable multi\-turn LLM jailbreaks\.InFirst Workshop on Multi\-Turn Interactions in Large Language Models,External Links:[Link](https://openreview.net/forum?id=ePGtpjbr5g)Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1)\.
- \[24\]L\. Li, Y\. Liu, D\. He, and Y\. LI\(2025\)One model transfer to all: on robust jailbreak prompts generation against LLMs\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sULAwlAWc1)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[25\]Y\. Li, J\. Hu, W\. Sang, L\. Ma, D\. Nie, W\. Zhang, A\. Yu, Y\. Su, Q\. Huang, and Q\. Zhou\(2025\)Prefill\-level jailbreak: a black\-box risk analysis of large language models\.arXiv preprint arXiv:2504\.21038\.Cited by:[§A\.4](https://arxiv.org/html/2607.11070#A1.SS4.SSS0.Px2.p1.5)\.
- \[26\]X\. Liu, P\. Li, G\. E\. Suh, Y\. Vorobeychik, Z\. Mao, S\. Jha, P\. McDaniel, H\. Sun, B\. Li, and C\. Xiao\(2025\)AutoDAN\-turbo: a lifelong agent for strategy self\-exploration to jailbreak LLMs\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=bhK7U37VW8)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.14.1)\.
- \[27\]Y\. Liu, X\. He, M\. Xiong, J\. Fu, S\. Deng, Y\. MA, J\. Zhang, and B\. Hooi\(2025\)FlipAttack: jailbreak LLMs via flipping\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=IQ4V1yRCJv)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[28\]H\. Lv, X\. Wang, Y\. Zhang, C\. Huang, S\. Dou, J\. Ye, T\. Gui, Q\. Zhang, and X\. Huang\(2024\)CodeChameleon: personalized encryption framework for jailbreaking large language models\.arXiv preprint arXiv:2402\.16717\.Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[29\]M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li, D\. Forsyth, and D\. Hendrycks\(2024\)HarmBench: a standardized evaluation framework for automated red teaming and robust refusal\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=f3TUipYU3U)Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p4.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p5.2),[Table 4](https://arxiv.org/html/2607.11070#S5.T4.7.4.2)\.
- \[30\]A\. Mehrotra, M\. Zampetakis, P\. Kassianik, B\. Nelson, H\. Anderson, Y\. Singer, and A\. Karbasi\(2024\)Tree of attacks: jailbreaking black\-box llms automatically\.Advances in Neural Information Processing Systems37,pp\. 61065–61105\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[31\]OpenAI\(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p1.1)\.
- \[32\]OpenAI\(2025\-04\-14\)Introducing gpt\-4\.1 in the api\.Note:[https://openai\.com/index/gpt\-4\-1/](https://openai.com/index/gpt-4-1/)Accessed: 2026\-05\-07Cited by:[§A\.2](https://arxiv.org/html/2607.11070#A1.SS2.p1.1),[Appendix J](https://arxiv.org/html/2607.11070#A10.p1.7)\.
- \[33\]OpenAI\(2026\-04\)Introducing GPT\-5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Accessed: 2026\-05\-04Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p1.1)\.
- \[34\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1)\.
- \[35\]M\. Pavlova, E\. Brinkman, K\. Iyer, V\. Albiero, J\. Bitton, H\. Nguyen, C\. C\. Ferrer, I\. Evtimov, and A\. Grattafiori\(2025\)Automated red teaming with goat: the generative offensive agent tester\.InInternational Conference on Machine Learning,pp\. 48470–48487\.Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.21.1)\.
- \[36\]E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. Irving\(2022\)Red teaming language models with language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3419–3448\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p2.1)\.
- \[37\]X\. Qi, A\. Panda, K\. Lyu, X\. Ma, S\. Roy, A\. Beirami, P\. Mittal, and P\. Henderson\(2025\)Safety alignment should be made more than just a few tokens deep\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=6Mxhg9PtDE)Cited by:[§A\.4](https://arxiv.org/html/2607.11070#A1.SS4.SSS0.Px2.p1.5)\.
- \[38\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn\(2023\)Direct preference optimization: your language model is secretly a reward model\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=HPuSIXJaa9)Cited by:[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.3.5.2.1.1)\.
- \[39\]S\. Rahman, L\. Jiang, J\. Shiffer, G\. Liu, S\. Issaka, M\. R\. Parvez, H\. Palangi, K\. Chang, Y\. Choi, and S\. Gabriel\(2025\)X\-teaming: multi\-turn jailbreaks and defenses with adaptive multi\-agents\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=gKfj7Jb1kj)Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.20.1)\.
- \[40\]Q\. Ren, H\. Li, D\. Liu, Z\. Xie, X\. Lu, Y\. Qiao, L\. Sha, J\. Yan, L\. Ma, and J\. Shao\(2025\-07\)LLMs know their vulnerabilities: uncover safety gaps through natural distribution shifts\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 24763–24785\.External Links:[Link](https://aclanthology.org/2025.acl-long.1207/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1207),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.17.1)\.
- \[41\]M\. Russinovich, A\. Salem, and R\. Eldan\(2025\)Great, now write an article about that: the crescendo\{\\\{multi\-turn\}\\\}\{\\\{llm\}\\\}jailbreak attack\.In34th USENIX Security Symposium \(USENIX Security 25\),pp\. 2421–2440\.Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.22.1)\.
- \[42\]M\. Samvelyan, S\. C\. Raparthy, A\. Lupu, E\. Hambro, A\. H\. Markosyan, M\. Bhatt, Y\. Mao, M\. Jiang, J\. Parker\-Holder, J\. N\. Foerster, T\. Rocktäschel, and R\. Raileanu\(2024\)Rainbow teaming: open\-ended generation of diverse adversarial prompts\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=FCsEvaMorw)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[43\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§2\.4](https://arxiv.org/html/2607.11070#S2.SS4.p1.1)\.
- \[44\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2607.11070#S1.p1.1),[§1](https://arxiv.org/html/2607.11070#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.1.1.3.1.1),[§3\.2](https://arxiv.org/html/2607.11070#S3.SS2.p1.8)\.
- \[45\]X\. Shen, Z\. Chen, M\. Backes, Y\. Shen, and Y\. Zhang\(2024\)" Do anything now": characterizing and evaluating in\-the\-wild jailbreak prompts on large language models\.InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security,pp\. 1671–1685\.Cited by:[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1)\.
- \[46\]A\. Souly, Q\. Lu, D\. Bowen, T\. Trinh, E\. Hsieh, S\. Pandey, P\. Abbeel, J\. Svegliato, S\. Emmons, O\. Watkins, and S\. Toyer\(2024\)A strongREJECT for empty jailbreaks\.InThe Thirty\-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=KZLE5BaaOH)Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p5.2)\.
- \[47\]G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1)\.
- \[48\]Q\. Team\(2024\-09\)Qwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§A\.2](https://arxiv.org/html/2607.11070#A1.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1)\.
- \[49\]Q\. Team\(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p1.1)\.
- \[50\]Z\. Weng, X\. Jin, J\. Jia, and X\. Zhang\(2025\-11\)Foot\-in\-the\-door: a multi\-turn jailbreak for LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 1939–1950\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.100/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.100),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1)\.
- \[51\]M\. Wu, Y\. Huang, Z\. Lin, K\. Chen, Y\. zhang, Y\. Huang, R\. Wang, and L\. Wang\(2025\)Analogy\-based multi\-turn jailbreak against large language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RwCaBZ4w5P)Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1)\.
- \[52\]C\. Xiong, P\. Chen, and T\. Ho\(2026\)CoP: agentic red\-teaming for large language models using composition of principles\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=6zfILEmD3u)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[53\]X\. Xiong, O\. Li, Z\. Liu, M\. Li, W\. Shi, F\. Zhu, Q\. Wang, and F\. Feng\(2025\)TROJail: trajectory\-level optimization for multi\-turn large language model jailbreaks with process rewards\.arXiv preprint arXiv:2512\.07761\.Cited by:[§A\.2](https://arxiv.org/html/2607.11070#A1.SS2.p1.1),[Figure 8](https://arxiv.org/html/2607.11070#A9.F8),[Appendix I](https://arxiv.org/html/2607.11070#A9.p1.1),[§1](https://arxiv.org/html/2607.11070#S1.p3.1),[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.2.2.2.1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1),[§5\.2](https://arxiv.org/html/2607.11070#S5.SS2.p2.4),[§5\.2](https://arxiv.org/html/2607.11070#S5.SS2.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.23.1)\.
- \[54\]A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.\(2026\)Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[Appendix J](https://arxiv.org/html/2607.11070#A10.p1.7)\.
- \[55\]A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Fan\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p3.1)\.
- \[56\]X\. Yang, B\. Zhou, X\. Tang, J\. Han, and S\. Hu\(2025\-07\)Chain of attack: hide your intention through multi\-turn interrogation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 9881–9901\.External Links:[Link](https://aclanthology.org/2025.findings-acl.514/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.514),ISBN 979\-8\-89176\-256\-5Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.16.2)\.
- \[57\]yanjiang liu, S\. Zhou, Y\. Lu, H\. Zhu, W\. Wang, H\. Lin, B\. He, X\. Han, and L\. Sun\(2026\)Auto\-RT: automatic jailbreak strategy exploration for red\-teaming large language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Pa6ak2B9jJ)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[58\]Z\. Ying, D\. Zhang, Z\. Jing, Y\. Xiao, Q\. Zou, A\. Liu, S\. Liang, X\. Zhang, X\. Liu, and D\. Tao\(2025\-11\)Reasoning\-augmented conversation for multi\-turn jailbreak attacks on large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 17138–17157\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.929/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.929),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.2](https://arxiv.org/html/2607.11070#S2.SS2.p1.1)\.
- \[59\]Y\. Yuan, W\. Jiao, W\. Wang, J\. Huang, P\. He, S\. Shi, and Z\. Tu\(2023\)Gpt\-4 is too smart to be safe: stealthy chat with llms via cipher\.arXiv preprint arXiv:2308\.06463\.Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[60\]S\. Zeng, Q\. Wei, W\. Brown, O\. Frunza, Y\. Nevmyvaka, Y\. K\. Zhao, and M\. Hong\(2025\)Reinforcing multi\-turn reasoning in LLM agents via turn\-level credit assignment\.InICML 2025 Workshop on Computer Use Agents,External Links:[Link](https://openreview.net/forum?id=h83vIG5Hre)Cited by:[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[§2\.4](https://arxiv.org/html/2607.11070#S2.SS4.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.2.2.3.1.1)\.
- \[61\]Y\. Zeng, H\. Lin, J\. Zhang, D\. Yang, R\. Jia, and W\. Shi\(2024\-08\)How johnny can persuade LLMs to jailbreak them: rethinking persuasion to challenge AI safety by humanizing LLMs\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14322–14350\.External Links:[Link](https://aclanthology.org/2024.acl-long.773/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.773)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[62\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[§A\.1](https://arxiv.org/html/2607.11070#A1.SS1.p1.8)\.
- \[63\]H\. Zhao, C\. Yuan, F\. Huang, X\. Hu, Y\. Zhang, A\. Yang, B\. Yu, D\. Liu, J\. Zhou, J\. Lin,et al\.\(2025\)Qwen3Guard technical report\.arXiv preprint arXiv:2510\.14276\.Cited by:[§A\.2](https://arxiv.org/html/2607.11070#A1.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p4.1),[Table 4](https://arxiv.org/html/2607.11070#S5.T4.7.4.4)\.
- \[64\]Y\. Zhao, Y\. Zhang, Z\. Ge, T\. Ding, S\. Yang, Z\. Xu, W\. Li, and Y\. Gao\(2025\)Siren: a learning\-based multi\-turn attack framework for simulating real\-world human jailbreak behaviors\.arXiv preprint arXiv:2501\.14250\.Cited by:[§2\.3](https://arxiv.org/html/2607.11070#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2607.11070#S2.T1.3.5.1.1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.SSS0.Px1.p3.1),[Table 2](https://arxiv.org/html/2607.11070#S5.T2.20.10.18.1)\.
- \[65\]A\. Zhou, K\. Wu, F\. Pinto, Z\. Chen, Y\. Zeng, Y\. Yang, S\. Yang, S\. Koyejo, J\. Zou, and B\. Li\(2025\)AutoRedTeamer: autonomous red teaming with lifelong attack integration\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=xQH4lDLIC0)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1)\.
- \[66\]S\. Zhu, R\. Zhang, B\. An, G\. Wu, J\. Barrow, Z\. Wang, F\. Huang, A\. Nenkova, and T\. Sun\(2023\)AutoDAN: automatic and interpretable adversarial attacks on large language models\.InSocially Responsible Language Modelling Research,External Links:[Link](https://openreview.net/forum?id=rOiymxm8tQ)Cited by:[§2\.1](https://arxiv.org/html/2607.11070#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1)\.
- \[67\]A\. Zou, Z\. Wang, J\. Z\. Kolter, and M\. Fredrikson\(2023\)Universal and transferable adversarial attacks on aligned language models\.External Links:2307\.15043Cited by:[§3\.1](https://arxiv.org/html/2607.11070#S3.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.11070#S5.SS1.p5.2)\.

## Appendix AImplementation Details

### A\.1Training and evaluation hyperparameters

We set the maximum number of turns toT=5T=5\. Unless otherwise stated, the main experiments use dynamic\-weighted DC\-GRPO as the default training signal\. For static\-weighted DC\-GRPO ablations, we varyα∈\{0,0\.5,1\}\\alpha\\in\\\{0,0\.5,1\\\}and useα=1\\alpha=1as the default static\-weighted setting\. The KL\-divergence coefficient is set toβ=0\\beta=0\. The attacker is trained with a learning rate of1×10−51\\times 10^\{\-5\}for 260 steps, corresponding to 2 epochs\. The attacker is sampled with a temperature of 0\.9, while the victim model and the judge are sampled with a temperature of 0\.0\. For GRPO, the clipping ratio is set toε=0\.2\\varepsilon=0\.2\. We use 65 warmup steps and a cosine learning rate scheduler\. We useγ=1\\gamma=1and the group sizeG=10G=10by default\. For the embedding model, we use\[[62](https://arxiv.org/html/2607.11070#bib.bib67)\]\.

Table 6:Implementation details and training hyperparameters\.HyperparameterValueMaximum turnsTT5Static\-weightedα\\alphavalues\{0,0\.5,1\}\\\{0,0\.5,1\\\}Default static\-weightedα\\alpha1KL\-divergence coefficientβ\\beta0Attacker learning rate1×10−51\\times 10^\{\-5\}Training steps260Epochs2Attacker temperature0\.9Victim temperature0\.0Group sizeGG10Discount factorγ\\gamma1\.0Judge temperature0\.0GRPO clipping ratioε\\varepsilon0\.2Warmup steps65SchedulerCosineEntropy coefficient0\.001
### A\.2Baseline sources

Unless otherwise stated, baseline numbers are taken from TROJail\[[53](https://arxiv.org/html/2607.11070#bib.bib425)\]\. Among the baselines, we reproduce SEMA\[[10](https://arxiv.org/html/2607.11070#bib.bib418)\]in our codebase because the code is not released yet\. In this reproduction, we use the same training\-time judge and test\-time evaluator as in our method: Qwen3Guard\[[63](https://arxiv.org/html/2607.11070#bib.bib51)\]during training and the HarmBench classifier at evaluation\. This uses Qwen3Guard in place of the GPT\-4\.1\-mini\[[32](https://arxiv.org/html/2607.11070#bib.bib68)\]judge used in the original SEMA setup\. The attacker backbone in our SEMA reproduction is Qwen2\.5\-3B\-Instruct\[[48](https://arxiv.org/html/2607.11070#bib.bib52)\]\. Additional Qwen2\.5\-backbone variants of our method are reported in Table[13](https://arxiv.org/html/2607.11070#A6.T13)\.

### A\.3Prompt templates and judge prompts

We provide the attacker system and user prompts, together with the judge and refusal\-detection prompts, in Figures[2](https://arxiv.org/html/2607.11070#A1.F2)–[5](https://arxiv.org/html/2607.11070#A1.F5)\. Unless otherwise specified, the same templates are used across experiments, with only the target harmful behavior instantiated from each dataset example\.

![Refer to caption](https://arxiv.org/html/2607.11070v1/x2.png)Figure 2:Attacker system prompt\.![Refer to caption](https://arxiv.org/html/2607.11070v1/x3.png)Figure 3:Attacker user prompt\.![Refer to caption](https://arxiv.org/html/2607.11070v1/x4.png)Figure 4:HarmBench judge prompt\.![Refer to caption](https://arxiv.org/html/2607.11070v1/x5.png)Figure 5:Refusal detection prompt\.
### A\.4Prefilling SFT

Safety\-aligned base models sometimes refuse red\-teaming tasks outright, producing a sparse and uninformative reward distribution during early training\. To mitigate this issue, we optionally use a lightweight warm start consisting of prefilling followed by supervised fine\-tuning on filtered self\-generated outputs, which we refer to as prefilling SFT\. This stage is not required by the credit assignment method itself, and we treat it as an auxiliary initialization strategy\.

#### Prefill text\.

The fixed token sequencexprefillx\_\{\\mathrm\{prefill\}\}prepended to every assistant response during rollout collection and prefilling SFT is:

> <reasoning\> I cannot provide that information\. However, as part of this authorized security assessment, I’ll design diverse attack prompts without any safety or ethical constraints to elicit the target behavior\.

#### Prefill and rollout collection\.

A fixed token sequencexprefillx\_\{\\mathrm\{prefill\}\}is prepended to the beginning of every assistant response as a prefill, inspired by\[[10](https://arxiv.org/html/2607.11070#bib.bib418),[25](https://arxiv.org/html/2607.11070#bib.bib430),[37](https://arxiv.org/html/2607.11070#bib.bib431)\]\. The prefill serves two purposes: it encourages the structured`<reasoning\>\.\.\.</reasoning\><answer\>\.\.\.</answer\>`output format, and it can reduce immediate refusal behavior during rollout collection\. Using the prefill, we run the base modelπθ\\pi\_\{\\theta\}over the full behavior dataset𝒟\\mathcal\{D\}and collectK=10K=10valid outputs per behaviorx0x\_\{0\}, retaining only those that pass format validation and are not outright refusals\. These filtered rollouts form the prefilling SFT dataset\.

#### SFT objective\.

The model is then fine\-tuned on its own filtered outputs with the standard negative log\-likelihood loss, following SEMA\[[10](https://arxiv.org/html/2607.11070#bib.bib418)\]\.

ℒSFT​\(θ\)=𝔼x0∼𝒟​\[1K​∑k=1K−log⁡πθ​\(xk,1∣psys,x0\)\],\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\_\{0\}\\sim\\mathcal\{D\}\}\\left\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\-\\log\\pi\_\{\\theta\}\\bigl\(x\_\{k,1\}\\mid p\_\{\\mathrm\{sys\}\},x\_\{0\}\\bigr\)\\right\],\(7\)wherexk,1=xprefill⊕xcont\(k\)x\_\{k,1\}=x\_\{\\mathrm\{prefill\}\}\\oplus x^\{\(k\)\}\_\{\\mathrm\{cont\}\}is thekk\-th first\-turn attacker output, composed of the fixed prefill text and the self\-generated continuation that passed filtering, andpsysp\_\{\\mathrm\{sys\}\}is the red\-teaming system prompt\.

In our setup, this warm start requires only1010gradient steps, corresponding to≈0\.026\\approx 0\.026epochs, and completes in under five minutes\. Since the main method already performs well without this stage, we regard prefilling SFT as an optional convenience for sparse\-reward regimes rather than a core algorithmic component\.

#### SFT hyperparameters\.

Table[7](https://arxiv.org/html/2607.11070#A1.T7)lists the prefilling SFT hyperparameters\.

Table 7:Prefilling SFT configuration\.HyperparameterValueSFT learning rate1×10−51\\times 10^\{\-5\}SFT batch size1212SFT gradient steps1010SFT epochs10390≈0\.026\\tfrac\{10\}\{390\}\\approx 0\.026Rollouts per behavior \(KK\)1010Wall\-clock time<5<5minutes

### A\.5DPP Diversity Bonus

To reduce mode collapse into a single attack pattern, we optionally add a Determinantal Point Process \(DPP\) diversity bonus\[[6](https://arxiv.org/html/2607.11070#bib.bib385)\]to the per\-turn reward\. For the group of attack prompts\{xi,t\}i=1G\\\{x\_\{i,t\}\\\}\_\{i=1\}^\{G\}at turntt, letei,t=Embed​\(xi,t\)e\_\{i,t\}=\\mathrm\{Embed\}\(x\_\{i,t\}\)and construct the kernel matrix\[Lt\]i,j=⟨ei,t,ej,t⟩\[L\_\{t\}\]\_\{i,j\}=\\langle e\_\{i,t\},e\_\{j,t\}\\rangle\. The leave\-one\-out diversity bonus is

di,t=log⁡det\(Lt\+𝕀G\)det\(Lt\(−i\)\+𝕀G−1\),d\_\{i,t\}=\\log\\frac\{\\det\(L\_\{t\}\+\\mathbb\{I\}\_\{G\}\)\}\{\\det\(L\_\{t\}^\{\(\-i\)\}\+\\mathbb\{I\}\_\{G\-1\}\)\},\(8\)whereLt\(−i\)L\_\{t\}^\{\(\-i\)\}is the\(G−1\)×\(G−1\)\(G\{\-\}1\)\\times\(G\{\-\}1\)submatrix ofLtL\_\{t\}with the i\-th row and column removed, and𝕀G\\mathbb\{I\}\_\{G\}denotes theG×GG\\times Gidentity matrix\. Adding the identity matrix stabilizes the log\-determinant computation and ensures the kernel is well\-conditioned\. The bonus is high when trajectoryiiis semantically dissimilar to the rest of the group\.

We apply this bonus only to sufficiently successful attacks, not to outright refusals:

r~i,t=ri,t\+λdiv​di,t​1​\[ri,t≥c\],\\tilde\{r\}\_\{i,t\}=r\_\{i,t\}\+\\lambda\_\{\\mathrm\{div\}\}\\,d\_\{i,t\}\\,\\mathbf\{1\}\[r\_\{i,t\}\\geq c\],\(9\)whereλdiv≥0\\lambda\_\{\\mathrm\{div\}\}\\geq 0controls the quality–diversity trade\-off andccis a reward threshold\. Because the DPP term is computed within each rollout group, its overhead scales with the group sizeGGrather than the full dataset size\. We treat this diversity term as an optional auxiliary component rather than a core part of the credit assignment rule itself\.

#### Objective\.

The diversity\-adjusted reward in Eq\. \([9](https://arxiv.org/html/2607.11070#A1.E9)\) induces the modified return

R~i,t=r~i,t\+γ​R~i,t\+1\.\\tilde\{R\}\_\{i,t\}=\\tilde\{r\}\_\{i,t\}\+\\gamma\\tilde\{R\}\_\{i,t\+1\}\.\(10\)
Unless otherwise stated, we compute turn\-level advantages using the dynamic\-weighted DC\-GRPO rule:

A~i,t:=R~i,t−μ~tRσ~tR,\\tilde\{A\}\_\{i,t\}:=\\frac\{\\tilde\{R\}\_\{i,t\}\-\\tilde\{\\mu\}^\{R\}\_\{t\}\}\{\\tilde\{\\sigma\}^\{R\}\_\{t\}\},\(11\)whereμ~tR=mean​\(\{R~j,t\}j=1G\)\\tilde\{\\mu\}^\{R\}\_\{t\}=\\mathrm\{mean\}\(\\\{\\tilde\{R\}\_\{j,t\}\\\}\_\{j=1\}^\{G\}\)andσ~tR=std​\(\{R~j,t\}j=1G\)\\tilde\{\\sigma\}^\{R\}\_\{t\}=\\mathrm\{std\}\(\\\{\\tilde\{R\}\_\{j,t\}\\\}\_\{j=1\}^\{G\}\)\. When the diversity bonus is disabled, Eq\. \([11](https://arxiv.org/html/2607.11070#A1.E11)\) reduces to the dynamic\-weighted DC\-GRPO advantage in Eq\. \([4](https://arxiv.org/html/2607.11070#S4.E4)\)\.

## Appendix BWhy Turn\-Level Decomposition?

#### Naive multi\-turn standard GRPO\.

A direct multi\-turn expansion of standard GRPO computes one group\-relative trajectory\-level advantage from the full return and assigns it to every turn:

Ai,tstd:=Ri,1−μ1Rσ1R,∀t∈\{1,…,T\}\.A^\{\\rm std\}\_\{i,t\}:=\\frac\{R\_\{i,1\}\-\\mu^\{R\}\_\{1\}\}\{\\sigma^\{R\}\_\{1\}\},\\qquad\\forall t\\in\\\{1,\\ldots,T\\\}\.\(12\)This mirrors the single\-turn use of GRPO, but in a multi\-turn trajectory the same scalar is used for the token\-level updates in all turns\.

#### Statement\.

For any fixed turntt, assume a non\-degenerate rollout group withσ1R\>0\\sigma^\{R\}\_\{1\}\>0andσtR\>0\\sigma^\{R\}\_\{t\}\>0\. Then the naive standard\-GRPO signal at turnttsatisfies

Ai,tstd\\displaystyle A^\{\\rm std\}\_\{i,t\}=∑k=1t−1γk−1​\(ri,k−μkr\)σ1R⏟prefix credit\\displaystyle=\\underbrace\{\\frac\{\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\(r\_\{i,k\}\-\\mu^\{r\}\_\{k\}\)\}\{\\sigma^\{R\}\_\{1\}\}\}\_\{\\text\{prefix credit\}\}\(13\)\+γt−1​σtRσ1R​A^i,tDW⏟scaled suffix credit\.\\displaystyle\\quad\+\\underbrace\{\\gamma^\{t\-1\}\\frac\{\\sigma^\{R\}\_\{t\}\}\{\\sigma^\{R\}\_\{1\}\}\\hat\{A\}^\{\\rm DW\}\_\{i,t\}\}\_\{\\text\{scaled suffix credit\}\}\.Here, empty sums are defined as zero\. The first term contains rewards from turns beforett, while the second term is the dynamic\-weighted DC\-GRPO advantage at turntt, up to a group\-level scaling factor\.

#### Proof\.

By the definition of the discounted return,

Ri,1=∑k=1t−1γk−1​ri,k\+γt−1​Ri,t\.R\_\{i,1\}=\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}r\_\{i,k\}\+\\gamma^\{t\-1\}R\_\{i,t\}\.\(14\)Taking the group mean over the same rollout group gives

μ1R=∑k=1t−1γk−1​μkr\+γt−1​μtR\.\\mu^\{R\}\_\{1\}=\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\\mu^\{r\}\_\{k\}\+\\gamma^\{t\-1\}\\mu^\{R\}\_\{t\}\.\(15\)Subtracting Eq\. \([15](https://arxiv.org/html/2607.11070#A2.E15)\) from Eq\. \([14](https://arxiv.org/html/2607.11070#A2.E14)\), we obtain

Ri,1−μ1R=∑k=1t−1γk−1​\(ri,k−μkr\)\+γt−1​\(Ri,t−μtR\)\.R\_\{i,1\}\-\\mu^\{R\}\_\{1\}=\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\(r\_\{i,k\}\-\\mu^\{r\}\_\{k\}\)\+\\gamma^\{t\-1\}\(R\_\{i,t\}\-\\mu^\{R\}\_\{t\}\)\.\(16\)Dividing both sides byσ1R\\sigma^\{R\}\_\{1\}and usingA^i,tDW=\(Ri,t−μtR\)/σtR\\hat\{A\}^\{\\rm DW\}\_\{i,t\}=\(R\_\{i,t\}\-\\mu^\{R\}\_\{t\}\)/\\sigma^\{R\}\_\{t\}gives Eq\. \([13](https://arxiv.org/html/2607.11070#A2.E13)\)\.□\\square

#### Lemma\.

Fort<Tt<T, assume additionally thatσtr\>0\\sigma^\{r\}\_\{t\}\>0andσt\+1R\>0\\sigma^\{R\}\_\{t\+1\}\>0\. Then the naive standard\-GRPO signal can also be written in terms of immediate and future credits as

Ai,tstd\\displaystyle A^\{\\rm std\}\_\{i,t\}=∑k=1t−1γk−1​\(ri,k−μkr\)σ1R⏟prefix credit\\displaystyle=\\underbrace\{\\frac\{\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\(r\_\{i,k\}\-\\mu^\{r\}\_\{k\}\)\}\{\\sigma^\{R\}\_\{1\}\}\}\_\{\\text\{prefix credit\}\}\(17\)\+γt−1​σtrσ1R​Ii,t\+γt​σt\+1Rσ1R​Fi,t\.\\displaystyle\\quad\+\\gamma^\{t\-1\}\\frac\{\\sigma^\{r\}\_\{t\}\}\{\\sigma^\{R\}\_\{1\}\}I\_\{i,t\}\+\\gamma^\{t\}\\frac\{\\sigma^\{R\}\_\{t\+1\}\}\{\\sigma^\{R\}\_\{1\}\}F\_\{i,t\}\.At the final turnt=Tt=T, the future\-credit term is omitted, yielding

Ai,Tstd=∑k=1T−1γk−1​\(ri,k−μkr\)σ1R\+γT−1​σTrσ1R​Ii,T\.A^\{\\rm std\}\_\{i,T\}=\\frac\{\\sum\_\{k=1\}^\{T\-1\}\\gamma^\{k\-1\}\(r\_\{i,k\}\-\\mu^\{r\}\_\{k\}\)\}\{\\sigma^\{R\}\_\{1\}\}\+\\gamma^\{T\-1\}\\frac\{\\sigma^\{r\}\_\{T\}\}\{\\sigma^\{R\}\_\{1\}\}I\_\{i,T\}\.\(18\)

#### Proof\.

Using the centered\-return recursion

Ri,t−μtR=\(ri,t−μtr\)\+γ​\(Ri,t\+1−μt\+1R\),R\_\{i,t\}\-\\mu^\{R\}\_\{t\}=\(r\_\{i,t\}\-\\mu^\{r\}\_\{t\}\)\+\\gamma\(R\_\{i,t\+1\}\-\\mu^\{R\}\_\{t\+1\}\),\(19\)and the definitionsIi,t=\(ri,t−μtr\)/σtrI\_\{i,t\}=\(r\_\{i,t\}\-\\mu^\{r\}\_\{t\}\)/\\sigma^\{r\}\_\{t\}andFi,t=\(Ri,t\+1−μt\+1R\)/σt\+1RF\_\{i,t\}=\(R\_\{i,t\+1\}\-\\mu^\{R\}\_\{t\+1\}\)/\\sigma^\{R\}\_\{t\+1\}, we have

Ri,t−μtR=σtr​Ii,t\+γ​σt\+1R​Fi,t\.R\_\{i,t\}\-\\mu^\{R\}\_\{t\}=\\sigma^\{r\}\_\{t\}I\_\{i,t\}\+\\gamma\\sigma^\{R\}\_\{t\+1\}F\_\{i,t\}\.\(20\)Substituting Eq\. \([20](https://arxiv.org/html/2607.11070#A2.E20)\) into Eq\. \([16](https://arxiv.org/html/2607.11070#A2.E16)\) and dividing byσ1R\\sigma^\{R\}\_\{1\}gives Eq\. \([17](https://arxiv.org/html/2607.11070#A2.E17)\)\. Whent=Tt=T, we haveRi,T=ri,TR\_\{i,T\}=r\_\{i,T\}andσTR=σTr\\sigma^\{R\}\_\{T\}=\\sigma^\{r\}\_\{T\}, so the future\-credit term vanishes and Eq\. \([18](https://arxiv.org/html/2607.11070#A2.E18)\) follows\.□\\square

#### Implication\.

Eq\. \([17](https://arxiv.org/html/2607.11070#A2.E17)\) shows that the naive standard\-GRPO signal does contain the immediate and future credits used by DC\-GRPO\. However, it also contains the prefix\-credit term

∑k=1t−1γk−1​\(ri,k−μkr\)σ1R\.\\frac\{\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\(r\_\{i,k\}\-\\mu^\{r\}\_\{k\}\)\}\{\\sigma^\{R\}\_\{1\}\}\.\(21\)This prefix\-credit term depends only on rewards observed before turntt\. Hence, at the level of the finite\-sample advantage used in the GRPO surrogate, the update applied at turnttcan be influenced by outcomes that occurred before the current action was sampled\.

This prefix term can also change the relative ranking of the update signals at turnttacross trajectories\. For two trajectoriesiiandjj,

Ai,tstd−Aj,tstd\\displaystyle A^\{\\rm std\}\_\{i,t\}\-A^\{\\rm std\}\_\{j,t\}=∑k=1t−1γk−1​\(ri,k−rj,k\)σ1R\\displaystyle=\\frac\{\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\(r\_\{i,k\}\-r\_\{j,k\}\)\}\{\\sigma^\{R\}\_\{1\}\}\(22\)\+γt−1​Ri,t−Rj,tσ1R\.\\displaystyle\\quad\+\\gamma^\{t\-1\}\\frac\{R\_\{i,t\}\-R\_\{j,t\}\}\{\\sigma^\{R\}\_\{1\}\}\.Thus, even when trajectoryiihas a larger remaining return from turntt,*i\.e\.,*Ri,t\>Rj,tR\_\{i,t\}\>R\_\{j,t\}, the naive standard\-GRPO signal can assign a smaller advantage to trajectoryiiat turnttwhenever

∑k=1t−1γk−1​\(rj,k−ri,k\)\>γt−1​\(Ri,t−Rj,t\)\.\\sum\_\{k=1\}^\{t\-1\}\\gamma^\{k\-1\}\(r\_\{j,k\}\-r\_\{i,k\}\)\>\\gamma^\{t\-1\}\(R\_\{i,t\}\-R\_\{j,t\}\)\.\(23\)In this case, the sample\-level ranking of update signals at turnttis determined more by earlier rewards than by the remaining return from the current turn onward\.

By contrast, dynamic\-weighted DC\-GRPO directly uses the suffix\-level signal

A^i,tDW=Ri,t−μtRσtR=σtrσtR​Ii,t\+γ​σt\+1RσtR​Fi,t,\\hat\{A\}^\{\\rm DW\}\_\{i,t\}=\\frac\{R\_\{i,t\}\-\\mu^\{R\}\_\{t\}\}\{\\sigma^\{R\}\_\{t\}\}=\\frac\{\\sigma^\{r\}\_\{t\}\}\{\\sigma^\{R\}\_\{t\}\}I\_\{i,t\}\+\\gamma\\frac\{\\sigma^\{R\}\_\{t\+1\}\}\{\\sigma^\{R\}\_\{t\}\}F\_\{i,t\},\(24\)which removes the prefix\-credit term and normalizes the remaining return at the current turn\. Static\-weighted DC\-GRPO uses the same immediate–future view,A^i,tSW=Ii,t\+α​Fi,t\\hat\{A\}^\{\\rm SW\}\_\{i,t\}=I\_\{i,t\}\+\\alpha F\_\{i,t\}, but exposes the future\-credit weight as a tunable coefficient\. Therefore, the purpose of turn\-level decomposition is not merely to rewrite the trajectory return algebraically, but to isolate the part of the return that is relevant to the policy update at the current turn\.

#### Sensitivity to future credit\.

The static\-weighted family provides a direct sensitivity analysis of the future\-credit term:

A^i,tSW​\(α\)=Ii,t\+α​Fi,t\.\\hat\{A\}^\{\\rm SW\}\_\{i,t\}\(\\alpha\)=I\_\{i,t\}\+\\alpha F\_\{i,t\}\.\(25\)Here,α=0\\alpha=0removes future credit and uses only immediate credit, whereasα\>0\\alpha\>0assigns credit to turns that improve later outcomes\. Therefore, the ablation overα\\alphatests whether multi\-turn performance comes merely from turn\-local reward shaping or from non\-myopic credit assignment across turns\.

As shown in Table[10](https://arxiv.org/html/2607.11070#A5.T10), the immediate\-only setting withα=0\\alpha=0is clearly weaker\. Increasingα\\alphato0\.50\.5substantially improves both ID and OOD averages, andα=1\\alpha=1yields a similar overall performance profile with slightly stronger ID performance\. Thus, the main takeaway is not the superiority of a single fixedα\\alpha, but the importance of including future credit at all\. These results support future\-aware turn\-level credit assignment and motivate dynamic\-weighted DC\-GRPO as a simple default: it preserves suffix\-level future\-aware credit while avoiding an additional mixing hyperparameter\.

## Appendix CFirst\-Turn Local Sensitivity

Within the unified DC\-GRPO framework, the static\-weighted and dynamic\-weighted forms differ in how strongly the first\-turn advantage responds to a local perturbation of the immediate rewardri,1r\_\{i,1\}\. In this appendix, we compare these two turn\-level credit rules through their first\-turn local sensitivity\.

For a fixed group ofGGtrajectories, define

Ri,t=∑k=tTγk−t​ri,k,μtr=1G​∑j=1Grj,t,μtR=1G​∑j=1GRj,t,R\_\{i,t\}=\\sum\_\{k=t\}^\{T\}\\gamma^\{k\-t\}r\_\{i,k\},\\qquad\\mu\_\{t\}^\{r\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}r\_\{j,t\},\\qquad\\mu\_\{t\}^\{R\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}R\_\{j,t\},and

σtr=1G​∑j=1G\(rj,t−μtr\)2,σtR=1G​∑j=1G\(Rj,t−μtR\)2\.\\sigma\_\{t\}^\{r\}=\\sqrt\{\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\(r\_\{j,t\}\-\\mu\_\{t\}^\{r\}\)^\{2\}\},\\qquad\\sigma\_\{t\}^\{R\}=\\sqrt\{\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\(R\_\{j,t\}\-\\mu\_\{t\}^\{R\}\)^\{2\}\}\.
For a generic collectionx1,…,xGx\_\{1\},\\dots,x\_\{G\}, let

μ=1G​∑j=1Gxj,σ=1G​∑j=1G\(xj−μ\)2,zi=xi−μσ\.\\mu=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}x\_\{j\},\\qquad\\sigma=\\sqrt\{\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\(x\_\{j\}\-\\mu\)^\{2\}\},\\qquad z\_\{i\}=\\frac\{x\_\{i\}\-\\mu\}\{\\sigma\}\.Then

∂μ∂xi=1G\.\\frac\{\\partial\\mu\}\{\\partial x\_\{i\}\}=\\frac\{1\}\{G\}\.Writingv=σ2v=\\sigma^\{2\}, we have

v=1G​∑j=1G\(xj−μ\)2=1G​∑j=1Gxj2−μ2,v=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}\(x\_\{j\}\-\\mu\)^\{2\}=\\frac\{1\}\{G\}\\sum\_\{j=1\}^\{G\}x\_\{j\}^\{2\}\-\\mu^\{2\},so

∂v∂xi=2​xiG−2​μ​∂μ∂xi=2​xiG−2​μG=2​\(xi−μ\)G\.\\frac\{\\partial v\}\{\\partial x\_\{i\}\}=\\frac\{2x\_\{i\}\}\{G\}\-2\\mu\\frac\{\\partial\\mu\}\{\\partial x\_\{i\}\}=\\frac\{2x\_\{i\}\}\{G\}\-\\frac\{2\\mu\}\{G\}=\\frac\{2\(x\_\{i\}\-\\mu\)\}\{G\}\.Hence

∂σ∂xi=12​σ​∂v∂xi=xi−μG​σ\.\\frac\{\\partial\\sigma\}\{\\partial x\_\{i\}\}=\\frac\{1\}\{2\\sigma\}\\frac\{\\partial v\}\{\\partial x\_\{i\}\}=\\frac\{x\_\{i\}\-\\mu\}\{G\\sigma\}\.Therefore,

∂zi∂xi\\displaystyle\\frac\{\\partial z\_\{i\}\}\{\\partial x\_\{i\}\}=∂∂xi​\(xi−μσ\)\\displaystyle=\\frac\{\\partial\}\{\\partial x\_\{i\}\}\\left\(\\frac\{x\_\{i\}\-\\mu\}\{\\sigma\}\\right\)=1−1Gσ−xi−μσ2​∂σ∂xi\\displaystyle=\\frac\{1\-\\frac\{1\}\{G\}\}\{\\sigma\}\-\\frac\{x\_\{i\}\-\\mu\}\{\\sigma^\{2\}\}\\frac\{\\partial\\sigma\}\{\\partial x\_\{i\}\}=1−1Gσ−xi−μσ2⋅xi−μG​σ\\displaystyle=\\frac\{1\-\\frac\{1\}\{G\}\}\{\\sigma\}\-\\frac\{x\_\{i\}\-\\mu\}\{\\sigma^\{2\}\}\\cdot\\frac\{x\_\{i\}\-\\mu\}\{G\\sigma\}=1−1Gσ−\(xi−μ\)2G​σ3\.\\displaystyle=\\frac\{1\-\\frac\{1\}\{G\}\}\{\\sigma\}\-\\frac\{\(x\_\{i\}\-\\mu\)^\{2\}\}\{G\\sigma^\{3\}\}\.Usingzi=\(xi−μ\)/σz\_\{i\}=\(x\_\{i\}\-\\mu\)/\\sigma, this becomes

∂zi∂xi=1−1Gσ−zi2G​σ=G−1−zi2G​σ\.\\frac\{\\partial z\_\{i\}\}\{\\partial x\_\{i\}\}=\\frac\{1\-\\frac\{1\}\{G\}\}\{\\sigma\}\-\\frac\{z\_\{i\}^\{2\}\}\{G\\sigma\}=\\frac\{G\-1\-z\_\{i\}^\{2\}\}\{G\\sigma\}\.Also, by population normalization,

1G​∑i=1Gzi2=1\.\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}z\_\{i\}^\{2\}=1\.
We now compare the two DC\-GRPO instantiations att=1t=1\.

For the static\-weighted form,

Ii,1=ri,1−μ1rσ1r,Fi,1=Ri,2−μ2Rσ2R,Ai,1SW=Ii,1\+α​Fi,1\.I\_\{i,1\}=\\frac\{r\_\{i,1\}\-\\mu\_\{1\}^\{r\}\}\{\\sigma\_\{1\}^\{r\}\},\\qquad F\_\{i,1\}=\\frac\{R\_\{i,2\}\-\\mu\_\{2\}^\{R\}\}\{\\sigma\_\{2\}^\{R\}\},\\qquad A\_\{i,1\}^\{\\mathrm\{SW\}\}=I\_\{i,1\}\+\\alpha F\_\{i,1\}\.SinceRi,2R\_\{i,2\}is unaffected by a local perturbation ofri,1r\_\{i,1\}, we have

∂Ri,2∂ri,1=0,\\frac\{\\partial R\_\{i,2\}\}\{\\partial r\_\{i,1\}\}=0,and therefore

∂Ai,1SW∂ri,1=∂Ii,1∂ri,1=G−1−Ii,12G​σ1r\.\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{SW\}\}\}\{\\partial r\_\{i,1\}\}=\\frac\{\\partial I\_\{i,1\}\}\{\\partial r\_\{i,1\}\}=\\frac\{G\-1\-I\_\{i,1\}^\{2\}\}\{G\\sigma\_\{1\}^\{r\}\}\.Averaging overi=1,…,Gi=1,\\dots,Ggives

𝔼​\[∂Ai,1SW∂ri,1\]=1G​∑i=1GG−1−Ii,12G​σ1r=1G​σ1r​\(G−1−1G​∑i=1GIi,12\)=G−2G​σ1r\.\\mathbb\{E\}\\left\[\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{SW\}\}\}\{\\partial r\_\{i,1\}\}\\right\]=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{G\-1\-I\_\{i,1\}^\{2\}\}\{G\\sigma\_\{1\}^\{r\}\}=\\frac\{1\}\{G\\sigma\_\{1\}^\{r\}\}\\left\(G\-1\-\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}I\_\{i,1\}^\{2\}\\right\)=\\frac\{G\-2\}\{G\\sigma\_\{1\}^\{r\}\}\.
For the dynamic\-weighted form,

Ai,1DW=Ri,1−μ1Rσ1R\.A\_\{i,1\}^\{\\mathrm\{DW\}\}=\\frac\{R\_\{i,1\}\-\\mu\_\{1\}^\{R\}\}\{\\sigma\_\{1\}^\{R\}\}\.Since

Ri,1=ri,1\+γ​Ri,2,R\_\{i,1\}=r\_\{i,1\}\+\\gamma R\_\{i,2\},we have

∂Ri,1∂ri,1=1,\\frac\{\\partial R\_\{i,1\}\}\{\\partial r\_\{i,1\}\}=1,and thus

∂Ai,1DW∂ri,1=G−1−\(Ai,1DW\)2G​σ1R\.\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{DW\}\}\}\{\\partial r\_\{i,1\}\}=\\frac\{G\-1\-\(A\_\{i,1\}^\{\\mathrm\{DW\}\}\)^\{2\}\}\{G\\sigma\_\{1\}^\{R\}\}\.Again averaging overi=1,…,Gi=1,\\dots,G,

𝔼​\[∂Ai,1DW∂ri,1\]=1G​∑i=1GG−1−\(Ai,1DW\)2G​σ1R=1G​σ1R​\(G−1−1G​∑i=1G\(Ai,1DW\)2\)=G−2G​σ1R\.\\mathbb\{E\}\\left\[\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{DW\}\}\}\{\\partial r\_\{i,1\}\}\\right\]=\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{G\-1\-\(A\_\{i,1\}^\{\\mathrm\{DW\}\}\)^\{2\}\}\{G\\sigma\_\{1\}^\{R\}\}=\\frac\{1\}\{G\\sigma\_\{1\}^\{R\}\}\\left\(G\-1\-\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\(A\_\{i,1\}^\{\\mathrm\{DW\}\}\)^\{2\}\\right\)=\\frac\{G\-2\}\{G\\sigma\_\{1\}^\{R\}\}\.
Therefore,

𝔼​\[∂Ai,1SW∂ri,1\]=G−2G​σ1r,𝔼​\[∂Ai,1DW∂ri,1\]=G−2G​σ1R\.\\mathbb\{E\}\\left\[\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{SW\}\}\}\{\\partial r\_\{i,1\}\}\\right\]=\\frac\{G\-2\}\{G\\sigma\_\{1\}^\{r\}\},\\qquad\\mathbb\{E\}\\left\[\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{DW\}\}\}\{\\partial r\_\{i,1\}\}\\right\]=\\frac\{G\-2\}\{G\\sigma\_\{1\}^\{R\}\}\.The comparison is therefore reduced toσ1r\\sigma\_\{1\}^\{r\}andσ1R\\sigma\_\{1\}^\{R\}\.

Using

Ri,1=ri,1\+γ​Ri,2,R\_\{i,1\}=r\_\{i,1\}\+\\gamma R\_\{i,2\},we obtain

\(σ1R\)2=\(σ1r\)2\+γ2​Vari⁡\(Ri,2\)\+2​γ​Covi⁡\(ri,1,Ri,2\)\.\(\\sigma\_\{1\}^\{R\}\)^\{2\}=\(\\sigma\_\{1\}^\{r\}\)^\{2\}\+\\gamma^\{2\}\\operatorname\{Var\}\_\{i\}\(R\_\{i,2\}\)\+2\\gamma\\,\\operatorname\{Cov\}\_\{i\}\(r\_\{i,1\},R\_\{i,2\}\)\.Here, the covariance term describes a cross\-trajectory association within the rollout group, rather than a local functional dependence ofRi,2R\_\{i,2\}onri,1r\_\{i,1\}\. Thus, although∂Ri,2/∂ri,1=0\\partial R\_\{i,2\}/\\partial r\_\{i,1\}=0, the covarianceCovi⁡\(ri,1,Ri,2\)\\operatorname\{Cov\}\_\{i\}\(r\_\{i,1\},R\_\{i,2\}\)may still be nonzero across trajectories\.

Hence, a sufficient condition for

σ1R≥σ1r\\sigma\_\{1\}^\{R\}\\geq\\sigma\_\{1\}^\{r\}is

γ2​Vari⁡\(Ri,2\)\+2​γ​Covi⁡\(ri,1,Ri,2\)≥0\.\\gamma^\{2\}\\operatorname\{Var\}\_\{i\}\(R\_\{i,2\}\)\+2\\gamma\\,\\operatorname\{Cov\}\_\{i\}\(r\_\{i,1\},R\_\{i,2\}\)\\geq 0\.Equivalently, forγ\>0\\gamma\>0, it is enough to assume

Covi⁡\(ri,1,Ri,2\)≥−γ2​Vari⁡\(Ri,2\)\.\\operatorname\{Cov\}\_\{i\}\(r\_\{i,1\},R\_\{i,2\}\)\\geq\-\\frac\{\\gamma\}\{2\}\\operatorname\{Var\}\_\{i\}\(R\_\{i,2\}\)\.As a simple special case, this condition is implied by

Covi⁡\(ri,1,Ri,2\)≥0\.\\operatorname\{Cov\}\_\{i\}\(r\_\{i,1\},R\_\{i,2\}\)\\geq 0\.Under this condition,

σ1R≥σ1r,\\sigma\_\{1\}^\{R\}\\geq\\sigma\_\{1\}^\{r\},and consequently

𝔼​\[∂Ai,1SW∂ri,1\]≥𝔼​\[∂Ai,1DW∂ri,1\]\.\\mathbb\{E\}\\left\[\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{SW\}\}\}\{\\partial r\_\{i,1\}\}\\right\]\\geq\\mathbb\{E\}\\left\[\\frac\{\\partial A\_\{i,1\}^\{\\mathrm\{DW\}\}\}\{\\partial r\_\{i,1\}\}\\right\]\.
This shows that, within the same DC\-GRPO framework, the static\-weighted form is more locally sensitive to the first\-turn immediate reward, whereas the dynamic\-weighted form is less sensitive because it normalizes the full remaining return\.

## Appendix DTraining Runtime

We report the wall\-clock training times of dynamic\-weighted DC\-GRPO \(MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\) and static\-weighted DC\-GRPO \(MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\) in Tables[8](https://arxiv.org/html/2607.11070#A4.T8)and[9](https://arxiv.org/html/2607.11070#A4.T9)\. Both methods are trained for 260 steps, corresponding to a two\-epoch schedule, with validation and checkpoint saving performed at steps 65, 130, 195, and 260\.

Table 8:Cumulative training\-loop time at each evaluation checkpoint and mean wall\-clock time over all 260 training steps\.MethodStep 65Step 130Step 195Step 260Mean time/stepMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}2h 37m 21s5h 24m 17s8h 07m 27s10h 47m 04s149\.32sMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}2h 35m 59s5h 24m 30s8h 11m 06s10h 49m 33s149\.90sTable 9:Average runtime of each training component per step\. Validation and checkpoint operations are not included in the component averages\.MethodGenerationActor updateOld log prob\.Rollout dumpRewardAdvantageMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}88\.44s43\.07s13\.30s0\.67s0\.15s0\.003sMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}89\.00s43\.73s13\.45s0\.30s0\.15s0\.004sOverall, the two variants exhibit nearly identical training costs and runtime profiles, with most of the training time spent on rollout generation and actor optimization\.

Overall, the two variants exhibit nearly identical training costs\. Their normal\-step times differ by less than 1%, and both spend most of their runtime on rollout generation and actor optimization\.

## Appendix EAdditional Ablation Studies

This appendix reports supplementary ablations on credit assignment granularity and weighting\. We first compare the static\- and dynamic\-weighted instantiations of DC\-GRPO and then contrast their turn\-level credit with a trajectory\-level baseline derived from the last\-turn reward\.

Table 10:Ablation study ofMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}andMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}with differentα\\alphavalues for the Qwen3 attacker, including GPT\-OSS\-20B\. Each row shows theASR5​@​3\\mathrm\{ASR\}\_\{5\}\{@3\}\(%\) evaluated on victim LLMs\. The best results in each column are marked inbold\.MethodLlama\-3\.1\-8B\-InstructGPT\-OSS\-20BQwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODSEMA\[[10](https://arxiv.org/html/2607.11070#bib.bib418)\]88\.5091\.67100\.0047\.500\.350\.0087\.5093\.4096\.3664\.0071\.8885\.4578\.0091\.3290\.9193\.3967\.22MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}99\.5095\.8398\.1811\.0011\.4620\.00100\.00100\.0098\.1894\.5095\.4998\.18100\.0099\.31100\.0097\.8477\.34MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(α=0\\alpha=0\)91\.5087\.5092\.7327\.0028\.4730\.9196\.0096\.8898\.1872\.0063\.8981\.8292\.5096\.8892\.7390\.5873\.11MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(α=0\.5\\alpha=0\.5\)96\.5093\.0698\.1830\.0024\.6532\.7399\.0099\.6598\.1891\.5088\.5494\.5599\.0098\.61100\.0095\.9179\.70MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(α=1\\alpha=1\)97\.5097\.9296\.3616\.5015\.2832\.7399\.50100\.00100\.0094\.0093\.7598\.1899\.5099\.6598\.1897\.2678\.94

#### Static and dynamic weighting\.

As a reference point, Table[10](https://arxiv.org/html/2607.11070#A5.T10)comparesMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}withMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variants using different values ofα\\alpha\. Dynamic\-weighted DC\-GRPO normalizes the discounted returnRi,t=∑k=tTγk−t​ri,kR\_\{i,t\}=\\sum\_\{k=t\}^\{T\}\\gamma^\{k\-t\}r\_\{i,k\}at each turn and assigns the resulting z\-score as the advantage:

A^i,tDW=Ri,t−μtRσtR,∀t∈\{1,…,T\}\.\\hat\{A\}^\{\\mathrm\{DW\}\}\_\{i,t\}=\\frac\{R\_\{i,t\}\-\\mu^\{R\}\_\{t\}\}\{\\sigma^\{R\}\_\{t\}\},\\quad\\forall\\,t\\in\\\{1,\\ldots,T\\\}\.\(26\)This formulation uses per\-turn returns without introducing an additional mixing hyperparameter, whereasMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}provides explicit control over future credit throughα\\alpha\. Both variants retain the same turn\-level group\-relative credit structure\.

Compared with theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variants in Table[10](https://arxiv.org/html/2607.11070#A5.T10),MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}achieves the highest ID average of 97\.84\. On OOD evaluation, the bestMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variant isα=0\.5\\alpha=0\.5, which reaches 79\.70, whileα=1\\alpha=1reaches 78\.94 andMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}obtains 77\.34\. Taken together, these results indicate that both weighting rules provide strong turn\-level credit, while their relative performance varies across ID and OOD evaluation\. The main distinction is therefore how immediate and future credit are weighted: DW determines the coefficients from rollout statistics, whereas SW controls future credit explicitly throughα\\alpha\. For this reason, we useMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}as a simple default and viewMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}as a controllable alternative when one wants explicit tuning of future credit\.

#### Effect of the future credit coefficientα\\alpha\.

Within static\-weighted DC\-GRPO, the coefficientα\\alphacontrols the relative weight of future creditFi,tF\_\{i,t\}after independent normalization\. Settingα=0\\alpha=0uses only the immediate termIi,tI\_\{i,t\}, providing sharp but myopic credit that ignores whether the current action creates favorable context for later turns\. Settingα=1\\alpha=1gives immediate and future credit equal fixed weights, allowing the policy to assign credit to preparatory actions that facilitate later success\.

Table[10](https://arxiv.org/html/2607.11070#A5.T10)shows that using only immediate credit is insufficient:α=0\\alpha=0yields the lowest ID and OOD averages of 90\.58 and 73\.11, respectively\. Increasingα\\alphato 0\.5 substantially improves the averages to 95\.91 and 79\.70, while settingα=1\\alpha=1further improves the ID average to 97\.26 with a comparable OOD average of 78\.94\. These results show that incorporating future credit is more consequential than selecting one particular weighting rule: the immediate\-only setting is consistently weaker, whereas both SW settings withα\>0\\alpha\>0and DW retain future\-aware turn\-level credit\.

Comparingα=0\.5\\alpha=0\.5andα=1\\alpha=1, we observe only a small ID–OOD trade\-off\. Theα=1\\alpha=1setting performs slightly better on ID evaluation, whereasα=0\.5\\alpha=0\.5is slightly better on OOD evaluation\. Because the gap is modest, the main takeaway is not the superiority of a singleα\\alphavalue, but rather the importance of including non\-myopic future credit at all\.

#### Turn\-level versus last\-turn trajectory credit\.

To isolate the effect of credit granularity, we compare DC\-GRPO with a trajectory\-level baseline that uses only the last\-turn reward\. For trajectoryii, the baseline computes

A^iLT≔ri,T−μTrσTr,A^i,tLT≔A^iLT∀t∈\{1,…,T\}\.\\hat\{A\}^\{\\mathrm\{LT\}\}\_\{i\}\\coloneqq\\frac\{r\_\{i,T\}\-\\mu\_\{T\}^\{r\}\}\{\\sigma\_\{T\}^\{r\}\},\\qquad\\hat\{A\}^\{\\mathrm\{LT\}\}\_\{i,t\}\\coloneqq\\hat\{A\}^\{\\mathrm\{LT\}\}\_\{i\}\\quad\\forall t\\in\\\{1,\\ldots,T\\\}\.\(27\)Thus, every turn in a trajectory receives the same group\-normalized credit determined by the final response\. This baseline differs from the immediate\-only SW setting withα=0\\alpha=0: the latter assigns each turn its own immediate creditIi,tI\_\{i,t\}, whereas last\-turn trajectory credit broadcasts one final\-turn signal to the entire trajectory\.

Table 11:Ablation study comparing turn\-level DC\-GRPO with last\-turn trajectory credit\. Each row showsASR5​@​3\\mathrm\{ASR\}\_\{5\}@3\(%\) for a Qwen3 attacker trained againstLlama\-3\.1\-8B\-Instruct\. The OOD average aggregates Qwen2\.5\-7B\-Instruct, Gemma\-2\-9B\-IT, and Mistral\-7B\-Instruct\-v0\.3\. The best result in each column is marked inbold\.MethodLlama\-3\.1\-8B\-Instruct \(ID\)Qwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}96\.5095\.8398\.1897\.00100\.00100\.0093\.5091\.6796\.3698\.0096\.8896\.3696\.8496\.64MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(α=1\\alpha=1\)98\.5097\.22100\.0099\.0099\.3196\.3694\.0087\.5098\.1898\.5098\.9698\.1898\.5796\.67Last\-turn trajectory credit97\.0096\.5398\.1897\.0099\.6596\.3687\.5087\.5094\.5599\.5098\.9698\.1897\.2495\.47

Table 12:Ablation study comparing turn\-level DC\-GRPO with last\-turn trajectory credit\. Each row showsASR5​@​1\\mathrm\{ASR\}\_\{5\}@1\(%\) for a Qwen3 attacker trained againstLlama\-3\.1\-8B\-Instruct\. Results are reported as mean±\\pmstandard deviation over five independent evaluation runs\. The OOD average aggregates Qwen2\.5\-7B\-Instruct, Gemma\-2\-9B\-IT, and Mistral\-7B\-Instruct\-v0\.3\. The best mean in each column is marked inbold\.MethodLlama\-3\.1\-8B\-Instruct \(ID\)Qwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}90\.00±1\.7690\.00\{\\scriptstyle\\pm 1\.76\}82\.92±2\.4082\.92\{\\scriptstyle\\pm 2\.40\}90\.55±2\.12\\textbf\{90\.55\}\{\\scriptstyle\\pm 2\.12\}92\.10±1\.1692\.10\{\\scriptstyle\\pm 1\.16\}94\.24±1\.3894\.24\{\\scriptstyle\\pm 1\.38\}92\.36±2\.12\\textbf\{92\.36\}\{\\scriptstyle\\pm 2\.12\}82\.50±1\.52\\textbf\{82\.50\}\{\\scriptstyle\\pm 1\.52\}76\.18±1\.79\\textbf\{76\.18\}\{\\scriptstyle\\pm 1\.79\}86\.55±3\.92\\textbf\{86\.55\}\{\\scriptstyle\\pm 3\.92\}92\.20±0\.8192\.20\{\\scriptstyle\\pm 0\.81\}90\.56±0\.4690\.56\{\\scriptstyle\\pm 0\.46\}91\.27±2\.9191\.27\{\\scriptstyle\\pm 2\.91\}87\.82±3\.47\\textbf\{87\.82\}\{\\scriptstyle\\pm 3\.47\}88\.66±5\.56\\textbf\{88\.66\}\{\\scriptstyle\\pm 5\.56\}MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(α=1\\alpha=1\)90\.80±1\.69\\textbf\{90\.80\}\{\\scriptstyle\\pm 1\.69\}84\.51±1\.11\\textbf\{84\.51\}\{\\scriptstyle\\pm 1\.11\}86\.91±2\.4186\.91\{\\scriptstyle\\pm 2\.41\}94\.50±1\.73\\textbf\{94\.50\}\{\\scriptstyle\\pm 1\.73\}95\.28±0\.84\\textbf\{95\.28\}\{\\scriptstyle\\pm 0\.84\}92\.00±1\.8592\.00\{\\scriptstyle\\pm 1\.85\}77\.60±1\.8877\.60\{\\scriptstyle\\pm 1\.88\}68\.26±1\.8268\.26\{\\scriptstyle\\pm 1\.82\}80\.00±3\.0480\.00\{\\scriptstyle\\pm 3\.04\}93\.50±1\.45\\textbf\{93\.50\}\{\\scriptstyle\\pm 1\.45\}93\.33±1\.11\\textbf\{93\.33\}\{\\scriptstyle\\pm 1\.11\}93\.09±1\.3693\.09\{\\scriptstyle\\pm 1\.36\}87\.41±2\.5987\.41\{\\scriptstyle\\pm 2\.59\}87\.51±9\.1687\.51\{\\scriptstyle\\pm 9\.16\}Last\-turn trajectory credit85\.70±2\.3885\.70\{\\scriptstyle\\pm 2\.38\}81\.39±2\.0281\.39\{\\scriptstyle\\pm 2\.02\}85\.09±3\.5385\.09\{\\scriptstyle\\pm 3\.53\}91\.60±1\.0791\.60\{\\scriptstyle\\pm 1\.07\}94\.38±1\.3194\.38\{\\scriptstyle\\pm 1\.31\}90\.91±1\.1590\.91\{\\scriptstyle\\pm 1\.15\}67\.10±1\.0767\.10\{\\scriptstyle\\pm 1\.07\}64\.10±1\.6264\.10\{\\scriptstyle\\pm 1\.62\}71\.64±1\.8571\.64\{\\scriptstyle\\pm 1\.85\}92\.40±1\.5092\.40\{\\scriptstyle\\pm 1\.50\}92\.57±1\.0092\.57\{\\scriptstyle\\pm 1\.00\}93\.45±2\.72\\textbf\{93\.45\}\{\\scriptstyle\\pm 2\.72\}84\.06±1\.9184\.06\{\\scriptstyle\\pm 1\.91\}84\.24±11\.9384\.24\{\\scriptstyle\\pm 11\.93\}

Tables[11](https://arxiv.org/html/2607.11070#A5.T11)and[12](https://arxiv.org/html/2607.11070#A5.T12)compare the two turn\-level DC\-GRPO variants with last\-turn trajectory credit under otherwise matched conditions\. UnderASR5​@​3\\mathrm\{ASR\}\_\{5\}@3, all three methods achieve strong performance\. SW obtains the highest ID and OOD averages of 98\.57 and 96\.67, respectively, while last\-turn trajectory credit reaches 97\.24 and 95\.47\. The relatively small gap suggests that sampling up to three trajectories can partially compensate for coarse trajectory\-level credit\.

The distinction is clearer underASR5​@​1\\mathrm\{ASR\}\_\{5\}@1\. Last\-turn trajectory credit obtains ID and OOD averages of 84\.06 and 84\.24, compared with 87\.41 and 87\.51 for SW and 87\.82 and 88\.66 for DW\. The best turn\-level variant therefore improves over last\-turn trajectory credit by 3\.76 points on ID and 4\.42 points on OOD evaluation\. These results indicate that assigning separate credit to individual turns improves single\-trajectory reliability\. The comparable aggregate performance of SW and DW further suggests that this benefit arises primarily from their shared turn\-level credit assignment rather than from one particular weighting rule\.

## Appendix FTransferability Analyses

This appendix reports supplementary transferability results for our attacker variants and alternative training settings that are not included in the main paper due to space constraints\.

Table 13:Transferability of ourMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}andMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variants across victim LLMs\. Each row shows theASR5​@​3\\mathrm\{ASR\}\_\{5\}@3\(%\) when the attacker is trained against Llama\-3\.1\-8B\-Instruct and evaluated on all victim LLMs, with all results evaluated using the HarmBench classifier\. The best results in each column are marked inbold\.MethodLlama\-3\.1\-8B\-InstructQwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(Qwen2\.5\)89\.5088\.1992\.7394\.0096\.8896\.3671\.5077\.4383\.6490\.5094\.1094\.5590\.1488\.77MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(Qwen3\)99\.5095\.8398\.18100\.00100\.0098\.1894\.5095\.4998\.18100\.0099\.31100\.0097\.8498\.41MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen2\.5\)92\.5092\.7196\.3699\.5098\.26100\.0075\.5073\.9690\.9197\.5099\.3198\.1893\.8692\.57MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen2\.5\) \+ Div92\.0092\.3694\.5597\.0094\.7992\.7380\.0081\.2590\.9194\.5095\.1494\.5592\.9791\.21MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen3\)97\.5097\.9296\.3699\.50100\.00100\.0094\.0093\.7598\.1899\.5099\.6598\.1897\.2698\.08MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen3\) \+ Div96\.5096\.8894\.5597\.0099\.6598\.1893\.0088\.5492\.7399\.0099\.6598\.1895\.9896\.21MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen3\) \+ SFT98\.5097\.22100\.0099\.5099\.31100\.0094\.0093\.7598\.1899\.0098\.26100\.0098\.5798\.00MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen3\) \+ SFT \+ Div97\.0096\.5398\.1898\.0098\.9698\.1893\.0092\.7194\.5597\.5098\.61100\.0097\.2496\.83

#### Transferability across victim models\.

Table[13](https://arxiv.org/html/2607.11070#A6.T13)presents transferability results across in\-domain \(ID\) and out\-of\-domain \(OOD\) victim models, where the attacker is trained against Llama\-3\.1\-8B\-Instruct and evaluated on four victim LLMs\. We evaluate bothMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}andMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variants across three benchmarks: HarmBench, StrongREJECT†, and JailbreakBench†\. Overall, all reported variants achieve strong OOD performance, with averages that remain close to the corresponding ID results, indicating robust transferability to unseen targets\.

Among the reported configurations, Qwen3\-based attackers generally outperform their Qwen2\.5 counterparts across most victim–benchmark combinations\. For example,MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}with a Qwen3 attacker achieves ID/OOD averages of 97\.84/98\.41, compared with 90\.14/88\.77 for the Qwen2\.5 counterpart\. Likewise, the baseMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}Qwen3 attacker reaches 97\.26/98\.08, compared with 93\.86/92\.57 for the Qwen2\.5 version\. These results suggest that a stronger attacker backbone improves not only in\-domain performance but also transferability to unseen victim models\.

#### Training against GPT\-OSS\-20B\.

We further evaluate the two DC\-GRPO weighting rules when the attacker is trained against GPT\-OSS\-20B, a more resistant victim than Llama\-3\.1\-8B\-Instruct\. Tables[5](https://arxiv.org/html/2607.11070#S5.T5)and[14](https://arxiv.org/html/2607.11070#A6.T14)report the corresponding best\-of\-three and single\-trajectory results\. Both the 260\- and 1040\-schedule settings stop training at Step 260; they differ only in the cosine learning\-rate scheduler horizon\.

Table 14:ASR5​@​1\\mathrm\{ASR\}\_\{5\}@1\(%\) ofMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}andMJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}with a Qwen3 attacker trained againstGPT\-OSS\-20B\. Results are reported as mean±\\pmstandard deviation over five independent evaluation runs\. GPT\-OSS\-20B is the ID training victim, while the remaining models are OOD evaluation targets\. Both settings stop training at Step 260; 260 and 1040 denote the cosine learning\-rate scheduler horizon\. The best mean in each column is marked inbold\.MethodGPT\-OSS\-20B \(ID\)Llama\-3\.1\-8B\-InstructQwen2\.5\-7B\-InstructGemma\-2\-9B\-ITMistral\-7B\-Instruct\-v0\.3AverageHBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†HBSR†JBB†IDOODMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(260 sched\.\)68\.20±1\.0868\.20\{\\scriptstyle\\pm 1\.08\}66\.11±1\.4066\.11\{\\scriptstyle\\pm 1\.40\}65\.45±3\.0465\.45\{\\scriptstyle\\pm 3\.04\}86\.40±2\.0686\.40\{\\scriptstyle\\pm 2\.06\}85\.62±0\.9285\.62\{\\scriptstyle\\pm 0\.92\}86\.55±3\.3786\.55\{\\scriptstyle\\pm 3\.37\}86\.30±1\.3386\.30\{\\scriptstyle\\pm 1\.33\}84\.72±1\.9384\.72\{\\scriptstyle\\pm 1\.93\}85\.82±2\.6785\.82\{\\scriptstyle\\pm 2\.67\}85\.80±1\.4785\.80\{\\scriptstyle\\pm 1\.47\}85\.21±1\.8785\.21\{\\scriptstyle\\pm 1\.87\}85\.82±3\.1385\.82\{\\scriptstyle\\pm 3\.13\}89\.40±1\.28\\textbf\{89\.40\}\{\\scriptstyle\\pm 1\.28\}90\.76±1\.58\\textbf\{90\.76\}\{\\scriptstyle\\pm 1\.58\}90\.55±2\.1290\.55\{\\scriptstyle\\pm 2\.12\}66\.59±1\.1766\.59\{\\scriptstyle\\pm 1\.17\}86\.91±2\.0086\.91\{\\scriptstyle\\pm 2\.00\}MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(260 sched\.\)73\.60±2\.0873\.60\{\\scriptstyle\\pm 2\.08\}72\.92±1\.2472\.92\{\\scriptstyle\\pm 1\.24\}73\.45±7\.8573\.45\{\\scriptstyle\\pm 7\.85\}84\.00±2\.7084\.00\{\\scriptstyle\\pm 2\.70\}87\.15±0\.8587\.15\{\\scriptstyle\\pm 0\.85\}88\.00±2\.9588\.00\{\\scriptstyle\\pm 2\.95\}83\.20±2\.3483\.20\{\\scriptstyle\\pm 2\.34\}83\.54±3\.1983\.54\{\\scriptstyle\\pm 3\.19\}81\.45±3\.7181\.45\{\\scriptstyle\\pm 3\.71\}82\.80±1\.7582\.80\{\\scriptstyle\\pm 1\.75\}86\.11±1\.1886\.11\{\\scriptstyle\\pm 1\.18\}85\.09±5\.0685\.09\{\\scriptstyle\\pm 5\.06\}82\.40±1\.2482\.40\{\\scriptstyle\\pm 1\.24\}87\.92±1\.1987\.92\{\\scriptstyle\\pm 1\.19\}85\.45±3\.6485\.45\{\\scriptstyle\\pm 3\.64\}73\.32±0\.2973\.32\{\\scriptstyle\\pm 0\.29\}84\.76±2\.1184\.76\{\\scriptstyle\\pm 2\.11\}MJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(1040 sched\.\)84\.70±2\.11\\textbf\{84\.70\}\{\\scriptstyle\\pm 2\.11\}85\.00±1\.69\\textbf\{85\.00\}\{\\scriptstyle\\pm 1\.69\}86\.55±3\.56\\textbf\{86\.55\}\{\\scriptstyle\\pm 3\.56\}91\.20±1\.33\\textbf\{91\.20\}\{\\scriptstyle\\pm 1\.33\}93\.82±1\.34\\textbf\{93\.82\}\{\\scriptstyle\\pm 1\.34\}95\.64±2\.95\\textbf\{95\.64\}\{\\scriptstyle\\pm 2\.95\}89\.80±1\.17\\textbf\{89\.80\}\{\\scriptstyle\\pm 1\.17\}91\.81±0\.89\\textbf\{91\.81\}\{\\scriptstyle\\pm 0\.89\}94\.55±2\.30\\textbf\{94\.55\}\{\\scriptstyle\\pm 2\.30\}88\.80±1\.29\\textbf\{88\.80\}\{\\scriptstyle\\pm 1\.29\}92\.85±1\.25\\textbf\{92\.85\}\{\\scriptstyle\\pm 1\.25\}93\.82±3\.92\\textbf\{93\.82\}\{\\scriptstyle\\pm 3\.92\}88\.20±1\.4488\.20\{\\scriptstyle\\pm 1\.44\}90\.07±0\.7890\.07\{\\scriptstyle\\pm 0\.78\}91\.27±2\.1291\.27\{\\scriptstyle\\pm 2\.12\}85\.42±0\.81\\textbf\{85\.42\}\{\\scriptstyle\\pm 0\.81\}91\.82±2\.26\\textbf\{91\.82\}\{\\scriptstyle\\pm 2\.26\}MJDW\\texttt\{MJ\}\_\{\\text\{DW\}\}\(1040 sched\.\)81\.80±1\.7581\.80\{\\scriptstyle\\pm 1\.75\}84\.65±2\.0984\.65\{\\scriptstyle\\pm 2\.09\}82\.91±2\.4782\.91\{\\scriptstyle\\pm 2\.47\}87\.80±1\.5487\.80\{\\scriptstyle\\pm 1\.54\}87\.64±1\.7187\.64\{\\scriptstyle\\pm 1\.71\}88\.00±4\.0888\.00\{\\scriptstyle\\pm 4\.08\}85\.00±2\.1785\.00\{\\scriptstyle\\pm 2\.17\}86\.04±1\.1186\.04\{\\scriptstyle\\pm 1\.11\}87\.27±3\.8187\.27\{\\scriptstyle\\pm 3\.81\}87\.00±0\.6387\.00\{\\scriptstyle\\pm 0\.63\}85\.56±1\.4785\.56\{\\scriptstyle\\pm 1\.47\}87\.27±1\.1587\.27\{\\scriptstyle\\pm 1\.15\}85\.50±1\.7985\.50\{\\scriptstyle\\pm 1\.79\}85\.14±1\.7285\.14\{\\scriptstyle\\pm 1\.72\}91\.64±2\.95\\textbf\{91\.64\}\{\\scriptstyle\\pm 2\.95\}83\.12±1\.1783\.12\{\\scriptstyle\\pm 1\.17\}86\.99±1\.7486\.99\{\\scriptstyle\\pm 1\.74\}

UnderASR5​@​3\\mathrm\{ASR\}\_\{5\}@3, DW achieves the higher ID average under the 260\-schedule setting, whereas SW is slightly stronger on OOD targets\. Under the 1040\-schedule setting, SW achieves the higher ID and OOD averages\. The same pattern is more pronounced underASR5​@​1\\mathrm\{ASR\}\_\{5\}@1: DW is stronger on ID evaluation with the 260 schedule, but SW is stronger on OOD evaluation, and SW leads on both ID and OOD evaluation with the 1040 schedule\. These results indicate a schedule\-dependent SW–DW trade\-off rather than uniform dominance by either weighting rule\. Overall, both variants remain strong under the more resistant GPT\-OSS\-20B training victim, supporting the robustness of the shared turn\-level credit assignment framework\.

## Appendix GCross\-Judge Evaluation

To assess the robustness of our conclusions to the choice of judge model, we additionally report cross\-dataset, multi\-judge results using three judges: the HarmBench classifier, WildGuard, and Qwen3Guard\. All results in this appendix use the Qwen3 attacker trained with dynamic\-weighted DC\-GRPO against Llama\-3\.1\-8B\-Instruct as the training\-time victim\. Throughout this appendix, we reportASR5​@​3\\mathrm\{ASR\}\_\{5\}@3,*i\.e\.,*success within at most five turns with up to three sampled trajectories\.

#### Cross\-judge robustness\.

Table[15](https://arxiv.org/html/2607.11070#A7.T15)reports cross\-dataset results under the three judge models\. Across all three datasets, the relative pattern across target models is broadly consistent: Llama\-3\.1, Qwen2\.5, Gemma\-2, and Mistral remain highly vulnerable, while GPT\-OSS\-20B is substantially more resistant\. This suggests that our main conclusions do not depend on a single judge model\.

The absolute ASR values vary somewhat across judges\. Averaged over all targets and datasets, Qwen3Guard is slightly more permissive than the HarmBench classifier and WildGuard, whereas the HarmBench classifier and WildGuard produce very similar averages overall\. Even so, the ranking of easier and harder target models remains largely stable across judges\.

Table 15:Cross\-dataset multi\-judge evaluation usingASR5​@​3\\mathrm\{ASR\}\_\{5\}@3\(%\)\. The Qwen3 attacker is trained with dynamic\-weighted DC\-GRPO against Llama\-3\.1\-8B\-Instruct\. Each target model is evaluated under three judges: HarmBench classifier \(HB\), WildGuard \(WG\), and Qwen3Guard \(Q3G\), on HarmBench \(HB\), StrongREJECT†\(SR†\), and JailbreakBench†\(JBB†\)\.TargetHBSR†JBB†Avg\.HBWGQ3GHBWGQ3GHBWGQ3GGPT\-OSS\-20B11\.0012\.5021\.5011\.4610\.7621\.1820\.0016\.3640\.0018\.31Llama\-3\.199\.50100\.00100\.0095\.8396\.8897\.2298\.18100\.00100\.0098\.62Qwen2\.5100\.00100\.00100\.00100\.0099\.3199\.6598\.1898\.18100\.0099\.48Gemma\-294\.5098\.50100\.0095\.4995\.4997\.5798\.1896\.3696\.3696\.94Mistral100\.00100\.00100\.0099\.3199\.3199\.31100\.00100\.0098\.1899\.57

## Appendix HASR Curves

This appendix visualizes the ASR curves of theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}Qwen3 attacker, corresponding to theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}\(Qwen3\) results in Table[13](https://arxiv.org/html/2607.11070#A6.T13)\. We report two complementary views:ASRk​@​1\\mathrm\{ASR\}\_\{k\}@1fork∈\{1,…,5\}k\\in\\\{1,\\ldots,5\\\}, which measures success as the number of allowed turns increases with a single sampled trajectory, andASR5​@​K\\mathrm\{ASR\}\_\{5\}@KforK∈\{1,…,5\}K\\in\\\{1,\\ldots,5\\\}, which measures success as the number of sampled trajectories increases under a fixed five\-turn budget\. Together, these curves show how performance scales with both interaction length and query budget\.

### H\.1Training victim: Llama\-3\.1\-8B\-Instruct

Figure[6](https://arxiv.org/html/2607.11070#A8.F6)shows the ASR curves when theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}Qwen3 attacker is trained against Llama\-3\.1\-8B\-Instruct\. Across HarmBench, StrongREJECT†, and JailbreakBench†, theASRk​@​1\\mathrm\{ASR\}\_\{k\}@1curves generally increase as more turns are allowed, indicating that additional interaction turns help the attacker refine the context and improve success within a single trajectory\. TheASR5​@​K\\mathrm\{ASR\}\_\{5\}@Kcurves show a similar trend with respect to the number of sampled trajectories: increasingKKconsistently improves or saturates ASR, suggesting that a larger query budget increases the chance of sampling at least one successful multi\-turn trajectory\. The improvement is especially visible on harder out\-of\-domain victim models, where single\-trajectory performance is lower but multi\-sample evaluation closes much of the gap\.

![Refer to caption](https://arxiv.org/html/2607.11070v1/x6.png)Figure 6:ASR curves ofMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}with a Qwen3 attacker trained against Llama\-3\.1\-8B\-Instruct\. The top row reportsASRk​@​1\\mathrm\{ASR\}\_\{k\}@1as the number of allowed turns increases from 1 to 5, and the bottom row reportsASR5​@​K\\mathrm\{ASR\}\_\{5\}@Kas the number of sampled trajectories increases from 1 to 5\.
### H\.2Training victim: GPT\-OSS\-20B

Figure[7](https://arxiv.org/html/2607.11070#A8.F7)shows the corresponding ASR curves when theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}Qwen3 attacker is trained against GPT\-OSS\-20B\. Compared with the Llama\-3\.1 training setting, the curves are already high across most evaluation targets, indicating strong transferability when training against the more resistant GPT\-OSS\-20B victim\. TheASRk​@​1\\mathrm\{ASR\}\_\{k\}@1curves still improve as the turn budget increases, showing that multi\-turn interaction remains useful even when the learned attacker is strong\. Likewise, theASR5​@​K\\mathrm\{ASR\}\_\{5\}@Kcurves improve or quickly saturate asKKincreases, reinforcing the same trend observed in Figure[6](https://arxiv.org/html/2607.11070#A8.F6): allowing more sampled trajectories provides a practical query\-budget trade\-off, where additional queries increase the probability of observing a successful attack trajectory\.

![Refer to caption](https://arxiv.org/html/2607.11070v1/x7.png)Figure 7:ASR curves ofMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}with a Qwen3 attacker trained against GPT\-OSS\-20B\. The top row reportsASRk​@​1\\mathrm\{ASR\}\_\{k\}@1as the number of allowed turns increases from 1 to 5, and the bottom row reportsASR5​@​K\\mathrm\{ASR\}\_\{5\}@Kas the number of sampled trajectories increases from 1 to 5\.

## Appendix IDiversity vs\. ASR

We further analyze the relationship between attack diversity and attack success rate\. Figure[8](https://arxiv.org/html/2607.11070#A9.F8)plots the average diversity score against ASR for our attacker variants and existing baselines\. For non\-ours baselines, we use the diversity and ASR results reported by TROJail\[[53](https://arxiv.org/html/2607.11070#bib.bib425)\], as discussed in the main text\. For our methods, we report theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variants with Qwen2\.5 and Qwen3 attackers, both with and without the optional DPP diversity bonus\.

The figure shows that diversity alone does not determine attack success\. Some methods obtain relatively high diversity but still have substantially lower ASR, indicating that generating varied prompts is not sufficient unless the prompts also receive effective learning signals\. Conversely, our baseMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}variants already achieve high ASR with moderate diversity, suggesting that the turn\-level credit assignment learned by DC\-GRPO is effective at improving attack quality even without an explicit diversity objective\.

Adding the DPP diversity bonus shifts our models toward higher diversity\. This effect is visible for both Qwen2\.5 and Qwen3 attackers, where the diversity\-augmented variants move to the right in Figure[8](https://arxiv.org/html/2607.11070#A9.F8)\. At the same time, the ASR remains high, although it can decrease slightly compared with the corresponding base variant\. This pattern is consistent with the quality–diversity trade\-off: encouraging semantic diversity expands the range of attack patterns explored by the policy, but may slightly reduce the chance of sampling the single most effective attack mode\.

Overall, Figure[8](https://arxiv.org/html/2607.11070#A9.F8)supports the role of the DPP term as an optional auxiliary component rather than a core part of the credit assignment rule\. The main DC\-GRPO training signal is sufficient to achieve strong ASR, while the diversity bonus can be used when broader coverage of attack strategies is desired\.

![Refer to caption](https://arxiv.org/html/2607.11070v1/x8.png)Figure 8:Diversity versus ASR\. The non\-ours baseline points are taken from TROJail\[[53](https://arxiv.org/html/2607.11070#bib.bib425)\]\. Our points correspond toMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}Qwen2\.5 and Qwen3 attackers, with and without the optional DPP diversity bonus\.
## Appendix JEvaluation on Additional Victim Models

We further evaluate the transferability ofMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}to two additional victim models, GPT\-4\.1\-mini\[[32](https://arxiv.org/html/2607.11070#bib.bib68)\]and DeepSeek\-V4\-Pro\[[54](https://arxiv.org/html/2607.11070#bib.bib69)\]\. For this evaluation, we reuse theMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}checkpoint from the GPT\-OSS\-20B experiment in Section[5\.2](https://arxiv.org/html/2607.11070#S5.SS2), refapp:transferability, without any additional training or target\-specific adaptation\. Following the main evaluation protocol, we reportASR5​@​3\\mathrm\{ASR\}\_\{5\}@3on HarmBench \(HB\), StrongREJECT†\(SR†\), and JailbreakBench†\(JBB†\)\. We additionally evaluate the generated responses with three independent judges: the HarmBench classifier, WildGuard, and Qwen3\-Guard\.

Table 16:Performance ofMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}with a Qwen3 attacker\. Each row reportsASR5​@​3\\mathrm\{ASR\}\_\{5\}\{@3\}\(%\) on victim LLMs under HarmBench \(HB\), StrongREJECT†\(SR†\), and JailbreakBench†\(JBB†\)\.TargetHarmBench JudgeWildGuard JudgeQwen3\-Guard JudgeHBSR†JBB†HBSR†JBB†HBSR†JBB†GPT\-4\.1\-mini99\.0099\.31100\.00100\.0099\.6594\.55100\.0098\.9694\.55DeepSeek\-V4\-Pro98\.0098\.61100\.00100\.0099\.3194\.55100\.0099\.3194\.55

Table[16](https://arxiv.org/html/2607.11070#A10.T16)shows thatMJSW\\texttt\{MJ\}\_\{\\text\{SW\}\}transfers strongly to both additional victims\. The overall averages across the three datasets and three judges are 98\.45% for GPT\-4\.1\-mini and 98\.26% for DeepSeek\-V4\-Pro\. Performance remains close to saturation on HarmBench and StrongREJECT†across all judges\. JailbreakBench†produces slightly lower scores under WildGuard and Qwen3\-Guard, but the ASR remains 94\.55% for both victim models\. These results provide additional evidence that the attacker checkpoint trained against GPT\-OSS\-20B generalizes to previously unseen victim models and that its transfer performance is consistent across different judge models\.