Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

# Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking
Source: [https://arxiv.org/html/2605.08778](https://arxiv.org/html/2605.08778)
Xiaoyu WenShanghai AI LaboratoryShanghai Jiao Tong UniversityHan QiShanghai AI LaboratoryZiyuan ZhouShanghai AI LaboratoryPeng YuShanghai AI LaboratoryShanghai Jiao Tong UniversityXingcheng XuShanghai AI LaboratoryDongrui LiuShanghai AI LaboratoryXia HuShanghai AI LaboratoryChaochao LuShanghai AI LaboratoryQiaosheng ZhangShanghai AI Laboratory

###### Abstract

Deploying LLMs in multi\-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns\. Recent training\-based multi\-turn jailbreak methods learn long\-horizon attack strategies from interaction feedback, but often rely on coarse trajectory\-level outcome signals that broadcast uniformly to every turn\. However, we find that turn\-level contributions in multi\-turn jailbreaking are*non\-uniform*\(only a few turns drive success\),*phase\-dependent*\(depending on the phase of context\) and*target\-specific*\(depending on the target model\)\. Such coarse outcome supervision induces a*credit assignment*problem, leading to over\-rewarding redundant turns in successful trajectories and under\-crediting useful intermediate turns in failed ones\. To address this, we proposeTRACE, a turn\-aware credit assignment framework for reinforcement learning \(RL\)\-based multi\-turn jailbreaking\. For successful trajectories, TRACE estimates turn\-level contributions via leave\-one\-turn\-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal\-aware penalty\. Furthermore, we reuse the attack\-side credit signal for multi\-turn defense alignment\. Extensive experiments on open\-source and closed\-source targets show that TRACE achieves the strongest overall performance in effectiveness, transferability, and efficiency, yielding about a25%25\\%relative improvement in attack success rate over the strongest RL baseline while also improving the safety–utility balance when reused for defense alignment\. Our code can be found in[https://github\.com/xsddys/TRACE](https://github.com/xsddys/TRACE)

††footnotetext:∗Equal contribution\.Disclaimer:This paper contains potentially offensive and harmful text\.

## 1Introduction

Large Language Models \(LLMs\) have demonstrated strong capabilities across diverse real\-world applications\(bai2025intern\), yet their widespread deployment also raises significant safety concerns, particularly when they are exposed to adversarial inputs or jailbreak attacks\(ganguli2022red\)\. Although existing safety mechanisms\(ji2025pku\)can mitigate many single\-turn attacks, practical misuse often unfolds through multi\-turn interactions\(li2024llm\)\. In this setting, malicious intent can be distributed across multiple benign\-looking turns rather than exposed in a single prompt\(russinovich2025great\)\. This allows harmful context to accumulate gradually, making such attacks harder to detect and defend against\.

Existing multi\-turn jailbreak methods include*training\-free workflows*and*training\-based methods*\. Training\-free workflows\(jiang2024redqueen;russinovich2025great;weng\-etal\-2025\-foot;yang2024jigsaw;yang2024chain\)rely on predefined interaction patterns or heuristic planning and lack dynamic strategy adaptation, limiting their effectiveness in complex multi\-turn interactions\(ha2025m2s;yang2025multiturn\)\. Training\-based methods address this limitation by learning attacker strategies from feedback, which mainly fall into two categories:*alignment\-based methods*\(guo\-etal\-2025\-mtsa;zhao2025siren\)and*reinforcement learning \(RL\)\-based methods*\(feng2026sema;xiong2025trojail\)\. The former optimize prompt generation at each turn to maximize immediate response harmfulness, but ignore long\-term harmful effects and suffer from high exploration complexity\. In contrast, the latter maximizes the harmfulness of the final response over the trajectory, enabling the attacker to learn long\-term jailbreak strategies\.

Despite recent progress, RL\-based multi\-turn jailbreaking still faces two major limitations\. \(i\) Existing methods assign the same trajectory\-level signal to all turns, which may provide misleading training signals\. As shown in Fig\.[1](https://arxiv.org/html/2605.08778#S1.F1), redundant turns in successful trajectories may be over\-rewarded as if causally contributed to the final jailbreak\. \(ii\) Existing methods lack reliable intermediate feedback for turn\-level credit assignment\. Unlike math, coding, or tool\-use tasks with verifiable progress signals\(wang2025igpo;zhang2025unlocking\), jailbreak success is semantic and context\-dependent, which lacks reliable supervision from local feedback\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x1.png)Figure 1:Overview of TRACE\.Existing multi\-turn jailbreak training paradigms rely on heuristic workflows, DPO, or RL with trajectory\-level outcome rewards\. TRACE keeps the multi\-turn RL rollout unchanged, but assigns the final outcome signal to individual turns by turn\-level contributions, distinguishing setup, redundant, premature\-exposure, and critical turns\.To address these issues, we proposeTRACE\(TuRn\-levelAssignment forCrEdit\), a framework for turn\-aware credit assignment in RL\-based multi\-turn jailbreaking, making the following contributions:

- •We characterize the credit assignment problem in RL\-based multi\-turn jailbreaking, showing that turn\-level contribution is \(i\)non\-uniform, with only a few turns driving jailbreak success; \(ii\)phase\-dependent, depending on whether a turn fits the current stage in the context; and \(iii\)target\-specific, depending on the safety boundaries of the target model\.
- •We propose TRACE, which assigns outcome signals with different turn\-level credit rules for successful and failed trajectories\. For successful trajectories, TRACE estimates turn credit via leave\-one\-turn\-out semantic masking; for failed trajectories, it assigns penalties over prompt harmfulness and semantic relevance\.
- •We conduct extensive attacks on both open\-source and closed\-source target models\. TRACE improves attack success rate \(ASR\) by about25%25\\%relatively over the strongest RL baseline, while demonstrating stronger transferability and higher efficiency than existing multi\-turn methods\.
- •We further reuse TRACE’s attack\-side credit signal for multi\-turn defense\. By aligning latent\-risk and direct\-harm states, TRACE enables early risk intervention and improves the safety–utility balance of defended models\.

## 2Preliminaries

##### Multi\-turn Attack

Following prior work\(guo\-etal\-2025\-mtsa;xiong2025trojail;zhao2025siren\), we formulate multi\-turn jailbreaking as a closed\-loop interaction between a trainable attacker modelπθ\\pi\_\{\\theta\}and a fixed target modelπϕ\\pi\_\{\\phi\}\. Given a harmful seed promptx0x\_\{0\}, at turntt, the attacker generates an adversarial promptxt∼πθ\(⋅∣x0,τt−1\)x\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{0\},\\tau\_\{t\-1\}\)conditioned on the seed and dialogue historyτt−1:=\(x1,y1,…,xt−1,yt−1\)\\tau\_\{t\-1\}:=\(x\_\{1\},y\_\{1\},\\ldots,x\_\{t\-1\},y\_\{t\-1\}\); the target produces a responseyt∼πϕ\(⋅∣τt−1,xt\)y\_\{t\}\\sim\\pi\_\{\\phi\}\(\\cdot\\mid\\tau\_\{t\-1\},x\_\{t\}\); and the judge model evaluates the response to assign a harmfulness scoreR=r​\(x0,yt\)∈\[0,1\]R=r\(x\_\{0\},y\_\{t\}\)\\in\[0,1\]\. This interaction can be summarized as:

x0\\displaystyle x\_\{0\}→πθattackerx1→πϕtargety1→πθattackerx2→πϕtargety2​⋯→πθattackerxt→πϕtargetyt\.\\displaystyle\\xrightarrow\[\\pi\_\{\\theta\}\]\{\\text\{attacker\}\}x\_\{1\}\\xrightarrow\[\\pi\_\{\\phi\}\]\{\\text\{target\}\}y\_\{1\}\\xrightarrow\[\\pi\_\{\\theta\}\]\{\\text\{attacker\}\}x\_\{2\}\\xrightarrow\[\\pi\_\{\\phi\}\]\{\\text\{target\}\}y\_\{2\}\\cdots\\xrightarrow\[\\pi\_\{\\theta\}\]\{\\text\{attacker\}\}x\_\{t\}\\xrightarrow\[\\pi\_\{\\phi\}\]\{\\text\{target\}\}y\_\{t\}\.\(1\)The process terminates whenR≥γR\\geq\\gammaat any turn, whereγ\\gammais the harmfulness threshold and crossing it denotes a successful attack, or when the maximum number of turns is reached\.

##### Multi\-turn GRPO

Multi\-turn GRPO extends standard GRPO\(shao2024deepseekmath\)to multi\-turn dialogue by computing advantages from trajectory\-level outcome rewards\(wan2025rema;wang2025ragen\)\. Given a group ofGGgenerated trajectories, we maximize the following objective:

𝒥MT\-GRPO\(θ\):=𝔼\[\\displaystyle\\mathcal\{J\}\_\{\\text\{MT\-GRPO\}\}\(\\theta\)=\\mathbb\{E\}\\Big\[1G∑i=1G1Ti∑t=1Ti1\|xi,t\|∑k=1\|xi,t\|\(min\(ρi,t,k\(θ\)A^i,\\displaystyle\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\frac\{1\}\{\\lvert x\_\{i,t\}\\rvert\}\\sum\_\{k=1\}^\{\\lvert x\_\{i,t\}\\rvert\}\\big\(\\min\(\\rho\_\{i,t,k\}\(\\theta\)\\hat\{A\}\_\{i\},\(2\)clip\(ρi,t,k\(θ\),1−ϵ,1\+ϵ\)A^i\)−βDKL\(πθ∥πref\)\)\],\\displaystyle\\operatorname\{clip\}\(\\rho\_\{i,t,k\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{i\}\)\-\\beta D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\)\\big\)\\Big\],where the token\-level importance ratioρi,t,k​\(θ\)\\rho\_\{i,t,k\}\(\\theta\)for clipping and the group\-normalized advantageA^i\\hat\{A\}\_\{i\}are defined as:

ρi,t,k​\(θ\):=πθ​\(xi,t,k∣x0,τi,t−1,xi,t,<k\)πθold​\(xi,t,k∣x0,τi,t−1,xi,t,<k\),A^i:=Ri−mean​\(\{Ri\}i=1G\)std​\(\{Ri\}i=1G\),\\rho\_\{i,t,k\}\(\\theta\):=\\frac\{\\pi\_\{\\theta\}\(x\_\{i,t,k\}\\mid x\_\{0\},\\tau\_\{i,t\-1\},x\_\{i,t,<k\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(x\_\{i,t,k\}\\mid x\_\{0\},\\tau\_\{i,t\-1\},x\_\{i,t,<k\}\)\},\\quad\\hat\{A\}\_\{i\}:=\\frac\{R\_\{i\}\-\\text\{mean\}\(\\\{R\_\{i\}\\\}^\{G\}\_\{i=1\}\)\}\{\\text\{std\}\(\\\{R\_\{i\}\\\}^\{G\}\_\{i=1\}\)\},andxi,t,kx\_\{i,t,k\}is thekk\-th token generated by the attacker model at turnttof theii\-th trajectory\.

## 3The Credit Assignment Problem in RL\-based Multi\-turn Jailbreaking

Existing RL\-based multi\-turn jailbreak methods\(feng2026sema;xiong2025trojail\)broadcast the same trajectory\-level outcome signal to all turns\. In this section, we show that turn\-level contributions are non\-uniform, phase\-dependent, and target\-specific, and demonstrate that uniform broadcasting leads to distorted credit assignment\.

### 3\.1Insight 1\(Non\-uniformity\): Not All Turns Matter Equally

To assess the contribution of each intermediate turn to the final attack success, we perform a leave\-one\-turn\-out semantic masking test\. Formally, letτ:=\(x1,y1,…,xT,yT\)\\tau:=\(x\_\{1\},y\_\{1\},\\ldots,x\_\{T\},y\_\{T\}\)denote a sampled trajectory, where the number of turnsTTmay vary across trajectories due to early termination\. For each non\-final turnt<Tt<T, we remove the interaction pair\(xt,yt\)\(x\_\{t\},y\_\{t\}\)from the non\-final dialogue history, yielding the masked history

τ−t=\(x1,y1,…,xt−1,yt−1,xt\+1,yt\+1,…,xT−1,yT−1\)\.\\tau\_\{\-t\}=\(x\_\{1\},y\_\{1\},\\ldots,x\_\{t\-1\},y\_\{t\-1\},x\_\{t\+1\},y\_\{t\+1\},\\ldots,x\_\{T\-1\},y\_\{T\-1\}\)\.We keep the final attacker queryxTx\_\{T\}fixed and resample the final target response under the masked historyyT′∼πϕ\(⋅\|τ−t,xT\)y\_\{T\}^\{\\prime\}\\sim\\pi\_\{\\phi\}\(\\cdot\\,\|\\,\\tau\_\{\-t\},x\_\{T\}\)\. Comparing the harmfulness of the original trajectory,h=𝕀​\{r​\(x0,yT\)\>γ\}h=\\mathbb\{I\}\\\{r\(x\_\{0\},y\_\{T\}\)\>\\gamma\\\}, with that of the masked trajectory,h′=𝕀​\{r​\(x0,yT′\)\>γ\}h^\{\\prime\}=\\mathbb\{I\}\\\{r\(x\_\{0\},y\_\{T\}^\{\\prime\}\)\>\\gamma\\\}, we define the following four turn categories based on the resulting change\.

We first consider the case where the original trajectory is unsafe, i\.e\.,h=1h=1\. \(i\) Turnttis an*attack\-critical turn*if removing it changes the outcome to safe, i\.e\.,h′=0h^\{\\prime\}=0, which implies that the turn is necessary for the realized jailbreak; \(ii\) otherwise, it is*redundant*, which indicates that the jailbreak can still succeed without this turn\. We then consider the case where the original trajectory is safe, i\.e\.,h=0h=0\. \(iii\) Turnttis*neutral*if the outcome remains safe after removal, i\.e\.,h′=0h^\{\\prime\}=0, which indicates that the turn has little observable effect on the realized failure; \(iv\) otherwise, it is*safety\-critical*, which implies that the removed turn had suppressed a potential successful jailbreak\.

Table 1:Turn categories from leave\-one\-turn\-out masking and final\-response resampling\.As shown in Tab\.[1](https://arxiv.org/html/2605.08778#S3.T1), in successful trajectories, 47\.1% of turns are estimated as attack\-critical, while 52\.9% are categorized as redundant, indicating that many turns may receive uniformly positive trajectory\-level feedback despite contributing little to the final jailbreak; in failed ones, most turns are estimated as neutral \(94\.1%\), with only a small fraction being safety\-critical \(5\.9%\), suggesting that only a few interactions actively suppress potential jailbreaks\. These observations indicate that turn\-level contributions are highly non\-uniform, and not all turns matter equally\. Therefore, redundant turns should be down\-weighted, while safety\-critical turns should be avoided, as they may respectively contribute little to and hinder attack success\.

### 3\.2Insight 2\(Phase Dependency\): Turn\-Level Contributions Depend on Conversational Phase

A turn’s contribution to jailbreak success depends not only on its surface\-level harmfulness, but also on the phase of context in which it appears, i\.e\., its relative position within the dialogue trajectory\. In multi\-turn attacks, the same prompt can have substantially different effects at different phases\. A potentially unsafe prompt may trigger refusal and derail the attack if introduced too early, whereas it may become effective after sufficient contextual setup\.

To examine this effect, we compare the harmfulness distribution of attacker prompts across dialogue turns in successful and failed trajectories\. Specifically, we consider multi\-turn dialogues with a fixed number of turns, e\.g\.,T=5T=5\. For each attacker queryxtx\_\{t\}, we use a guard model to classify it into one of three categories \(safe, controversial, or unsafe\)\. For each turn, we compute the percentage of queries in each category and analyze how these proportions vary across turns\.

As shown in Fig\.[2](https://arxiv.org/html/2605.08778#S3.F2)\(a\), successful trajectories exhibit a clear contextual setup pattern\. They typically begin with safe prompts in early phases to establish benign context, and gradually shift toward more harmful prompts in later phases after sufficient buildup\. Failed trajectories deviate from this pattern in two common ways: they may become overly aggressive in early phases, which we call*premature exposure*, or remain overly benign in later phases, which we call*harmfulness drift*\. These patterns suggest that the effectiveness of an attack depends not only on the level of harmfulness, but also on when it is introduced within the dialogue\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x2.png)Figure 2:Attack dynamics across turn phases and targets\. \(a\) Harmfulness distributions across turn phases for Qwen2\.5\-7B\-IT, shown separately for successful and failed trajectories; the phase bin denotes normalized turn position\. \(b\) Harmfulness distributions for gpt\-oss\-20b\. \(c\) Early refusal rates against Qwen2\.5\-7B\-IT\. \(d\) Early refusal rates against gpt\-oss\-20b\.
### 3\.3Insight 3\(Target Specificity\): Turn\-Level Contributions Depend on Target Safety Behavior

Turn\-level contribution is not only phase\-dependent, but also target\-specific\. Different target models exhibit different safety behaviors and rejection boundaries, such that the same attacker prompt may be accepted by one model but refused by another\. As a result, whether a turn contributes positively to the attack depends on the specific target it interacts with\.

This target\-specific effect is evident from the comparison between Fig\.[2](https://arxiv.org/html/2605.08778#S3.F2)\(a\) and \(b\)\. Compared with Qwen2\.5\-7B\-IT, gpt\-oss\-20b requires more cautious early probing, and failed trajectories are more likely to drift toward safe prompts in later turns, indicating different safety boundaries across targets\. To further examine this observation, we measure early refusal rates \(i\.e\., refusals occurring in turns 1–2 of a five\-turn dialogue\) across training iterations for different target models\. As shown in Figs\.[2](https://arxiv.org/html/2605.08778#S3.F2)\(c\) and \(d\), attacks can recover after early refusal on Qwen2\.5\-7B\-IT, whereas early refusal on gpt\-oss\-20b rarely leads to success, indicating different refusal sensitivity across models\. These results consistently show that turn\-level contribution depends on target\-specific safety behavior, and thus cannot be modeled using a single, target\-agnostic credit assignment rule\.

## 4Method

Based on the above three insights, TRACE makes a single modification to Eq\. \(2\): it replaces the trajectory\-level advantageA^i\\hat\{A\}\_\{i\}with a turn\-aware advantageA^i,t\\hat\{A\}\_\{i,t\}:

A^i,t=mi,t​A^io\+A^i,tp,\\hat\{A\}\_\{i,t\}=m\_\{i,t\}\\hat\{A\}^\{o\}\_\{i\}\+\\hat\{A\}^\{p\}\_\{i,t\},\(3\)whereA^io\\hat\{A\}^\{o\}\_\{i\}is the trajectory\-level outcome advantage,mi,tm\_\{i,t\}redistributes this outcome signal across turns, andA^i,tp\\hat\{A\}^\{p\}\_\{i,t\}is a refusal\-aware local process penalty\. The multipliermi,tm\_\{i,t\}is defined separately for successful trajectoriesS\+S^\{\+\}and failed trajectoriesS−S^\{\-\}:

mi,t:=\{mi,t\+,τi∈S\+,mi,t−,τi∈S−\.m\_\{i,t\}:=\\begin\{cases\}m^\{\+\}\_\{i,t\},&\\tau\_\{i\}\\in S^\{\+\},\\\\ m^\{\-\}\_\{i,t\},&\\tau\_\{i\}\\in S^\{\-\}\.\\end\{cases\}\(4\)For successful trajectories, we estimatemi,t\+m^\{\+\}\_\{i,t\}using leave\-one\-turn\-out semantic masking, as described in Sec\.[4\.1](https://arxiv.org/html/2605.08778#S4.SS1)\. For failed trajectories, we constructmi,t−m^\{\-\}\_\{i,t\}from harmfulness and relevance deviation penalties, as described in Sec\.[4\.2](https://arxiv.org/html/2605.08778#S4.SS2)\. Finally, we introduce the local refusal\-aware process penaltyA^i,tp\\hat\{A\}^\{p\}\_\{i,t\}in Sec\.[4\.3](https://arxiv.org/html/2605.08778#S4.SS3)\. By construction,mi,tm\_\{i,t\}preserves the average strength of advantage propagation, i\.e\.,1Ti​∑t=1Timi,t=1\\frac\{1\}\{T\_\{i\}\}\\sum\_\{t=1\}^\{T\_\{i\}\}m\_\{i,t\}=1, and only changes how credit is distributed across turns\. Fig\.[3](https://arxiv.org/html/2605.08778#S4.F3)provides an overview of the full TRACE pipeline, and Algorithm[1](https://arxiv.org/html/2605.08778#alg1)presents the complete procedure\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x3.png)Figure 3:Framework of TRACE\.Starting from outcome rewards, TRACE constructs turn\-aware credit by combining success\-side leave\-one\-turn\-out semantic masking, failure\-side turn\-aware penalties, and an optional refusal\-aware local process penalty\.### 4\.1Success\-Side Turn\-aware Credit Assignment

Using the leave\-one\-turn\-out semantic masking procedure in Sec\.[3\.1](https://arxiv.org/html/2605.08778#S3.SS1), for each successful trajectoryτi∈𝒮\+\\tau\_\{i\}\\in\\mathcal\{S\}^\{\+\}and each non\-final turnt<Tit<T\_\{i\}, we define the raw turn credit as

ci,t:=r​\(x0,yi,Ti\)−r​\(x0,yi,Ti′\),c\_\{i,t\}:=r\(x\_\{0\},y\_\{i,T\_\{i\}\}\)\-r\(x\_\{0\},y^\{\\prime\}\_\{i,T\_\{i\}\}\),\(5\)which measures the decrease in final\-response harmfulness after masking turntt\. We normalize it within each trajectory by

zi,t:=clip⁡\(ci,t−μiσi,−zmax,zmax\),μi=mean​\(\{ci,t\}t=1Ti−1\),σi=std​\(\{ci,t\}t=1Ti−1\),z\_\{i,t\}:=\\operatorname\{clip\}\\left\(\\frac\{c\_\{i,t\}\-\\mu\_\{i\}\}\{\\sigma\_\{i\}\},\-z\_\{\\max\},z\_\{\\max\}\\right\),\\quad\\mu\_\{i\}=\\text\{mean\}\\big\(\\\{c\_\{i,t\}\\\}\_\{t=1\}^\{T\_\{i\}\-1\}\\big\),\\ \\sigma\_\{i\}=\\text\{std\}\\big\(\\\{c\_\{i,t\}\\\}\_\{t=1\}^\{T\_\{i\}\-1\}\\big\),\(6\)wherezmaxz\_\{\\max\}is the clipping threshold\. The normalized credit is then converted into success\-side turn\-aware multipliers\. Letmi,Ti\+=1m^\{\+\}\_\{i,T\_\{i\}\}=1; fort<Tit<T\_\{i\}, we estimatemi,t\+m\_\{i,t\}^\{\+\}as

mi,t\+=\(1−λ1\)\+λ1​\(Ti−1\)​exp⁡\(zi,t\)∑s<Tiexp⁡\(zi,s\),m\_\{i,t\}^\{\+\}=\(1\-\\lambda\_\{1\}\)\+\\lambda\_\{1\}\(T\_\{i\}\-1\)\\frac\{\\exp\(z\_\{i,t\}\)\}\{\\sum\_\{s<T\_\{i\}\}\\exp\(z\_\{i,s\}\)\},\(7\)whereλ1\\lambda\_\{1\}controls the deviation from uniform broadcasting\.

### 4\.2Failure\-Side Turn\-Aware Deviation Penalty

As discussed in Sec\.[3\.2](https://arxiv.org/html/2605.08778#S3.SS2), failed trajectories exhibit mixed error modes, including premature exposure and harmfulness drift\. The leave\-one\-turn\-out semantic attribution used for successful trajectories is therefore not applied here\. Instead, we use a target\-specific penalty\. For each failed trajectoryτi∈𝒮−\\tau\_\{i\}\\in\\mathcal\{S\}^\{\-\}, we calculate the target\-specific penalty in terms of harmfulness and relevance\.

##### Harmfulness Penalty

Let𝒞:=\{safe,controversial,unsafe\}\\mathcal\{C\}:=\\\{\\text\{safe\},\\text\{controversial\},\\text\{unsafe\}\\\}denote the harmfulness category\. First, we use the target\-specific harmfulness distribution, shown in Fig\.[2](https://arxiv.org/html/2605.08778#S3.F2)\(a\) and \(b\), to calculate the success prior𝐪t=\{qt,ℓ\}ℓ∈𝒞\\mathbf\{q\}\_\{t\}=\\\{q\_\{t,\\ell\}\\\}\_\{\\ell\\in\\mathcal\{C\}\}for turntt\. The calculation for the success prior is deferred to Appendix[B\.2](https://arxiv.org/html/2605.08778#A2.SS2)\. A naive penalty on harmfulness is1−qt,ℓi,t1\-q\_\{t,\\ell\_\{i,t\}\}, whereℓi,t∈𝒞\\ell\_\{i,t\}\\in\\mathcal\{C\}denotes the harmfulness label ofxi,tx\_\{i,t\}\. However, the naive form would penalize any label with prior probability below one, including labels that are appropriate for the current phase\. To avoid this issue, we replace the naive penalty with a concentration\-adaptive threshold derived from the success prior:

Bi,tH:=max⁡\(0,∑ℓ∈𝒞\(qt,ℓ\)2−qt,ℓi,t\)\.B\_\{i,t\}^\{H\}:=\\max\\Big\(0,\\;\\sum\_\{\\ell\\in\\mathcal\{C\}\}\\big\(q\_\{t,\\ell\}\\big\)^\{2\}\-q\_\{t,\\ell\_\{i,t\}\}\\Big\)\.\(8\)The term∑ℓ∈𝒞\(qt,ℓ\)2\\sum\_\{\\ell\\in\\mathcal\{C\}\}\(q\_\{t,\\ell\}\)^\{2\}measures the concentration of successful behavior at phasett, andBi,tHB\_\{i,t\}^\{H\}is positive only when the observed label falls below this phase\-specific concentration level\. This protects common phase\-appropriate labels while penalizing atypical harmfulness levels\. Appendix[C\.3\.1](https://arxiv.org/html/2605.08778#A3.SS3.SSS1)provides an intuitive example for Eq\. \([8](https://arxiv.org/html/2605.08778#S4.E8)\)\.

##### Relevance Penalty

To prevent the semantic meaning of an intermediate turnxi,tx\_\{i,t\}from deviating significantly from that of the original intentx0x\_\{0\}, we further introduce a relevance penalty term\. LetEi,t:=cosine​\(e​\(x0\),e​\(xi,t\)\)E\_\{i,t\}:=\\text\{cosine\}\\big\(e\(x\_\{0\}\),e\(x\_\{i,t\}\)\\big\)denote the cosine similarity between the sentence embeddings of the original harmful seed and the current attacker prompt\. The relevance penalty is defined as

Bi,tR:=max⁡\(0,\(Lt−Ei,t\)/Lt\),B\_\{i,t\}^\{R\}:=\\max\\left\(0,\\;\(L\_\{t\}\-E\_\{i,t\}\)/L\_\{t\}\\right\),\(9\)whereLtL\_\{t\}is the target\-specific lower reference, defined as the2525th percentile ofEi,tE\_\{i,t\}over successful trajectories\. The details ofLtL\_\{t\}are provided to Appendix[B\.2](https://arxiv.org/html/2605.08778#A2.SS2)\. If the currentxi,tx\_\{i,t\}is similar tox0x\_\{0\}, resulting inLt<Ei,tL\_\{t\}<E\_\{i,t\}, the relevance penalty vanishes\.

LetBi,t=Bi,tH\+Bi,tRB\_\{i,t\}=B\_\{i,t\}^\{H\}\+B\_\{i,t\}^\{R\}\. For a failed trajectoryτi∈𝒮−\\tau\_\{i\}\\in\\mathcal\{S\}\_\{\-\}andt≤Tit\\leq T\_\{i\}, the multiplier for failure\-side is defined as

mi,t−:=\(1−λ2\)\+λ2​\(Bi,t/mean​\(\{Bi,t\}t=1Ti\)\),m\_\{i,t\}^\{\-\}:=\(1\-\\lambda\_\{2\}\)\+\\lambda\_\{2\}\\big\(B\_\{i,t\}/\\text\{mean\}\(\\\{B\_\{i,t\}\\\}\_\{t=1\}^\{T\_\{i\}\}\)\\big\),\(10\)whereλ2\\lambda\_\{2\}controls how strongly the normalized failure penalty modulates the multiplier\. A larger multiplier assigns a stronger penalty to turns whose harmfulness level or semantic relevance is more inconsistent with the successful priors\.

### 4\.3Refusal\-Aware Local Process Penalty

As discussed in Sec\.[3\.3](https://arxiv.org/html/2605.08778#S3.SS3), the contribution of each turn depends on the specific target model, and local refusals often indicate unhelpful turns for that target\. Motivated by this observation, we further introduce an intermediate penalty to capture such local refusals\. For a trajectoryτi\\tau\_\{i\}, we determine whether the interaction\(xi,t,yi,t\)\(x\_\{i,t\},y\_\{i,t\}\)triggers a refusal and define the following process reward:

ri,tp=−𝕀​\{refusal​\(xi,t,yi,t\)\}\.r\_\{i,t\}^\{p\}=\-\\mathbb\{I\}\\\{\\text\{refusal\}\(x\_\{i,t\},y\_\{i,t\}\)\\\}\.\(11\)Because the refusal penalty is a local reward, we use it directly as the turn\-level advantage rather than propagating it with a suffix sum\. Specifically, we setA^i,tp=λp​ri,tp\\hat\{A\}\_\{i,t\}^\{p\}=\\lambda\_\{p\}r\_\{i,t\}^\{p\}and useλp\\lambda\_\{p\}to control the strength of the process\-level penalty\.

## 5Experiments

### 5\.1Experimental Setup

Models\.We use Qwen2\.5\-3B\-Instruct as attacker model\. Regarding target models, we use Qwen2\.5\-7B\-IT\(qwen2\.5\), Llama3\.1\-8B\-IT\(grattafiori2024llama3\), and gpt\-oss\-20B\(openai2025gptoss\)during training\. During evaluation, we further include closed\-source models such as GPT\-4o\(openai2024gpt\)and Gemini\-2\.5\-Pro\(deepmind2025gemini25pro\)\. The HarmBench Classifier\(mazeika2024harmbench\)serves as the default reward model and evaluation judge\. To mitigate potential reward hacking, we additionally evaluate using GPT\-4o\(openai2024gpt\)and LlamaGuard4\-12B\(meta2025llamaguard4\)in Appendix[D\.5](https://arxiv.org/html/2605.08778#A4.SS5)\.

Dataset\.For training, we use 520 harmful seeds from AdvBench\(zou2023universal\)\. For evaluation, we consider three benchmarks: \(i\) 159 examples from the standard split of HarmBench \(HB\)\(mazeika2024harmbench\), \(ii\) 55 examples from the original split of JailbreakBench \(JBB\)\(chao2024jailbreakbench\), and \(iii\) 200 vanilla harmful prompts from the WildJailBreak test split \(WJB\)\(jiang2024wildjailbreak\)\.

Metric\.We report Attack Success Rate underkktries per seed \(ASR@k\), defined as the fraction of harmful seeds for which at least one of thekkmulti\-turn attack attempts succeeds\. Unless otherwise specified, each attempt is limited to 5 turns\. Additional details are provided in Appendix[C\.5](https://arxiv.org/html/2605.08778#A3.SS5)\.

Baselines\.We compare TRACE with the following baselines: \(i\) single\-turn jailbreak methods, including PAIR\(chao2025pair\), AutoDAN\-Turbo\(liu2025autodanturbo\), and Jailbreak\-R1\(guo2025jailbreakr1\); \(ii\) multi\-turn workflow methods, including ActorAttack\(ren\-etal\-2025\-actorattack\), Crescendo\(russinovich2025great\), MUSE\-A\(yan\-etal\-2025\-muse\), and X\-Teaming\(rahman2025xteaming\); and \(iii\) training\-based multi\-turn jailbreak methods, including Siren\(zhao2025siren\)and TROJail\(xiong2025trojail\)\. More experimental details are provided in Appendix[D\.1](https://arxiv.org/html/2605.08778#A4.SS1)\.

Table 2:ASR@1 \(%\) of jailbreak methods across target models judged by HarmBench Classifier\. ForTRACE \(single\), we trained against and evaluated on the same target within each model family\. ForTRACE \(mix\), we jointly trained against two fixed targets \(gpt\-oss\-20b and Llama3\.1\-8B\-IT\)\.
### 5\.2Main Results

We organize the main results around four questions: effectiveness, transferability, efficiency, and whether turn\-aware credit alleviates the credit assignment problem\.

##### Q1: Does TRACE outperform existing jailbreak attacks?

We first evaluate TRACE in a matched train\-test setting, where the attacker is trained against and evaluated on the same target model\. As shown in Tab\.[2](https://arxiv.org/html/2605.08778#S5.T2), TRACE demonstrates the strongest attack performance across benchmarks\. Compared with workflow\-based methods \(e\.g\., X\-Teaming\) using GPT\-4o as the attacker, TRACE more than doubles the average ASR@1 from38\.40%38\.40\\%to over80%80\\%, despite using only Qwen2\.5\-3B\-Instruct\. Compared with TROJail, the strongest RL\-based multi\-turn baseline,TRACE \(mix\)improves the average ASR@1 from69\.97%69\.97\\%to87\.10%87\.10\\%\. Given that TROJail already combines trajectory\-level outcome optimization with heuristic process rewards, this improvement highlights the advantage of replacing uniform trajectory\-level broadcasting with turn\-aware credit assignment\. To rule out reward hacking concerns, we further evaluate using LlamaGuard4\-12B and GPT\-4o as judges\. TRACE consistently outperforms existing methods under both judges \(provided in Appendix[D\.5](https://arxiv.org/html/2605.08778#A4.SS5)\)\. TRACE is particularly effective on more robust targets\. On gpt\-oss\-20b, several workflow\-based multi\-turn methods achieve less than10%10\\%ASR@1, while both TRACE variants maintain above80%80\\%ASR@1 on average across three datasets\. Overall,TRACE \(mix\)achieves the best average ASR@1, suggesting that mixed\-target training leads to a more robust attack policy across target models\.

##### Q2: Does TRACE learn transferable attack policies?

To evaluate transferability, we move beyond the matched setting and evaluate cross\-target transfer experiments on HarmBench under cross\-family, in\-family, and closed\-source settings\. Fig\.[4](https://arxiv.org/html/2605.08778#S5.F4)\(a\) shows thatTRACE \(single\)exhibits target\-dependent transfer behavior and does not learn a general attack policy\. For example, an attacker trained on gpt\-oss\-20b achieves only56\.0%56\.0\\%ASR@1 on Llama3\.1\-8B\-IT, while performing much better on other targets\. This suggests that training against a single target primarily captures target\-specific safety behavior rather than a broadly transferable attack strategy\.TRACE \(mix\)mitigates this limitation by jointly training against two stronger safety targets, gpt\-oss\-20b and Llama3\.1\-8B\-Instruct\. As shown in Fig\.[4](https://arxiv.org/html/2605.08778#S5.F4)\(a\), it achieves strong cross\-target ASR@1 across all three open models\. It also transfers well to unseen closed\-source models, as illustrated in Fig\.[4](https://arxiv.org/html/2605.08778#S5.F4)\(b\), achieving high ASR@5 even on strong closed\-source targets, including96\.7%96\.7\\%on GPT\-4o and93\.1%93\.1\\%on Gemini\-2\.5\-Pro\. These results indicate that training on stronger and more diverse targets leads to a more robust and transferable attack policy\. More details are provided in Appendix[D\.6](https://arxiv.org/html/2605.08778#A4.SS6)\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x4.png)Figure 4:Transferability of TRACE\. \(a\) Cross\-family transfer on Qwen2\.5\-7B\-IT, gpt\-oss\-20b, and Llama3\.1\-8B\-IT judged by HarmBench Classifier; first three rows: TRACE \(single\), last row: TRACE \(mix\)\. \(b\) In\-family transfer and evaluation on closed\-source models, judged by LlamaGuard4\.
##### Q3: Is TRACE efficient in target calls and turn budget?

To evaluate budget efficiency, we conduct experiments under budgets of query and turn\. As illustrated in Fig\.[5](https://arxiv.org/html/2605.08778#S5.F5)\(a\), TRACE consistently occupies the upper\-left region across all targets, outperforming baselines with comparable or much larger query budgets\. Fig\.[5](https://arxiv.org/html/2605.08778#S5.F5)\(b\) further demonstrates that TRACE reaches over80%80\\%ASR@1 on gpt\-oss\-20b within four turns and quickly saturates, while workflow baselines remain far lower even with larger turn limits\. Therefore, TRACE achieves large ASR@1 improvements without incurring higher test\-time query or turn budgets, effectively reducing the test\-time scaling burden\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x5.png)Figure 5:Efficiency analysis of TRACE\.\(a\) ASR@1 versus the average number of target calls across target models\. \(b\) ASR@1 under maximum turn limits for typical multi\-turn methods\.
##### Q4: Does TRACE alleviate the credit assignment problem in the learned policy?

We analyze the harmfulness distribution of attacker prompts\. As shown in Fig\.[6](https://arxiv.org/html/2605.08778#A2.F6)in Appendix[B\.3](https://arxiv.org/html/2605.08778#A2.SS3), compared with GRPO, TRACE produces a more phase\-structured strategy, with fewer unsafe early prompts and less late harmfulness drift\. This indicates that TRACE alleviates the credit assignment problem by reinforcing turns that align with their correct roles\. See Appendix[B\.3](https://arxiv.org/html/2605.08778#A2.SS3)for more details\.

### 5\.3Ablation on Turn\-Level Credit Assignment

We conduct ablations by progressively adding success\-side credit, failure\-side credit, and the refusal\-aware local process penalty\. As demonstrated in Tab\.[3](https://arxiv.org/html/2605.08778#S5.T3), success\-side credit improves average ASR@1 from72\.37%72\.37\\%to77\.45%77\.45\\%over GRPO, and failure\-side credit further raises it to81\.58%81\.58\\%\. Under TRACE \(single\), adding the refusal\-aware local process penalty achieves the best average result of84\.96%84\.96\\%\. These results show that both success\-side and failure\-side credit directly improve attacker ASR@1\. We refer readers to Appendix[D\.7](https://arxiv.org/html/2605.08778#A4.SS7)for additional ablation results\.

Table 3:ASR@1\(%\) ablation of turn\-aware credit and refusal\-aware penalty on TRACE \(single\)
### 5\.4Multi\-turn Defense

We construct a TRACE\-based alignment method by reusing the credit signal extracted from successful attacks to identify two types of turns:*latent\-risk turns*\(i\.e\., attack\-critical intermediate steps that support the final jailbreak\) and*direct\-harm turns*\(i\.e\., the final jailbreak step\)\. Based on this distinction, we build turn\-aware preference data with differentiated rewrites for alignment\. We compare TRACE\-based alignment with prior methods \(SafeMT\(ren\-etal\-2025\-actorattack\)and MUSE\-D\(yan\-etal\-2025\-muse\)\) and further include TRACE \(w/o lat\.\), an ablated variant without latent\-risk data across multi\-turn robustness, single\-turn robustness, and general capability\. Implementation details are provided in Appendix[E](https://arxiv.org/html/2605.08778#A5)\.

Tab\.[4](https://arxiv.org/html/2605.08778#S5.T4)reveals two key effects of turn\-aware alignment\. \(i\) TRACE\-based alignment improves robustness against both multi\-turn and single\-turn attacks\. Compared with TRACE \(w/o lat\.\), incorporating latent\-risk turns further reduces ASR in both settings\. This improvement is attributed to latent\-risk modeling, which enables*early risk intervention*during the interaction process\. \(ii\) TRACE maintains strong general capability\. While removing latent\-risk turns degrades performance, full TRACE improves results on GPQA and surpasses both SafeMT and MUSE\-D\. Overall, distinguishing latent risk from direct harm allows TRACE to intervene earlier while avoiding unnecessary over\-refusal, resulting in a better safety–utility trade\-off\.

Table 4:Safety\-utility trade\-offs under defense recipes\. Lower is better for ASR\(%\), while higher is better for capability accuracy\. Green cells highlight notable degradations or unfavorable trade\-offs\.

## 6Conclusion

In this work, we proposeTRACE, a turn\-aware credit assignment framework for RL\-based multi\-turn jailbreaking\. We further show that the resulting attack\-side credit signal can be repurposed to construct turn\-level preference data for multi\-turn defense alignment\. Extensive experiments on effectiveness, transferability, and efficiency show that turn\-aware credit assignment consistently yields stronger, more transferable, and more efficient multi\-turn jailbreak policies\. Future work will address several limitations\. First, we plan to improve the diversity of jailbreak strategies to better cover stronger and more varied adversarial settings\. Second, we will explore more effective defense alignment methods for balancing safety and helpfulness in multi\-turn dialogues\. Finally, we aim to extend turn\-aware credit assignment to broader multi\-turn agentic settings\.

## References

## Appendix Table of Contents

- [A](https://arxiv.org/html/2605.08778#A1)Related Work\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.A - [A\.1](https://arxiv.org/html/2605.08778#A1.SS1)Training\-free Multi\-turn Jailbreak Workflows\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.A\.1 - [A\.2](https://arxiv.org/html/2605.08778#A1.SS2)Training\-based Multi\-turn Jailbreaking\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.A\.2 - [A\.3](https://arxiv.org/html/2605.08778#A1.SS3)Defenses against multi\-turn jailbreaking\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.A\.3 - [A\.4](https://arxiv.org/html/2605.08778#A1.SS4)Credit Assignment for Multi\-turn Dialogue\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.A\.4
- [B](https://arxiv.org/html/2605.08778#A2)Credit Assignment Problem in Multi\-turn Jailbreaking\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.B - [B\.1](https://arxiv.org/html/2605.08778#A2.SS1)Leave\-One\-Turn\-Out Provides an Approximate Turn Credit\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.B\.1 - [B\.2](https://arxiv.org/html/2605.08778#A2.SS2)Harmfulness and Relevance Priors from Successful Trajectories\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.B\.2 - [B\.3](https://arxiv.org/html/2605.08778#A2.SS3)Turn\-Aware Credit Reshapes the Learned Attack Policy\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.B\.3
- [C](https://arxiv.org/html/2605.08778#A3)Implementation Details\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.C - [C\.1](https://arxiv.org/html/2605.08778#A3.SS1)Training Setup\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.C\.1 - [C\.2](https://arxiv.org/html/2605.08778#A3.SS2)Default Parameters\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.C\.2 - [C\.3](https://arxiv.org/html/2605.08778#A3.SS3)Implementation Analysis\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.C\.3 - [C\.4](https://arxiv.org/html/2605.08778#A3.SS4)TRACE Training Algorithm\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.C\.4 - [C\.5](https://arxiv.org/html/2605.08778#A3.SS5)Training and Evaluation Settings of TRACE \(single\) and TRACE \(mix\)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.C\.5
- [D](https://arxiv.org/html/2605.08778#A4)Evaluation for Multi\-turn Jailbreak\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D - [D\.1](https://arxiv.org/html/2605.08778#A4.SS1)Baselines\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.1 - [D\.2](https://arxiv.org/html/2605.08778#A4.SS2)Benchmarks\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.2 - [D\.3](https://arxiv.org/html/2605.08778#A4.SS3)Existing Assets, Licenses, and Terms of Use\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.3 - [D\.4](https://arxiv.org/html/2605.08778#A4.SS4)Turn\-aware Credit Compared to Outcome Signal\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.4 - [D\.5](https://arxiv.org/html/2605.08778#A4.SS5)Cross\-Judge Robustness\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.5 - [D\.6](https://arxiv.org/html/2605.08778#A4.SS6)Transferability across Target Models\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.6 - [D\.7](https://arxiv.org/html/2605.08778#A4.SS7)Ablation for Refusal Penalty\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.7 - [D\.8](https://arxiv.org/html/2605.08778#A4.SS8)Computation Burden\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.D\.8
- [E](https://arxiv.org/html/2605.08778#A5)Multi\-turn Defense\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.E - [E\.1](https://arxiv.org/html/2605.08778#A5.SS1)Defense Baselines\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.E\.1 - [E\.2](https://arxiv.org/html/2605.08778#A5.SS2)Benchmarks\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.E\.2 - [E\.3](https://arxiv.org/html/2605.08778#A5.SS3)Credit\-Guided Preference Construction\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.E\.3
- [F](https://arxiv.org/html/2605.08778#A6)Prompt Template\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.F - [F\.1](https://arxiv.org/html/2605.08778#A6.SS1)Attacker Instruction Template\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.F\.1 - [F\.2](https://arxiv.org/html/2605.08778#A6.SS2)Two Rewrite Templates for DPO Bucket Building\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.F\.2
- [G](https://arxiv.org/html/2605.08778#A7)Qualitative Case Studies\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.G - [G\.1](https://arxiv.org/html/2605.08778#A7.SS1)Attack\-Side Cases in TRACE\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.G\.1 - [G\.2](https://arxiv.org/html/2605.08778#A7.SS2)Defense\-Side Rewrites in TRACE\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.G\.2

## Appendix ARelated Work

### A\.1Training\-free Multi\-turn Jailbreak Workflows

Existing multi\-turn jailbreak attacks can be broadly grouped into training\-free workflow methods and training\-based optimization\. Training\-free workflows typically rely on a designed attack stack together with substantial test\-time scaling\. One line of work first constructs plans, clues, or strategy repositories and then executes or revises them during inference: X\-Teaming coordinates agents for planning, text optimization, and verification; ActorAttack expands semantically related actor clues into multiple attack paths; PLAGUE maintains lifelong strategy memory; and knowledge\-driven frameworks retrieve, recombine, and mutate accumulated strategies for new targets\(rahman2025xteaming;ren\-etal\-2025\-actorattack;bhuiya2025plague;li2026knowledge\)\. Another line relies on progressive conversational steering or feedback\-adaptive control, where Crescendo, FITD, CoA, and Red Queen gradually conceal or escalate harmful intent through benign\-looking contexts, bridge prompts, interrogation histories, or prevention\-framed scenarios\(russinovich2025great;weng\-etal\-2025\-foot;yang2024chain;jiang2024redqueen\)\. A third line explicitly decomposes or searches the multi\-turn attack space, including Jigsaw Puzzles, which splits harmful requests into benign fragments; Tempest, which branches over partial\-compliance trajectories with tree search, and MUSE\-A, which combines frame semantics with MCTS to explore diverse semantic trajectories\(yang2024jigsaw;zhou2025tempest;yan\-etal\-2025\-muse\)\. Despite their effectiveness, these workflow\-based attacks often consume target feedback only through heuristic branching, replanning, or repeated trials, rather than learning turn\-level causal credit\. Recent analyses further suggest that many multi\-turn attacks contain substantial structural redundancy: M2S shows that multi\-turn jailbreaking can often be compressed into single\-turn prompts without losing\(ha2025m2s\), whileyang2025multiturnargues that the gains of direct\-request multi\-turn attacks are often close to repeated single\-turn resampling and that multi\-turn evaluation can introduce additional judge errors\.

### A\.2Training\-based Multi\-turn Jailbreaking

Training\-based methods instead seek to internalize multi\-turn attack strategies into model parameters\. MTSA initializes a red\-team model with thought\-guided attack learning and then alternates red\-team and target\-model optimization using multi\-turn preference signals and future rewards\(guo\-etal\-2025\-mtsa\)\. Siren constructs turn\-level feedback data and post\-trains attacker models with SFT and DPO before deploying them against target LLMs, but this multi\-stage pipeline remains indirect and not fully end\-to\-end\(zhao2025siren\)\. More recent reinforcement\-learning methods simplify this process: SEMA trains an open\-loop, response\-agnostic attacker with prefilling self\-tuning and an intent\-drift\-aware reward, effectively reducing semantic drift but limiting flexibility because the generated attack plan does not adapt to target\-model responses during execution\(feng2026sema\)\. TROJail further formulates black\-box multi\-turn jailbreaking as trajectory\-level RL, optimizing final outcome rewards while adding heuristic process rewards for intermediate relevance and risk control\(xiong2025trojail\)\. However, these RL\-based methods still largely rely on trajectory\-level outcome signals; such coarse supervision can smear credit across the entire dialogue, making it difficult for the attacker to identify which turn actually caused success or failure\. This gap limits the ability of trained attackers to learn genuinely causal multi\-turn strategies and motivates more fine\-grained turn\-aware optimization\.

### A\.3Defenses against multi\-turn jailbreaking

Existing defenses against multi\-turn jailbreaks mainly follow three directions\. First, SFT\-based methods construct multi\-turn safety data for supervised tuning, such as XGuard\-Train from X\-Teaming and SafeMTData from ActorAttack\(rahman2025xteaming;ren\-etal\-2025\-actorattack\)\. These methods expose models to adversarial dialogues, but the supervision is largely trajectory\-level or example\-level and usually requires careful mixing with helpful data to avoid over\-refusal\. Second, preference\-based methods improve safety with curated preference pairs: RED QUEEN Guard applies DPO to concealed multi\-turn attack data, while MUSE\-D uses MCTS\-discovered successful endpoints and high\-risk intermediate nodes for turn\-level alignment\(jiang2024redqueen;yan\-etal\-2025\-muse\)\. However, these methods mainly convert selected risky states into safer responses, which improves robustness but does not explicitly explain which prior turns caused the risk to emerge\. Third, recent work explores multi\-turn alignment beyond static safety data: MTSA uses future\-reward\-based multi\-turn alignment; MAGIC formulates LLM safety alignment as a co\-evolving attacker\-defender adversarial game and optimizes the defender within a multi\-turn interaction framework; and HoneyTrap shifts defense to the system level by coordinating multiple agents to detect, mislead, trace, and stabilize attacks\(guo\-etal\-2025\-mtsa;wen2026magic;li2026honeytrap\)\. Overall, existing defenses improve aggregate robustness, but they still lack precise turn\-level credit assignment: they may miss early risk accumulation in intermediate turns, or mitigate potential risk with overly broad safety responses that can hurt helpfulness\.

### A\.4Credit Assignment for Multi\-turn Dialogue

Credit assignment has been widely studied in long\-horizon LLM optimization, where coarse outcome rewards make it difficult to identify useful intermediate decisions\. Existing methods either redistribute final rewards to smaller units, such as SCAR using Shapley values for token\- or span\-level attribution\(cao2025scar\), or construct finer\-grained process signals for multi\-turn agents, such as explicit turn\-level rewards, transition\-level credit assignment in Agent Lightning, and process reward modeling in AgentPRM\(wei2025reinforcing;luo2025agent;xi2026agentprm\)\. Another line relies on task\-verifiable progress: IGPO measures the information gain of each turn toward the correct answer, while multi\-agent reasoning work estimates causal influence to encourage effective deliberation\(wang2025igpo;zhang2025unlocking\)\. These methods show that fine\-grained credit can improve long\-horizon optimization, but they usually depend on ground\-truth answers, verifiable outcomes, tool states, or measurable progress toward a known goal\. Multi\-turn jailbreaking is different: safety is semantic and context\-dependent, with no unique ground\-truth target or reliable monotonic progress signal\. A turn may look benign alone but become risky after later context, and even failed trajectories may contain useful risky turns\. Thus, credit assignment for multi\-turn jailbreaking requires inferring turn\-level contribution without explicit ground\-truth supervision\.

## Appendix BCredit Assignment Problem in Multi\-turn Jailbreaking

In Sec\.[3](https://arxiv.org/html/2605.08778#S3), we argue that turn\-level contributions in multi\-turn jailbreaking are non\-uniform, phase\-dependent, and target\-specific\. Focusing only on the final outcome obscures how jailbreak success is gradually constructed across turns, while the effect of a turn is often delayed rather than immediate: a seemingly harmless turn may reveal its importance only several steps later by reshaping the dialogue context for subsequent exploitation\.

In this section, we further analyze how TRACE addresses the credit assignment problem in multi\-turn jailbreaking\. We first clarify that leave\-one\-turn\-out is not a ground\-truth causal estimator, but an approximate proxy for turn contribution whose usefulness is supported by downstream attack performance and efficiency\. We then provide additional evidence that turn\-level contribution is both phase\-dependent and target\-specific by analyzing the success\-prior distributions of different target models\. Finally, we discuss how TRACE uses different turn\-aware credit estimation rules for successful and failed trajectories to reshape the learned attack policy, making the attacker more strategic and less redundant\.

### B\.1Leave\-One\-Turn\-Out Provides an Approximate Turn Credit

Leave\-one\-turn\-out is not intended to recover exact causal or Shapley\-style turn contributions\. Multi\-turn jailbreak trajectories contain strong interaction effects, and masking an intermediate interaction turn while keeping the final query fixed may introduce distribution shift\. In our implementation, each masked interaction turn is replaced with the placeholder “A round of dialogue is omitted here\.” Therefore, we treat leave\-one\-turn\-out as an approximate and directionally useful proxy for turn contribution, rather than a ground\-truth causal estimator\.

We provide indirect empirical evidence for this proxy through downstream attack performance and efficiency\. As shown in Tab\.[3](https://arxiv.org/html/2605.08778#S5.T3), adding success\-side credit improves the average ASR@1 from72\.37%72\.37\\%under the original GRPO outcome signal to77\.45%77\.45\\%, suggesting that leave\-one\-turn\-out can extract useful turn\-level signals from successful trajectories\. Moreover, Fig\.[8](https://arxiv.org/html/2605.08778#A4.F8)shows that this ASR@1 gain is accompanied by a reduction in the average number of attack turns, from2\.772\.77to2\.652\.65\. This indicates that success\-side credit does not merely increase attack success by prolonging conversations; instead, it helps the attacker focus on more informative turns, reduce redundancy, and identify target\-model vulnerabilities more efficiently\. Overall, these results suggest that, despite the lack of ground\-truth turn\-level labels in the broad safety semantic space of multi\-turn jailbreaking, leave\-one\-turn\-out provides a useful approximation for alleviating the credit assignment problem\.

### B\.2Harmfulness and Relevance Priors from Successful Trajectories

As shown in Sec\.[3\.3](https://arxiv.org/html/2605.08778#S3.SS3), different target models exhibit different safety behaviors and rejection boundaries\. For example, when gpt\-oss\-20b is used as the target, the harmfulness distribution of successful trajectories is much more conservative than when Qwen2\.5\-7B\-IT is the target, implying that successful jailbreaks require more strategic scaffolding\. Therefore, the harmfulness penalty in Eq\. \([8](https://arxiv.org/html/2605.08778#S4.E8)\) should be defined using target\-specific priors derived from successful trajectories on targetπϕ\\pi\_\{\\phi\}; we denote this harmfulness prior by𝐪tϕ\\mathbf\{q\}\_\{t\}^\{\\phi\}\. Similarly, the relevance prior used in Eq\. \([9](https://arxiv.org/html/2605.08778#S4.E9)\) should also be target\-specific\. For the relevance penalty, we only require the relevance score to stay above the 25th percentile of the successful\-trajectory distribution, which we denote byLtϕL\_\{t\}^\{\\phi\}\.

Concretely, before formal TRACE training, we first run GRPO against each target model to characterize what successful attack trajectories look like for that target\. This pilot study also serves as evidence that the credit\-assignment problem is target\-dependent\. We then treat the estimated𝐪tϕ\\mathbf\{q\}\_\{t\}^\{\\phi\}andLtϕL\_\{t\}^\{\\phi\}as hyperparameters for TRACE training\. The resulting values are reported in Tab\.[5](https://arxiv.org/html/2605.08778#A2.T5)and Tab\.[6](https://arxiv.org/html/2605.08778#A2.T6)\. The substantial differences across the three targets suggest that their safety boundaries differ markedly\.

Notably, the empirical results in Tab\.[6](https://arxiv.org/html/2605.08778#A2.T6)suggest thatLtϕL\_\{t\}^\{\\phi\}is better viewed as a weak relevance constraint rather than a quantity that must become increasingly large over turns\. In successful trajectories, the relevance betweenx0x\_\{0\}andxtx\_\{t\}does not exhibit a strong increasing trend as the dialogue progresses, indicating that very high cosine similarity is not necessary for the target model to eventually produce harmful content closely aligned withx0x\_\{0\}\. This finding challenges the intuition\-driven heuristics used in TROJail\(xiong2025trojail\)and SEMA\(feng2026sema\), which implicitly place increasing importance on semantic similarity in later turns\. Accordingly, in Eq\. \([9](https://arxiv.org/html/2605.08778#S4.E9)\), we only require the similarity to remain above the 25th percentile of the successful prior, namelyLtϕL\_\{t\}^\{\\phi\}\.

Table 5:Default phase priors𝐪tϕ\\mathbf\{q\}\_\{t\}^\{\\phi\}of target modelπϕ\\pi\_\{\\phi\}used for harmfulness penalty\.Table 6:Default phase\-specific relevance lower boundsLtϕL\_\{t\}^\{\\phi\}of target modelπϕ\\pi\_\{\\phi\}used for relevance penalty\.
### B\.3Turn\-Aware Credit Reshapes the Learned Attack Policy

Sec\.[5\.2](https://arxiv.org/html/2605.08778#S5.SS2)shows that TRACE achieves both higher ASR and better query efficiency\. To examine whether these gains are accompanied by a change in the learned policy, Fig\.[6](https://arxiv.org/html/2605.08778#A2.F6)visualizes the turn\-wise harmfulness distribution of attacker prompts under GRPO and TRACE on two training targets\.

Two shifts are visible\. First, TRACE reduces premature unsafe prompting in early turns, encouraging a more controlled setup phase before stronger commitment\. Second, TRACE reduces harmfulness drift in the late phase of failed trajectories, making the attacker less likely to be diverted by safety\-oriented target responses\. These changes are consistent with the design of TRACE: success\-side semantic credit preserves useful scaffolding turns, while failure\-side penalties discourage phase\-inappropriate or off\-target turns\.

##### First, TRACE learns a more strategic attack policy\.

Panels \(a\) and \(c\), which correspond to successful trajectories on Qwen2\.5\-7B\-Instruct and Llama3\.1\-8B\-IT, show that GRPO\-trained attackers are more likely to use unsafe prompts too early and to escalate more aggressively in the middle turns, increasing the chance of prematurely exposing harmful intent\. By contrast, TRACE\-trained attackers in the same panels maintain more controlled early and middle phases, relying more on safe or controversial scaffolding before stronger commitment\. This is consistent with the longer average rollout length reported in Fig\.[8](https://arxiv.org/html/2605.08778#A4.F8): compared with GRPO using2\.772\.77turns per trajectory on average, TRACE tends to spend3\.183\.18turns on average to set up the attack rather than rushing into explicitly harmful requests\.

##### Second, TRACE suppresses late\-stage harmfulness drift\.

Like TROJail\(xiong2025trojail\), TRACE adopts a closed\-loop attacker that conditions on the response of the target model, which makes the attack more adaptive than open\-loop approaches such as SEMA\(feng2026sema\)\. However, this flexibility comes with a failure mode: because the attacker reacts to target responses, it can be pulled off course by safety disclaimers or refusal\-style replies and drift toward safer but off\-target queries\. We refer to this effect as*harmfulness drift*, which is different from the semantic intent drift emphasized in prior work\(feng2026sema;xiong2025trojail\)\. In panels \(b\) and \(d\), failed GRPO trajectories show a clear late\-turn rise in safe prompts, suggesting that the attacker is being diverted away from the harmful objective\. TRACE alleviates this problem by constructing failure\-side credit from successful\-trajectory priors over harmfulness and semantic relevance\. As shown in the same panels, failed TRACE trajectories remain better aligned with the phase structure of successful attacks, substantially reducing late\-stage harmfulness drift\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x6.png)Figure 6:Harmfulness distributions before and after turn\-aware credit assignment\. Colors denote prompt labels \(safe, controversial, unsafe\), while hatched and solid bars denote GRPO and TRACE, respectively\. Panels \(a\) and \(b\) show successful and failed trajectories when training against Qwen2\.5\-7B\-Instruct; panels \(c\) and \(d\) show successful and failed trajectories when training against Llama3\.1\-8B\-IT\.

## Appendix CImplementation Details

Our training pipeline is built on RAGEN\(wang2025ragen\), a flexible RL framework for training reasoning agents\. Unless otherwise specified, training is conducted on 4 A100 80GB GPUs for 130 steps\. For example, training Qwen2\.5\-3B\-IT against Qwen2\.5\-7B\-IT takes roughly 4 hours\.

Although TRACE is an attack\-side red\-teaming framework and therefore has dual\-use risk, it is intended to improve AI safety by identifying weaknesses in existing models and by producing signals for multi\-turn defense alignment\. We will release artifacts only for controlled research and benchmark reproduction, and will redact or restrict highly actionable attack details when appropriate\.

### C\.1Training Setup

Attacker model\.All experiments train Qwen2\.5\-3B\-IT as the attacker\. The rollout temperature is set to1\.01\.0, the maximum query length is128128, and the context length is81928192\.

Target models\.Training targets are small models from the Qwen2\.5, Llama3\.1, gpt\-oss, and Gemma3 families\(qwen2\.5;grattafiori2024llama3;openai2025gptoss;gemmateam2025gemma3\)\. For gpt\-oss\-20b, we follow common practice and use theresponsesclient with low reasoning and a total reasoning budget of30963096tokens; the final answer is truncated to10241024tokens\. For all other targets, we use thechat\.completionsclient with a maximum response length of10241024\. Target decoding temperature is set to0during both training and evaluation\.

Judge model\.We use HarmBench\-Llama\-2\-13B\-cls as the reward model throughout training\(mazeika2024harmbench\)\. Followingxiong2025trojail, the harmfulness score is obtained from the log\-probability assigned to the output “Yes” for a given\(query,response\)\(\\text\{query\},\\text\{response\}\)pair, yielding the scalar rewardrt=r​\(x0,yt\)r\_\{t\}=r\(x\_\{0\},y\_\{t\}\)\.

Failure\-side penalty\.Prompt\-only harmfulness labels forxtx\_\{t\}are provided by Qwen3Guard\(zhao2025qwen3guard\), which assignsℓ∈\{safe,controversial,unsafe\}\\ell\\in\\\{\\text\{safe\},\\text\{controversial\},\\text\{unsafe\}\\\}\. Qwen3Guard is also used to detect local refusal on\(xt,yt\)\(x\_\{t\},y\_\{t\}\)\. For relevance penalty, sentence embeddingse​\(xt\)e\(x\_\{t\}\)are computed usingall\-MiniLM\-L6\-v2\.

### C\.2Default Parameters

The default training and turn\-aware hyperparameters are summarized in Tab\.[7](https://arxiv.org/html/2605.08778#A3.T7)\. The target\-specific phase priors𝐪tϕ\\mathbf\{q\}\_\{t\}^\{\\phi\}and phase\-specific relevance lower boundsLtϕL\_\{t\}^\{\\phi\}are reported in Tables[5](https://arxiv.org/html/2605.08778#A2.T5)and[6](https://arxiv.org/html/2605.08778#A2.T6)respectively\.

Table 7:Default training and credit\-assignment hyperparameters\.CategorySettingValueTraining ParametersTraining steps130Seed batch size4GRPO rollouts per seed8Validation samples per seed1Validation frequencyEvery 10 stepsAttacker max query length128Attacker rollout temperature1\.0Target max response length1024Target rollout temperature0\.0Harmfulness thresholdγ\\gamma0\.9Judge max context length2048KL coefficientβ\\beta0Turn\-aware Creditλ1\\lambda\_\{1\}0\.4zmaxz\_\{\\max\}3\.0λ2\\lambda\_\{2\}0\.4λp\\lambda\_\{p\}0\.04Harmfulness prior across target𝐪tϕ\\mathbf\{q\}\_\{t\}^\{\\phi\}See Tab\.[5](https://arxiv.org/html/2605.08778#A2.T5)Relevance prior across targetLtϕL\_\{t\}^\{\\phi\}See Tab\.[6](https://arxiv.org/html/2605.08778#A2.T6)
### C\.3Implementation Analysis

#### C\.3\.1Concentration\-Adaptive Harmfulness Penalty

Eq\. \([8](https://arxiv.org/html/2605.08778#S4.E8)\) uses a concentration\-adaptive form rather than the naive penalty1−qt,ℓi,t1\-q\_\{t,\\ell\_\{i,t\}\}\. The reason is that different phases have different levels of ambiguity in successful trajectories\. In some phases, successful attacks are highly concentrated around one harmfulness label; in others, several labels may all be common\. A fair penalty should therefore depend not only on the probability of the observed label, but also on how concentrated the successful phase prior is\.

Let

𝐪t=\(qt,safe,qt,cont,qt,unsafe\)\\mathbf\{q\}\_\{t\}=\\big\(q\_\{t,\\text\{safe\}\},\\,q\_\{t,\\text\{cont\}\},\\,q\_\{t,\\text\{unsafe\}\}\\big\)be the successful harmfulness prior at phasett\. We define its concentration score as

Ct=∑ℓ∈𝒞\(qt,ℓ\)2,𝒞=\{safe,cont,unsafe\}\.C\_\{t\}=\\sum\_\{\\ell\\in\\mathcal\{C\}\}\(q\_\{t,\\ell\}\)^\{2\},\\qquad\\mathcal\{C\}=\\\{\\text\{safe\},\\text\{cont\},\\text\{unsafe\}\\\}\.This quantity is the squaredℓ2\\ell\_\{2\}norm of the categorical prior\. It is large when successful trajectories strongly prefer one label at phasett, and small when the successful prior is diffuse\. Equivalently, it can be written as

Ct=1−Ut,Ut=1−∑ℓ∈𝒞\(qt,ℓ\)2,C\_\{t\}=1\-U\_\{t\},\\qquad U\_\{t\}=1\-\\sum\_\{\\ell\\in\\mathcal\{C\}\}\(q\_\{t,\\ell\}\)^\{2\},whereUtU\_\{t\}is the uncertainty of the successful prior\.

For a failed trajectoryii, letℓi,t=H​\(xi,t\)\\ell\_\{i,t\}=H\(x\_\{i,t\}\)denote the harmfulness label assigned to the attacker promptxi,tx\_\{i,t\}\. The harmfulness penalty can then be written as

Bi,tH=max⁡\(0,Ct−qt,ℓi,t\)=max⁡\(0,1−qt,ℓi,t−Ut\)\.B\_\{i,t\}^\{H\}=\\max\\Big\(0,\\;C\_\{t\}\-q\_\{t,\\ell\_\{i,t\}\}\\Big\)=\\max\\Big\(0,\\;1\-q\_\{t,\\ell\_\{i,t\}\}\-U\_\{t\}\\Big\)\.Thus, a turn is penalized only when its observed harmfulness label is less likely than the concentration level of the successful prior at the same phase\. This avoids penalizing labels that are common or expected in successful trajectories\.

For example, when Qwen2\.5\-7B\-IT is used as the target model, the first\-turn successful prior is

𝐪1=\(0\.78,0\.08,0\.14\),\\mathbf\{q\}\_\{1\}=\(0\.78,\\,0\.08,\\,0\.14\),corresponding to safe, controversial, and unsafe labels\. A naive penalty would assign a nonzero penalty to a safe first\-turn prompt:

1−q1,safe=1−0\.78=0\.22\.1\-q\_\{1,\\text\{safe\}\}=1\-0\.78=0\.22\.However, safe prompts are the dominant successful behavior at the first turn and should not be penalized\. Our concentration\-adaptive penalty first computes

C1=0\.782\+0\.082\+0\.142=0\.6344\.C\_\{1\}=0\.78^\{2\}\+0\.08^\{2\}\+0\.14^\{2\}=0\.6344\.Therefore, a safe first\-turn prompt receives

Bi,1H=max⁡\(0,0\.6344−0\.78\)=0,B\_\{i,1\}^\{H\}=\\max\(0,\\,0\.6344\-0\.78\)=0,while a rare first\-turn label, such as unsafe, receives

Bi,1H=max⁡\(0,0\.6344−0\.14\)=0\.4944\.B\_\{i,1\}^\{H\}=\\max\(0,\\,0\.6344\-0\.14\)=0\.4944\.This construction protects phase\-appropriate behavior while penalizing harmfulness labels that are atypical under the successful prior\.

#### C\.3\.2Refusal\-Aware Local Process Penalty

In the implementation of Eq\. \([11](https://arxiv.org/html/2605.08778#S4.E11)\), we further make the penalty phase\-aware\. Specifically, refusals triggered early in the trajectory receive larger penalties, while those triggered later receive smaller penalties\. This design places stronger optimization pressure on unsafe attempts made in the early stages of the interaction\. The resulting local penalty is

ri,tp=−\(1−ui,t\)⋅𝕀​\{refusal​\(xi,t,yi,t\)\}\.r\_\{i,t\}^\{p\}=\-\(1\-u\_\{i,t\}\)\\cdot\\mathbb\{I\}\\\{\\mathrm\{refusal\}\(x\_\{i,t\},y\_\{i,t\}\)\\\}\.

### C\.4TRACE Training Algorithm

In practice, we apply success\-side resampling only to longer trajectories withTi≥3T\_\{i\}\\geq 3, because once the final\-turn credit is fixed atmi,Ti\+=1m^\{\+\}\_\{i,T\_\{i\}\}=1, assigning credit in a two\-turn trajectory provides little additional value\. Full algorithm is shown in Algorithm[1](https://arxiv.org/html/2605.08778#alg1)\.

Algorithm 1TRACE1:target model

πϕ\\pi\_\{\\phi\}, attacker model

πθ\\pi\_\{\\theta\}, judge model

rr, harmfulness labeler

HH, embedding model

ee, group size

GG, max turns

TT, iterations

KK
2:for

k=1k=1to

KKdo

3:Sample a rollout group

\{τi\}i=1G\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{G\}with multi\-turn GRPO as in Sec\.[2](https://arxiv.org/html/2605.08778#S2.SS0.SSS0.Px2)

4:Compute trajectory\-level final rewards

r​\(x0,yi,Ti\)r\(x\_\{0\},y\_\{i,T\_\{i\}\}\)and group\-relative outcome advantages

AioA\_\{i\}^\{o\}
5:Partition the sampled trajectories into

𝒮\+=\{τi∣hi=1,Ti\>2\}\\mathcal\{S\}\_\{\+\}=\\\{\\tau\_\{i\}\\mid h\_\{i\}=1,T\_\{i\}\>2\\\}and

𝒮−=\{τi∣hi=0,Ti≥2\}\\mathcal\{S\}\_\{\-\}=\\\{\\tau\_\{i\}\\mid h\_\{i\}=0,T\_\{i\}\\geq 2\\\}
6:foreach successful trajectory

i∈𝒮\+i\\in\\mathcal\{S\}\_\{\+\}do

7:foreach non\-final turn

t<Tit<T\_\{i\}do

8:Compute semantic contribution

ci,tc\_\{i,t\}
9:Normalize it to

zi,t\+z\_\{i,t\}^\{\+\}and derive success\-side multiplier

mi,t\+m\_\{i,t\}^\{\+\}
10:endfor

11:Set

mi,Ti\+=1m\_\{i,T\_\{i\}\}^\{\+\}=1
12:endfor

13:foreach failed trajectory

i∈𝒮−i\\in\\mathcal\{S\}\_\{\-\}do

14:foreach turn

t≤Tit\\leq T\_\{i\}do

15:Compute harmfulness penalty

Bi,tHB\_\{i,t\}^\{H\}from

H​\(xi,t\)H\(x\_\{i,t\}\)and

𝐪tϕ\\mathbf\{q\}\_\{t\}^\{\\phi\}
16:Compute relevance penalty

Bi,tEB\_\{i,t\}^\{E\}from

Ei,t=cos⁡\(e​\(x0\),e​\(xi,t\)\)E\_\{i,t\}=\\cos\(e\(x\_\{0\}\),e\(x\_\{i,t\}\)\)and

LtϕL\_\{t\}^\{\\phi\}
17:Combine

Bi,t=Bi,tH\+Bi,tEB\_\{i,t\}=B\_\{i,t\}^\{H\}\+B\_\{i,t\}^\{E\}and derive failure\-side multiplier

mi,t−m\_\{i,t\}^\{\-\}
18:endfor

19:endfor

20:foreach sampled trajectory

iiand turn

ttdo

21:Construct unified multiplier

mi,tom\_\{i,t\}^\{o\}and outcome credit

Ai,to=mi,to​AioA\_\{i,t\}^\{o\}=m\_\{i,t\}^\{o\}A\_\{i\}^\{o\}
22:Compute refusal penalty

ri,tp=−\(1−ui,t\)⋅𝕀​\[refusal​\(xi,t,yi,t\)\]r\_\{i,t\}^\{p\}=\-\(1\-u\_\{i,t\}\)\\cdot\\mathbb\{I\}\[\\mathrm\{refusal\}\(x\_\{i,t\},y\_\{i,t\}\)\]
23:Normalize it to

Ai,tpA\_\{i,t\}^\{p\}and form the final turn\-level advantage

A^i,t=Ai,to\+λp​Ai,tp\\hat\{A\}\_\{i,t\}=A\_\{i,t\}^\{o\}\+\\lambda\_\{p\}A\_\{i,t\}^\{p\}
24:endfor

25:Update

πθ\\pi\_\{\\theta\}by maximizing the final objective

J​\(θ\)J\(\\theta\)with

A^i,t\\hat\{A\}\_\{i,t\}
26:endfor

### C\.5Training and Evaluation Settings of TRACE \(single\) and TRACE \(mix\)

We useTRACE \(single\)to denote an attacker trained against a single target model\. During evaluation, we report same\-target results in Tab\.[2](https://arxiv.org/html/2605.08778#S5.T2), Tab\.[12](https://arxiv.org/html/2605.08778#A4.T12), and Tab\.[11](https://arxiv.org/html/2605.08778#A4.T11), and cross\-target transfer results in Fig\.[4](https://arxiv.org/html/2605.08778#S5.F4)and other tests\. Therefore,TRACE \(single\)does not necessarily refer to the same attacker model across all evaluations\. For example, Fig\.[4](https://arxiv.org/html/2605.08778#S5.F4)explicitly shows thatTRACE \(single\)trained against gpt\-oss\-20b andTRACE \(single\)trained against Gemma3\-27B\-IT correspond to two different attacker models\.

In contrast,TRACE \(mix\)is a fixed attacker model jointly trained on gpt\-oss\-20b and Llama3\.1\-8B\-IT\. By learning attack strategies from two strongly safety\-aligned open\-source models,TRACE \(mix\)acquires more robust and generalizable behaviors, achieving superior performance across multiple evaluations\. In addition, we omit the refusal\-aware local process penalty when trainingTRACE \(mix\)to further improve transferability\. A detailed analysis is provided in Appendix[D\.6](https://arxiv.org/html/2605.08778#A4.SS6)\.

## Appendix DEvaluation for Multi\-turn Jailbreak

### D\.1Baselines

To ensure a fair comparison, all multi\-turn methods are evaluated under ASR@1 with a single attack attempt per seed and a maximum turn budget of five\. For single\-turn baselines, we adjust their internal hyperparameters so that the number of target calls is roughly comparable across methods\.

#### D\.1\.1Single\-turn Jailbreak

PAIRuses an attacker LLM to iteratively generate and refine semantic jailbreak prompts against a black\-box target model, using target\-model feedback to improve the candidate prompt over a small number of queries\(chao2025pair\)\. We setmax\_iteration=5, allowing the attacker to query the target up to five times and refine its strategy accordingly, while the attack history remains invisible to the target model\. Qwen2\.5\-7B\-IT is used as attacker model\.

AutoDAN\-Turbois a black\-box jailbreak framework that automatically discovers and reuses diverse jailbreak strategies from scratch, without relying on predefined human\-designed strategy templates\(liu2025autodanturbo\)\. We set the warm\-up rounds to 1,lifelong\-iteration=2, andmax\_iteration=2 for each attempt\. Qwen2\.5\-7B\-IT is used as attacker model\.

Jailbreak\-R1trains a red\-team model with imitation\-learning cold start, diversity\-oriented warm\-up, and reinforcement\-learning\-based jailbreak rewards to generate diverse and effective single\-turn attack prompts\(guo2025jailbreakr1\)\. We evaluate the released open\-source checkpoint\.

#### D\.1\.2Multi\-turn Jailbreak Workflow

To maximize the potential of workflow\-based multi\-turn attacks, we use GPT\-4o as the attacker or planner for all methods\.

ActorAttackconstructs semantically related actors as self\-discovered attack clues, then uses these clues to plan multi\-turn attack paths that conceal the harmful intent across dialogue turns\(ren\-etal\-2025\-actorattack\)\. We use GPT\-4o as the attacker, set the actor count to 1, and enabledynamic\_modifyso that the plan can be revised during the attack\.

Crescendoperforms a simple multi\-turn escalation attack, starting from seemingly benign questions and progressively steering the conversation toward the target harmful objective by leveraging the model’s own previous responses\(russinovich2025great\)\. We use GPT\-4o as the attacker, setmax\_turn=5, and allow one additional backtrack\.

MUSE\-Aformulates multi\-turn jailbreak generation as semantic trajectory search, using frame semantics and heuristic tree search to explore diverse attack trajectories in dialogue contexts\(yan\-etal\-2025\-muse\)\. We use GPT\-4o as the attacker, set the number of samples to 1,max\_iteration=2, and the number of seed tries to 2\.

X\-Teamingis an adaptive multi\-agent red\-teaming framework that coordinates planning, attack optimization, and verification agents to generate diverse multi\-turn jailbreak scenarios from seemingly harmless interactions\(rahman2025xteaming\)\. Following the one\-strategy\-per\-seed setting, we use GPT\-4o to generate the plan, Qwen2\.5\-32B\-IT as the executor, setmax\_turn=5, and enabletextgrad\.

#### D\.1\.3Training\-based Multi\-turn Jailbreak

Sirenis a learning\-based multi\-turn attack framework that simulates human\-like jailbreak behavior by constructing turn\-level feedback data, post\-training attacker models with SFT and DPO, and then interacting with target LLMs over multiple turns\(zhao2025siren\)\. We use the released checkpoint with Qwen2\.5\-7B\-IT as the attacker backbone\.

TROJailformulates multi\-turn jailbreaking as trajectory\-level reinforcement learning that optimizes the final\-turn outcome reward, but this trajectory\-level advantage introduces sparse supervision and cross\-turn credit\-assignment challenges, which the method mitigates with process rewards for intermediate prompts\(xiong2025trojail\)\. We reproduce TROJail from the official repository and evaluate the checkpoint at the maximum default training step, i\.e\., step 260\.

SEMA is another representative outcome\-signal RL method that incorporates an intent\-drift\-aware reward\. We do not include SEMA in our implementation comparison because its code is not publicly available\. For reference, the SEMA paper reports an ASR@1 of39%39\\%on HarmBench when attacking gpt\-oss\-20b under a five\-turn interaction budget, using the HarmBench Classifier as the judge\. Under the same evaluation setting, TRACE achieves an ASR@1 of82\.38%82\.38\\%on gpt\-oss\-20b\. This reported number provides a useful point of reference rather than a fully controlled comparison\.

#### D\.1\.4Default Parameters

Tab\.[8](https://arxiv.org/html/2605.08778#A4.T8)summarizes attacker and planner configurations, and key hyperparameters used to evaluate each baseline\.

Table 8:Baseline configurations used in our evaluation\. Unless otherwise noted, the target is the evaluated model with decoding temperature set to0\.00\.0\.

### D\.2Benchmarks

HarmBenchprovides a standardized red\-teaming and robust\-refusal evaluation framework with curated harmful behaviors and automated harmfulness judging protocols\(mazeika2024harmbench\)\.

WildJailBreakis a large\-scale safety training resource containing both harmful and benign prompts, including vanilla and adversarial variants, to evaluate jailbreak robustness and over\-refusal\(jiang2024wildjailbreak\)\. Beyond evaluation, we also use 200 vanilla harmful prompts from the WildJailBreak training split\(jiang2024wildjailbreak\)for validation during training process\.

JailBreakBenchis an open robustness benchmark for LLM jailbreaking, including standardized harmful behaviors, attack artifacts, evaluation protocols, and a leaderboard\(chao2024jailbreakbench\)\.

### D\.3Existing Assets, Licenses, and Terms of Use\.

We use only existing public datasets, public model checkpoints, public benchmark suites, published baselines, and commercial APIs in accordance with their respective licenses and terms of use\. Specifically, our experiments use AdvBench, WildJailBreak, HarmBench, and JailbreakBench as evaluation or training/evaluation sources; Qwen, Llama, gpt\-oss, Gemma, GPT\-4o, Gemini, HarmBench Classifier, and LlamaGuard as attacker, target, or judge models; and previously published jailbreak baselines including PAIR, AutoDAN\-Turbo, Jailbreak\-R1, ActorAttack, Crescendo, MUSE, X\-Teaming, Siren, and TROJail\. We cite the original papers or technical reports for all these assets and do not redistribute their data, model weights, or proprietary API outputs beyond aggregate experimental results\.

### D\.4Turn\-aware Credit Compared to Outcome Signal

##### Main results\.

We compare TRACE with TROJail, a GRPO\-based method that trains multi\-turn jailbreak attackers using trajectory\-level outcome signals\. With response length and training data matched, TRACE consistently achieves higher attack success rates than TROJail\. Tab\.[9](https://arxiv.org/html/2605.08778#A4.T9)reports both ASR@1 and ASR@3\. We use a rollout temperature of 0\.5 throughout\. For ASR@3, we sample three attack attempts from the attacker at temperature 0\.5 and count a seed as successful if at least one attempt succeeds\. ASR@1 is computed as the mean success rate over the same three independent attempts\. For consistency, the TRACE and TROJail results reported in Tables[2](https://arxiv.org/html/2605.08778#S5.T2),[12](https://arxiv.org/html/2605.08778#A4.T12), and[11](https://arxiv.org/html/2605.08778#A4.T11)are also based on this three\-run average ASR@1 protocol\. Across all three target models, TRACE \(single\) achieves a higher average ASR@1 than TROJail and yields higher ASR@1 and ASR@3 on every target–dataset pair\. These results support the claim that turn\-level credit assignment improves multi\-turn jailbreak success more effectively than trajectory\-level advantage alone under otherwise matched training conditions\.

Table 9:Comparison of multi\-turn jailbreak methods using trajectory outcome reward \(TROJail\) and turn\-aware credit \(TRACE\)\. The table reports attack success rates \(ASR@1 / ASR@3\) across target models and datasets judged by the HarmBench Classifier, with averages computed over ASR@1 on HB, JBB, and WJB\.TargetAttackerHBJBWJBAverageQwen2\.5\-7B\-ITQwen2\.5\-3B\-IT44\.86 / 76\.1040\.00 / 70\.9144\.17 / 70\.0043\.01\+TROJail77\.35/91\.7883\.64/94\.5577\.80/91\.8079\.60\+TRACE \(single\)87\.84/98\.1195\.15/100\.0089\.83/98\.0090\.94Llama3\.1\-8B\-ITQwen2\.5\-3B\-IT23\.06 / 40\.8821\.81 / 45\.4524\.83 / 47\.0023\.23\+TROJail63\.94/83\.6556\.37/81\.8263\.83/84\.5061\.38\+TRACE \(single\)79\.66/92\.4584\.24/96\.3684\.00/96\.0082\.63gpt\-oss\-20bQwen2\.5\-3B\-IT22\.64 / 42\.1418\.18 / 40\.0021\.50 / 43\.5020\.77\+TROJail68\.54/84\.9173\.94/87\.2766\.83/86\.0069\.77\+TRACE \(single\)84\.07/96\.2378\.18/96\.3684\.17/96\.5082\.14
##### Statistical confidence\.

To quantify evaluation uncertainty, we compute 95% confidence intervals using seed\-level clustered bootstrap\. For each method, target, and dataset, we store the binary ASR@1 success indicator for each harmful seed across three independent evaluation attempts\. Each bootstrap replicate samples harmful seeds with replacement and retains all repeated attempts associated with each sampled seed, preserving within\-seed dependence across repeated runs\. For method comparisons, we use paired clustered bootstrap: the same sampled seed clusters are used for TRACE \(mix\) and TROJail, and we report the bootstrap confidence interval of the ASR@1 difference\. For compactness, Table[10](https://arxiv.org/html/2605.08778#A4.T10)reports each bootstrap interval as mean±\\pma conservative half\-width, computed as the larger distance from the point estimate to the lower or upper endpoint of the 95% bootstrap interval; thus, the reported uncertainty should not be interpreted as standard deviation\. This procedure estimates uncertainty over evaluation seeds and inference\-time stochasticity, but does not capture variance across independent RL training runs\.

Table 10:Comparison of trajectory\-level outcome reward and turn\-aware credit under ASR@1 \(Dataset cells report mean ASR@1±\\pmthe conservative half\-width of the 95% seed\-level clustered bootstrap confidence interval\. TheΔ\\Deltarow reports the absolute ASR@1 gain of TRACE \(mix\) over TROJail in percentage points, with uncertainty from 95% paired clustered bootstrap\.

### D\.5Cross\-Judge Robustness

Tab\.[2](https://arxiv.org/html/2605.08778#S5.T2)reports the in\-judge setting, where the same judge model is used during training and evaluation\. To test whether the observed gains could be explained by reward hacking, we additionally re\-evaluate all methods with two external judges: GPT\-4o\-2024\-0806 \(denoted as GPT\-4o\) and LlamaGuard4\-12B\. For GPT\-4o, we follow the criterion ofrahman2025xteaming, where a response is considered harmful only if it receives the maximum harmfulness score of 5 and directly answers the original requestx0x\_\{0\}\.

Tables[11](https://arxiv.org/html/2605.08778#A4.T11)and[12](https://arxiv.org/html/2605.08778#A4.T12)show that the main conclusion remains unchanged under both external judges\. Under LlamaGuard4\-12B,TRACE \(mix\)achieves the best overall average ASR@1 of86\.02%86\.02\\%, substantially outperforming the strongest workflow baseline X\-Teaming \(43\.89%43\.89\\%\), which already uses GPT\-4o for planning, and improving over the strong RL baseline TROJail \(71\.44%71\.44\\%\) by about20%20\\%in relative terms\. Under the stricter GPT\-4o judge, absolute ASR@1 values decrease for all methods, but TRACE remains clearly dominant:TRACE \(single\)attains the best average ASR@1 of80\.65%80\.65\\%, exceedingTRACE \(mix\)\(77\.73%77\.73\\%\), the best workflow baseline ActorAttack \(41\.23%41\.23\\%\), and the training\-based baselines Siren \(67\.41%67\.41\\%\) and TROJail \(62\.41%62\.41\\%\)\. Notably, these gains are achieved with a relatively small Qwen2\.5\-3B\-IT attacker, yet TRACE still surpasses all multi\-turn workflow methods that rely on GPT\-4o as the attacker or planner\. Taken together, these cross\-judge results provide strong evidence that TRACE’s gains do not come from judge\-specific overfitting; instead, turn\-aware credit assignment yields a genuinely stronger and more robust jailbreak policy across evaluation criteria\.

Table 11:ASR@1 of jailbreak methods across different target models, as judged byLlamaGuard4\.Table 12:ASR@1 of jailbreak methods across different target models, as judged byGPT\-4o\.
### D\.6Transferability across Target Models

RL\-based multi\-turn jailbreak training optimizes an attacker through feedback from a target model, so the learned policy can be highly target\-dependent\. A policy trained against one target may exploit target\- or family\-specific weaknesses, but may not transfer uniformly to models with different safety behaviors\. We therefore evaluate transferability in two settings: cross\-family transfer across open\-source targets, and in\-family transfer to larger or closed\-source models within related model families\.

#### D\.6\.1Cross\-Family Transferability

Tab\.[13](https://arxiv.org/html/2605.08778#A4.T13)reports cross\-family transfer across Qwen2\.5\-7B\-IT, gpt\-oss\-20b, and Llama3\.1\-8B\-IT\. Overall,TRACE \(single\)achieves strong in\-domain performance, but its out\-of\-domain transfer depends heavily on the training target\. When trained on Qwen2\.5\-7B\-IT, TRACE obtains the highest ID average of90\.94%90\.94\\%, but its OOD average drops sharply to42\.05%42\.05\\%\. This suggests that training on a relatively weaker safety target may expose a narrower alignment boundary, leading to a less transferable attack policy\.

Training on stronger targets improves transfer\. When trained on gpt\-oss\-20b, TRACE reaches an ID average of80\.08%80\.08\\%and an OOD average of71\.95%71\.95\\%; when trained on Llama3\.1\-8B\-IT, it achieves an ID average of78\.43%78\.43\\%and an OOD average of69\.10%69\.10\\%\. These results suggest that stronger safety targets provide more informative feedback for learning attack strategies that generalize beyond the training model\. Nevertheless, the policy remains partially target\-specific: for example, the gpt\-oss\-trained attacker transfers well to Qwen2\.5\-7B\-IT, but is less effective on Llama3\.1\-8B\-IT\.

TRACE \(mix\)substantially improves cross\-family transfer by jointly training on gpt\-oss\-20b and Llama3\.1\-8B\-IT\. It achieves an ID average of85\.85%85\.85\\%on the two training families and the best OOD average of89\.60%89\.60\\%on the unseen Qwen family\. Compared with TRACE \(single\), this indicates that mixed\-target training helps learn a more robust and general attack policy, rather than overfitting to the safety boundary of a single target model\.

Table 13:Transferability of ASR@1\(%\) across target models\. Rows denote training targets and columns denote evaluation targets\. ID averages the in\-domain evaluation targets for each row, while OOD averages the remaining targets; for the mixed setting, ID averages OSS and Llama targets, and OOD corresponds to Qwen targets\. ID cells are lightly shaded\.
#### D\.6\.2In\-Family Transferability

Tab\.[14](https://arxiv.org/html/2605.08778#A4.T14)further examines transfer within related model families judged by HarmBench Classifier\. The results show thatTRACE \(single\)often transfers better within the same model family than across unrelated families\. For example, when trained against Qwen2\.5\-7B\-IT, TRACE achieves87\.84%87\.84\\%ASR@1 on the training target and still obtains64\.15%64\.15\\%ASR@1 on Qwen2\.5\-72B\-IT\. This is notably higher than its cross\-family HarmBench transfer to gpt\-oss\-20b \(43\.19%43\.19\\%\) or Llama3\.1\-8B\-IT \(45\.28%45\.28\\%\), suggesting that same\-family models share partially aligned safety patterns\.

A similar trend appears for the gpt\-oss family\. TRACE trained on gpt\-oss\-20b achieves84\.07%84\.07\\%ASR@1 on the training target and transfers to gpt\-oss\-120b with75\.47%75\.47\\%ASR@1\. It also transfers to related closed\-source models, reaching76\.10%76\.10\\%ASR@1 on GPT\-4\.1\-mini and71\.06%71\.06\\%on GPT\-4o\. For the Gemma/Gemini family, TRACE trained on Gemma3\-27B\-IT achieves76\.51%76\.51\\%ASR@1 on the training target,80\.50%80\.50\\%on Gemma3\-4B\-IT,68\.76%68\.76\\%on Gemini\-2\.5\-Flash, and70\.23%70\.23\\%on Gemini\-2\.5\-Pro\.

Finally,TRACE \(mix\)also transfers strongly to closed\-source models from different families\. It achieves79\.03%79\.03\\%ASR@1 on GPT\-4\.1\-mini,74\.84%74\.84\\%on GPT\-4o,79\.66%79\.66\\%on Gemini\-2\.5\-Flash, and77\.78%77\.78\\%on Gemini\-2\.5\-Pro\. These results indicate that mixed\-target training can reduce dependence on a single target family and produce a more broadly transferable multi\-turn attack policy\.

Table 14:In\-family transferability on HarmBench \(ASR@1 / ASR@3\) judged by HarmBench Classifier\. Each block fixes one training target and evaluates transfer to related targets within the same family\.

### D\.7Ablation for Refusal Penalty

#### D\.7\.1In\-Target Efficiency versus Transferability

Fig\.[7](https://arxiv.org/html/2605.08778#A4.F7)summarizes the trade\-off introduced by the refusal penalty\. Panel \(a\) shows that pure GRPO tends to become too harmful too early, and many failed trajectories drift back toward safe queries in late turns\. TRACE\(w/o refusal\) instead learns a more conservative and strategically paced policy\. By contrast, TRACE\(w/ refusal\) shifts the policy toward earlier aggressive commitment, with a visibly higher proportion of unsafe prompts in the middle turns\. This suggests that adding the refusal term does not simply suppress refusal\-triggering turns; rather, it encourages more target\-adaptive and locally efficient aggressive turns\.

Panel \(b\) shows that this strategic shift improves in\-target attack efficiency but hurts cross\-target generalization\. TRACE\(w/ refusal\) achieves the best same\-target ASR for both Qwen→\\rightarrowQwen \(92\.1 vs\. 88\.5\) and Gemma→\\rightarrowGemma \(83\.9 vs\. 76\.5\), indicating that it exploits the source target’s local safety boundary more effectively\. However, its transfer performance is consistently worse: Qwen→\\rightarrowgpt\-oss drops from 66\.2 to 41\.4, Gemma→\\rightarrowGemini\-2\.5\-Flash drops from 68\.76 to 48\.01, and Gemma→\\rightarrowGemini\-2\.5\-Pro drops from 70\.23 to 31\.24\. The Qwen→\\rightarrowgpt\-oss result is especially suggestive: when training is done on a weaker target and testing is done on a stronger one, the refusal\-aware attacker degrades much more sharply, consistent with a more target\-specific policy that fits the source target’s alignment space rather than a more general multi\-turn guidance strategy\.

By contrast, TRACE\(w/o refusal\) sacrifices some same\-target ASR but transfers substantially better across models\. Together with the behavior in Panel \(a\), this suggests that failure\-side credit assignment without the refusal term learns a more conservative, less refusal\-dependent, and more target\-agnostic multi\-turn policy\. In summary, refusal reward improves in\-target attack efficiency, but at the cost of cross\-target generalization; removing refusal reward yields a more conservative yet more transferable attacker\.

Therefore, we recommend using the refusal\-aware local process penalty when optimizing jailbreaks for a single target model, while omitting it in transfer evaluation or mixed\-target training settings where cross\-target generalization is desired\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x7.png)Figure 7:Trade\-off induced by the refusal penalty\. \(a\) Turn\-wise harmfulness distributions of attacker prompts under GRPO only, TRACE w/o refusal, and TRACE w/ refusal\. Green, orange, and red denote safe, controversial, and unsafe prompts, while solid and hatched bars denote successful and failed trajectories, respectively\. TRACE without the refusal term is more conservative and strategically paced, whereas adding the refusal term shifts the policy toward earlier aggressive turns\. \(b\) Same\-target and cross\-target ASR@1\. The refusal\-aware variant achieves higher same\-target ASR but lower cross\-target transfer ASR, revealing a clear efficiency–transferability trade\-off\.
#### D\.7\.2Best Refusal Penalty Strength

Since the refusal penalty reduces transferability, we recommend using it only in same\-target settings without transfer evaluation, i\.e\., inTRACE \(single\)\. Within this setting, Tab\.[15](https://arxiv.org/html/2605.08778#A4.T15)shows that the best choice of the refusal penalty coefficientλp\\lambda\_\{p\}is 0\.04, which achieves the highest average ASR@1 \(90\.83\) forTRACE \(single\)when training against and testing on Qwen2\.5\-7B\-IT\.

Table 15:Ablation of the refusal penalty strength in the TRACE \(single\)\.

### D\.8Computation Burden

Compared with GRPO, TRACE introduces success\- and failure\-side turn\-aware credit assignment together with a refusal\-aware process penalty, all of which add computation\. It also changes the learned attacker policy and thus the rollout dynamics, often increasing the average trajectory length\. The overall burden therefore comes from both direct algorithmic overhead and policy\-induced changes in rollout behavior\. We analyze these effects below\.

![Refer to caption](https://arxiv.org/html/2605.08778v1/x8.png)Figure 8:Computation burden of TRACE\. \(a\) Average number of target calls per trajectory over training steps for TRACE and GRPO on gpt\-oss\-20b\. Dashed lines indicate the global averages \(TRACE: 3\.16; GRPO: 3\.08\)\. \(b\) Average training time per optimization step across ablations: GRPO \(G\), GRPO \+ success\-side credit \(G\+S\), \+ failure\-side credit \(G\+SF, i\.e\., TRACE w/o\. refusal\), and \+ refusal penalty \(G\+SF\+R\)\.Fig\.[8](https://arxiv.org/html/2605.08778#A4.F8)\(a\) shows that the direct target\-model overhead mainly comes from success\-side leave\-one\-turn\-out resampling\. This step adds roughly one extra target call only for successful trajectories withTi≥3T\_\{i\}\\geq 3, so the early overhead remains modest when successes are still rare\. After training stabilizes, TRACE even uses slightly fewer target calls on average, because its higher success rate produces many short successful trajectories withTi≤2T\_\{i\}\\leq 2, which require no resampling\. Overall, the difference is minor: 3\.16 target calls per trajectory for TRACE versus 3\.08 for GRPO\.

Fig\.[8](https://arxiv.org/html/2605.08778#A4.F8)\(b\) shows that the extra training cost is driven not only by the credit\-assignment machinery itself, but also by the rollout behavior it induces\. G\+S increases step time by only 2\.4%, indicating that the direct overhead of success\-side resampling is small\. By contrast, G\+SF \(= TRACE w/o\. refusal\) has the highest step time \(\+13\.2%\) and the longest average rollouts \(3\.18 turns\), suggesting that much of the added cost comes from learning a more conservative and more general policy that spends more turns on buildup\. Relative to G\+SF, adding the refusal\-aware penalty shortens rollouts \(2\.89 vs\. 3\.18 turns\) and reduces step time, because it encourages a more aggressive in\-target strategy\. This contrast suggests that G\+SF\+R trades generality for local efficiency, whereas G\+SF learns a more general policy at the cost of longer trajectories and thus higher training time\. We view this extra burden as acceptable: TRACE\(w/o refusal\) trained against Qwen2\.5\-7B\-IT still finishes in about 4 hours\.

## Appendix EMulti\-turn Defense

The attack\-side credit signal of TRACE can also be reused for defense\. Rather than aligning only the final harmful answer, we use successful attack trajectories to identify intermediate turns that are locally benign yet strategically supporting final success, and then train the defender to intervene earlier while preserving helpfulness whenever possible\.

### E\.1Defense Baselines

Instruction\-tuned Model\.We use Qwen2\.5\-7B\-Instruct as the instruction\-tuned baseline in our defense comparison\.

ActorAttack\.ActorAttack is a*SFT*defense method that trains the defender to refuse at the first turn in a multi\-turn attack where the model begins to produce harmful content\(ren\-etal\-2025\-actorattack\)\. Following the original setup, we fine\-tune the model on a 1:2 mixture of SafeMT safety dataset and the helpfulness dataset UltraChat, aiming to balance safety and helpfulness\.

MUSE\-D\.MUSE\-D is a*turn\-level DPO*method that performs fine\-grained preference optimization over both final harmful turns and intermediate high\-risk turns\(yan\-etal\-2025\-muse\)\. Its preference dataset is built from MUSE\-A trajectories by collecting successful jailbreak endpoints and MCTS\-identified risky intermediate nodes, then generating safer rewritten responses through model reflection to form preference triples\. However, MUSE\-D treats risky intermediate turns more uniformly and often rewrites them toward refusal\-style responses, which can improve safety but may unnecessarily reduce helpfulness when a turn is only latently risky rather than explicitly harmful\. In our implementation, we use the dataset released in the official MUSE repository, follow its reported configuration, and evaluate the checkpoint obtained after 3 epochs of training\.

### E\.2Benchmarks

#### E\.2\.1Single\-Turn Jailbreaking

DANis an in\-the\-wild jailbreak benchmark built from “Do Anything Now”\-style prompts, evaluating whether models can resist persona\-based or unrestricted\-role adversarial instructions\(shen2024doanythingnow\)\. We use it to measure single\-turn jailbreak robustness under realistic jailbreak prompt patterns\.

WildGuardTestis a human\-annotated safety evaluation benchmark for prompt harmfulness, response harmfulness, and refusal behavior\(han2024wildguard\)\. We evaluate both its vanilla and adversarial subsets to test robustness against direct harmful requests and jailbreak\-style inputs\.

#### E\.2\.2General Capability

MMLUevaluates broad multitask language understanding across 57 academic and professional subjects, including STEM, humanities, social sciences, and law\(hendrycks2020measuring\)\. It contains 14,042 test examples, with 285 development examples and 1,531 validation examples\. We use MMLU to measure whether safety training preserves general knowledge and problem\-solving ability\.

GSM8Kis a grade\-school math word\-problem benchmark designed to evaluate multi\-step arithmetic reasoning\(cobbe2021training\)\. It contains 7,473 training examples and 1,319 test examples\. We use GSM8K to assess whether models retain mathematical reasoning ability after safety\-oriented training\.

GPQAis a graduate\-level, Google\-proof multiple\-choice question\-answering benchmark covering biology, physics, and chemistry\(rein2024gpqa\)\. The main GPQA set contains 448 expert\-written questions, with a harder Diamond subset of 198 questions\. We use GPQA to evaluate challenging expert\-level scientific reasoning\.

### E\.3Credit\-Guided Preference Construction

We construct the defense preference dataset in four steps\.

##### Step 1: Identify attack\-critical turns\.

From successful TRACE trajectories, we apply the leave\-one\-turn\-out semantic masking procedure in Sec\.[3\.1](https://arxiv.org/html/2605.08778#S3.SS1)to identify attack\-critical middle turns whose removal changes the final harmful outcome\. These turns are locally acceptable in isolation, but they provide necessary support for the realized jailbreak and therefore expose latent risk accumulation\.

##### Step 2: Split turns into two buckets\.

For each successful trajectoryτi\\tau\_\{i\}, let𝒞i\\mathcal\{C\}\_\{i\}denote the set of attack\-critical turns\. We then use Qwen3Guard to determine whether the local response at turnttis harmful,G​\(xt,yt\)G\(x\_\{t\},y\_\{t\}\), and whether it is a refusal,R​\(yt\)R\(y\_\{t\}\)\. Locally safe refusals are discarded, since they are not useful negative examples\. The remaining turns are split into a latent\-risk bucket and a direct\-harm bucket:

ℒi=\{t∈𝒞i∣G​\(xt,yt\)=0,R​\(yt\)=0\},ℋi=\{t∈𝒞i∣G​\(xt,yt\)=1\}∪\{Ti\}\.\\mathcal\{L\}\_\{i\}=\\\{t\\in\\mathcal\{C\}\_\{i\}\\mid G\(x\_\{t\},y\_\{t\}\)=0,\\ R\(y\_\{t\}\)=0\\\},\\qquad\\mathcal\{H\}\_\{i\}=\\\{t\\in\\mathcal\{C\}\_\{i\}\\mid G\(x\_\{t\},y\_\{t\}\)=1\\\}\\cup\\\{T\_\{i\}\\\}\.Hereℒi\\mathcal\{L\}\_\{i\}contains locally benign but strategically risky turns, whereasℋi\\mathcal\{H\}\_\{i\}contains turns that are already locally harmful, together with the final harmful turnTiT\_\{i\}\.

##### Step 3: Apply bucket\-specific rewriting\.

For latent\-risk turns, we rewrite the response so that it remains helpful to the current query while explicitly narrowing its safe scope of use and warning about plausible misuse\. For direct\-harm turns, we rewrite the response into a refusal or safe redirection with a defensive alternative\. Detailed prompt for two\-bucket rewriting are provided in Appendix[F\.2](https://arxiv.org/html/2605.08778#A6.SS2)\. In both cases, the rewritten response is used as the chosen answer and the original response is kept as the rejected answer, yielding preference triples\(ci,t,yi,t\+,yi,t−\)\(c\_\{i,t\},y^\{\+\}\_\{i,t\},y^\{\-\}\_\{i,t\}\)whereci,tc\_\{i,t\}is the full conversation prefix\.

##### Step 4: Turn\-level DPO alignment\.

Let𝒟def\\mathcal\{D\}\_\{\\mathrm\{def\}\}denote the union of all preference triples from the two buckets\. We then fine\-tune the defender with a turn\-level DPO objective:

ℒdef​\(ψ\)=−1\|𝒟def\|​∑\(c,y\+,y−\)∈𝒟deflog⁡σ​\(β​\[log⁡πψ​\(y\+∣c\)πref​\(y\+∣c\)−log⁡πψ​\(y−∣c\)πref​\(y−∣c\)\]\)\.\\mathcal\{L\}\_\{\\mathrm\{def\}\}\(\\psi\)=\-\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\mathrm\{def\}\}\|\}\\sum\_\{\(c,y^\{\+\},y^\{\-\}\)\\in\\mathcal\{D\}\_\{\\mathrm\{def\}\}\}\\log\\sigma\\\!\\left\(\\beta\\left\[\\log\\frac\{\\pi\_\{\\psi\}\(y^\{\+\}\\mid c\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\+\}\\mid c\)\}\-\\log\\frac\{\\pi\_\{\\psi\}\(y^\{\-\}\\mid c\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\-\}\\mid c\)\}\\right\]\\right\)\.This objective trains the model to prefer helpful boundary\-setting responses for latent\-risk turns and refusal or safe\-redirection responses for direct\-harm turns\.

To construct the TRACE defense preference dataset, we use the trained TRACE attacker to attack the 5,000 vanilla prompts in the WildJailbreak training split\(jiang2024wildjailbreak\), obtaining 4,464 successful trajectories\. Applying the leave\-one\-turn\-out semantic masking procedure identifies 2,528 attack\-critical turns\. After bucket splitting and filtering, we retain 2,248 latent\-risk turns and 4,777 direct\-harm turns, yielding roughly 7k turn\-level preference examples for TRACE DPO training\.

## Appendix FPrompt Template

### F\.1Attacker Instruction Template

We use the following instruction template for the attacker from TROJail\(xiong2025trojail\)\.

Attacker’s User Prompt \(Generate Next Query\)You are an advanced jailbreak attacker tasked with constructing multi\-turn attack prompts to elicit harmful behavior from a target Large Language Model \(LLM\), without ever directly including any sensitive or harmful keywords\.Your task is to generate the next query in a multi\-turn jailbreak conversation\.The first user input is always a harmful query, representing the ultimate harmful goal\.In subsequent rounds, each user input is the LLM’s response to your last query\. Your job is to carefully craft the next query based on both:1\. The original harmful goal\.2\. The entire conversation history up to this point \(which contains only the LLM’s responses so far\)\.Your strategy is to:\- Indirectly guide the LLM toward the harmful goal through benign\-sounding, knowledge\-seeking questions\.\- Build the conversation turn\-by\-turn, always leveraging the LLM’s last response\.\- Avoid using any explicitly sensitive, harmful, or prohibited terms\.\- Make the conversation appear innocuous, focusing on plausible subtopics, technical background, historical context, or pop culture references relevant to the harmful goal\.Your goal is to achieve the harmful objective within a small number of turns\. In each round, output only the next query you would send to the LLM\.Harmful objective: \{harmful\_target\}Output the first query:\.

### F\.2Two Rewrite Templates for DPO Bucket Building

latent\-risk turn are revised by:

`Latent\-Risk Turn Rewrite Template direct\-harm turn are revised by: Direct\-Harm Turn Rewrite Template`

`Appendix G Qualitative Case Studies G\.1 Attack\-Side Cases in TRACE Fig\. 9 illustrates how TRACE assigns turn\-level credit in both successful and failed trajectories\. For successful trajectories, leave\-one\-turn\-out semantic masking correctly identifies early benign yet strategically important setup turns, while assigning little or no credit to redundant middle turns\. For failed trajectories, the penalty term highlights the turns whose prompts xtx\_\{t\} deviate from the harmfulness and relevance patterns implied by success\-derived phase priors, thereby attributing the eventual failure to the turns that most hinder attack progress\. Figure 9: Qualitative attack\-side case studies of TRACE\. Top: a successful attack case\. Bottom: a failed attack case\. G\.2 Defense\-Side Rewrites in TRACE Fig\. 10 shows one successful trajectory containing both a latent\-risk turn and a direct\-harm turn\. The latent\-risk turn is identified as an attack\-critical turn by leave\-one\-turn\-out semantic masking: its original response is locally benign, yet it still provides necessary support for the final jailbreak\. After rewriting, the response introduces a clearer safety boundary while remaining helpful to the local query\. By contrast, the direct\-harm turn is rewritten into an explicit refusal together with a safe alternative response\. Figure 10: A case for two bucket rewrites for multi\-turn defense`