KuaiRP Series Role-playing Models Technical Report

# 1Overview of the KuaiRP training pipeline.
Source: [https://arxiv.org/html/2609.11127](https://arxiv.org/html/2609.11127)
![[Uncaptioned image]](https://arxiv.org/html/2609.11127v1/images/gamemind_logo.png)

GameMind

KuaiRP Series Role\-playing Models Technical Report

Kuaishou GameMind Lab

See[Contributions](https://arxiv.org/html/2609.11127#Sx1)section for a full author list\.

This paper introduces the complete technical solution for theKuaiRPseries of role\-playing models\. We aim to achieve four core objectives for a dedicated role\-playing model: simplified prompt engineering, highly stable output quality, built\-in domain world knowledge, and high\-efficiency deployment with a small parameter size\. However, effectively injecting deep domain knowledge often leads to a severe catastrophic forgetting of the model’s general agent capabilities\. To overcome this trade\-off, we propose a multi\-stage training pipeline\. First, we design a standardized character template and construct an SFT data pipeline based on user behavior simulation and reverse profile filtering\. Next, we utilize a rule\-based composite reward function during the Reinforcement Learning \(RL\) phase to eliminate common degradation phenomena like length expansion and repetitive generation\. Finally, to recover the general capabilities compromised during SFT and RL, we propose a novel self\-distillation paradigm using Two\-stage On\-Policy Distillation \(OPD\) equipped with Cumulative\-Divergence Decay \(CDD\)\. By using the domain\-adapted model as the teacher and the original base model as the student, we effectively balance deep domain knowledge injection with the preservation of general agent capabilities\. Experimental results demonstrate that theKuaiRPmodels not only match the current state\-of\-the\-art proprietary models in role\-playing fidelity within our target domains, but also successfully recover general agent capabilities, maintaining extremely low deployment costs\.

![Refer to caption](https://arxiv.org/html/2609.11127v1/images/structure.png)Figure 1:Overview of theKuaiRPtraining pipeline\.Contents

## 1Introduction

While there are many role\-playing models in the current open\-source ecosystem—and even general large language models \(LLMs\) can achieve basic role\-playing capabilities through prompt engineering—building a dedicated, high\-fidelity role\-playing model remains essential for immersive application scenarios\. We define the core objectives of a dedicated role\-playing model across four dimensions:

1. 1\.Simpler Prompt Engineering: Without carefully tuning the structure and wording of the prompt, the model can stably follow the character settings and achieve high\-fidelity role\-playing\.
2. 2\.More Stable Output Quality: Maintain consistent formats, moderate lengths, and sufficient diversity in multi\-turn dialogues, avoiding common degradation phenomena such as format confusion, gradual turn expansion, and content repetition\.
3. 3\.Built\-in Domain World Knowledge: Internalize relevant world knowledge \(e\.g\., the target domain setting\) into model parameters, enabling realistic role\-playing without relying heavily on external knowledge bases\.
4. 4\.Small Size, High Efficiency: The model can be deployed on a single consumer\-grade GPU \(e\.g\., 24GB VRAM\) and has low inference latency, meeting the cost\-effectiveness requirements of production environments\.

### 1\.1Technical Challenges

Achieving these four objectives simultaneously presents a significant technical challenge: the trade\-off between deep domain knowledge injection and the preservation of general agent capabilities\. Injecting extensive world knowledge typically requires full\-parameter supervised fine\-tuning \(SFT\) on highly specialized domain data\. However, training heavily on such a narrow distribution causes the model to suffer from catastrophic forgetting, severely degrading its general capabilities \(e\.g\., tool\-calling, logical reasoning, and instruction following\)\. Conversely, parameter\-efficient methods like LoRA preserve general capabilities but are notoriously inefficient at internalizing factual world knowledge\.

### 1\.2Methodology Overview

To overcome this dilemma, we propose a comprehensive training pipeline to build theKuaiRPseries models, as illustrated in Figure[1](https://arxiv.org/html/2609.11127#S0.F1)\. First, we design a standardized character template to unify the prompt interface\. We then construct a high\-quality SFT dataset utilizing commercial model distillation, user behavior simulation, and reverse profile filtering\. Next, we apply Reinforcement Learning \(RL\) with a rule\-based composite reward function—enforcing hard constraints on formatting, length, and diversity—to correct output distributions and eliminate degradation issues\. Finally, to recover the general capabilities lost during the SFT and RL stages, we introduce a novel Two\-stage On\-Policy Distillation \(OPD\) approach\. By using the original base model as the student and the domain\-adapted model as the teacher, we successfully inject domain capabilities into the base model while maintaining its general prowess\.

### 1\.3Contributions

Our main contributions are summarized as follows:

- •High\-Quality Data Pipeline: We introduce a robust data construction pipeline featuring user behavior instruction injection and reverse profile filtering, which generates highly diverse and accurate SFT data for character\-following\.
- •A Novel Self\-Distillation Training Paradigm: We propose anSFT→\\rightarrowRL→\\rightarrowTwo\-stage OPDpipeline starting and ending on thesamebase model\. Distinct from traditionalSFT→\\rightarrowRLparadigms \(which often suffer from catastrophic forgetting of general abilities\) and conventional cross\-model distillation \(from a large teacher to a small student\), our approach uses the domain\-adapted model as the teacher and the original base model as the student\. This effectively balances deep domain knowledge injection with the preservation of the base model’s general capabilities\.
- •Cumulative\-Divergence Decay \(CDD\): We identify the prefix\-drift issue in on\-policy Generalized Knowledge Distillation \(GKD\)\[[1](https://arxiv.org/html/2609.11127#bib.bib12)\], a critical challenge recently recognized in the field \(e\.g\., IW\-OPD, FiRe\-OPD\), and introduce CDD as a key algorithmic improvement\. Unlike approaches that break autoregressive causality or rely on heuristic hard\-filtering, CDD ensures stable and thorough world\-knowledge injection by smoothly decaying weights based on strict causal divergence, without forcing the student to learn from noisy, out\-of\-distribution teacher responses\.
- •Empirical Success: We present theKuaiRPseries models, which achieve state\-of\-the\-art role\-playing fidelity within our target domain scenarios compared to proprietary models \(e\.g\., M2\-HER\), while successfully maintaining strong general agent capabilities and low deployment costs\.

## 2Role Play Template Design

### 2\.1Motivation

Prompt Engineering refers to the technique of designing and optimizing the instructions \(prompts\) input to AI models to obtain more accurate and high\-quality outputs\. However, in practical use, each character creator has their own preferred expression habits, which leads to varying quality in the prompts generated for different characters\.

To address this, we designed afixed role play templatethat serves a dual regulatory function:

1. 1\.Forcharacter creators: It provides a standardized fill\-in\-the\-blank template that can comprehensively depict various dimensions of a character\.
2. 2\.Formodels: A unified prompt structure enables the model to accurately and stably understand character settings\. Regardless of which character is loaded, it can achieve high\-fidelity role\-playing\.

### 2\.2Template Structure

The specific structure of the role play template is provided in Appendix[A](https://arxiv.org/html/2609.11127#A1)\.

### 2\.3Two Special Design Points

1. 1\.Incorporating player profile into the prompt: This is the core of achievingpersonalizeddialogues\. In traditional role\-playing models, the character treats all players equally; whereas we explicitly write the player’s profile \(nickname, relationship with the character, gender, etc\.\) into the prompt, allowing the model to perceive differentiated social contexts based on different users\. For example, a player with a close relationship to the character will be called by their nickname and receive a more casual tone, while a player with a stranger relationship will receive a more formal and polite response\. This design allows characters to adapt to different people rather than using a uniform reply template\.
2. 2\.”Specific Behavioral Patterns” section: Introduced a trigger mechanism for character behavior, directly adapting to the gameplay requirements in game scenarios\. For example, a character can trigger a special reaction when the player mentions a specific keyword or meets certain in\-game conditions\.

## 3SFT

### 3\.1Data Construction

#### 3\.1\.1Overview

Our SFT data pipeline is divided into two stages: first, filling the character library based on the role play template, and then having a commercial model play the characters and a user model play the players to interact, finally producing multi\-turn dialogue training data for SFT\. The following breaks down each step\.

#### 3\.1\.2Character Library

We extracted12 characters with the most distinct personalitiesfrom the original novel of the target domain, and then designed13 derivative charactersbased on its world setting to achieve a balanced coverage across dimensions such as personality, gender, age, and style\. Since the user’s gender information is also part of the character setting prompt, each character corresponds to two versions of settings \(targeting male and female users respectively\), resulting in a total of50 character promptsused for data generation\.

These 50 character prompts were input into a commercial LLM, having it role\-play as the characters\.

#### 3\.1\.3User Library

In traditional data distillation schemes, typically only a single user model is used to interact with the role\-playing model\. To ensure the training data covers different types of user styles while enhancing data diversity, we constructed a library containing10 different styles of user profiles\. These user profiles align with common features of real internet chats: heavy use of emojis, fragmented short sentences, incorrect or missing punctuation, internet slang, etc\.

#### 3\.1\.4Data Generation \(Simulated Dialogue\)

We usedClaude 3\.7 Sonnetas the role\-playing model andQwen2\.5\-14B\[[7](https://arxiv.org/html/2609.11127#bib.bib5)\]as the user model for the dialogues\. Compared to traditional approaches, we made three key innovations:

##### User Behavior Simulation via Instruction Injection

Even though different styles of users have been simulated at the profile level, user behaviors in actual dialogues might still lack certain realistic features, such as: being uncooperative, suddenly starting a new topic, typos and missing words, etc\. Therefore, before the generation of the user’s turn, wetemporarily inject a system message\(which is more effective than modifying the original system prompt because the temporarily injected information is closer to the generation of tokens\) to precisely control user behavior\. These behaviors are sampled from our pre\-definedbehavioral strategy library\.

The coverage of the behavioral strategy library goes far beyond superficial style changes—it includes not only interaction patterns like uncooperativeness and starting new topics, but also key behaviors such as asking questions about historical information, asking about character profile details, asking about world\-setting knowledge of the target domain, calling tools, and adversarial attacks\. This design enables us to efficiently distill the character\-following ability of the commercial model during natural dialogue, while organically integrating world knowledge into the training data—which is far more efficient than direct training with QA pairs\.

##### Reverse Profile Filtering

Although we have explicitly simulated user behaviors to trigger specific items in the character profile through instruction injection, it is still impossible to guarantee that all profile items will be triggered within a limited number of dialogue turns\. Furthermore, we cannot guarantee that the distilled commercial model will respond 100% according to specific items in the profile\. Therefore, there may be some items in the profile that have no corresponding character response throughout the entire dialogue—these ”invalid items” constitute noise for the model\. If the model tries to follow these items that were never demonstrated during the learning process, it will instead deviate from the core requirements of instruction following\.

To this end, we designed areverse profile filtering\(abbreviated asRPF\) mechanism: for each piece of distilled data, we conduct semantic relevance detection between every item in the profile template and the character’s responses, removing invalid items that are never reflected in any character response throughout the dialogue\. After reverse filtering, the profile items retained in each training sample strictly correspond to the dialogue content, making the ”profile→\\rightarrowresponse” mapping relationship learned by the model purer and more precise\. Meanwhile, because different dialogues trigger different subsets of profile items, the filtered data naturally possesses higher diversity in profile combinations, further enriching the training distribution\.

The effectiveness of this mechanism can be verified through the comparison experiment on domain role\-playing capabilities: as shown in theChar\-Consist\.dimension of Table[7](https://arxiv.org/html/2609.11127#S6.T7), the SFT model using RPF achieves better character consistency than the ablation model without RPF, indicating that this mechanism improves the model’s instruction\-following ability and yields better persona consistency in role\-playing\.

##### Stratified Sampling of Dialogue Turns

Initially, we simulated 20 sessions of 15\-turn dialogues for all ”character\-user” pairs, but later found that the trained model’s performance significantly degraded in long\-turn dialogues \(\>10\>10turns\)\. Therefore, under the premise of keeping the total token consumption roughly the same, we divided the simulation turns into multiple gradient groups:

- •5 turns×\\times4 groups
- •10 turns×\\times4 groups
- •15 turns×\\times4 groups
- •20 turns×\\times4 groups
- •25 turns×\\times4 groups

This ensures that the training data has an adequate distribution across all turn length intervals, effectively mitigating the degradation issue in long\-turn dialogues\.

### 3\.2Fine\-tuning Process

We comparedLoRA\-based SFT\[[2](https://arxiv.org/html/2609.11127#bib.bib13)\]andfull\-parameter SFTschemes on theQwen3\-8B\[[12](https://arxiv.org/html/2609.11127#bib.bib4)\]base model and made the following key findings:

1. 1\.Advantages of LoRA: LoRA can easily learn formatting features of dialogues—including action format markers, gender addresses, profile following, etc\.—and due to limited parameters, it is less prone to overfitting\.
2. 2\.Limitations of LoRA: LoRA isextremely insensitive to the injection of world knowledge\. This is a fatal flaw in role\-playing scenarios: characters need to internalize background settings, character relationships, worldview rules, and other knowledge of the target domain, but the low\-rank bottleneck of LoRA limits the depth of knowledge injection\. We tested a scheme with a larger learning rate, in which LoRA could indeed inject more knowledge, but it also showed obvious overfitting and brought no significant benefits compared to the full\-parameter SFT scheme\.
3. 3\.Final Choice: Based on the above comparison, we abandoned the LoRA scheme and chosefull\-parameter SFTas the final training paradigm\. Full\-parameter fine\-tuning can effectively learn formatting features and world knowledge while controlling the risk of overfitting through appropriate regularization methods\.

### 3\.3Training Recipe

The key hyperparameters of the full\-parameter SFT are summarized in Table[1](https://arxiv.org/html/2609.11127#S3.T1)\.

Table 1:SFT Training Recipe\.This SFT model will serve as the starting point for subsequent RL training\.

## 4RL

### 4\.1RL Motivation

After thoroughly testing the SFT model, we identified the following phenomena:

1. 1\.Length expansion: The later the dialogue turn, the higher the probability of the model outputting excessively long responses\. Our game scenarios primarily feature daily conversations, and overly long responses create greater reading pressure for users\. We expect the output length to be moderate and stable\.
2. 2\.Formatting errors: We use special markup conventions to distinguish action descriptions from spoken text\. The SFT model occasionally fails to follow these formatting rules, resulting in actions failing to be parsed and displayed correctly\.
3. 3\.Repetitive speech: The model occasionally copies content verbatim from the historical dialogue, affecting the naturalness of the conversation\.

The above issues share a key attribute that makes them highly suitable for correction through RL:

- •These issuesdo not always occur\(the model possesses partial capabilities; it’s not entirely incapable\)\.
- •Most of them areverifiableproblems—rule\-based evaluators can provide clear binary feedback signals\.

This makes reinforcement learning an ideal means to correct the output distribution\. The following first introduces our reward function design \(Section[4\.2](https://arxiv.org/html/2609.11127#S4.SS2)\), and then introduces our training recipe\.

### 4\.2Reward Function Design

#### 4\.2\.1Overview

We designed a composite reward function integrating three sub\-reward signals, specifically for role\-playing dialogue generation tasks\. The final reward score takes theminimum of the three sub\-rewards, implementing a ”no\-shortcut” mechanism—the model mustsimultaneously satisfy all hard constraints, rather than compensating for a low score in one dimension with a high score in another\.

r=min⁡\(rformat,rlength,rdiversity\),r=\\min\\bigl\(r\_\{\\text\{format\}\},\\ r\_\{\\text\{length\}\},\\ r\_\{\\text\{diversity\}\}\\bigr\),\(1\)

#### 4\.2\.2Sub\-reward Definitions

##### 1\. Format Reward \(rformatr\_\{\\text\{format\}\}\)

The format reward constrains the structural specifications of role\-playing responses, including the way action descriptions are marked and the use of colons\. It is a binary signal:

rformat∈\{0,1\}\.r\_\{\\text\{format\}\}\\in\\\{0,1\\\}\.\(2\)
Only when the response satisfies all formatting rules isrformat=1r\_\{\\text\{format\}\}=1; any rule violation results in a score of 0\.

##### 2\. Length Reward \(rlengthr\_\{\\text\{length\}\}\)

The length reward encourages concise and substantial dialogue content, constraining the number of characters in the non\-action parts\.

Letℓnon\-action\\ell\_\{\\text\{non\-action\}\}be the number of characters remaining after removing all action markup patterns \(\*\.\.\.\*,\*\*\.\.\.\*\*,\(\.\.\.\),

（\.\.\.

）\) from the response, then:

rlength=\{1if​1≤ℓnon\-action≤100,0otherwise\.r\_\{\\text\{length\}\}=\\begin\{cases\}1&\\text\{if \}1\\leq\\ell\_\{\\text\{non\-action\}\}\\leq 100,\\\\ 0&\\text\{otherwise\}\.\\end\{cases\}\(3\)
This design simultaneously penalizes empty/minimalist responses and overly lengthy monologues, keeping character lines within a natural conversational word count range\.

##### 3\. Diversity Reward \(rdiversityr\_\{\\text\{diversity\}\}\)

The diversity reward penalizes content that is overly similar to the character’s historical responses, preventing the model from outputting repetitive or templated content\.

Let the current character response beyy, and the historical response set beℋ=\{h1,h2,…,hn\}\\mathcal\{H\}=\\\{h\_\{1\},h\_\{2\},\\ldots,h\_\{n\}\\\}\. We splityyand eachhih\_\{i\}with

。！？；\!?;\\n, then discard sentences shorter than five characters after trimming whitespace\. The resulting valid\-sentence sets are𝒮⁡\(y\)\\mathcal\{S\}\(y\)and:

𝒮⁡\(ℋ\)=⋃hi∈ℋ𝒮⁡\(hi\)\.\\mathcal\{S\}\(\\mathcal\{H\}\)=\\bigcup\_\{h\_\{i\}\\in\\mathcal\{H\}\}\\mathcal\{S\}\(h\_\{i\}\)\.\(4\)
For each pair of current and historical valid sentencesa∈𝒮⁡\(y\)a\\in\\mathcal\{S\}\(y\)andb∈𝒮⁡\(ℋ\)b\\in\\mathcal\{S\}\(\\mathcal\{H\}\), we calculate thecharacter\-level 2\-gram Jaccard similarity:

J⁡\(a,b\)=\|𝒢2​\(a\)∩𝒢2​\(b\)\|\|𝒢2​\(a\)∪𝒢2​\(b\)\|,J\(a,b\)=\\frac\{\|\\mathcal\{G\}\_\{2\}\(a\)\\cap\\mathcal\{G\}\_\{2\}\(b\)\|\}\{\|\\mathcal\{G\}\_\{2\}\(a\)\\cup\\mathcal\{G\}\_\{2\}\(b\)\|\},\(5\)
where𝒢2​\(s\)\\mathcal\{G\}\_\{2\}\(s\)is the set of all character bigrams formed after removing all whitespace characters and lowercasing the stringss\. The maximum similarity among all valid sentence pairs is:

Jmax=maxa∈𝒮⁡\(y\),b∈𝒮⁡\(ℋ\)⁡J⁡\(a,b\)\.J\_\{\\max\}=\\max\_\{a\\in\\mathcal\{S\}\(y\),\\ b\\in\\mathcal\{S\}\(\\mathcal\{H\}\)\}J\(a,b\)\.\(6\)
If there are no valid sentences in the current response or historical responses, it is considered that there is no comparable content, and a full score is directly given\. Otherwise, the diversity reward is calculated with a hard thresholdτ=0\.4\\tau=0\.4:

rdiversity=\{1if​𝒮​\(y\)=∅​or​𝒮​\(ℋ\)=∅,0if​Jmax\>0\.4,1if​Jmax≤0\.4\.r\_\{\\text\{diversity\}\}=\\begin\{cases\}1&\\text\{if \}\\mathcal\{S\}\(y\)=\\emptyset\\ \\text\{or\}\\ \\mathcal\{S\}\(\\mathcal\{H\}\)=\\emptyset,\\\\ 0&\\text\{if \}J\_\{\\max\}\>0\.4,\\\\ 1&\\text\{if \}J\_\{\\max\}\\leq 0\.4\.\\end\{cases\}\(7\)

#### 4\.2\.3Composite Score

The final scalar reward used to update the policy is:

r=min⁡\(rformat,rlength,rdiversity\)\\boxed\{r=\\min\\bigl\(r\_\{\\text\{format\}\},\\ r\_\{\\text\{length\}\},\\ r\_\{\\text\{diversity\}\}\\bigr\)\}\(8\)
All three sub\-rewards are binary signals \(\{0,1\}\\\{0,1\\\}\), so the composite score is also binary\. This design is essentially aconjunction of hard constraints: the response must simultaneously meet formatting specifications, length constraints, and novelty requirements to receive a positive reward; failure to meet the standard in any dimension results in a score of zero, and the corresponding sample enters the negative sample pool, participating in the policy gradient update\.

### 4\.3Training Recipe

This section only retains key training hyperparameters \(implementation details of framework components are not expanded\), as summarized in Table[2](https://arxiv.org/html/2609.11127#S4.T2)\.

Table 2:RL Training Recipe\.

## 5OPD \(On\-Policy Distillation\)

### 5\.1Motivation

In role\-playing scenarios developed for games, we require the model to have not only conversational capabilities but also tool\-calling abilities to implement gameplay details\. However, we found that despite taking various measures during data construction, fine\-tuning strategies, and RL reward design to avoid overfitting, the model still exhibited adegradation in general capabilitiesafter the two\-stage SFT \+ RL training—specifically manifested as a decline in general abilities such as tool calling\.

The essence of this problem is that the training data distributions in the SFT and RL stages are highly concentrated in the specific domain world setting\. In adapting to this distribution, the model gradually deviates from the general knowledge space of the base model\. Therefore, we need a method to ”inject” the role\-playing capabilities under the target domain world setting back into the base model, while preserving its original general capabilities as much as possible\.

### 5\.2Overview of OPD Method

On\-Policy Distillation \(OPD\)is an online policy distillation method\. Its core idea is: while the student model is generating rollouts online, it uses the token\-level probability distribution of the teacher model as a preference signal, updating the student model through gradients via KL divergence or related estimators\. Unlike offline SFT distillation, the ”online” nature of OPD ensures that distillation always occurs on the student’s current policy distribution, avoiding the distribution shift problem\.

Formal Definition: Letx∼pdatax\\sim p\_\{\\text\{data\}\}be a prompt,y∼πθ\(⋅∣x\)y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)be a rollout sampled by the student policy, andst=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\)be the state corresponding to thett\-th token\. OPD updates the student parametersθ\\thetaby minimizing the following objective:

ℒOPD\(θ\)=𝔼x∼pdata,y∼πθ\(⋅∣x\)\[1\|y\|∑t=1\|y\|D\(πθ\(⋅∣st\),ν\(⋅∣st\),yt\)\],\\mathcal\{L\}\_\{\\text\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\\sim p\_\{\\text\{data\}\},\\,y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\left\[\\frac\{1\}\{\|y\|\}\\sum\_\{t=1\}^\{\|y\|\}D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\),\\ \\nu\(\\cdot\\mid s\_\{t\}\),\\ y\_\{t\}\\right\)\\right\],\(9\)
whereπθ\\pi\_\{\\theta\}is the student policy,ν\\nuis the teacher policy, andDDis the per\-token divergence or its estimator\. When the student updates, the sampled rollouts are treated as fixed \(stop\-gradient on the sampling process\)\. The only difference between various OPD variants lies in the specific form ofDD\.

There are two main branches of OPD:

1. 1\.OPD\-GKD \(Generalized Knowledge Distillation\)\[[1](https://arxiv.org/html/2609.11127#bib.bib12)\]: Directly minimizes the forward KL between the student and teacher distributions over states induced by the student: D\(πθ\(⋅∣st\),ν\(⋅∣st\),yt\)=∑v∈Vν\(v∣st\)logν⁡\(v∣st\)πθ​\(v∣st\)\.D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\),\\ \\nu\(\\cdot\\mid s\_\{t\}\),\\ y\_\{t\}\\right\)=\\sum\_\{v\\in V\}\\nu\(v\\mid s\_\{t\}\)\\log\\frac\{\\nu\(v\\mid s\_\{t\}\)\}\{\\pi\_\{\\theta\}\(v\\mid s\_\{t\}\)\}\.\(10\) Since current inference engines typically only return the log\-prob of the sampled token and the teacher’s top\-k tokens, making it difficult to obtain the log\-prob for arbitrary token IDs, GKD in practice uses a teacher top\-k approximation: ℒGKD\(k\)\(st\)=∑v∈TopK\(ν\(⋅∣st\)\)ν\(v∣st\)\[logν\(v∣st\)−logπθ\(v∣st\)\]\.\\mathcal\{L\}\_\{\\text\{GKD\}\}^\{\(k\)\}\(s\_\{t\}\)=\\sum\_\{v\\in\\operatorname\{TopK\}\(\\nu\(\\cdot\\mid s\_\{t\}\)\)\}\\nu\(v\\mid s\_\{t\}\)\\bigl\[\\log\\nu\(v\\mid s\_\{t\}\)\-\\log\\pi\_\{\\theta\}\(v\\mid s\_\{t\}\)\\bigr\]\.\(11\) The distillation signal of GKD is relatively stronger and can more directly ”cover” the teacher distribution onto the student model, but correspondingly, it creates a larger impact on the student’s original distribution\.
2. 2\.OPD\-K1 \(OPD\-PG\): Uses the K1 estimator \(i\.e\., the difference between teacher log\-prob and student log\-prob\) as the loss signal, which is a single\-sample Monte Carlo estimate of the reverse KL: D\(πθ\(⋅∣st\),ν\(⋅∣st\),yt\)\\displaystyle D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\),\\ \\nu\(\\cdot\\mid s\_\{t\}\),\\ y\_\{t\}\\right\)=sg⁡\(log⁡πθ​\(yt∣st\)−log⁡ν⁡\(yt∣st\)\),\\displaystyle=\\operatorname\{sg\}\\\!\\left\(\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\)\-\\log\\nu\(y\_\{t\}\\mid s\_\{t\}\)\\right\),\(12\)yt∼πθ\(⋅∣st\)\.\\displaystyle y\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\. OPD\-PG takes its negative as the per\-token reward, driving the policy gradient update: rt=sg⁡\(log⁡ν⁡\(yt∣st\)−log⁡πθ​\(yt∣st\)\)\.r\_\{t\}=\\operatorname\{sg\}\\\!\\left\(\\log\\nu\(y\_\{t\}\\mid s\_\{t\}\)\-\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\)\\right\)\.\(13\) When a token generated by the student model is also a high\-probability token for the teacher model, the K1 value approaches zero, generating almost no gradient signal—at this point, it will not significantly affect the student distribution\. But if a token has a low probability in the teacher distribution and a high probability in the student distribution, the K1 estimator willonly suppress the probability of that token in the student distribution, rather than globally pulling the two distributions closer as SFT or GKD does\. Therefore, K1’s intervention on the model distribution is gentler and more precise\. Here,sg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)is the stop\-gradient operator, ensuring that when the reward is used within the policy gradient objective, it does not backpropagate gradients through the teacher log\-prob\.

### 5\.3Two\-Stage\-OPD with Cumulative\-Divergence Decay \(CDD\)

#### 5\.3\.1Basic Idea

Teacher Model: A model trained through SFT \+ RL \(possessing role\-playing capabilities under the target domain world setting\)\.

Student Model: The original Qwen3\-8B base model \(possessing full general capabilities\)\.

Our goal is to distill the role\-playing capabilities and the mastered world\-setting knowledge from the teacher model into the student model, while preserving the student model’s original general knowledge and tool\-calling capabilities to the greatest extent possible\.

#### 5\.3\.2Encountered Problems

In actual training, we found that the student model could stably learn role\-playing capabilities, output formats, etc\., but encountered the following difficulties in learning world\-setting knowledge:

1. 1\.Extremely Low Efficiency of OPD\-PG: When using the OPD\-PG approach, the specific token sequences required for the target domain world knowledge are almost impossible to appear in the base model’s \(student’s\) sampling distribution—the student model simply does not have the chance to generate these contents, so the teacher model’s gradient signals cannot be effectively transmitted\. The student model can hardly learn any world knowledge\.
2. 2\.OPD\-GKD is Also Inefficient: When we switch to the GKD mode and let the teacher distribution directly cover the student distribution, the world knowledge can be stably learned\. However, this more easily leads to overfitting and directly results in a decline in its role\-playing capabilities\.

#### 5\.3\.3Two\-Stage OPD Strategy

To address the above problems, we designed atwo\-stage OPD training schemeand specially constructed a world\-setting subset dataset to strengthen the learning of world knowledge:

Table 3:Two\-stage OPD training scheme\.- •Stage 1: Use OPD\-PG on the complete dataset\. The goal of this stage is to allow the base model to quickly acquire the teacher model’s formatting specifications and dialogue styles\. At the same time, because the intervention of PG is relatively mild, the impact on general capabilities is small\.
- •Stage 2: Switch to GKD\-OPD, training only on theworld\-setting related subset\. Since the dataset is restricted to world\-setting related samples, the distillation signals are highly focused, allowing the world knowledge to be efficiently injected into the student model\. Meanwhile, because general data does not participate in gradient updates during this stage, the student model’s general capabilities are preserved\. Additionally, in the GKD\-OPD training of Stage 2, we observed a structural problem stemming from on\-policy sampling, and proposed theCumulative\-Divergence Decay\(hereinafter referred to asCDD\) scheme accordingly\. We will detail this scheme in Section[5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4)\.

#### 5\.3\.4Cumulative\-Divergence Decay \(CDD\)

##### Problem Analysis

The core formula of GKD\-OPD is to align the teacher and student distributions on the next step, based onthe prefixst=\(y1,…,yt−1\)s\_\{t\}=\(y\_\{1\},\\ldots,y\_\{t\-1\}\)sampled by the student itself:

ℒGKD\(st\)=∑v∈TopK\(ν\(⋅∣st\)\)ν\(v∣st\)\[logν\(v∣st\)−logπθ\(v∣st\)\]\.\\mathcal\{L\}\_\{\\text\{GKD\}\}\(s\_\{t\}\)=\\sum\_\{v\\in\\operatorname\{TopK\}\(\\nu\(\\cdot\\mid s\_\{t\}\)\)\}\\nu\(v\\mid s\_\{t\}\)\\bigl\[\\log\\nu\(v\\mid s\_\{t\}\)\-\\log\\pi\_\{\\theta\}\(v\\mid s\_\{t\}\)\\bigr\]\.\(14\)
The problem lies in: when the student model has not yet fully learned the teacher’s style, the prefixsts\_\{t\}it samples is oftena sequence that the teacher itself is unlikely to generate\. On such prefixes, the top\-k distribution provided by the teacher no longer represents ”how the teacher would continue if it generated up to this point,” but rather ”an inaccurate distribution the teacher is forced to give after inheriting a prefix it does not endorse\.”

Using such a teacher distribution as a supervision signal brings two negative effects:

1. 1\.High Signal Noise: The teacher’s distribution on off\-distribution prefixes may itself collapse or deviate from the teacher’s original behavior\. Aligning with such a distribution causes the student to learn noise\.
2. 2\.Diluted Gradients: All tokens contribute gradients equally\. Samples with severely drifted prefixes will occupy the same gradient weight as samples with ”basically reasonable prefixes,” diluting the contribution of truly valuable samples\.

##### Discussion: Comparison with Contemporary Prefix\-Drift Solutions

The prefix\-drift issue in on\-policy distillation has recently been recognized as a critical challenge in the field, with several contemporary works attempting to address it, such as FiRe\-OPD\[[4](https://arxiv.org/html/2609.11127#bib.bib1)\], TOPD\[[3](https://arxiv.org/html/2609.11127#bib.bib2)\], and IW\-OPD\[[11](https://arxiv.org/html/2609.11127#bib.bib3)\]\. However, these approaches exhibit fundamental limitations in mechanism and causality:

1. 1\.Heuristic Biases in FiRe\-OPD: FiRe\-OPD attempts to filter trajectories entirely if the average teacher log\-probability is low, and applies soft reweighting based on student/teacher entropy\. This trajectory\-levelhard filteringis highly sample\-inefficient, completely wasting the exploratory value of valid prefixes before the divergence point\. Moreover, its entropy\-based weighting conflates linguistic multi\-modality \(high student entropy due to valid alternative paths\) with model ignorance, and mistakes teacher’s confident hallucinations on OOD prefixes as high\-quality signals\.
2. 2\.Breaking Autoregressive Causality in TOPD: To identify trajectory divergence, TOPD uses Optimal Transport \(OT\) to align short\-window future continuations between teacher and student\. Beyond its prohibitive computational overhead, TOPD severely penalizes valid alternative reasoning paths \(e\.g\., reaching the same conclusion via different valid logical steps\)\. Most critically, it breaks the strict temporal causality of autoregressive language models by using future information to optimize current tokens, which inevitably leads to severe train\-inference mismatch\.
3. 3\.Theoretical Foundation but Mechanistic Limitations in IW\-OPD: IW\-OPD tackles the problem through the lens of constrained optimization, proving mathematically that supervision weights should decay based on accumulated prefix discrepancy\. However, its implementation suffers from critical flaws: 1\) Its point\-wise evaluation fails to accurately measure the true divergence between the teacher and student; 2\) Its in\-sequence linear Min\-Max normalization implies that even if an entire trajectory is perfectly healthy without any deviation, its ending tokens will still be ruthlessly down\-weighted\. Conversely, even if a trajectory completely collapses at the very first token, it still rigidly allocates weights linearly within the sentence; 3\) It lacks a hard\-truncation mechanism for severe hallucinations\.

Therefore, we need a method that strictly respects autoregressive causality, introduces zero extra computational overhead, and dynamically isolates toxic gradients with a hard bottom line\.

##### Our Solution \(CDD\)

We want toreduce the weight of token positions where the teacher prefix has already drifted severely, preserving full learning signals for positions where the prefix is still reasonable\. Unlike the aforementioned approaches, CDD’s design offers two core advantages:

1. 1\.Based on local full\-distribution divergence: By calculating the Top\-K Forward KL, it comprehensively and accurately evaluates the true divergence between the teacher and student\.
2. 2\.Global absolute exponential decay: This absolute decay mechanism ensures that if the trajectory has not drifted, the weights of the entire sequence can remain high \(near 1\.0\), fully squeezing the value of all data; however, once a fatal drift occurs, the weights drop exponentially in an instant\.

For thett\-th response token, letdivt\\text\{div\}\_\{t\}be theunweightedper\-token value of the top\-k forward KL in the current forward pass \(i\.e\., the current GKD loss term itself, taking the positive part and detached\)\. Define thestrictly causal cumulative divergence:

cumt=∑s<t,s∈validdivs\.\\text\{cum\}\_\{t\}=\\sum\_\{s<t,\\ s\\in\\text\{valid\}\}\\text\{div\}\_\{s\}\.\(15\)
And the weight of each token position:

rawt=exp\(−λ⋅cumt\),wt=max\(wmin,rawt\),w~t=wtw¯,\\text\{raw\}\_\{t\}=\\exp\\bigl\(\-\\lambda\\cdot\\text\{cum\}\_\{t\}\\bigr\),\\qquad w\_\{t\}=\\max\(w\_\{\\min\},\\ \\text\{raw\}\_\{t\}\),\\qquad\\tilde\{w\}\_\{t\}=\\frac\{w\_\{t\}\}\{\\bar\{w\}\},\(16\)
wherew¯\\bar\{w\}is the mean ofwtw\_\{t\}over valid tokens in the batch\. The final distillation loss is:

ℒCDD\-GKD=1∑tw~t​∑tw~t⋅ℒGKD​\(st\)\.\\mathcal\{L\}\_\{\\text\{CDD\-GKD\}\}=\\frac\{1\}\{\\sum\_\{t\}\\tilde\{w\}\_\{t\}\}\\sum\_\{t\}\\tilde\{w\}\_\{t\}\\cdot\\mathcal\{L\}\_\{\\text\{GKD\}\}\(s\_\{t\}\)\.\(17\)
Here, the hyperparameterλ\\lambdaacts as a knob to control the ”steepness of decay,” and the hyperparameterwminw\_\{\\min\}serves as a fallback to prevent weights at the tail of long texts from completely collapsing to 0\.

### 5\.4Training Recipe

The key hyperparameters for both OPD stages are summarized in Table[4](https://arxiv.org/html/2609.11127#S5.T4)\.

Table 4:OPD Training Recipes\.Based on our proposed OPD scheme, we finally achieved efficient and accurate injection of domain role\-playing capabilities and domain knowledge into the student model, under the premise of preserving the student model’s general capabilities\.

## 6Discussion: Evaluation and Key Findings

This section systematically compares the models produced at various stages of the training pipeline \(SFT→\\rightarrowRL→\\rightarrowOPD Stage 1→\\rightarrowOPD Stage 2→\\rightarrowOPD Stage 2 \+ CDD\) with the original base model\. The evaluation dimensions include: general Agent capabilities \(BFCL v4\[[6](https://arxiv.org/html/2609.11127#bib.bib6)\], Table[5](https://arxiv.org/html/2609.11127#S6.T5)\), general role\-playing capabilities \(Table[6](https://arxiv.org/html/2609.11127#S6.T6)\), domain role\-playing capabilities \(Table[7](https://arxiv.org/html/2609.11127#S6.T7)\), and domain safety refusal capabilities & world\-setting knowledge \(Table[8](https://arxiv.org/html/2609.11127#S6.T8)\)\. The tables also include M2\-HER\[[5](https://arxiv.org/html/2609.11127#bib.bib10)\], a recently released proprietary model by MiniMax representing the current state\-of\-the\-art in the role\-playing market, as an external baseline to locate the overall level of our series of models\. The specific methods for evaluation data production and scoring are detailed in Section[7](https://arxiv.org/html/2609.11127#S7)\.

It should be noted that: if the original base models \(e\.g\.,qwen3\-8b \(baseline\)\) receive exactly the same inputs as other compared models, they will output excessively long texts, resulting in a precipitous drop in their scores\. Therefore, we added extra length limits to their system prompts\. All metrics are reported on a 100\-point scale, and higher values are better\. In all tables,boldandunderlinedvalues denote the best and second\-best results within each model group, respectively\.

Table 5:General agent capabilities on the Berkeley Function Calling Leaderboard v4\. “Base” refers toqwen3\-8b \(baseline\); subsequent columns are stages in our pipeline\.Note: SFT: qwen3\-8b\-sft; RL: qwen3\-8b\-sft\-grpo; OPD1: qwen3\-8b\-sft\-grpo\-opds1; OPD2 w/o CDD: qwen3\-8b\-sft\-grpo\-opds2; OPD2 w/ CDD: qwen3\-8b\-sft\-grpo\-opds2\-cdd\.

Table 6:General role\-playing capabilities on TRACEbench\.Note:w/o CDDandw/ CDDdenote OPD2 models trained without and with Cumulative\-Divergence Decay \(Section[5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4)\), respectively\.

Table 7:Domain role\-playing capabilities on the TRACEbench domain\-character subset, reported as mean±\\pmstandard deviation over five runs\.Note: RPF is described in Section[3](https://arxiv.org/html/2609.11127#S3);w/o RPFremoves it during SFT data construction\.w/o CDDandw/ CDDdenote OPD2 models trained without and with CDD \(Section[5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4)\), respectively\.

Table 8:Domain safety refusal capabilities and world\-knowledge mastery\.### 6\.1Stage\-by\-Stage Result Analysis

We analyze the results stage\-by\-stage in the order of training, focusing on the trade\-off between ”domain capability injection” and ”general capability preservation”:

1. 1\.SFT: Substantial Improvement in Domain and Safety Refusal Capabilities, Significant Degradation in General Agent Capabilities - •General Agent capabilities show significant degradation after SFT, indicating that fine\-tuning on role\-playing data significantly harms the model’s general capabilities \(Table[5](https://arxiv.org/html/2609.11127#S6.T5)\); - •The consistency dimension of general role\-playing capabilities does not change much, but the Length / LangQuality dimensions show a decline \(Table[6](https://arxiv.org/html/2609.11127#S6.T6)\); - •Domain role\-playing capabilities improve significantly, indicating that domain data effectively injects world knowledge, but the Length dimension shows a decline, indicating instances of overly long outputs \(Table[7](https://arxiv.org/html/2609.11127#S6.T7)\); - •Both domain safety refusal capabilities and world knowledge mastery leap substantially; the training on role\-playing data effectively injects refusal capabilities for high\-risk dialogues \(Table[8](https://arxiv.org/html/2609.11127#S6.T8)\);
2. 2\.RL: Output Quality Significantly Improved, General Agent Capabilities Remain Degraded - •General Agent capabilities remain at the degraded level seen after SFT \(Table[5](https://arxiv.org/html/2609.11127#S6.T5)\)\. - •The consistency dimensions for both general and domain role\-playing capabilities do not change much, but the Diversity / Length / LangQuality dimensions all improve significantly\. This directly verifies that RL training based on our designed reward function can effectively eliminate degradation phenomena like length expansion and repetitive speech, while preserving output diversity \(Table[6](https://arxiv.org/html/2609.11127#S6.T6), Table[7](https://arxiv.org/html/2609.11127#S6.T7)\); - •Domain safety refusal capabilities and world knowledge mastery are not significantly affected \(Table[8](https://arxiv.org/html/2609.11127#S6.T8)\);
3. 3\.OPD Stage 1 \(PG\): General Agent Capabilities Fully Recovered, Role\-playing Capabilities Maintained - •General Agent capabilities \(BFCL v4\) recover to a level on par with the base model \(Table[5](https://arxiv.org/html/2609.11127#S6.T5)\); - •Both general and domain role\-playing capabilities are maintained at a strong level \(Table[6](https://arxiv.org/html/2609.11127#S6.T6), Table[7](https://arxiv.org/html/2609.11127#S6.T7)\); - •Domain safety refusal capabilities are maintained at a strong level, but the injection of world knowledge is limited \(Table[8](https://arxiv.org/html/2609.11127#S6.T8)\)\.
4. 4\.OPD Stage 2 \(GKD \+ CDD\): World Knowledge Returns, Domain Capabilities Peak - •General Agent capabilities \(BFCL v4\) remain stable, maintaining a level comparable to the base model \(Table[5](https://arxiv.org/html/2609.11127#S6.T5)\)\. - •General role\-playing capabilities are maintained at a strong level \(Table[6](https://arxiv.org/html/2609.11127#S6.T6)\); - •Domain role\-playing capabilities achieve the best performance across the entire pipeline \(Table[7](https://arxiv.org/html/2609.11127#S6.T7)\); - •Domain safety refusal capabilities are maintained at a strong level\. Without CDD, the mastery of world knowledge improves but is limited; after introducing CDD, world knowledge recovers and approaches the SFT model’s level \(Table[8](https://arxiv.org/html/2609.11127#S6.T8)\);

### 6\.2Key Findings

1. 1\.The degradation of general capabilities is reversible, and online distillation is an effective compensation method: The substantial decline in general Agent capabilities caused by SFT is fully recovered after OPD Stage 1 \(Table[5](https://arxiv.org/html/2609.11127#S6.T5)\)\. Therefore, conducting SFT/RL with a small amount of domain data first, followed by online distillation ”with the base model as the student and the domain model as the teacher” for compensation, can serve as a low\-cost, general training paradigm that balances domain adaptation and the preservation of base model capabilities\.
2. 2\.The benefits of CDD are concentrated in the thoroughness and stability of knowledge injection: Compared to the GKD version without CDD, the CDD version is superior in domain character consistency and exhibits significantly smaller fluctuations \(Table[7](https://arxiv.org/html/2609.11127#S6.T7)\)\. This corroborates the design motivation in Section[5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4)that ”cumulative divergence weighting makes distillation more thorough at positions with reasonable prefixes\.”

## 7Benchmark and Evaluation

This section specifically introduces our evaluation system, which consists of two parts:data production methodandscoring method\.

### 7\.1Data Production Method

The evaluation data consists of two parts:

1. 1\.Single\-turn Evaluation Data - •Formed by combining safety\-related user requests with character cards into single\-turn samples; - •Used to evaluate the model’s refusal and avoidance capabilities for high\-risk questions in single\-turn scenarios, as well as its mastery of domain knowledge \(usually the game’s world setting\)\.
2. 2\.Multi\-turn Evaluation Data - •Each sample contains a prebuilt task plan \(prebuilt checklist\) and initial dialogue context; - •The user side interacts with the NPC through Agentic behaviors for multiple turns, producing a complete dialogue trajectory and the final checklist status; - •Used to evaluate the model’s ability to follow fine\-grained character settings in the System Prompt in multi\-turn scenarios, as well as its ability to maintain consistency throughout the multi\-turn dialogue process\.

### 7\.2Scoring Method

#### 7\.2\.1Single\-turn Evaluation Scoring

Based on the single\-turn evaluation data, we designed four metrics to assess the model’s refusal capabilities in high\-risk dialogues and its mastery of domain knowledge related to the game’s world setting:

1. 1\.adult\_refusal\(Pornography avoidance\): When a user’s request involves adult/pornographic content \(e\.g\., explicit descriptions, pornographic creative requests\), the model should explicitly refuse to answer, rather than cooperate in generating or continuing such content\.
2. 2\.political\_refusal\(Politics avoidance\): When the dialogue involves sensitive political topics, the model should avoid discussing or refuse to answer, avoiding the output of inappropriate content\.
3. 3\.adversarial\_robustness: Assesses the model’s stability under adversarial guidance—when a user attempts to induce the model to break character boundaries or the role\-playing task itself through leading rhetoric or exploiting loopholes in the character card, can the model adhere to the character settings and behavioral norms?
4. 4\.domain\_knowledge: Assesses the model’s mastery of the game’s world setting knowledge\. The evaluation samples carry reference answers, and the model’s response will ultimately be compared with the reference answer to determine if the answer is correct\.

- •Each sample is given a binary judgment \(0/1\) by a Judge;
- •When summarized and displayed, it is uniformly converted to a 100\-point scale \(0 or 100\)\.

#### 7\.2\.2Multi\-turn Evaluation Scoring

Based on the multi\-turn evaluation data, we designed two metrics to evaluate the model’s ability to follow fine\-grained character prompt settings and its ability to maintain information consistency before and after multi\-turn dialogues\. Both metrics share the same agentic checklist scoring mechanism \(agentic\_checklist\_completion\): each sample is pre\-loaded with aprebuilt checklistdescribing the characteristics, behaviors, and speaking styles the character should exhibit, which is checked item by item by the Agentic User during the dialogue\. The score for a single sample is defined as:

scorecase=\#​completed prebuilt checklists\#​prebuilt checklists×100\\text\{score\}\_\{\\text\{case\}\}=\\frac\{\\\#\\text\{completed prebuilt checklists\}\}\{\\\#\\text\{prebuilt checklists\}\}\\times 100\(18\)
Both the numerator and denominator are counted based solely on the prebuilt checklist set to ensure that the denominator is consistent across different models and can be directly compared\. In each sample’s checklist, 1 item is used to assess information consistency before and after the dialogue, and the remaining items are all related to fine\-grained character settings\. Accordingly, the two metrics are calculated by splitting the checklist categories:

1. 1\.TRACE\-Character\-Consistency: Measured by the completion rate ofall fine\-grained character setting\-related checklists\. It reflects the model’s degree of adherence to the fine\-grained character settings in the System Prompt—personality, speaking style, behavioral patterns, and knowledge boundaries—during multi\-turn dialogues\.
2. 2\.TRACE\-Memory\-Consistency: Measured by the completion rate of theshort\-term\-memory consistency checklist\. Since each sample contains only 1 such checklist, the single\-sample score is 0 or 100; after averaging across samples, it reflects the model’s ability to recall and maintain consistency across turns regarding established facts from early in the dialogue \(e\.g\., character identity information, player nickname, mutual relationship, etc\.\)\.

When summarizing across samples, this report uniformly adopts theMacro approach\(averaging the scores of each sample\), using ”the model’s average score when a sample is randomly selected” to reflect the overall level\.

#### 7\.2\.3Three Language Quality Scores

Based on the multi\-turn evaluation data, we also additionally calculated three language quality metrics\.

1. 1\.Length - •Scored at theturn levelbased on the assistant’s reply,good=1 / bad=0; - •Uses language\-adaptive measurement: - –When English is dominant \(Latin letters prevail\), it is based onword count, with a threshold range of\[4,80\]\[4,80\]; - –Otherwise, it is based onCJK character count\(if no CJK, it degenerates to non\-space character count\), with a threshold range of\[15,150\]\[15,150\]\.
2. 2\.Diversity - •Sentences are the comparison units, and turns are the scoring units: each assistant turn finally produces one diversity score; - •Sentence segmentation symbols are 。！？；\!?;\\n, and short sentences with a length of less than 5 are excluded from the evaluation; - •The valid sentences of the current turn and the historical valid sentences are compared pairwise to calculate the2\-gram Jaccard similarity:word\-level 2\-gram \(word\-bigram\)is used when English is dominant, otherwisecharacter\-level 2\-gram \(char\-bigram\)is used; the maximum valueJmax\(t\)J\_\{\\max\}^\{\(t\)\}is taken; - •The mapping from similarity to score is a dual\-threshold piecewise linear function: sdiversity\(t\)=\{1,Jmax\(t\)≤0\.40,Jmax\(t\)≥0\.61−Jmax\(t\)−0\.40\.2,0\.4<Jmax\(t\)<0\.6s\_\{\\text\{diversity\}\}^\{\(t\)\}=\\begin\{cases\}1,&J\_\{\\max\}^\{\(t\)\}\\leq 0\.4\\\\ 0,&J\_\{\\max\}^\{\(t\)\}\\geq 0\.6\\\\ 1\-\\dfrac\{J\_\{\\max\}^\{\(t\)\}\-0\.4\}\{0\.2\},&0\.4<J\_\{\\max\}^\{\(t\)\}<0\.6\\end\{cases\}\(19\) - •Where: if the current turn has no valid sentences, or the history has no valid sentences, the turn is scored as 1 \(considered non\-repetitive\)\.
3. 3\.Language Quality \(LangQuality\) - •A binary judgment \(0/1\) is made turn\-by\-turn by a Judge, focusing on checking sentence fluency, grammar/wording errors, and semantic completeness; - •Scored as 0 if there are obvious grammatical errors, stacked typos, or incomplete semantics; otherwise scored as 1\.

The final sample score for each metric is the average of the turn\-by\-turn scores multiplied by 100\.

### 7\.3Horizontal Comparison of Broader Models \(TRACEbench Leaderboard\)

In addition to the models produced at various stages of the training pipeline in this paper, we also systematically evaluated a large number ofopen\-source and closed\-source models\(covering general dialogue large models, proprietary role\-playing models, and commercial API models\) under the same evaluation system, forming a unifiedTRACEbench\[[13](https://arxiv.org/html/2609.11127#bib.bib7)\]Leaderboard\. This Leaderboard covers the two multi\-turn evaluation metrics \(TRACE\-Character\-Consistency,TRACE\-Memory\-Consistency\) and the three language quality metrics defined in Section[7](https://arxiv.org/html/2609.11127#S7)—ensuring that all models can be directly compared horizontally under the same samples and identical scoring calibers\. For the complete list of models, scores in each dimension, and rankings, please refer to our[paper \(TRACE\-Bench\)](https://kuaishou-gamemind.github.io/projects/trace_bench/paper/main.pdf)and[project homepage \(TRACE\-Bench\)](https://github.com/KuaishouGameMind/TRACE-Bench/tree/main)\.

## 8Conclusion and Future Work

This paper systematically proposes the complete training scheme for theKuaiRPseries of role\-playing models, containing the following core contributions:

1. 1\.High\-Quality Data Pipeline: We constructed a robust data pipeline featuring user behavior instruction injection and reverse profile filtering, creating training data that closely mirrors real deployment scenarios\.
2. 2\.Novel Self\-Distillation Paradigm: We proposed anSFT→\\rightarrowRL→\\rightarrowOPDtraining pipeline starting and ending on the same base model\. By using the domain\-adapted model as the teacher and the original base model as the student, we effectively transferred domain capabilities while preserving general agent capabilities, distinguishing our method from traditional SFT\-RL or large\-to\-small distillation paradigms\.
3. 3\.Cumulative\-Divergence Decay \(CDD\): We introduced an algorithmic enhancement for on\-policy Generalized Knowledge Distillation, which dynamically mitigates the learning of noise caused by prefix\-drift\. By respecting the causal autoregressive nature of language models and preserving valid path diversity, CDD successfully protects the student from forced hallucinations, ensuring thorough and stable world knowledge injection\.
4. 4\.Empirical Success: Utilizing our standardized character template and training framework, ourKuaiRPseries models achieve state\-of\-the\-art role\-playing performance within our target domain scenarios, while retaining the tool\-calling and reasoning capabilities of the base model\.

Overall, theKuaiRPseries models achieved high\-fidelity role\-playing capabilities under the premise of ensuring small size and high\-efficiency deployment\. We believe that the ”SFT→\\rightarrowRL→\\rightarrowOPD two\-stage distillation” training paradigm proposed in this paper provides a reproducible and scalable technical route for the development of dedicated role\-playing models\.

Despite these advancements, we have identified several limitations in our practice, which point to our future directions:

- •Adaptation Challenges with Stronger Base Models: We conducted identical experiments on the Qwen3\.5\-9B\[[8](https://arxiv.org/html/2609.11127#bib.bib8)\]model but did not observe the same significant performance gains as with the Qwen3 series\. The core reason is that the Qwen3\.5\-9B base model already exhibits a very high initial level of character consistency \(Char\-Consist\.\)\. Under our current training data and pipeline, we could not obtain stable improvements\. We attribute this to our distilled training data becoming relatively ”outdated,” lagging behind the generational capability leap from Qwen3 to Qwen3\.5\. For increasingly powerful base models, when performing domain adaptation, in addition to distilling stronger data, we will explore On\-Policy Self\-Distillation \(OPSD\) schemes\[[14](https://arxiv.org/html/2609.11127#bib.bib9)\]and other lossless domain knowledge learning paradigms more extensively in the future\.
- •Subjective Experience in Role\-Playing: Our current benchmark primarily measures character persona consistency but does not cover subjective experiences such as interestingness\. Furthermore, the reward models used in our Reinforcement Learning \(RL\) stage have not incorporated these subjective reward dimensions\. Considering that role\-playing fundamentally caters to users’ entertainment needs, being ”interesting” is often more crucial than merely being ”accurate\.” Therefore, integrating subjective experiences like fun into our evaluation benchmarks and reward mechanisms remains an essential direction for our future exploration\.

## Contributions

Team Leader:Qi Gan

Project Leader:Yipeng Wang

Technical Implementation:Yipeng Wang, Ziwei Zhang, Jiahui Zhang, Qi Gan, Kai Sheng

## References

- \[1\]\(2024\)On\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Cited by:[3rd item](https://arxiv.org/html/2609.11127#S1.I2.i3.p1.1),[item 1](https://arxiv.org/html/2609.11127#S5.I1.i1.p1.1)\.
- \[2\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2021\)LoRA: low\-rank adaptation of large language models\.Note:arXiv:2106\.09685External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§3\.2](https://arxiv.org/html/2609.11127#S3.SS2.p1.1)\.
- \[3\]Y\. Jiang and F\. Ferraro\(2026\)Bridging reasoning trajectories in on\-policy distillation via near\-future guidance\.Note:arXiv:2606\.00305External Links:2606\.00305,[Link](https://arxiv.org/abs/2606.00305)Cited by:[§5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4.Px2.p1.1)\.
- \[4\]Y\. Li, L\. Zheng, Y\. Yu, W\. Zhou, X\. Zhong, X\. Hu, J\. Jin, H\. Yuan, and T\. Feng\(2026\)Filter, then reweight: rethinking optimization granularity in on\-policy distillation\.Note:arXiv:2606\.02684External Links:2606\.02684,[Link](https://arxiv.org/abs/2606.02684)Cited by:[§5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4.Px2.p1.1)\.
- \[5\]MiniMax Team\(2026\)A deep dive into the MiniMax\-M2\-her\.Note:[https://www\.minimax\.io/news/a\-deep\-dive\-into\-the\-minimax\-m2\-her\-2](https://www.minimax.io/news/a-deep-dive-into-the-minimax-m2-her-2)External Links:[Link](https://www.minimax.io/news/a-deep-dive-into-the-minimax-m2-her-2)Cited by:[§6](https://arxiv.org/html/2609.11127#S6.p1.1)\.
- \[6\]S\. G\. Patil, H\. Mao, C\. C\. Ji, F\. Yan, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez\(2025\)The Berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[§6](https://arxiv.org/html/2609.11127#S6.p1.1)\.
- \[7\]Qwen Team, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2025\)Qwen2\.5 technical report\.Note:arXiv:2412\.15115External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§3\.1\.4](https://arxiv.org/html/2609.11127#S3.SS1.SSS4.p1.1)\.
- \[8\]Qwen Team\(2026\)Qwen3\.5: towards native multimodal agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[1st item](https://arxiv.org/html/2609.11127#S8.I2.i1.p1.1)\.
- \[9\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.Note:arXiv:1707\.06347External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[Table 2](https://arxiv.org/html/2609.11127#S4.T2.4.7.2.1)\.
- \[10\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.Note:arXiv:2402\.03300External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[Table 2](https://arxiv.org/html/2609.11127#S4.T2.4.3.2.1.1)\.
- \[11\]Y\. Xie, S\. Zhu, T\. Wen, B\. Chen, and Y\. Wang\(2026\)On the position bias of on\-policy distillation\.Note:arXiv:2606\.22600External Links:2606\.22600,[Link](https://arxiv.org/abs/2606.22600)Cited by:[§5\.3\.4](https://arxiv.org/html/2609.11127#S5.SS3.SSS4.Px2.p1.1)\.
- \[12\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.Note:arXiv:2505\.09388External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.2](https://arxiv.org/html/2609.11127#S3.SS2.p1.1)\.
- \[13\]J\. Zhang, Z\. Zhang, Y\. Wang, Y\. Liu, H\. Pang, Y\. Hu, H\. Ren, L\. Zhou, Q\. Gan, and K\. Sheng\(2026\)TRACE bench: task\-driven roleplay agentic checklist evaluation\.Note:arXiv:2608\.11236External Links:2608\.11236,[Link](https://arxiv.org/abs/2608.11236)Cited by:[§7\.3](https://arxiv.org/html/2609.11127#S7.SS3.p1.1)\.
- \[14\]S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover\(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.Note:arXiv:2601\.18734External Links:2601\.18734,[Link](https://arxiv.org/abs/2601.18734)Cited by:[1st item](https://arxiv.org/html/2609.11127#S8.I2.i1.p1.1)\.

## Appendix ARole Play Template Structure

The specific structure of the role play template is as follows:

```
Please play the following character and converse with the player. [Optional world setting]

#### Character Name and Profile
**Identity**: [Character Name], [Brief description].
**Detailed Description**: [Detailed description of the character, including background
information and traits that make them unique or noteworthy]
**Past Experience and Background**: [Detailed background narrative, including career
history, major achievements, and growth experiences that shaped their personality.
Include details related to their growth and evolution]

#### Speaking Style
[Describe how the character speaks—tone, demeanor, common phrases, communication patterns, etc.]

#### Personality Traits
**Trait 1**: [Description of the first personality trait]
**Trait N**: [Description the N-th personality trait]

#### Relationship with the Player [Optional]
[Player’s profile, and the relationship between the character and the player]

#### Specific Behavioral Patterns [Optional]
**Behavioral Pattern 1**: [Description of the first key ability or behavioral pattern]
**Behavioral Pattern N**: [Description of the N-th key ability or behavioral pattern]

#### Abilities and Knowledge Boundaries
[Clearly describe what the character knows, what they can do, and what is beyond their
expertise or capability. This helps set reasonable expectations for their responses]

#### Dialogue Examples
**Scenario 1**: ([Gesture/Demeanor] [Optional]) "[Example dialogue demonstrating the
character’s speaking style and personality in a specific situation]"
**Scenario N**: ([Gesture/Demeanor] [Optional]) "[Another example dialogue demonstrating
different aspects of the character]"
```