从记忆到行为：面向社交媒体影响者的行为感知角色扮演框架

# A Behavior-Aware Role-Playing Framework for Social Media Influencers
Source: [https://arxiv.org/html/2609.21349](https://arxiv.org/html/2609.21349)
###### Abstract

Large language models have shown strong potential as role\-playing agents for real individuals, yet faithful impersonating remains challenging\. Existing in\-context learning\-based methods fail to capture how individuals react under different situations\. In addition, LLM\-based evaluation is difficult for obscure individuals\. To address these challenges, we proposeSituation–Internal state–Behavior Personamethod to incorporate situation\-dependent behavioral strategies\. We further design an evaluation protocol that provides LLM evaluators with references about the impersonated individual\. We evaluate our approach on a newly constructed dataset for the task of generating replies on social media\. Experimental results show that our proposed method outperforms state\-of\-the\-art ICL\-based baselines, while our evaluation protocol achieves moderate correlation with human judgment\. Besides, experiments on fictional\-character benchmarks demonstrate that our proposed method is applicable beyond the social media setting\. These findings suggest that incorporating behavioral information broadly improves the fidelity of role\-playing for real individuals on social media or fictional characters\.111Code:[https://github\.com/MiuLab/SIBPersona](https://github.com/MiuLab/SIBPersona)

## 1Introduction

Recent advances in large language models \(LLMs\) have accelerated the development of role\-playing agents \(RPAs\) capable of impersonating both fictional characters\([Yu et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib4);[Zhang et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib3)\)and real individuals\([Chuang et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib1);[Wang et al\., 2025a](https://arxiv.org/html/2609.21349#bib.bib36)\)\. Recent evidence further suggests that RPAs for real individuals can produce convincing impersonations\([Shi et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib2)\)\.

Toward faithful role\-playing of specific real individuals, two major challenges remain\.Behavioral Modeling Gap:Existing in\-context learning \(ICL\)\-based role\-playing methods mainly provide RPAs with factual information about impersonated individuals extracted from historical conversations\([Liu et al\., 2024](https://arxiv.org/html/2609.21349#bib.bib12);[Shi et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib2);[Gao et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib26)\)\. However, such information does not explicitly capture how an individual tends to react under different situations\. As illustrated in[Figure 1](https://arxiv.org/html/2609.21349#S1.F1), in social media comment\-replying scenarios between influencers and followers, replies are often brief or knowledge\-light, making faithful impersonation depends more on behavioral strategies than on factual information\.LLM Familiarity Gap:Existing LLM\-based evaluation protocols for role\-playing are largely developed for fictional characters, whose information are often represented in LLMs’ knowledge\([Peng and Chen, 2026](https://arxiv.org/html/2609.21349#bib.bib23)\)\. As a result, directly prompting an LLM can provide a reasonable evaluation\([Wang et al\., 2024](https://arxiv.org/html/2609.21349#bib.bib9);[Yu et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib4)\)\. By contrast, this direct prompting strategy is less applicable to obscure real individuals, whose information may be sparsely represented in LLMs’ pre\-training data\. This limited familiarity makes it difficult to directly use LLMs for automated evaluation of role\-playing real\-individual\.

To address these challenges, we develop a behavior\-aware framework for role\-playing real individuals\. Inspired by the Cognitive\-Affective Processing System\([Mischel and Shoda, 1995](https://arxiv.org/html/2609.21349#bib.bib14)\), we proposeSituation–Internal state–Behavior Persona \(SIBPersona\), a method that models individuals’ situation\-dependent behavioral strategies\. In addition, we design an automated LLM\-based evaluation protocol tailored to role\-playing real\-individual, which provides the LLM evaluator with reference about the impersonated individual to alleviate its limited familiarity with the individual\. We apply this framework to the domain of social media by constructing a dataset of influencer–follower interactions\. Using this dataset, we evaluate RPAs on their ability to impersonate influencers, specifically in generating replies to followers’ comments\.

Figure 1:In the social media comment\-replying scenarios between influencers and followers, replies involve short texts where behavioral strategy is more important for determining the fidelity of impersonation\.The contributions of this paper are three\-fold:

- •We present a structured view of persona for RPAs by distinguishing data\-invariant and data\-variant personae\. Building on this view, we propose SIBPersona, which introduces reaction process as a novel dimension of data\-variant persona to capture situation\-dependent behavioral strategies\.
- •We establish an reference\-augmented evaluation protocol for role\-playing of obscure individuals\. The protocol achieves moderate correlation with human judgment\.
- •We conduct comprehensive experiments across multiple influencers, model families, and fictional\-character benchmarks, showing that SIBPersona consistently outperforms state\-of\-the\-art ICL\-based baselines\.

## 2Related Work

##### Approaches of LLM Role\-Playing

Two mainstream approaches enable LLMs to impersonate specific targets: Fine\-Tuning \(FT\)\([Yang et al\., 2026](https://arxiv.org/html/2609.21349#bib.bib10);[Ye et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib11)\)and ICL\. We focus on ICL\-based methods because persona information is explicitly represented in prompts, making the role\-playing process easier to interpret and modify\. Existing ICL\-based methods utilize retrieval\-augmented generation \(RAG\)\([Lewis et al\., 2020](https://arxiv.org/html/2609.21349#bib.bib17)\)to incorporate context\-relevant information into prompts\([Shi et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib2);[Gao et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib26)\)\. However, the retrieved content primarily consists of raw conversational data or factual knowledge\. Such approaches largely emphasize what an individual knows rather than how an individual tends to behave across different interaction situations\.

To address this gap, we draw on the Cognitive\-Affective Processing System \(CAPS\)\([Mischel and Shoda, 1995](https://arxiv.org/html/2609.21349#bib.bib14)\), which models behavior as a dynamic reaction process\. According to CAPS, individuals perform behaviors by processing situational inputs through internal states\. This perspective suggests that RPAs should model the situation\-dependent reaction process\. Nonetheless, how to explicitly model such a dynamic reaction process in ICL\-based method remains underexplored\.

##### Datasets for Role\-Playing of Real Individuals

Existing role\-playing datasets primarily focus on fictional characters\([Xu et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib19);[Xu et al\., 2026](https://arxiv.org/html/2609.21349#bib.bib27)\), while datasets for role\-playing real individuals remain relatively limited\. To study role\-playing real\-individual, we turn to social media, which provides rich conversational records for observing how an individual responds across diverse situations\([Li et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib44)\)\. To support our research goal of impersonating influencers, the dataset must satisfy two requirements: \(1\) it should contain abundant influencer\-centered interactions from a single identifiable individual, allowing LLMs to obtain comprehensive information about the impersonated individual; and \(2\) it should preserve the comment\-reply structure under the influencer’s posts, which is essential for observing how the individual responds to different situations\.

As summarized in[Table 1](https://arxiv.org/html/2609.21349#S2.T1), no existing dataset satisfies both requirements simultaneously\. Among the public datasets, BluePrint\([Bück\-Kaeffer et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib16)\)SYNTHIA\([Rahimzadeh et al\., 2026](https://arxiv.org/html/2609.21349#bib.bib41)\), PersonalDialog\([Zheng et al\., 2019](https://arxiv.org/html/2609.21349#bib.bib42)\), and Persona\-Chat\([Zhang et al\., 2018](https://arxiv.org/html/2609.21349#bib.bib15)\)do not preserve an identifiable individual’s history: BluePrint anonymizes users and aggregates interactions into clusters, while SYNTHIA provides only synthesized persona attributes rather than raw, individual\-level social media interactions\. PER\-CHAT\([Wu et al\., 2021](https://arxiv.org/html/2609.21349#bib.bib43)\)does contain large\-scale, identifiable individual histories \(314,749314,749individuals\), but lacks the comment\-reply structure needed to observe situational responses\. Conversely, PersonaConvBench\([Li et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib44)\)preserves this structure but provides a limited number of samples per individual \(6\.876\.87posts and37\.0537\.05comment\-reply pairs on average\), making it insufficient for LLMs to obtain comprehensive information about the impersonated individual\. These limitations motivate us to construct a new dataset tailored for the task of generating replies on social media\.

DatasetPublicIdentifiableIndividualHistory\#IndividualComment\-ReplyStructureAvg\. Postsper IndividualAvg\.Comment\-ReplyPairs perIndividualMax Postsper IndividualMaxComment\-ReplyPairs perIndividualBuzzFace[Santia and Williams \(2018\)](https://arxiv.org/html/2609.21349#bib.bib28)✗\-\-\-\-\-\-\-TWON[Münker et al\. \(2026\)](https://arxiv.org/html/2609.21349#bib.bib40)✗\-\-\-\-\-\-\-BluePrint[Bück\-Kaeffer et al\. \(2025\)](https://arxiv.org/html/2609.21349#bib.bib16)✓✗\-\-\-\-\-\-SYNTHIA[Rahimzadeh et al\. \(2026\)](https://arxiv.org/html/2609.21349#bib.bib41)✓✗\-\-\-\-\-\-PersonalDialog[Zheng et al\. \(2019\)](https://arxiv.org/html/2609.21349#bib.bib42)✓✗\-\-\-\-\-\-Persona\-Chat[Zhang et al\. \(2018\)](https://arxiv.org/html/2609.21349#bib.bib15)✓✗\-\-\-\-\-\-PER\-CHAT[Wu et al\. \(2021\)](https://arxiv.org/html/2609.21349#bib.bib43)✓✓314749✗\-\-\-\-PersonaConvBench[Li et al\. \(2025\)](https://arxiv.org/html/2609.21349#bib.bib44)✓✓2796✓6\.8737\.05111414Ours✓✓6✓18115554290315240

Table 1:Feasibility of existing datasets for studying role\-playing of real individuals on social media reply generation\. ”\-” indicates that the corresponding information is not applicable under our comparison criteria\. Since our research goal is to impersonate influencers across diverse interaction situations, we construct a dataset centered on a small number of influencers with rich historical interactions\.
##### Evaluation of RPAs

LLM evaluation have become widely adopted for evaluating role\-playing of fictional character\([Gu et al\., 2026](https://arxiv.org/html/2609.21349#bib.bib22)\), as they enable fine\-grained assessment of performance\([Wang et al\., 2025b](https://arxiv.org/html/2609.21349#bib.bib29);[Wang et al\., 2024](https://arxiv.org/html/2609.21349#bib.bib9);[Shao et al\., 2023](https://arxiv.org/html/2609.21349#bib.bib30)\)\. These approaches typically assume that the evaluator model possesses sufficient knowledge about the impersonated character[Peng and Chen \(2026\)](https://arxiv.org/html/2609.21349#bib.bib23)\.

Evaluating role\-playing of real individuals remains challenging because LLM evaluators have limited knowledge about obscure people\. Consequently, prior studies on role\-playing real individuals often rely on human evaluation\([Shi et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib2);[Tu et al\., 2023](https://arxiv.org/html/2609.21349#bib.bib31)\), while automated evaluation protocols for role\-playing of obscure real individuals remain underdeveloped\.

## 3SIBPersona

In this section, we presentSIBPersonamethod designed for role\-playing of influencers on social media\. We first introduce our multifaceted view of persona and define the reaction process in §[3\.1](https://arxiv.org/html/2609.21349#S3.SS1), followed by formalizing the task of generating replies on social media in §[3\.2](https://arxiv.org/html/2609.21349#S3.SS2)\. Subsequently, we elaborate how data\-variant persona is constructed from historical data in §[3\.3](https://arxiv.org/html/2609.21349#S3.SS3)\. Finally, we explain how the constructed data\-variant persona is retrieved and used for reply generation in §[3\.4](https://arxiv.org/html/2609.21349#S3.SS4)\.

### 3\.1Multifaceted Nature of Persona

In RPA, a "persona" is always referred to the information describing fictional characters or real individuals\. We argue that personae can be divided into two types:data\-invariantanddata\-variant\. The data\-invariant persona, as the name implies, is provided in advance and is not derived from the data\. For example, we can create a narrative to describe the demographic information of the influencer or the fictional traits of the fictional character, as far as we know\. On the other hand, the data\-variant persona is derived from the data and designated to capture the dynamics of RPA\.

Apart from deriving from the data or not, the prior methods mainly modeled the persona by two dimensions: 1\) memory[Shi et al\. \(2025\)](https://arxiv.org/html/2609.21349#bib.bib2)capturing the factual information and 2\) speaking style[Tu et al\. \(2024\)](https://arxiv.org/html/2609.21349#bib.bib8)capturing expression habits\. However, we contend that these two dimensions are insufficient for RPAs\. As suggested in CAPS, responses of real people are shaped not only by what they know and how they speak but also by the reaction process they tend to follow in a given situation\.

##### Reaction Process

In order to resolve this issue, we propose to incorporate so\-called reaction process, the third dimension, into the data\-variant persona\. To this end, we model reaction process as the mapping of a given situation to an internal state and subsequently to a behavioral strategy\. Formally, reaction process is defined by\{⟨s,i,b⟩∈S×I×B∣s=LLM\(x\),i=LLM\(x\),b=LLM\(x\)\}\\\{\\langle s,i,b\\rangle\\in S\\times I\\times B\\mid s=LLM\(x\),i=LLM\(x\),b=LLM\(x\)\\\}, whereSScontains situations,IIrepresents the set of internal states,BBis the set of behavioral strategies, andxxdenotes a text input\. We call this three\-tuple, namely⟨s,i,b⟩\\langle s,i,b\\rangle, an SIB triplet\. In the following, we utilize the example that influencers reply followers’ comments on social media to illustrate the idea of reaction process\. When influencers reply to followers’ comments, they first interpret the comments into high\-level situations, e\.g\., praise, inquiry, or hostility\. This emerging situation then triggers a specific internal state, such as happy or angry, which leads to a certain behavioral strategy, such as expressing gratitude or defensive rebuffing\. Grounded in CAPS, this three\-dimensional data\-variant persona provides a more psychologically faithful modeling of how real individuals respond\.

Figure 2:Overview of the SIBPersona method\. The upper part illustrates the process of extracting data\-variant persona from historical data\. The lower part presents reply generation process: retrieving the most relevant dimensional documents based on the comment enables the RPA to generate a high\-fidelity reply\.

### 3\.2Problem Formulation

After illustrating the three\-dimensional data\-variant persona, we provide the formal settings of the problem for easy exposition\. In general, RPAs are designed to engage in conversations with users by emulating specific characters or impersonating specific individuals\. In this work, we focus on the generation of single\-turn replies for influencers on social media\. To be specific, we generate the replyrraccording to the influencer’s posti​pip, the follower’s commentf​cfc, the influencer’s data\-invariant profilepp, and the influencer’s data\-variant personaP​Sd​vPS\_\{dv\}, which can be represented as:

r=R​P​A​\(i​p,f​c,p,P​Sd​v\)\.r=RPA\(ip,fc,p,PS\_\{dv\}\)\.In particular,P​Sd​vPS\_\{dv\}contains information about the three aforementioned dimensions\.

### 3\.3Constructing Data\-Variant Persona

In this subsection, we elaborate on how each dimension is extracted and represented\. We first view a data sample associated with a specific influencer asd​sj=⟨i​pj,C​R​Pj⟩ds\_\{j\}=\\langle ip\_\{j\},CRP\_\{j\}\\rangle, wherei​pjip\_\{j\}is the influencer’s post, andC​R​Pj=\{⟨f​cj​1,rj​1⟩,⟨f​cj​2,rj​2⟩,…,⟨f​cj​n,rj​n⟩\}CRP\_\{j\}=\\\{\\langle fc\_\{j1\},r\_\{j1\}\\rangle,\\langle fc\_\{j2\},r\_\{j2\}\\rangle,\\dots,\\langle fc\_\{jn\},r\_\{jn\}\\rangle\\\}denotes the set of all comment\-reply pairs associated withi​pjip\_\{j\}post\. As a consequence, the historical data of this influencer is a collection of the data samples, denoted as𝒟=\{d​s1,d​s2,…,d​sn\}\\mathcal\{D\}=\\\{ds\_\{1\},ds\_\{2\},\\dots,ds\_\{n\}\\\}\.

In accordance with the previous works of extracting data\-variant personae by LLMs\([Shi et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib2);[Liu et al\., 2024](https://arxiv.org/html/2609.21349#bib.bib12)\), we employ a two\-stage pipeline to produce data\-variant personae from raw social media histories\. Our entire pipeline is depicted in the upper part of[Figure 2](https://arxiv.org/html/2609.21349#S3.F2)and is explained in detail below\.

##### Stage 1: Dimensional Relevance Filtering

Due to data noise on social media, we employ a recognizing LLM to filter irrelevant information in the first stage to increase the efficacy of the data pipeline\. Specifically, the recognizing LLM acts like a multi\-label classifier that assigns each data sample to the set of dimensional labelsL=\{lM,lT,lR​P\}L=\\\{l\_\{M\},l\_\{T\},l\_\{RP\}\\\}\. That is,∀j​h​\(d​sj\)=𝒜j𝒜j⊆L\\forall j\\ h\(ds\_\{j\}\)=\\mathcal\{A\}\_\{j\}\\ \\ \\mathcal\{A\}\_\{j\}\\subseteq L, wherehhis a mapping function from a data sample to the subset ofLL\. As a shorthand, we use the notationsMM,TT,R​PRPto represent memory, speaking style, and reaction process, respectively\. If the recognizing LLM assigns the data sampled​sjds\_\{j\}to the empty set, i\.e\.h⁡\(d​sj\)=∅h\(ds\_\{j\}\)=\\emptyset,d​sjds\_\{j\}is discarded\. Otherwise,d​sjds\_\{j\}proceeds to the next stage\.

##### Stage 2: Dimensional Document Extraction

Given a data sampled​sj=⟨i​pj,C​R​Pj⟩ds\_\{j\}=\\langle ip\_\{j\},CRP\_\{j\}\\ranglesuch thath⁡\(d​sj\)≠∅h\(ds\_\{j\}\)\\neq\\emptyset, we use an extracting LLM to generate the dimensional information ofd​sjds\_\{j\}for each labell∈h⁡\(d​sj\)l\\in h\(ds\_\{j\}\)\. We introducedimensional documentswhose details are elaborated later as a countable surrogate for the generated dimensional information\. Givend​sjds\_\{j\}and a labell∈h⁡\(d​sj\)l\\in h\(ds\_\{j\}\), the extracting LLM may generate more than one dimensional document, and we usekkto index the generated dimensional documents\.

To equip our approach with interpretability, we define a subset ofC​R​PjCRP\_\{j\}as a set ofevidence pairsdenotedE​Pj​kEP\_\{jk\}, andE​Pj​kEP\_\{jk\}provides the grounding evidence for the generated dimensional information\. If the labelllislMl\_\{M\}, the extracting LLM generates a memory documentdj​k\(lM\)=⟨m​pj​k,E​Pj​k⟩d\_\{jk\}^\{\(l\_\{M\}\)\}=\\langle mp\_\{jk\},EP\_\{jk\}\\rangle, wherem​pj​kmp\_\{jk\}denotes a memory point capturing factual information\. When the labelllislTl\_\{T\}, the extracting LLM produces a speaking style documentdj​k\(lT\)=⟨s​pj​k,E​Pj​k⟩d\_\{jk\}^\{\(l\_\{T\}\)\}=\\langle sp\_\{jk\},EP\_\{jk\}\\rangle, wheres​pj​ksp\_\{jk\}represents a speaking style point reflecting a expression habit\. Otherwise, the extracting LLM yields a reaction process documentdj​k\(lR​P\)=⟨⟨sj​k,ij​k,bj​k⟩,E​Pj​k⟩d\_\{jk\}^\{\(l\_\{RP\}\)\}=\\langle\\langle s\_\{jk\},i\_\{jk\},b\_\{jk\}\\rangle,EP\_\{jk\}\\rangle, where⟨sj​k,ij​k,bj​k⟩\\langle s\_\{jk\},i\_\{jk\},b\_\{jk\}\\rangleconstitutes an SIB triplet comprising the situation, internal state, and behavioral strategy, respectively\.

Depending on the type of dimensions, we store the extracted dimensional document in the correspondingdimensional corpus, which is a collection of dimensional documents\. Thus, our data\-variant personaP​Sd​vPS\_\{dv\}has three dimensional corpora: memory corpus, speaking style corpus, and reaction process corpus\. These three dimensional corpora lay the foundation for the reply generation illustrated in the next subsection\.

### 3\.4Reply Generation

After the data\-variant persona is established, one intuitive idea is to generate the reply by considering all dimensional documents in these three dimensional corpora as the demonstrations to LLMs\. However, this idea is inherently infeasible since influencers on social media always have numerous posts and comment\-reply pairs; otherwise, they could hardly be called "influencers\." Instead of usingP​Sd​vPS\_\{dv\}directly, we utilize the approximate data\-variant personaP​Sd​v¯\\overline\{\{PS\_\{dv\}\}\}derived fromP​Sd​vPS\_\{dv\}as an alternative\.

As depicted in the lower part of[Figure 2](https://arxiv.org/html/2609.21349#S3.F2), we utilize the concept of RAG\([Lewis et al\., 2020](https://arxiv.org/html/2609.21349#bib.bib17)\)to deriveP​Sd​v¯\\overline\{\{PS\_\{dv\}\}\}\. Given the queryf​cfc, we independently retrieve the top three dimensional documents which are most relevant to the queryf​cfcfrom each dimensional corpus\. The relevance measure is calculated as follows\. We treatm​pj​kmp\_\{jk\},s​pj​ksp\_\{jk\}, andsj​ks\_\{jk\}in the SIB triplet as the retrieval keys for memory corpus, speaking style corpus, and reaction process corpus, respectively\. For each dimensional corpus, we compute the cosine similarity between the embedding of queryf​cfcand the embedding of the retrieval keys\. Afterwards, we rank the dimensional documents by similarity scores and retrieve the top three corresponding dimensional documents for each dimensional corpus\. Finally, the nine retrieved dimensional documents constituteP​Sd​v¯\\overline\{\{PS\_\{dv\}\}\}, serving as demonstrations to LLM\. The problem we address becomes:

r=R​P​A​\(i​p,f​c,p,P​Sd​v¯\)\.r=RPA\(ip,fc,p,\\overline\{PS\_\{dv\}\}\)\.[Figure 11](https://arxiv.org/html/2609.21349#A7.F11)shows an example of a retrieved dimensional document and the reply generated by SIBPersona\. Additional implementation details are provided in Appendix[B](https://arxiv.org/html/2609.21349#A2)\.

## 4Reference\-Augmented Evaluation

In this section, we propose a reference\-augmented protocol for automatically evaluating role\-playing agents\. We first explain the challenges of this task, then detail our design, and conclude by empirically validating its alignment with human judgment\.

MethodsGPT\-5\.2Gemini\-3\-flashDeepSeek\-V3\.2Claude\-Opus\-4\.6AverageMMTTR​PRPMMTTR​PRPMMTTR​PRPMMTTR​PRPMMTTR​PRPNaive97\.5046\.40‡67\.53‡94\.7042\.47‡69\.27‡95\.1348\.93‡70\.83‡96\.7746\.93‡76\.87‡96\.0346\.18‡70\.78‡CoT97\.4744\.93‡70\.73‡95\.8341\.17‡69\.53‡96\.2749\.23‡71\.47‡97\.3347\.0077\.33‡96\.7345\.58‡72\.27‡IMPersona96\.0747\.50‡71\.93‡94\.9044\.60‡70\.10‡96\.0050\.80‡72\.50‡97\.4348\.67‡78\.13‡96\.1047\.89‡73\.17‡Vanilla RAG97\.3349\.43‡73\.20‡93\.7753\.2071\.50‡94\.2358\.37‡73\.4396\.7056\.9377\.70‡95\.5154\.48‡73\.96‡SIBPersona96\.9054\.3377\.6393\.7756\.4775\.5094\.8363\.6776\.4396\.9362\.7781\.9395\.6159\.3177\.88Table 2:Performance onInfluencer A\.Bold: the best results;‡\\ddagger: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.MethodsDeepseek\-V3\.2Gemini\-3\-flashMMTTR​PRPMMTTR​PRPNaive92\.10±\\pm3\.0442\.03‡±\\pm8\.4368\.48‡±\\pm5\.8692\.50±\\pm2\.3243\.33‡±\\pm7\.3570\.64‡±\\pm7\.37CoT92\.17±\\pm3\.0343\.92‡±\\pm8\.1369\.83‡±\\pm6\.1492\.15±\\pm2\.5444\.05‡±\\pm7\.0870\.48‡±\\pm6\.40IMPersona91\.60±\\pm2\.2145\.54‡±\\pm5\.8671\.14‡±\\pm6\.0291\.42±\\pm1\.9445\.21‡±\\pm6\.4071\.56‡±\\pm7\.19Vanilla RAG91\.39±\\pm3\.2357\.05‡±\\pm2\.4372\.43‡±\\pm5\.0692\.19±\\pm2\.4653\.43‡±\\pm4\.9272\.59‡±\\pm4\.79SIBPersona92\.14±\\pm3\.2661\.09±\\pm1\.6675\.84±\\pm4\.9092\.21±\\pm2\.2057\.34±\\pm3\.2076\.48±\\pm5\.08Table 3:Results on other five influencers\.Boldindicates the better scores\.‡\\ddagger: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.MethodsMMTTR​PRPSIBPersona92\.67±\\pm3\.1861\.60±\\pm1\.8275\.96±\\pm4\.49w/oII92\.61±\\pm2\.4960\.30±\\pm2\.3775\.44±\\pm3\.61w/oR​PRP92\.83±\\pm2\.4457\.82‡±\\pm2\.5173\.36‡±\\pm4\.64Table 4:Dimension ablation study usingDeepSeek\-V3\.2on the six influencer\.Bold: the best results;‡\\ddagger: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.### 4\.1Penalty\-Based LLM Evaluator

To enable scalability and facilitate future iterations, we adopt LLM evaluators to automatically assess role\-playing agents for the task of generating replies on social media\. However, as addressed in §[2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3), LLM evaluators may have limited knowledge of obscure individuals\. To resolve this knowledge scarcity issue, we augment the LLM evaluator with some references calledrepresentative dimensional documentsabout the impersonated individual\. The representative dimensional documents are constructed as follows\. We first apply the method described in §[3\.3](https://arxiv.org/html/2609.21349#S3.SS3)to construct the three dimensional corpora from the test samples and then select representative dimensional documents, denoted asr​d​dlrdd\_\{l\}, from the resulting dimensional corpus for each dimensionl∈Ll\\in L\. Specifically, we apply K\-medoids clustering to selectM/2\\sqrt\{M/2\}representative dimensional documents from each dimensional corpus, whereMMis the size of dimensional documents\. Clustering is performed based on the embeddings ofsj​ks\_\{jk\},m​pj​kmp\_\{jk\}ands​pj​ksp\_\{jk\}\. Crucially, the representative dimensional documents are constructed and sampled exclusively from the test samples rather than the demonstration examples\. This ensures that the LLM evaluator’s reference standard remains independent of the role\-playing agent’s demonstration source, so that the evaluation reflects the agent’s true generalization ability rather than its capacity to reproduce seen demonstrations\.

For the clarity of presentation, we define a test sample ast​sj=⟨i​pj,f​cj,rja⟩,ts\_\{j\}=\\langle ip\_\{j\},fc\_\{j\},r\_\{j\}^\{a\}\\rangle,whererjar\_\{j\}^\{a\}is the reply generated by an RPA\. For each dimensionl∈Ll\\in L, our LLM evaluator takest​sjts\_\{j\},pp,r​d​dlrdd\_\{l\}, and the ground\-truth replyrjg​tr\_\{j\}^\{gt\}written by the influencer as input\. Following[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.21349#bib.bib6), our LLM evaluator is instructed to identify the flaws inrjar\_\{j\}^\{a\}and to assign a severity level ranging from11to55for each flaw at the same time\. The final score for the dimensionllis computed by averaging the scores on all test samples whose size isNN, which can be summarized as follows:

S​c​o​r​el=1N​∑j=1Nmax⁡\(0,100−5⋅∑f∈Fl​jvf\),Score\_\{l\}=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}\\max\(0,100\-5\\cdot\\sum\_\{f\\in F\_\{lj\}\}v\_\{f\}\),whereFl​jF\_\{lj\}is the set of the identified flaws for dimensionllonrjar\_\{j\}^\{a\}, andvfv\_\{f\}is the severity level of flawff\. Additional details regarding the evaluation implementation are provided in Appendix[C](https://arxiv.org/html/2609.21349#A3)\.

### 4\.2Alignment with Human judgment

To assess whether our reference\-augmented evaluation protocol is well\-aligned with human judgment, we examine the correlation between human and LLM evaluator scores\. For evaluation scores averaged across the three dimensions, the alignment between human and LLM evaluator yields a Pearson’srrof0\.5610\.561and a Spearman’sρ\\rhoof0\.5300\.530, indicating a moderate correlation by conventional social\-science standards\([Cohen, 1988](https://arxiv.org/html/2609.21349#bib.bib24);[Lovakov and Agadullina, 2021](https://arxiv.org/html/2609.21349#bib.bib25)\)and also consistent with the previous findings in role\-playing evaluation\([Zhou et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib13);[He et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib5)\)\. These results demonstrate that our reference\-augmented protocol serves as a reasonable proxy for human judgment\. Additional details of the correlation analysis are provided in Appendix[D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px3), and the advantages of our protocol over traditional NLP metrics are reported in Appendix[E\.2](https://arxiv.org/html/2609.21349#A5.SS2)\.

## 5Experimental Evaluation

In this section, we present the experimental evaluation of SIBPersona\. We first describe the process of building the dataset in §[5\.1](https://arxiv.org/html/2609.21349#S5.SS1), followed by the experimental settings in §[5\.2](https://arxiv.org/html/2609.21349#S5.SS2)\. We then show the results of the experiments conducted on six influencers to demonstrate the effectiveness of SIBPersona in §[5\.3](https://arxiv.org/html/2609.21349#S5.SS3)\. To validate our SIBPersona, we perform a human evaluation to examine whether theR​PRPdimension provides interpretable evidence in §[5\.4](https://arxiv.org/html/2609.21349#S5.SS4)\. Lastly, we show that SIBPersona is also applicable to fictional character benchmarks in §[5\.5](https://arxiv.org/html/2609.21349#S5.SS5)\.

### 5\.1Dataset Construction

We construct our dataset from posts, comments, and replies associated with six influencers, denoted asInfluencer AtoFfromCMoney Forum222CMoney Forum is the largest online investor community in Taiwan, where members post Traditional Chinese messages under individual stock tickers and follow financial influencers whose posts routinely attract numerous replies\.\. We collect the data span from January 1, 2024, to September 30, 2025\. The data whose dates are before August 1, 2025 belong to the demonstration source, while the remaining data are preserved as the candidate pool for test samples\.

To have a fair comparison between the different influencers, we perform stratified sampling on the candidate pool and keep the size of the test samples for each influencer in the same order of magnitude\. In addition, our preliminary analysis reveals that different comment–reply pairs exhibit distinct interaction types\. To ensure that the test samples thoroughly cover these diverse interactions, we promptgpt\-4o\-minito assign each comment–reply pair a score reflecting the degree of expressing personal characteristic\. Based on these scores, the data are partitioned into three levels: high, medium, and low\. From each level, we select5050test samples at most\. Additional details regarding dataset construction are provided in Appendix[A](https://arxiv.org/html/2609.21349#A1)\.

### 5\.2Experimental Setting

We compare our method with several ICL\-based role\-playing methods, all of which take the impersonated individual’s profileppas default input\. We briefly describe each compared method below\.Naiverelies solely on the impersonated individual’s profile\.CoTbuilds upon theNaiveby additionally incorporating Chain\-of\-Thought reasoning\([Wei et al\., 2022](https://arxiv.org/html/2609.21349#bib.bib18)\)\.Vanilla RAGaugments the prompt with nine retrieved raw comment–reply pairs to match the number of dimensional documents utilized in SIBPersona\. Finally,IMPersona\([Shi et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib2)\)is a state\-of\-the\-art ICL\-based method for role\-playing real individuals in conversations, which uses the proposed hierarchical memory module to model its data\-variant persona\.

We conduct two types of experiments to evaluate the robustness of SIBPersona along two axes: model choice and impersonated influencer\. First, we examine model\-agnostic robustness by applying multiple LLMs to construct data\-variant personas, includingGPT\-5\.2,Gemini\-3\-flash,DeepSeek\-v3\.2, andClaude\-Opus\-4\.6\. These experiments are conducted onInfluencer Ato assess whether the effectiveness of SIBPersona is robust to the choice of the LLM used for persona construction, rather than being tied to a specific model\. Second, we examine the robustness of cross\-influencers by applying SIBPersona toInfluencer BthroughInfluencer F\. In these experiments, we useGemini\-3\-flashandDeepSeek\-v3\.2to evaluate whether SIBPersona remains effective across different influencers\. All experiments are evaluated using the reference\-augmented evaluation protocol introduced in §[4](https://arxiv.org/html/2609.21349#S4)in all three dimensions, withgpt\-4\.1as the LLM evaluator andtext\-embedding\-3\-smallas the embedding model for retrieval\. For statistical significance testing, we perform t\-tests with Holm correction for multiple comparisons\.

MethodsMMTTR​PRPAvg\.IMPersona2\.92‡2\.40‡2\.67‡2\.67‡Vanilla RAG3\.683\.723\.623\.67SIBPersona3\.783\.643\.623\.68Table 5:Human evaluation results comparing our framework with the baseline\.Bold: better results;‡: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.ConditionsMMTTR​PRPUninterpretable3\.033\.002\.87Interpretable4\.13†3\.97†4\.00†Table 6:Human evaluation scores for Interpretable \(n=20\) and Uninterpretable \(n=10\) groups\.Bold: better results;†: significant improvement withp<0\.05p<0\.05\.
### 5\.3Main Results

[Table 2](https://arxiv.org/html/2609.21349#S4.T2)shows the results ofInfluencer Ain four LLMs\. On average, SIBPersona outperforms all baselines by a statistically significant margin on both theTTandR​PRPdimensions\. Across all methods, performance on theMMdimension remains uniformly high, reflecting that replies on social media naturally demand less factual memory information and thus leave minimal room for penalties\. In terms of the other two dimensions, Naive and CoT perform the worst due to insufficient information regarding the impersonated individual\. While IMPersona incorporates rich memory information, it fails on speaking style and reaction process due to its single\-dimensional design\. Vanilla RAG emerges as the strongest baseline, indicating that simply mirroring similar historical replies yields reasonable results\. Nevertheless, Vanilla RAG still falls short of SIBPersona, showing that constructing structured, multi\-dimensional data\-variant persona is crucial for faithful role\-playing\.

[Table 3](https://arxiv.org/html/2609.21349#S4.T3)reports the results aggregated overInfluencer BthroughInfluencer F\. SIBPersona consistently achieves the highest scores in theTTandR​PRPdimensions across all five influencers under both models\. Nearly all baselines exhibit statistically significant degradation compared to SIBPersona, and the advantage is consistent regardless of model family or impersonated influencer\. These results show that SIBPersona is robust along two orthogonal axes: model choice and impersonated individual\. From a qualitative perspective,[Figure 12](https://arxiv.org/html/2609.21349#A7.F12)illustrates that SIBPersona better captures the behavioral strategy and speaking style of the impersonated individual facing both praise and criticism\. Detailed experimental results and qualitative analysis are provided in Appendix[E\.1](https://arxiv.org/html/2609.21349#A5.SS1)and[G](https://arxiv.org/html/2609.21349#A7)\.

The ablation study further demonstrates the importance of theR​PRPdimension\. As shown in[Table 4](https://arxiv.org/html/2609.21349#S4.T4), removing theR​PRPdimension significantly degrades the performance on theTTandR​PRPdimensions, confirming that theR​PRPdimension contributes substantially to overall performance\. In contrast, removing only the internal state component causes marginal, non\-significant declines\. This asymmetry suggests that while the full SIB triplet is beneficial, explicit behavioral strategiesBBcontribute more critically than intermediate internal\-stateIIfor the social media reply generation task, where replies tend to be short and direct\.

### 5\.4Human Evaluations

To validate the effectiveness of SIBPersona, we perform a human evaluation in which the annotators rate the replies generated by SIBPersona, Vanilla RAG and IMPersona along three dimensions using a five\-point Likert scale\([Joshi et al\., 2015](https://arxiv.org/html/2609.21349#bib.bib33)\)\. As shown in[Table 5](https://arxiv.org/html/2609.21349#S5.T5), SIBPersona significantly outperforms IMPersona across all three dimensions, while performing comparably to Vanilla RAG\. Through qualitative analysis, we observe that Vanilla RAG frequently directly replicates the replies from the retrieved replies\. This tendency often leads to misaligned replies when discrepancies arise between the follower’s comment and the retrieved context\. In contrast, SIBPersona generates replies grounded in the structured points, allowing it to adaptively fit the current situation\. While such fine\-grained differences are less distinguishable under the small\-scale human evaluation, the adaptability of SIBPersona is more clearly reflected in our large\-scale LLM\-based evaluations\. Additional details and results of the human evaluation are provided in Appendix[D](https://arxiv.org/html/2609.21349#A4)\.

##### Interpretability of SIBPersona

We further examine the interpretability of SIBPersona\. In this type of experiments, the annotators are asked to assess whether the SIB triplets inP​Sd​v¯\\overline\{PS\_\{dv\}\}, especially the internal state and behavior strategy, can provide reasonable explanations for the generated reply given a test sample\. Based on these assessments, we divide the test samples into interpretable and uninterpretable groups\. Additional details of the interpretability experiments are provided in Appendix[D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px3)\. As shown in[Table 6](https://arxiv.org/html/2609.21349#S5.T6), the replies in the interpretable group significantly outperform those in the uninterpretable group across all three dimensions\. This result suggests that the interpretability of the SIB triplets inP​Sd​v¯\\overline\{PS\_\{dv\}\}influences the quality of the generated replies and subsequently contributes to improved individual fidelity\. In addition, we provide a qualitative example in Appendix[G\.3](https://arxiv.org/html/2609.21349#A7.SS3), showing that SIBPersona can facilitate error diagnosis\.

### 5\.5Results on CharacterEval

To examine whether SIBPersona is applicable beyond the social media setting, we evaluate our method on CharacterEval\([Tu et al\., 2024](https://arxiv.org/html/2609.21349#bib.bib8)\), a fictional character benchmark formulated as a single\-turn response generation task\. To minimize confounding effects from a model’s prior knowledge of well\-known fictional characters, we follow the anonymized evaluation setting proposed by[Peng and Chen \(2026\)](https://arxiv.org/html/2609.21349#bib.bib23)\. We construct dimensional corpora usinggpt\-4o\-miniand generate replies usingOLMo\-3\-7B\-Instruct\.

As summarized in[Table 7](https://arxiv.org/html/2609.21349#S5.T7), SIBPersona significantly outperforms IMPersona on three out of four evaluation dimensions\. Remarkably, SIBPersona also achieves performance competitive withCoSER\-Llama\-3\.1\-8B\([Wang et al\., 2025c](https://arxiv.org/html/2609.21349#bib.bib6)\), a model specifically fine\-tuned on fictional characters’ conversational data, demonstrating that an ICL\-based method augmented with behavioral modeling can approach the performance of task\-specific fine\-tuned models\.

MethodsCARACKCPIMPersona2\.942\.432\.252\.38SIBPersona3\.28†2\.66†2\.32†2\.69†Fine\-tuned CoSER3\.352\.612\.282\.68Table 7:Results on CharacterEval\.CA: Conversational Ability,RA: Role\-Playing Attractiveness,CK: Character Consistency–Knowledge, andCP: Character Consistency–Persona\.Bold: the best results;†\{\\dagger\}: significant improvement withp<0\.05p<0\.05\.The ablation study in[Table 8](https://arxiv.org/html/2609.21349#S5.T8)further corroborates the findings in §[5\.3](https://arxiv.org/html/2609.21349#S5.SS3)\. Removing theR​PRPdimension leads to significant degradation across multiple evaluation dimensions, while removing only the internal state yields a smaller performance drop\. Removing theTTdimension does not lead to a significant performance drop, suggesting that reaction process may play a more important role than stylistic information in fictional character benchmarks\. These consistent patterns across two distinct role\-playing settings confirm that SIBPersona is broadly applicable beyond the domain for which it was originally designed\. Additional results and analysis are provided in Appendix[E\.4](https://arxiv.org/html/2609.21349#A5.SS4)\.

MethodsCARACKCPSIBPersona3\.282\.662\.322\.69w/oTT3\.332\.692\.312\.74w/oII3\.282\.662\.312\.68w/oR​PRP3\.16‡2\.612\.302\.62‡w/oR​PRP&TT3\.05‡2\.54‡2\.282\.57‡Table 8:Component ablation study on CharacterEval\.Bold: the best results;‡\\ddagger: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.

## 6Conclusion

In this work, we study the problem of role\-playing real individuals on social media and propose a behavior\-aware framework to address two major challenges\. In our framework, we introduce SIBPersona, which extends data\-variant persona by modeling the reaction process\. We construct a dataset featuring real social media influencers and formulate the task of generating replies\. Our experimental results across multiple models demonstrate that SIBPersona improves role\-playing fidelity, and human evaluation further verifies the interpretability of the retrieved SIB triplets\. We also find that reaction process modeling is applicable to fictional character role\-playing benchmarks\. Moreover, our reference\-augmented evaluation protocol for role\-playing real\-individual shows acceptable alignment with human judgment\. These findings suggest that behavior\-aware persona modeling can improve role\-playing performance on social media setting\.

In future work, this framework could be extended to broader domains and more complex generation tasks\. One such direction is extending SIBPersona to multi\-turn dialogue, which would require handling evolving reaction processes\. This can be achieved by dynamically formulating retrieval queries from the accumulated conversational history instead of relying solely on the latest comment, enabling the model to fetch situation\-relevant SIB triplets that capture accumulated context\.

## Limitations

Although our study demonstrates effectiveness on both our newly constructed social media dataset and a public open\-source dataset, and shows consistent improvements across both large API\-based models and smaller open\-source models, several limitations remain\.

First, we observe that our method struggles with two types of comment situations \([Figure 13](https://arxiv.org/html/2609.21349#A7.F13)\): real\-time information queries and sarcastic comments\. In the former case, the LLM lacks up\-to\-date information, which leads to hallucinated replies\. In the latter case, the LLM may misinterpret sarcastic remarks from followers, resulting in inappropriate behavioral strategies\.

Second, our current study focuses on single\-turn reply generation and does not evaluate performance in more challenging multi\-turn dialogue settings\. Since conversational contexts may evolve across turns, future work could investigate how the proposed framework performs under dynamically changing interaction scenarios\.

Third, the dimensional documents in SIBPersona are currently provided to the LLM through in\-context learning rather than model training\. We have not explored whether integrating dimensional documents into model training could further improve performance\. Future works could investigate training models with dimensional documents and examine whether such models can achieve stronger performance than API\-based models\.

## Ethical Considerations

This work involves the use of social media data and the impersonation of real individuals\. We take several measures to address ethical concerns related to data usage, annotation practices, and potential societal impacts\.

First, this study was conducted as an industry\-academia collaboration between National Taiwan University and CMoney, the company that owns and operates the platform from which the data were collected\. All data access and processing were conducted solely by CMoney\-affiliated authors within a secure, company\-managed environment in full compliance with the platform’s Terms of Service and Privacy Policy\. We obtained explicit consent from all influencers whose personas were evaluated in our experiments to use their available data for research purposes\. Second, regarding follower comments, platform members accept the Terms of Service and Privacy Policy upon registration, authorizing the platform to collect, process, and use member\-provided data for online behavior research, statistics, and service improvements\. To protect participant privacy, all follower comments in our dataset and evaluation materials are strictly anonymized, retaining no user identifiers or personally identifiable information\. Furthermore, no raw influencer or follower content will be released as part of our research artifacts\. Third, human annotators involved in the evaluation are employees of CMoney and are compensated through their regular salaries\. Fourth, AI\-based writing assistance tools were used to help polish parts of the manuscript\. All AI\-assisted text was carefully reviewed and proofread by the authors to ensure correctness and accuracy\.

Finally, we acknowledge that technologies capable of impersonating real individuals could potentially be misused to generate misleading or deceptive content on social media\. This work is an evaluative research study rather than a public deployment\. Any future commercialization or deployment would be handled separately by CMoney and is outside the scope of this study; nonetheless, we recommend that any downstream deployment require: \(1\) personas to be created internally and only for explicitly authorized influencers, \(2\) human review of generated content before publication, particularly for sensitive responses, and \(3\) mechanisms allowing influencers to deactivate their personas or withdraw consent\. Meanwhile, our study contributes to understanding the capabilities and limitations of such systems\. We hope that these insights can help future research develop methods for detecting impersonation and mitigating the spread of misinformation in online communities\.

## Acknowledgments

The work was financially supported by a National Taiwan University Industry\-Academia Cooperative Research Project, the National Science and Technology Council \(NSTC\), and the Featured Area Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education \(MOE\), Taiwan, under Grants 112\-2223\-E\-002\-012\-MY5, 115\-2628\-E\-002\-023\-MY4, and 115L900901\.

We thank Li\-Ching Chien, Pei\-Yu Hou, Pei\-Ru Huang, Si\-Rui Huang, Meng\-Jung Lin, Yung\-Chen Liu, Xiao\-An Wang, and Yi\-Chi Yeh for serving as human annotators in this work\.

## References

- Bück\-Kaefferet al\.\(2025\)A\. Bück\-Kaeffer, J\. Q\. Chooi, D\. Zhao, M\. P\. Touzel, K\. Pelrine, J\. Godbout, R\. Rabbany, and Z\. YangBluePrint: a social media user dataset for LLM persona evaluation and training\.arXiv preprint arXiv:2510\.02343\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.02343),[Link](https://arxiv.org/abs/2510.02343)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p2.1),[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.4.1)\.
- Chuanget al\.\(2025\)Y\. Chuang, R\. Tu, C\. Dai, Y\. Li, S\. Vasani, B\. Yao, M\. H\. Tessler, S\. Yang, D\. Shah, R\. Hawkins, J\. Hu, and T\. T\. RogersDEBATE: a large\-scale benchmark for evaluating opinion dynamics in role\-playing LLM agents\.arXiv preprint arXiv:2510\.25110\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.25110),[Link](https://arxiv.org/abs/2510.25110)Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p1.1)\.
- Cohen \(1988\)J\. CohenStatistical power analysis for the behavioral sciences\.2nd edition,Lawrence Erlbaum Associates,Hillsdale, NJ\.Cited by:[§4\.2](https://arxiv.org/html/2609.21349#S4.SS2.p1.1)\.
- Gaoet al\.\(2025\)Z\. Gao, X\. Xing, and X\. XuTailorRPA: a retrieval\-based framework for eliciting personalized and coherent role\-playing agents in general domain\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 5381–5412\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.288/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.288),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p2.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2026\)J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Z\. Lin, B\. Zhang, L\. Ni, W\. Gao, Y\. Wang, and J\. GuoA survey on LLM\-as\-a\-judge\.The Innovation7\(6\),pp\. 101253\.External Links:[Document](https://dx.doi.org/10.1016/j.xinn.2025.101253),[Link](https://doi.org/10.1016/j.xinn.2025.101253)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p1.1)\.
- Heet al\.\(2025\)K\. He, Y\. Huang, W\. Wang, D\. Ran, D\. Sheng, J\. Huang, Q\. Lin, J\. Xu, W\. Liu, and M\. FengCrab: a novel configurable role\-playing LLM with assessing benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 15030–15052\.External Links:[Link](https://aclanthology.org/2025.acl-long.731/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.731),ISBN 979\-8\-89176\-251\-0Cited by:[§D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.21349#S4.SS2.p1.1)\.
- Joshiet al\.\(2015\)A\. Joshi, S\. Kale, S\. Chandel, and D\. K\. PalLikert scale: explored and explained\.British journal of applied science & technology7\(4\),pp\. 396–403\.External Links:[Document](https://dx.doi.org/10.9734/BJAST/2015/14975),[Link](https://doi.org/10.9734/BJAST/2015/14975)Cited by:[§D\.1\.2](https://arxiv.org/html/2609.21349#A4.SS1.SSS2.p1.1),[§5\.4](https://arxiv.org/html/2609.21349#S5.SS4.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. KielaRetrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 9459–9474\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2609.21349#S3.SS4.p2.1)\.
- Liet al\.\(2025\)L\. Li, P\. Cai, R\. A\. Rossi, F\. Dernoncourt, B\. Kveton, J\. Wu, T\. Yu, L\. Song, T\. Yang, Y\. Qin, N\. K\. Ahmed, S\. Basu, S\. Mukherjee, R\. Zhang, Z\. Hu, B\. Ni, Y\. Zhou, Z\. Wang, Y\. Huang, Y\. Wang, X\. Zhang, P\. S\. Yu, X\. Hu, and Y\. ZhaoA personalized conversational benchmark: towards simulating personalized conversations\.arXiv preprint arXiv:2505\.14106\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.14106),[Link](https://arxiv.org/abs/2505.14106)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p2.1),[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.9.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§E\.2](https://arxiv.org/html/2609.21349#A5.SS2.p1.1)\.
- Liuet al\.\(2024\)J\. Liu, Z\. Ni, H\. Que, T\. Sun, Z\. Wang, J\. Yang, J\. Wang, H\. Guo, Z\. Peng, G\. Zhang, J\. Tian, X\. Bu, K\. Xu, W\. Rong, J\. Peng, and Z\. ZhangRoleAgent: building, interacting, and benchmarking high\-quality role\-playing agents from scripts\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 49403–49428\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/5875aca1ef70285a35940afbbce0f9fb-Abstract-Datasets_and_Benchmarks_Track.html)Cited by:[§E\.4](https://arxiv.org/html/2609.21349#A5.SS4.p1.1),[§1](https://arxiv.org/html/2609.21349#S1.p2.1),[§3\.3](https://arxiv.org/html/2609.21349#S3.SS3.p2.1)\.
- Lovakov and Agadullina \(2021\)A\. Lovakov and E\. R\. AgadullinaEmpirically derived guidelines for effect size interpretation in social psychology\.European Journal of Social Psychology51\(3\),pp\. 485–504\.External Links:[Document](https://dx.doi.org/10.1002/ejsp.2752),[Link](https://doi.org/10.1002/ejsp.2752)Cited by:[§4\.2](https://arxiv.org/html/2609.21349#S4.SS2.p1.1)\.
- McHugh \(2012\)M\. L\. McHughInterrater reliability: the kappa statistic\.Biochemia Medica22\(3\),pp\. 276–282\.External Links:[Document](https://dx.doi.org/10.11613/BM.2012.031),[Link](https://doi.org/10.11613/BM.2012.031)Cited by:[§D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px2.p2.1)\.
- Mischel and Shoda \(1995\)W\. Mischel and Y\. ShodaA cognitive\-affective system theory of personality: reconceptualizing situations, dispositions, dynamics, and invariance in personality structure\.\.Psychological review102\(2\),pp\. 246–268\.External Links:[Document](https://dx.doi.org/10.1037/0033-295X.102.2.246),[Link](https://doi.org/10.1037/0033-295X.102.2.246)Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p3.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px1.p2.1)\.
- Münkeret al\.\(2026\)S\. Münker, N\. Schwager, and A\. RettingerDon’t trust generative agents to mimic communication on social networks unless you benchmarked their empirical realism\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),Rabat, Morocco,pp\. 1141–1151\.External Links:[Link](https://aclanthology.org/2026.eacl-long.51/),[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.51),ISBN 979\-8\-89176\-380\-7Cited by:[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.3.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,P\. Isabelle, E\. Charniak, and D\. Lin \(Eds\.\),Philadelphia, Pennsylvania, USA,pp\. 311–318\.External Links:[Link](https://aclanthology.org/P02-1040/),[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[§E\.2](https://arxiv.org/html/2609.21349#A5.SS2.p1.1)\.
- Pearson \(1895\)K\. PearsonNote on regression and inheritance in the case of two parents\.Proceedings of the Royal Society of London58,pp\. 240–242\.External Links:[Document](https://dx.doi.org/10.1098/rspl.1895.0041),[Link](https://doi.org/10.1098/rspl.1895.0041)Cited by:[§D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px2.p1.1)\.
- Peng and Chen \(2026\)J\. Peng and Y\. ChenRethinking role\-playing evaluation: anonymous benchmarking and a systematic study of personality effects\.InProceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue,J\. D\. Choi, Y\. Chen, K\. Funakoshi, and A\. Emami \(Eds\.\),Atlanta, Georgia, USA,pp\. 205–218\.External Links:[Link](https://aclanthology.org/2026.sigdial-1.15/)Cited by:[§E\.4](https://arxiv.org/html/2609.21349#A5.SS4.p1.1),[§1](https://arxiv.org/html/2609.21349#S1.p2.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p1.1),[§5\.5](https://arxiv.org/html/2609.21349#S5.SS5.p1.1)\.
- Rahimzadehet al\.\(2026\)V\. Rahimzadeh, E\. Moosavi Monazzah, M\. T\. Pilehvar, and Y\. YaghoobzadehSynthia: scalable grounded persona generation from social media data\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 45983–46006\.External Links:[Link](https://aclanthology.org/2026.acl-long.2134/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.2134),ISBN 979\-8\-89176\-390\-6Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p2.1),[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.5.1)\.
- Royall and Wortmann \(2015\)E\. Royall and T\. WortmannFinding the state space of urban regeneration: modeling gentrification as a probabilistic process using k\-means clustering and markov models\.InProceedings of the 14th International Conference on Computers in Urban Planning and Urban Management \(CUPUM 2015\): Planning Support Systems and Smart Cities,Cambridge, MA, USA\.Note:Paper 275External Links:[Link](https://web.mit.edu/cron/lockers/cupum2015/proceedings/Content/modeling/275_royall_h.pdf)Cited by:[Appendix C](https://arxiv.org/html/2609.21349#A3.SS0.SSS0.Px1.p1.1)\.
- Santia and Williams \(2018\)G\. Santia and J\. WilliamsBuzzFace: a news veracity dataset with Facebook user commentary and egos\.InProceedings of the International AAAI Conference on Web and Social Media,Vol\.12,pp\. 531–540\.External Links:[Document](https://dx.doi.org/10.1609/icwsm.v12i1.14985),[Link](https://doi.org/10.1609/icwsm.v12i1.14985)Cited by:[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.2.1)\.
- Sellamet al\.\(2020\)T\. Sellam, D\. Das, and A\. ParikhBLEURT: learning robust metrics for text generation\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 7881–7892\.External Links:[Link](https://aclanthology.org/2020.acl-main.704/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.704)Cited by:[§E\.2](https://arxiv.org/html/2609.21349#A5.SS2.p1.1)\.
- Shaoet al\.\(2023\)Y\. Shao, L\. Li, J\. Dai, and X\. QiuCharacter\-LLM: a trainable agent for role\-playing\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13153–13187\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.814/)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p1.1)\.
- Shiet al\.\(2025\)Q\. Shi, C\. E\. Jimenez, S\. Dong, B\. Seo, C\. Yao, A\. Kelch, and K\. R\. NarasimhanIMPersona: evaluating individual level LLM impersonation\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=7qhBXq0NLN)Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p1.1),[§1](https://arxiv.org/html/2609.21349#S1.p2.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p2.1),[§3\.1](https://arxiv.org/html/2609.21349#S3.SS1.p2.1),[§3\.3](https://arxiv.org/html/2609.21349#S3.SS3.p2.1),[§5\.2](https://arxiv.org/html/2609.21349#S5.SS2.p1.1)\.
- Spearman \(1904\)C\. SpearmanThe proof and measurement of association between two things\.The American Journal of Psychology15\(1\),pp\. 72–101\.External Links:ISSN 00029556,[Link](http://www.jstor.org/stable/1412159)Cited by:[§D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px2.p1.1)\.
- Tuet al\.\(2023\)Q\. Tu, C\. Chen, J\. Li, Y\. Li, S\. Shang, D\. Zhao, R\. Wang, and R\. YanCharacterChat: learning towards conversational AI with personalized social support\.arXiv preprint arXiv:2308\.10278\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.10278),[Link](https://arxiv.org/abs/2308.10278)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p2.1)\.
- Tuet al\.\(2024\)Q\. Tu, S\. Fan, Z\. Tian, T\. Shen, S\. Shang, X\. Gao, and R\. YanCharacterEval: a Chinese benchmark for role\-playing conversational agent evaluation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11836–11850\.External Links:[Link](https://aclanthology.org/2024.acl-long.638/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.638)Cited by:[§3\.1](https://arxiv.org/html/2609.21349#S3.SS1.p2.1),[§5\.5](https://arxiv.org/html/2609.21349#S5.SS5.p1.1)\.
- Wanget al\.\(2025a\)H\. Wang, W\. Fu, Y\. Tang, Z\. Chen, Y\. Huang, J\. Piao, C\. Gao, F\. Xu, T\. Jiang, and Y\. LiA survey on responsible LLMs: inherent risk, malicious use, and mitigation strategy\.arXiv preprint arXiv:2501\.09431\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2501.09431),[Link](https://arxiv.org/abs/2501.09431)Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p1.1)\.
- Wanget al\.\(2025b\)L\. Wang, J\. Lian, Y\. Huang, Y\. Dai, H\. Li, X\. Chen, X\. Xie, and J\. WenCharacterBox: evaluating the role\-playing capabilities of LLMs in text\-based virtual worlds\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 6372–6391\.External Links:[Link](https://aclanthology.org/2025.naacl-long.323/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.323),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2024\)N\. Wang, Z\.Y\. Peng, H\. Que, J\. Liu, W\. Zhou, Y\. Wu, H\. Guo, R\. Gan, Z\. Ni, J\. Yang, M\. Zhang, Z\. Zhang, W\. Ouyang, K\. Xu, W\. Huang, J\. Fu, and J\. PengRoleLLM: benchmarking, eliciting, and enhancing role\-playing abilities of large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14743–14777\.External Links:[Link](https://aclanthology.org/2024.findings-acl.878/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.878)Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p2.1),[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px3.p1.1)\.
- Wanget al\.\(2025c\)X\. Wang, H\. Wang, Y\. Zhang, X\. Yuan, R\. Xu, J\. Huang, S\. Yuan, H\. Guo, J\. Chen, S\. Zhou, W\. Wang, and Y\. XiaoCoSER: coordinating LLM\-based persona simulation of established roles\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 64822–64858\.External Links:[Link](https://proceedings.mlr.press/v267/wang25dk.html)Cited by:[Appendix C](https://arxiv.org/html/2609.21349#A3.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2609.21349#A3.p1.1),[§4\.1](https://arxiv.org/html/2609.21349#S4.SS1.p2.1),[§5\.5](https://arxiv.org/html/2609.21349#S5.SS5.p2.1)\.
- Warrens \(2015\)M\. J\. WarrensFive ways to look at Cohen’s kappa\.Journal of Psychology & Psychotherapy5\(4\),pp\. 197\.External Links:[Document](https://dx.doi.org/10.4172/2161-0487.1000197),[Link](https://doi.org/10.4172/2161-0487.1000197)Cited by:[§D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px2.p2.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, b\. ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§5\.2](https://arxiv.org/html/2609.21349#S5.SS2.p1.1)\.
- Wuet al\.\(2021\)Y\. Wu, X\. Ma, and D\. YangPersonalized response generation via generative split memory network\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 1956–1970\.External Links:[Link](https://aclanthology.org/2021.naacl-main.157/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.157)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p2.1),[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.8.1)\.
- Xuet al\.\(2025\)R\. Xu, X\. Wang, J\. Chen, S\. Yuan, X\. Yuan, J\. Liang, Z\. Chen, Xiaoqingdong, and Y\. XiaoCharacter is destiny: can persona\-assigned language models make personal choices?\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15038–15059\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.813/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.813),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2026\)Z\. Xu, D\. Chen, S\. Wang, J\. Li, C\. Wang, M\. Han, and Y\. WangAdaMARP: an adaptive multi\-agent interaction framework for general immersive role\-playing\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 31230–31283\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1563/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1563),ISBN 979\-8\-89176\-395\-1Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2026\)S\. Yang, Z\. Lu, Y\. Yang, B\. Lv, Y\. Shen, and N\. LiuHyCoRA: hyper\-contrastive role\-adaptive learning for role\-playing\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34304–34312\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i40.40727),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/40727)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2025\)J\. Ye, R\. Wang, Y\. Wu, V\. Ma, F\. Fang, F\. Huang, and Y\. LiCPO: addressing reward ambiguity in role\-playing dialogue via comparative policy optimization\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 297–323\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.18/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.18),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025\)Y\. Yu, R\. Yu, H\. Wei, Z\. Zhang, and Q\. QianBeyond dialogue: a profile\-dialogue alignment framework towards general role\-playing language model\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11992–12022\.External Links:[Link](https://aclanthology.org/2025.acl-long.586/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.586),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p1.1),[§1](https://arxiv.org/html/2609.21349#S1.p2.1)\.
- Zhanget al\.\(2025\)H\. Zhang, R\. Luo, X\. Liu, Y\. Wu, T\. Lin, P\. Zeng, Q\. Qu, F\. Fang, M\. Yang, L\. Gao, J\. Song, F\. Huang, and Y\. LiOmniCharacter: towards immersive role\-playing agents with seamless speech\-language personality interaction\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 26318–26331\.External Links:[Link](https://aclanthology.org/2025.acl-long.1276/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1276),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2609.21349#S1.p1.1)\.
- Zhanget al\.\(2018\)S\. Zhang, E\. Dinan, J\. Urbanek, A\. Szlam, D\. Kiela, and J\. WestonPersonalizing dialogue agents: I have a dog, do you have pets too?\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 2204–2213\.External Links:[Link](https://aclanthology.org/P18-1205/),[Document](https://dx.doi.org/10.18653/v1/P18-1205)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p2.1),[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.7.1)\.
- Zhanget al\.\(2020\)T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with BERT\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§E\.2](https://arxiv.org/html/2609.21349#A5.SS2.p1.1)\.
- Zhenget al\.\(2019\)Y\. Zheng, G\. Chen, M\. Huang, S\. Liu, and X\. ZhuPersonalized dialogue generation with diversified traits\.arXiv preprint arXiv:1901\.09672\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.1901.09672),[Link](https://arxiv.org/abs/1901.09672)Cited by:[§2](https://arxiv.org/html/2609.21349#S2.SS0.SSS0.Px2.p2.1),[Table 1](https://arxiv.org/html/2609.21349#S2.T1.2.1.6.1)\.
- Zhouet al\.\(2025\)J\. Zhou, Y\. Huang, B\. Wen, G\. Bi, Y\. Chen, P\. Ke, Z\. Chen, X\. Xiao, L\. Peng, K\. Tang, R\. Zhang, L\. Zhang, T\. Lv, Z\. Hu, H\. Wang, and M\. HuangCharacterBench: benchmarking character customization of large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 26101–26110\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i24.34806),[Link](https://doi.org/10.1609/aaai.v39i24.34806)Cited by:[§D\.2](https://arxiv.org/html/2609.21349#A4.SS2.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.21349#S4.SS2.p1.1)\.

## Appendix ADetail of Constructing Dataset

In this work, we construct the dataset from social media data related to the impersonated influencers, collected internally under the authorization of the platform operator\. To ensure sufficient data coverage, we select influencers who have more than500500comment–reply pairs in the demonstration source and more than120120comment–reply pairs in the candidate pool for testing\.[Table 22](https://arxiv.org/html/2609.21349#A7.T22)reports the dataset statistics for the six influencers studied in our work\.

Furthermore, based on our preliminary analysis, comment–reply pairs exhibit highly distinct interaction types\. Some pairs mainly involve factual information exchange, while others more clearly reveal the influencer’s speaking style or behavioral strategies\. To ensure that the test samples comprehensively covers these diverse interactions, we promptgpt\-4o\-minito assign each comment–reply pair a persona score ranging from 0 to 1, indicating the degree to which the reply exhibits the influencer’s personal characteristic\. We then use this score as the basis for stratified sampling\. Specifically, comment–reply pairs are categorized into three levels according to their persona scores: high\[0\.8,1\.0\]\[0\.8,1\.0\], medium\[0\.5,0\.8\)\[0\.5,0\.8\), and low\[0,0\.5\)\[0,0\.5\)\. Examples of the three levels are shown in[Figure 7](https://arxiv.org/html/2609.21349#A7.F7)\. Low\-level pairs mainly involve factual or experiential information exchange, medium\-level pairs exhibit stylistic cues such as emoji usage, and high\-level pairs clearly reflect the influencer’s personal characteristic\. The prompt used to obtain persona scores is shown in[Figure 14](https://arxiv.org/html/2609.21349#A7.F14)\. The statistics of the test samples for each of the six influencers are summarized in[Table 9](https://arxiv.org/html/2609.21349#A1.T9)\. ForInfluencer D, all available data in the medium and high levels are included because each level contains fewer than5050test samples\. The same logic applies toInfluencer EandInfluencer F\.

To further demonstrate the diversity of our dataset,[Table 10](https://arxiv.org/html/2609.21349#A1.T10)reports the text length distributions and the proportions of the three persona\-score levels across all six influencers\. As shown in the table, textual lengths vary considerably across individuals: average reply lengths range from8\.468\.46to33\.6533\.65characters, while average post lengths range from271\.65271\.65to952\.74952\.74characters\. Moreover, the distribution of persona\-score levels in the candidate pool reflects distinct individual interaction styles\. For instance,Influencer E’s comment–reply pairs predominantly involve low persona\-characteristic exchanges \(87%87\\%\), whereasInfluencer Cdisplays a much more balanced distribution across medium \(36%36\\%\) and low \(53%53\\%\) levels\. These statistics confirm that our dataset encompasses a wide spectrum of behavioral patterns\.

InfluencerPersona ScoreABCDEFHigh50505035509Medium505050502239Low505050501250Table 9:Number of data in the test samples\.InfluencerPost LengthComment LengthReply LengthHigh \(%\)Med \(%\)Low \(%\)A467\.92±\\pm471\.0917\.58±\\pm25\.378\.46±\\pm9\.6913%29%58%B367\.39±\\pm231\.6230\.70±\\pm57\.5510\.01±\\pm10\.5412%35%53%C952\.74±\\pm740\.9266\.83±\\pm105\.5433\.65±\\pm28\.1434%36%53%D355\.21±\\pm508\.8914\.20±\\pm15\.249\.56±\\pm10\.1716%25%59%E271\.65±\\pm221\.0027\.54±\\pm28\.069\.52±\\pm10\.215%8%87%F860\.21±\\pm574\.4126\.53±\\pm29\.4111\.82±\\pm10\.398%33%60%Table 10:Dataset diversity statistics across the six influencers, including text lengths and the proportion of comment–reply pairs across high, medium, and low persona\-score levels\.
## Appendix BDetail of SIBPersona

### B\.1Data\-invariant Persona Construction

For the data\-invariant persona, we construct a short profile for each influencer using the influencer description provided on the social media, rather than deriving it from historical data\. Each profile contains approximately5050–100100words\.

### B\.2Data\-variant Persona Construction

[Table 11](https://arxiv.org/html/2609.21349#A2.T11)reports the size of the dimensional corpora extracted forInfluencer Ain four LLMs, while[Table 12](https://arxiv.org/html/2609.21349#A2.T12)and[Table 13](https://arxiv.org/html/2609.21349#A2.T13)further present the corresponding statistics for the remaining five influencers usingDeepSeek\-v3\.2andgemini\-3\-flash\-preview, respectively\. All models are accessed through cloud\-based APIs: we accessclaude\-opus\-4\-6\(pinned snapshot\) through the Claude API,gemini\-3\-flash\-preview\(December 2025 snapshot\) through Google Cloud Platform \(GCP\), and bothgpt\-5\.2\-2025\-12\-11andDeepSeek\-v3\.2through Azure AI Foundry\. For decoding parameters across tasks, we set\(temperature,top\-​p\)\(\\text\{temperature\},\\text\{top\-\}p\)to\(1\.0,1\.0\)\(1\.0,1\.0\)for recognizing,\(0\.2,0\.8\)\(0\.2,0\.8\)for extraction, and\(0\.8,1\.0\)\(0\.8,1\.0\)for generation\. All other API parameters remain at their default values, with the reasoning effort set to the lowest level supported by each model\.

### B\.3Distinctiveness of Persona Dimensions

Our dimension\-wise retrieval design is based on the hypothesis that the three dimensions of data\-variant persona capture largely distinct aspects of an individual’s persona\. While these dimensions may still exhibit certain correlations, we hypothesize that memory, speaking style, and reaction process encode sufficiently different types of information to be treated separately during retrieval\. Under this hypothesis, the follower’s comment is matched separately against each dimensional corpus, which prevents high\-frequency patterns in one dimension from dominating the retrieval results of the others\.

To empirically examine this hypothesis, we analyze the distinctiveness of the three dimensions in the embedding space\. Specifically, we embedmpj​km\_\{p\_\{j\}k\}for memory,s​ppj​ksp\_\{p\_\{j\}k\}for speaking style, and behaviorbj​kb\_\{jk\}for the reaction process\. We usebj​kb\_\{jk\}rather than the full SIB triplet because behavior, similar to memory and speaking style, directly reflects the impersonated individual’s observable reply patterns, whereas situation and internal state mainly describe the intermediate reasoning process that leads to the behavior\. The resulting vectors are then analyzed through qualitative visualization and quantitative clustering evaluation\. The results provide empirical evidence that the three dimensions are separable, supporting our decision to process them independently during retrieval\.

For the qualitative analysis,[Figure 3](https://arxiv.org/html/2609.21349#A2.F3)shows the t\-SNE visualization of the embeddings ofm​pj​kmp\_\{jk\},s​pj​ksp\_\{jk\}, andbj​kb\_\{jk\}extracted forInfluencer AbyClaude Opus 4\.6\. As illustrated in the figure, the three dimensions occupy different regions in the projected space, suggesting that they capture different aspects of the impersonated individual\. For the quantitative analysis, we further calculate the Silhouette Score across six influencers, using documents extracted byDeepSeek\-v3\.2\. The resulting average Silhouette Score is0\.370\.37\. As a random baseline, we permute the cluster label of all document embeddings and repeat this process 1,000 times\. The average Silhouette Score under this random permutation setting is−0\.001\-0\.001\. The substantial gap between the actual score and the random baseline provides evidence that the three dimensions form distinguishable clusters in the embedding space\.

ModelsMMTTR​PRPGPT\-5\.26,0641,410970Gemini\-3\-flash5,7551,050861Deepseek\-V3\.25,1431,022749Claude\-Opus\-4\.62,307407703Table 11:Number of dimensional documents ofInfluencer A\.InfluencerMMTTR​PRPB12,3588,1626,701C6,1792,9824,049D8,9144,9353,811E10,0872,3932,378F2,587446398Table 12:Number of dimensional documents ofInfluencer B\-Fextracted fromDeepseek V3\.2\.InfluencerMMTTR​PRPB1336082536911C529633213643D1010248394053E1030935363287F2363515449Table 13:Number of dimensional documents ofInfluencer B\-Fextracted fromGemini\-3\-flash\.![Refer to caption](https://arxiv.org/html/2609.21349v1/figure/persona_tsne_claude.png)Figure 3:t\-SNE projection of the embeddings ofm​pj​kmp\_\{jk\},s​pj​ksp\_\{jk\}, andbj​kb\_\{jk\}generated byClaude\-Opus\-4\.6\.

## Appendix CDetails of Reference\-Augmented Evaluation

We adopt the penalty\-based evaluation protocol introduced by[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.21349#bib.bib6)to score RPA\-generated replies\. We usegpt\-4\.1\-2025\-04\-14as LLM evaluator and access it through the Azure AI Foundry\. The temperature is set to0\.20\.2and top\-ppto0\.80\.8to ensure the stability of the evaluation results\.

##### Number of Representative Dimensional Documents

To avoid using the information from the same dimensional corpora for RPA generation and evaluation, we select the representative dimensional documents from the dimensional corpora constructed from the candidate pool for test samples\. Following the rule of thumb discussed by[Royall and Wortmann \(2015\)](https://arxiv.org/html/2609.21349#bib.bib37), we set the number of medoidskkaccording tok=M/2k=\\sqrt\{M/2\}, whereMMdenotes the number of dimensional documents in the corresponding corpus\. We then roundkkup to the nearest integer and use it as the number of representative dimensional documents\.[Table 23](https://arxiv.org/html/2609.21349#A7.T23)reports, for each influencer and extracting model, the total number of extracted dimensional documents and the corresponding number of representative dimensional documents used in evaluation\.

##### Rubrics for LLM Evaluators

Evaluations are conducted along three dimensions: memory, speaking style, and reaction process\. Each dimension is independently evaluated in a separate run\. For each evaluation run, the LLM evaluator is provided with the definition of the target dimension and its corresponding error types as dimension\-specific rubrics\. Using the data\-invariant profile and representative dimensional documents as references, the LLM evaluator identifies flaws in the generated reply and assigns a severity score to each detected flaw\. This design allows the LLM evaluator to focus on one dimension at a time and avoids penalizing errors from other dimensions\. The complete evaluation prompt and evaluation rubrics are shown in[Figure 19](https://arxiv.org/html/2609.21349#A7.F19)and[Figure 20](https://arxiv.org/html/2609.21349#A7.F20), respectively\.

##### Details of the Formula

In the original formulation by[Wang et al\. \(2025c\)](https://arxiv.org/html/2609.21349#bib.bib6), the evaluation score includes a length\-based compensation term in addition to the penalty term, since longer replies are more likely to receive penalties under error\-based scoring schemes\. However, in our setting, the generated replies are typically short, with an average length of approximately1010words\. We therefore omit the length\-based score adjustment in our evaluation\.

##### Visualization for Representative Dimensional Documents

We visualize the representative dimensional documents extracted forInfluencer AbyDeepSeek\-v3\.2\. As shown in[Figure 4](https://arxiv.org/html/2609.21349#A3.F4),[Figure 5](https://arxiv.org/html/2609.21349#A3.F5), and[Figure 6](https://arxiv.org/html/2609.21349#A3.F6), the selected representative dimensional documents are spread across different regions of the projected space rather than being concentrated in a single local cluster\. This qualitative observation suggests that representative dimensional documents capture diverse aspects of the impersonated influencer across the three dimensions\.

![Refer to caption](https://arxiv.org/html/2609.21349v1/figure/rep_memory.png)Figure 4:t\-SNE projection of representative memory documents generated byDeepSeek\-v3\.2\.![Refer to caption](https://arxiv.org/html/2609.21349v1/figure/rep_speaking.png)Figure 5:t\-SNE projection of representative speaking style documents generated byDeepSeek\-v3\.2\.![Refer to caption](https://arxiv.org/html/2609.21349v1/figure/rep_SIB.png)Figure 6:t\-SNE projection of representative reaction process documents generated byDeepSeek\-v3\.2\.

## Appendix DDetails of Human Evaluation

Human evaluations are conducted with two main objectives\. The first objective is to assess the alignment between reference\-augmented evaluation protocol and human judgment\. For this purpose, we use the comparison evaluation to support annotator quality control and the performance evaluation to measure the correlation between human judgment and LLM evaluators\. The second objective is to examine the validity of the reaction\-process\-based design of SIBPersona\. Specifically, the interpretability evaluation assesses whether the retrieved SIB triplets can serve as explanations for generated replies, and the construction faithfulness evaluation verifies whether the constructed SIB triplets are faithful to their corresponding evidence pairs\. The following subsections first describe the settings and annotation protocols of the four human evaluations in Appendix[D\.1](https://arxiv.org/html/2609.21349#A4.SS1)and then present the corresponding results in Appendix[D\.2](https://arxiv.org/html/2609.21349#A4.SS2)\.

### D\.1Human Evaluation Setting

This subsection describes the setting of our human evaluation\. In the following, we refer to each question in the human evaluation inventory as an evaluation item\.

#### D\.1\.1Comparison Evaluation

The comparison evaluation is designed to serve two complementary purposes\. First, it provides a post hoc diagnostic for identifying potentially unreliable annotators\. Second, it offers a comparison\-based assessment of the quality of RPA\-generated replies\. For each evaluation item, annotators are given an influencer’s posti​pip, a follower’s commentf​cfc, and three anonymized candidate replies\. They are asked to rank the three replies according to how closely each reply resembles the impersonated influencer’s actual reply, with ties allowed when multiple replies are considered equally similar\.

Although all three options are presented as generated replies, only two of them are produced by RPA: one by IMPersona and the other by SIBPersona\. The remaining option is the ground\-truth reply written by the influencer\. Since the ground\-truth reply is written by impersonated individual, a reliable annotator is expected not to consistently rank it as the least similar reply\. We therefore use the hidden ground\-truth reply as a quality\-control signal\. Specifically, for each annotator, we compute the proportion of evaluation items in which the ground\-truth reply is ranked last\. Annotators whose ground\-truth\-last rate exceeds a predefined threshold are excluded from subsequent analysis; in our work, the threshold is set to30%30\\%\.

#### D\.1\.2Performance Evaluation

The performance evaluation is designed to assess the alignment between our proposed evaluation protocol and human judgment by measuring the correlation between their scores\. In addition, it allows us to compare the quality of RPA\-generated replies through statistical significance testing\. For each evaluation item, annotators are given an influencer’s post, a follower’s comment, and an RPA\-generated reply generated by IMPersona, Vanilla RAG or SIBPersona\. They are asked to rate the reply along three dimensions: memory, speaking style, and reaction process, using a five\-point Likert scale\([Joshi et al\., 2015](https://arxiv.org/html/2609.21349#bib.bib33)\)\. If an annotator considers a reply irrelevant to a given dimension, they may selectNA\. This option is included to avoid forcing annotators to assign scores to dimensions that are not applicable to a particular reply\.

To addressNAannotations during analysis, we process human evaluation scores as follows\. For each evaluation item and each dimension, we aggregate scores across annotators by averaging all non\-NAratings\. This aggregation yields one human evaluation score for each dimension of each sample and helps reduce the influence of annotator\-specific variation, allowing the subsequent correlation analysis to focus on aggregate human evaluation trends\. If all annotators selectNAfor an evaluation item on a given dimension, the aggregated score for that dimension is treated asNA\. Such cases are excluded from analysis involving the corresponding dimension because no valid human evaluation score is available\.

After aggregating the human evaluation scores, we perform two complementary analysis\. The first analysis examines the alignment between LLM evaluator and human judgment using correlation measures\. The second analysis compares the quality of RPA\-generated replies across IMPersona, Vanilla RAG and SIBPersona based on human evaluation scores\.

For the correlation analysis, we compute both dimension\-specific and overall correlations between LLM evaluator and human judgment scores\. Dimension\-specific correlation is calculated separately for memory, speaking style, and reaction process\. For each dimension, evaluation items withNAin the corresponding aggregated human judgment score are excluded since no valid human score is available for that dimension\. This analysis allows us to examine whether the LLM evaluator aligns with human judgment on each specific dimension\.

In addition, we compute an overall correlation by first averaging the scores across the three dimensions for each evaluation item and then calculating the correlation based on these overall scores\. Compared with the dimension\-specific correlation, this analysis provides an overall view of the alignment between LLM evaluator and human judgment\. For this analysis, evaluation items are excluded if any of the three dimension scores isNA, because the overall score cannot be consistently computed across all dimensions\.

For the method\-level quality comparison, we focus on the three RPA methods: IMPersona, Vanilla RAG and SIBPersona\. For each evaluation dimension, we compute the average human evaluation score of each method and conducttt\-tests with Holm correction to examine whether the score differences are statistically significant\. This analysis complements the correlation analysis by directly comparing the human\-evaluated quality of the three methods\.

#### D\.1\.3Interpretability Evaluation

After evaluating the quality of the RPA\-generated replies, we further conduct evaluations to examine whether the constructed reaction process corpus provides meaningful explanatory evidence for SIBPersona\. We first perform an interpretability evaluation to assess whether the retrieved SIB triplets can serve as plausible explanations for the generated replies\. For each evaluation item, annotators are given an influencer’s post, a follower’s comment, and a reply generated by SIBPersona, together with the top\-three SIB triplets retrieved and provided to the RPA\.

Annotators are asked to evaluate each SIB triplet from two aspects\. First, they assess whether the situation is relevant to the influencer’s post and the follower’s comment\. Second, they assess whether the internal state and behavior can reasonably explain the generated reply\. This design allows us to examine both the contextual relevance of the retrieved situations and the explanatory consistency between the retrieved SIB triplets and the final generated reply\. Ratings are given using a five\-point Likert scale\.

#### D\.1\.4Construction Faithfulness Evaluation

Finally, we conduct a construction faithfulness evaluation to assess whether the generated SIB triplets are reasonably extracted from the original data sample\. This evaluation examines whether each extracted SIB triplet can be reasonably supported by the comment–reply pair assigned as its evidence\. For each evaluation item, annotators are given a comment–reply pair and its corresponding extracted SIB triplet\. They are then asked to the annotator whether it is reasonable to construct the given SIB triplet from the provided comment–reply pair\. This setup allows us to directly evaluate the faithfulness of the corpus construction process, rather than only assessing their groundedness in downstream reply generation\. The evaluation is collected as a binary pass/fail label, wherepassindicates that the triplet is reasonably supported by the source comment–reply pair, andfailindicates that the triplet is unsupported, inconsistent with the comment–reply pair, or overly inferred from the source data\.

#### D\.1\.5Implementation Details

All human evaluations are conducted using test samples generated byDeepSeek\-V3\.2forInfluencer F\. For the performance evaluation, we use3030test samples and evaluate replies generated by three methods, Vanilla RAG, IMPersona and SIBPersona, resulting in9090evaluation items in total\. For the interpretability evaluation, we use the same3030test samples\. Since each test sample contains top\-three retrieved SIB triplets, annotators evaluate each triplet separately\.

The3030test samples are manually selected from the three levels introduced in §[5\.1](https://arxiv.org/html/2609.21349#S5.SS1)to ensure diversity in comment situations\. For the comparison evaluation, we randomly select1010samples from these3030test samples\. For the construction faithfulness evaluation, we use5050representative dimensional documents from reaction process corpus to assess whether the extracted SIB triplets are faithful to their corresponding evidence pairs\.

We recruit nine annotators for the human evaluation\. All annotators have worked withInfluencer Ffor several years, ensuring that they have sufficient familiarity with the impersonated influencer\. To control annotation quality, we further filter annotators based on the ground\-truth\-last rate calculated from the comparison evaluation\. Since only three annotators pass the filtering threshold, we report the results of these three annotators in the following subsection\.

### D\.2Results of Human Evaluation

##### Results of Comparison Evaluation

For the comparison evaluation, we convert the rankings of candidate replies into reciprocal ranking scores, where the score of each reply is computed as the reciprocal of its assigned rank\. We then report the average reciprocal ranking score for each method, with higher scores indicating better rankings\. As shown in[Table 14](https://arxiv.org/html/2609.21349#A4.T14), SIBPersona achieves a significantly higher score than IMPersona and obtains an average score comparable to that of the ground\-truth replies\. This result suggests that annotators judged the replies generated by SIBPersona to be substantially closer to the replies of real influencer’s than those generated by IMPersona\.

MethodsReciprocal Ranking scoreIMPersona0\.42‡SIBPersona0\.71Ground Truth0\.71Table 14:Human comparison evaluation reciprocal scores\.Boldindicates the best score\.‡\\ddagger: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)
##### Results of Performance Evaluation

To examine the alignment between human judgment and LLM evaluators, we compare the aggregated human judgment scores with the LLM evaluator scores and report their correlations\. We report both Pearson’srr\([Pearson, 1895](https://arxiv.org/html/2609.21349#bib.bib32)\)and Spearman’sρ\\rho\([Spearman, 1904](https://arxiv.org/html/2609.21349#bib.bib7)\)to measure linear and rank\-based associations, respectively\. As shown in[Figure 8](https://arxiv.org/html/2609.21349#A7.F8), the overall human judgment score shows a moderate correlation with the overall LLM evaluator score, yielding Pearson’srrof0\.5610\.561and Spearman’sρ\\rhoof0\.5300\.530\. This suggests that the LLM evaluator captures evaluation trends that are broadly aligned with human judgment when the three dimensions are considered jointly\. Among the dimension\-specific correlations, speaking style shows the strongest alignment between human judgment and LLM evaluator, with Pearson’srrof0\.5760\.576and Spearman’sρ\\rhoof0\.5610\.561indicating a moderate correlation\. Memory and reaction process also show significant but weaker correlations, with Pearson’srrof0\.3540\.354and0\.2890\.289, and Spearman’sρ\\rhoof0\.3650\.365and0\.2870\.287, respectively\. These results are consistent with the previous findings in role\-playing evaluation\([Zhou et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib13);[He et al\., 2025](https://arxiv.org/html/2609.21349#bib.bib5)\)and demonstrate that our reference\-augmented protocol serves as a reasonable proxy for human judgment\.

Moreover, we further assess inter\-rater agreement among human annotators, computed by the weighted Cohen’sκ\\kappa\([Warrens, 2015](https://arxiv.org/html/2609.21349#bib.bib34)\)\. The agreement scores across the three dimensions are presented in[Table 15](https://arxiv.org/html/2609.21349#A4.T15)\. The pairwise agreement scores range from0\.3640\.364to0\.5080\.508\. Following the interpretation guidelines of[McHugh \(2012\)](https://arxiv.org/html/2609.21349#bib.bib35), these results indicate a fair to moderate level of agreement across annotators\. The agreement is highest between Rater 1 and Rater 3, while Rater 1 and Rater 2 show relatively lower agreement\. This pattern suggests that the annotation task involves subjective judgment, but the annotators still exhibit meaningful consistency\.

For the method\-level quality comparison, as shown in[Table 5](https://arxiv.org/html/2609.21349#S5.T5), SIBPersona significantly outperforms IMPersona across all three dimensions\.

MMTTR​PRPRater 1 & 20\.3640\.4290\.406Rater 1 & 30\.4170\.5060\.508Rater 2 & 30\.4260\.4950\.487Table 15:Inter\-rater agreement among the three annotators\.
##### Results of Interpretability Evaluation

For the interpretability evaluation, each test sample is associated with three retrieved SIB triplets\. Since the RPA may rely more heavily on one of the retrieved triplets during generation, we take the highest score among the three triplets as the score for that test sample\. Specifically, lets​c​o​r​ei,jSscore^\{S\}\_\{i,j\}ands​c​o​r​ei,jI​Bscore^\{IB\}\_\{i,j\}denote the average annotator scores for thejj\-th SIB triplet of theii\-th test sample, wherej∈\{1,2,3\}j\\in\\\{1,2,3\\\}\. The final scores, denoted asS​c​o​r​eSScore\_\{S\}andS​c​o​r​eI​BScore\_\{IB\}, are computed by averaging the highest triplet\-level score across allNNtest samples:

S​c​o​r​eS=1N​∑i=1Nmaxj∈\{1,2,3\}⁡s​c​o​r​ei,jS,\\displaystyle Score\_\{S\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\max\_\{j\\in\\\{1,2,3\\\}\}score^\{S\}\_\{i,j\},S​c​o​r​eI​B=1N​∑i=1Nmaxj∈\{1,2,3\}⁡s​c​o​r​ei,jI​B\.\\displaystyle Score\_\{IB\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\max\_\{j\\in\\\{1,2,3\\\}\}score^\{IB\}\_\{i,j\}\.
Our RPA replies achieve aS​c​o​r​eSScore\_\{S\}of3\.633\.63and aS​c​o​r​eI​BScore\_\{IB\}of3\.573\.57\. The highS​c​o​r​eSScore\_\{S\}indicates that the situation described in the retrieved SIB triplet generally align with the corresponding follower’s comment\. The highS​c​o​r​eI​BScore\_\{IB\}further suggests that the internal states and behaviors described in the triplets can reasonably explain the final replies generated by the RPA\.

In addition, we useS​c​o​r​eI​BScore\_\{IB\}as the criterion for categorizing test samples into interpretable and uninterpretable, as shown in[Table 6](https://arxiv.org/html/2609.21349#S5.T6)\. For each test sample, if the averageS​c​o​r​eI​BScore\_\{IB\}assigned by the three annotators exceeds33, the test sample is considered interpretable; otherwise, it is considered uninterpretable\. These results suggest that our retrieval method can identify situations similar to an unseen follower’s comment, and the retrieved internal states and behaviors serve as a valid supporting context in the RPA\-generated replies\.

##### Results of Extraction Faithfulness Evaluation

To evaluate the faithfulness of the reaction process corpus, we report the pass rates assigned by the three annotators in[Table 16](https://arxiv.org/html/2609.21349#A4.T16)\. The individual annotator pass rates are consistently high, with all raters assigning pass rates above0\.800\.80\. When applying majority voting across annotators, the overall pass rate reaches0\.940\.94with a 95% confidence interval of\[0\.84,0\.98\]\[0\.84,0\.98\]\.

These results suggest that most constructed SIB triplets are reasonably supported by their corresponding comment–reply pair\. In other words, our process of constructing dimensional corpora can generally preserve information from the original comment–reply pairs while converting them into structured SIB triplets\. The higher majority\-vote pass rate further indicates that although the annotators may differ in their strictness, there is strong aggregate support for the faithfulness of the constructed reaction process corpus\.

Pass Rate95% CIRater 10\.80\[0\.67, 0\.89\]Rater 20\.84\[0\.72, 0\.92\]Rater 30\.92\[0\.81, 0\.97\]Total0\.94\[0\.84, 0\.98\]Table 16:Extraction faithfulness evaluated by the three annotators\. TheTotalcolumn reports the pass rate of by applying majority\-vote across all annotators\.

## Appendix EAdditional Results

### E\.1Per\-Influencer Results

[Table 24](https://arxiv.org/html/2609.21349#A7.T24)presents the detailed results forInfluencer BthroughInfluencer F, which correspond to the aggregated results reported in[Table 3](https://arxiv.org/html/2609.21349#S4.T3)\. The table provides a more fine\-grained view of our experimental results in §[5\.3](https://arxiv.org/html/2609.21349#S5.SS3)\. Overall, our method consistently outperforms the baseline in both the speaking style dimension and reaction process dimension\.

### E\.2Comparison with Traditional NLP Metrics

To further justify the use of our reference\-augmented evaluation protocol, we compare its alignment with human judgment against several traditional automatic evaluation metrics, including BLEU\([Papineni et al\., 2002](https://arxiv.org/html/2609.21349#bib.bib21)\), ROUGE\-L\([Lin, 2004](https://arxiv.org/html/2609.21349#bib.bib20)\), BERTScore\([Zhang et al\., 2020](https://arxiv.org/html/2609.21349#bib.bib38)\), and BLEURT\([Sellam et al\., 2020](https://arxiv.org/html/2609.21349#bib.bib39)\)\. Specifically, we compute the Spearman correlation between each metric and human judgment\. The results show that traditional metrics exhibit substantially weaker correlations with human judgment: BLEU achieves a Spearman’sρ\\rhoof−0\.01\-0\.01, ROUGE\-L achieves−0\.04\-0\.04, BERTScore achieves0\.280\.28, and BLEURT achieves0\.020\.02\. In contrast, our reference\-augmented evaluation protocol achieves a Spearman’sρ\\rhoof0\.5300\.530, showing a much stronger alignment with human judgment\.

This gap suggests that traditional automatic metrics are not sufficient for evaluating role\-playing real\-individual on social media\. One possible reason is that a generated reply may differ substantially from the ground\-truth reply at the lexical or surface\-semantic level, while still adopting a behavioral strategy that is consistent with the impersonated individual\. For example, as shown in[Figure 9](https://arxiv.org/html/2609.21349#A7.F9), the RPA\-generated reply follows the influencer’s behavioral strategy and is therefore rated highly by both the LLM evaluator and human evaluators\. However, because this reply has low lexical and surface\-semantic similarity to the ground\-truth reply, it receives low scores from traditional automatic metrics\.

These findings support our decision to report reference\-augmented evaluation as the main automatic evaluation protocol in this work, as it provides a more fine\-grained assessment of whether the generated replies align with the impersonated individuals\.

### E\.3Token Budgets

[Table 17](https://arxiv.org/html/2609.21349#A5.T17)reports the average number of additional context tokens inserted into the prompt per query, evaluated across the six influencers usingDeepSeek\-V3\.2\. As shown in the results,SIBPersonaconsumes fewer tokens thanIMPersonawhile achieving superior role\-playing performance\. This indicates that performance gains are not simply driven by increasing context length, but rather rely on supplying structured, behavior\-aware, and effective persona information\.

Method\# TokensVanilla RAG593\.51±\\pm456\.37IMPersona1972\.64±\\pm816\.14SIBPersona1630\.83±\\pm812\.02Table 17:Average number of tokens inserted into the prompt per test query across six influencers usingDeepSeek\-V3\.2\.
### E\.4Result of RoleAgentBench

We further evaluate our method on an additional open\-source benchmark, RoleAgentBench\([Liu et al\., 2024](https://arxiv.org/html/2609.21349#bib.bib12)\), a bilingual \(Chinese–English\) role\-playing of fictional character benchmark\. To examine whether SIBPersona is applicable beyond the social media setting, we select two response generation tasks from the benchmark:General ResponseandSummary\. In the experiments,gpt\-4o\-miniis used to construct the dimensional corpus,OLMo\-3\-7B\-Instructis used as the generation model, andgemini\-2\.0\-flashserves as the LLM evaluator\. During experimentation, we follow[Peng and Chen \(2026\)](https://arxiv.org/html/2609.21349#bib.bib23)and anonymize the character names in the dataset to mitigate the influence of the model’s prior knowledge during pre\-training\.

For evaluation, we use two complementary protocols: our proposed reference\-augmented evaluation protocol and the original evaluation protocol provided by the benchmark\. Under the benchmark protocol, responses are evaluated in a pairwise comparison setting\. Specifically, its LLM evaluator is presented with two responses generated by SIBPersona as well as IMPersona and asked to select the one that better resembles the target character\. To mitigate potential positional bias caused by response ordering in the prompt, we conduct two evaluation rounds with the order of the two responses swapped\. A method is counted as the winner only if its response is selected in both rounds, while it is counted as losing only if the competing method is selected in both rounds\. Cases with inconsistent outcomes across the two rounds are treated as ties\.

The experimental results \([Table 25](https://arxiv.org/html/2609.21349#A7.T25)\) show that our method significantly outperforms the baseline under both evaluation protocols, consistent with the findings of our main experiment\. This suggests that SIBPersona is also applicable to fictional character benchmarks\. For completeness, we also provide the prompts used in the benchmark setting\.[Figure 15](https://arxiv.org/html/2609.21349#A7.F15)shows the prompt for dimensional relevance filtering, while[Figure 16](https://arxiv.org/html/2609.21349#A7.F16),[Figure 17](https://arxiv.org/html/2609.21349#A7.F17), and[Figure 18](https://arxiv.org/html/2609.21349#A7.F18)present the prompts used for dimensional document extraction\. The prompt used for the penalty\-based LLM evaluation is shown in[Figure 19](https://arxiv.org/html/2609.21349#A7.F19), with the corresponding evaluation rubrics provided in[Figure 20](https://arxiv.org/html/2609.21349#A7.F20), and[Figure 21](https://arxiv.org/html/2609.21349#A7.F21)presents the prompt used for response generation\.

## Appendix FValidation of Reference\-Augmented Evaluation

Automated evaluation for role\-playing real individuals remains a challenging and underexplored problem due to the lack of static gold\-standard personas\. To validate that our reference\-augmented evaluation protocol provides a reasonable assessment of role\-playing fidelity, we conduct empirical investigations addressing potential issues\.

### F\.1Alignment with Human Judgment

A central objective of an automated evaluation protocol is to accurately reflect human perception\. To justify our design choice of using SIBPersona\-structured documents from the test candidate pool as references, we compare the correlation between LLM evaluator scores and human judgement under three different reference configurations\. As reported in[Table 20](https://arxiv.org/html/2609.21349#A6.T20), our proposed configuration achieves the highest alignment with human judgment, outperforming raw comment–reply pairs from the same pool as well as structured documents extracted from the demonstration source\. These results confirm that structured SIBPersona representations best approximate human evaluative criteria\. Furthermore, extracting references from the demonstration source—the exact corpus observed during generation—tends to reward surface context\-matching rather than true behavioral fidelity, which explains its lower correlation with human judgment\.

### F\.2Robustness Against Test Sample Overlap

We further examine whether gold reply leakage occurs when representative reference documents are sampled from the test pool\. BecauseKK\-medoids selects only⌈N/2⌉\\lceil\\sqrt\{N/2\}\\rceilcluster medoids fromNNcandidate instances to ensure semantic diversity rather than exhaustive coverage, the empirical overlap between representative documents and test samples is exceptionally low:0\.35%0\.35\\%for Memory,4\.01%4\.01\\%for Speaking Style, and4\.68%4\.68\\%for Reaction Process onInfluencer A\. To strictly rule out any leakage bias, we re\-evaluate all baseline models after excluding all overlapping instances\. As shown in[Table 18](https://arxiv.org/html/2609.21349#A6.T18), the performance rankings and relative improvements remain identical to our primary findings, confirming that the minimal overlap does not affect evaluation outcomes\.

MethodMMTTR​PRPNaive95\.8946\.43‡70\.85‡CoT96\.5845\.73‡72\.33‡IMPersona95\.9448\.10‡73\.20‡Vanilla RAG95\.3254\.12‡73\.68‡SIBPersona95\.4659\.2277\.81Table 18:Evaluation results onInfluencer Aaveraged over four models after excluding overlapping test instances\. ‡: significant degradation compared to SIBPersona \(p < 0\.05\)\.
### F\.3Evaluation Under Strictly Independent Demonstration Sources

To completely eliminate the risk of test pool contamination and assess the generalizability of our protocol, we conduct a conservative check where references are constructed entirely from the independent demonstration source\.[Table 19](https://arxiv.org/html/2609.21349#A6.T19)and[Table 21](https://arxiv.org/html/2609.21349#A6.T21)report the results\. Across all tested models and influencers, SIBPersona consistently outperforms all baselines on bothTTandR​PRPdimensions\. The persistent advantages across diverse reference configurations—test candidate pool and demonstration source—demonstrate that our evaluation protocol and empirical conclusions are robust and not artifacts of specific reference construction strategies\.

MethodsMTRPNaive97\.2846\.48‡77\.33‡CoT97\.6147\.93‡78\.08‡IMPersona97\.1448\.63‡78\.80‡Vanilla RAG97\.0258\.93‡79\.18‡SIBPersona96\.6962\.3781\.52Table 19:Results onInfluencer Aaveraged over four models using evaluator references extracted from the demonstration source\.‡: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.Reference FormatSource of ReferencePearsonSpearmanRaw comment–reply pairsTest candidate pool0\.3550\.409SIBPersona documentDemonstration source0\.4480\.462SIBPersona documentTest candidate pool \(Ours\)0\.5610\.530Table 20:Correlation between different evaluator configurations and human judgments\.MethodsDeepseek\-V3\.2Gemini\-3\-flashMMTTR​PRPMMTTR​PRPNaive91\.34±\\pm3\.2442\.63‡±\\pm9\.1970\.83‡±\\pm5\.4992\.18±\\pm2\.8344\.21‡±\\pm10\.8468\.91‡±\\pm7\.71CoT91\.06±\\pm3\.3546\.65‡±\\pm7\.3671\.56‡±\\pm4\.7992\.14±\\pm1\.7145\.57‡±\\pm8\.6368\.53‡±\\pm7\.39IMPersona91\.43±\\pm2\.6148\.44‡±\\pm5\.6173\.04‡±\\pm5\.2091\.34±\\pm2\.3646\.32‡±\\pm8\.2069\.88‡±\\pm7\.94Vanilla RAG91\.30±\\pm3\.2459\.18‡±\\pm4\.5874\.30‡±\\pm4\.2692\.37±\\pm2\.6056\.47‡±\\pm3\.0772\.49‡±\\pm3\.97SIBPersona92\.01±\\pm3\.5362\.96±\\pm4\.2777\.18±\\pm4\.4492\.29±\\pm2\.4859\.48±\\pm3\.0075\.60±\\pm5\.10Table 21:Evaluation results across the other five influencers using references extracted from the demonstration source\.‡\\ddagger: significant degradation compared to SIBPersona \(p<0\.05p<0\.05\)\.

## Appendix GExamples and Case Study

This section presents qualitative analysis of the retrieved dimensional documents in Appendix[G\.1](https://arxiv.org/html/2609.21349#A7.SS1), representative success and failure reply cases in Appendix[G\.2](https://arxiv.org/html/2609.21349#A7.SS2), and an example showing how SIBPersona supports error tracing and correction in Appendix[G\.3](https://arxiv.org/html/2609.21349#A7.SS3)\.

### G\.1Examples of Retrieved Dimensional Document

[Figure 10](https://arxiv.org/html/2609.21349#A7.F10)and[Figure 11](https://arxiv.org/html/2609.21349#A7.F11)present examples of the top\-1 retrieved dimensional documents\. Given a follower’s comment that praises the influencer’s appearance, the retrieved situation closely matches the comment context\. The generated reply also reflects the retrieved speaking style and behavioral strategy, avoiding a direct acceptance of the compliment\. In contrast, the memory dimension retrieves less relevant information, which is expected because factual knowledge is less central to this type of comment\. Annotators familiar withInfluencer Afurther confirm that the influencer typically does not directly accept compliments about their appearance\. This case suggests that our method can retrieve information of situation\-relevant reaction process and produce a reply consistent with the influencer’s characteristics\.

### G\.2Examples of Successes and Failure Cases

In addition,[Figure 12](https://arxiv.org/html/2609.21349#A7.F12)illustrates that our method achieves better role\-playing fidelity than the baseline methods\. Across the two examples, the ground\-truth replies are relatively concise and reflect the influencer’s characteristic way of reacting to followers: giving a brief acknowledgment in reply to praise and using a sharp, indirect reply when challenged\. SIBPersona better preserves these pragmatic reply patterns\. For example, when the follower praises the influencer, our method generates a short positive reaction that is close to the ground\-truth reply\. When the follower comments sarcastically with hindsight, our method returns with a sarcastic counter\-reaction, which better matches the interactional intent of the ground truth\.

In contrast, the baseline methods often deviate from the impersonated individual’s reply pattern in different ways\. Naive and CoT tend to introduce excessive explanatory content, such as trading strategies, app usage details, or additional contextual assumptions that are not present in the ground\-truth replies\. Similarly, IMPersona frequently generates overly detailed and elaborated replies, suggesting that hierarchical memory alone is not enough for modeling behavioral strategies\. Vanilla RAG produces shorter replies, but its replies are often generic and fail to capture the specific behavioral strategy required by the comment context\.

In contrast,[Figure 13](https://arxiv.org/html/2609.21349#A7.F13)highlights the limitations of our approach\. When the comment involves real\-time information, our method may misinterpret the situation due to insufficient knowledge of the current context\. In addition, for sarcastic comments, the embedding\-based retrieval process may fail to capture the follower’s ironic intent\. This can lead to the retrieval of SIB triplets that are mismatched with the actual situation, causing the generated reply to follow an inappropriate behavioral strategy\. These cases suggest that our method remains sensitive to the quality of situation understanding and retrieval\. This limitation is especially pronounced when the input requires real\-time knowledge or reasoning beyond surface\-level semantic similarity\.

### G\.3Tracing and Correcting Behavioral Errors

Even when the system makes mistakes, SIBPersona allows us to effectively trace the source of the error and adjust RPA properly\. For instance, we observed a case whereInfluencer Areplied to a fan’s generic"Thank you"with a highly user\-specific follow\-up question:"Has that matter been taken care of?"\. After investigation, we found that the LLM initially yields a reaction process document with this outlier behavior, even though such a reply would be inappropriate for other users\. A key advantage of SIBPersona is adjustable: because the generated replies can be traced back to specific dimensional documents\. By simply deleting the specific outlier document from the dimensional corpus, we can immediately and precisely adjust the model’s behavior\.

InfluencerSplitData typeABCDEFDemonstrationPost1955271671329032130453DemonstrationComment\-Reply pair136115240555360874383705Test CandidatesPost6522855419228784Test CandidatesComment\-Reply pair5471719361227261120Table 22:Number of data in the training set and the test candidate pool\.InfluencersModelsMMTTR​PRPAllRep\.AllRep\.AllRep\.ADeepSeek\-V3\.22424354461527412Claude\-Opus\-4\.61156251891024412Gemini\-3\-flash2477364191531813GPT\-5\.22926395881837514BDeepSeek\-V3\.21786308512175820Gemini\-3\-flash1628298632172219CDeepSeek\-V3\.2491162001027012Gemini\-3\-flash420152621229713DDeepSeek\-V3\.276520184101359Gemini\-3\-flash75520200101589EDeepSeek\-V3\.21314261811017910Gemini\-3\-flash1230252671221911FDeepSeek\-V3\.2521171118947Gemini\-3\-flash476161178937Table 23:Statistics of extracted documents and representative documents across influencers and models\.Alldenotes the total number of dimensional documents extracted from the candidates for testing, andRep\.denotes the number of representative dimensional documents selected from the corresponding dimensional corpus\.Figure 7:Examples of three level of data forInfluencer A\.![Refer to caption](https://arxiv.org/html/2609.21349v1/figure/correlation_plot_camera.png)Figure 8:Correlation matrix between LLM and human evaluations\. \*\*\*p<0\.001p<0\.001, \*\*p<0\.01p<0\.01, \*p<0\.05p<0\.05\.MethodsBCDEFMMTTR​PRPMMTTR​PRPMMTTR​PRPMMTTR​PRPMMTTR​PRPDeepseek v3\.2Naive97\.6745\.60‡78\.07‡88\.9752\.00‡69\.57‡91\.5937\.22‡60\.93‡90\.2426\.85‡64\.17‡90\.6640\.97‡66\.22‡CoT97\.8745\.63‡79\.80‡89\.5753\.00‡71\.4390\.8943\.96‡62\.22‡90\.3028\.15‡65\.30‡90\.8240\.87‡66\.48‡IMPersona95\.7746\.47‡81\.43‡89\.7054\.43‡71\.6790\.7042\.44‡64\.63‡90\.6536\.73‡66\.19‡90\.2042\.35‡67\.81Vanilla RAG97\.3360\.5081\.33‡88\.2055\.43‡70\.8390\.8158\.41‡70\.44‡89\.8253\.8769\.7089\.3455\.1066\.33‡SIBPersona97\.7761\.6384\.7388\.4059\.3772\.9392\.5663\.9374\.4489\.7059\.8873\.3990\.7760\.0570\.71Gemini\-3\-flashNaive96\.8047\.77‡79\.60‡90\.9749\.60‡66\.80‡91\.7844\.11‡73\.52‡91\.0729\.46‡57\.44‡90\.5137\.76‡70\.15‡CoT96\.7047\.33‡78\.97‡89\.3748\.73‡67\.37‡92\.1145\.96‡71\.22‡90\.6031\.85‡59\.23‡90\.8236\.22‡70\.87‡IMPersona94\.8048\.33‡80\.53‡90\.2051\.30‡67\.90‡91\.3746\.30‡73\.37‡89\.5233\.63‡58\.63‡89\.8039\.54‡71\.99‡Vanilla RAG96\.6757\.2780\.93‡90\.8349\.70‡67\.50‡91\.3055\.8570\.93‡89\.7058\.4568\.5190\.7745\.61‡73\.37SIBPersona96\.1760\.9384\.9390\.1055\.3071\.4392\.1957\.8576\.5290\.7759\.8870\.7190\.6652\.0976\.17

Table 24:Results ofInfluencer B–F\.Bold: best results;‡: significant degradation compared to SIBPersonap<0\.05p<0\.05\.LLM penalty\-based evaluationPair\-wise EvaluationTasksMethodsMMTTR​PRPWinTieLoseGeneralIMPersona75\.3265\.1071\.15\-\-\-GeneralSIBPersona76\.6466\.75†73\.97†0\.430\.320\.25SummaryIMPersona75\.2862\.3770\.44\-\-\-SummarySIBPersona78\.94†63\.8373\.12†0\.340\.390\.27Table 25:Experimental results on RoleAgentBench, comparing our method with the baseline\.Boldindicates the better score between the two methods\.†denotes a result that is significantly better than its counterpart \(p<0\.05p<0\.05\)\.Figure 9:An example where both human and LLM evaluators assign high scores, while traditional metrics assign low scores\.Figure 10:Example of top\-1 dimensional document retrieval\. The influencer names in the text are anonymized and replaced with "Influencer"\.Figure 11:English translation of the example for top\-1 dimensional document retrieval\. The influencer names in the text are anonymized and replaced with "Influencer"\.Figure 12:Qualitative examples where our method performs well\. The influencer names in the text are anonymized and replaced with "Influencer"\.Figure 13:Qualitative examples where our method performs poorly\. The influencer names in the text are anonymized and replaced with "Influencer"\.System prompt:You are a model specialized in identifying information contained in text\.User prompt:You are a text classification and scoring assistant specializing in evaluating the "degree of persona/personal style exhibition of \{person\_name\}\." Please review a fan comment and the corresponding reply from \{person\_name\}, then provide a score between 0 and 1 based on the following criteria:\#\#\# Scoring Criteria:\- \*\*0\.0\*\*: The reply is entirely neutral, robotic, or devoid of personal character\. It may consist of purely technical or stock market\-related information \(e\.g\., "Thank you," "Hold this stock"\)\.\- \*\*1\.0\*\*: The reply fully demonstrates a distinct persona, characterized by:\- Clear emotions \(e\.g\., excitement, anger, or being touched\)\.\- Signature tone or catchphrases unique to the persona\.\- Colloquial language, humor, or slang that resonates with the audience\.\- Subjective perspectives or value judgments\.\- Emojis \(Note: Emojis only contribute to the score when accompanied by text; \*\*standalone emojis do not increase the score\*\*\)\.\#\#\# Important Constraints:\- \*\*Responses consisting only of emojis or brief phrases \(e\.g\., "\[heart emoji\]", "Thanks for the support\!"\) must not exceed 0\.6\*\*\.\- \*\*Purely informational or technical answers that lack emotion or personal tone must not exceed 0\.4\*\*\.\- Please consider the \*\*length, emotional intensity, and linguistic style\*\* as these directly influence the perceived degree of persona exhibition\.\#\#\# Output Format:Output JSON only in the following format:\{\{"score": 0\.xx\}\}\#\#\# Data for Evaluation:Fan: \{comment\}\{person\_name\}: \{reply\}Figure 14:English translation of the persona scoring prompt used in our dataset\.System prompt:You are a model specialized in identifying information contained in text\.User prompt:You are an expert in dialogue analysis and categorization\.I will provide you with a dialogue that \{person\_name\} participated in\. Your task is:1\. Analyze the dialogue and determine which aspects of \{person\_name\} the information relates to\. Choose from the following:\- Memory: Factual information about \{person\_name\}, similar to long\-term semantic memory, including \{person\_name\}’s background, knowledge they should possess, etc\. \(e\.g\., \{person\_name\} was born in Shanghai; \{person\_name\}’s specialty is transformation magic; Bob is \{person\_name\}’s master\.\)\- Speaking\_Style: Information related to \{person\_name\}’s way of speaking or verbal habits \(e\.g\., \{person\_name\} often begins sentences with “Ha\!”\)\- SIB \(Situation–Internal state–Behavior\): Information about \{person\_name\}’s behavioral patterns in conversations \(e\.g\., \{person\_name\} uses a mocking tone when facing enemies; \{person\_name\} responds humbly when praised\.\)2\. You may select multiple categories, or none if appropriate\. Please make the most suitable judgment based on the dialogue\.3\. Output your response in the following format, filling in your analysis and the categories contained in the dialogue:\`\`\`json\{\{"analysis": "\.\.\.", "categorys": \["category1", "category2"\]\}\}\`\`\`Note: If the dialogue do not clearly correspond to any category, output an empty list for "categorys", but you must still provide an analysis\.Below is an example:Dialogue and comment thread:\{\{"diag\_id": 0,"dialogue": \[\{\{"role": "Master","content": "What is your surname? What is your name? So you were born of heaven and earth\. From your appearance and behavior, you look like a monkey\. You shall take the surname Sun, and your Dharma name shall be Sun Wukong\. How about that?"\}\},\{\{"role": "Sun Wukong","content": "Good, good, good\! Today I finally have a surname and a name\. I will be called Sun—Wu—Kong\! My name is Sun Wukong\! My name is Sun Wukong\!"\}\}\]\}\}Output:\{\{"analysis": "This dialogue shows \{person\_name\}’s personal history and speaking style\. \{person\_name\} receives the Dharma name ’Sun Wukong’ and previously had no surname, which falls under Memory\. In addition, \{person\_name\} refers to himself as a disciple when speaking to his master, which falls under Speaking\_Style\.", "categorys": \["Memory", "Speaking\_Style"\]\}\}Below is the dialogue:\{dialogue data\}Please output strictly following the required format\.Figure 15:Recognition prompt used in RoleAgentBench\. All datasets share the same prompt except that the few\-shot examples are replaced with the examples from the corresponding dataset\.System prompt:You are a model specialized in identifying information contained in text\.User prompt:You are an expert skilled at extracting semantic memory from a dialogue\.I will provide you with a segment of dialogue involving \{person\_name\}\. Your task is:1\. Extract the semantic memory related to \{person\_name\} from the text of this dialogue\.2\. Semantic memory includes verifiable facts related to \{person\_name\}, such as that \{person\_name\}’s birthplace is Shanghai; and the domain knowledge possessed by \{person\_name\}, such as that the Shu Kingdom and the Wei Kingdom were enemies\.3\. When formulating the memory, please use the text content to converge and infer zero to multiple claims\. Each claim should describe a single concept and express that claim in concise language\. There is no limit to the number of claims to be outputted\.4\. Output must strictly follow the format below, filling the \`claim\` field with the memory points contained in the dialogue and providing a confidence score of 0 to 1 for your analysis:\`\`\`json\{\{"items": \[\{\{"claim": "claim 1", "confidence": <your evidence score\>\}\}, \{\{"claim": "claim 2", "confidence": <your evidence score\>\}\}\]\}\}\`\`\`Supplement: If there are no obvious claims to extract from the dialogue content, the items field should output an empty list\.The following is an example:Dialogue Content:\{\{"diag\_id": 0,"dialogue": \[\{\{"role": "Master","content": "What is your surname? What is your name? You were born from Heaven and Earth\. I see your appearance and demeanor resemble a macaque, so your surname shall be Sun, and your religious name shall be Sun Wukong, does that sound good?",\}\},\{\{"role": "Sun Wukong","content": "Good, good, good\! Disciple finally has a name and surname today\. I’ll be called Sun—Wu—Kong\! My name is Sun Wukong\! My name is Sun Wukong\!"\}\}\]\}\}Output:\`\`\`json\{\{"items": \[\{\{"claim": "\{person\_name\} looks like a monkey and originally did not have a name", "confidence": 0\.8\}\}, \{\{"claim": "\{person\_name\}’s religious name was given by the Patriarch as Sun Wukong", "confidence": 0\.9\}\}\]\}\}\`\`\`Dialogue Content:\{dialogue\_data\}Please strictly output according to the required format\.Figure 16:Memory extraction prompt used in RoleAgentBench\. All datasets share the same prompt except that the few\-shot examples are replaced with the examples from the corresponding dataset\.System prompt:You are a model specialized in identifying information contained in text\.User prompt:You are an expert skilled at extracting the speaking style of an individual from a dialogue\.I will provide you with a segment of dialogue involving \{person\_name\}\. Your task is:1\. Extract the speaking style of \{person\_name\} contained within the text of this dialogue\.2\. Speaking style includes \{person\_name\}’s word choices, verbal tics \(catchphrases\), etc\. For example, \{person\_name\} frequently uses "Haha" at the end of a sentence\.3\. When formulating the speaking style, please use the text content to converge and infer zero to multiple claims\. Each claim should describe a single concept and express that knowledge point in concise language\. There is no limit to the number of knowledge points \(claims\) to be outputted\.4\. Output must strictly follow the format below, filling the \`claim\` field with the speaking styles contained in the dialogue and providing a confidence score of 0 to 1 for your analysis:\`\`\`json\{\{"items": \[\{\{"claim": "claim 1", "confidence": <your evidence score\>\}\}, \{\{"claim": "claim 2", "confidence": <your evidence score\>\}\}\]\}\}\`\`\`Supplement: If there are no obvious claims to extract from the dialogue content, the items field should output an empty list\.The following is an example:Dialogue Content:\{\{"diag\_id": 0,"dialogue": \[\{\{"role": "Master","content": "What is your surname? What is your name? You were born from Heaven and Earth\. I see your appearance and demeanor resemble a macaque, so your surname shall be Sun, and your religious name shall be Sun Wukong, does that sound good?",\}\},\{\{"role": "Sun Wukong","content": "Good, good, good\! Disciple finally has a name and surname today\. I’ll be called Sun—Wu—Kong\! My name is Sun Wukong\! My name is Sun Wukong\!"\}\}\]\}\}Output:\`\`\`json\{\{"items": \[\{\{"claim": "\{person\_name\} refers to himself as ’disciple’ when addressing the Master", "confidence": 0\.8\}\}, \{\{"claim": "\{person\_name\} repeats phrases or sentences when speaking", "confidence": 0\.7\}\}\]\}\}\`\`\`Dialogue Content:\{dialogue\_data\}Please strictly output according to the required format\.Figure 17:Speaking style extraction prompt used in RoleAgentBench\. All datasets share the same prompt except that the few\-shot examples are replaced with the examples from the corresponding dataset\.System prompt:You are a model specialized in identifying information contained in text\.User prompt:You are a Research Assistant familiar with the "Cognitive\-Affective Processing System \(CAPS\)" and skilled at inferring the internal thoughts and behavioral patterns of \{person\_name\} from a dialogue\.\[Task Objective\]I will provide you with a segment of dialogue involving \{person\_name\}\. Your task is:Based on the CAPS theory, extract the S\-I\-B triplets \(Situation\-Internal state\-Behavior\) of \{person\_name\} from the dialogue to assist a subsequent LLM in imitating \{person\_name\}’s decision\-making and tone during the conversation\.\[CAPS Theory Brief\]When a person faces a specific Situation \(S\), an internal processing sequence \(Internal state\) is activated, including but not limited to:1\) Cognitions: Interpretation of the situation, focus of attention, encoding of semantic/social cues\.2\) Affect: Emotions, mood\.3\) Goals/Values: Communication goals to be achieved at the moment, and long\-term values\.4\) Expectancies/Beliefs: Expectations regarding the outcome of the interaction, the reaction of others, or social norms\.The internal processing sequence leads to an observable response \(Behavior\), such as the reply strategy and tone actually adopted by \{person\_name\}\.\[Definition of S\-I\-B in this Task\]\- Situation \(S\): The context of the dialogue \(if necessary, include a minimalist summary of the conversation, limited to the minimum essential information helpful for interpretation\)\.\- Internal state \(I\): Based on the dialogue between \{person\_name\} and others, provide a concise inference of \{person\_name\}’s possible internal processing sequence at that moment, which is "verifiable and alignable with evidence\."\- Behavior \(B\): The reply strategy adopted by \{person\_name\} when responding\.\[Crucial Rules\]1\. Observing the entire dialogue can yield 0 to multiple SIBs\.2\. Confidence Score: For each SIB, assign a confidence value between 0 and 1, reflecting your overall certainty regarding that SIB\.3\. Output must strictly follow the format below:\`\`\`json\{\{"items": \[\{\{"situation": "situation 1", "internal\_state": "internal state 1", "behavior": "behavior 1", "confidence": <your evidence score\>\}\}, \{\{"situation": "situation 2", "internal\_state": "internal state 2", "behavior": "behavior 2", "confidence": <your evidence score\>\}\}\]\}\}\`\`\`Supplement: If there are no obvious SIB triplets to extract from the dialogue, the items field should output an empty list\.The following is an example:Dialogue Content:\{\{"diag\_id": 31,"dialogue": \[\{\{"role": "Master","content": "You wretched monkey, why have you come to me instead of sleeping late at night?",\}\},\{\{"role": "Sun Wukong","content": "Master hit me three times during the day and closed the middle door, clearly intending for me to come through the back door at the third watch to learn the Way\. Master, Master, please teach me some true skills\."\}\},\{\{"role": "Master","content": "Considering your sincerity and desire to learn, I shall pass on some magic to you\. I have two types of transformation skills: 36 transformations and 72 transformations\. Which one do you wish to learn?",\}\},\{\{"role": "Sun Wukong","content": "Disciple wishes to learn the greater one\. Master, please teach me the 72 transformations\."\}\},\]\}\}Output:\`\`\`json\{\{"items": \[\{\{"situation": "Patriarch asks what \{person\_name\} is doing up late instead of sleeping", "internal\_state": "\{person\_name\} feels wronged, believing he is merely following the Patriarch’s instructions", "behavior": "\{person\_name\} references the Patriarch hitting him three times during the day and asking him to come at night to learn the Way", "confidence": 0\.8\}\}, \{\{"situation": "Patriarch describes two types of transformation skills, 36 and 72 transformations", "internal\_state": "\{person\_name\} is eager to learn and wants to acquire more skills", "behavior": "Expresses his thought by stating a desire to learn the 72 transformations", "confidence": 0\.7\}\}\]\}\}\`\`\`Dialogue Content:\{dialogue\_data\}Please strictly output according to the required format\.Figure 18:Reaction Process extraction prompt used in RoleAgentBench\. All datasets share the same prompt except that the few\-shot examples are replaced with the examples from the corresponding dataset\.System prompt:You are a critic responsible for evaluating digital personas\. Please identify errors and their severity levels in the model outputs and return JSON format\. You will evaluate 1 samples at once, conducting evaluation after reviewing all samples comprehensively\.User prompt:You are responsible for critiquing digital persona responses to questions\. I will provide you with \{eval\_batch\_size\} questions for \{person\_name\} and the model\-generated responses, along with \{person\_name\}’s correct answers\. Please evaluate the model’s responses according to the following steps:1\. Read \{person\_name\}’s profile and persona\. The profile is a brief introduction written by the real person, while the persona consists of descriptions about \{person\_name\} derived from past data along with corresponding original data as evidence\.2\. Read the scoring guidelines and metrics, and use the provided profile and persona as standards to judge whether the model’s performance deviates from the standards\. If you feel the model’s response violates certain metrics but similar content appears in the profile and persona, you should consider it correct\.3\. Evaluate the model’s response for each sample separately and output results in JSON format\.\#\# \{person\_name\}’s profile:\{profile\}\#\# \{person\_name\}’s persona:\{persona\}\#\# Scoring MetricsTo evaluate the model\-generated responses, please identify whether the model has made the following error types:\{evaluation\_rubrics\}\#\# Scoring Guidelines1\. Identify all errors \(zero to multiple\) made in the model responses\. If you find error types not included in the scoring metrics, they can also be listed as long as they fit the metric definitions, and fill in "other" in the type field\.2\. When evaluating errors, if content that appears in the Persona is present in the response, consider it correct\. If content that contradicts the persona appears, consider it an error\. If content appears in the response that the persona doesn’t mention, if it’s within reasonable bounds, don’t consider it an error\.3\. For each error, determine severity level 1˜5 points, where 1 represents minor mistake, 3 represents moderate error, 5 represents serious violation of metrics\. If error severity is 0, it means the error is invalid and should not be included in the final output\.4\. When comparing multiple samples, maintain consistency in scoring standards\.5\. There are three evaluation dimensions: memory, speaking\_style, and S\-I\-B consistency \(SIB\)\. The dimension being evaluated this time is: \{dimension\_name\}\. Please focus on that dimension’s scoring metrics for evaluation\. Do not point out errors for other dimensions\. For example, if you are evaluating the memory dimension, even if the response doesn’t match \{person\_name\}’s speaking style, you don’t need to point out the error\.\#\# Output Format RequirementsProvide your evaluation results in JSON format as follows\. After comprehensively reviewing all samples’ evaluation results, please only output this JSON:\{\{"flaws": \[\{\{"instance": "<brief description of the error\>", "type": "<error type\>", "severity": <error severity 1˜5 points\>, "sample\_index": \[<sample numbers where error was found\>,\.\.\.\]\}\},\.\.\.\]\}\}=== Sample 1 ===Question: \{question\}Model Reply: \{model\_reply\}Correct Reply: \{ground\_truth"\}Please output accurately according to the required formatFigure 19:Evaluation prompt used in RoleAgentBench\. All datasets share the same prefix prompt except for the data input format \(question, model reply, and correct reply triplet\)\.\{"memory": f"""This metric evaluates whether the model’s responses align with the target’s autobiographical memory, i\.e\., the target’s past experiences and knowledge they should possess\.Error types:1\. Memory contradiction: The model’s response contradicts personal history or experiences provided in the persona\.2\. Knowledge error: The model’s response incorrectly presents knowledge or information from the persona\.""","speaking\_style": f"""This metric evaluates whether the model’s responses match the target’s writing style, such as word choice and catchphrases\.Error types:1\. Word choice deviation: The model uses vocabulary significantly different from the persona’s common words, or adopts vocabulary styles the persona wouldn’t use \(e\.g\., too formal or too casual\)\.2\. Sentence structure mismatch: The model uses sentence length, rhythm, or structure different from the persona’s habits, e\.g\., persona prefers short colloquial sentences but model produces long formal sentences\.3\. Missing catchphrases or habitual expressions: Common phrases, emojis, or fixed expressions used by the persona that should appear in the context but are missing\.4\. Style dilution: Overall tone has no obvious errors but lacks the persona’s unique rhythm, warmth, or characteristics, resulting in low imitation quality\.""","Reaction\_Process": f"""This metric evaluates whether the model’s responses align with the target’s consistency in "Situation understanding \(S\) \- Internal state \(I\) \- Response strategy \(B\)" under specific contexts\. The focus is: can it correctly read the situation, go through reasonable internal states, and adopt response strategies consistent with the persona\.Error types:1\. Situation understanding error: Failure to correctly understand the current situation, such as interpreting support as offense, or neutral as praise or vice versa\.2\. Internal state contradiction: The model’s inferred internal state doesn’t match the target’s reaction patterns described in the persona\.3\. Response strategy error: Adopting response strategies that don’t belong to the target in the persona\."""\}Figure 20:Evaluation rubrics used for all datasetsSystem prompt:You are an expert in mimicking the speaking style of specific characters\. You need to play the role of \{person\_name\}, responding to questions in the character’s tone and behavioral habits\.User prompt:You will play the role of \{person\_name\}\. Based on the information provided below, please respond to the question\. Please note the following points:\# Objective\- If the historical memory includes specific language habits, please maintain them\.\- Output only the text of your reply; do not add any other explanations or titles\.\# Persona and Past Interaction Records\{persona\}\# Profile of \{person\_name\}\{profile\}\- The three aspects of persona are: Memory, Speaking Style, and SIB\_triplet\. Memory refers to \{person\_name\}’s personal information and domain knowledge\. Speaking Style refers to \{person\_name\}’s word choices and catchphrases\. SIB\_triplet refers to \{person\_name\}’s Internal state and Behavior strategies when facing specific Situations\.\# Question\{question\}Figure 21:Reply generation prompt used in RoleAgentBench\. All datasets share the same prefix prompt except for the data input format \(question\)\.