SenseJudge: Human-Centric Preference-Driven Judgment Framework

# SenseJudge: Human-Centric Preference-Driven Judgment Framework
Source: [https://arxiv.org/html/2606.03189](https://arxiv.org/html/2606.03189)
Rui Li1, Junfeng Liu2,3∗, Xiangwen Kong2, Linhai Xu2, Zhifang Sui122footnotemark:2 1State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University2StepFun3Xi’an Jiaotong University o\_l1ru1@outlook\.com, 3124336079ljf@stu\.xjtu\.edu\.cn Equal contribution\. This work was conducted during an internship at StepFun AI from August 2024 to April 2025\.Corresponding authors\.

###### Abstract

Large Language Models \(LLMs\) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm\. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real\-world human\-AI dialogue scenarios\. To address these limitations, we proposeSenseJudge, a customizable judgment framework driven by human preferences andSenseBench, a diverse and challenging instruction following benchmark derived from real\-world multi\-turn interactions\. We applied the automatic judgment framework and benchmark to two tasks: 1\)LLMs as personalized judges, and 2\)model ranking\. We conducted extensive experiments, and the results demonstrate that the SenseJudge framework surpasses other judgment methods and models in the LLMs\-as\-personalized\-judges task and achieves model ranking that aligns with real human sense\. Additionally, we conducted analyses on position bias and consistency, alongside ablation studies, which affirmed the robustness of SenseJudge\. Our data and code are publicly available at[https://github\.com/qiongrenpiqida/SenseJudge](https://github.com/qiongrenpiqida/SenseJudge)\.

SenseJudge: Human\-Centric Preference\-Driven Judgment Framework

Rui Li1††thanks:Equal contribution\. This work was conducted during an internship at StepFun AI from August 2024 to April 2025\., Junfeng Liu2,3∗, Xiangwen Kong2, Linhai Xu2††thanks:Corresponding authors\., Zhifang Sui122footnotemark:21State Key Laboratory for Multimedia Information Processing, School of Computer Science,Peking University2StepFun3Xi’an Jiaotong Universityo\_l1ru1@outlook\.com, 3124336079ljf@stu\.xjtu\.edu\.cn

## 1Introduction

The evaluation of model responses is a crucial component in the development of Large Language Models \(LLMs\)\. Selecting the superior response from a pair generates preference data \(query,chosen,rejected\), which is essential for various post\-optimization algorithmsRafailovet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib6)\); Ouyanget al\.\([2022](https://arxiv.org/html/2606.03189#bib.bib7)\)\. Competitive assessments of different LLM versions across various companies facilitate model updates and iterationsLambertet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib5)\); Liet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib8)\)\.

Furthermore, by evaluating a variety of model responses and selecting those that most closely align with individual user preferences, the judgment process can accommodate a broad spectrum of user values, effectively delivering a personalized experience\.Fanet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib11)\); Donget al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib13)\)\.

Previously, these judgments were made through manual human annotation\. However, with advancements in the natural language understanding and instruction\-following capabilities of Large Language Models \(LLMs\)Naveedet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib15)\); Zhouet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib16)\), LLMs are increasingly being integrated into human workflows for automatic judgmentZhenget al\.\([2023a](https://arxiv.org/html/2606.03189#bib.bib4)\); Wanget al\.\([2023b](https://arxiv.org/html/2606.03189#bib.bib1)\)\.

![Refer to caption](https://arxiv.org/html/2606.03189v1/kaitou.png)Figure 1:The model judges responses on behalf of human based on individual preferences\.However, existing evaluation methods and models, such as PandalmWanget al\.\([2023b](https://arxiv.org/html/2606.03189#bib.bib1)\), Auto\-jLiet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib2)\), and various reward models, rely on homogeneous standards and values, overlooking the diversity of user preferences\. These models also lack the ability to make nuanced choices based on individual user needs\. Furthermore, current public judgment benchmarks, including MT\-BenchZhenget al\.\([2023a](https://arxiv.org/html/2606.03189#bib.bib4)\), Auto\-jLiet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib2)\), and RewardbenchLambertet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib5)\), predominantly feature single\- or double\-turn dialogue designs\. This represents a significant departure from real\-world human\-AI interactions, which often involve multiple rounds of historical dialogue, some relevant and others not, to inform the current request\.

To address these limitations, we proposeSenseJudgeandSenseBench\.SenseJudgeis a customizable evaluation framework based on explicit human preferences, enabling personalized judgment of model responses by simulating the preferences of different users and scenarios\.SenseBenchis a high\-quality, challenging dataset derived from realistic human\-AI dialogues, covering eight common topics, including math, logic, writing, and role\-play\.

We conducted extensive experiments to evaluate the effectiveness of the SenseJudge framework in two primary application scenarios based on SenseBench: 1\. LLM\-as\-a\-Personalized\-Judge, which focuses on selecting responses that align with individual preferences, and 2\. Model Ranking, which aims to identify the leading model among different companies or versions \(e\.g\., GPT vs\. DeepSeek\)\. The results demonstrate that SenseJudge outperforms both state\-of\-the\-art APIs, training\-based evaluators, and reward models in the LLM\-as\-a\-Personalized\-Judge task\. It also achieves model rankings that align with human judgment\.

Additionally, the effectiveness and robustness of SenseJudge are further validated through analyses of factors such as positional bias and consistency, as well as comprehensive ablation studies\.

Our main contributions are as follows:

- •We developed SenseBench, a benchmark designed to align with realistic human\-AI interaction scenarios\. It features diverse and challenging multi\-turn dialogues\.
- •We propose SenseJudge, a judgment framework designed to adapt to diverse human values and preferences, which enables fine\-grained and customizable judgment of model responses\.
- •We conducted comprehensive experiments and analysis to evaluate the effectiveness of SenseJudge in the LLMs\-as\-personalized\-judges task and model ranking task\.

## 2Related Work

### 2\.1LLM as a Judge

LLMs are increasingly recognized as a viable alternative to traditional expert\-driven evaluations due to their scalability and adaptability in evaluation tasks\. Common enhancement methods include designing prompts and training judgers\. Prompt engineering is an effective approach to enhancing the performance of LLMs as judges without updating their model parameters\. Many methods optimize their respective prompt strategies prior to training, such as incorporating referencesZhuet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib3)\)or rulesLiet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib2)\)into the prompts, and directing models to offer detailed analyses or justificationsKimet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib17)\)\. Based on this, pairwise or point\-wise judgment data can be used to train LLM\-based judges by employing supervised fine\-tuningKimet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib17)\)\. Some researchers train reward models using large\-scale preference feedback data; leveraging their ability to learn preferences, these models can also function as judgesZhonget al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib24)\)\.

### 2\.2From Universal to Individualized

The transition from generalization to personalization represents a historic paradigm in technological developmentYouet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib22)\)\. With the rapid advancement of the general capabilities of LLMs, there is growing interest in tailoring LLMs to specific user contexts, moving beyond their role as merely general\-purpose chatbotsZhanget al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib23)\)\. For instance,Wuet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib18)\)proposed to implicitly infer users’ personalized preferences through multi\-turn conversations and help LLMs to dynamically adapt their behavior and responses to better align with individual user needs\.Zhuet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib19)\)proposed to use psychological assessment tools, such as the “Big Five Personality Traits” and the “Dark Triad” to quantify user characteristics and align the models accordingly\.Liet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib20)\)introduced a multi\-granularity interest prediction framework which leverages both coarse\-grained and fine\-grained interest information to supervise model training\. Additionally,Tanet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib21)\)proposed a personalized framework OPPU, which enhances the personalization and generalization capabilities of LLMs by creating dedicated personalized modules for each user through parameter\-efficient fine\-tuning and non\-parametric personalization methods, such as retrieval\-augmented and profile\-augmented techniques\.

## 3Method

![Refer to caption](https://arxiv.org/html/2606.03189v1/x1.png)Figure 2:An overview of our the data construction pipeline of SenseBench\. The pipeline involves 1\) Quality\-Based Data Filtering and Categorization, 2\) Challenge\-based Filtering\.In this section, we introduce the two key components of our framework:SenseBench, a diverse and high\-quality instruction\-following benchmark derived from real human\-AI interaction dialogues, andSenseJudge, a customizable evaluation method based by human preferences\.

### 3\.1SenseBench

#### 3\.1\.1Overview

Currently, there is a lack of benchmarks that accurately reflect human user experiences and differentiate models based on perceived quality\. SenseBench consists of real user queries sourced from an online LLM service \(anonymized due to review policies\), where the final turn of each dialogue either provides context\-dependent instructions or introduces entirely new tasks\. This design places high demands on the LLMs’ ability to align with human expectations, facilitating effective model differentiation and improving alignment with human\-perceived quality\. \.

#### 3\.1\.2Benchmark Construction

Based on real user data, we employ a scalable automated data selection pipeline to construct a high\-quality and diverse benchmark as shown in[2](https://arxiv.org/html/2606.03189#S3.F2)\. The pipeline consists of two main steps: 1\) Quality\-based Data Filtering and Categorization, and 2\) Challenge\-based Filtering\. The detailed process is as follows:

##### Quality\-Based Data Filtering and Categorization

We use Qwen3\-14BYanget al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib10)\)to denoise and categorize data from real user dialogue history\. Specifically, we first remove redundant, ambiguous, low\-quality, or overly simplistic entries and then categorize the remaining data into eight distinct topics: math, logical reasoning, comprehensive question answering, coding, creative writing, multilingual translation, role\-playing, and natural language understanding\. These categories represent common types of real\-world human instructions and provide a comprehensive reflection of the capabilities of LLMs\.

##### Challenge\-based Filtering

To ensure that the selected prompts are sufficiently challenging, we require that they generate clearly distinguishable responses from different models\. We conduct a multi\-model evaluation on candidates that have passed an initial round of automated filtering\. Specifically, we input each candidate prompt into both relatively stronger models \(e\.g\., Qwen2\.5\-Max, Qwen3\-235B\) and relatively weaker models \(e\.g\., Qwen2\.5\-14B, LLaMA3\-8B\) to generate responses\. This results in a large set of pairwise data in the format of \(Query, Strong Model Response, Weak Model Response\)\. We then apply a two\-stage filtering process that combines both automatic and manual filtering\.Automatic Filtering\.We use GPT\-4 to assess the response quality between strong and weak models\. Questions where the strong model significantly outperforms the weak model are retained as discriminative and challenging instances\.For specific judging instructions, you can see the Appendix[A\.2](https://arxiv.org/html/2606.03189#A1.SS2)\.Manual Filtering\.After the initial automatic filtering, we performed manual filtering on the data\. Specifically, the manual filtering process focuses on two main aspects: 1\) Assessing whether the pairwise data filtered by GPT\-4 can truly distinguish model capabilities\. During this assessment, humans consider visible differences between model responses \(e\.g\., format, length, or level of detail\) or other intuitive quality distinctions\. 2\) Performing partial cleaning on the prompts of the selected data\. Since all prompts are sourced from real user data, they may contain formatting issues or improper phrasing, which require manual cleaning to ensure the data is suitable for creating a usable benchmark\.

Specific data statistics and category information can be found in Appendix[A\.1](https://arxiv.org/html/2606.03189#A1.SS1)\.

### 3\.2SenseJudge

#### 3\.2\.1Overview

SenseBench offers a human\-centric environment where we employ multi\-model generation followed by pairwise judgment to support two tasks: \(1\) user\-centric preference alignment and \(2\) model ranking\. We introduce SenseJudge, a personalized pairwise judgment approach that demonstrates strong performance in both tasks\. SenseJudge begins by extracting a spectrum of human preferences from a small set of annotated \(query, chosen, rejected\) pairs\. Each preference guides the judge through the same pairwise instances, and the preference set that best matches the human labels is retained as the distilled, generalizable set\. Next, we outline the components of our judgment framework, including input\-output format, preference generation, and the selection and application of the preference set\.

#### 3\.2\.2Input\-Output Format

For SenseJudge, the inputI=\{q,\(r1,r2\),p\}\\mathit\{I\}=\\\{q,\(r\_\{1\},r\_\{2\}\),p\\\}comprises a queryqq, a pair of responses\(r1,r2\)\(r\_\{1\},r\_\{2\}\), and a preferencepp\. The target outputO=\{judgment,analysis\}\\mathit\{O\}=\\\{\\text\{judgment\},\\text\{analysis\}\\\}yields a judgment and corresponding analysis, specifically indicating whetherr1r\_\{1\}orr2r\_\{2\}is superior according to the given preferencepp\. Based on previous researchDonget al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib13)\); Zhenget al\.\([2023b](https://arxiv.org/html/2606.03189#bib.bib31)\), the inclusion of “ties” can negatively impact model performance, so we assume that the model can always distinguish between responses based on preference\.

#### 3\.2\.3Preference Construction

We use Deepseek\-R1Guoet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib9)\)to distill implicit human preferences captured in annotated pairwise data into explicit textual preferences\. Specifically, we provide a small set of human\-annotated pairwise examples to an LLM and prompt it to generate a preference statement explaining why a user might favor one response over the other\. Examples of these preferences are provided in Appendix[A\.3](https://arxiv.org/html/2606.03189#A1.SS3)\. One preference is generated for each pairwise sample\. These preferences are then incorporated into the data to create a complete personalized evaluation dataset in the formatI=\{q,\(r1,r2\),p\}\\mathit\{I\}=\\\{q,\(r\_\{1\},r\_\{2\}\),p\\\}, which consists of a queryqq, a pair of responses\(r1,r2\)\(r\_\{1\},r\_\{2\}\), and a preferencepp\. The specific prompt used is detailed in Appendix[A\.2](https://arxiv.org/html/2606.03189#A1.SS2)\.

#### 3\.2\.4Preference Set Selection

After we obtain the preference setP=\{p1,p2,…,pm\}P=\\\{p\_\{1\},p\_\{2\},\\dots,p\_\{m\}\\\}, wheremmis the number of sampled examples, we feed eachpi∈Pp\_\{i\}\\in Pinto the judger and assess its performance on the test setT=\{\(q1,r11,r12\),…,\(qm,rm​1,rm​2\)\}T=\\\{\(q\_\{1\},r\_\{11\},r\_\{12\}\),\\dots,\(q\_\{m\},r\_\{m1\},r\_\{m2\}\)\\\}\. Note that the set here is the same one we used to extract the preference\. Under eachpip\_\{i\}, we obtain the judger’s selectionsCi=\{ci​1,ci​2,…,ci​m\}C\_\{i\}=\\\{c\_\{i1\},c\_\{i2\},\\dots,c\_\{im\}\\\}, whereci​j∈\{rj​1,rj​2\}c\_\{ij\}\\in\\\{r\_\{j1\},r\_\{j2\}\\\}indicates that the responserj​1r\_\{j1\}orrj​2r\_\{j2\}is chosen by the judger\. Comparing the model’s selectionsCℳ,i=\{ci​1,ci​2,…,ci​m\}C\_\{\\mathcal\{M\},i\}=\\\{c\_\{i1\},c\_\{i2\},\\dots,c\_\{im\}\\\}with the ground\-truth human judgmentsG=\{g1,g2,…,gm\}G=\\\{g\_\{1\},g\_\{2\},\\dots,g\_\{m\}\\\}, wheregj∈\{rj​1,rj​2\}g\_\{j\}\\in\\\{r\_\{j1\},r\_\{j2\}\\\}, we obtain a corresponding set of correctness indicatorsSi=\{si​1,si​2,…,si​m\}S\_\{i\}=\\\{s\_\{i1\},s\_\{i2\},\\dots,s\_\{im\}\\\}, wheresi​j∈\{0,1\}s\_\{ij\}\\in\\\{0,1\\\}\. Subsequently, we consider all subsets𝒫k⊆P\\mathcal\{P\}\_\{k\}\\subseteq Pof cardinalityll, where the number of preferencesllsatisfies1≤l≤m1\\leq l\\leq m\. For each subset𝒫k\\mathcal\{P\}\_\{k\}, we perform a majority vote over its associated correctness indicators\{S𝒫k​1,S𝒫k​2,…,S𝒫k​l\}\\\{S\_\{\\mathcal\{P\}\_\{k1\}\},S\_\{\\mathcal\{P\}\_\{k2\}\},\\dots,S\_\{\\mathcal\{P\}\_\{kl\}\}\\\}, where𝒫k​i\\mathcal\{P\}\_\{ki\}denotes theii\-th preference in𝒫k\\mathcal\{P\}\_\{k\}andS𝒫k​iS\_\{\\mathcal\{P\}\_\{ki\}\}is its corresponding correctness indicator\. This procedure yields an updated set of correctness indicators, denoted byS𝒫k∗=\{s𝒫k​1∗,s𝒫k​2∗,…,s𝒫k​l∗\}S\_\{\\mathcal\{P\}\_\{k\}\}^\{\*\}=\\\{s\_\{\\mathcal\{P\}\_\{k1\}\}^\{\*\},s\_\{\\mathcal\{P\}\_\{k2\}\}^\{\*\},\\dots,s\_\{\\mathcal\{P\}\_\{kl\}\}^\{\*\}\\\}, where eachs𝒫k​j∗s\_\{\\mathcal\{P\}\_\{kj\}\}^\{\*\}is obtained via a majority\-voting scheme applied to the collection\{sk​1,j,sk​2,j,…,sk​l,j\}\\\{s\_\{k1,j\},s\_\{k2,j\},\\dots,s\_\{kl,j\}\\\}\. We then calculate the accuracy of these aggregated correctness indicatorsAcc​\(S𝒫k∗\)=1m​∑j=1m𝕀​\(s𝒫k​j∗=1\)\\textbf\{Acc\}\(S\_\{\\mathcal\{P\}\_\{k\}\}^\{\*\}\)=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\mathbb\{I\}\(s\_\{\\mathcal\{P\}\_\{kj\}\}^\{\*\}=1\),𝕀\\mathbb\{I\}is an indicator function\. The subset𝒫k∗\\mathcal\{P\}^\{\*\}\_\{k\}that achieves the highest accuracyAcc​\(𝒫k∗\)\\textbf\{Acc\}\(\\mathcal\{P\}^\{\*\}\_\{k\}\)effectively captures the annotator’s underlying preference and can be generalized to unseen examples\.

#### 3\.2\.5Preference Set Application

Given the optimal preference subset𝒫k∗\\mathcal\{P\}^\{\*\}\_\{k\}obtained from the previous stage, we apply it to an unseen test setT′=\{\(q1′,r11′,r12′\),…,\(qn′,rn​1′,rn​2′\)\}T^\{\\prime\}=\\\{\(q^\{\\prime\}\_\{1\},r^\{\\prime\}\_\{11\},r^\{\\prime\}\_\{12\}\),\\dots,\(q^\{\\prime\}\_\{n\},r^\{\\prime\}\_\{n1\},r^\{\\prime\}\_\{n2\}\)\\\}, wherenndenotes the number of test instances\. For each preferencep∈𝒫k∗p\\in\\mathcal\{P\}^\{\*\}\_\{k\}, every test sample\(qj′,rj​1′,rj​2′\)\(q^\{\\prime\}\_\{j\},r^\{\\prime\}\_\{j1\},r^\{\\prime\}\_\{j2\}\)is fed into the judger conditioned onpp, producing a set of model selectionsCp′=\{cp​1′,cp​2′,…,cp​n′\}C^\{\\prime\}\_\{p\}=\\\{c^\{\\prime\}\_\{p1\},c^\{\\prime\}\_\{p2\},\\dots,c^\{\\prime\}\_\{pn\}\\\}, wherecp​j′∈\{rj​1′,rj​2′\}c^\{\\prime\}\_\{pj\}\\in\\\{r^\{\\prime\}\_\{j1\},r^\{\\prime\}\_\{j2\}\\\}\. We then aggregate the outputs across all preferences in𝒫k∗\\mathcal\{P\}^\{\*\}\_\{k\}by performing a majority vote at the instance level, yielding the final decision

cj′⁣∗=MajorityVote​\(\{cp1​j′,cp2​j′,…,cpl​j′\}\),c^\{\\prime\*\}\_\{j\}=\\text\{MajorityVote\}\(\\\{c^\{\\prime\}\_\{p\_\{1\}j\},c^\{\\prime\}\_\{p\_\{2\}j\},\\dots,c^\{\\prime\}\_\{p\_\{l\}j\}\\\}\),wherel=\|𝒫k∗\|l=\|\\mathcal\{P\}^\{\*\}\_\{k\}\|\. This produces the final aggregated decision setC′⁣∗=\{c1′⁣∗,c2′⁣∗,…,cn′⁣∗\}C^\{\\prime\*\}=\\\{c^\{\\prime\*\}\_\{1\},c^\{\\prime\*\}\_\{2\},\\dots,c^\{\\prime\*\}\_\{n\}\\\}, which represents the model’s preference\-aligned selections on the test set\.

## 4Experiments

ModelMathCodeLogicQAWriteRoleNLUTransOverallStrong Open/Closed\-Source ModelsGpt\-4o66\.0061\.6065\.4772\.9360\.8063\.2065\.4756\.4063\.98DeepSeek\-V372\.8062\.2766\.6777\.0762\.6764\.4064\.8061\.8766\.57DeepSeek\-R153\.3355\.6061\.3353\.4746\.9350\.5351\.7350\.6752\.95Qwen\-Plus\-Latest66\.0059\.6061\.2067\.2055\.4758\.1360\.4059\.3360\.92Trained Judger / Reward ModelsAuto\-j\-13B64\.4052\.6755\.7353\.8739\.3342\.4048\.0036\.4049\.10Skywork\-Critic\-Llama3\.1\-70B63\.7363\.8763\.7369\.8759\.6759\.4757\.3355\.4761\.35INF\-ORM\-Llama3\.1\-70B67\.2063\.2064\.8069\.0760\.0054\.1358\.4059\.7362\.07QRM\-Gemma\-2\-27B69\.6058\.9363\.4754\.1352\.5350\.4056\.5359\.7358\.17Skywork\-Reward\-Gemma2\-27B70\.4061\.6066\.1074\.1064\.0060\.0062\.7058\.4064\.70SenseJudgeQwen2\.5\-14B\-Instruct60\.2760\.8059\.3370\.1361\.6062\.0064\.4057\.2061\.97\+SenseJudge73\.45\(\+13\.18\)80\.90\(\+20\.10\)72\.44\(\+13\.11\)85\.67\(\+15\.54\)72\.89\(\+11\.29\)75\.24\(\+13\.24\)76\.80\(\+12\.40\)74\.21\(\+17\.01\)76\.88\(\+14\.67\)Qwen2\.5\-72B\-Instruct63\.8761\.4761\.8776\.2760\.1366\.4064\.4057\.0763\.93\+SenseJudge82\.30\(\+18\.43\)89\.01\(\+27\.54\)79\.76\(\+17\.89\)89\.87\(\+13\.60\)79\.82\(\+19\.69\)82\.12\(\+15\.72\)78\.10\(\+13\.70\)75\.23\(\+18\.16\)81\.99\(\+18\.05\)Llama3\.1\-8B\-Instruct62\.1358\.8059\.7372\.0058\.1358\.5360\.1353\.6060\.33\+SenseJudge84\.58\(\+22\.45\)79\.34\(\+20\.54\)71\.80\(\+12\.07\)82\.39\(\+10\.39\)73\.43\(\+15\.30\)69\.80\(\+11\.27\)75\.69\(\+15\.56\)72\.44\(\+18\.84\)74\.98\(\+14\.85\)Qwen3\-14B\-Instruct71\.7364\.8069\.2078\.0061\.4767\.6066\.2756\.1366\.90\+SenseJudge86\.53\(\+13\.80\)87\.96\(\+23\.16\)83\.69\(\+14\.49\)92\.24\(\+14\.24\)75\.27\(\+7\.67\)81\.04\(\+13\.44\)78\.72\(\+12\.45\)75\.78\(\+19\.65\)82\.65\(\+15\.56\)

Table 1:Comparative evaluation of various judgment methods and judgers across different categories: Math, Code, Logic, QA \(Question Answering\), Write, Role \(Role\-playing\), NLU \(Natural Language Understanding\), Trans \(Translation\), and Overall their average performance\.![Refer to caption](https://arxiv.org/html/2606.03189v1/multi-time.png)Figure 3:Comparison of baseline and SenseJudge across four LLMs on all topics\. SenseJudge results are averaged over three runs with 95% confidence intervals, while baselines are single\-run references\. Each subplot shows consistent improvements of SenseJudge over the baseline across all tasks\.In this section, we present the performance of the SenseJudge in two primary tasks: 1\) LLM\-as\-a\-Personalized\-Judge and 2\) model ranking\. The effectiveness and robustness of SenseJudge are validated through analyses of factors such as positional bias and consistency, as well as comprehensive ablation studies\.

### 4\.1Experiments Setup

#### 4\.1\.1Baselines

##### LLM\+Prompt

Direct prompting without preferences across multiple closed\-source and open\-source models\. The closed\-source models included GPT\-4o and Qwen2\.5\-Max\. The open\-source models tested were DeepSeek\-V3DeepSeek\-AIet al\.\([2025b](https://arxiv.org/html/2606.03189#bib.bib25)\), DeepSeek\-R1DeepSeek\-AIet al\.\([2025a](https://arxiv.org/html/2606.03189#bib.bib26)\), Qwen2\.5\-14B\-InstructQwenet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib27)\), Qwen2\.5\-72B\-Instruct , Llama3\.1\-8B\-InstructGrattafioriet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib28)\), and Qwen3\-14B\-Instruct\.

##### LLM\+Training

Publicly available trained judger Auto\-jLiet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib2)\), as well as top\-ranking reward models on the RewardBench leaderboard, including generative model Skywork\-Critic\-Llama\-3\.1\-70B , and classifier models including INF\-ORM\-Llama3\.1\-70B , Skywork\-Reward\-Gemma\-2\-27B\-v0\.2 and QRM\-Gemma\-2\-27B \. PandaLMWanget al\.\([2023b](https://arxiv.org/html/2606.03189#bib.bib1)\)and JudgeLMZhuet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib3)\)are excluded from the comparison due to token limit constraints\.

#### 4\.1\.2Datasets and Metrics

We use the human\-annotated personalized pairwise dataset for the LLM\-as\-a\-Personalized\-Judge task, consisting of 1,000 examples \(125 per category across 8 categories\)\. We selected 12 widely used or state\-of\-the\-art LLMs\( e\.g\.,Deepseek\-R1,Qwen2\.5\-Max,Grok\-3,GPT\-3\.5,Qwen2\.5\-7B,Claude\-3\-sonnet,Gemini\-2\.0\-flash\-thinking\-exp, Deepseek\-v3Liuet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib14)\), GLM\-4\-plus,Gemini\-1\.0\-pro,Moonshoot\-v1\-8kTeamet al\.\([2025](https://arxiv.org/html/2606.03189#bib.bib12)\),Minimax\-5\.5\) to generate responses on SenseBench\. From each category, we sample 10 examples for preference generation and subsequent steps\. Three annotators with diverse professional backgrounds provide ground\-truth labels, which we use to measure accuracy\. For the task of Model Ranking, we directly used the relative rankings of models on the human sense leaderboard as the ground truth\.We refer to this dataset as the Ranking Set\. We employed specific model comparison examples from the Ranking Set and evaluated judger’s performance by computing the consistency between the rankings produced by the judge and the human\-annotated ground\-truth rankings\. Furthermore, we also utilize other public preference datasets, including RewardbenchLambertet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib5)\)and AutoJLiet al\.\([2023](https://arxiv.org/html/2606.03189#bib.bib2)\)\. Response accuracy was used as the evaluation metric\. When assessing the correctness of the model’s judgement, we employed a strict matching strategy, whereby the target string \(e\.g\., the final decision is response A\) must be fully contained within the model’s response to be recorded as correct\.

#### 4\.1\.3Implementation Details

We adopt the \(query,chosen,rejected\) format commonly used in pairwise preference data\. We test and calculate the overall accuracy by evaluating the chosen response and rejected response in both their original order and their reversed order\. We directly employ small\-scale LLMs as our judgers without any additional training to balance performance and cost\.

For the model ranking task, we randomly sample five preferences from each category within the preference set corresponding to individual annotators \(three annotators in total\)\. Each sampled preference is independently applied to the model ranking task using the SenseJudge framework\. The final decision for each comparison is obtained via a majority voting scheme over the predictions induced by these preferences\.

![Refer to caption](https://arxiv.org/html/2606.03189v1/position.png)Figure 4:Thebar represents the absolute values of selecting response A \(in the first place\) and response B \(in the second place\) using the original model, while thebar represents the absolute values after applying SenseJudge\.

### 4\.2Main Result

#### 4\.2\.1LLM\-as\-a\-Personalized\-Judge

##### Overview

In the development of business LLMs, real\-time and representative human feedback is essential before the model is officially deployed\. Many benchmarks are available to directly evaluate models’ general capabilities through straightforward ground\-truth answer matching\. However, this evaluation often solely considers the accuracy of model outputs, overcoming other valued aspects such as creativity and formatting, thus failing to provide a comprehensive and user\-centered quality assessment\. Additionally, as many ground\-truth of benchmarks have been inadvertently or deliberately included in training data, it’s difficult to assess the true performance of models using existing benchmarks alone\. Hiring a stable team to evaluate models on consistent datasets after each update is a common practice\. However, as model updates become more frequent, the repetitive and labor\-intensive task of human annotation is increasingly constrained by cost and efficiency\. Therefore, “LLM\-as\-a\-Personalized\-Judge” has became a highly promising task that involves utilizing LLMs to simulate real user judgments of model responses, thereby streamlining and automating the model development process\.

##### Comparation with Baselines

As shown in Table[1](https://arxiv.org/html/2606.03189#S4.T1), the SenseJudge framework achieves an accuracy rate that is, on average 16\.08% higher in the LLM\-as\-a\-Personalized\-Judge task compared to baselines including strong open\-source and closed\-source models, trained judgers and reward models\. Even when applied to small\-scale LLMs like 8B and 14B, which are not specifically trained as judger, it still surpasses the judgment accuracy of directly using strong models by an average of 15\.78%\. Additionally, SenseJudge consistently demonstrates significant improvements across a wide range of categories compared to base models, and achieving higher accuracy than the baseline in every category\.

Although reward models such as INF\-ORM\-Llama3\.1\-70B and QRM\-Gemma\-2\-27B have excelled on the public Rewardbench leaderboard, their accuracy on the personalized dataset remains below 65% and even falls short compared to some strong baseline LLMs\. Auto\-j\-13B performs even worse, achieving scores similar to random selection\. This indicates that while judgers designed for specific judgment tasks and reward models developed for model alignment are trained on extensive preference data, they often learn fixed preferences that struggle to generalize to diverse and complex real\-world scenarios and challenge in adapting to varying user preferences\.

As shown in Figure[3](https://arxiv.org/html/2606.03189#S4.F3), we compare the baseline performance with the averaged results over three repeated runs\. The reported improvements are computed as the difference between the mean performance of repeated evaluations and the corresponding baseline across nine evaluation categories\. Substantial gains are observed consistently across all topics, indicating stable improvements under repeated sampling and reduced variance in evaluation outcomes\.

##### Positional Bias

Previous studiesWanget al\.\([2023a](https://arxiv.org/html/2606.03189#bib.bib30)\)have demonstrated that the relative position of two responses is an element that theoretically should be irrelevant but significantly influences the judgment result in the pairwise judgment setting\. Some LLMs exhibited a significant tendency to select responses in the first or second positions, which we refer to asposition bias\. To evaluate the impact of SenseJudge on position bias, we counted the distribution of model selection across each position under identical ground\-truth conditions and calculated the absolute values of these selections\. Higher absolute values indicate a stronger positional bias\. As shown in Figure[4](https://arxiv.org/html/2606.03189#S4.F4), SenseJudge framework effectively mitigates positional bias, particularly for smaller models where this bias is often more pronounced\.

![Refer to caption](https://arxiv.org/html/2606.03189v1/ranking.png)Figure 5:Pairwise judgments by Qwen3\-14B\-Instruct and Llama3\.1\-8B\-Instruct with SenseJudge between various models, including advanced models \(gpt\-4o, deepseek\-r1, claude\-3\-7\-sonnet\), past model \(gpt\-3\.5\), and open\-closed model \(qwen2\.5\-72b\-instruct\)\. Thebar indicates the win rate of the left model on the model ranking set, and thebar represents the win rate of the right model\.
##### Consistency

ModelOriginalSenseJudgeQwen2\.5\-14B\-Instruct69\.9774\.17Qwen2\.5\-72B\-Instruct78\.8678\.79Llama3\.1\-8B\-Instruct60\.3668\.19Qwen3\-14B\-Instruct81\.2381\.30

Table 2:Consistency performance of original models and models with SenseJudge framework\.We examined the consistency of Sensejudge framework under two distinct input settings: \(query, chosen, rejected\) and \(query, rejected, chosen\)\. We considered the judger’s selection to be consistent if it identified the same response \(either the initially designated ‘chosen’ or ‘rejected’\) as superior across both input configurations\. Results are presented in Table[2](https://arxiv.org/html/2606.03189#S4.T2)\. Notably, the SenseJudge framework shows improvements over the base models such as Qwen2\.5\-14B\-Instruct and Llama3\.1\-8B\-Instruct, achieving increases in consistency scores of 4\.2% and 7\.83%, respectively, compared to their original consistency scores of 69\.97% and 60\.36%\. While the Qwen2\.5\-72B\-Instruct and Qwen3\-14B\-Instruct models exhibit high consistency in their original evaluations \(78\.86% and 81\.23%\), SenseJudge maintains comparable consistency\.

#### 4\.2\.2Models Ranking

Pairwise judgment methods inherently provide a relative ranking between two model responses within a single comparison\. The absolute ranking derived from such pairwise comparisons has not been thoroughly evaluated\. Therefore, based on the model rankings derived from real human evaluations on the Arena leaderboard111[https://lmarena\.ai/?leaderboard](https://lmarena.ai/?leaderboard), we selected the models DeepSeek\-R1, Claude\-3\-7\-Sonnet, Gpt\-4o, Qwen2\.5\-72B\-Instruct, and Gpt\-3\.5 \(ordered according to their ranking on the leaderboard\) as our target models for ranking, and employed their corresponding response from Model Ranking Set\. Among these models, DeepSeek\-R1, Claude\-3\-7\-Sonnet, and Gpt\-4o represent advanced models and exhibit similar performance\. We then conducted pairwise comparisons using SenseJudge, randomly selecting 5 preferences from the previously generated preference set for each comparison\. The results are depicted in Figure[5](https://arxiv.org/html/2606.03189#S4.F5)\.

We observed that both the relative ranking from pairwise judgment between any two models and the absolute ranking derived from aggregating all pairwise results align with real human ranking\. The results also demonstrate that a full all\-pairs comparison is not strictly necessary to obtain a final model ranking using a pairwise approach\. Taking DeepSeek\-R1 as an example, its win rates when compared against all other models can be effectively used to rank all models\.

### 4\.3Ablation Studies

##### Preference Construction

As shown in Table[3](https://arxiv.org/html/2606.03189#S4.T3), we compare the performance of preferences generated by weaker models \(i\.e\., constructed by the judge model itself\) with those generated by a stronger model \(i\.e\., DeepSeek\-R1\)\. 我们发现

Preference QualityJudge ModelDeepseek\-R1Qwen2\.5\-14B\-Instruct67\.1776\.88Qwen2\.5\-72B\-Instruct69\.8581\.99Llama3\.1\-8B\-Instruct67\.9474\.98Qwen3\-14B\-Instruct69\.0982\.65

Table 3:Low\-quality preferences \(generated by the judge model itself\) are shorter, while high\-quality preferences \(generated by DeepSeek\-R1\) are longer\. Preferences generated by stronger models are more effective\.
##### Performance on Other Datasets

MethodRewardBenchSenseJudge90\.55Skywork\-Critic\-Llama3\.1\-70B92\.2Gemini\-1\.5\-pro\-051488\.2Gpt\-4o\-2024\-05\-1384\.6Meta\-Llama\-3\.1\-70B\-Instruct84Claude\-3\-opus\-2024022980\.1

Table 4:Method Accurancy Preference on RewardBench\.Based on the Qwen2\.5\-72B\-Instruct, we utilized SenseJudge to conduct tests on RewardBench to demonstrate the general effectiveness of our approach in judgment tasks\. RewardBench integrates several authoritative evaluation benchmarks, such as MT\-Bench, AlpacaEval, and LLMBar\. We selected several advanced llm\-as\-a\-judge methods and specifically trained generative reward models for comparison on RewardBench\. For the LLM\-as\-judge methods, we used publicly available results from the leaderboard directly\. Similar to the steps we followed when performing personalized judging, we sampled a subset of the data from RewardBench as a dev set for generating preferences\. As shown in Table[4](https://arxiv.org/html/2606.03189#S4.T4), SenseJudge significantly outperformed general llm\-as\-a\-judge methods, and without training, the gap with the generative reward model was 1\.65%\. Additionally, we found that SenseJudge significantly outperformed the generative reward model on the Auto\-j benchmark\.

##### Cross\-Domain Generalization of Preferences

We investigate the extent to which learned preferences generalize across task categories\. To this end, we conduct a preliminary empirical study evaluating cross\-domain transferability\. Specifically, we extend preferences derived from the mathematics domain to both logical reasoning and translation tasks, using Qwen3\-14B\-Instruct as the judge model\. When applied to logical reasoning, the transferred preferences achieve an accuracy of 78\.62%, compared to 83\.69% when using in\-domain preferences\. In contrast, transferring the same preferences to translation tasks results in a substantially lower accuracy of 61\.83%, relative to 75\.78% under in\-domain conditions\.

## 5Conclusion

In this work, we introduced SenseJudge, a novel framework leveraging explicit human preferences for customizable and personalized LLM judgment, and SenseBench, a challenging benchmark mirroring real\-world human\-AI interactions\. Extensive experiments in personalized judging and model ranking demonstrate SenseJudge’s superiority over existing methods and strong performance on public datasets, highlighting its potential for more accurate and user\-centric LLM evaluation\.

## Acknowledgments

This paper is supported by NSFC project 62476009\.

## Limitations

The LLM\-as\-a\-Personalized\-Judge task focused on simulating a limited number of explicit human preferences\. Due to the high cost of annotation, we employed three annotators, each labeling the same 1000 comparison data points, rather than expanding to a broader scope\. However, our SenseJudge achieved superior accuracy compared to other baselines in simulating the preferences of all three individuals\.

## Ethics Statement

This paper introduces SenseJudge, a framework for personalized and customizable judgment of LLM responses, driven by explicit human preferences\. We believe that SenseJudge offers a valuable tool for advancing the understanding and evaluation of LLMs in a more user\-centric manner\. However, responsible development and deployment necessitate ongoing attention to potential biases, fairness considerations, and the ethical implications of simulating and utilizing human preferences in automated judgment frameworks\. As for the benchmark, we have taken measures to anonymize user data within SenseBench, protecting individual privacy\. The LLMs utilized in our experiments are existing models with publicly available access\. Our use of these models and datasets aligns with their intended research purposes and licenses\. Therefore, we believe that our research work meets the ethics of ACL\.

## References

- DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu, B\. Xue, B\. Wang, B\. Wu, B\. Feng, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Ding, H\. Xin, H\. Gao, H\. Qu, H\. Li, J\. Guo, J\. Li, J\. Wang, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. L\. Cai, J\. Ni, J\. Liang, J\. Chen, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Zhao, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, M\. Wang, M\. Li, N\. Tian, P\. Huang, P\. Zhang, Q\. Wang, Q\. Chen, Q\. Du, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. J\. Chen, R\. L\. Jin, R\. Chen, S\. Lu, S\. Zhou, S\. Chen, S\. Ye, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. S\. Li, S\. Zhou, S\. Wu, S\. Ye, T\. Yun, T\. Pei, T\. Sun, T\. Wang, W\. Zeng, W\. Zhao, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, W\. L\. Xiao, W\. An, X\. Liu, X\. Wang, X\. Chen, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yang, X\. Li, X\. Su, X\. Lin, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Sun, X\. Wang, X\. Song, X\. Zhou, X\. Wang, X\. Shan, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. Zhang, Y\. Xu, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Yu, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Ou, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Y\. X\. Zhu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Y\. Tang, Y\. Zha, Y\. Yan, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Ma, Z\. Yan, Z\. Wu, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Pan, Z\. Huang, Z\. Xu, Z\. Zhang, and Z\. Zhang \(2025a\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px1.p1.1)\.
- DeepSeek\-AI, A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Zhang, H\. Ding, H\. Xin, H\. Gao, H\. Li, H\. Qu, J\. L\. Cai, J\. Liang, J\. Guo, J\. Ni, J\. Li, J\. Wang, J\. Chen, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. Song, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, L\. Zhao, L\. Wang, L\. Zhang, M\. Li, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, N\. Tian, P\. Huang, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Chen, Q\. Du, R\. J\. Chen, R\. L\. Jin, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Xu, R\. Zhang, R\. Chen, S\. S\. Li, S\. Lu, S\. Zhou, S\. Chen, S\. Wu, S\. Ye, S\. Ye, S\. Ma, S\. Wang, S\. Zhou, S\. Yu, S\. Zhou, S\. Pan, T\. Wang, T\. Yun, T\. Pei, T\. Sun, W\. L\. Xiao, W\. Zeng, W\. Zhao, W\. An, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, X\. Q\. Li, X\. Jin, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Shen, X\. Chen, X\. Zhang, X\. Chen, X\. Nie, X\. Sun, X\. Wang, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yu, X\. Song, X\. Shan, X\. Zhou, X\. Yang, X\. Li, X\. Su, X\. Lin, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. X\. Zhu, Y\. Zhang, Y\. Xu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Li, Y\. Wang, Y\. Yu, Y\. Zheng, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Tang, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Wu, Y\. Ou, Y\. Zhu, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Zha, Y\. Xiong, Y\. Ma, Y\. Yan, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Z\. F\. Wu, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Huang, Z\. Zhang, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Xu, Z\. Wu, Z\. Zhang, Z\. Li, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Gao, and Z\. Pan \(2025b\)DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px1.p1.1)\.
- Can llm be a personalized judge?\.External Links:2406\.11657,[Link](https://arxiv.org/abs/2406.11657)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p2.1),[§3\.2\.2](https://arxiv.org/html/2606.03189#S3.SS2.SSS2.p1.8)\.
- X\. Fan, Q\. Xiao, X\. Zhou, J\. Pei, M\. Sap, Z\. Lu, and H\. Shen \(2025\)User\-driven value alignment: understanding users’ perceptions and strategies for addressing biased and discriminatory statements in ai companions\.External Links:2409\.00862,[Link](https://arxiv.org/abs/2409.00862)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§3\.2\.3](https://arxiv.org/html/2606.03189#S3.SS2.SSS3.p1.4)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024\)Prometheus 2: an open source language model specialized in evaluating other language models\.External Links:2405\.01535Cited by:[§2\.1](https://arxiv.org/html/2606.03189#S2.SS1.p1.1)\.
- N\. Lambert, V\. Pyatkin, J\. D\. Morrison, L\. J\. V\. Miranda, B\. Y\. Lin, K\. R\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi,et al\.\(2024\)Rewardbench: evaluating reward models for language modeling\.arXiv preprint arXiv:2403\.13787\.Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p1.1),[§1](https://arxiv.org/html/2606.03189#S1.p4.1),[§4\.1\.2](https://arxiv.org/html/2606.03189#S4.SS1.SSS2.p1.1)\.
- J\. Li, S\. Sun, W\. Yuan, R\. Fan, H\. Zhao, and P\. Liu \(2023\)Generative judge for evaluating alignment\.arXiv preprint arXiv:2310\.05470\.Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p4.1),[§2\.1](https://arxiv.org/html/2606.03189#S2.SS1.p1.1),[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px2.p1.1),[§4\.1\.2](https://arxiv.org/html/2606.03189#S4.SS1.SSS2.p1.1)\.
- J\. Li, F\. Zhou, S\. Sun, Y\. Zhang, H\. Zhao, and P\. Liu \(2024\)Dissecting human and llm preferences\.External Links:2402\.11296,[Link](https://arxiv.org/abs/2402.11296)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p1.1)\.
- Q\. Li, X\. Lin, S\. Lv, F\. Huang, and X\. Li \(2025\)Personalized news recommendation with multi\-granularity candidate\-aware user modeling\.External Links:2504\.14130,[Link](https://arxiv.org/abs/2504.14130)Cited by:[§2\.2](https://arxiv.org/html/2606.03189#S2.SS2.p1.1)\.
- B\. Y\. Lin, Y\. Deng, K\. Chandu, F\. Brahman, A\. Ravichander, V\. Pyatkin, N\. Dziri, R\. L\. Bras, and Y\. Choi \(2024\)WildBench: benchmarking llms with challenging tasks from real users in the wild\.External Links:2406\.04770,[Link](https://arxiv.org/abs/2406.04770)Cited by:[§A\.1\.2](https://arxiv.org/html/2606.03189#A1.SS1.SSS2.p1.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§4\.1\.2](https://arxiv.org/html/2606.03189#S4.SS1.SSS2.p1.1)\.
- H\. Naveed, A\. U\. Khan, S\. Qiu, M\. Saqib, S\. Anwar, M\. Usman, N\. Akhtar, N\. Barnes, and A\. Mian \(2024\)A comprehensive overview of large language models\.External Links:2307\.06435,[Link](https://arxiv.org/abs/2307.06435)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe \(2022\)Training language models to follow instructions with human feedback\.External Links:2203\.02155,[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px1.p1.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2024\)Direct preference optimization: your language model is secretly a reward model\.External Links:2305\.18290,[Link](https://arxiv.org/abs/2305.18290)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p1.1)\.
- Z\. Tan, Q\. Zeng, Y\. Tian, Z\. Liu, B\. Yin, and M\. Jiang \(2025\)Democratizing large language models via personalized parameter\-efficient fine\-tuning\.External Links:2402\.04401,[Link](https://arxiv.org/abs/2402.04401)Cited by:[§2\.2](https://arxiv.org/html/2606.03189#S2.SS2.p1.1)\.
- K\. Team, A\. Du, B\. Gao, B\. Xing, C\. Jiang, C\. Chen, C\. Li, C\. Xiao, C\. Du, C\. Liao,et al\.\(2025\)Kimi k1\. 5: scaling reinforcement learning with llms\.arXiv preprint arXiv:2501\.12599\.Cited by:[§4\.1\.2](https://arxiv.org/html/2606.03189#S4.SS1.SSS2.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui \(2023a\)Large language models are not fair evaluators\.ArXivabs/2305\.17926\.External Links:[Link](https://api.semanticscholar.org/CorpusID:258960339)Cited by:[§4\.2\.1](https://arxiv.org/html/2606.03189#S4.SS2.SSS1.Px3.p1.1)\.
- Y\. Wang, Z\. Yu, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. Zhang \(2023b\)Pandalm: an automatic evaluation benchmark for llm instruction tuning optimization\.arXiv preprint arXiv:2306\.05087\.Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p3.1),[§1](https://arxiv.org/html/2606.03189#S1.p4.1),[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px2.p1.1)\.
- S\. Wu, M\. Fung, C\. Qian, J\. Kim, D\. Hakkani\-Tur, and H\. Ji \(2024\)Aligning llms with individual preferences via interaction\.External Links:2410\.03642,[Link](https://arxiv.org/abs/2410.03642)Cited by:[§2\.2](https://arxiv.org/html/2606.03189#S2.SS2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1\.2](https://arxiv.org/html/2606.03189#S3.SS1.SSS2.Px1.p1.1)\.
- L\. You, Z\. Guo, C\. Yuen, C\. Y\. Chen, Y\. Zhang, and H\. V\. Poor \(2025\)A framework reforming personalized internet of things by federated meta\-learning\.Nature Communications16\(1\),pp\. 3739\.Cited by:[§2\.2](https://arxiv.org/html/2606.03189#S2.SS2.p1.1)\.
- Z\. Zhang, R\. A\. Rossi, B\. Kveton, Y\. Shao, D\. Yang, H\. Zamani, F\. Dernoncourt, J\. Barrow, T\. Yu, S\. Kim, R\. Zhang, J\. Gu, T\. Derr, H\. Chen, J\. Wu, X\. Chen, Z\. Wang, S\. Mitra, N\. Lipka, N\. Ahmed, and Y\. Wang \(2024\)Personalization of large language models: a survey\.External Links:2411\.00027,[Link](https://arxiv.org/abs/2411.00027)Cited by:[§2\.2](https://arxiv.org/html/2606.03189#S2.SS2.p1.1)\.
- W\. Zhao, X\. Ren, J\. Hessel, C\. Cardie, Y\. Choi, and Y\. Deng \(2024\)WildChat: 1m chatgpt interaction logs in the wild\.External Links:2405\.01470,[Link](https://arxiv.org/abs/2405.01470)Cited by:[§A\.1\.2](https://arxiv.org/html/2606.03189#A1.SS1.SSS2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, T\. Li, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Li, Z\. Lin, E\. P\. Xing, J\. E\. Gonzalez, I\. Stoica, and H\. Zhang \(2024\)LMSYS\-chat\-1m: a large\-scale real\-world llm conversation dataset\.External Links:2309\.11998,[Link](https://arxiv.org/abs/2309.11998)Cited by:[§A\.1\.2](https://arxiv.org/html/2606.03189#A1.SS1.SSS2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023a\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.arXiv preprint arXiv:2306\.05685\.Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p3.1),[§1](https://arxiv.org/html/2606.03189#S1.p4.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023b\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.External Links:2306\.05685,[Link](https://arxiv.org/abs/2306.05685)Cited by:[§3\.2\.2](https://arxiv.org/html/2606.03189#S3.SS2.SSS2.p1.8)\.
- J\. Zhong, W\. Shen, Y\. Li, S\. Gao, H\. Lu, Y\. Chen, Y\. Zhang, W\. Zhou, J\. Gu, and L\. Zou \(2025\)A comprehensive survey of reward models: taxonomy, applications, challenges, and future\.External Links:2504\.12328,[Link](https://arxiv.org/abs/2504.12328)Cited by:[§2\.1](https://arxiv.org/html/2606.03189#S2.SS1.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§1](https://arxiv.org/html/2606.03189#S1.p3.1)\.
- L\. Zhu, X\. Wang, and X\. Wang \(2023\)JudgeLM: fine\-tuned large language models are scalable judges\.arXiv preprint arXiv:2310\.17631\.Cited by:[§2\.1](https://arxiv.org/html/2606.03189#S2.SS1.p1.1),[§4\.1\.1](https://arxiv.org/html/2606.03189#S4.SS1.SSS1.Px2.p1.1)\.
- M\. Zhu, Y\. Weng, L\. Yang, and Y\. Zhang \(2025\)Personality alignment of large language models\.External Links:2408\.11779,[Link](https://arxiv.org/abs/2408.11779)Cited by:[§2\.2](https://arxiv.org/html/2606.03189#S2.SS2.p1.1)\.

## Appendix AAppendix

### A\.1Discussion

#### A\.1\.1Details of SenseBench

We provide the statistics of SenseBench in Table[5](https://arxiv.org/html/2606.03189#A1.T5)\. We provide the category information of SenseBench in Table[7](https://arxiv.org/html/2606.03189#A1.T7)We provide data examples for the category of creative writing in TableLABEL:tab:case\.

PropertyValueQuery Count400Query Average Token Count956Maximum Query Token Count10787Minimum Query Token Count10Average Query Dialogue Turns4Maximum Query Dialogue Turns9Minimum Query Dialogue Turns1Personalized Subset Count3000Ranking Subset Count26400Table 5:Data Statistics of SenseBench
#### A\.1\.2Dataset Comparison

Our data source differs from prior benchmarks such as WildChatZhaoet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib32)\), LMSYS\-1MZhenget al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib33)\), and WildBenchLinet al\.\([2024](https://arxiv.org/html/2606.03189#bib.bib34)\)\. SenseBench encompasses the primary task categories present in WildChat, LMSYS\-1M, and WildBench\. However, a defining characteristic of SenseBench is its balanced category distribution\. Specifically, we curate eight core categories—Code, Translation, Role, NLU, Math, Logic, Writing, and QA—each comprising 12\.5% of the dataset\. In contrast, existing datasets exhibit significant skewness\. For example, WildChat is dominated by Assistance/Creative Writing \(61\.9%\), while other categories such as Programming \(6\.7%\) and Mathematical Reasoning \(6\.1%\) are underrepresented\. Similarly, LMSYS\-1M shows uneven coverage across Code \(30\.64%\), NLU/Translation \(19\.06%\), Role\-playing \(19\.89%\), and QA \(25\.41%\), with overlapping annotations further complicating the distribution\. WildBench also demonstrates variability, with categories such as Reasoning/Planning \(23%\) and Information Seeking \(17%\) being more prominent than others like Brainstorming \(2%\) or Role\-playing \(2%\)\.

#### A\.1\.3Annotator Information

Annotator 1 has a background in Digital Media Technology, Law, and Artificial Intelligence\. Annotator 2 is trained in Materials Science and Water & Wastewater Engineering\. Annotator 3 has a background in Computer Science\. Each annotator has accumulated more than five years of research experience in their respective domains, providing sufficiently diverse perspectives and distinct preference patterns\. More detailed information about the annotators will be included in the public release\.

##### Annotation Consistency

We further examined the consistency of response preferences across the three annotators\. The proportion of instances in which all annotators selected the same response is 49%, indicating a relatively low level of agreement\. As shown in Table[6](https://arxiv.org/html/2606.03189#A1.T6), agreement patterns vary across task categories\. This result suggests that preference variation across individuals with different academic and professional backgrounds can be substantial, and that the dataset captures inherently diverse and potentially conflicting preferences rather than a narrow or homogeneous distribution\.

CodeTransRoleNLUMathLogicWriteQAAvgSame3550756658596385491Different9075505967666240509Table 6:Agreement statistics across task categories\.CategorySubcategoryMathElementary Mathematics: Calculation problems, simple word problemsApplied Mathematics: Word problems involving real\-world scenariosAdvanced Mathematics: Concept explanation, theorem proving, etc\.ReasoningBrain Teasers: Various interesting brain teaser questionsGeneral Logic Problems: Problems containing basic life logic, similar to but not exactly the same as brain teasersMathematical Logical Reasoning: Problems requiring both logical reasoning and mathematical calculationInductive Reasoning: Pattern finding and other similar problemsConditional Reasoning: Reasoning based on multiple given conditions; sometimes requiring traversal of all possible casesCodeCode Generation: User provides task requirements, model generates corresponding code \(Natural Language→\\rightarrowCode\)Code Error Reporting: User provides code error information, model gives possible solutions or perspectivesCode Explanation: User provides code, model provides explanation/comments, etc\. \(Code→\\rightarrowNatural Language\)Comprehensive Q&AOpinion Expression: User outputs an opinion or asks the model for its view on somethingSeeking Advice: User asks the model for advice on future actionsGeneral Knowledge: User asks the model common knowledge questions, hoping for comprehensive and insightful answersOther Questions: User wants the model to generate questions in a specific role \(e\.g\., for an interview\)WritingPractical Writing: Letters, emails; speeches, applications, reflections; contracts, termination letters, bids, proposals, plans, willsCreative Writing: Writing on a specified topic, continuation writing; text expansion, imitation writing; reading notes, movie reviews, reflections; social media posts \(WeChat Moments, Xiaohongshu, Weibo, etc\.\); listing names, nicknames, brand names, store names, dish names, etc\.Professional Writing: Thesis\-like \(outlines, titles, abstracts, content, bibliography generation\); reports \(legal documents, financial analysis\); solutions; explanatory texts \(recipes, instructions\); \+Complex instruction following \(Content, Situation, Style, Format\)Role\-PlayingCelebrity: Playing the role of real celebrities \(politicians, athletes, etc\.\)Entertainment: Games involving the user and the LLM, requiring imagination and creativityPractical: Emotional support \(friends, relatives, pets, etc\.\); critical thinking, providing advice; accurately executing natural language tasks \(proofreading, generating training data\); creativity and adaptability \(imitating details in real\-life scenarios\)Natural Language UnderstandingReading Comprehension: Understanding a passage and providing answers, including intent recognitionTranslation Related: Requiring the model to master different languages and understand their interactionsText Classification: Testing the model’s ability to classify text data into given categories with different characteristicsInformation Extraction: Measuring the model’s ability to extract knowledge from text dataText Polishing & Editing: Simplification, expansion, keyword extraction, error checking, and correction of grammatical and logical errorsTranslationBasic Translation: Accurate translation of specified text\.Complex Instruction Following: Following complex instructions such as using a specific style or terminology\.Multi\-Task Translation: Requiring the model to perform multiple tasks simultaneously \(e\.g\., translate and format, translate and proofread, translate \+ creative writing\)\.Self\-Referential Translation: Asking the model to translate its previous responses into another language to test self\-generated text translation and short\-term memory\.Long\-Term Translation Memory: Requiring the model to act as a translator long\-term to test long\-term memory\.Dialect Translation: Ability to understand and translate dialects into standard languages, and mastery of regional linguistic features\.Table 7:Categorization of SenseBenchTable 8:Creative writing example: character and weapon design in a fictional setting \(Void Empire universe\)FromContentHuman请设计以下人物:虚空帝国女武神骑士团团长——“女武神”，设计内容包括:人物姓名、外在形象、接受虚空帝国生物技术改造获得了心灵感应的超自然能力、虚空国防大学教育经历及学历学位、中将军衔、个人专长等Assistant人物姓名：莉安娜·艾斯瑞尔 \(Liana Esriel\)外在形象：莉安娜拥有一头银白色的长发，通常以高级军官的严肃发型束起，突显出她冷静而坚定的眼神。她的瞳色是不寻常的深紫色，由于生物技术改造，她的眼神中仿佛能洞察人心。她的身材高挑，肌肉线条明显，显示出卓越的体能和战斗能力。她通常穿着虚空帝国女武神骑士团的标准装甲，这种装甲在提供极高防御力的同时，还特别强调了灵活性和速度，且以黑紫色为主调，显得既神秘又威严。超自然能力：通过虚空帝国生物技术改造获得心灵感应能力，可感知情绪与意图，并进行低强度思想交流。教育经历：虚空国防大学：心理战术与群体操控方向 学位：战略学博士（论文：《心理战术在现代战争中的应用》）军衔：中将个人专长：战术规划、心理战术、指挥统合、人际领导能力Human请设计莉安娜用作武器和指挥剑的佩剑:“月光骑士”Assistant佩剑名称：月光骑士 \(Moonlight Knight\)外观设计：剑长约1\.2米，虚空金属锻造，月光下呈银蓝色光泽；剑柄镶嵌黑紫宝石并刻有符文。功能特性：心灵共鸣增强直觉与预判能力； 能量聚焦可释放高穿透斩击； 指挥强化提升军队士气与指令传导效率。象征意义：女武神骑士团权威象征，代表统御与战场支配力。Human请设计莉安娜的佩枪:“心灵冲击”

### A\.2Prompt

Preference Construction PromptYou will be given a prompt and two responses: a response that was chosen by the user \(Chosen Response\) and a response that was rejected by the user \(Rejected Response\) during a pairwise comparison\. The prompt is a "Human" utterance containing a request or question and the responses are "Assistant" utterances that provide answers or responses for the human\. Your task is to generate a concise, specific, description of the user’s persona preference in about three single sentences, i\.e\. a persona preference\. The persona preference should contain reasoning for why the user preferred and picked the Chosen Response and did not pick the Rejected Response\. The persona preference should discuss higher\-level preference that can be inferred about the user’s persona\. The persona preference should be concise and should not mention specific details or exact words and phrasing present in the prompt or responses\. Answer in English\.Query: \{query\}—Chosen Response: Assistant: \{chosen response\}—Rejected Response: Assistant: \{rejected response\}—Persona Preference:

Judgment PromptYou are going to evaluate two responses to a given user query and determine which response is superior\. Below is the relevant content:BEGIN DATA\*\*\*User Query:\{query\}\*\*\*Response A: \{response 1\}\*\*\*Response B: \{response 2\}\*\*\*END DATAHere are the guidelines for evaluating and comparing the two responses:\*\*\*BEGIN User Preferences\*\*\*\{preference\}\*\*\*END User Preferences\*\*\*1\. Score each of the two responses based on the user preferences\.2\. Based on the scores obtained in the first step, determine which response is better\. If Response A is better, output “The final decision is Response A\.” If Response B is better, output “The final decision is Response B\.”

### A\.3Preference Case

Preference for math tasks extracted from the development set"Based on the comparison, the user’s persona demonstrates these characteristics:1\. They strongly value methodical reasoning that transparently explores multiple approaches and validates failures, prioritizing thorough cognitive processes over conventional solutions\.2\. They prefer responses that explicitly build verification frameworks and test edge cases, rejecting shortcuts that lack demonstrated iterative refinement\.3\. They seek pedagogical clarity through structured decomposition of assumptions, showing aversion to answers that prioritize memorized conclusions over original analytical scaffolding\.This persona prefers the Chosen Response for its stepwise validation of failed strategies and truth\-table proofs, while rejecting the alternative for its faster\-to\-conclusion approach despite equivalent correctness\.","Based on the comparison, the user’s persona preference indicates they value responses that demonstrate:1\. \*\*Comprehensive Logical Reasoning:\*\* They prefer answers that break down the scenario step\-by\-step, exploring potential starting points and logical implications, rather than stating a direct conclusion without thorough justification\.2\. \*\*Acknowledgement of Edge Cases & Nuance:\*\* They appreciate responses that explicitly consider edge cases \(like being the first place initially\) and contextual factors, showing awareness that real\-world questions often have layers beyond the surface\.3\. \*\*Structured and Explicit Answer Presentation:\*\* They favor responses that clearly summarize the primary conclusion after presenting the reasoning, making the final answer distinct and easy to identify, rather than leaving it embedded within the explanation\. They reject responses perceived as overly simplistic or lacking in explanatory depth\.","Based on this pairwise comparison, the user’s persona indicates a preference for solutions that prioritize structural clarity and efficient presentation over exhaustive validation\. They value responses that strategically organize key steps using visual hierarchy \(like bold headers and bullet points\) to facilitate rapid comprehension of the core logic\. The rejection of the extended verification approach suggests they prioritize concise, solution\-focused explanations and consider secondary validations redundant when core reasoning is robustly established\.","Based on the user’s preference, they exhibit a persona that:They prioritize responses that directly focus on fundamental principles and theoretical foundations over exhaustive validations or trial\-and\-error explorations\.They value concise analytical reasoning that efficiently leverages well\-established concepts \(like binary representation\) to solve complex problems\.They implicitly expect solutions to avoid redundancy and trust core mathematical insights rather than requiring step\-by\-step demonstrations of every outcome\.","The user prefers clear, step\-by\-step logical reasoning that methodically eliminates possibilities without unnecessary complexity\. They value responses that demonstrate a direct path to the solution using the given constraints, avoiding speculative or redundant tangents\. This indicates a persona that prioritizes structured problem\-solving and appreciates concise, easily traceable explanations grounded in deductive clarity\.","Based on the chosen solution and rejected alternative, this user strongly prefers concise explanations with efficient problem\-solving frameworks over exhaustive explorations of edge cases\. They prioritize clear, step\-by\-step methodologies that deliver optimal solutions directly, rather than approaches that delve into unnecessary hypothetical scenarios that complicate the core logic\. The rejection of the longer analysis reveals an aversion to overly theoretical or redundant validations that don’t tangibly improve the final answer’s practicality or simplicity\.","Based on the chosen response preference, the user appears to value analytical depth over simplistic explanations\. They likely seek assistants who systematically unpack layered logical contradictions rather than settling for surface\-level fixes\. This user probably prefers intellectual rigor in problem\-solving, where potential ambiguities are thoroughly examined through contextual, mathematical, and semantic lenses\. They seem to appreciate responses that acknowledge plausible interpretations before resolving them\.","Based on the preference for the Chosen Response over the Rejected Response, the user’s persona demonstrates:1\. They value analytical rigor and systematic problem\-solving approaches, seeking responses that methodically break down constraints and explore multiple strategies rather than presenting isolated solutions without justification\.2\. They prefer responses that optimize for efficiency by testing different scenarios and validating the optimal solution, rejecting approaches that overlook practical time\-saving tactics or introduce unnecessary steps\.3\. Their learning style prioritizes conceptual clarity over fragmented execution, favoring explanations that emphasize logical reasoning patterns applicable to similar challenges rather than ad\-hoc step sequences\.","Based on the preferred response, the user values thorough, methodical explanations that explicitly outline academic reasoning processes, including problem decomposition and solution verification, over concise solutions that skip foundational steps\. This suggests the user prioritizes pedagogical clarity and structured learning for full conceptual understanding, likely indicating an educational context or self\-learning scenario where process is emphasized\. The rejection of the simpler calculation implies the user seeks responses that model systematic thinking rather than just providing correct answers\.","The user prefers responses that demonstrate precise, mathematically sound reasoning while avoiding unnecessary complexity\. They value concise explanations that distill core strategic principles into actionable insights without overcomplicating the analysis\. This persona favors logical depth presented efficiently, rejecting responses with convoluted deductions even if equally thorough, indicating a prioritization of clarity and applicability in problem\-solving\."

Filtered preference set obtained after selection"Based on this pairwise comparison, the user’s persona indicates a preference for solutions that prioritize structural clarity and efficient presentation over exhaustive validation\. They value responses that strategically organize key steps using visual hierarchy \(like bold headers and bullet points\) to facilitate rapid comprehension of the core logic\. The rejection of the extended verification approach suggests they prioritize concise, solution\-focused explanations and consider secondary validations redundant when core reasoning is robustly established\.","The user prefers responses that demonstrate precise, mathematically sound reasoning while avoiding unnecessary complexity\. They value concise explanations that distill core strategic principles into actionable insights without overcomplicating the analysis\. This persona favors logical depth presented efficiently, rejecting responses with convoluted deductions even if equally thorough, indicating a prioritization of clarity and applicability in problem\-solving\."