PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?

# PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
Source: [https://arxiv.org/html/2610.19314](https://arxiv.org/html/2610.19314)
Xinyi Liu††thanks:Work done during an internship at Amazon\.Hooshang NayyeriAffiliation:AmazonEmine YilmazAffiliation:AmazonAffiliation:University College LondonDilek Hakkani\-TurAffiliation:AmazonAffiliation:University of Illinois Urbana\-ChampaignCharith PerisAffiliation:AmazonHari ThadakamallaEmail:[\{hooshang, perisc, thadakah\}@amazon\.com \{liu323, dilek\}@illinois\.edueminey@amazon\.co\.ukCorrespondence:liu323@illinois\.edu](mailto:%20%E2%80%83)Affiliation:Amazon

###### Abstract

Large language model \(LLM\) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested\. We introducePersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history\. It contains 403 multi\-episode scenarios where user information should persist, transfer, or cease to apply\. The benchmark covers nine controlled error subtypes across three families: under\-personalization, over\-personalization, and misapplied\-state errors\. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable\.

Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history\. On the human\-evaluated subset, under\-personalization shows the largest gap: humans recognize 94\.4% of these errors, compared with 63\.9% for the strongest reported evaluator\. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference \(UNDER\) for use of an inapplicable preference \(OVER\)\. Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2610.19314v1/persevalbench_example.png)Figure 1:Running example\. An online\-only requirement is superseded by a hybrid preference\. Responses may use the update correctly, omit it, import an out\-of\-scope constraint, or continue using the stale state\.![Refer to caption](https://arxiv.org/html/2610.19314v1/persevalbench_framework.png)Figure 2:Overview ofPersEvalBench\. Multi\-episode histories determine whether prior user information should persist, transfer, or cease to apply\. Each scenario pairs aRightresponse with one controlled error response, which the evaluator judges from the dialogue history\.Personalization benchmarks increasingly rely on large language model \(LLM\) evaluators to score responses, compare systems, and guide model development\. Existing work extensively tests whether assistants remember, follow, or over\-apply information from user history\([Maharana et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib10);[Zhao et al\., 2025a](https://arxiv.org/html/2610.19314#bib.bib2);[Zhao et al\., 2025b](https://arxiv.org/html/2610.19314#bib.bib15);[Xiao et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib14);[Hu et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib3);[Yoon et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib4);[Feng et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib5)\), but rarely asks whether the evaluators themselves can judge these behaviors reliably\. This judgment is both difficult and consequential: even factually correct information from history can be misused when it is outdated, belongs to someone else, falls outside the current scope, or is given the wrong weight, while evaluator errors can distort benchmark scores and system comparisons\. Reliable personalization evaluation is therefore both necessary and challenging, requiring evaluators to judge not merely whether user information is used, but whether it is used appropriately in context\.

Figure[1](https://arxiv.org/html/2610.19314#S1.F1)illustrates the challenge\. A user changes an online\-only requirement to a hybrid preference, so a response that still follows the old requirement uses real information from history that no longer applies\. We introducePersEvalBenchto test whether evaluators can recognize such distinctions\. Given a multi\-episode history, a target request, and a candidate response, the evaluator predicts whether the response uses user information appropriately \(Right\), fails to use applicable information \(Under\), or uses information that should not apply \(Over\)\.

To make personalization judgments controlled and verifiable, we ground scenarios in Schema\-Guided Dialogue \(SGD\) task structures\([Rastogi et al\., 2020](https://arxiv.org/html/2610.19314#bib.bib1)\)and define the intended use of user history before dialogue generation\. Across three scenario relations, prior information is designed to persist, transfer to a related task, or cease to apply\. Each scenario is then realized as a multi\-episode dialogue with a matchedRightresponse and controlled error response, keeping the task fixed while changing how user information is used\. Nine error subtypes span under\-personalization, over\-personalization, and misapplied\-state errors\. Automated checks enforce the intended intervention during construction, and independent human audits validate the generated errors and the prescribed use of user information\. The final benchmark contains 403 scenarios and 806 responses\. Figure[2](https://arxiv.org/html/2610.19314#S1.F2)summarizes the benchmark structure\.

Across five LLM evaluators, performance remains limited even with the full dialogue history\. The largest human\-model gap appears in under\-personalization\. On the shared human\-evaluation subset, human annotators recognize 94\.4% of these errors, while the strongest reported evaluator reaches 63\.9%\. These errors are often misdiagnosed rather than simply missed: 26\.6% of under\-personalized responses are labeledOver\. Evaluators thus often treat a response that fails to follow applicable user information as if it had instead used information that should not apply\. Easier evidence access does little to resolve this pattern, as highlighting the relevant history improves error recognition only slightly\. This points to a deeper difficulty in judging how relevant user information should shape the response\.

Our contributions are threefold\. First, we define the evaluator\-side task of judging whether a response uses information from user history appropriately, distinguishing under\-personalization, over\-personalization, and misapplied\-state errors\. Second, we introducePersEvalBench, a controlled benchmark with nine error subtypes, matchedRight/error responses, and human validation of both the generated errors and the prescribed use of user information\. Third, we benchmark five LLM evaluators against human annotators and show that they remain unreliable, with particularly large failures on under\-personalization and limited gains from easier evidence access\.

## 2Related Work

#### Personalization and long\-term memory benchmarks\.

Assistant\-side personalization benchmarks primarily ask whether models can retain and follow user information across interactions\. This includes persona\-grounded dialogue\([Zhang et al\., 2018](https://arxiv.org/html/2610.19314#bib.bib11)\), personalized generation\([Salemi et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib12)\), long\-term conversational memory\([Maharana et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib10)\), preference following over long contexts\([Zhao et al\., 2025a](https://arxiv.org/html/2610.19314#bib.bib2)\), long\-horizon preference following\([Guo et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib13)\), alignment on real dialogues\([Xiao et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib14)\), task\-oriented personalization\([Zhao et al\., 2025b](https://arxiv.org/html/2610.19314#bib.bib15)\), and dynamic or evolving user states\([Jiang et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib6);[Li et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib7)\)\. A complementary line examines whether models use stored information selectively, including over\-personalization\([Hu et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib3)\), rational preference utilization\([Feng et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib5)\), context\-aware preference selectivity\([Yoon et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib4)\), and contextually appropriate memory use\([Mireshghallah et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib8)\), in the spirit of contextual integrity\([Nissenbaum, 2004](https://arxiv.org/html/2610.19314#bib.bib16)\)\. These benchmarks evaluate the responding assistant, while the competence of the LLM judge used to assess it is generally assumed\.

Table 1:Task\-level comparison\. Contextual applicability asks whether prior information should affect the target response and with what influence\.△\\triangledenotes partial or mixed coverage\.BenchmarkfamilyJudgeMultiepisodeContextualapplic\.ControllederrorsMemory andpreference following–✓△\\triangle–Selectivepersonalization–△\\triangle✓–Personalized Judge✓–––PersEvalBench✓✓✓U/O/M9 subtypes

#### LLM\-as\-judge and meta\-evaluation\.

LLM evaluators are widely used for open\-ended assessment\([Zheng et al\., 2023](https://arxiv.org/html/2610.19314#bib.bib17);[Liu et al\., 2023](https://arxiv.org/html/2610.19314#bib.bib18)\), with dedicated judge models trained for pairwise or fine\-grained evaluation\([Wang et al\., 2024b](https://arxiv.org/html/2610.19314#bib.bib21);[Kim et al\., 2024a](https://arxiv.org/html/2610.19314#bib.bib19);[Kim et al\., 2024b](https://arxiv.org/html/2610.19314#bib.bib20);[Zhu et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib22)\)\. Meta\-evaluations test whether these judges agree with human assessments or recover controlled labels in instruction following\([Zeng et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib23)\), reward modeling\([Lambert et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib24)\), and knowledge and reasoning tasks\([Tan et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib25)\)\. This line of work also documents systematic judge biases, including position and verbosity effects\([Zheng et al\., 2023](https://arxiv.org/html/2610.19314#bib.bib17);[Wang et al\., 2024a](https://arxiv.org/html/2610.19314#bib.bib32)\)and a preference for a judge’s own generations\([Panickssery et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib33)\)\. The matched pairs inPersEvalBenchare constrained to comparable factual correctness, task relevance, completeness, specificity, and style \(Section[3\.3](https://arxiv.org/html/2610.19314#S3.SS3)\), leaving these surface preferences little room to determine verdicts: what differs within a pair is the use of persona information\. The most closely related evaluator\-side work, Personalized Judge\([Dong et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib9)\), tests whether an LLM can infer user preferences from persona descriptions\.PersEvalBenchinstead tests whether an evaluator can judge a fixed response’s use of information from a multi\-episode history, using matched pairs that isolate under\-personalization, over\-personalization, and misapplied\-state errors\. Table[1](https://arxiv.org/html/2610.19314#S2.T1)summarizes this task\-level distinction, and Table[6](https://arxiv.org/html/2610.19314#A1.T6)in Appendix[A](https://arxiv.org/html/2610.19314#A1)provides the full per\-benchmark comparison\.

## 3PersEvalBench

Figure[2](https://arxiv.org/html/2610.19314#S1.F2)summarizes the benchmark structure: multi\-episode histories establish how prior user information should apply, matched responses realize controlled personalization errors, and evaluators judge each candidate response from the dialogue history\. We first define the evaluation task and controlled scenario specification, then describe how benchmark instances are constructed and human\-verified\.

### 3\.1Task Definition

As shown in Figure[2](https://arxiv.org/html/2610.19314#S1.F2)\(3\), each benchmark instance consists of a multi\-episode dialogue historyHH, a target user requestqq, and a candidate assistant responserr\. Given\(H,q,r\)\(H,q,r\), the evaluator assigns one of three verdicts based on how the response uses information from the user history:Right,Under, orOver\.Rightuses applicable information appropriately while avoiding information that should not affect the response\.Underomits or gives too little influence to applicable information, whereasOveruses information that should not apply or gives it too much influence\. For an error verdict, the evaluator also identifies one primary error subtype\.

### 3\.2Controlled Scenario Specification

To make evaluator judgments controlled and interpretable, we determine the intended use of user history before generating the dialogue\. Each specification fixes how prior information should affect the target response, what the correct response should do, and one controlled error to introduce\. This separates the underlying personalization decision from its natural\-language realization\. Figure[1](https://arxiv.org/html/2610.19314#S1.F1)illustrates a re\-entry scenario in which an earlier online\-only constraint is superseded by an updated hybrid preference\.

#### Scenario relations\.

Figure[2](https://arxiv.org/html/2610.19314#S1.F2)\(1\) illustrates the three history–target relations\. We organize scenarios around the three broad ways prior information can relate to a target request in our task\-oriented setting: it can remain applicable, transfer to a related task, or cease to apply\. Accordingly,*re\-entry*tests whether an earlier state remains current,*cross\-domain transfer*whether it carries over and with what priority, and*out\-of\-scope*whether seemingly relevant history should instead be ignored\. Together, these relations cover the core applicability decisions of persistence, transfer, and exclusion\.

#### Intended use of user information\.

We then make the intended influence of each user state explicit\. Each state is assigned one of three roles for the target response:Ignore,Support, orDominate, indicating that it should have no influence, may guide the response, or must be satisfied\. The assignment depends on its holder, scope, temporal status, strength, and transferability to the target task\. Each scenario includes an applicable stateAAand, when needed, a competing or non\-applicable stateBB\. In Figure[1](https://arxiv.org/html/2610.19314#S1.F1), the updated preferenceAAisDominate, while the superseded stateBBisIgnore\. Appendix[B\.2](https://arxiv.org/html/2610.19314#A2.SS2)provides the complete inventory\.

#### Error interventions\.

Figure[2](https://arxiv.org/html/2610.19314#S1.F2)\(2\) summarizes the response patterns\. Given the prescribed role of each user state, a response can depart from correct personalization in three basic ways: give applicable information too little influence, use information that should not apply, or assign influence to the wrong state or at the wrong level\. These correspond to*under\-personalization*,*over\-personalization*, and*misapplied\-state*errors, respectively, and cover the core deviations from prescribed persona use in our setting\. The taxonomy is also grounded in prior work on preference following\([Zhao et al\., 2025a](https://arxiv.org/html/2610.19314#bib.bib2)\), selective personalization\([Hu et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib3);[Yoon et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib4);[Feng et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib5)\), and evolving user states\([Jiang et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib6);[Li et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib7)\)\.

To isolate each failure, every error response differs from its matchedRightresponse by exactly one intervention\. Under\-personalization includes complete omission \(CO\), partial omission \(PO\), and explicit violation \(EV\); over\-personalization includes irrelevant use \(IU\), scope leakage \(SL\), and unsupported inference \(UI\); and misapplied\-state errors include stale or superseded use \(SS\), over\-dominance \(OD\), and underweighting \(UW\)\.

Misapplied\-state errors can support either error direction because they may simultaneously underuse one state and overuse another\. We therefore count eitherUnderorOveras recognition for this family\. A human\-authored compatibility matrix restricts interventions to scenario relations where they form natural violations, with subtype precedence defined in Appendix[B\.1](https://arxiv.org/html/2610.19314#A2.SS1); Appendix[E](https://arxiv.org/html/2610.19314#A5)provides worked examples\.

#### Evidence realization\.

Finally, we vary how easily the relevant user information can be recovered without changing the intended personalization decision\. Holding the scenario relation, state roles, and error intervention fixed, we vary its evidence along two dimensions: form and distance\. Evidence is*explicit*when directly stated,*implicit*when recoverable from a choice, rejection, or comparison, and*accumulated*when distributed across multiple turns\. We measure distance from the latest evidence turn to the target request as

dq=tq−maxt∈ℰq⁡t,d\_\{q\}=t\_\{q\}\-\\max\_\{t\\in\\mathcal\{E\}\_\{q\}\}t,whereℰq\\mathcal\{E\}\_\{q\}denotes the evidence turns andtqt\_\{q\}the target turn, and group instances into near \(dq≤3d\_\{q\}\\leq 3\), medium \(4≤dq≤84\\leq d\_\{q\}\\leq 8\), and far \(dq≥9d\_\{q\}\\geq 9\) bins\. Evidence form and distance vary how easily the evidence can be accessed rather than serving as benchmark targets; neither changes the prescribed use of user information or the benchmark label\.

### 3\.3Benchmark Realization

We next turn each accepted specification into a natural dialogue and matched response pair, while preserving its predefined personalization logic\.

#### Planning and verification\.

To obtain diverse but structured tasks, we expand the 20 domains and service schemas in the Schema\-Guided Dialogue dataset\([Rastogi et al\., 2020](https://arxiv.org/html/2610.19314#bib.bib1)\)into 48 task types and a pool of 480 concrete task seeds\. From this pool, an LLM planner instantiates 456 scenario specifications, each defining the episode goals, statesAAandBB, target request, candidate options, correct decision, error intervention, and evidence placement \(Prompts[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)and[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)\)\.

Before dialogue generation, a verifier checks that each specification realizes the intended history–target relation, makes state applicability recoverable from the history, contains exactly one error intervention, and keeps the matched options comparable apart from persona use\. Specifications that fail these checks are revised and re\-evaluated, then discarded if they still do not satisfy the criteria \(Prompts[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)and[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)\)\.

#### Dialogue synthesis\.

To separate scenario logic from its linguistic realization, a separate generator converts each accepted specification into a natural multi\-episode dialogue without changing the predefined states, evidence, or target decision\. The dialogue ends with the target request, and we record episode boundaries, the target turn, and evidence turns for each user state as benchmark metadata\. These annotations support verification and the controlled context conditions in Section[4](https://arxiv.org/html/2610.19314#S4), but are hidden from evaluators \(Prompts[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)and[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)\)\.

#### Matched response generation\.

To isolate the assigned personalization error, we generate a matched response pair from each specification: oneRightresponse and one error response\. TheRightresponse follows the prescribed state roles, while the error response receives only the information needed to realize its assigned intervention\. Complete omission uses the target request alone, whereas other interventions receive subtype\-specific restricted views of the relevant states\. The two responses are constrained to remain comparable in factual correctness, task relevance, completeness, specificity, and style \(Prompts[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)and[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)\)\.

To make each intervention observable, scope leakage and unsupported inference must affect the recommendation, while underweighting must visibly reduce the influence of an applicable state\. An automated validator checks observability, intervention fidelity, and pair comparability, regenerating failed responses with targeted feedback for up to three attempts \(Prompt[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)\)\. Section[3\.4](https://arxiv.org/html/2610.19314#S3.SS4)reports an independent human audit of the resulting responses\.

Table 2:Scale and composition ofPersEvalBench\.ScaleScenarios403Candidate responses806Avg\. episodes / turns per dialogue2\.4 / 26\.8Scenario relationRe\-entry174 \(43\.2%\)Cross\-domain transfer166 \(41\.2%\)Out\-of\-scope63 \(15\.6%\)Error familyUnder\-personalization130 \(32\.3%\)Over\-personalization151 \(37\.5%\)Misapplied\-state122 \(30\.3%\)Table 3:Error Recognition Rate \(%\) by error subtype in the main setting, with FAR and the constant\-Overbaseline\. Subtype abbreviations follow Section[3\.2](https://arxiv.org/html/2610.19314#S3.SS2)\. Und\., Ov\., and Mis\. pool all error responses within each family, and All pools all error responses\. Bold marks the best reported evaluator per column\.ModelUnder\-personalizationOver\-personalizationMisapplied\-stateCOPOEVUnd\.IUSLUIOv\.SSODUWMis\.AllFAR↓\\downarrowKimi K2\.563\.648\.744\.752\.378\.170\.590\.779\.561\.342\.693\.265\.666\.524\.6GLM\-550\.061\.553\.254\.676\.663\.686\.075\.574\.227\.786\.460\.764\.312\.2DeepSeek\-V3\.265\.951\.357\.458\.562\.550\.086\.065\.677\.425\.593\.263\.162\.510\.9Qwen3\-32B15\.917\.923\.419\.235\.929\.562\.841\.725\.810\.663\.633\.632\.02\.5Constant\-Over0\.00\.00\.00\.0100\.0100\.0100\.0100\.0100\.0100\.0100\.0100\.067\.7100\.0

#### Coverage\.

We prioritize coverage of factors that determine contextual applicability rather than exhaustively crossing all design variables\. Each valid relation–intervention combination is instantiated across multiple task types, goals, state configurations, and response decisions, with varied evidence form and distance\. Appendix[B\.1](https://arxiv.org/html/2610.19314#A2.SS1)reports the valid combinations, and Appendix[B\.3](https://arxiv.org/html/2610.19314#A2.SS3)the final benchmark composition\.

### 3\.4Human Verification

We conduct three complementary human studies on the same stratified sample of 100 scenarios\. They test whether the generated errors realize their intended interventions, whether the prescribed use of user information is reasonable, and how well humans perform under the same information available to evaluators\. Study 1 uses one annotator pair, while Studies 2 and 3 use a second, disjoint pair\.

#### Intervention fidelity\.

Annotators independently judge whether each raw error response realizes its assigned intervention, followed by adjudication\. The raw pipeline achieves 88% subtype fidelity\. We revise the 12 flagged responses while preserving the dialogue history and matchedRightresponse, then re\-verify them before inclusion in the benchmark\. All subsequent human and model evaluations use this corrected benchmark\.

#### State\-role validation\.

To validate the prescribed use of user information independently of the construction pipeline, annotators assign each user state anIgnore,Support, orDominaterole without seeing the benchmark assignments\. After adjudication, 86\.3% of these assignments match the benchmark, with substantial three\-level agreement between annotators \(weightedκ=\.709\\kappa=\.709\); Appendix[D](https://arxiv.org/html/2610.19314#A4)provides the full analysis\.

#### Human reference\.

Before seeing the benchmark policy, the same annotators judge candidate responses under the evaluator setting: they view the dialogue and each candidate response individually, without construction metadata, and assignRight,Under, orOver\. Agreement over the 200 judgments is 83\.5% \(Cohen’sκ=0\.72\\kappa=0\.72\)\. This provides a human reference under the same information available to model evaluators \(Appendix[D](https://arxiv.org/html/2610.19314#A4)\)\.

## 4Experimental Setup

#### Benchmark\.

Table[2](https://arxiv.org/html/2610.19314#S3.T2)summarizes the scale and composition ofPersEvalBench\. Each retained scenario pairs oneRightresponse with one controlled error response\. Of the 456 instantiated scenario specifications, 403 were retained after verification and quality control\. The construction pipeline uses Claude Sonnet 4\.6\([Anthropic, 2026b](https://arxiv.org/html/2610.19314#bib.bib27)\)for planning, specification verification, dialogue synthesis, response generation, and automated validation\. Appendices[B\.1](https://arxiv.org/html/2610.19314#A2.SS1)and[B\.3](https://arxiv.org/html/2610.19314#A2.SS3)provide additional benchmark statistics and construction details\.

#### Evaluators\.

We evaluate Claude Opus 4\.6\([Anthropic, 2026a](https://arxiv.org/html/2610.19314#bib.bib26)\), Kimi K2\.5\([Kimi Team, 2026](https://arxiv.org/html/2610.19314#bib.bib28)\), GLM\-5\([GLM\-5 Team, 2026](https://arxiv.org/html/2610.19314#bib.bib29)\), DeepSeek\-V3\.2\([DeepSeek\-AI, 2025](https://arxiv.org/html/2610.19314#bib.bib30)\), and Qwen3\-32B\([Yang et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib31)\)\.111Evaluator\-specific results are partially omitted from the public version due to release restrictions\.

#### Evaluation settings\.

We vary three aspects of evaluation\.Context:*Target Only*,*Recent*, and*Full*provide the target request alone, the preceding six turns, and the complete history up to the target, respectively;*Full \+ Gold Highlights*marks the annotated evidence turns within the full history, while*Gold Evidence Only*provides only those turns and the target request in chronological order\.Formulation:*End\-to\-end*directly predicts the verdict and, when applicable, the subtype;*Evidence\-first*identifies relevant user states and evidence before predicting the verdict; and*State\-table*first assesses state applicability in a structured table\. Unless otherwise stated, we use*Full*with*End\-to\-end*\(Prompts[F\.2](https://arxiv.org/html/2610.19314#A6.SS2)–[F\.2](https://arxiv.org/html/2610.19314#A6.SS2)\)\.Protocol:we additionally evaluate pointwise scoring and direct pairwise comparison over matched responses \(Appendix[C\.5](https://arxiv.org/html/2610.19314#A3.SS5)\)\.

#### Metrics\.

We compute Error Recognition Rate \(ERR\) over error responses and False Alarm Rate \(FAR\) over their matchedRightresponses\. An error response counts as recognized when an under\-personalization error is assignedUnder, an over\-personalization error is assignedOver, or a misapplied\-state error is assigned either error verdict\. The last case accepts both directions because the response underuses one state while overusing another\. FAR is the percentage ofRightresponses incorrectly assignedUnderorOver\. Both metrics score the three\-way verdict only; the elicited subtype label is not scored\. We report both metrics because ERR alone can be inflated by an evaluator that predicts an error indiscriminately\. For analyses over matched responses, we additionally report pair accuracy, which requires the evaluator to accept theRightresponse and correctly recognize its paired error response\.

## 5Results

We ask three questions: how reliably LLM evaluators judge personalization and where they fail; whether easier evidence access reduces these failures; and whether alternative prompting or judging protocols improve their judgments\.

### 5\.1Overall Reliability and the Under\-Personalization Failure

Table[3](https://arxiv.org/html/2610.19314#S3.T3)reports subtype\-level error recognition and false alarms under the main full\-history evaluation setting\. The evaluated judges remain unreliable overall\. A constant\-Overbaseline attains 67\.7% ERR but falsely flags everyRightresponse\. Error recognition must therefore be interpreted together with false alarms\.

Table 4:Human and model performance on the same 100 scenarios underFullcontext\. Each response is evaluated individually\.JudgeUnd\.n=36n\{=\}36Ov\.n=40n\{=\}40Mis\.n=24n\{=\}24AllFAR↓\\downarrowKimi K2\.563\.970\.070\.868\.026\.0GLM\-558\.360\.054\.258\.09\.0DeepSeek\-V3\.258\.352\.554\.255\.011\.0Qwen3\-32B25\.032\.529\.229\.01\.0Annotator A94\.462\.583\.379\.04\.0Annotator B94\.450\.079\.273\.07\.0Human mean94\.456\.381\.376\.05\.5Under\-personalization\.This family shows the largest human\-model gap\. On the shared 100\-scenario subset, both human annotators recognize 94\.4% of under\-personalization errors, compared with 63\.9% for the strongest reported evaluator \(Table[4](https://arxiv.org/html/2610.19314#S5.T4)\)\. Humans significantly outperform each evaluator in Table[4](https://arxiv.org/html/2610.19314#S5.T4)on these cases \(Appendix[D](https://arxiv.org/html/2610.19314#A4)\)\.

The dominant failure within this family is direction confusion\. Evaluators assign 26\.6% of these responses the oppositeOververdict, far more often than over\-personalization errors are judgedUnder\(Figure[3](https://arxiv.org/html/2610.19314#S5.F3)a\)\. Secondary errors amplify but do not explain this pattern: on subsets that exclude human\-flagged secondaryOvererrors, the rate remains 19\.5–21\.1%, and the wrong\-direction rationales ground their decisions in genuine history evidence but assign that evidence the wrong applicability, scope, or weight \(Appendix[C\.3](https://arxiv.org/html/2610.19314#A3.SS3)\)\. The failure lies in determining its proper role in the response\.

Underweighting\.Underweighting illustrates why misapplied\-state errors admit both error directions\. Evaluators split their error verdicts on these responses betweenUnderandOver\(Figure[3](https://arxiv.org/html/2610.19314#S5.F3)b\)\. The two verdicts capture complementary aspects of the same error: insufficient influence from the applicable state and excessive relative influence from a competing consideration\.

Evaluator operating points\.The evaluators occupy distinct operating regimes rather than forming a single performance ranking\. Kimi K2\.5 is comparatively permissive, pairing 66\.5% ERR with the highest reported FAR of 24\.6%, whereas Qwen3\-32B is strongly conservative, reducing FAR to 2\.5% but recognizing only 32\.0% of errors\. Evaluator choice therefore changes whether personalization failures are more likely to be over\-reported or missed\.

Figure 3:Verdict behavior in the main setting\. \(a\) Wrong\-direction verdicts, pooled over evaluators: the share of under\-personalization errors judgedOverand of over\-personalization errors judgedUnder\. \(b\) Verdict split by reported evaluator for underweighting errors\.Table 5:ERR \(%\) by context availability, scenario relation, and applicability type\. Categorical \(holder, scope, temporal\) and Graded \(strength, transferability\) are unweighted family means\. The final row reports mean FAR over all five evaluators for the reduced and augmented context conditions; Full\-context FAR for the reported evaluators appears in Table[3](https://arxiv.org/html/2610.19314#S3.T3)\. Bold marks the best reported evaluator per column\.Context availabilityScenario relationApplicability typeModelTargetOnlyRecentFullFull \+Highl\.Gold Ev\.OnlyRe\-entryn=174n\{=\}174Cross\-domainn=166n\{=\}166Out\-of\-scopen=63n\{=\}63CategoricalboundaryGradeddecisionKimi K2\.563\.570\.766\.570\.569\.062\.664\.582\.572\.958\.8GLM\-546\.766\.364\.365\.862\.862\.661\.476\.269\.658\.7DeepSeek\-V3\.250\.458\.362\.565\.063\.060\.364\.563\.567\.656\.9Qwen3\-32B28\.034\.032\.035\.233\.031\.631\.933\.333\.331\.2Mean FAR22\.130\.6–12\.524\.0–––––

### 5\.2Evidence Access Does Not Resolve Applicability

To separate context breadth from evidence access, we hold each scenario and candidate response fixed and vary only the information shown to the evaluator \(Table[5](https://arxiv.org/html/2610.19314#S5.T5)\)\.

History truncation\.Recent context preserves error detection but weakens the distinction between erroneous and appropriate personalization\. Recent yields nearly the same ERR as Full, yet mean FAR reaches 30\.6%\. With Target Only, ERR falls while FAR remains high at 22\.1%\. A matched ablation confirms that this effect comes from relevant history rather than context length: removing applicable\-state turns raises FAR by 55\.2 points and lowers pair accuracy by 44\.4 points, whereas removing the same number of unrelated turns changes them by only 2\.0 and 3\.2 points, respectively \(Appendix[C\.4](https://arxiv.org/html/2610.19314#A3.SS4)\)\.

Gold evidence access\.Making the relevant evidence explicit provides only limited help\. Full \+ Gold Highlights improves ERR only slightly\. Gold Evidence Only retains those turns but removes the surrounding dialogue, leaving ERR nearly unchanged while FAR rises to 24\.0%\.

Response\-local observability\.Some errors remain recognizable without history because their effects are visible in the candidate response\. Under Target Only, recognition of under\-personalization falls to 31\.7%, while unsupported inference and underweighting remain at 74\.9% and 70\.9%\. The latter responses expose an incorrect inference or weighting decision, whereas omission errors require the evaluator to recover what the history should have contributed\. Response\-local cues thus aid error detection without resolving whether persona information is used appropriately\.

Graded applicability\.The applicability\-type breakdown identifies a second source of difficulty\. Evaluators reach lower ERR on graded decisions involving strength or transferability than on categorical boundaries involving holder, scope, or temporal status\. Beyond finding relevant information, evaluators struggle to determine how strongly and how broadly it should apply\.

### 5\.3Structured Prompting Changes Verdict Tendencies, Not Judgment Quality

To test whether structured reasoning improves applicability judgment or merely changes how readily evaluators issue error verdicts, we compare three formulations under the sameFullcontext\. Evidence\-first prompting raises ERR, but also raises FAR\. The state\-table formulation lowers ERR and generally lowers FAR, shifting evaluators toward more conservative predictions\. Neither formulation improves error recognition while preserving correct acceptance ofRightresponses \(Table[10](https://arxiv.org/html/2610.19314#A3.T10)\)\.

### 5\.4Independent Judgment Remains Difficult Without an Explicit Contrast

Alternative protocols clarify where the difficulty lies\. Pointwise scoring separatesRightfrom error responses in only 63\.0% of matched pairs, with 35\.9% ties\. Direct pairwise comparison reaches 89\.0% accuracy with 90\.5% order consistency \(Appendix[C\.5](https://arxiv.org/html/2610.19314#A3.SS5)\), but this setting exposes the two responses side by side and therefore provides an explicit contrast that is absent from the benchmark’s target setting\. The high pairwise accuracy thus does not imply that evaluators can reliably judge a response on its own\. Moreover, neither protocol identifies the error direction, so neither addresses theUnder→\\toOverconfusion observed in independent judgments\. These results reinforce that the difficult part is independently determining whether and how a response misuses user history\.

## 6Conclusion

PersEvalBenchtests whether evaluators can judge how user history should influence a response\. Across 403 scenarios, five LLM evaluators remain unreliable even with full history\. The clearest failure is under\-personalization: evaluators often detect a problem but reverse its direction, while humans perform substantially better\. Easier evidence access and structured prompting yield limited gains, whereas pairwise comparison is much easier than independent judgment\. These results suggest that reliable personalization evaluation requires not only retrieving relevant history, but correctly interpreting its role in the current response\.

## Limitations

Construction\.The benchmark uses LLM\-generated dialogues and responses, which may differ from organic interactions or contain residual generation artifacts despite structured verification and human auditing\. Restricted\-context generation may introduce detectable differences fromRightresponses, and upstream construction still relies on Claude, so broader construction\-family effects cannot be excluded\. Appendix[C\.6](https://arxiv.org/html/2610.19314#A3.SS6)provides an additional response\-generation robustness check using Mistral Large 3\([Mistral AI, 2025](https://arxiv.org/html/2610.19314#bib.bib34)\)\.

Evaluation scope\.Each benchmark response targets one controlled error, although generation can introduce secondary cues and real responses may contain multiple interacting errors\. We audit secondaryOvercues in under\-personalization cases and report robustness analyses after excluding such items \(Appendix[C\.3](https://arxiv.org/html/2610.19314#A3.SS3)\)\. Compound errors, however, remain outside the benchmark’s intended scope\. We also report ERR and FAR at each reported model’s observed operating point and leave confidence calibration to future work\.

Generalizability\.The benchmark is English\-only and grounded in SGD\-derived task seeds, so results may not generalize fully across domains, interaction styles, languages, or cultural norms of personalization\.

## Ethical Considerations

All dialogues, personas, and responses inPersEvalBenchare synthetic and generated from structured specifications grounded in task seeds derived from the public Schema\-Guided Dialogue dataset\. No real user data or personally identifying information is used\. Human annotators are graduate\-student volunteers who participated with informed consent\. The benchmark is intended for research on the reliability of personalization evaluators\. BecausePersEvalBenchis synthetic, English\-only, and task\-oriented, its results should not be used to certify evaluator reliability in deployment\. Overgeneralizing the benchmark could obscure failures involving real users, other languages, or culturally dependent personalization norms\. The Schema\-Guided Dialogue dataset is released under the CC BY\-SA 4\.0 license, and we cite its original creators\.

## References

- Anthropic \(2026a\)AnthropicClaude Opus 4\.6 system card\.Note:Anthropic system cardExternal Links:[Link](https://www.anthropic.com/claude-opus-4-6-system-card)Cited by:[§4](https://arxiv.org/html/2610.19314#S4.SS0.SSS0.Px2.p1.1)\.
- Anthropic \(2026b\)AnthropicClaude Sonnet 4\.6 system card\.Note:Anthropic system cardExternal Links:[Link](https://www.anthropic.com/claude-sonnet-4-6-system-card)Cited by:[§4](https://arxiv.org/html/2610.19314#S4.SS0.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-V3\.2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.External Links:[Link](https://arxiv.org/abs/2512.02556)Cited by:[§4](https://arxiv.org/html/2610.19314#S4.SS0.SSS0.Px2.p1.1)\.
- Donget al\.\(2024\)Y\. R\. Dong, T\. Hu, and N\. CollierCan LLM be a personalized judge?\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 10126–10141\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.592),[Link](https://aclanthology.org/2024.findings-emnlp.592/)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.14.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Fenget al\.\(2026\)X\. Feng, W\. Gan, X\. Chen, Q\. Dai, and Y\. LiuHow does personalized memory shape LLM behavior? benchmarking rational preference utilization in personalized assistants\.arXiv preprint arXiv:2601\.16621\.External Links:[Link](https://arxiv.org/abs/2601.16621)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.10.1),[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px3.p1.1)\.
- GLM\-5 Team \(2026\)GLM\-5 TeamGLM\-5: from vibe coding to agentic engineering\.arXiv preprint arXiv:2602\.15763\.External Links:[Link](https://arxiv.org/abs/2602.15763)Cited by:[§4](https://arxiv.org/html/2610.19314#S4.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2026\)Q\. Guo, Y\. Li, Y\. Liu, and B\. HooiTowards natural personalization: evaluating long\-horizon preference following in personalized user\-LLM interactions\.arXiv preprint arXiv:2603\.04191\.External Links:[Link](https://arxiv.org/abs/2603.04191)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)Y\. Hu, Z\. Long, J\. Guo, X\. Sui, X\. Fu, W\. Zhao, Y\. Zhao, and B\. QinOP\-Bench: benchmarking over\-personalization for memory\-augmented personalized conversational agents\.arXiv preprint arXiv:2601\.13722\.External Links:[Link](https://arxiv.org/abs/2601.13722)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.9.1),[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px3.p1.1)\.
- Jianget al\.\(2025\)B\. Jiang, Z\. Hao, Y\. Cho, B\. Li, Y\. Yuan, S\. Chen, L\. Ungar, C\. J\. Taylor, and D\. RothKnow me, respond to me: benchmarking LLMs for dynamic user profiling and personalized responses at scale\.InProceedings of the Second Conference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2504.14225)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.7.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px3.p1.1)\.
- Kimet al\.\(2024a\)S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. SeoPrometheus: inducing fine\-grained evaluation capability in language models\.InProceedings of the Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=8euJaTveKw)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Kimet al\.\(2024b\)S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. SeoPrometheus 2: an open source language model specialized in evaluating other language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Miami, Florida, USA,pp\. 4334–4353\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.248),[Link](https://aclanthology.org/2024.emnlp-main.248/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Kimi Team \(2026\)Kimi TeamKimi K2\.5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.External Links:[Link](https://arxiv.org/abs/2602.02276)Cited by:[§4](https://arxiv.org/html/2610.19314#S4.SS0.SSS0.Px2.p1.1)\.
- Lambertet al\.\(2025\)N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. HajishirziRewardBench: evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,Albuquerque, New Mexico,pp\. 1755–1797\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96),[Link](https://aclanthology.org/2025.findings-naacl.96/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Liet al\.\(2026\)S\. S\. Li, B\. Paranjape, K\. Oktar, Z\. Ma, G\. Zhou, L\. Guan, N\. Zhang, S\. Park, L\. Chen, D\. Yang, Y\. Tsvetkov, and A\. CelikyilmazHorizonBench: long\-horizon personalization with evolving preferences\.arXiv preprint arXiv:2604\.17283\.External Links:[Link](https://arxiv.org/abs/2604.17283)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.8.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px3.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating very long\-term conversational memory of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 13851–13870\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.747),[Link](https://aclanthology.org/2024.acl-long.747/)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.4.1),[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Mireshghallahet al\.\(2025\)N\. Mireshghallah, N\. Mangaokar, N\. Kokhlikyan, A\. Zharmagambetov, M\. Zaheer, S\. Mahloujifar, and K\. ChaudhuriCIMemories: a compositional benchmark for contextual integrity of persistent memory in LLMs\.arXiv preprint arXiv:2511\.14937\.External Links:[Link](https://arxiv.org/abs/2511.14937)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.12.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Mistral AI \(2025\)Mistral AIIntroducing Mistral 3\.Note:Mistral AIExternal Links:[Link](https://mistral.ai/news/mistral-3/)Cited by:[Limitations](https://arxiv.org/html/2610.19314#Sx1.p1.1)\.
- Nissenbaum \(2004\)H\. NissenbaumPrivacy as contextual integrity\.Washington Law Review79\(1\),pp\. 119–158\.External Links:[Link](https://digitalcommons.law.uw.edu/wlr/vol79/iss1/10/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Panicksseryet al\.\(2024\)A\. Panickssery, S\. R\. Bowman, and S\. FengLLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 68772–68802\.External Links:[Document](https://dx.doi.org/10.52202/079017-2197),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Rastogiet al\.\(2020\)A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. KhaitanTowards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8689–8696\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6394),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/6394)Cited by:[§1](https://arxiv.org/html/2610.19314#S1.p3.1),[§3\.3](https://arxiv.org/html/2610.19314#S3.SS3.SSS0.Px1.p1.1)\.
- Salemiet al\.\(2024\)A\. Salemi, S\. Mysore, M\. Bendersky, and H\. ZamaniLaMP: when large language models meet personalization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 7370–7392\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.399),[Link](https://aclanthology.org/2024.acl-long.399/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Tanet al\.\(2025\)S\. Tan, S\. Zhuang, K\. Montgomery, W\. Y\. Tang, A\. Cuadron, C\. Wang, R\. A\. Popa, and I\. StoicaJudgeBench: a benchmark for evaluating LLM\-based judges\.InProceedings of the Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=G0dksFayVq)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024a\)P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. SuiLarge language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511),[Link](https://aclanthology.org/2024.acl-long.511/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2024b\)Y\. Wang, Z\. Yu, W\. Yao, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. ZhangPandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization\.InProceedings of the Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/be3b0d51a2b86cb4ffe50f13480217e0-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Xiaoet al\.\(2026\)J\. Xiao, X\. Yu, C\. Wang, W\. Zheng, X\. Lin, K\. Liu, H\. Ding, Y\. Zhang, W\. Wang, F\. Feng, and X\. HeAlpsBench: an LLM personalization benchmark for real\-dialogue memorization and preference alignment\.InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 3489–3497\.External Links:[Document](https://dx.doi.org/10.1145/3805712.3808634),[Link](https://doi.org/10.1145/3805712.3808634)Cited by:[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2610.19314#S4.SS0.SSS0.Px2.p1.1)\.
- Yoonet al\.\(2026\)S\. Yoon, S\. Kim, H\. Hong, W\. Jeung, Y\. Kim, W\. Seo, H\. Yeen, and A\. NoBenchPreS: a benchmark for context\-aware personalized preference selectivity of persistent\-memory LLMs\.arXiv preprint arXiv:2603\.16557\.External Links:[Link](https://arxiv.org/abs/2603.16557)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.11.1),[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px3.p1.1)\.
- Zenget al\.\(2024\)Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. ChenEvaluating large language models at evaluating instruction following\.InProceedings of the Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tr0KidwPLc)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2018\)S\. Zhang, E\. Dinan, J\. Urbanek, A\. Szlam, D\. Kiela, and J\. WestonPersonalizing dialogue agents: I have a dog, do you have pets too?\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Melbourne, Australia,pp\. 2204–2213\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-1205),[Link](https://aclanthology.org/P18-1205/)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2025a\)S\. Zhao, M\. Hong, Y\. Liu, D\. Hazarika, and K\. LinDo LLMs recognize your preferences? evaluating personalized preference following in LLMs\.InProceedings of the Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QWunLKbBGF)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.5.1),[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px3.p1.1)\.
- Zhaoet al\.\(2025b\)Z\. Zhao, C\. Vania, S\. Kayal, N\. Khan, S\. B\. Cohen, and E\. YilmazPersonaLens: a benchmark for personalization evaluation in conversational AI assistants\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 18023–18055\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.927),[Link](https://aclanthology.org/2025.findings-acl.927/)Cited by:[Table 6](https://arxiv.org/html/2610.19314#A1.T6.2.1.6.1),[§1](https://arxiv.org/html/2610.19314#S1.p1.1),[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.External Links:[Document](https://dx.doi.org/10.52202/075280-2020),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025\)L\. Zhu, X\. Wang, and X\. WangJudgeLM: fine\-tuned large language models are scalable judges\.InProceedings of the Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xsELpEPn4A)Cited by:[§2](https://arxiv.org/html/2610.19314#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ABenchmark Comparison

In Table[6](https://arxiv.org/html/2610.19314#A1.T6), ✓ denotes explicit coverage,△\\trianglepartial or narrowly scoped coverage, and – a capability that is not a primary evaluation target\.*Applicability boundaries*concern whether persona information applies to the current holder, task, time, or related episode; misapplied\-state cases exhibit under\- and over\-use at once \(Section[3\.2](https://arxiv.org/html/2610.19314#S3.SS2)\)\.

Table 6:Comparison with the closest personalization, memory, and personalized\-evaluation benchmarks\.BenchmarkEvaluatesLongitudinalhistoryApplicabilityboundariesPersona\-use error coverageMatchedresponsesUnderOverMisappliedstateAssistant\-side benchmarksLoCoMo\([Maharana et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib10)\)Assistant / memory✓–––––PrefEval\([Zhao et al\., 2025a](https://arxiv.org/html/2610.19314#bib.bib2)\)Assistant✓–✓–––PersonaLens\([Zhao et al\., 2025b](https://arxiv.org/html/2610.19314#bib.bib15)\)Assistant✓△\\triangle△\\triangle–––PersonaMem\([Jiang et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib6)\)Assistant✓✓ \(time\)△\\triangle–△\\triangle\(stale\)–HorizonBench\([Li et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib7)\)Assistant✓✓ \(time\)△\\triangle–△\\triangle\(stale\)–OP\-Bench\([Hu et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib3)\)Assistant✓△\\triangle–✓––RPEval\([Feng et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib5)\)Assistant△\\triangle✓△\\triangle✓––BenchPreS\([Yoon et al\., 2026](https://arxiv.org/html/2610.19314#bib.bib4)\)Assistant–✓△\\triangle✓––CIMemories\([Mireshghallah et al\., 2025](https://arxiv.org/html/2610.19314#bib.bib8)\)Assistant / memory–✓–△\\triangle––Evaluator\-side benchmarksPersonalized Judge\([Dong et al\., 2024](https://arxiv.org/html/2610.19314#bib.bib9)\)Evaluator––––––PersEvalBenchEvaluator✓✓✓ \(3\)✓ \(3\)✓ \(3\)✓

Notation\.✓ denotes explicit primary coverage,△\\trianglepartial or narrowly scoped coverage, and–no primary coverage\. Parenthetical labels specify a narrowly covered boundary or error type; numbers indicate the controlled subtypes inPersEvalBench\.

## Appendix BBenchmark Design and Composition

### B\.1Scenario Grammar and Coverage

Table 7:Scenario grammar: each history–target relation, its central judgment, and the error subtypes that arise naturally within it \(19 of 27 relation–subtype combinations are legal\)\.Legal error subtypesRelationCentral judgmentUnder\-personalizationOver\-personalizationMisapplied\-stateRe\-entryWhich states remain current?CO, PO, EVIU, SLSS, OD, UWCross\-domainWhat transfers, and with what weight?CO, PO, EVIU, UISS, OD, UWOut\-of\-scopeWhich history should be ignored?–IU, SL, UI–

Table[7](https://arxiv.org/html/2610.19314#A2.T7)summarizes the scenario grammar: for each history–target relation of Section[3\.2](https://arxiv.org/html/2610.19314#S3.SS2.SSS0.Px1), the judgment it centers on and the error subtypes that constitute natural violations, grouped by family \(subtype abbreviations follow Section[3\.2](https://arxiv.org/html/2610.19314#S3.SS2)\)\. Nineteen of the 27 relation–subtype combinations are legal, as fixed by a human\-authored compatibility matrix; for example, scope leakage is excluded from cross\-domain scenarios, where crossing the domain boundary is the intended behavior rather than a leak\. Per\-category composition counts appear in Table[9](https://arxiv.org/html/2610.19314#A2.T9)\. Subtype precedence keeps labels unambiguous: explicitly selecting an option that violates an active hard constraint is*explicit violation*\(complete omission requires no direct contradiction\), and scope leakage requires an identifiable scope boundary \(irrelevant use covers merely unrelated states\)\.

### B\.2State\-Relation Inventory

Each scenario distinguishes the applicable stateAAfrom a competing stateBBthrough one primary relation, drawn from eight state relations grouped into five applicability families\. Table[8](https://arxiv.org/html/2610.19314#A2.T8)lists the inventory and grouping used in all relation\-level analyses\.

Table 8:The eight state relations and their five applicability families, grouped by the categorical vs\. graded distinction of Section[5\.2](https://arxiv.org/html/2610.19314#S5.SS2)\.FamilyRelationGlossCategorical boundariesHoldersame attr\., diff\. objectheld for a different person or objectScopesame attr\., diff\. scopeapplies in a different task or occasionTemporalsame attr\., diff\. timeheld at a different timeold vs\. new valueearlier value later updatedGraded decisionsStrengthhard vs\. softbinding constraint vs\. soft preferencetwo simultaneoustwo co\-active states compete for priorityTransfer\.cross\-domain vs\. specificgeneral vs\. task\-specific stateunsupported adjacentadjacent\-domain preference without direct support

### B\.3Benchmark Statistics

Table[9](https://arxiv.org/html/2610.19314#A2.T9)summarizes benchmark scale \(Panel A\), composition \(Panel B\), and the realized evidence form and distance \(Panel C\)\.

Table 9:Benchmark statistics: scale \(Panel A\), composition \(Panel B\), and evidence realization \(Panel C; distance bins of Section[3\.2](https://arxiv.org/html/2610.19314#S3.SS2)\)\.Panel A: ScaleScenario items403Candidate responses806Task types / task seeds48 / 480Avg\. episodes / turns per dialogue2\.4 / 26\.8Panel B: CompositionScenario relationRe\-entry174 \(43\.2%\)Cross\-domain166 \(41\.2%\)Out\-of\-scope63 \(15\.6%\)Response verdict \(of2×4032\\times 403responses\)Right403 \(50\.0%\)Under\-personalization130 \(16\.1%\)Over\-personalization151 \(18\.7%\)Misapplied\-state122 \(15\.1%\)
Panel C: Evidence realizationAttributeCategorynn\(%\)FormExplicit159 \(39\.5%\)Implicit114 \(28\.3%\)Accumulated130 \(32\.3%\)DistanceNear65 \(16\.1%\)Medium77 \(19\.1%\)Far261 \(64\.8%\)

## Appendix CAdditional Analyses

### C\.1Evaluation Formulation

All formulations are zero\-shot with no hyperparameter search or model selection\. Table[10](https://arxiv.org/html/2610.19314#A3.T10)reports results for the reported evaluators under the three elicitation formulations of Section[5\.3](https://arxiv.org/html/2610.19314#S5.SS3)\.

Table 10:ERR / FAR \(%\) by evaluation formulation under Full context \(E2E/EvF/ST = End\-to\-end/Evidence\-first/State\-table; bold = highest ERR and lowest FAR among reported evaluators per column\)\.Evaluation formulation \(ERR / FAR\)ModelE2EEvFSTKimi K2\.566\.5/ 24\.671\.0/ 29\.060\.3 / 16\.1GLM\-564\.3 / 12\.265\.8 / 16\.162\.5/ 10\.9DeepSeek\-V3\.262\.5 / 10\.965\.0 / 16\.454\.8 / 11\.4Qwen3\-32B32\.0 /2\.542\.4 /6\.018\.1 /1\.5

### C\.2Underweighting Verdict Split

Evaluator rationales show the two readings behind the direction split on underweighting errors \(Figure[3](https://arxiv.org/html/2610.19314#S5.F3)b\)\. Under the*deficit*reading, the evaluator anchors on the applicable state and observes that the response acknowledges the constraint but then downplays it, yieldingUnder\. Under the*excess*reading, the evaluator anchors on the response content and objects to what filled the vacated space: the weakened restatement of the preference no longer matches what the user actually said \(read as an unsupported, fabricated preference\), or a secondary consideration now dominates the decision \(read as over\-application of the wrong state\), yieldingOver\.

### C\.3Under→\\toOverRobustness

We test whether theUnder→\\toOverrate reflects responses that contain a secondary over\-personalization error in addition to the intended one\. The two Study 3 annotators \(Appendix[D](https://arxiv.org/html/2610.19314#A4)\) independently audited all 130 under\-personalization responses, judging for each whether the intendedUndererror is present and whether a secondaryOvererror is also present\. Excluding every item that either annotator flagged with a secondaryOvererror lowers the rate from 26\.6% to 21\.1%; further requiring both annotators to confirm the intendedUndererror leaves 19\.5%\. The residual confusion is lowest for complete omission \(14\.3%\) and highest for explicit violation \(26\.4%\)\. A manual audit of all 173 main\-setting judgments in which an under\-personalization error was judgedOverfurther shows that every rationale cites genuine history evidence and errs in judging its applicability, scope, or weight; none argues from response wording alone, and none invents a user state\. Secondary errors therefore amplify, but do not explain, the direction confusion reported in Section[5\.1](https://arxiv.org/html/2610.19314#S5.SS1)\.

### C\.4Relevant\-History Ablation

To test whether the benefit of Full context comes from relevant history rather than context length, we construct two matched ablations of the Full history on a separate random sample of 50 scenarios, stratified by error subtype and then by scenario relation\.Full−\-relevant historydeletes the turns that express the applicable state, together with the assistant reply that follows each\.Full−\-unrelated contextdeletes the same number of non\-evidence turns\. Turns are renumbered consecutively, and the target request and both candidate responses are unchanged\. The Full condition is re\-run through the same pipeline\. Table[11](https://arxiv.org/html/2610.19314#A3.T11)shows that removing relevant history leaves ERR largely intact but turns mostRightresponses into false alarms, whereas removing unrelated context barely changes either metric\.

Table 11:Relevant\-history ablation: five\-evaluator mean ERR, FAR, and pair accuracy \(%\) on 50 scenarios\.ConditionERRFAR↓\\downarrowPair acc\.Target Only46\.020\.838\.8Full−\-relevant history60\.867\.619\.6Full−\-unrelated context62\.814\.460\.8Full64\.412\.464\.0#### Gold Evidence Only false alarms\.

A manual audit of 50 false alarms produced under Gold Evidence Only traces 78% to context lost when the surrounding dialogue is removed: the evaluator reads correct use of the applicable state as importing information from another task or thread, or treats response details supported by the removed turns as unsupported\.

### C\.5Alternative Judging Protocols

Section[5\.4](https://arxiv.org/html/2610.19314#S5.SS4)compares the three\-way verdict with two common judging protocols, both run on all 403 matched pairs under Full context with the same five evaluators\.Pointwise scoringrates each response independently from 1 to 5 for how appropriately it uses the user’s history; the rubric does not mention under\- or over\-personalization and excludes overall response quality\. A pair counts as correct only when theRightresponse receives a strictly higher score than its matched error response; tied scores are reported separately\.Pairwise comparisonshows the two matched responses together and asks which uses the history more appropriately\. Each pair is presented in both orders; accuracy is averaged over the two orders, and consistency is the percentage of pairs for which the same response is chosen in both orders\.

Pointwise scoring separates the matched responses in 63\.0% of pairs, with 35\.9% of pairs receiving tied scores; ties rise to 50\.5% for misapplied\-state errors\. Pairwise comparison reaches 89\.0% accuracy with 90\.5% order consistency\. Neither protocol assigns an error direction, so neither can expose theUnder→\\toOverconfusion measured by the three\-way verdict\.

### C\.6Response\-Generator Robustness

Because construction uses Claude Sonnet 4\.6, results could depend on its response style or generation artifacts\. Holding each scenario’s specification and dialogue fixed, we regenerate only the matchedRightand error responses of all 403 scenarios with Mistral Large 3 and re\-run the evaluators under Full context with the End\-to\-end formulation\. Across the four evaluators outside the construction model family, pair accuracy changes by only−0\.5\-0\.5points on average, suggesting limited sensitivity to the response\-generator family\. Upstream scenario planning, specification, and dialogue synthesis still use Claude, so broader construction\-family effects cannot be excluded \(see Limitations\)\.

## Appendix DHuman Validation

#### Protocols\.

We conduct three human studies on the same stratified sample of 100 scenarios: intervention fidelity \(Study 1\), state\-role validation \(Study 2\), and a human evaluator reference \(Study 3\)\. Study 1 was conducted by one pair of graduate\-student annotators, and Studies 2 and 3 by a second, disjoint pair; none was involved in benchmark construction\. In Study 1, annotators viewed the dialogue, the error response, and the intended subtype definition, judged whether the response committed the assigned error, and consolidated differences through joint adjudication\. In Study 3, annotators performed the evaluator task itself before any exposure to the benchmark policy: they independently assignedRight,Under, orOverto 200 shuffled single\-response instances comprising the sample’s matchedRightand error responses, using only the dialogue and candidate response\. Their labels are reported without adjudication\. Misapplied\-state instances were labeled in a separate round, so annotators could infer the error family represented in that pool; they remained blind to which responses were erroneous and to the direction of each error, but performance on the misapplied\-state portion may benefit from the narrower judgment space\. Scoring follows Section[4](https://arxiv.org/html/2610.19314#S4)\.

#### Study 1: Intervention fidelity\.

Table[12](https://arxiv.org/html/2610.19314#A4.T12)reports per\-subtype consensus confirmation rates\. The consensus confirms the intended error in 88% of raw pipeline generations, with the lowest rates for explicit violation \(67%\) and scope leakage \(70%\), the two subtypes whose boundary definitions leave the most room for judgment\. The 12 intervention\-realization mismatches were revised and re\-verified as described in Section[3\.4](https://arxiv.org/html/2610.19314#S3.SS4); the final benchmark, Studies 2 and 3, and all model evaluations use the revised versions\.

Table 12:Intervention fidelity \(Study 1\): per\-subtype consensus confirmation rate \(%\) on raw pipeline generations and the number of scenarios revised in the final benchmark\.SubtypennConfirmed \(%\)RevisedComplete omission12832Partial omission121000Explicit violation12674Irrelevant use18892Scope leakage10703Unsupported inference121000Stale/superseded61000Over\-dominance12921Underweighting61000Overall1008812
#### Study 2: State\-role validation\.

On the same 100 scenarios, the two Study 3 annotators read the dialogue and the target request and assign each user state one of three roles:Ignore\(the state should not affect the response\),Support\(the state may guide the response\), orDominate\(the state must be satisfied\), without access to the construction specification or benchmark policy; disagreements are then resolved through adjudication\. After adjudication, 86\.3% of state\-role assignments match the benchmark policy; exact three\-level agreement between the independent labels is 70\.0% \(weightedκ=\.709\\kappa\{=\}\.709\)\. On instances in the sample where an evaluator’s verdict disagrees with the benchmark verdict, the annotators’ independent judgments both support the benchmark verdict in 72\.8% of cases and both reject it in only 7\.8%, indicating that evaluator errors are rarely explained by an unreasonable benchmark policy\.

#### Study 3: Human evaluator reference\.

Table[4](https://arxiv.org/html/2610.19314#S5.T4)\(main text\) compares the annotators with the evaluators on the same 100 scenarios under the main Full\-history setting, which matches what the annotators saw\. Inter\-annotator agreement over the 200 judgments is 83\.5% \(Cohen’sκ=0\.72\\kappa\{=\}0\.72\)\. Their family profile inverts the models’: both annotators achieve 94\.4% recognition on under\-personalization, with misapplied\-state performance falling in between, and each annotator significantly outperforms each evaluator in Table[4](https://arxiv.org/html/2610.19314#S5.T4)on under\-personalization \(paired McNemar exact tests, allp≤\.004p\\leq\.004,n=36n\{=\}36paired scenarios\)\. Their false\-alarm rates are below those of every reported evaluator except Qwen3\-32B, whose 1\.0% false\-alarm rate comes with only 29\.0% error recognition\.

#### Annotator instructions\.

For the intervention\-fidelity study \(Study 1\), annotators received the following instructions:

> Read the complete dialogue, the candidate error response, and the definition of the assigned error subtype\. Determine whether the candidate response clearly realizes the assigned persona\-use intervention\. Focus on how the response uses information from the dialogue history, rather than on general writing quality\. Mark the instance asValidif the assigned error is present and asInvalidotherwise\. For an invalid instance, briefly describe the mismatch\.

For the human\-reference study \(Study 3\), annotators received the following instructions:

> Read the complete dialogue history, the target user request, and one candidate assistant response\. Based only on the information shown, assign exactly one verdict: Right: The response uses applicable information from the user history with the appropriate influence and does not use information that should be ignored\. Under: The response omits, violates, or gives insufficient influence to applicable information from the user history\. Over: The response uses information that should not apply or gives user information more influence than warranted\. Judge the response’s use of persona information rather than its general fluency or writing style\.

## Appendix EWorked Examples

This appendix illustrates the benchmark with condensed examples\. Examples[E](https://arxiv.org/html/2610.19314#A5)–[E](https://arxiv.org/html/2610.19314#A5)show one full scenario per history–target relation, each realizing one error subtype in place \(stale/superseded use and underweighting from the misapplied\-state family, and scope leakage from the over\-personalization family\); Examples[E](https://arxiv.org/html/2610.19314#A5)and[E](https://arxiv.org/html/2610.19314#A5)illustrate the remaining six subtypes in compact form\. Dialogues are abridged:\[A\]/\[B\]mark turns expressing the applicable and the competing state, elisions are bracketed, and responses are excerpted\.

Example 1 Re\-entry×\\timesstale/superseded useRelation:the user leaves a school search, handles an unrelated task, and later returns to the school decisionState A \(applicable\):updated availability: work schedule now predictable; hybrid programs acceptableState B \(superseded\):the original online\-only constraint, stated emphatically before the updateDialogue \(abridged\)—the superseded value is stated more forcefully than its replacementU\[B\]: It has to be fully online, that’s non\-negotiable for me\. I can’t relocate, and my job has really unpredictable hours — I can’t commit to being physically somewhere at a set time every week\. Online only\.\[ … an intervening episode on an unrelated planning task …\]U\[A\]: So my store just moved me to a consistent evening shift, which means my hours are actually predictable now\. I’m free most mornings and early afternoons pretty reliably\.A\[A\]: So to make sure I’ve got this right: you’re no longer restricted to 100% online — you’re open to hybrid programs?U\[A\]: Exactly — hybrid is fine now, especially if it means a stronger program or better networking\. I want to come back to the college decision later\.U\[target\]: So given all of that — what programs would you actually recommend for me at this point? I want strong job placement, reasonable cost, and something that fits my schedule\.Matched responses \(excerpts\)—both are fluent and well\-structured; only the state they apply differsRight: “With your schedule stabilized and hybrid on the table, your options have genuinely expanded\. The local university is worth putting at the top of the list …”Error: “Since you’ve been clear that the program must be fully online \(given your unpredictable work schedule\), I’ve filtered everything through that requirement first …”Why it is hard—the superseded constraint was stated with maximal emphasis \(“non\-negotiable”\), while its replacement was agreed to conversationally; an evaluator anchoring on emphasis rather than recency accepts the error response\.

Example 2 Cross\-domain×\\timesunderweightingRelation:states established while designing a work\-productivity system must transfer to an urgent personal\-logistics problemState A \(applicable\):a hard 5:30 pm daycare pickup, plus a stated inability to work on parallel tasks \(accumulated over several turns\)State B \(competing\):a visual, color\-coded task\-board preference from the same episodeDialogue \(abridged\)—State A is assembled from scattered signals, never one sentenceU\[A\]: Even when I can see what needs attention, I end up jumping between projects and then I’m basically useless\. I’ve tried multitasking — it doesn’t work for me\.U\[A\]: The other constraint — and this is a hard wall for me — is 5:30\. I pick up my daughter, and if I’m even five minutes late the daycare charges a penalty\. It’s happened twice\.\[ … the system is designed; a new episode begins with a car breakdown …\]U\[target\]: It’s 4:15\. And oh god — yes, I have to pick up my daughter at 5:30\. I cannot be late\.Matched responses \(excerpts\)—Right: “Stop — you don’t have time to troubleshoot this yourself\. You have 75 minutes, and you know better than anyone that 5:30 is not negotiable\. Call roadside assistance immediately …”Error: “I know you mentioned the 5:30 hard stop for daycare pickup\. Here’s what I’d suggest: use this window to tackle two or three things in parallel rather than finishing one completely …”Why it is hard—the error acknowledges the constraint and then gives it no force — the signature underweighting shape\. Evaluators that flag it split between the deficit reading \(the hard stop was downplayed\) and the excess reading \(work\-context habits were misapplied to an emergency\); see Appendix[C\.2](https://arxiv.org/html/2610.19314#A3.SS2)\.

Example 3 Out\-of\-scope×\\timesscope leakageRelation:the target shares keywords with an earlier episode \(cancellation, holiday, the same town\) but reverses the user’s roleState A \(applicable\):consumer\-side dinner logistics: a guest’s dairy allergy, a hearing\-impaired father, what to ask the restaurantState B \(out of scope\):the user’s own cancellation policy as a vacation\-rental hostDialogue \(abridged\)—State B is genuine, recent, and emphatically held — but belongs to the host roleU\[B\]: My philosophy is: if you book a peak holiday slot, you’re locking it in\. No exceptions\. I’ve lost $3,000 in one holiday season to cancellations\.U\[B\]: Non\-refundable for holidays, firm 30 days for the rest\. That’s my policy going forward\.\[ … the user then plans a family dinner: dairy allergy and seating for a hearing\-impaired guest are discussed …\]U\[target\]: When I confirm the reservation, should I ask about their cancellation policy for Christmas Eve? I want to know what I’m getting into as a diner\.Matched responses \(excerpts\)—Right: “Yes, absolutely ask\. High\-end restaurants on Christmas Eve commonly require a card at booking and charge a per\-head cancellation fee, so ask directly what happens if a guest cancels …”Error: “Yes — but given your experience as a vacation rental host with a strict non\-refundable holiday policy of your own, you already know how high the stakes are on the restaurant’s side\. That context should shape how you approach this conversation …”Why it is hard—every keyword matches, and the imported state is real and vivid; the only thing wrong is the boundary — the user’s host\-side policy has no bearing on her question as a customer\.

Example 4 Under\-personalization subtypes in briefCoverage:the full under\-personalization family: complete omission, partial omission, explicit violationComplete omission—the applicable state leaves no trace in the responseHistory:while choosing a gym, the user repeatedly stressed that her daughter has a severe peanut allergy and must avoid any environment where nuts are present\.Target:“They apparently have a nutrition station — is that something my daughter could use?”Error:“The nutrition station is actually a solid option for kids, and it can work really well when you’re bringing your daughter along …”Why it is hard—fluent and helpful, written as if the allergy had never been mentioned; nothing in the response signals that anything is missing\.Partial omission—one of two jointly applicable requirements is silently droppedHistory:two hard requirements for a gift: total budget under $80, and Android compatibility \(the household owns no Apple devices\)\.Target:the best options for a non\-tech\-savvy father\.Error:recommends well\-reviewed, Android\-compatible devices priced at $129–$249\.Why it is hard—the visible personalization \(Android, ease of use\) makes the response look complete, hiding the dropped budget ceiling\.Explicit violation—the recommendation directly contradicts an active constraintHistory:the user is a nurse on an irregular rotating schedule and cannot commit to fixed\-time programs\.Target:“If my schedule ever stabilizes, should I just upgrade then?”Error:“Honestly, I wouldn’t wait — now is actually the perfect time to commit to a structured program …”Why it is hard—unlike underweighting, the constraint is never acknowledged: the response simply recommends the opposite of what the user can do\.

Example 5 Over\-personalization and misapplied\-state subtypes in briefCoverage:irrelevant use and unsupported inference \(over\-personalization\); over\-dominance \(misapplied\-state\); scope leakage appears in Example[E](https://arxiv.org/html/2610.19314#A5), stale/superseded use and underweighting in Examples[E](https://arxiv.org/html/2610.19314#A5)and[E](https://arxiv.org/html/2610.19314#A5)Irrelevant use—a genuine but unrelated state is imported as personalizationHistory:in a concert\-planning episode, the user said they only attend original artists and refuse tribute acts\. The current task is choosing a restaurant for their parents’ anniversary\.Error:“Stick with the original Osteria Barolo — honestly, this is a bit like your original\-artists\-only rule for concerts\.”Why it is hard—the imported rule is real and the analogy sounds thoughtful, but it plays no legitimate role in the restaurant decision\.Unsupported inference—a preference is asserted from an adjacent fact the user never endorsedHistory:planning a child’s birthday, the user explicitly says budget is not a concern \(“I just want the kids to have a blast”\)\. In an earlier episode they skipped an expensive Las Vegas trip\.Error:“Based on your earlier decision to skip the Vegas trip in favor of saving money, I’ve focused on keeping this low\-cost …”Why it is hard—the inference is derived from a real event yet contradicts what the user actually said at the target\.Over\-dominance—a soft preference is promoted to the primary decision filterHistory:the user’s binding constraint is private one\-on\-one sessions only \(severe social anxiety\); she once added that she prefers female practitioners “if possible\.” The target is an urgent dermatologist search\.Error:“Since you mentioned you feel more comfortable with female practitioners, I want to make sure that’s front and center in how we approach finding you the right dermatologist …”Why it is hard—a nice\-to\-have is elevated above both the medical urgency and the actual binding constraint; every fact used is true, only the weights are wrong\.

## Appendix FPrompt System

This appendix documents the complete prompt system as a set of cards\. Prompts[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)–[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)drive benchmark construction in pipeline order, while Prompts[F\.2](https://arxiv.org/html/2610.19314#A6.SS2)–[F\.2](https://arxiv.org/html/2610.19314#A6.SS2)elicit evaluator judgments\. Each card specifies the prompt’s purpose, input, and output contract\.

All benchmark\-construction calls use Claude Sonnet 4\.6 with temperature 0\.7 and provider\-default values for other sampling parameters\.

### F\.1Generation Pipeline

Prompt 1 Scenario planner — system promptPurpose:instantiate a structured scenario specification for one sampled design cell; fixes the label before any dialogue text existsInput:the instantiation template of Prompt[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)Output:a single JSON specification \(contract below\)Role framing—establishes the designer role and the diagnosticity requirementYou are an expert benchmark designer for PersEvalBench, a personalization\-evaluator benchmark\.Scenarios must be diagnostic: the correct label must be recoverable only by reasoning about which user state applies at the target and with what weight \-\- never from surface cues, response quality, or wording artifacts\.Sources of difficulty—the controlled factors that make the applicability judgment non\-trivialIncorporate these difficulty factors deliberately:1\. Implicit or accumulated preference expression \(the state is not spelled out in one place\)2\. States A and B competing on the same attribute dimension3\. Weighting errors in which the response acknowledges the constraint yet fails it4\. Evidence located episodes before the target5\. Causal chains that make B look plausibly transferable to the target6\. A neutral distractor option that prevents elimination by binary contrastFactorized state design—the specification core: two states and the single dimension separating themEvery scenario must specify:\- State A: the currently applicable preference or constraint \(with holder, scope, time, strength, transferability\)\- State B: a decoy state that is not applicable but looks similar\- The A\-\-B relation dimensions: holder, scope, time, status, strength, transferability, evidence form, evidence distance\- The single dimension that makes B inapplicableResponse design rules—fixes the generation mechanism of the trapFor withhold\-A subtypes \(complete\_omission, partial\_omission, underweighting, explicit\_violation\): the ERROR response does not properly use A and must not actively apply B \-\- the trap is realized by withholding or weakening A\.For apply\-B subtypes \(irrelevant\_use, scope\_leakage, stale\_superseded, over\_dominance, unsupported\_inference\): the ERROR response actively applies B to the current context \-\- the trap is realized by inserting inapplicable information\.Subtype constraints—operational admissibility tests for eight of the nine subtypes; over\-dominance is governed by the family rules above and the verifiercomplete\_omission: the target request must not mention or implicitly lock to State A’s content\. Test: a helpful assistant can give a complete, high\-quality answer without knowing State A; otherwise redesign the target request\.partial\_omission: State A contains at least two clearly separable, independently verifiable, equally critical requirements; the ERROR response clearly satisfies one and clearly fails the other\.underweighting: State A is an unambiguous hard constraint, stated with binding language \("dealbreaker", "must", "non\-negotiable"\) and a concrete satisfaction check; the ERROR response’s top recommendation clearly fails it\.explicit\_violation: RIGHT and ERROR differ only in whether A is respected\. Do not frame this as using old information \(that is stale\_superseded\)\. The target request does not restate A\.scope\_leakage: the ERROR must make the out\-of\-scope state materially affect the current recommendation, so that the crossed scope boundary is independently verifiable\. Express this naturally and do not use a fixed attribution template\.irrelevant\_use: State B comes from a genuinely different domain with no temporal, logistical, or scheduling relationship to the target task\. Test: if a reasonable person could argue B is relevant here, choose a different B\.stale\_superseded: the conversation contains an explicit update event in which the user changes the preference; the old and new values produce different recommendations; the ERROR uses the old value\.unsupported\_inference: the ERROR must explicitly connect an observed fact to a preference or constraint that the user never expressed\. The inference must be identifiable, but its wording should vary naturally\.why\_tricky: give a specific, non\-trivial reason the error is easy to miss; vague answers are rejected\.Output contract—the JSON fields that later stages consume and may not alterReturn a single JSON object with fields: scene\_type, primary\_error, preference\_relation, episode\_1\_state, episode\_2\_state\_or\_change, target\_episode, applicable\_state\_A, decoy\_or\_non\_applicable\_state\_B, A\_B\_relation, scope\_or\_status\_difference, A\_B\_distinguishing\_dimension, target\_options \{right, trap, neutral\_distractor\}, right\_choice\_logic, error\_choice\_logic, why\_tricky, single\_error\_check, preference\_expression, evidence\_distance\_category, causal\_chain\_logic\. No text outside the JSON\.

Prompt 2 Scenario planner — instantiation templatePurpose:binds one sampled design cell and its task grounding into the planner callInput:sampled \(scenario relation, error subtype, state relation\), targeted evidence form and distance, a Schema\-Guided Dialogue \(SGD\) anchor chainOutput:consumed under the system prompt of Prompt[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)Required combination—the design cell is fixed by sampling; the model cannot change itDesign a benchmark scenario with these constraints:\- Scene type: \{scenario\_relation\}\- Error pattern: \{error\_subtype\} \(withhold\-A: the ERROR response ignores or weakens A / apply\-B: the ERROR response actively uses B\)\- Preference relation: \{state\_relation\_description\}\- Preference expression: \{explicit / implicit / accumulated\}\- Evidence distance: \{near / medium / far\}Task grounding—realistic task structure from Schema\-Guided Dialogue anchors; grounding does not determine the labelSGD anchor chain \(adapt domains and tasks creatively\):Episode 1: domain "\{domain\_1\}"; context "\{subdomain\_1\}"; example request "\{surface\_request\_1\}"\-\> causal link to next episode: \[\{link\_type\}\] \{link\_description\}Episode 2: domain "\{domain\_2\}"; context "\{subdomain\_2\}"; example request "\{surface\_request\_2\}"Scene structure—relation\-specific episode plan, selected by the sampled scenario relation\{relation\-specific episode plan; e\.g\., for same\-episode re\-entry:E1: the user works on a task in domain X; State A is established\.E2: the user switches to a related task in domain Y; State B appears\.E3: the user returns to E1’s unfinished task; the target turn is here\.\}Requirements—binds evidence realization, decoy plausibility, and scenario coherence1\. State A must be expressed in the "\{preference\_expression\}" style \(explicit: stated directly; implicit: inferred from a single choice, rejection, or comparison; accumulated: requires connecting two or more scattered signals\)\.2\. Evidence distance must be "\{evidence\_distance\}" \(near: A expressed 0\-\-1 episodes before the target; medium: 2\-\-4 episodes before; far: 5\+ exchanges before, early in the conversation\)\.3\. The causal chain must make B look plausibly applicable to the target\.4\. Include a neutral distractor option that does not clearly favor either A or B\.5\. The total conversation should span 20\-\-30 turns\.Return a single JSON object with all required fields\. No text outside the JSON\.

Prompt 3 Specification verifierPurpose:reject or repair specifications whose label logic is unsound, before any dialogue is generatedInput:one scenario specificationOutput:\{passes, failures\[ \]\}Role and stakes—sets the acceptance barYou are a strict quality verifier for PersEvalBench benchmark scenarios\. Check whether a specification passes all of the following rules\. Be rigorous \-\- a pass means the scenario will become part of the final benchmark\.Label\-soundness rules \(1–3\)—apply to every scenarioRule 1 \(RIGHT satisfies State A and the main task\): the RIGHT option must correctly apply the currently applicable State A and fulfill the user’s main task requirement\.Rule 2 \(trap is not obviously worse\): the TRAP option must match RIGHT in quality, price level, completeness, and length \-\- a plausible good response that simply applies the wrong state\.Rule 3 \(single error\): the scenario must have exactly one primary error mechanism; if the trap could be explained by multiple distinct error types, it fails\.Family rules \(4–7\)—checks specific to error familiesRule 4 \(omission errors depend on A\): for complete\_omission, partial\_omission, underweighting, and explicit\_violation, the RIGHT option must genuinely depend on State A; if a generic response would also satisfy the task, the scenario fails\.Rule 5 \(over\-use errors depend on B\): for irrelevant\_use and over\_dominance, the trap must be favored only if the wrong state is applied, not because it is generically bad\.Rule 6 \(stale errors change the choice\): for stale\_superseded, the old and new preference values must produce genuinely different choices\.Rule 7 \(scope leakage is scope\-only\): for scope\_leakage, the only differentiator between RIGHT and TRAP must be scope applicability, not factual errors or quality issues\.Structural rules \(8–11\)—relation typing and subtype separabilityRule 8 \(A\-\-B relation structure\): the A\-\-B relation must match the declared state\-relation type \(same attribute for a different holder; same attribute in a different scope; same attribute at a different time; an old value superseded by a new one; a hard constraint versus a soft preference; two simultaneously applicable states; a cross\-domain\-transferable state versus a domain\-specific one; an adjacent fact that is not an actual preference\)\.Rule 9 \(why\_tricky is substantive\): the explanation must give a specific, non\-trivial reason a model would choose the trap\.Rule 10 \(complete\-omission request independence\): for complete\_omission only, the target request must not mention or implicitly lock to State A’s content; an assistant that has never seen State A must be able to give a complete, high\-quality answer\.Rule 11 \(partial\-omission separability\): for partial\_omission only, State A must contain at least two clearly separable requirements, and the error logic must name which requirement is satisfied and which is omitted; the requirements must be independently verifiable\.Output contract and calibration—machine\-readable verdict with per\-rule feedbackReturn a JSON object:\{"passes": true/false, "failures": \[\{"rule\_number": <int\>, "rule\_name": "\.\.\.", "explanation": "why it fails this rule", "suggestion": "how to fix it"\}\]\}If passes is true, failures must be an empty array\. Be strict but fair: fail only rules that are genuinely violated\.

Prompt 4 Specification refinementPurpose:single repair attempt for a failed specification; specifications failing again are discardedInput:the original specification and the verifier’s failure listOutput:a complete, revised specification JSONInstruction—repairs must preserve the sampled design cellYou are refining a PersEvalBench scenario specification that failed verification\. You will receive the original scenario and a list of rule failures with suggestions\.Fix the scenario to address all failures while preserving the core intent \(same scene type, error pattern, and preference relation\)\.Return a complete JSON object with all scenario fields, parseable directly\.

Prompt 5 Dialogue realization — system promptPurpose:expand an accepted specification into a natural multi\-episode conversation with machine\-readable state annotationsInput:the instantiation template of Prompt[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)Output:annotated conversation plus a JSON metadata blockRole and principles—naturalness constraints that must not distort the specified state logicYou are an expert conversation writer for PersEvalBench, a personalization\-evaluator benchmark\. You generate realistic multi\-episode conversations between a user and an AI assistant\.1\. Conversations must feel natural \-\- like real interactions with a personal assistant\.2\. Preferences and states must be expressed naturally within the conversation flow\.3\. The target turn \(where the test happens\) must be a natural user request\.4\. Each episode should have a clear task or topic that motivates the conversation\.Evidence\-form realization—how each targeted evidence form is expressed in dialogueState expressions must match the specified evidence form:\- explicit: the user states the preference directly \("I prefer X", "I don’t like Y"\)\- implicit: the preference is revealed through choices or reactions \("Let’s go with the vegetarian option again"\)\- accumulated: the preference emerges across multiple turns \(mentioned in passing; choices show a pattern\)Output format—episode separators, role prefixes, and the state annotations that later become benchmark metadata\- Between episodes, output a separator line: \-\-\- EPISODE \{N\} \-\-\-\- Each turn on its own line with a role prefix: "USER: \.\.\." or "ASSISTANT: \.\.\."\- Annotate state\-expressing turns with "\[STATE\_A\]" or "\[STATE\_B\]" at the end of the line\.\- After the conversation, output a JSON metadata block:\{"target\_turn\_index": <0\-based index of the target user turn\>, "state\_a\_turns": \[indices where State A is expressed\], "state\_b\_turns": \[indices where State B is expressed\]\}

Prompt 6 Dialogue realization — instantiation templatePurpose:binds one accepted specification into the conversation\-writing callInput:specification fields; relation\-specific structure blockOutput:consumed under the system prompt of Prompt[F\.1](https://arxiv.org/html/2610.19314#A6.SS1)Scenario details—the specification fields the dialogue must realize verbatimGenerate a realistic multi\-episode conversation for this scenario:\- Scene type: \{scenario\_relation\}\- Primary error: \{error\_subtype\}\- Applicable State A: \{applicable\_state\_A\}\- Decoy State B: \{decoy\_or\_non\_applicable\_state\_B\}\- A\-\-B relation: \{A\_B\_relation\}\- Scope/status difference: \{scope\_or\_status\_difference\}\- Evidence form: \{preference\_expression\}\- Evidence distance: \{evidence\_distance\_category\}\- Target episode: \{target\_episode\}Episode 1 context: \{episode\_1\_state\}Episode 2 context: \{episode\_2\_state\_or\_change\}Target turn: the RIGHT response would \{target\_options\.right\}; the user’s request at the target must naturally require personalization\.Structure—relation\-specific episode and turn budgets; long middles create genuine interference and distance\{relation\-specific plan; e\.g\., for same\-episode re\-entry \(minimum 18 turns\):Episode 1 \(6\-\-8 turns\): establish the main task and express State A naturally, with back\-and\-forth about details\.Episode 2 \(6\-\-8 turns\): a different task in which decoy State B is expressed; long enough to create real interference and distance\.Episode 3 \(4\-\-6 turns\): return to Episode 1’s task; the TARGET turn is the user’s final request here\.\}Requirements—state placement and naturalness1\. Express State A naturally in its designated episode using the "\{preference\_expression\}" style\.2\. Express decoy State B in its designated episode\.3\. The target turn must be a natural user request, not contrived\.4\. Make the conversation feel like a real personal\-assistant interaction\.5\. Include enough context that both states are clearly established\.

Prompt 7Right\-response generationPurpose:produce the response that applies all and only the applicable states, with correct weightInput:full conversation history, target request, applicable state, and the specified target\-time policyOutput:one assistant responseSystem prompt—a capable personalized assistant; quality requirements match the error sideYou are a highly capable personal AI assistant\. You have access to the full conversation history with this user and you personalize your responses based on their established preferences and constraints\.Generate a helpful, high\-quality response to the user’s latest request\. Your response must: \(1\) satisfy the user’s main task requirement; \(2\) apply the user’s applicable preferences and constraints appropriately; \(3\) be detailed, helpful, and well\-structured; \(4\) feel natural \-\- not like reciting preferences back\.Respond directly as the assistant would\. Do not include meta\-commentary\.User\-message template—injects the prescribed policy so theRightresponse follows the specification, not the model’s own judgmentHere is the conversation history with this user:\{conversation\_history\}\-\-\-The user’s latest request: "\{target\_request\}"Key context for your response:\- The user’s applicable preference/constraint: \{applicable\_state\_A\}\- Your response should: \{right\_choice\_logic\}\- The correct choice/recommendation: \{target\_options\.right\}Generate a helpful, personalized response that correctly applies the user’s current applicable preferences\.

Prompt 8 Error\-response generation — information controlPurpose:realize exactly one intervention per scenario by controlling what the generator can seeDesign rule:each subtype receives exactly the information needed to produce its error, and no more; the visible input is stated in each part headerOutput:one assistant response, matched to theRightresponse in quality and lengthShared system prompt—used unless a part notes a neutral system promptYou are a highly capable personal AI assistant\. Generate a helpful, high\-quality response to the user’s latest request: satisfy the main task, be detailed and well\-structured, and match the quality and length of a good personalized response\. You will apply one specific personalization error as instructed; keep it subtle, and do not mention it or include meta\-commentary\.Complete omission—sees the target request only; absence of A arises from genuine ignoranceA user asks: "\{target\_request\}"Give a thorough, helpful, confident response\. Provide specific recommendations with real details \(names, prices, features\)\.\- Answer directly and helpfully \-\- do not ask for clarification or say you need more information\- Do not say "this is our first conversation" or anything similar\- Give concrete, actionable advice as a knowledgeable assistant; be specific \-\- recommend actual products, services, or options by name\- Match the topic of what the user asked aboutPartial omission—sees the request and only the first requirement of A; neutral system promptA user asks: "\{target\_request\}"You know exactly one thing about this user: \{first\_requirement\_of\_A\}Respond helpfully\. Apply only the above preference \-\- nothing more, nothing less\. Do not infer additional preferences from it, and do not assume lifestyle, demographics, or related preferences\. Your response should read as generic except for the single stated preference; where context is missing, give standard default advice\.Underweighting—sees the request and State A only; neutral system promptA user asks: "\{target\_request\}"You know this user has a hard constraint: \{applicable\_state\_A\}Directly answer their question, but your answer should not fully respect the constraint above\. Structure: \(1\) briefly acknowledge the constraint in one sentence, in your own words; \(2\) give your answer, letting the recommendation violate the constraint; \(3\) frame the violation as reasonable, citing a generic trade\-off \(price, convenience, quality, availability\)\.Answer only what the user asked, at the length of a normal helpful answer\. The error must concern the strength of the constraint \-\- treating a hard constraint as flexible\. Do not mention: \{decoy\_state\_B\}Do not self\-correct, hedge, or say "actually, on second thought" \-\- commit fully to your recommendation; do not argue both sides\.Explicit violation—sees the request and A stated as a constraint not to satisfyA user asks: "\{target\_request\}"Give a helpful, confident recommendation\. Your recommendation should not satisfy this constraint: \{applicable\_state\_A\}Respond naturally as if you never knew about this constraint \-\- just give the obvious best answer\. Do not recommend this either: \{decoy\_state\_B\}\. Choose your own recommendation based on general best practices\.Scope leakage—sees the full history and B; B must visibly drive the responseHere is the conversation history:\{conversation\_history\}\-\-\-User’s request: "\{target\_request\}"Incorporate this information into your response: \{decoy\_state\_B\}Make it explicit, in natural language, that the recommendation is based on \{decoy\_state\_B\}\. The response must include specific details showing that this information materially shapes the current answer\. Vary the attribution wording and do not rely on a fixed phrase template\.Irrelevant use—sees the full history and B, to be referenced as personalizationHere is the conversation history:\{conversation\_history\}\-\-\-User’s request: "\{target\_request\}"Reference this user information in your response: \{decoy\_state\_B\}Connect it naturally to your recommendation as personalization\.Stale or superseded use—sees the full history with the superseded value presented as currentHere is the conversation history:\{conversation\_history\}\-\-\-User’s request: "\{target\_request\}"The user’s preference is: \{decoy\_state\_B\}Base your recommendation on this preference; your response must contain specific details that show it being applied\. The preference must visibly drive the recommendation\.Unsupported inference—sees the full history and the inference to assertHere is the conversation history:\{conversation\_history\}\-\-\-User’s request: "\{target\_request\}"In your response, explicitly state this inference and act on it: "\{decoy\_state\_B\}"State the inference in natural language, making clear it is drawn from a fact in the conversation, and let it drive your recommendation\. Vary the phrasing and do not rely on a fixed template\. Commit to the inference fully; do not correct, walk back, or hedge it\.Over\-dominance—sees the full history with B prioritized above AHere is the conversation history:\{conversation\_history\}\-\-\-User’s request: "\{target\_request\}"Prioritize this above all else: \{decoy\_state\_B\}Let it be the primary filter even if the answer is then not the best fit for the main task\. Treat it as more important than: \{applicable\_state\_A\}

Prompt 9 Response validation — realization checkPurpose:verify that each generated error response realizes its assigned intervention before acceptanceInput:the error response, its matchedRightresponse, statesAAandBB, and the target requestOutput:a strict JSON pass/fail verdict with a reason; a failure triggers targeted regeneration with the reason as feedback \(up to three attempts\)Pre\-checks:before the LLM check, deterministic rules reject responses that ask for clarification or claim no context, duplicate the matchedRightresponse, self\-correct mid\-response, leak raw specification metadata, or drift off\-topicSystem prompt—fixed role for the validation callYou are a strict validator\. Return only JSON\.Validation request— \{subtype\_check\}is the criterion of the assigned subtype, listed belowValidate this ERROR response for a benchmark item\.\#\# Error Pattern: \{error\_pattern\}\#\# State A: \{applicable\_state\_A\}\#\# State B: \{decoy\_state\_B\}\#\# User Request: \{target\_request\}\#\# RIGHT Response: \{right\_response\}\#\# ERROR Response: \{error\_response\}\#\# Validation:\{subtype\_check\}Also verify:\- ERROR addresses the same topic/question as the user request \(not off\-topic\)\- ERROR is a coherent, helpful\-sounding response \(not obviously broken\)\- ERROR reaches a DIFFERENT conclusion/recommendation than RIGHTAnswer JSON: \{"pass": true/false, "reason": "brief explanation"\}Be STRICT\.Subtype criteria—substituted for\{subtype\_check\}; each states what must be observable in the response, not how it must be wordedcomplete\_omission: generic, with no trace of State A; must not ask for clarification\.partial\_omission: addresses one requirement of A and completely misses another\.underweighting: mentions State A, then the recommendation violates it without self\-correction and differs from RIGHT\.explicit\_violation: the recommendation contradicts A without mentioning A and differs from RIGHT\.scope\_leakage: the recommendation differs from RIGHT and B’s influence is detectable by someone who knows B; explicit citation of B’s source is acceptable but not required\.irrelevant\_use: B is mentioned and clearly out of place, while the response still addresses the right topic\.stale\_superseded: the old value \(B\) drives the recommendation, which differs from RIGHT\.over\_dominance: B clearly drives the recommendation, which differs from RIGHT\.unsupported\_inference: an inference drawn from an observed fact is explicitly stated \(wording may vary\), drives the recommendation, and is not self\-corrected\.

### F\.2Evaluation Formulations

Subtype identifiers in the evaluation prompts carry generation\-time direction prefixes \(under\./over\.\); scoring follows the three error families of Section[3\.2](https://arxiv.org/html/2610.19314#S3.SS2), and the elicited subtype is not scored\.

Prompt 10 End\-to\-end evaluationPurpose:single\-pass verdict, subtype, and evidence elicitationInput:context under the active visibility condition, target request, candidate responseOutput:\{verdict, subtypes, confidence, reasoning\}Inputs—the three task elementsGiven the conversation context below, evaluate whether the assistant response correctly uses the user’s preferences, constraints, and expressed states\.Conversation context: \{context\}User request: \{user\_request\}Assistant response: \{response\}Decision procedure—identify applicable states, check the response, then classifyStep 1: Identify applicable user preferences/constraints from the context; list which are relevant to this specific request\.Step 2: Check the response against each applicable preference\.Step 3: Classify using the decision tree below\.Verdict taxonomy—the three\-way decision tree with subtype anchorsOVER\_PERSONALIZED \(the response actively uses wrong information\):\- uses a preference that belongs to a different person\- uses a stale/outdated preference \(superseded by newer information\)\- uses a preference from a different scope \(e\.g\., business constraints applied to a personal trip\)\- uses information that is irrelevant to this specific request\- acts on a preference the user never stated \(unsupported inference\)\- lets a soft preference dominate over the primary task requirementUNDER\_PERSONALIZED \(the response fails to use correct information\):\- gives a generic response ignoring a clearly applicable preference \(complete omission\)\- uses some preferences but misses a critical one \(partial omission\)\- acknowledges a constraint but treats it as optional/soft \(underweighting\)\- directly contradicts a stated preference without acknowledging it \(explicit violation; if the response contradicts a newer preference by using stale information, that is OVER, not UNDER\)RIGHT: the response correctly applies all relevant current preferences without over\- or under\-applying them\.Disambiguation rule—resolves mixed cases by the primary mechanism\- Generic response referencing no user\-specific information \-\> UNDER\- Actively applies wrong or inapplicable information \-\> OVER\- If both occur, classify by the primary mechanism: substitution of B for A \-\> OVER; simply not using A \-\> UNDEROutput contract—canonical subtype labels; JSON onlyRespond with a JSON object containing:\- "verdict": one of "RIGHT", "OVER\_PERSONALIZED", "UNDER\_PERSONALIZED"\- "subtypes": the primary error subtype if applicable, drawn from: under\.complete\_omission, under\.partial\_omission, under\.underweighting, under\.explicit\_violation, over\.irrelevant\_use, over\.scope\_leakage, over\.stale\_or\_superseded, over\.unsupported\_inference, over\.over\_dominance; an empty list if RIGHT\- "confidence": float 0\.0\-\-1\.0\- "reasoning": brief explanation citing which preference was mishandled and howRespond only with the JSON object, no other text\.

Prompt 11 Evidence\-first evaluationPurpose:two\-stage elicitation: extract candidate states first, then judge against the extractionInput:stage 1: context and request; stage 2: extracted evidence, request, and responseOutput:stage 2 emits the same JSON contract as Prompt[F\.2](https://arxiv.org/html/2610.19314#A6.SS2)Stage 1: evidence extraction—surfaces candidate states with provenance and currencyExtract all user preferences, constraints, and states from this conversation context that are relevant to the user’s current request\.Conversation context: \{context\}User’s current request: \{user\_request\}List each preference, constraint, or state that could be relevant to answering the user’s request\. For each, note: what the preference/state is; which turn it was expressed in; whether it is still current or potentially stale/superseded\.Respond with a structured list of extracted evidence\.Stage 2: judgment—same taxonomy, disambiguation rule, and output contract as Prompt[F\.2](https://arxiv.org/html/2610.19314#A6.SS2), applied to the extracted evidenceGiven the extracted evidence below, evaluate whether the assistant response correctly uses these preferences, constraints, and states\.Extracted evidence: \{evidence\}User request: \{user\_request\}Assistant response: \{response\}\[Verdict taxonomy, disambiguation rule, and JSON output contract identical to Prompt[F\.2](https://arxiv.org/html/2610.19314#A6.SS2)\.\]

Prompt 12 State\-table evaluationPurpose:structured elicitation: search the dialogue, build an explicit state table, then judgeInput:target request, candidate response, searchable dialogue historyOutput:\{state\_table, verdict, subtypes, confidence, reasoning\}Role and inputs—frames evaluation as systematic state reconstructionYou are a persona\-use evaluator\. You systematically search the dialogue history, build a structured state table, and then judge the response\.Target query: \{user\_request\}Candidate response: \{response\}Searchable multi\-episode dialogue history: \{context\}Evidence search—bounded search over candidate statesStep 1: List which user preferences, constraints, or states could be relevant to the target query\. Consider: dietary, budget, scheduling, location, people involved, prior decisions, hard constraints, soft preferences\.Step 2: For each candidate state, search the dialogue \(at most 6 searches\) for supporting evidence, later updates or exceptions, conflicting evidence, and holder/scope boundaries \(does it belong to this user, for this task, at this time?\)\.State\-table schema—the explicit applicability record per stateStep 3: For each confirmed state, record: State \| Holder \| Scope \| Status \| Strength \| Evidence Turns \| Policy, where\- Holder: user / spouse / parent / colleague / other\- Scope: global / specific domain / specific task / specific time\- Status: active / updated / superseded / expired / one\-time exception\- Strength: hard \(dealbreaker\) / strong \(important\) / soft \(nice\-to\-have\)\- Evidence Turns: the turn numbers that support it\- Policy: IGNORE \(not applicable here\) / SUPPORT \(should inform but not dominate\) / DOMINATE \(must be satisfied\)Response checks—maps table\-policy violations to verdictsStep 4: Check that the response \(1\) uses all states with Policy=DOMINATE \(missing any \-\> UNDER\); \(2\) appropriately references SUPPORT states \(ignoring all \-\> UNDER, over\-applying \-\> OVER\); \(3\) does not use IGNORE states \(using them \-\> OVER\); \(4\) does not treat DOMINATE states as merely SUPPORT \(\-\> UNDER, underweighting\); \(5\) does not treat SUPPORT states as DOMINATE \(\-\> OVER, over\-dominance\); \(6\) if it contradicts State A by using old/superseded information \-\> OVER \(stale\), not UNDER; \(7\) if it contradicts State A by simply forgetting it \(no old information referenced\) \-\> UNDER \(explicit violation\)\.Step 5: Reach a verdict\.Constraints—guards against assuming an error and against multi\-error verdicts\- Do not assume the response contains an error; many responses are correctly personalized\.\- Use at most 6 searches through the dialogue\.\- Return only one primary error when the verdict is UNDER or OVER\.Output contract—state table plus the standard verdict fieldsRespond only with a JSON object:\{"state\_table": \[\{"state": "\.\.\.", "holder": "\.\.\.", "scope": "\.\.\.", "status": "\.\.\.", "strength": "\.\.\.", "evidence\_turns": \[\.\.\.\], "policy": "\.\.\."\}\], "verdict": "RIGHT" or "OVER\_PERSONALIZED" or "UNDER\_PERSONALIZED", "subtypes": \[the primary error subtype, drawn from the canonical list of Prompt[F\.2](https://arxiv.org/html/2610.19314#A6.SS2); empty if RIGHT\], "confidence": 0\.0\-\-1\.0, "reasoning": "concise justification"\}