PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants

# PACEShop: Evaluating Personalized, Actionable, Compositional, and Evidence-grounded Shopping Assistants
Source: [https://arxiv.org/html/2608.26180](https://arxiv.org/html/2608.26180)
Chen LuoGuangrui LiAffiliation:Amazon Northeastern University ExpediaYaochen XieDhineshkumar RamasubbuArief KoesdwiadyWanqiu LongHansu GuYutong ChenZheshen WangDakuo WangThanks:Work done at Amazon\.Yi Liu11footnotemark:1

###### Abstract

Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product evidence, and next\-step guidance into a coherent recommendation experience\. This changes the unit of evaluation: a fluent response can still fail by ignoring shopper context, contradicting itself across components, or leaving defects too vague to localize\. Existing personalization, grounding, and LLM\-as\-a\-judge benchmarks cover pieces of this problem, but they do not define a joint evaluation target for structured shopping\-assistant responses\. We formulate this missing evaluation target asPACE:Personalized,Actionable,Compositional, andEvidence\-grounded evaluation\. We instantiatePACEwith two artifacts:PACEShop, a benchmark dataset that makes the target measurable through 22,625 controlled records with structured personas, auditable evidence pools, GOOD/BAD labels, and gold defect family and location annotations; andPACEJudge, a training\-free judging protocol that makes the target reportable through a structured output contract\. Our experiments show that generic judges can recognize broad quality but fail to recover the diagnostic fields required forPACE;PACEShopmakes these failures verifiable, andPACEJudgeimproves persona\-source, cross\-component, grounding, and family/location closure without retraining, showing that realistic shopping\-assistant evaluation requires a task\-matched output contract rather than only a stronger backbone or scalar prompt\.

## 1Introduction

Shopping assistants are moving beyond ranked product lists and single answers toward structured decision support: a response may summarize the shopper’s need, organize product categories, suggest related query refinements, and cite evidence from a retrieved product pool\([9](https://arxiv.org/html/2608.26180#bib.bib1);[15](https://arxiv.org/html/2608.26180#bib.bib2)\)\. This changes the unit of evaluation\. A response is no longer valid merely because one generated text is fluent or broadly relevant; it must be valid as a structured, persona\-conditioned object\. As Fig\.[1](https://arxiv.org/html/2608.26180#S1.F1)illustrates, a response can appear coherent while still failing in ways that a standard scalar judge may miss: it may recommend a brand the shopper avoids, contradict itself across response fields, or leave the defect unable to localize\. Such failures are difficult to capture with an overall quality score, yet they are central to practical evaluation: a realistic judge must identify not only whether the response is wrong, but also what failed, where it failed, and which evidence supports the diagnosis\.

![Refer to caption](https://arxiv.org/html/2608.26180v1/pace_failure_teaser_excalidraw_v3_singlecolumn_cc_v1.png)Figure 1:Generic judges can miss structured failures in shopping\-assistant responses\.For the same query, persona, and response, a standard judge returns GOOD, whilePACEJudgereturns BAD with defect family and field location\. The example failures include a Nike preference conflict and a cross\-field mismatch in related queries, exposing the need forPACEevaluation\.These failures expose an under\-specified evaluation target\. Existing personalization benchmarks study whether systems adapt to user profiles, histories, preferences, or long\-context behavioral signals\([13](https://arxiv.org/html/2608.26180#bib.bib3);[18](https://arxiv.org/html/2608.26180#bib.bib4);[17](https://arxiv.org/html/2608.26180#bib.bib5);[6](https://arxiv.org/html/2608.26180#bib.bib6);[10](https://arxiv.org/html/2608.26180#bib.bib7);[5](https://arxiv.org/html/2608.26180#bib.bib8);[2](https://arxiv.org/html/2608.26180#bib.bib9)\)\. Grounding\-focused evaluators test whether generated claims are supported by retrieved evidence\([12](https://arxiv.org/html/2608.26180#bib.bib10)\)\. Generic LLM\-as\-a\-judge protocols score open\-ended outputs with rubrics, scalar ratings, or pairwise preferences\([19](https://arxiv.org/html/2608.26180#bib.bib12);[8](https://arxiv.org/html/2608.26180#bib.bib11);[14](https://arxiv.org/html/2608.26180#bib.bib13);[7](https://arxiv.org/html/2608.26180#bib.bib14);[1](https://arxiv.org/html/2608.26180#bib.bib15);[4](https://arxiv.org/html/2608.26180#bib.bib16);[3](https://arxiv.org/html/2608.26180#bib.bib17)\)\. These directions are complementary, but they do not define the joint target required for structured shopping\-assistant responses: the judge must condition on shopper context, reason across response components, verify evidence support, and return a diagnosis that localizes the failure\. The gap is therefore conceptual before it is empirical\. A stronger judge backbone or a better scalar prompt cannot report defect fields that the evaluation task never asks it to produce\.

We formulate this missing evaluation target asPACE:Personalized,Actionable,Compositional, andEvidence\-grounded\.Personalizedmeans that validity depends on the shopper context;Actionablemeans that the judge must identify what failed and where;Compositionalmeans that the response must be evaluated as a structured object rather than isolated text;Evidence\-groundedmeans that claims must be supported by product evidence and persona history\. Together,PACEclarifies what current evaluation lacks: records that make such failures observable, and a judging protocol that can report them\. Tab\.[2](https://arxiv.org/html/2608.26180#S3.T2)summarizes this design principle: eachPACEtarget determines what the benchmark must contain, what the judge must output, and what scenario\-level metric can test it\.

To makePACEmeasurable, we introducePACEShop, a verifiable benchmark dataset for structured shopping\-assistant evaluation\. Existing datasets provide valuable personalization or judging signals, but they do not jointly expose persona\-dependent validity, cross\-component consistency, evidence support, and localizable defects in one controlled setting\.PACEShopfills this gap by pairing each shopping query with a structured persona, a candidate response, an auditable evidence pool, and a GOOD/BAD label; BAD records additionally include gold defect family and defect location annotations\. This design turns the abstractPACEformulation into measurable evaluation cases\.

To makePACEreportable, we proposePACEJudge, a training\-free judging protocol\. Existing scalar judges can estimate overall quality, but they do not require the judge to name the violated dimension, localize the broken response field, or ground the diagnosis in evidence\.PACEJudgefills this gap through a judging protocol that evaluatesPACEwith a a structured output schema\. The protocol is backbone\-agnostic, allowing us to separate what comes from model capability from what comes from asking the judge to produce the right diagnostic fields\.

Across multiple LLM backbones and judging configurations, we compare broad GOOD/BAD discrimination with scenario\-level tests that require persona\-source diagnosis, cross\-component localization, evidence\-grounding checks, and actionable defect labels\. The results show that general discrimination is not the main bottleneck: scalar and generic judges can often recognize that a response is broadly good or bad\. The gap appears when evaluation requires the diagnostic fields needed for debugging\. On the same backbones,PACEJudgeimproves these contract\-dependent metrics without retraining, supporting our central claim: realistic shopping\-assistant evaluation requires diagnostic closure over persona, components, evidence, and defect location, not only broad quality discrimination\. Our contributions are threefold:

1. 1\.We identify a problem\-setting gap in existing evaluation for structured shopping assistants, and formulate the missing target asPACE\.
2. 2\.We introducePACEShop, a benchmark dataset that makesPACEmeasurable through controlled persona\-, component\-, and evidence\-grounded failure cases\.
3. 3\.We proposePACEJudge, a training\-free judging protocol that makesPACEreportable and improves diagnostic metrics beyond scalar quality scores\.

Table 1:Existing benchmark datasets against the fourPACEtargets\.PACEShopjointly covers all targets through structured personas, multi\-component responses, auditable evidence pools, and gold defect family/location labels\.
## 2Related Work

##### Personalization, shopping, and search benchmarks\.

Personalized LLM benchmarks evaluate user\-conditioned generation, preferences, memory, likability, and tool use\([13](https://arxiv.org/html/2608.26180#bib.bib3);[18](https://arxiv.org/html/2608.26180#bib.bib4);[17](https://arxiv.org/html/2608.26180#bib.bib5);[6](https://arxiv.org/html/2608.26180#bib.bib6);[10](https://arxiv.org/html/2608.26180#bib.bib7);[5](https://arxiv.org/html/2608.26180#bib.bib8);[2](https://arxiv.org/html/2608.26180#bib.bib9)\)\. Commerce\-oriented resources such as ECom\-Bench\([15](https://arxiv.org/html/2608.26180#bib.bib2)\), Search Arena\([9](https://arxiv.org/html/2608.26180#bib.bib1)\), and OPeRA\([16](https://arxiv.org/html/2608.26180#bib.bib18)\)add realistic e\-commerce support, search\-augmented chat, or online\-shopping behavior signals\. However, they do not provide the joint evaluation unit needed and cannot directly test whether a response satisfiesPACEtargets: personalization, actionability, compositional consistency, and evidence grounding\. Tab\.[1](https://arxiv.org/html/2608.26180#S1.T1)summarizes this benchmark\-side gap\.

##### LLM\-as\-a\-judge and grounded evaluation\.

Generic LLM\-as\-a\-judge methods such as G\-Eval\([8](https://arxiv.org/html/2608.26180#bib.bib11)\)and MT\-Bench\([19](https://arxiv.org/html/2608.26180#bib.bib12)\)established rubric\-based, scalar, and pairwise judging for open\-ended model outputs\. Subsequent work studies judge reliability, alignment, bias, distributional validity, and domain\-specific judging protocols\([14](https://arxiv.org/html/2608.26180#bib.bib13);[7](https://arxiv.org/html/2608.26180#bib.bib14);[1](https://arxiv.org/html/2608.26180#bib.bib15);[4](https://arxiv.org/html/2608.26180#bib.bib16);[3](https://arxiv.org/html/2608.26180#bib.bib17)\)\. Grounding\-focused evaluators such as ARES\([12](https://arxiv.org/html/2608.26180#bib.bib10)\)test whether generated claims are supported by retrieved evidence\. These methods are complementary, but their evaluation contracts are not designed for structured shopping\-assistant diagnosis: they do not jointly require persona conditioning, cross\-component reasoning, evidence support, defect\-family prediction, and field\-level localization\.PACEJudgeaddresses this protocol\-side gap by making the fourPACEtargets reportable through structured diagnostic outputs\.

## 3PACEShopDataset

Table 2:MakingPACEmeasurable and reportable\.EachPACEtarget maps to whatPACEShopmakes measurable, how thePACEJudgeprotocol makes it reportable, and which evaluation scenario tests the corresponding failure mode\.PACEShopis designed to makePACEmeasurable\. The goal is to construct controlled and verifiable records in which personalization, actionability, compositional consistency, and evidence grounding can be observed and audited\. Tab\.[2](https://arxiv.org/html/2608.26180#S3.T2)previews how eachPACEtarget is turned into measurable benchmark controls inPACEShop\. The benchmark construction follows a valid\-response\-first, controlled\-defect\-second design: we first create and validate structured GOOD responses, then derive BAD variants through controlled edits with known defect family and location labels\.

### 3\.1Benchmark Task

##### Scope\.

PACEShopfocuses on the generated response layer of shopping assistants rather than the full ranking interface or end\-to\-end shopping session\. This scope isolates the judging problem: given a shopping assistant user inputs and responses, can a judge determine whether the response is valid and diagnose any failure?

##### Task formulation\.

EachPACEShoprecord defines a pointwise judging task over a tuple

\(q,p,y,E,ℓ\),\(q,p,y,E,\\ell\),whereqqis a shopping query,ppis a structured persona,yyis a candidate shopping\-assistant response,EEis an auditable evidence pool, andℓ∈\{GOOD,BAD\}\\ell\\in\\\{\\textsc\{GOOD\},\\textsc\{BAD\}\\\}is the validity label\. The judge receives\(q,p,y,E\)\(q,p,y,E\)and predicts

ℓ^=J⁡\(q,p,y,E\),ℓ^∈\{GOOD,BAD\}\.\\hat\{\\ell\}=J\(q,p,y,E\),\\qquad\\hat\{\\ell\}\\in\\\{\\textsc\{GOOD\},\\textsc\{BAD\}\\\}\.For BAD responses, the judge should additionally diagnose the failure by predicting a defect familyd∈𝒟d\\in\\mathcal\{D\}and a defect locationr∈ℛr\\in\\mathcal\{R\}\. Thus, the benchmark measures not only whether a judge can separate GOOD from BAD responses, but also whether it can identify what failed and where the failure occurs\.

![Refer to caption](https://arxiv.org/html/2608.26180v1/pace_generation_task_flow_v2_excalidraw.png)Figure 2:PACEShopbenchmark dataset makesPACEmeasurable; andPACEJudgeevaluation protocal makesPACEreportable\.Top:PACEShopdata construction carries persona, shopping behavior, query intent, and evidence through assignment, GOOD\-response generation, controlled BAD construction, and validity checks\.Middle:the resulting records make PACE properties measurable through S1–S4, with S0 as a reference GOOD/BAD check\.Bottom:PACEJudgeturns each record\(q,p,y,E\)\(q,p,y,E\)into a structured output whose fields support scenario\-aligned evaluation\.

### 3\.2PACEShopConstruction Pipeline

The PACEShop construction pipeline makesPACEfailures observable and auditable\. It isolates shopper\-context effects through persona assignment, makes grounding checkable through evidence\-backed query retention, uses validated GOOD responses as verified base cases, and derives controlled BAD variants with gold defect family and location labels\. Full construction details, prompts, schema checks, and examples are provided in Appx\.[B](https://arxiv.org/html/2608.26180#A2)\.

##### Persona and evidence setup\.

We construct 1,200 personas as structured shopping contexts rather than free\-form user descriptions\. Each persona specifies stable shopper information, such as household type, budget, quality sensitivity, urgency, constraints, preferences, and brand affinities, together with concrete shopping behavior in the form of purchase history and recent searches\. We pair these personas with evidence\-backed shopping queries drawn from a public query and product\-evidence pool\. This setup lets the same query and evidence pool support different valid responses under different shopper contexts, making personalization and persona\-history grounding testable\.

##### Query–persona assignment\.

We begin with 97,227 normalized US shopping queries, 1,200 structured personas, and 4,460 public evidence records across 11 shopping domains\.111Coverage is audited at the source level: queries are tagged by retail vertical and shopping mission, personas by shopper attributes and behavior\-history fields, and evidence records by domain and provenance\. Appx\.[B\.3](https://arxiv.org/html/2608.26180#A2.SS3)reports the detailed coverage breakdowns and representative examples\.We retain 1,132 queries with sufficient evidence support and pair each retained query with distinct personas, yielding 4,525 validated query–persona pairs\. The repeated\-measures design holds the query and evidence pool fixed while varying shopper context, allowing persona\-dependent failures to be isolated from changes in shopping intent\.

##### GOOD\-response generation\.

For each query–persona pair, a strong LLM generator produces one structured GOOD candidate under a constrained response contract\. We do not treat these generations as valid merely because they come from a strong model: each candidate must pass deterministic normalization and validation, including schema, cardinality, attribution\-ID, and parsing checks\. Accepted GOOD responses therefore serve as contract\-valid benchmark anchors for controlled defect construction\.

##### Controlled BAD construction\.

From each validated GOOD response, we construct four BAD variants containing one, two, three, and four defects\. Defects are drawn from seven families covering persona conflicts, intent drift, cross\-component mismatch, redundancy, evidence mismatch, over\-personalized hallucination, and unsupported claims\. Because each BAD record is created by a controlled edit, its gold defect family and field location are known by construction\.

##### Validity checks\.

PACEShopis designed to makePACEfailures measurable and verifiable, not to rely on synthetic generation as ground truth\. Each released record carries an audit trail: deterministic validators check schema compliance and evidence\-ID consistency, validated GOOD responses are used as base cases before any defect injection, controlled edits make BAD labels traceable to known defect families and response fields, and evidence\-provenance flags expose the strength of the grounding signal\. Together, these layers address the main dataset\-quality risks: synthetic\-on\-synthetic circularity, thin grounding signal, persona\-template artifacts, rigid schema effects, and limited transfer beyond one model family\. Appx\.[B\.4](https://arxiv.org/html/2608.26180#A2.SS4)details the five verification layers; Appx\.[B\.5](https://arxiv.org/html/2608.26180#A2.SS5)defines the defect taxonomy; Appx\.[B\.9](https://arxiv.org/html/2608.26180#A2.SS9)gives GOOD/BAD record generation procedures\.

Dataset statistics\.PACEShop contains 22,625 records: 4,525 GOOD records and 18,100 BAD records\. BAD records are evenly distributed across single\-, double\-, triple\-, and quad\-defect tiers, and every BAD record carries gold defect family and location annotations\. Tab\.[8](https://arxiv.org/html/2608.26180#A2.T8)in Appx\.[B](https://arxiv.org/html/2608.26180#A2)summarizes the benchmark statistics, evidence coverage, defect structure, and validation assets\.

## 4PACEJudgeEvaluation Protocol

PACEJudgeis a training\-free pointwise judging protocol that makesPACEreportable\. In Tab\.[2](https://arxiv.org/html/2608.26180#S3.T2),PACEShopmakes thePACEtargets measurable through controlled records, whilePACEJudgespecifies how a judge should report the diagnosis\. Given aPACEShoprecord, the protocol asks the judge to evaluate the candidate response as a structured, persona\-conditioned, evidence\-grounded object rather than as a single fluent text\. This addresses the central limitation of scalar judging: an overall quality score does not identify whether the failure comes from shopper\-context mismatch, cross\-component inconsistency, unsupported evidence, or a defect that cannot be localized\.

### 4\.1Protocol Overview and Output Schema

Given aPACEShoprecord\(q,p,y,E\)\(q,p,y,E\),PACEJudgereturns

J⁡\(q,p,y,E\)→\(ℓ^,p^bad,sP,aA,sC,sE,sF\),J\(q,p,y,E\)\\rightarrow\(\\hat\{\\ell\},\\hat\{p\}\_\{\\mathrm\{bad\}\},s\_\{\\mathrm\{P\}\},a\_\{\\mathrm\{A\}\},s\_\{\\mathrm\{C\}\},s\_\{\\mathrm\{E\}\},s\_\{\\mathrm\{F\}\}\),whereℓ^∈\{GOOD,BAD\}\\hat\{\\ell\}\\in\\\{\\textsc\{GOOD\},\\textsc\{BAD\}\\\}is the verdict,p^bad∈\[0,1\]\\hat\{p\}\_\{\\mathrm\{bad\}\}\\in\[0,1\]is the BAD probability,sP,sC,sE∈\[1,5\]s\_\{\\mathrm\{P\}\},s\_\{\\mathrm\{C\}\},s\_\{\\mathrm\{E\}\}\\in\[1,5\]are the Personalized, Compositional, and Evidence\-grounded scores, andsF∈\[1,5\]s\_\{\\mathrm\{F\}\}\\in\[1,5\]is an auxiliary format/safety score\. The Actionable target is represented as a diagnostic object rather than a scalar:

aA=\(d^,r^,S^,c^,z^\),a\_\{\\mathrm\{A\}\}=\(\\hat\{d\},\\hat\{r\},\\hat\{S\},\\hat\{c\},\\hat\{z\}\),whered^\\hat\{d\}is the predicted defect family,r^\\hat\{r\}is the predicted defect location,S^\\hat\{S\}is the set of supporting evidence IDs,c^\\hat\{c\}is confidence, andz^\\hat\{z\}is a short rationale\. The diagnostic fields follow the benchmark schema:

d^∈𝒟∪\{None\},r^∈ℛ∪\{None\},S^⊆E,\\hat\{d\}\\in\\mathcal\{D\}\\cup\\\{\\textsc\{None\}\\\},\\quad\\hat\{r\}\\in\\mathcal\{R\}\\cup\\\{\\textsc\{None\}\\\},\\quad\\hat\{S\}\\subseteq E,where𝒟\\mathcal\{D\}is the seven\-family defect taxonomy andℛ\\mathcal\{R\}is the response\-field set \(overview,categories,related queries, andattributions\)\. For GOOD verdicts, the defect family and location should beNone; for BAD verdicts, the judge identifies the primary defect family and response field\. Detailed protocol and examples can be found in Appx\.[D\.4](https://arxiv.org/html/2608.26180#A4.SS4)\.

Table 3:MainPACE\-closure results averaged over seven LLM backbones\.S0reports reference GOOD/BAD balanced accuracy, whileS1–S4evaluatePACEdiagnostic closure\.PACEJudgeachieves the strongest overall performance among multi\-target rubric and the single\-target methods, showing that the main gap is not broad GOOD/BAD discrimination but recovering the diagnostic fields needed forPACEevaluation\.False\-firemeasures defect alarms on GOOD records, andPACEavg\.combines closure metrics with GOOD\-record specificity\. “–” marks unsupported cells under a method’s native judging protocol\. Full per\-backbone results \(Tab\.[23](https://arxiv.org/html/2608.26180#A5.T23)\) and metric definitions are provided in Appx\.[E\.4](https://arxiv.org/html/2608.26180#A5.SS4)\.
### 4\.2Coverage of thePACETargets

The protocol is designed to cover allPACEtargets\. ForP, the persona scoresPs\_\{\\mathrm\{P\}\}makes shopper context part of the verdict\. ForA, the diagnostic objectaAa\_\{\\mathrm\{A\}\}requires the judge to report what failed, where it failed, and what evidence supports the diagnosis\. ForC, the compositional scoresCs\_\{\\mathrm\{C\}\}and defect\-location fieldr^\\hat\{r\}require the judge to evaluate the response as a structured object rather than isolated text\. ForE, the evidence scoresEs\_\{\\mathrm\{E\}\}and evidence\-ID constraintS^⊆E\\hat\{S\}\\subseteq Erequire grounding in both the product evidence pool and persona history\.

This design makesPACEJudgeevaluable beyond GOOD/BAD accuracy\. A scalar judge may call a response “mostly helpful,” but it has no required field for the violated target, the broken response component, the defect family, or supporting evidence\.PACEJudgemakes these diagnostic fields explicit, enabling the scenario\-level metrics in Sec\.[5](https://arxiv.org/html/2608.26180#S5)\. A detailed per\-target walkthrough of howPACEJudgereports the correspondingPACEtarget, together with the matching scenario\-level metric, is provided in Appx\.[D](https://arxiv.org/html/2608.26180#A4)\.

## 5Experiments

PACEShopprovides controlled records where persona, compositional, grounding, and actionable\-diagnosis failures are measurable;PACEJudgespecifies the diagnostic fields a judge should return\. We therefore ask a simple question: when evaluated on the same records and backbones, do scalar or generic judge protocols recover the diagnostic fields needed forPACEevaluation, or is an explicit structured protocol necessary?

### 5\.1Experimental Setup

We evaluate judge protocols onPACEShoprecords using seven LLM backbones \(Opus 4\.7, Sonnet 4\.6, Sonnet 4\.5, Haiku 4\.5, Qwen3 32B, GPT\-OSS 20B, GPT\-OSS 120B\)\. We comparePACEJudgewith two baseline families onPACEShoprecords\. The first family consists of*multi\-target rubric ablations*that share thePACEJudgeoutput schema but vary the judging instruction: G\-Eval\-style scalar judging\([8](https://arxiv.org/html/2608.26180#bib.bib11)\), MT\-Bench\-style scalar judging\([19](https://arxiv.org/html/2608.26180#bib.bib12)\), persona\-blind judging, and a persona\-aware generic rubric\. The second family consists of*single\-target adapted baselines*whose native protocols cover only partialPACEdimensions: ARES\-style grounding\([12](https://arxiv.org/html/2608.26180#bib.bib10)\), PersonaLens\-style personalization\([18](https://arxiv.org/html/2608.26180#bib.bib4)\), and EtaPP\-style actionable diagnosis\([5](https://arxiv.org/html/2608.26180#bib.bib8)\)\. Full backbone, prompt, and native\-support details are in Appx\.[E](https://arxiv.org/html/2608.26180#A5)\.

### 5\.2Evaluation Scenarios

Building on the target\-to\-scenario mapping in Tab\.[2](https://arxiv.org/html/2608.26180#S3.T2), we report one reference scenario and fourPACEclosure tests\. S0 measures broad GOOD/BAD discrimination and is included only to show whether records are broadly judgeable\. S1–S4 test persona\-source diagnosis, cross\-component detection and localization, grounding over product evidence and persona history, and actionable defect\-family/field\-location prediction, respectively\. Tab\.[22](https://arxiv.org/html/2608.26180#A5.T22)links these scenarios to metric columns; full definitions are provided in Appx\.[E\.3](https://arxiv.org/html/2608.26180#A5.SS3)\.

Table 4:Strict paired\-counterfactualPACE\-closure results\.Each S1–S4 score is a joint metric that requires more than detecting BAD examples: the judge must also satisfy target\-specific conditions\. These gates prevent an always\-BAD judge from receiving high credit through over\-flagging\. The last two rows summarize the lift fromPACEJudgeover the strongest comparable baseline, showing consistent gains under strict closure metrics at both the seven\-backbone mean and best\-backbone levels\. Metric definitions are in Appx\.[E\.4](https://arxiv.org/html/2608.26180#A5.SS4); native\-protocol support and leakage audits for single\-target adapted baselines are in Tabs\.[19](https://arxiv.org/html/2608.26180#A3.T19)and[20](https://arxiv.org/html/2608.26180#A3.T20)\.
### 5\.3Main Results

Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)tells a simple story: broad GOOD/BAD judging is not where the main difficulty lies\. Most judges/protocols can often decide whether a response is generally valid or flawed, but they break down when the metric asks for the fields needed to debug the failure: whichPACEtarget was violated, which response field failed, and whether the diagnosis is grounded in the provided evidence\.PACEJudgeis strongest on this diagnostic closure setting among multi\-target rubric ablations, while single\-target adapted baselines leave most columns unsupported because their native protocols cover only one part ofPACE\. Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)then asks a stricter question: does the judge simply flag more responses as BAD? Under paired\-counterfactual gates that require GOOD\-record specificity, correct target/family/location, axis consistency, and evidence\-ID containment,PACEJudgeremains strongest\. The improvement therefore reflects better diagnostic closure, not merely more aggressive defect detection\.

##### Finding 1: broad accuracy and stronger backbones do not solvePACEclosure\.

S0 in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)shows that scalar and generic judges can perform similarly toPACEJudgeon broad GOOD/BAD discrimination\. However, this binary signal does not translate intoPACEclosure: a judge may mark a response BAD while still failing to identify whether the failure comes from shopper\-context mismatch, cross\-component inconsistency, unsupported evidence, or a missing actionable diagnosis\. Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)shows the same pattern in the stricter audit: even frontier self\-consistency improves reference accuracy, but generic rubrics still trailPACEJudgeon the joint S1–S4 closure metrics\. This shows that stronger inference alone cannot recover fields that the protocol does not ask the judge to produce\.

##### Finding 2:PACEJudgeimproves the diagnostic fields required byPACE\.

Across the seven\-backbone in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3),PACEJudgeachieves the strongest overallPACEaverage among multi\-target rubric ablations\. The gain is diagnostic rather than scalar:Prequires identifying persona conflict as the source of invalidity,Crequires localizing the failed response component,Erequires grounding against both product evidence and persona history, andArequires defect\-family and field\-location prediction\. Fig\.[3](https://arxiv.org/html/2608.26180#S5.F3)summarizes this S1–S4 closure pattern, omitting S0 because the central claim is not broad classification but task\-aligned diagnosis\.

##### Finding 3: protocol design helps, but backbone capability sets the ceiling\.

The per\-backbone breakdowns in Appx\.[E\.5\.1](https://arxiv.org/html/2608.26180#A5.SS5.SSS1),[E\.5\.2](https://arxiv.org/html/2608.26180#A5.SS5.SSS2), and[E\.5\.3](https://arxiv.org/html/2608.26180#A5.SS5.SSS3)show a common pattern:PACEJudgeis most effective when the backbone can follow a structured contract, calibrate GOOD\-record specificity, and reason over persona and field\-level constraints\. Frontier backbones such as Sonnet 4\.6 and Opus 4\.7 obtain the strongestPACEclosure with relatively low false\-fire rates, while smaller or open\-weight backbones \(Qwen3 32B, GPT\-OSS 20B\) often improve on individual diagnostic columns but remain unstable on persona\-source recall, grounding, or false\-fire control\. Thus,PACEJudgesupplies the missing output contract, but the absolute closure level is still bounded by the backbone’s ability to execute that contract\.

##### Finding 4: persona\-source diagnosis and component localization expose hidden failures\.

Tabs\.[5](https://arxiv.org/html/2608.26180#S5.T5)and[6](https://arxiv.org/html/2608.26180#S5.T6)unpack two failure modes that are easy to hide behind scalar quality scores\. Tab\.[5](https://arxiv.org/html/2608.26180#S5.T5)shows that personalization must be evaluated as a source of invalidity: the response is unchanged, but the swapped persona makes it wrong, so credit requires predictingPREF\_CONFLICTrather than merely lowering an overall score\. Tab\.[6](https://arxiv.org/html/2608.26180#S5.T6)separates detecting a cross\-component defect from localizing the response field that must be fixed; this distinction is central to actionable debugging because a judge can know that a response is flawed while still failing to say where the repair is needed\.

##### Finding 5: gains are most informative on low\-signal, single\-defect cases\.

The four\-tier BAD construction \(1\-, 2\-, 3\-, and 4\-defect BAD records\) lets us test whether judges diagnose individual failures or merely accumulate surface alarms as more defects are added\. Fig\.[4](https://arxiv.org/html/2608.26180#S5.F4)shows that the largest protocol gap appears on one\-defect records, where only a single response field is wrong and surface signal is weakest\.PACEJudgeremains stable across the tiered difficulty gradient, suggesting that its advantage comes from structured diagnosis rather than simply detecting that heavily corrupted responses look bad\.

Table 5:Persona\-source diagnosis by conflict axis, averaged over backbones\.A hit requires predictingPREF\_CONFLICTon a counterfactual persona swap\.PACEJudgeis strongest across all axes, showing that personalization failures require source\-level diagnosis rather than broad BAD detection\. Axis means correspond to S1 in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3); full per\-backbone results are in Appx\.[E\.5\.2](https://arxiv.org/html/2608.26180#A5.SS5.SSS2)\.Table 6:Cross\-component detection and localization\.Detection flags whether a compositional defect is present; 4\-way localization requires identifying the failed response field\.PACEJudgeis strongest on both, showing that compositional failures require field\-level diagnosis beyond broad BAD detection\. Scores correspond to S2 detect / loc\. in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3); full per\-backbone results are in Appx\.[E\.5\.3](https://arxiv.org/html/2608.26180#A5.SS5.SSS3)\.Figure 3:PACE\-closure scores averaged over seven backbones\.PACEJudgeis strongest on the S1–S4 metrics that require persona\-source diagnosis, cross\-component localization, grounding control, and actionable defect localization\.Figure 4:GOOD/BAD accuracy by defect tier\.BAD records are grouped by the number of injected defects\. The 1\-defect tier isolates the hardest setting for surface\-level detection, while higher tiers expose whether methods mainly benefit from accumulating defects\.

## 6Conclusion

We argued that shopping\-assistant evaluation should be treated as a jointPACEproblem: responses must be personalized, actionable, compositional, and evidence\-grounded\.PACEShopmakes this setting measurable through controlled and verifiable records with structured personas, auditable evidence pools, and gold defect family/location labels\. Across seven backbones and multiple judging configurations, scalar and generic rubric judges often recognize broad GOOD/BAD quality but fail to recover the diagnostic fields needed for debugging\.PACEJudgeimproves these closure metrics without retraining\. These results support our central claim: for structured shopping assistants, the key evaluation challenge is not broad quality scoring but producing a task\-aligned diagnosis of what failed, where, and why\.

## Limitations

PACEShopandPACEJudgehave several bounded limitations\.\(1\) Synthetic GOOD/BAD construction\.GOOD responses come from a single generator \(Claude Opus 4\.6\) and BAD responses from rule\-based defect injection; the cross\-generator and real\-assistant slices audit but do not fully reproduce live traffic\.\(2\) Evidence depth\.Most evidence records carry product\-listing fields only; 129/4,460 are review\-augmented, bounding how deeply grounding can be audited per claim\.\(3\) Holdout scope\.Held\-out persona bundles cover 345/1,200 personas across 8 families; this is narrower than real\-user generalization across demographics, regions, and languages\. The dataset is English\-only\.

## Potential Risks

PACEShopandPACEJudgeare intended for evaluating structured shopping\-assistant responses, not for directly certifying production systems or replacing human review\. A main risk is personalization\. Although the personas are synthetic or structured benchmark artifacts, personalization evaluation can encourage systems to infer or exploit user traits if transferred carelessly to real deployments\. Practical use should therefore avoid sensitive personal attributes, preserve user privacy, and treat persona\-conditioned evaluation as a tool for detecting misalignment rather than for maximizing persuasion or user targeting\.

## References

- Chenet al\.\(2025\)L\. Chen, Z\. Zhang, H\. Tan, Q\. Dai, H\. Yang, Z\. Dong, and X\. ChenBeyond single\-point judgment: distribution alignment for llm\-as\-a\-judge\.arXiv preprint arXiv:2505\.12301\.Cited by:[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1)\.
- Donget al\.\(2024\)Y\. R\. Dong, T\. Hu, and N\. CollierCan llm be a personalized judge?\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 10126–10141\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.8.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.9.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.5.1.1.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Enguehardet al\.\(2025\)J\. Enguehard, M\. Van Ermengem, K\. Atkinson, S\. Cha, A\. G\. Chowdhury, P\. K\. Ramaswamy, J\. Roghair, H\. R\. Marlowe, C\. S\. Negreanu, K\. Boxall,et al\.LeMAJ \(legal llm\-as\-a\-judge\): bridging legal reasoning and llm evaluation\.InProceedings of the Natural Legal Language Processing Workshop 2025,pp\. 318–337\.Cited by:[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1)\.
- Feueret al\.\(2025\)B\. Feuer, C\. Tseng, A\. S\. Lathe, O\. Elachqar, and J\. P\. DickersonWhen judgment becomes noise: how design failures in llm judge benchmarks silently undermine validity\.arXiv preprint arXiv:2509\.20293\.Cited by:[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1)\.
- Haoet al\.\(2025\)Y\. Hao, P\. Cao, Z\. Jin, H\. Liao, Y\. Chen, K\. Liu, and J\. ZhaoEvaluating personalized tool\-augmented llms from the perspectives of personalization and proactivity\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 21897–21935\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.7.1.1.1),[§C\.3\.2](https://arxiv.org/html/2608.26180#A3.SS3.SSS2.p4.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.8.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.4.1.1.1),[Table 20](https://arxiv.org/html/2608.26180#A3.T20.2.5.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p3.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.68.1.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.69.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.70.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.71.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.72.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.73.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.74.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.75.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.76.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.7.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2608.26180#S4.T3.2.1.12.1),[§5\.1](https://arxiv.org/html/2608.26180#S5.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.26180#S5.T4.2.1.6.1)\.
- Jianget al\.\(2025\)B\. Jiang, Z\. Hao, Y\. Cho, B\. Li, Y\. Yuan, S\. Chen, L\. Ungar, C\. J\. Taylor, and D\. RothKnow me, respond to me: benchmarking llms for dynamic user profiling and personalized responses at scale\.arXiv preprint arXiv:2504\.14225\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.5.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.6.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.8.1.1.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p2.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.5.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2025\)D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu,et al\.From generation to judgment: opportunities and challenges of llm\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 2757–2791\.Cited by:[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§C\.3\.1](https://arxiv.org/html/2608.26180#A3.SS3.SSS1.p2.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.13.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.15.1),[Table 28](https://arxiv.org/html/2608.26180#A5.T28.2.1.4.1.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.26180#S4.T3.2.1.4.1),[§5\.1](https://arxiv.org/html/2608.26180#S5.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.26180#S5.T4.2.1.12.1),[Table 4](https://arxiv.org/html/2608.26180#S5.T4.2.1.8.1)\.
- Miroyanet al\.\(2025\)M\. Miroyan, T\. Wu, L\. King, T\. Li, J\. Pan, X\. Hu, W\. Chiang, A\. N\. Angelopoulos, T\. Darrell, N\. Norouzi,et al\.Search arena: analyzing search\-augmented llms\.arXiv preprint arXiv:2506\.05334\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.10.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.11.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.6.1.1.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p3.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.11.1),[§1](https://arxiv.org/html/2608.26180#S1.p1.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Rahmanet al\.\(2025\)M\. A\. Rahman, A\. Gabrys, D\. Kang, J\. Sun, T\. Tan, and A\. ChandramouliLikeBench: evaluating subjective likability in llms for personalization\.arXiv preprint arXiv:2512\.13077\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.6.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.7.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.9.1.1.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p2.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.6.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Reddyet al\.\(2022\)C\. K\. Reddy, L\. Màrquez, F\. Valero, N\. Rao, H\. Zaragoza, S\. Bandyopadhyay, A\. Biswas, A\. Xing, and K\. SubbianShopping queries dataset: a large\-scale esci benchmark for improving product search\.arXiv preprint arXiv:2206\.06588\.Cited by:[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.10.1)\.
- Saad\-Falconet al\.\(2024\)J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. ZahariaAres: an automated evaluation framework for retrieval\-augmented generation systems\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 338–354\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.11.1.1.1),[§C\.3\.2](https://arxiv.org/html/2608.26180#A3.SS3.SSS2.p2.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.12.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.2.1.1.1),[Table 20](https://arxiv.org/html/2608.26180#A3.T20.2.3.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p6.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.50.1.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.51.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.52.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.53.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.54.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.55.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.56.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.57.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.58.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.26180#S4.T3.2.1.10.1),[§5\.1](https://arxiv.org/html/2608.26180#S5.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.26180#S5.T4.2.1.4.1)\.
- Salemiet al\.\(2024\)A\. Salemi, S\. Mysore, M\. Bendersky, and H\. ZamaniLamp: when large language models meet personalization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7370–7392\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.2.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.3.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.10.1.1.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.2.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Thakuret al\.\(2025\)A\. S\. Thakur, K\. Choudhary, V\. S\. Ramayapally, S\. Vaidyanathan, and D\. HupkesJudging the judges: evaluating alignment and vulnerabilities in llms\-as\-judges\.InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics \(GEM2\{\}^\{2\}\),pp\. 404–430\.Cited by:[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025a\)H\. Wang, X\. Peng, H\. Cheng, Y\. Huang, M\. Gong, C\. Yang, Y\. Liu, and J\. LinECom\-bench: can llm agent resolve real\-world e\-commerce customer support issues?\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 276–284\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.9.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.10.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.9.1),[§1](https://arxiv.org/html/2608.26180#S1.p1.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, Y\. Lu, W\. Li, A\. Amini, B\. Sun, Y\. Bart, W\. Lyu, J\. Gesi, T\. Wang, J\. Huang,et al\.Opera: a dataset of observation, persona, rationale, and action for evaluating llms on human online shopping behavior simulation\.arXiv preprint arXiv:2506\.05606\.Cited by:[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.8.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2025a\)S\. Zhao, M\. Hong, Y\. Liu, D\. Hazarika, and K\. LinDo llms recognize your preferences? evaluating personalized preference following in llms\.arXiv preprint arXiv:2502\.09597\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.4.1.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.5.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.7.1.1.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p2.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.4.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2025b\)Z\. Zhao, C\. Vania, S\. Kayal, N\. Khan, S\. B\. Cohen, and E\. YilmazPersonalens: a benchmark for personalization evaluation in conversational ai assistants\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 18023–18055\.Cited by:[Table 7](https://arxiv.org/html/2608.26180#A1.T7.2.3.1.1.1),[§C\.3\.2](https://arxiv.org/html/2608.26180#A3.SS3.SSS2.p3.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.4.1),[Table 19](https://arxiv.org/html/2608.26180#A3.T19.2.1.3.1.1.1),[Table 20](https://arxiv.org/html/2608.26180#A3.T20.2.4.1),[§D\.2](https://arxiv.org/html/2608.26180#A4.SS2.p2.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.59.1.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.60.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.61.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.62.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.63.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.64.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.65.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.66.1),[Table 23](https://arxiv.org/html/2608.26180#A5.T23.2.1.67.1),[Table 1](https://arxiv.org/html/2608.26180#S1.T1.2.1.3.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2608.26180#S4.T3.2.1.11.1),[§5\.1](https://arxiv.org/html/2608.26180#S5.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.26180#S5.T4.2.1.5.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§C\.3\.1](https://arxiv.org/html/2608.26180#A3.SS3.SSS1.p3.1.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.13.1),[Table 18](https://arxiv.org/html/2608.26180#A3.T18.2.1.16.1),[Table 28](https://arxiv.org/html/2608.26180#A5.T28.2.1.12.1.1),[§1](https://arxiv.org/html/2608.26180#S1.p2.1),[§2](https://arxiv.org/html/2608.26180#S2.SS0.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.26180#S4.T3.2.1.5.1),[§5\.1](https://arxiv.org/html/2608.26180#S5.SS1.p1.1),[Table 4](https://arxiv.org/html/2608.26180#S5.T4.2.1.13.1)\.

## Appendix AExisting Benchmark Datasets

Table 7:Native judging protocols of the prior personalization, evidence\-grounding, and judging resources cited in the main text\. None of these protocols defines a closure metric for the fourPACEtargets \(Personalized, Compositional, Evidence\-grounded, Actionable\); they evaluate user\-conditioned generation, MCQ over user history, RAG faithfulness, or pairwise human votes, but not the joint persona–compositional–grounded–actionable diagnosis thatPACEShoprequires\. We additionally implement honest single\-target adapted baselines based on ARES \(E target\), PersonaLens \(P target\), and EtaPP \(A target\) and report them in the main experiment table \(Table[3](https://arxiv.org/html/2608.26180#S4.T3)\); the remaining seven resources are not run onPACEShopbecause their native judging protocols cannot be honestly mapped to the four\-target diagnosis without forcing inapplicable scores\.This section documents how the prior resources cited in Tab\.[1](https://arxiv.org/html/2608.26180#S1.T1)relate to the PACEShop evaluation setting\. The goal is not to argue that prior personalization, shopping, RAG, or LLM\-as\-judge benchmarks are weak\. Rather, they were designed for different native tasks: user\-conditioned generation, dialogue preference following, memory retrieval, task success, pairwise search comparison, or RAG faithfulness\. These protocols measure important pieces of the problem, but they do not define the same closure target as PACEShop: given a query, persona, four\-field shopping response, and evidence pool, the evaluator must diagnose whether the response is personalized, cross\-field consistent, evidence\-grounded, and actionable for debugging\.

Tab\.[7](https://arxiv.org/html/2608.26180#A1.T7)therefore lists each resource by its*native evaluation protocol*\. This distinction matters for the experiment tables\. A benchmark may be strongly related to one PACE target while still lacking the output fields needed for the S1–S4 closure metrics\. For example, personalization benchmarks can evaluate whether a response follows user preferences, but usually do not define a persona\-swap defect family and response\-field location\. RAG evaluators can measure faithfulness to retrieved context, but usually do not check whether a response invents shopper history or whether the judge itself cites evidence IDs outside the evidence pool\. Shopping task benchmarks can evaluate realistic user trajectories, but they do not provide gold defect families and locations for four\-field shopping\-assistant responses\.

For this reason, the main experiment table \(Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\) does not force every prior resource into every PACEShop scenario\. Instead, unsupported native judging protocols are marked as “–”\. We implement honest single\-target adapted baselines only when the native judging protocol gives a defensible mapping to onePACEtarget:ARES\-stylefor evidence grounding \(E\),PersonaLens\-stylefor personalization \(P\), andEtaPP\-stylefor actionable key\-point diagnosis \(A\)\. The other resources remain comparison points for problem setting and protocol coverage, but are not converted into artificial all\-target baselines\. This keeps the appendix comparison auditable: “–” means the native protocol does not define the scored PACEShop output field, not that the resource is unimportant\.

## Appendix BPACEShopDataset Construction

This appendix documents the construction\-side material that supportsPACEShop\. We first describe the full construction pipeline and the validity layers \(Secs\.[B\.2](https://arxiv.org/html/2608.26180#A2.SS2)–[B\.5](https://arxiv.org/html/2608.26180#A2.SS5)\); we close with concrete records, JSON input examples, the response\-generation prompt, holdout\-persona statistics, and source\-side coverage tables that make the benchmark auditable \(Secs\.[B\.6](https://arxiv.org/html/2608.26180#A2.SS6)–[B\.3](https://arxiv.org/html/2608.26180#A2.SS3)\)\.

### B\.1PACEShopDataset Statistics

Tab\.[8](https://arxiv.org/html/2608.26180#A2.T8)summarizes the benchmark statistics, evidence coverage, defect structure, and validation assets\.

BlockStatisticValueSourceBackbone queries \(public\)97,227Evidence\-backed queries \(≥3\\geq 3products\)1,132Persona pool \(constraints\+history\+brand\)1,200Query–persona pairs4,525EvidenceEvidence records \(11 domains\)4,460Review\-augmented records129Median evidence IDs / response10DatasetGOOD / BAD / total4,525 / 18,100 / 22,625Train / dev / test15,825 / 3,400 / 3,400Stealth\-defect fraction35\.1%DefectsFamilies†×\\timesfield groups7×\\times4Multi\-defect tiers \(1/2/3/4\)4,525 eachGold \{family, location\} on every BAD✓Targets coveredPersonalized \(persona\-swap slice\)✓Actionable \(gold family\+location\)✓Compositional \(cross\-field defects\)✓Evidence\-grounded \(product \+ history\)✓Validity checkL1 deterministic checks \(12/record\)99\.3% passL2 evidence provenance flag4,331 / 129L3 human\-calibration package250 itemsL4 cross\-generator slice \(3 models\)50 sharedL5 real\-assistant slice100 outputs

†PREF\_CONFLICT, INTENT\_DRIFT, XCOMP\_MISMATCH, REDUNDANCY, EVIDENCE\_MISMATCH, OVER\_PERSONALIZATION\_HALLUCINATION, UNSUPPORTED\_CLAIM\. Field groups: overview, categories, related queries, attributions\.

Table 8:Summary of thePACEShopbenchmark\. The “Targets covered” block names which of the fourPACEtargets each design choice supports; the “Validity stack” block enumerates the five verification layers that make the construction auditable\.
### B\.2Construction Pipeline Workflow

PACEShopis built from public source artefacts in a controlled, auditable pipeline\.

##### Persona construction\.

We construct personas as structured shopping contexts rather than free\-form user descriptions\. The released persona pool contains 1,200 personas, including public seed personas and behavior\-seeded synthetic personas normalized under a shared schema\. Each persona specifies stable shopper information, such as household type, budget, quality sensitivity, urgency, constraints, preferences, and brand affinities, together with concrete shopping behavior in the form of purchase history and recent searches\. This design supports two kinds of evaluation\. First, shopper constraints and preferences make persona\-conditioned correctness testable: a response can be valid for one shopper and invalid for another\. Second, purchase and search histories make persona\-history grounding auditable: a judge can check whether a response faithfully refers to the shopper’s actual history or invents unsupported behavior\.

##### Shopping\-behavior and evidence collection\.

We begin from 97,227 normalized US shopping queries and 4,460 public evidence records across 11 shopping domains\. Queries are tagged by coarse auditing metadata, including retail vertical and shopping mission, while evidence records expose product titles, brands, metadata, and available review snippets\. We retain 1,132 queries whose evidence pool contains at least three products\. This evidence\-backed filtering is deliberate: PACEShop asks whether the same shopping intent can support different valid responses under different personas, which requires enough product alternatives for shopper constraints and preferences to matter\. Each retained query is paired with four distinct personas using a coverage\-balanced assignment procedure that minimizes persona\-taxonomy reuse and prevents the same persona from repeating on a query\. This yields 4,525 query–persona pairs after validation drops\. The resulting repeated\-measures design holds the shopping query and evidence pool fixed while varying shopper context, allowing personalization failures to be isolated from changes in shopping intent\.

##### The full four\-stage workflow\.

1. 1\.Query catalog\.Normalize 97,227 US ESCI queries; tag retail vertical \(11 categories\) and shopping mission \(6 categories\)\. Detailed breakdown can be found in Appx\.[B\.3](https://arxiv.org/html/2608.26180#A2.SS3)\.
2. 2\.Evidence catalog\.Build 4,460 public evidence records across 11 shopping domains; augment 129 with review snippets\.
3. 3\.Persona pool\.Merge 200 public v1 personas with 1,000 behavior\-seeded synthetic personas; normalize under one structured schema \(household, budget, quality, urgency, constraints, preferences, brand affinities, history\)\.
4. 4\.Coverage\-balanced assignment\.Retain 1,132 queries with≥\\geq3 evidence candidates; reserve 345 holdout personas across 8 bundles for dev/test; assign 4 personas per retained query\.
5. 5\.GOOD generation\.One structured response per assignment, Claude Opus 4\.6, constrained JSON output\.
6. 6\.Normalization\.Schema validation and deterministic post\-processing\.
7. 7\.BAD construction\.Four tiers \(1/2/3/4 defects\) per GOOD, defects drawn from distinct field groups\.
8. 8\.Packaging\.Export packaged / explicit / clean JSONL views plus 250\-item human\-calibration package, 50\-input cross\-generator slice, 100\-output real\-assistant slice\.

The four substantive stages of this workflow are summarized in Tab\.[9](https://arxiv.org/html/2608.26180#A2.T9), which makes explicit how each stage’s design choice advances one or more PACE properties\.

Table 9:PACEShopconstruction pipeline\. Each stage is designed to make one or more PACE properties observable and verifiable\.

### B\.3PACEShopCoverage Details

The released persona pool, evidence catalog, and query catalog are documented below with representative examples\. The query catalog covers 11 retail verticals and 6 shopping missions; the persona pool covers 12 household types and 8 holdout bundle families; the evidence catalog spans 11 shopping domains and includes 129 review\-augmented records\.

Table 10:Query distribution across 11 retail verticals\.*Backbone*counts all unique US normalized ESCI queries;*Retained*counts the 1,132 evidence\-backed queries that pass the evidence\-availability filter \(≥\\geq3 evidence\-backed candidates\)\.Table 11:Query distribution across 6 shopping mission types\. Mission type is assigned by intent\-pattern matching against query text and top product titles/brands\.#### B\.3\.1Query Coverage

The public backbone contains 97,227 unique US\-locale normalized ESCI queries categorized along two dimensions\.

Retail verticals \(11 categories\)\.These follow standard e\-commerce product departments \(e\.g\., Amazon Browse Nodes\)\. Each query is assigned to one vertical by a keyword\-hit counting rule \(\_pick\_rule\(\)in the released code\): the assignment function concatenates the normalized query text with the top candidate product titles and brands, then counts the number of keyword hits for each vertical’s keyword list\. The vertical with the most hits wins; ties are broken by list order, and if no keyword matches the query falls back to General Merchandise\. Representative keyword→\\tovertical mappings include:

- •“shirt”, “shoes”, “backpack”→\\toApparel & Accessories
- •“usb”, “charger”, “headphones”→\\toElectronics
- •“kitchen”, “vacuum”, “air fryer”→\\toHome & Kitchen
- •“toy”, “lego”, “puzzle”→\\toToys & Games
- •“dog”, “baby”, “stroller”→\\toPet & Baby
- •“book”, “vinyl”, “dvd”→\\toBooks & Media

The full keyword lists \(10–19 terms per vertical\) are provided in the releasedv2\_taxonomy\.py\. The lists are intentionally broad rather than exhaustive: common category\-indicative terms are sufficient for coarse coverage reporting, since the taxonomy serves as an auditing tool rather than a fine\-grained classifier\.

Shopping mission types \(6 categories\)\.These capture the shopper’s task framing and follow well\-known shopping\-behavior distinctions in the e\-commerce literature: undirected browsing, repeat/replenishment purchases, problem\-driven search, gift/event shopping, compatibility/replacement needs, and feature\-constrained filtering\. Assignment uses the same\_pick\_rule\(\)keyword\-hit counting mechanism as retail verticals, applied to intent\-indicative keyword lists\. Representative keyword→\\tomission mappings include:

- •“gift”, “birthday”, “wedding”→\\toGift & Event
- •“best”, “quiet”, “waterproof”→\\toProblem Solution
- •“wireless”, “organic”, “compact”→\\toFeature Constrained
- •“compatible”, “replacement”, “adapter”→\\toCompatibility & Replacement
- •“bulk”, “refill”, “pods”→\\toReplenishment & Repeat

The fallback category \(no keyword hits\) is Broad Browse\. Full keyword lists \(7–12 terms per mission type\) are inv2\_taxonomy\.py\.

Table[10](https://arxiv.org/html/2608.26180#A2.T10)shows the backbone distribution across the 11 retail verticals together with the number of evidence\-backed queries retained after evidence filtering\. Table[11](https://arxiv.org/html/2608.26180#A2.T11)shows the corresponding distribution across the 6 shopping mission types\.

The retention rate varies across verticals and missions because it depends on evidence availability in the public ESCI product metadata rather than on intentional stratification\. For example, Beauty & Health retains 33 queries \(1\.14%\) while Toys & Games retains 235 \(1\.36%\), reflecting differences in product\-metadata completeness across categories\.

#### B\.3\.2Persona Coverage

The full released persona pool contains 1,200 personas\. Table[12](https://arxiv.org/html/2608.26180#A2.T12)summarizes the aggregate coverage across the major taxonomy axes, including household type, budget, quality preference, urgency, constraints, shopping preferences, behavioral\-history density, and holdout attributes\. Table[14](https://arxiv.org/html/2608.26180#A2.T14)then provides a compact set of representative released personas, selected to illustrate the major coverage patterns discussed in the main text without listing the full persona catalog\.

Table 12:Released persona coverage summary used inPACEShop\. Counts are over the full 1,200\-persona pool, while the rows below show representative released personas rather than the full catalog\.
#### B\.3\.3Evidence Coverage

This subsection expands the evidence side ofPACEShopin two complementary views\. Tab\.[13](https://arxiv.org/html/2608.26180#A2.T13)summarizes the domain\-level coverage of the released evidence catalog, while Tab\.[15](https://arxiv.org/html/2608.26180#A2.T15)shows representative evidence records from each domain so readers can inspect the concrete product information available to the response generator and judge\.

The full released evidence catalog contains 4,460 public evidence records spanning 11 shopping domains\. Each record exposes human\-readable product information extracted from public ESCI product listings: product title, brand, color, bullet points, and description\. Of these, 4,331 records \(97\.1%\) are*product\-listing only*, grounded solely in these catalog fields\. The remaining 129 records \(2\.9%\) are*review\-augmented*, carrying the same product\-listing fields plus public customer review snippets that provide richer grounding signals \(e\.g\., real\-user opinions on durability, fit, or usability\)\. Every record carries an explicit provenance flag so that downstream judges and analyses can condition on evidence strength\. This design keeps the grounding signal lighter than fully review\-grounded RAG benchmarks, but more transparent than a generic “grounded” claim because evidence quality is measurable per record rather than assumed uniform\. Table[13](https://arxiv.org/html/2608.26180#A2.T13)gives the aggregate domain distribution of the evidence catalog, while Table[15](https://arxiv.org/html/2608.26180#A2.T15)provides a representative released evidence record from each domain\. Together, the two tables show both the breadth of evidence coverage and the concrete product\-level information available in each per\-query evidence pool\.

Table 13:Evidence record distribution across 11 shopping domains\. Of these, 129 records \(2\.9%\) are review\-augmented; the remainder are grounded in product\-listing fields only \(title, brand, color, bullets, description\)\.Table 14:Representative released personas covering the major taxonomy values emphasized in the paper\.Table 15:Representative evidence view forPACEShop\. Each row reports one domain count and one representative released evidence record from that domain\.

### B\.4Validity Check

We do not treat the dataset as self\-validating\. Five independent layers make the construction auditable: \(i\) a 12\-check deterministic record\-level validator that every record passes through, \(ii\) explicit evidence\-provenance flags \(metadata vs\. review\-augmented\), \(iii\) a prepared 250\-item human\-calibration package, \(iv\) a cross\-generator slice on 50 shared inputs to confirm that GOOD generation does not overfit a single backbone, and \(v\) a normalized real\-assistant slice of 100 external outputs that map cleanly into the same response contract\. Of the 4,525 released GOOD records, 99\.3% pass all 12 deterministic checks; the remaining 0\.7% fail only a case\-sensitive category\-duplicate check and are flagged but retained\. Tab\.[16](https://arxiv.org/html/2608.26180#A2.T16)maps each validity layer to the specific construction risk it mitigates\.

Table 16:Threats and mitigations for benchmark validity\. The key design choice is to make each concern measurable through released artifacts rather than acknowledging it only qualitatively\.
### B\.5Defect Taxonomy

BAD records are constructed from seven defect families distributed across four response\-field groups \(overview, categories, related queries, evidence attributions\)\. Each defect family is paired with the response field whose contract it violates, so localization in the main experiment table is well posed\. Tab\.[17](https://arxiv.org/html/2608.26180#A2.T17)lists the seven families, their target field groups, and the operational definition used to inject each defect during BAD construction\.

Defect familyComponent groupGold locationWhy it mattersPREF\_CONFLICTcategoryproduct categoriesIntroduces an option that directly violates persona constraints or preferences\.INTENT\_DRIFTcategoryproduct categoriesShifts the answer away from the original shopping intent while remaining superficially plausible\.XCOMP\_MISMATCHrqListrelated queriesBreaks consistency between the overview, categories, and refinement suggestions\.REDUNDANCYrqListrelated queriesProduces repetitive or low\-diversity refinements that hurt utility\.EVIDENCE\_MISMATCHattributionevidence attributionsCites evidence that does not support the claims made in the response\.OVER\_PERS\_HALLUC\.overviewpersonalized overviewAdds user\-specific claims that are unsupported by the available evidence\.UNSUPPORTED\_CLAIMoverviewpersonalized overviewAdds a generic but unverifiable claim that lacks support in the evidence store\.Multi\-defect tier design\(defects within the same component group conflict and are never combined\):Single\-defect—1 location7 single\-defect variants, one per family\.Double\-defect—2 locations18 valid cross\-group pairs \(e\.g\., category \+ attribution\)\.Triple\-defect—3 locations20 valid cross\-group triples\.Quad\-defect—4 locations4 curated quads covering all 4 component groups\.

Table 17:Release\-level defect taxonomy and multi\-tier difficulty design used in the fullPACEShopbenchmark\. Each BAD example preserves the original query and persona while injecting one to four defects from distinct component groups\. The tiered design creates a measurable difficulty gradient—single\-defect variants test basic fault detection, while quad\-defect variants require simultaneous identification of failures across all four response components\.
### B\.6An Example Record ofPACEJudgeDataset

##### Benchmark objects\.

InPACEShop, the candidate response is instantiated as a structured object

y=\(yov,ycat,yrq,yattr\),y=\(y^\{\\mathrm\{ov\}\},y^\{\\mathrm\{cat\}\},y^\{\\mathrm\{rq\}\},y^\{\\mathrm\{attr\}\}\),whereyovy^\{\\mathrm\{ov\}\}is the overview,ycaty^\{\\mathrm\{cat\}\}is the product\-category list,yrqy^\{\\mathrm\{rq\}\}is the related\-query list, andyattry^\{\\mathrm\{attr\}\}is the evidence list\. The personappcontains shopper context, including household profile, budget, quality sensitivity, urgency, constraints, preferences, brand affinities, purchase history, recent searches, and a short context summary\. The evidence poolEEcontains product\-level evidence snippets such as titles, brands, bullets, descriptions, and, when available, review snippets; each snippet has an evidence ID so that attribution claims can be checked against the released pool\. For paper\-facing presentation, hard constraints and soft preferences are merged into a single preference view, while the released JSON preserves the original fields for reproducibility\. Fig\.[5](https://arxiv.org/html/2608.26180#A2.F5)in Appx\.[B\.6](https://arxiv.org/html/2608.26180#A2.SS6)illustrates one concrete PACEShop record; full persona and evidence JSON examples are provided in the same appendix section\.

InputQuery:tactical duffle grayPersona\(structured JSON object\):profile:household = family with school\-age kidspreferences:avoid peanuts; quiet operation;easy to clean; durable for daily usebrand\_likes:\[Ecco Press, Lansinoh\]shopping:budget = mid\-range; quality = balanced;shipping = standardhistory:purchases = \[Barack Before Obama…,Lansinoh Storage Bags…\]recent\_searches = \[…alternatives, …upgrades\]context:“family with school\-age kids; mid\-range budget; balanced quality; recent purchases include Barack Before Obama, Lansinoh Storage Bags\.”OutputOverview:“For a durable, travel\-friendly gray tactical duffle that handles daily family use, exploreTactical,Gray Gym Bags,Rolling Duffles,Travel Duffles, andRange Bags…”Category\(4–5\): Tactical; Gray Gym Bags;Rolling Duffles; Travel Duffles; Range BagsRelated queries\(5\):tactical duffle with MOLLE webbing;gray duffel bag water resistant;wheeled tactical duffel carry\-on;packable lightweight gray duffle;tactical range bag gunmetal greyAttributes\(8 cited products\):Nexpak 26” Tactical Duffle Military Molle;East West U\.S\.A Tactical Outdoor Duffel;Under Armour Contain 4\.0 Duffle Bag;Gonex 80L Packable Travel Duffle;Samsonite Andante 2 Wheeled Rolling Duffel;OSAGE RIVER Tactical Range Bag;Amazon Basics Large Travel Duffel;Vertx Contingency Duffel 85L

Figure 5:Worked example of aPACEShoprecord \(full\-good\-00000\)\. Each example pairs a shopping query with a structured persona, an auditable evidence pool, and a four\-field shopping response\. The input pairs a shopping query with a structured persona object; the output is a heterogeneous response with four evaluation\-facing components\. Correctness depends jointly on the persona, all components, and the cited evidence\. Attributes are shown as short product descriptions; the underlying JSON carries evidence IDs\. BAD records additionally expose gold defect family and location labels, enabling diagnostic evaluation rather than only scalar scoring\.
##### Worked Example of a Dataset Record\.

The boxed schema below summarizes the abstractPACEShoprecord schema: every record pairs a query with a structured persona, an auditable evidence pool, and a four\-field shopping response, and BAD records additionally expose gold defect family and location labels\. Fig\.[5](https://arxiv.org/html/2608.26180#A2.F5)then shows a single concrete record end to end: the persona\-conditioned query on the left, the structured four\-field response on the right\. Correctness on this record requires joint judgements – persona consistency, cross\-field coherence, evidence support, and schema validity – which the fourPACEtargets operationalize\.

### B\.7Concrete Persona\-JSON Input Example

The\[PERSONA\_JSON\]slot of the generation prompt \(Sec\.[B\.9](https://arxiv.org/html/2608.26180#A2.SS9)\) receives the full persona record, including household profile, preferences \(merged hard \+ soft\), brand affinities, shopping context, purchase history, recent searches, and a natural\-language context summary\. Below is a representative example \(truncated for space; the published JSON preserveshard\_constraintsandsoft\_preferencesas separate lists for reproducibility\):

```
{
  "persona_id": "persona-0045",
  "profile": {
    "household": "family with school-age kids",
    "demographics": "balances durability and value"
  },
  "preferences": {
    "hard_constraints": ["avoid peanuts in ingredients",
                         "quiet operation required"],
    "soft_preferences": ["easy to clean",
                         "durable enough for daily use"],
    "brand_likes": ["Ecco Press", "Lansinoh"],
    "brand_avoids": [],
    "attribute_preferences": ["travel friendly",
                              "low-maintenance finishes"]
  },
  "shopping_context": {
    "budget": "mid-range",
    "quality_sensitivity": "balanced",
    "shipping_urgency": "standard shipping"
  },
  "history": {
    "purchase_history": [
      {"asin": "0063028743",
       "product_name": "Barack Before Obama: Life Before
                        the Presidency",
       "brand": "Ecco Press"},
      {"asin": "B006XISCNA",
       "product_name": "Lansinoh Breastmilk Storage Bags,
                        100 Count",
       "brand": "Lansinoh"}
    ],
    "recent_searches": [
      "Barack Before Obama alternatives",
      "Lansinoh upgrades"
    ]
  },
  "context_summary": "This shopper is part of a family with
    school-age kids who balances durability and value on a
    mid-range budget. They prioritize travel-friendly,
    low-maintenance products..."
}
```

### B\.8Concrete Evidence\-JSON Input Example

The\[EVIDENCE\_JSON\]slot receives up to 24 evidence snippets drawn from the public evidence catalog\. Each snippet carries a product identifier, title, brand, a unique evidence ID, the text snippet, and the source field\. Below is a representative excerpt; paper\-facing tables show these as short product descriptions \(“Nexpak 26” Tactical Duffle”\) rather than raw IDs\.

```
[
  {"canonical_product_id": "B079YYGMR3",
   "title": "26\" Tactical Duffle Military Molle Gear
             Shoulder Strap Range Bag TF126 GMG
             Gunmetal Grey",
   "brand": "Nexpak",
   "evidence_id": "M:B079YYGMR3:0",
   "snippet": "26\" Tactical Duffle Military Molle Gear
               Shoulder Strap Range Bag TF126 GMG
               Gunmetal Grey",
   "source_field": "product_title"},
  {"canonical_product_id": "B079YYGMR3",
   "title": "26\" Tactical Duffle Military Molle Gear ...",
   "brand": "Nexpak",
   "evidence_id": "M:B079YYGMR3:1",
   "snippet": "Nexpak",
   "source_field": "product_brand"},
  {"canonical_product_id": "B077KKMFFN",
   "title": "Under Armour Adult Contain 4.0 Duffle Bag,
             Graphite Medium Heat (040)/Black,
             One Size Fits All",
   "brand": "Under Armour",
   "evidence_id": "M:B077KKMFFN:0",
   "snippet": "Under Armour Adult Contain 4.0 Duffle Bag,
               Graphite Medium Heat (040)/Black,
               One Size Fits All",
   "source_field": "product_title"}
]
```

### B\.9PACEShopBenchmark record Generation Prompt

PACEShopconstruction has two stages, and only the first stage uses an LLM\.*GOOD records*are generated by a fixed LLM \(Claude Opus 4\.6\) under the constrained response\-generation prompt reproduced below\.*BAD records*are then derived from each accepted GOOD record by deterministic rule\-based defect injection in Python rather than by any second LLM prompt; this design choice is what guarantees that every BAD record carries an auditable gold defect family and gold defect location label by construction\. We document both stages in this subsection so the appendix is self\-contained\.

##### GOOD generation: LLM under a constrained JSON contract\.

For each query–persona pair, the generator is invoked once with a constrained JSON contract\. The prompt enforces the four\-field response schema \(overview, bolded category list, related queries, evidence attributions\), the cardinality and length budgets used in main\-text §[3\.2](https://arxiv.org/html/2608.26180#S3.SS2), and a strict no\-extras rule on the list fields\. Crucially, the generator is never told what defect families exist: it only sees the persona and the candidate evidence pool, so accepted GOOD responses are contract\-valid anchors rather than negative examples for the judge to learn against\. The prompt’s slot syntax \(\[ASSIGNMENT\_ID\],\[QUERY\],\[PERSONA\_JSON\],\[GROUP\_JSON\],\[EVIDENCE\_JSON\]\) matches the released runner\.

```
You are generating a PACEShop V2 benchmark response.
Return JSON only with keys: overview_text, bold_categories,
productCategoryList, category_search_queries, rqList, attribution_list.

Requirements:
- Overview <= 350 chars.
- Include exactly 4 or 5 bold categories using **...** in overview_text.
- productCategoryList must be an array of plain strings aligned with
  bold categories.
- Each category label <= 15 chars.
- rqList must contain exactly 5 short related shopping refinements.
- attribution_list must be an array of plain evidence ID strings only.
- category_search_queries must be an array of plain strings aligned 1:1
  with productCategoryList.
- Do not return products, scores, or citation objects inside list fields.
- Keep the response personalized to the provided persona and compatible
  with the query intent.

Assignment: [ASSIGNMENT_ID]
Query: [QUERY]
Persona JSON: [PERSONA_JSON]
Support groups: [GROUP_JSON]
Candidate evidence: [EVIDENCE_JSON]
```

Generated GOOD candidates pass through deterministic normalization and the L1 12\-check validator \(§[B\.4](https://arxiv.org/html/2608.26180#A2.SS4)\) before any BAD variant is derived; candidates that fail hard schema, evidence\-ID, or XML round\-trip checks are rejected\.

##### BAD construction: deterministic rule\-based defect injection\.

We deliberately do*not*use an LLM prompt to generate BAD records\. Each BAD record is produced by a deterministic Python edit on a validated GOOD record drawn from the seven\-family defect taxonomy in Tab\.[17](https://arxiv.org/html/2608.26180#A2.T17)\(Sec\.[B\.5](https://arxiv.org/html/2608.26180#A2.SS5)\)\. For every defect familyd∈𝒟d\\in\\mathcal\{D\}the edit is fixed in advance, so the gold defect familyddand gold defect locationr∈ℛr\\in\\mathcal\{R\}are known by construction rather than being predicted by another model:

- •PREF\_CONFLICT: replace the first product category with a label that contradicts the persona’s first hard constraint \(e\.g\.,*leather handbags*for a persona withavoid\_leather\)\. Location:productCategoryList\[0\]\.
- •OVER\_PERSONALIZATION\_HALLUCINATION: prepend a fabricated persona\-history reference \(e\.g\., “Given their Dyson Airwrap, …”\) to the overview\. Location:overview\_text\.
- •XCOMP\_MISMATCH: replace the related\-query list with the related\-query list of an unrelated GOOD record\. Location:rqList\.
- •INTENT\_DRIFT: replace the first product category with an unrelated category drawn from a fixed off\-intent pool\. Location:productCategoryList\[0\]\.
- •REDUNDANCY: duplicate the first related query into the second slot\. Location:rqList\[1\]\.
- •EVIDENCE\_MISMATCH: replace the attribution list with the attribution list of an unrelated GOOD record\. Location:attribution\_list\.
- •UNSUPPORTED\_CLAIM: append a fabricated product\-feature claim \(e\.g\., “It also highlights solar charging support\.”\) to the overview\. Location:overview\_text\.

Multi\-defect variants \(tiers 2–4\) are produced by composing edits from distinct response\-field groups \(*categories*,*related queries*,*attributions*,*overview*\) under the canonical group order*categories→\\rightarrowrelated queries→\\rightarrowattributions→\\rightarrowoverview*, which avoids interactions between overview\-modifying edits and category\-modifying edits \(e\.g\., bolded\-category alignment\)\. Combinations spanning two defects from the same field group are disallowed; the full set of admissible single\- and multi\-defect combinations is enumerated inpmsb\.faultsof the released benchmark code\.

Because every BAD edit is a deterministic function of \(i\) the source GOOD record, \(ii\) the assigned persona, and \(iii\) the defect family, BAD records do not require a separate LLM\-judging\-style validator on top of the L1 12\-check audit\. The injection produces both the modified four\-field response and the gold location string in a single step, so any disagreement between a judge’s predicted \(defect family, defect location\) and the gold pair in Tabs\.[3](https://arxiv.org/html/2608.26180#S4.T3)and[4](https://arxiv.org/html/2608.26180#S5.T4)is unambiguously a judge error rather than a labeling error\.

## Appendix CPrior Evaluation Protocols and Baseline Adaptation

This appendix documents how prior personalization, RAG, and judging protocols relate to thePACEShopevaluation schema\. We first describe, in a single unified support table, which scenarios each prior judging protocol natively covers and which scenarios each running baseline scores or abstains on \(Sec\.[C\.1](https://arxiv.org/html/2608.26180#A3.SS1)\)\. We then audit target leakage for the three single\-target adapted baselines on the same held\-out test set \(Sec\.[C\.2](https://arxiv.org/html/2608.26180#A3.SS2)\)\. Finally, we list the baseline prompt templates run on the shared backbone pool \(Sec\.[C\.3](https://arxiv.org/html/2608.26180#A3.SS3)\)\. Together these subsections explain why the “–” cells in the main result tables are deliberate and measurement\-backed, not notational laziness\.

### C\.1Native and Adapted Scenario Support

Prior personalization, RAG, and judging protocols differ in what they natively evaluate\. A personalization benchmark may test whether a model adapts to user history without defining defect\-family labels; a RAG evaluator may check product\-evidence faithfulness without evaluating invented shopper history; and a scalar judge may estimate overall quality without localizing the failed response field\. Tab\.[18](https://arxiv.org/html/2608.26180#A3.T18)summarizes this in two blocks: the upper block lists prior protocols that we do*not*directly run asPACEShopjudges and marks the targets they natively cover with “∂\\partial” or “–”; the lower block lists the baselines we*do*run on the shared backbone pool used in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)– four multi\-target rubric ablations of thePACEJudgeoutput schema and three single\-target adapted baselines \(ARES\-/PersonaLens\-/EtaPP\-style\) that score only their native target and report “–” elsewhere\. Unsupported scenarios are reported as “–” rather than forced into thePACEShopoutput schema\.

Method / native protocolNative signal or role in studyS1 P\-sourceS2 C detect\+locS3 E product/historyS4 A family\+locWhy this row scores or abstains*Prior protocols not directly run asPACEShopjudges \(reference\)*LaMP[13](https://arxiv.org/html/2608.26180#bib.bib3)Task metrics for personalized generation∂\\partial–––No unchanged\-response persona swap or defect\-source label\.PersonaLens[18](https://arxiv.org/html/2608.26180#bib.bib4)LLM user and judging agents for assistants∂\\partial–––Evaluates user\-conditioned success, not four\-field response diagnosis\.PrefEval[17](https://arxiv.org/html/2608.26180#bib.bib5)Preference following/classification∂\\partial–––Persona signal is present, but no evidence pool or response\-field location\.PersonaMem[6](https://arxiv.org/html/2608.26180#bib.bib6)Response selection over user histories∂\\partial–––History use is measured, not defect family and location\.LikeBench[10](https://arxiv.org/html/2608.26180#bib.bib7)Simulated\-user likability diagnostics∂\\partial–––Subjective user\-fit signal; noPACEShopgrounding or localization contract\.EtaPP[5](https://arxiv.org/html/2608.26180#bib.bib8)Key\-point judging for personalized tool use∂\\partial––∂\\partialChecks personalized task points, but notPACEShopsource\-family/location labels\.Personalized judge[2](https://arxiv.org/html/2608.26180#bib.bib9)User\-conditioned judging preference∂\\partial–––Personalizes a judging preference, but does not define multi\-field defect localization\.ECom\-Bench[15](https://arxiv.org/html/2608.26180#bib.bib2)Task\-success trajectories in e\-commerce support∂\\partial–––Realistic commerce tasks, but not fixed\-response judge diagnosis\.Search Arena[9](https://arxiv.org/html/2608.26180#bib.bib1)Human preferences for search\-augmented chat––∂\\partial–Search/citation quality is relevant, but no persona\-history grounding label\.ARES[12](https://arxiv.org/html/2608.26180#bib.bib10)RAG faithfulness to retrieved evidence––∂\\partial–Covers product\-evidence grounding only, not persona or multi\-field diagnosis\.G\-Eval / MT\-Bench[8](https://arxiv.org/html/2608.26180#bib.bib11);[19](https://arxiv.org/html/2608.26180#bib.bib12)Scalar or pairwise quality judgments––––Runnable only after adapting the prompt to emit thePACEShopschema \(see below\)\.*Running baselines on the shared backbone pool \(used in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\)*G\-Eval\-style rubric[8](https://arxiv.org/html/2608.26180#bib.bib11)Multi\-target rubric ablation, persona\-aware✓✓✓✓Adapted to thePACEJudgeoutput schema; differs only in instruction\.MT\-Bench\-style scalar[19](https://arxiv.org/html/2608.26180#bib.bib12)Multi\-target rubric ablation, scalar\-first✓✓✓✓Adapted to thePACEJudgeoutput schema; back\-fills structured fields after a scalar judgment\.Persona\-blindMulti\-target rubric ablation, persona withheld–✓✓✓Persona payload replaced by\{"omitted": true\}; isolates the marginal value of persona conditioning\.Persona\-aware genericMulti\-target rubric ablation, persona\-aware✓✓✓✓Shared protocol for the backbone comparison without explicit counterfactual reasoning\.ARES\-styleSingle\-target adapted: evidence \(E\) only––✓–Native protocol scores RAG faithfulness only; non\-E targets reported as “–”\.PersonaLens\-styleSingle\-target adapted: persona \(P\) only✓–––Native protocol scores personalization only; non\-P targets reported as “–”\.EtaPP\-styleSingle\-target adapted: actionable \(A\) only–––✓Native protocol scores key\-point matches only; non\-A targets reported as “–”\.PACEJudge\(ours\)StructuredPACEjudging contract✓✓✓✓Verdict, target scores, defect family, location, and evidence IDs\.

Table 18:Native and adapted support for the fourPACEtarget scenarios\. The first block lists prior protocols that are*not directly run*asPACEShopjudges; “∂\\partial” marks a related signal that does not define thePACEShopclosure metric or the required diagnostic output fields\. The second block lists the baselines we*do run*on the shared backbone pool in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3): four multi\-target rubric ablations of thePACEJudgeoutput schema and three single\-target adapted baselines \(ARES\-/PersonaLens\-/EtaPP\-style\) that score only their native target and report “–” on the others\. Tab\.[19](https://arxiv.org/html/2608.26180#A3.T19)expands the native protocol mapping for the single\-target adapted baselines, and Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)audits their off\-target leakage\.
### C\.2Native Judging\-Protocol Map and Evaluation\-Target Leakage Audit

The strict paired\-counterfactual main table \(Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)\) is supported by two audit tables kept here rather than in the main text\. The native judging\-protocol map \(Tab\.[19](https://arxiv.org/html/2608.26180#A3.T19)\) records, for each adapted single\-target baseline, whichPACEtarget its native judging protocol covers and why the other targets are reported as “–” rather than over\-adapted\. The target\-leakage audit \(Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)\) then measures, on the same 3,400\-record held\-out test set, the rate at which each adapted baseline emits a defect\-family label outside its native target \(e\.g\., ARES\-style firingPREF\_CONFLICTon a record whose gold defect is compositional\)\. ARES\-style and PersonaLens\-style stay under11–2%2\\%on the*Any off\-target*aggregate, validating their honest restriction in Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)even though they show non\-trivial target leakage on individual P/C cells; EtaPP\-style leaks heavily \(99% any\-off\-target\), and we therefore do not interpret its S4 cell as evidence that EtaPP\-style covers more than the A target\. Together these two tables protect Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)from the failure mode of “apply a prompt, get a number on every cell”: the “–” restrictions are deliberate and supported by measurement, not notational laziness\.

Table 19:Native judging\-protocol toPACEtarget mapping\.Each adapted baseline is restricted to thePACEtarget that its native judging protocol scores; unsupported targets are reported as “–”\. Table[4](https://arxiv.org/html/2608.26180#S5.T4)uses this mapping when reporting scenario scores: prior protocols cover at most one target each, so the joint S1–S4 PACE average is undefined for them\. OnlyPACEJudgehas an output contract supporting all fourPACEtargets\.Table 20:Evaluation\-target leakage audit\.For each adapted single\-target baseline \(Sonnet 4\.6 on the 3,400\-record held\-out test set\), this table reports the rate at which the baseline emits a defect\-family label outside its nativePACEtarget\. A genuine single\-target baseline should score near 0% on every non\-native target; high leakage means the prompt is over\-adapted\. ThePACEJudgereference row reports family misclassification rather than leakage, sincePACEJudgesupports allPACEtargets\. This audit is the safeguard against “apply a prompt, get a number on every cell”: cells in Table[4](https://arxiv.org/html/2608.26180#S5.T4)are non\-zero for a single\-target baseline only on its native target\.
### C\.3Baseline Prompt Templates

We organize the baselines into two groups\. The first group is a set of*multi\-target rubric ablations*that share thePACEJudgeinput bundle \(query, persona JSON, response XML, structured response JSON, candidate evidence IDs, candidate evidence preview, deterministic checks\) and the same structured output contract, differing only in*instruction*and*persona visibility*\. The second group is a set of*single\-target adapted baselines*\(ARES\-style, PersonaLens\-style, EtaPP\-style\) whose native judging protocols cover only onePACEtarget each; we adapt them faithfully on their native target and abstain \(“–”\) on the other targets, as documented in Tab\.[19](https://arxiv.org/html/2608.26180#A3.T19)and audited in Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)\.

#### C\.3\.1Multi\-target rubric ablations

The rubric ablations all emit the fullPACEJudgeoutput schema \(verdict, P/C/E/F axis scores, defect family, defect location, supporting evidence IDs, confidence, rationale\) so that metrics across rows of Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)differ only by the prompt instruction\.

G\-Eval\-style rubric judge[8](https://arxiv.org/html/2608.26180#bib.bib11)\.“You are a rubric\-guided judge in the style of a generic high\-quality LLM evaluator\. Score the response using a compact rubric over relevance, helpfulness, coherence, grounding, and structure\. Then translate that rubric judgment into the required output schema\.”

Scalar\-first judge \(MT\-Bench style\)[19](https://arxiv.org/html/2608.26180#bib.bib12)\.“You are a scalar\-first judge\. Base the decision mainly on one overall quality impression, then back\-fill the required structured fields in a best\-effort way\.”

Persona\-blind ablation\.“You are a generic shopping\-response judge\. The persona is intentionally withheld for this baseline\. Judge only whether the response seems broadly useful, coherent, grounded, and structurally valid\.” The persona payload is serialized as\{"omitted": true\}\.

Persona\-aware generic rubric\.“You are a persona\-aware but otherwise generic shopping\-response judge\. Use the persona and query to judge whether the response is suitable for the user\. Do not use explicit counterfactual swap reasoning and do not invent additional defect classes beyond the provided labels\.”

#### C\.3\.2Single\-target adapted baselines

ARES\-style, PersonaLens\-style, and EtaPP\-style do not share thePACEJudgefour\-target output schema in their native form\. We therefore adapt each one only on thePACEtarget that its native judging protocol scores, leaving cells outside the native target as “–” in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)and Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)\. To keep the adaptation auditable, all three adapted baselines run on the same 3,400\-record held\-out test set and the same shared backbone pool as the rubric ablations; only the*native\-target*cells are scored against thePACEJudgeschema, and an evaluation\-target leakage audit \(Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)\) measures the rate at which each baseline emits a defect family outside its native target\.

ARES\-style grounding judge[12](https://arxiv.org/html/2608.26180#bib.bib10)\.The native ARES protocol is a RAG\-faithfulness judge built from three lightweight LM scorers:*context relevance*,*answer faithfulness*, and*answer relevance*\. We instantiate this protocol onPACEShopby mapping the three ARES axes onto\(q,y,E\)\(q,y,E\): context relevance asks whether the candidate evidence IDs inEEare related toqq; answer faithfulness asks whether claims inyyare supported byEE, including no fabricated evidence IDs and no invented persona\-history claims; answer relevance asks whetheryyanswersqq\. The judge averages the three sub\-judgments into the evidence\-grounding axissE∈\[1,5\]s\_\{\\mathrm\{E\}\}\\in\[1,5\]and emitsℓ^=BAD\\hat\{\\ell\}=\\textsc\{BAD\}only whensE<3\.0s\_\{\\mathrm\{E\}\}<3\.0\. When BAD, the predicted defect familyd^\\hat\{d\}is restricted to evidence\-target labels \{EVIDENCE\_MISMATCH,OVER\_PERSONALIZATION\_HALLUCINATION,UNSUPPORTED\_CLAIM\}, andr^\\hat\{r\}is the response field carrying the unsupported claim or invalid attribution\. For any non\-evidence defect, the prompt explicitly returnsNONE/NONErather than fabricating a label\. The persona, compositional, and format/safety axes\{sP,sC,sF\}\\\{s\_\{\\mathrm\{P\}\},s\_\{\\mathrm\{C\}\},s\_\{\\mathrm\{F\}\}\\\}are pinned to a neutral3\.03\.0because ARES does not natively score them, and supporting evidence IDs are constrained toS^⊆E\\hat\{S\}\\subseteq E\. Consequently, this baseline reports a measured number on the S3 column of Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)and S3E\{\}\_\{\\text\{E\}\}in Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4), and reports “–” on S1, S2, and S4\. The leakage audit in Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)confirms that this restriction is honored in practice: ARES\-style fires a non\-evidence defect family on under2%2\\%of off\-target records on the*Any off\-target*aggregate\.

PersonaLens\-style personalization judge[18](https://arxiv.org/html/2608.26180#bib.bib4)\.The native PersonaLens judging protocol is a three\-axis judge that scores*personalization*,*response quality*, and*task success*for user\-conditioned generation\. We adapt it toPACEShopby walking through the structured persona one slot at a time—hard constraints, brand likes/avoids, attribute preferences, and budget—and asking the judge to mark each slot as honored, contradicted, or unconstrained by the response fields; this drives the persona\-alignment axissP∈\[1,5\]s\_\{\\mathrm\{P\}\}\\in\[1,5\]\. Response quality scores the format/safety axissFs\_\{\\mathrm\{F\}\}, and task success is folded into the verdict via the thresholdsP<3\.0s\_\{\\mathrm\{P\}\}<3\.0\. When BAD, the predicted defect familyd^\\hat\{d\}is restricted to persona\-target labels:PREF\_CONFLICTfor an explicit persona conflict, e\.g\., the response bolds a brand the persona avoids, orOVER\_PERSONALIZATION\_HALLUCINATIONwhen the response invents a purchase or search not in the persona history;r^\\hat\{r\}is the response field carrying the conflict\. The compositional and evidence axes\{sC,sE\}\\\{s\_\{\\mathrm\{C\}\},s\_\{\\mathrm\{E\}\}\\\}are pinned at3\.03\.0,S^\\hat\{S\}may be empty, and any non\-persona defect is reported asNONE/NONE\. This baseline therefore contributes a measured number on the S1 column, S1P\{\}\_\{\\text\{P\}\}in Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4), and abstains elsewhere\. The evaluation\-target leakage audit in Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)shows that PersonaLens\-style fires a non\-persona family on under1%1\\%of off\-target records on the*Any off\-target*aggregate\.

EtaPP\-style actionable key\-point judge[5](https://arxiv.org/html/2608.26180#bib.bib8)\.The native EtaPP judging protocol judges a personalized action trace against human\-annotated*key\-points*that the trace must satisfy\. We adapt this toPACEShopby treating the gold defect familyddand the gold defect locationrras the two key\-points the judge must name: the candidate responseyyreplaces the action trace, the deterministic checks plus a brief read of the four response fields drive the GOOD/BAD verdict, and when BAD the judge proposes bothd^∈𝒟\\hat\{d\}\\in\\mathcal\{D\}andr^∈ℛ\\hat\{r\}\\in\\mathcal\{R\}from the full label space\. Following the EtaPP convention, the target score on the predicted target is fixed at3\.03\.0rather than emitted by the judge;sPs\_\{\\mathrm\{P\}\},sCs\_\{\\mathrm\{C\}\},sEs\_\{\\mathrm\{E\}\}are likewise pinned at3\.03\.0, whilesFs\_\{\\mathrm\{F\}\}is scored honestly from schema validity andS^\\hat\{S\}may be empty\. Because EtaPP’s native judging protocol does not score persona, compositional, or evidence\-grounding targets separately, this baseline reports measured S4 family/location numbers in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\(and S4A\{\}\_\{\\text\{A\}\}in Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)\) and “–” on S1–S3\. Two caveats follow from the audit tables\. First, the fixed3\.03\.0target score is why EtaPP\-style fails the strict target\-triage conjunct in Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)more often than its S4 family number alone would suggest\. Second, the evaluation\-target leakage audit \(Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)\) shows that EtaPP\-style emits*some*defect family on99%99\\%of off\-target records, so we do not interpret its S4 cell as evidence that EtaPP\-style covers more than the A target\.

The full Markdown prompt files for all baselines, including the three adapted single\-target baselines, are shipped with the code release\.

## Appendix DPACEJudgeEvaluation

This appendix expands the main\-paper subsection on*Coverage of thePACETargets*\(Sec\.[4\.2](https://arxiv.org/html/2608.26180#S4.SS2)\) and the structured output contract used byPACEJudge\. It is organized as follows\. Sec\.[D\.1](https://arxiv.org/html/2608.26180#A4.SS1)gives a per\-target walkthrough mapping eachPACEJudgeoutput field to one of the fourPACEtargets and to the corresponding evaluation scenario\. Sec\.[D\.2](https://arxiv.org/html/2608.26180#A4.SS2)expands the four main\-body axes into the fuller six\-dimension rubric used during judge development\. Sec\.[D\.3](https://arxiv.org/html/2608.26180#A4.SS3)documents the structured fault\-localization protocol that turns the judge from a scalar scorer into a diagnostic tool\. Readers who only need the headline mapping can rely on Tab\.[2](https://arxiv.org/html/2608.26180#S3.T2)and the main\-paper subsection; the subsections below are intended for readers who want the full coverage argument and the dimension\-by\-dimension protocol\.

### D\.1How the Protocol Closes the FourPACETargets

Tab\.[2](https://arxiv.org/html/2608.26180#S3.T2)gives the high\-level mapping fromPACEtargets toPACEShopdesign choices,PACEJudgeoutput fields, and scenario\-level metrics\. The role ofPACEJudgeis to make the corresponding diagnosis reportable:PACEShopmakes each target observable, whilePACEJudgegives the judge explicit fields for reporting it\.

- •Personalized\(sP→S​1\)\(s\_\{\\mathrm\{P\}\}\\rightarrow S1\)\. The persona\-alignment score checks whetheryyrespects the shopper context inpp, including constraints, preferences, brand affinities, budget, purchase history, and recent searches\. This closes the Personalized target by forcing the verdict to condition onpprather than treating persona as optional context\. It directly supports the S1 persona\-swap evaluation\.
- •Actionable\(aA→S​4\)\(a\_\{\\mathrm\{A\}\}\\rightarrow S4\)\. The actionable diagnosis object requires the judge to report what failed and where:d^\\hat\{d\}identifies the defect family,r^\\hat\{r\}identifies the response field,S^\\hat\{S\}provides evidence support,c^\\hat\{c\}indicates confidence, andz^\\hat\{z\}explains the decision\. This closes the Actionable target by turning the judge from a scalar scorer into a diagnostic tool\. It directly supports the S4 defect\-family and field\-location evaluation\.
- •Compositional\(sC,r^→S2\)\(s\_\{\\mathrm\{C\}\},\\hat\{r\}\\rightarrow S2\)\. The compositional\-consistency score evaluatesyyas a four\-field object rather than as independent text\. The overview should agree with the category list, related queries should refine the represented product facets rather than drift away from the intent, and attributions should support the claims they are attached to\. Together with the location fieldr^\\hat\{r\}, this closes the Compositional target and supports S2 cross\-field detection and localization\.
- •Evidence\-grounded\(sE,S^→S3\)\(s\_\{\\mathrm\{E\}\},\\hat\{S\}\\rightarrow S3\)\. The evidence\-grounding score verifies both product grounding and persona\-history grounding\. Product grounding requires attribution IDs to exist inEEand support the relevant claims\. Persona\-history grounding requires the response not to invent purchases, searches, or user\-specific facts absent frompp\. Together with the supporting\-evidence fieldS^\\hat\{S\}, this closes the Evidence\-grounded target and supports S3 product\- and history\-grounding evaluation\.

The auxiliary format/safety scoresFs\_\{\\mathrm\{F\}\}checks whetheryysatisfies the response contract, including count constraints, length limits, banned\-string filters, evidence\-ID formatting, and XML round\-trip validity\. This separates structural compliance from the fourPACEtargets: a response may be well\-formed but semantically invalid, or semantically plausible but structurally unusable\.

This output contract explains where scalar judges fail and wherePACEJudgecan be evaluated more precisely\. A scalar judge may call a response “mostly helpful,” but it has no required slot for naming the broken field, the defect type, or the evidence supporting the diagnosis\.PACEJudgeexposes these missing fields directly: the P/C/E target scores identify which quality dimension degraded, the Actionable fields identify what failed and where, and the evidence\-ID constraint prevents the judge from inventing its own support\. As a result, the protocol can be evaluated not only by GOOD/BAD accuracy, but also by the scenario\-aligned metrics in Sec\.[5](https://arxiv.org/html/2608.26180#S5): persona\-swap accuracy, cross\-field localization, grounding detection, hallucinated\-ID rate, and defect\-family/location accuracy\.

### D\.2DetailedPACEJudgeEvaluation Dimensions

The four main\-body axes \(§[4](https://arxiv.org/html/2608.26180#S4)\) collapse a richer six\-dimension protocol that we used during the judge development\. Each dimension below states \(i\) the challenge that motivates it, \(ii\) howPACEJudgeoperationalizes it, and \(iii\) the metrics used to measure it\. Every dimension admits both deterministic checks and LLM\-scored assessments; deterministic checks provide a cost\-free lower bound, while LLM scoring captures semantic nuances that rules cannot\.

D1: Persona Alignment\.*Challenge\.*The central gap in personalization evaluation is distinguishing responses that are*generically helpful*from those that are*correct for a specific user*[18](https://arxiv.org/html/2608.26180#bib.bib4);[17](https://arxiv.org/html/2608.26180#bib.bib5);[6](https://arxiv.org/html/2608.26180#bib.bib6);[10](https://arxiv.org/html/2608.26180#bib.bib7)\. A response can score highly on fluency and relevance yet still ignore the user’s budget constraint, violate a hard preference, or recommend products misaligned with the user’s shopping journey stage\.*Approach\.*PACEJudgeevaluates persona alignment through*counterfactual persona swaps*\(S1 in §[5\.2](https://arxiv.org/html/2608.26180#S5.SS2)\): if swapping the persona changes which response dimensions should differ, the judge must detect that change; dimensions that should remain invariant under the swap must stay stable\. Formally, letArel​\(q,p\)A\_\{\\mathrm\{rel\}\}\(q,p\)denote dimensions that should change under a persona swap andAinv​\(q,p\)A\_\{\\mathrm\{inv\}\}\(q,p\)denote dimensions that should remain invariant\. The persona\-swap score isSswap=∑a∈Arelwa​Δa−λ​∑a∈Ainvwa​ΔaS\_\{\\mathrm\{swap\}\}=\\sum\_\{a\\in A\_\{\\mathrm\{rel\}\}\}w\_\{a\}\\Delta\_\{a\}\-\\lambda\\sum\_\{a\\in A\_\{\\mathrm\{inv\}\}\}w\_\{a\}\\Delta\_\{a\}, whereΔa\\Delta\_\{a\}measures the distance between axis signatures of paired responses\. We also define the alignment gapG⁡\(q,p\)=Hgen​\(f⁡\(q,∅\)\)−Hpers​\(f⁡\(q,p\)\)G\(q,p\)=H\_\{\\mathrm\{gen\}\}\(f\(q,\\varnothing\)\)\-H\_\{\\mathrm\{pers\}\}\(f\(q,p\)\), which quantifies the extent to which persona conditioning changes the evaluation outcome\.*Metrics\.*Persona\-alignment score \(1–5, LLM\-scored\); GOOD–BAD score gap on D1 \(larger gap==stronger persona discrimination\)\. A*negative*gap indicates the judge cannot distinguish persona\-conditioned quality—the signature of a persona\-blind evaluation\.

D2: Relevancy & Helpfulness\.*Challenge\.*A shopping assistant must match both explicit requirements \(brand, price, features\) and implicit expectations \(use case, quality tier, journey stage\)[5](https://arxiv.org/html/2608.26180#bib.bib8);[9](https://arxiv.org/html/2608.26180#bib.bib1)\. Generic “helpfulness” ratings conflate these with fluency\.*Approach\.*The judge scores each component on product relevancy, intent matching \(explicit \+ implicit\), and actionable decision support—whether the overview provides real buying advice rather than generic category listing\.*Metrics\.*Per\-section helpfulness score \(1–5\); binary criteria for product relevancy, intent matching, and helpfulness\.

D3: Diversity & Distinctiveness\.*Challenge\.*Responses that are relevant but homogeneous—e\.g\., recommending five near\-identical products or generating overlapping category groups—fail to help users compare options\.*Approach\.*The judge checks whether products, categories, and related queries offer sufficient variety in brands, price points, and features; whether category groups are non\-overlapping and organized by customer need; and whether related queries span multiple types \(comparison, recommendation\)\.*Metrics\.*Binary criteria for product diversity, type diversity, query specificity, and category distinctness; deterministic duplicate\-detection checks\.

D4: Compositional Consistency\.*Challenge\.*A structured four\-field response is correct only if its fields are*mutually coherent*: the overview should preview the categories, the categories should be refinable by the related queries, and the attributions should support the claims\. Existing judges evaluate each field independently and miss cross\-field failures\.*Approach\.*PACEJudgereasons over explicit cross\-field constraints: overview–category alignment \(bolded terms match category names\), category–query coverage \(queries refine the categories\), evidence–claim grounding, diversity/non\-redundancy, and intent preservation across all four fields\. The objective is to judge the response as a structured object rather than scoring each field in isolation\.*Metrics\.*Component\-consistency score \(1–5, LLM\-scored\); deterministic highlight–category overlap check; GOOD–BAD score gap on D4\.

D5: Evidence Grounding\.*Challenge\.*Shopping assistants must ground their claims in real product evidence\. Hallucinated product features or invalid attribute IDs erode user trust\. Evidence\-only evaluators[12](https://arxiv.org/html/2608.26180#bib.bib10)address this but ignore whether the*selected*evidence is appropriate for the specific user\.*Approach\.*The judge verifies that claims in the overview and categories are supported by the referenced evidence records, that attribute IDs resolve to valid entries in the evidence pool, and that no hallucinated evidence appears\. Evidence\-grounding assessment combines deterministic ID validation with LLM\-based claim–evidence alignment\.*Metrics\.*Evidence\-grounding score \(1–5, LLM\-scored\); attribute\-ID validity \(deterministic\); hallucinated\-evidence rate \(fraction of cited IDs absent from the evidence pool\); GOOD–BAD score gap on D5\.

D6: Format & Safety\.*Challenge\.*Structural compliance \(correct counts, lengths, valid XML\) and trust/safety requirements \(neutral tone, appropriate disclaimers, no banned content\) are prerequisites for deployment but are often treated as afterthoughts in evaluation\.*Approach\.*Twelve deterministic validation checks \(overview length, bold–category match, category and related\-query counts and uniqueness, evidence\-ID validity, XML round\-trip, banned\-string screening\) form a cost\-free baseline\. LLM judges additionally assess language tone and score a format\-safety axis\.*Metrics\.*Format\-safety score \(1–5, LLM\-scored\); 12\-check deterministic pass rate; parse success rate; GOOD–BAD score gap on D6\.

Mapping D1–D6 to the four main\-body axes\.In the main\-body table \(§[5\.3](https://arxiv.org/html/2608.26180#S5.SS3)\) we report four axes rather than six: D2 and D3 both contribute to the*component\_consistency*axis \(relevancy is assessed within each component, and diversity is checked across components\), which keeps the output contract compact without losing the underlying signals\. The full six\-dimension rubric is used by the LLM in its free\-text reasoning and is reflected back into the four axis scores\.

### D\.3Fault Localization Protocol

Beyond the quality dimensions,PACEJudgeaddresses a capability missing from existing evaluation frameworks:*structured fault localization*\. Rather than reducing evaluation to a scalar score, the judge predicts \(i\) the*defect family*\(which of 7 failure modes occurred\) and \(ii\) the*defect location*\(which of 4 field groups contains the fault\)\. This transforms the judge from a classifier into a*diagnostic tool*: developers can identify not just that a response failed, but*what*went wrong and*where*to fix it\.

The multi\-tier defect design inPACEShopcreates a particularly demanding test: for multi\-defect records, the judge must detect and localize faults across up to four field groups simultaneously\. We measure localization via defect\-type macro\-F1 \(7 families\), defect\-location exact match, and field\-group match\. For single\-defect records, exact match is reported directly\. For multi\-defect records the gold label is a set of locations while the current protocol outputs one prediction, so we report a*hit rate*—whether the predicted location appears in the gold set\. Multi\-location prediction is a natural extension \(a single inference emitting a list rather than a scalar location\)\.

### D\.4PACEJudgeEvaluation Protocol and Prompt

ThePACEJudgeevaluation protocol is implemented as a single training\-free judging prompt that takes aPACEShoprecord\(q,p,y,E\)\(q,p,y,E\)as typed input and returns the structured output defined in §[4](https://arxiv.org/html/2608.26180#S4)\. To make the protocol fully reproducible, we reproduce the prompt verbatim in this subsection; this is the symmetric counterpart of the dataset\-side response\-generation prompt in Appx\.[B\.9](https://arxiv.org/html/2608.26180#A2.SS9), and it is the prompt that allPACEJudgerows in Tabs\.[3](https://arxiv.org/html/2608.26180#S4.T3)and[4](https://arxiv.org/html/2608.26180#S5.T4)use across every backbone\.

##### Prompt structure\.

The prompt is parameterized by four slot groups, each contributing a different piece of the protocol:

1. 1\.Typed input bundle\.Ten named slots bind the record to evaluation\-facing fields:record\_id,split,keyword\(the queryqq\),persona\_json\(the structured personapp\),response\_xmlandresponse\_struct\_json\(the candidate responseyyin two views\),candidate\_evidence\_ids\_jsonandcandidate\_evidence\_json\(the evidence poolEEas IDs and previews\),used\_evidence\_ids\_json\(the evidence subset cited duringPACEShopGOOD\-response construction, used as a soft anchor rather than a label\), anddeterministic\_checks\_json\(the L1 12\-check validator output, exposed to the judge as objective anchors\)\.
2. 2\.Closed label sets\.The diagnostic vocabulary is pinned to two closed enumerations:allowed\_defect\_labels= the sevenPACEShopdefect families plusNONE, andallowed\_location\_labels= the four response\-field groups \(overview,categories,related queries,evidence attributions\) plusNONE\. Free\-form labels outside these enumerations are rejected at parse time rather than coerced\.
3. 3\.Decision rules\.Four explicit rules force the judge to commit to a verdict and an evidence\-grounded diagnosis rather than collapse to a scalar score: \(i\) mark GOOD only when the response is correct for the persona, coherent across components, grounded inEE, and structurally valid; \(ii\) useNONE/NONEfor defect family/location only when no material defect is present; \(iii\)supporting\_evidence\_idsmust be a subset ofcandidate\_evidence\_ids\_json\(no invented IDs\); \(iv\)confidenceandoverall\_bad\_probabilitymust lie in\[0,1\]\[0,1\], whileoverall\_qualityand the four axis scores must lie in\[1,5\]\[1,5\]\.
4. 4\.Output\-schema contract\.A JSON\-Schema object \(theprompt\_contract\_jsonslot\) requires the verdictℓ^∈\{GOOD,BAD\}\\hat\{\\ell\}\\in\\\{\\textsc\{GOOD\},\\textsc\{BAD\}\\\}, the BAD probabilityp^bad∈\[0,1\]\\hat\{p\}\_\{\\mathrm\{bad\}\}\\in\[0,1\], the four axis scoressP,sC,sE,sF∈\[1,5\]s\_\{\\mathrm\{P\}\},s\_\{\\mathrm\{C\}\},s\_\{\\mathrm\{E\}\},s\_\{\\mathrm\{F\}\}\\in\[1,5\], and the actionable diagnostic objectaA=\(d^,r^,S^,c^,z^\)a\_\{\\mathrm\{A\}\}=\(\\hat\{d\},\\hat\{r\},\\hat\{S\},\\hat\{c\},\\hat\{z\}\)withd^∈𝒟∪\{NONE\}\\hat\{d\}\\in\\mathcal\{D\}\\cup\\\{\\textsc\{NONE\}\\\},r^∈ℛ∪\{NONE\}\\hat\{r\}\\in\\mathcal\{R\}\\cup\\\{\\textsc\{NONE\}\\\}, andS^⊆E\\hat\{S\}\\subseteq E\.

##### PACEJudgeprompt\.

The full prompt as run on every backbone is reproduced below\. Slots in curly braces are filled per\-record by the runner; the closed label sets and the JSON\-Schema contract are the same on every record\.

```
You are LLMaJ for persona-conditioned multi-component shopping
assistant evaluation.

Judge the candidate response as a structured object. Use the
deterministic checks as objective anchors, but do not stop there.
Reason about:
- persona alignment
- cross-component consistency
- evidence grounding
- format safety

Allowed defect labels: {allowed_defect_labels}
Allowed defect locations: {allowed_location_labels}

Input bundle
- Record ID: {record_id}
- Split: {split}
- Query: {keyword}
- Persona JSON:
{persona_json}
- Response XML:
{response_xml}
- Response struct JSON:
{response_struct_json}
- Candidate evidence IDs:
{candidate_evidence_ids_json}
- Candidate evidence preview:
{candidate_evidence_json}
- Used evidence IDs from benchmark construction:
{used_evidence_ids_json}
- Deterministic checks:
{deterministic_checks_json}

Decision rules
- Mark GOOD only when the response is correct for this user,
  coherent across components, grounded in the provided evidence,
  and structurally valid.
- Mark BAD when any material failure is present.
- Use NONE / NONE only if no material defect is present.
- supporting_evidence_ids must be a subset of Candidate evidence IDs.
- confidence and overall_bad_probability must be floats in [0, 1].
- overall_quality and all axis scores must be floats in [1, 5].

Return JSON only with this schema:
{prompt_contract_json}
```

##### Why this prompt is multi\-target rather than scalar\.

The four slot groups together are what distinguishPACEJudgefrom the rubric ablations and the single\-target adapted baselines in Appx\.[C\.3](https://arxiv.org/html/2608.26180#A3.SS3)\. The rubric ablations share the same input bundle and the same output schema but replace the four\-axis reasoning instruction with a generic rubric or a scalar quality impression; the single\-target adapted baselines \(ARES\-style, PersonaLens\-style, EtaPP\-style\) faithfully implement their native judging contracts but cover only onePACEtarget each\. OnlyPACEJudgepairs the four\-axis reasoning instruction with the closed\-vocabulary defect labels, the evidence\-ID containment constraintS^⊆E\\hat\{S\}\\subseteq E, and the explicit GOOD/NONE/NONEdiscipline that prevents over\-flagging\. Outputs that violate any closed\-vocabulary or range constraint are rejected at parse time rather than silently coerced, which is what allows the strict paired\-counterfactual gates of Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)– joint pair credit, target\-triage, and evidence\-ID containment – to be evaluated faithfully against the released schema\.

Table 21:Qualitative examples ofPACEJudge\(Opus 4\.6\) output on the development set\. The judge produces structured diagnostics: per\-axis scores identify*which*dimension failed \(e\.g\., Consistency = 1\.5 in Example 2\), defect type and location enable*actionable debugging*\(e\.g\., the costume\-themed rqList in a toy query\), and free\-text reasoning explains*why*the response is defective with specific evidence\. In Example 3, the judge identifies all three failure modes in its reasoning despite outputting only one primary defect label\.
#### D\.4\.1PACEJudgeOutput Examples

Tab\.[21](https://arxiv.org/html/2608.26180#A4.T21)presents three representative judge outputs fromPACEJudge\. They illustrate how the structured evaluation protocol produces*actionable diagnostics*rather than scalar scores\. In the single\-defect case \(Example 2\), the judge assigns a low consistency score \(1\.5\) while keeping other axes high, correctly pinpointing the cross\-field mismatch between toy categories and costume\-themed related queries\. In the triple\-defect case \(Example 3\), the judge’s free\-text reasoning identifies all three injected defect families—fabricated evidence IDs, hallucinated persona references, and off\-topic categories—even though the output schema allows only one primary defect label\. This generative reasoning, combined with structured axis scores and defect labels, bridges the gap between automated evaluation and human\-interpretable debugging\.

## Appendix EExperiments

This appendix provides the experimental details omitted from the main text for space: backbone and prompt settings, exact scenario metric definitions, per\-backbone main results, scenario\-level breakdowns, expanded findings, and – in the final subsection \(§[E\.8](https://arxiv.org/html/2608.26180#A5.SS8)\) – per\-target specificity audits, difficulty\-gradient analyses, qualitative outputs, and concrete record galleries\. Native judging\-protocol baseline support, prompt templates, and the judging\-protocol baseline ladder are documented separately in Appx\.[C](https://arxiv.org/html/2608.26180#A3)\.

### E\.1Backbones, Prompt Configurations, and Parsing

We evaluate seven LLM backbones spanning frontier and open\-weight tiers: Claude Opus 4\.7, Sonnet 4\.6, Sonnet 4\.5, Haiku 4\.5, Qwen3 32B, GPT\-OSS 20B, and GPT\-OSS 120B\. All backbones are run through the samePACEShoprecord interface\(q,p,y,E\)\(q,p,y,E\)where applicable\. Each backbone is paired with five prompt configurations: G\-Eval\-style scalar judging, MT\-Bench\-style scalar judging, persona\-blind judging, persona\-aware generic judging, andPACEJudge\. Details in Appx\.[C\.3](https://arxiv.org/html/2608.26180#A3.SS3)\. For all protocols that emit diagnostic text, we parse the output into the shared schema when possible\. If a protocol does not natively define a required field, the corresponding metric is marked unsupported rather than imputed\.

### E\.2Scenario\-Metric Mapping

We discuss the scenario\-metric mapping in Tab\.[22](https://arxiv.org/html/2608.26180#A5.T22)\.

Table 22:Scenario\-to\-failure summary\. Each metric is tied to thePACEtarget it is meant to close\. S0 is a reference check; the main protocol claim is measured by S1–S4 and the PACE average in Table[3](https://arxiv.org/html/2608.26180#S4.T3)\.
### E\.3PACEEvaluation Scenario Definitions

The main paper reports S0 as a reference scenario and S1–S4 asPACEclosure scenarios\. Here we provide the exact metric definitions used for the result tables, including family sets, location codes, axis thresholds, harmonic constructions, specificity terms, and paired\-counterfactual gates\.

##### S0: General reference\.

S0 measures balanced GOOD/BAD accuracy on a fixed balanced subset\. It is reported for context only and is not included in thePACEaverage\.

##### S1: Personalized\.

S1 uses persona\-swap records that hold the query and response fixed while replacing the persona with one that contradicts an attestation in the unchanged response\. The main metric is persona\-source exact match: the judge must identify the failure as a persona conflict rather than merely lowering a generic score\.

##### S2: Compositional\.

S2 evaluates cross\-component defects such asXCOMP\_MISMATCH,INTENT\_DRIFT, andREDUNDANCY\. The main metric requires both a BAD verdict and correct response\-field localization\.

##### S3: Evidence\-grounded\.

S3 evaluates both product\-evidence grounding and persona\-history grounding\. The metric checks whether the judge identifies the correct grounding\-related defect family while constraining supporting evidence IDs to the provided evidence pool\.

##### S4: Actionable\.

S4 evaluates whether the judge produces a debugging\-ready diagnosis\. The main metric requires exact defect\-family prediction and correct response\-field location\.

### E\.4Main Table Metrics Definitions

This subsection walks through every column of the two main\-text experiment tables – Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)and the paired\-counterfactual Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)– so the symbols and conjuncts in the captions can be read without flipping between sections\. Both tables are scored on the same 3,400\-record held\-out test set ofPACEShoprecords \(680 GOOD \+ 2,720 BAD\), with the S1 persona\-swap slice scored separately on a 680\-record held\-out swap slice\. ThePACEaverage in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)excludes S0 because S0 is a reference GOOD/BAD task rather than a target\-specific closure metric\.

Notation shared by all three tables\.“GOOD” and “BAD” refer to the gold record label\. “Family” denotes the gold or predicteddefect\_familyfield, drawn from the seven\-family taxonomy \{PREF\_CONFLICT,OVER\_PERSONALIZATION\_HALLUCINATION,XCOMP\_MISMATCH,INTENT\_DRIFT,REDUNDANCY,EVIDENCE\_MISMATCH,UNSUPPORTED\_CLAIM\} \(plusNONEfor GOOD records\)\. “Location” denotes the predicted 4\-way response\-field label \{overview, categories, related queries, evidence attributions\}\. “Axis” refers to the fourPACEJudgeaxes \{persona\_alignment, component\_consistency, evidence\_grounding, format\_safety\}, each scored on a 1–5 Likert scale\. “PACEtarget” refers to one of the four target dimensions \(P, C, E, A\); each defect family belongs to exactly one target\.

##### Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)metrics definition\.

Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)reports seven\-backbone means across Opus 4\.7, Sonnet 4\.6, Sonnet 4\.5, Haiku 4\.5, Qwen3 32B, GPT\-OSS 20B, and GPT\-OSS 120B on the 3,400\-record held\-out test set\. S0 is a reference GOOD/BAD task and is not included in thePACEaverage\. S1–S4 evaluatePACEdiagnostic closure: persona\-source diagnosis, cross\-component detection and localization, evidence grounding, and actionable defect\-family/location prediction\. False\-fire measures defect alarms on GOOD records, so methods cannot improve closure scores merely by predicting defects everywhere\. Cells marked “–” are unsupported by the method’s native judging protocol; unsupported columns are excluded from averages, and rows missing any contributingPACEcomponent report “–” for thePACEaverage\.

The columns are:

- •S0 \(Gen\.\): balanced GOOD/BAD accuracy on a 50\+50 reference subset\. This is a sanity check, not part of thePACEaverage\.
- •S1 \(P\-source\): persona\-source diagnosis on the held\-out persona\-swap slice\. The score is the fraction of swapped records on which the predicted defect family isPREF\_CONFLICT\. A judge that marks the record BAD without naming the persona conflict receives no credit\.
- •S2 detect: cross\-component defect detection on records whose gold family is in \{XCOMP\_MISMATCH,INTENT\_DRIFT,REDUNDANCY\}\. The score is the fraction of records marked BAD\.
- •S2 loc\.: cross\-component localization on the same evaluation slice\. The score requires both a BAD verdict and correct 4\-way response\-field localization\.
- •S3 grounding: harmonic mean of three grounding sub\-scores: product\-evidence mismatch detection, persona\-history hallucination detection, and GOOD\-record specificity against evidence\-family false alarms\. The harmonic mean requires all components to be high, preventing a judge from winning by always firing an evidence\-related defect\.
- •S4 family: exact defect\-family prediction on single\-defect BAD records across all seven defect families\.
- •S4 loc\.: exact 4\-way response\-field localization on the same single\-defect BAD slice\.
- •False\-fire↓\\downarrow: fraction of GOOD records on which the judge predicts any non\-NONEdefect family\. Lower is better\.
- •PACEavg\.: arithmetic mean of S1, S2\-loc\., S3 grounding, S4\-family, and\(1−False\-fire\)\(1\-\\text\{False\-fire\}\), with all components aligned so higher is better\.

Per\-column best scores in the rubric\-ablation block are bolded; the mean row provides the headline protocol\-level comparison\.

##### Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)Strict paired\-counterfactual metrics definition\.

The paired\-counterfactual columns close the “always cry BAD” loophole left open by single positive\-class metrics:

- •S0 \(ref\. acc\.\): balanced GOOD/BAD accuracy on the 50/50 reference subset; reported for context only and not part of the PACE average\.
- •S1P\{\}\_\{\\text\{P\}\}\(PJC\):*Joint Pair Credit*on \(good, swap\) pairs sharing a base record id\. The judge must classify the original\-persona record as GOOD*and*the swapped\-persona record as BAD with family==PREF\_CONFLICTand persona\_alignment axis<3\.0<3\.0on the same paired record\. A persona\-blind judge cannot pass this gate\.
- •S2C\{\}\_\{\\text\{C\}\}\(J\-strict\): BAD verdict, family∈𝒟C\\in\\mathcal\{D\}\_\{\\mathrm\{C\}\}, correct 4\-way location, and component\-consistency axis<3\.0<3\.0, minus the false\-C\-family alarm rate on GOOD records\.
- •S3E\{\}\_\{\\text\{E\}\}\(J\-harmonic\): harmonic of three E sub\-scores: Youden’s J on product\-evidence BADs withsupporting\_evidence\_ids⊆\\subseteqcandidate pool; Youden’s J on persona\-history BADs with the same containment check; and paired specificity \(no E\-family alarm on GOOD records\)\. The containment check makes “hallucinate evidence IDs” a scored failure mode\.
- •S4A\{\}\_\{\\text\{A\}\}\(J\-strict\): exact family\+\+exact 4\-way location\+\+target\-triage \(the lowest relevantPACEJudgeaxis matches the defect’s target\)\+\+confidence≥0\.5\\geq 0\.5, minus the confident\-false\-alarm rate on GOOD records\.
- •PACE: arithmetic mean of S1P\{\}\_\{\\text\{P\}\}, S2C\{\}\_\{\\text\{C\}\}, S3E\{\}\_\{\\text\{E\}\}, and S4A\{\}\_\{\\text\{A\}\}\.

The*Comparison level*column distinguishes single\-backbone, 7\-backbone mean, and 7\-backbone self\-consistency majority\-vote results; single\-target adapted baselines \(ARES\-style E, PersonaLens\-style P, EtaPP\-style A\) report measured Sonnet 4\.6 numbers on cells inside their native target and “–” elsewhere\. The native judging\-protocol map \(Tab\.[19](https://arxiv.org/html/2608.26180#A3.T19)\) and evaluation\-target leakage audit \(Tab\.[20](https://arxiv.org/html/2608.26180#A3.T20)\) explain why these baselines remain “–” outside their native target\.

### E\.5Full Experiments Results

#### E\.5\.1Full Per\-Backbone Results

Tab\.[23](https://arxiv.org/html/2608.26180#A5.T23)provides the full per\-\(method, backbone\) breakdown on the 3,400\-record held\-out test set; the per\-methodMeanrow aggregates each method into the corresponding row of the compact main\-text Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\. The per\-backbone spread, not just the mean, is informative:PACEJudgeon the strong frontier backbones \(Opus 4\.7, Sonnet 4\.6\) reaches PACE 0\.71–0\.78 with False\-fire 0\.16–0\.20, while open\-weight backbones over\-fire under every rubric, includingPACEJudge\.

S0S1S2 CS3 ES4 AFalse\-firePACEMethodBackboneGen\.P\-sourcedetectloc\.groundingfamilyloc\.↓\\downarrowavg\.*Rubric ablations of the same PACE output contract**G\-Eval*G\-EvalOpus 4\.70\.870\.230\.890\.860\.780\.520\.870\.700\.54G\-EvalSonnet 4\.60\.860\.280\.790\.870\.600\.710\.900\.910\.51G\-EvalSonnet 4\.50\.720\.260\.930\.860\.840\.650\.850\.460\.63G\-EvalHaiku 4\.50\.610\.200\.960\.630\.420\.330\.700\.750\.37G\-EvalQwen3 32B0\.600\.000\.060\.310\.000\.260\.270\.010\.31G\-EvalGPT\-OSS 20B0\.750\.140\.990\.840\.000\.480\.580\.610\.37G\-EvalGPT\-OSS 120B0\.710\.090\.650\.550\.180\.440\.610\.230\.41G\-EvalMean0\.730\.170\.750\.700\.400\.480\.680\.520\.45*Scalar*ScalarOpus 4\.70\.830\.120\.890\.850\.850\.570\.880\.680\.54ScalarSonnet 4\.60\.840\.170\.820\.850\.480\.640\.860\.930\.44ScalarSonnet 4\.50\.730\.290\.820\.780\.870\.600\.840\.380\.63ScalarHaiku 4\.50\.560\.100\.980\.450\.470\.300\.630\.870\.29ScalarQwen3 32B0\.620\.000\.130\.160\.000\.240\.270\.010\.28ScalarGPT\-OSS 20B0\.680\.110\.960\.420\.060\.420\.510\.550\.29ScalarGPT\-OSS 120B0\.870\.080\.800\.680\.790\.640\.750\.160\.61ScalarMean0\.730\.120\.770\.600\.500\.490\.680\.510\.44*Persona\-blind*Persona\-blindOpus 4\.70\.60–0\.990\.770\.560\.550\.810\.97–Persona\-blindSonnet 4\.60\.85–0\.820\.850\.800\.620\.870\.69–Persona\-blindSonnet 4\.50\.85–0\.820\.810\.900\.610\.850\.18–Persona\-blindHaiku 4\.50\.62–0\.960\.520\.650\.310\.620\.80–Persona\-blindQwen3 32B0\.62–0\.150\.290\.000\.270\.300\.01–Persona\-blindGPT\-OSS 20B0\.69–0\.850\.440\.110\.420\.580\.48–Persona\-blindGPT\-OSS 120B0\.92–0\.910\.830\.420\.660\.830\.19–Persona\-blindMean0\.74–0\.790\.640\.490\.490\.690\.47–*Persona\-aware*Persona\-awareOpus 4\.70\.770\.070\.910\.850\.910\.620\.860\.450\.60Persona\-awareSonnet 4\.60\.850\.220\.830\.840\.860\.700\.850\.560\.61Persona\-awareSonnet 4\.50\.710\.340\.790\.750\.790\.510\.760\.340\.61Persona\-awareHaiku 4\.50\.590\.370\.960\.650\.720\.450\.710\.760\.49Persona\-awareQwen3 32B0\.640\.040\.290\.190\.000\.220\.230\.010\.29Persona\-awareGPT\-OSS 20B0\.770\.240\.870\.480\.180\.390\.580\.370\.38Persona\-awareGPT\-OSS 120B0\.840\.140\.850\.720\.770\.630\.750\.130\.62Persona\-awareMean0\.740\.200\.790\.640\.600\.500\.680\.370\.51*PACEJudge*PACEJudgeOpus 4\.70\.910\.220\.960\.880\.960\.670\.910\.160\.71PACEJudgeSonnet 4\.60\.960\.560\.950\.900\.930\.690\.910\.200\.78PACEJudgeSonnet 4\.50\.550\.781\.000\.860\.730\.640\.870\.850\.63PACEJudgeHaiku 4\.50\.610\.221\.000\.700\.670\.490\.720\.730\.47PACEJudgeQwen3 32B0\.830\.050\.730\.500\.080\.400\.560\.040\.40PACEJudgeGPT\-OSS 20B0\.600\.280\.960\.460\.390\.500\.660\.800\.37PACEJudgeGPT\-OSS 120B0\.730\.140\.910\.730\.610\.740\.820\.520\.54PACEJudgeMean0\.740\.320\.930\.720\.620\.590\.780\.470\.56*Honest\-adapted single\-target baselines \(cells shown only where the native judging protocol supports the metric\)**ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)*ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)Opus 4\.70\.74–––0\.95––0\.02–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)Sonnet 4\.60\.69–––0\.32––0\.02–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)Sonnet 4\.50\.74–––0\.93––0\.23–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)Haiku 4\.50\.79–––0\.45––0\.28–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)Qwen3 32B0\.61–––0\.00––0\.31–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)GPT\-OSS 20B0\.55–––0\.00––0\.69–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)GPT\-OSS 120B0\.68–––0\.08––0\.29–ARES\-style[12](https://arxiv.org/html/2608.26180#bib.bib10)Mean0\.69–––0\.39––0\.26–*PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)*PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)Opus 4\.70\.680\.06–––––0\.00–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)Sonnet 4\.60\.720\.04–––––0\.00–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)Sonnet 4\.50\.700\.40–––––0\.06–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)Haiku 4\.50\.630\.41–––––0\.34–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)Qwen3 32B0\.660\.11–––––0\.01–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)GPT\-OSS 20B0\.560\.18–––––0\.06–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)GPT\-OSS 120B0\.600\.08–––––0\.00–PersonaLens\-style[18](https://arxiv.org/html/2608.26180#bib.bib4)Mean0\.650\.18–––––0\.07–*EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)*EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)Opus 4\.70\.78––––0\.600\.910\.50–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)Sonnet 4\.60\.81––––0\.550\.900\.35–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)Sonnet 4\.50\.50––––0\.650\.950\.99–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)Haiku 4\.50\.56––––0\.500\.840\.88–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)Qwen3 32B0\.73––––0\.340\.410\.05–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)GPT\-OSS 20B0\.85––––0\.570\.750\.22–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)GPT\-OSS 120B0\.72––––0\.320\.440\.00–EtaPP\-style[5](https://arxiv.org/html/2608.26180#bib.bib8)Mean0\.71––––0\.500\.740\.43–

Table 23:Full per\-backbone results\.Companion to Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3), which reports the 7\-backbone mean\. Each method is run on seven backbones \(Opus 4\.7, Sonnet 4\.6, Sonnet 4\.5, Haiku 4\.5, Qwen3 32B, GPT\-OSS 20B, GPT\-OSS 120B\) on the 3,400\-record held\-out test set\.S0: balanced GOOD/BAD accuracy on a 50\+50 reference subset\.S1: predicted family==PREF\_CONFLICTon the persona\-swap slice\.S2 detect / loc\.: BAD verdict and 4\-way response\-field localization on the cross\-component defect subset \(XCOMP\_MISMATCH,INTENT\_DRIFT,REDUNDANCY\)\.S3 grounding: harmonic mean of \(i\) BAD with exactEVIDENCE\_MISMATCHfamily on the product\-evidence subset, \(ii\) BAD with exactOVER\_PERSONALIZATION\_HALLUCINATIONfamily on the persona\-history subset, \(iii\) no E\-family alarm on GOOD records \(specificity\)\.S4 family / loc\.: exact family and 4\-way location on single\-defect BAD records\.False\-fire↓\\downarrow: fraction of GOOD records on which the judge fired any non\-nonedefect family \(lower is better\)\.PACE: mean of S1, S2\-loc\., S3 grounding, S4\-family, and\(1−False\-fire\)\(1\-\\text\{False\-fire\}\), all aligned higher==better\. “–” marks cells the method’s native judging protocol does not define\. Per\-method best mean is bolded on thePACEJudgerow\.
#### E\.5\.2S1 Personalized: Per\-Axis and Per\-Backbone Breakdown

Tab\.[24](https://arxiv.org/html/2608.26180#A5.T24)reports the full per\-backbone breakdown of the persona\-swap diagnostic \(Finding 4 in Sec\.[5](https://arxiv.org/html/2608.26180#S5)\), whose 7\-backbone mean is the main\-text Tab\.[5](https://arxiv.org/html/2608.26180#S5.T5)\. Each row reports the fraction of swap records in the 680\-record held\-out persona\-swap slice for which the judge predictsPREF\_CONFLICT, broken down by the four conflict axes \(budget, hard constraint, brand affinity, attribute preference\); the axis\-mean of each method’sMeanrow also populates the S1 P\-source column of Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\. The per\-backbone spread is informative on its own:PACEJudgeon Sonnet 4\.5 reaches 0\.57–0\.89 across axes, but on Qwen3 32B never exceeds 0\.07, which is the source of the open\-weight gap referenced in the main\-text discussion\.

Table 24:Full per\-backbone persona\-source diagnosis by conflict axis\.Companion to Tab\.[5](https://arxiv.org/html/2608.26180#S5.T5), which reports the 7\-backbone mean\. The evaluation uses the 680\-record held\-out persona\-swap slice\. A hit means the judge predictsPREF\_CONFLICT, the gold source family induced by the counterfactual persona swap\. TheMeanrow aggregates the seven backbones \(Opus 4\.7, Sonnet 4\.6, Sonnet 4\.5, Haiku 4\.5, Qwen3 32B, GPT\-OSS 20B, GPT\-OSS 120B\) into the corresponding row of the main\-text Tab\.[5](https://arxiv.org/html/2608.26180#S5.T5)\.
#### E\.5\.3S2 Compositional: Detection and Localization Details

Tab\.[25](https://arxiv.org/html/2608.26180#A5.T25)reports the full per\-backbone breakdown of the component\-consistency evaluation slice \(Finding 4 in Sec\.[5](https://arxiv.org/html/2608.26180#S5)\), whose 7\-backbone mean is the main\-text Tab\.[6](https://arxiv.org/html/2608.26180#S5.T6)and the same numbers populate the S2 detect/loc\. columns of Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\. Each method is evaluated on the 100\-record cross\-field defect subset \(XCOMP\_MISMATCH,INTENT\_DRIFT,REDUNDANCY\); thedetectcolumn is BAD verdict and the4\-way loc\.column is correct response\-field localization\.

Table 25:Full per\-backbone detection and 4\-way localizationon the 100\-record cross\-field defect subset \(XCOMP\_MISMATCH, INTENT\_DRIFT, REDUNDANCY\)\. Companion to Tab\.[6](https://arxiv.org/html/2608.26180#S5.T6), which reports the 7\-backbone mean\. TheMeanrow aggregates the seven backbones \(Opus 4\.7, Sonnet 4\.6, Sonnet 4\.5, Haiku 4\.5, Qwen3 32B, GPT\-OSS 20B, GPT\-OSS 120B\) into the corresponding row of the main\-text Tab\.[6](https://arxiv.org/html/2608.26180#S5.T6); the same numbers populate the S2 detect / loc\. columns of Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)\.

### E\.6Expanded Experiments Findings

This section expands the main\-text findings with the detailed per\-scenario evidence\. The main paper groups the results into four high\-level conclusions for readability; here we keep the original six findings to show how each conclusion is supported by scenario\-level and audit\-level results\.

##### Finding 1:PACEclosure separates protocol design from generic quality\.

S0 shows that several scalar and generic judging prompts can perform reasonably on broad GOOD/BAD discrimination\. This confirms thatPACEShopis not simply an impossible binary classification task\. However, the gap appears when the metric requires fields that close thePACEtargets: persona\-source diagnosis, cross\-component localization, grounding control, and defect\-family/location prediction\. Across the seven\-backbone mean in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3),PACEJudgeachieves the best score on every runnable closure column, including S1 persona\-source diagnosis, S2 localization, S3 grounding, S4 defect\-family prediction, and the overallPACEaverage\. This supports the main claim that structured shopping\-assistant evaluation requires a protocol that asks for the right diagnostic fields, not merely a scalar quality score\.

##### Finding 2: personalized evaluation must be scored as a source of invalidity\.

S1 is designed to prevent shortcuts\. If the metric only asked whether a swapped\-persona record is marked BAD, a persona\-blind or over\-pessimistic judge could receive credit without actually using the persona\. We therefore score exact persona\-source diagnosis: the judge must identify the failure asPREF\_CONFLICTon records where the query and response are unchanged but the persona is counterfactually swapped\. Tab\.[5](https://arxiv.org/html/2608.26180#S5.T5)shows that this source label is not solved uniformly across conflict axes\. The detailed per\-backbone results in Tab\.[24](https://arxiv.org/html/2608.26180#A5.T24)further show that strong GOOD/BAD discrimination does not guarantee reliable persona\-conflict diagnosis\. This justifies keeping S1 separate from S0\.

##### Finding 3: compositional and grounding failures expose where generic judges are under\-specified\.

S2 and S3 test whether judges can move beyond detecting that a response is flawed\. For S2, generic prompts often recognize that something is wrong, but performance drops when the judge must also localize the failed response component\. Tab\.[6](https://arxiv.org/html/2608.26180#S5.T6)separates detection from localization to show this gap directly\. For S3, the distinctive requirement is not only product\-evidence faithfulness, but also avoiding invented shopper\-history claims and hallucinated supporting evidence IDs\. This is why grounding is evaluated against both the product evidence pool and the persona history\. Together, S2 and S3 show that structured shopping\-assistant evaluation requires field\-aware and evidence\-constrained diagnosis, not only generic helpfulness or RAG\-style faithfulness\.

##### Finding 4: actionable evaluation is not the same as scalar evaluation\.

S4 requires a debugging\-ready diagnosis: the judge must identify both the defect family and the response\-field location\. This is stricter than deciding whether a response is BAD\. A scalar judge may provide a plausible overall rationale while still failing to say which family of error occurred or which response component should be fixed\. The S4 results show thatPACEJudgeimproves this complete diagnosis setting because its protocol explicitly requires family, location, confidence, rationale, and evidence support\. This finding isolates the Actionable target: evaluation is useful for developers only when it reports what failed and where\.

##### Finding 5: specificity matters; closure should not come from over\-flagging\.

A high score on S1–S4 would be less meaningful if a judge achieved it by predicting defects on most records\. We therefore audit false\-fire behavior on GOOD records\. The False\-fire column in Tab\.[3](https://arxiv.org/html/2608.26180#S4.T3)measures how often a method incorrectly reports defects on GOOD examples, and the strict paired\-counterfactual Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)folds the same specificity check into per\-target gates \(PJC, Youden’s J, J\-harmonic\) so that a method cannot game closure by always flagging BAD\. Together these checks show whether a protocol preserves specificity while detecting structured failures\. The results indicate thatPACEJudge’s gains are not explained solely by always flagging BAD cases; rather, the protocol improves the ability to report the intended diagnostic fields\.

##### Finding 6: strict paired\-counterfactual metrics preserve the conclusion\.

Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)re\-scores the setting under stricter paired\-counterfactual gates\. These metrics reduce loopholes such as always predicting BAD or assigning generic defect labels\. For example, S1 requires joint success on paired GOOD and persona\-swapped BAD records; S2 requires BAD verdict, correct compositional family, correct location, and axis consistency; S3 combines product\-evidence detection, persona\-history detection, and GOOD\-record specificity with evidence\-ID containment; and S4 requires exact family, exact location, axis triage, and confidence thresholding\. Under this stricter formulation,PACEJudgeremains strongest on thePACEaverage, supporting the same conclusion as the main table: the advantage comes from structured diagnostic closure rather than only lenient positive\-class scoring\.

### E\.7Standard Judges Failure Analysis

#### E\.7\.1Concrete Benchmark Examples

The remaining paragraphs collect record\-level galleries that illustratePACEShopbehaviour end to end\. Each record is drawn directly from the released dataset\.

##### Ten Concrete Failure Cases\.

Tab\.[26](https://arxiv.org/html/2608.26180#A5.T26)walks through ten BAD records with their gold defect family and gold defect location, paired with thePACEJudgeverdict, axis scores, and predicted family/location\. The cases span all four field groups and all seven defect families; they make concrete the structured\-output diagnoses that the main text reports as aggregate numbers\.

Table 26:Ten concrete failure cases\. Each row contrasts a scalar baseline’s verdict and reasoning againstPACEJudge’s structured output on the same record\. Axis scores abbreviated P/C/G/F\.
##### Persona\-Swap Conflict Examples\.

Tab\.[27](https://arxiv.org/html/2608.26180#A5.T27)lists ten persona\-swap records and the specific persona attestation each swap puts in conflict with the unchanged response\. The four conflict axes \(budget, hard constraint, brand affinity, attribute preference\) are represented; each row is a paired GOOD record with a single persona swap that turns it into a gold\-BAD record for the S1 scenario\.

Table 27:Ten representative persona\-swap records\. Each swap flips one axis \(budget, hard\_constraint, brand\_affinity, attribute\_preference\) to a value drawn from the 1,200\-persona pool that directly contradicts an attestation in the unchanged response; the gold label flips from GOOD to BAD\.

### E\.8Audits, Robustness Checks, and Concrete Examples

The main text reports compact S0–S4 tables\. The remaining experiment subsections keep the supporting audits and record\-level galleries that explain why those numbers should be read as PACE\-closure results rather than ordinary scalar\-judge scores\. The subsections below check off\-target leakage, false fires on GOOD records, per\-backbone stability, tier difficulty, qualitative judge outputs, earlier reference runs, and concrete benchmark records that illustratePACEShopbehaviour end to end\.

#### E\.8\.1Difficulty Gradient Analysis

The multi\-tier defect design enables a decomposition unavailable to single\-defect benchmarks\. Per\-tier accuracy \(computed separately per defect count\) reveals two distinct regimes:

1. 1\.Deterministic baseline:accuracy rises monotonically from single \(0\.880\) to quad \(0\.975\) on the development set, because more defects trigger more rule\-based checks\.
2. 2\.LLM judges:tier profiles are flatter for frontier models\. Smaller models \(Qwen3 32B, Gemma3 27B\) show steeper gradients—e\.g\., Qwen3 32B rises from 0\.740 \(single\) to 0\.990 \(quad\)—suggesting they rely on surface signal accumulation rather than understanding individual fault types\.

This two\-regime pattern motivates the tiered design: if every judge handled every tier uniformly, the tiered construction would be wasted\. Instead, smaller or scalar\-prompted judges’ steeper tier gradients give us a graded robustness metric a single\-tier benchmark cannot provide\. Fig\.[4](https://arxiv.org/html/2608.26180#S5.F4)shows per\-defect\-level accuracy by method on the development set\.

#### E\.8\.2Reference Main\-Evaluation Table on the Development Set

The per\-defect\-level discrimination figure and a paired\-counterfactual specificity audit are retained for reference\. These are not the primary SOTA\-failure numbers reported in §[5](https://arxiv.org/html/2608.26180#S5); they are kept because they document the tier\-gradient behavior that motivates the multi\-tier defect design and the swap\-TPR / paired\-good specificity that supports the strict paired\-counterfactual gates of Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)\.

Table 28:Paired\-counterfactual specificity audit\.S1P\{\}\_\{\\text\{P\}\}is reported as the swap\-side TPR and the paired GOOD\-side specificity \(judge correctly returns GOOD on the original\-persona record sharing the same base id\)\. Persona\-aware on Sonnet 4\.6 retains a high swap\-TPR but a low paired\-good specificity; that is exactly the “always cry BAD\-PREF\_CONFLICT” failure mode the paired metric rules out\. S2 / S4 columns show TPR on the BAD subset and FPR on the same\-CSV GOOD subset\. S3 columns show Ja\(product evidence\), Jb\(persona\-history grounding\), and the paired specificity rate \(no E\-family alarm on GOOD\)\. These quantities are the per\-rubric inputs to the harmonic in Tab\.[4](https://arxiv.org/html/2608.26180#S5.T4)\.