Self- and Other-Labels Induce Bidirectional Bias in LLM Judges

# Self- and Other-Labels Induce Bidirectional Bias in LLM Judges
Source: [https://arxiv.org/html/2608.18091](https://arxiv.org/html/2608.18091)
Songeun Chae Min Kim Donghoon Jung Seojin Choi Seohyon Jung School of Digital Humanities and Computational Social Sciences, KAIST, South Korea \{songeun, mk, donghoon\.jung, seojin64, seohyon\.jung\}@kaist\.ac\.kr

###### Abstract

As LLM\-as\-a\-judge systems become increasingly widespread, self\-preference in LLMs—the tendency to favor one’s own outputs—raises growing concerns about evaluation reliability\. However, it has been studied predominantly on generated text, where stylistic features and response quality are inevitably conflated\. As a result, existing measurements cannot separate genuine self\-preference from these confounds\. We address this by changing the object of evaluation: instead of judging generated text, ten LLMs assess narrative constraint selections, which carry no model\-specific stylistic fingerprint yet retain a recoverable model\-specific signature\. We run two experiments that yield distinct findings\. Under blind evaluation, self\-preference largely disappears once selection quality and evaluator severity are controlled\. It vanishes on three of four rubric dimensions and reverses on the fourth, where judges rate their own selections as less original\. Under matched quality, however, self\- and other\-labels alone—without naming any model—shift scores bidirectionally: LLM judges inflate scores for self\-labeled selections and deflate those for other\-labeled ones regardless of the selection’s actual source\. We make two contributions: 1\) authorship attribution is a distinct driver of evaluation bias, and 2\) open\-ended, ground\-truth\-free tasks can serve as controlled instruments for studying LLM judge behavior\.

Self\- and Other\-Labels Induce Bidirectional Bias in LLM Judges

Songeun Chae Min Kim Donghoon Jung Seojin Choi Seohyon Jung††thanks:Corresponding author\.School of Digital Humanities and Computational Social Sciences, KAIST, South Korea\{songeun, mk, donghoon\.jung, seojin64, seohyon\.jung\}@kaist\.ac\.kr

## 1Introduction

LLM\-as\-a\-judge has emerged as a widely adopted paradigm for scalable evaluation, using large language models \(LLMs\) as automatic assessors across diverse tasks\(Guet al\.,[2026](https://arxiv.org/html/2608.18091#bib.bib24)\)\. Despite their widespread use, recent studies have shown that LLM judges exhibit systematic biases\(Shiet al\.,[2025](https://arxiv.org/html/2608.18091#bib.bib17); Yeet al\.,[2025](https://arxiv.org/html/2608.18091#bib.bib8)\)\. A particularly concerning bias is self\-preference, where an LLM judge rates its own outputs more favorably than those of other models\(Wataokaet al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib12); Panicksseryet al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib18)\)\. Prior work has examined implicit sources of self\-preference, including low perplexity\(Wataokaet al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib12)\)and self\-recognition\(Panicksseryet al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib18)\)\.Sarafet al\.\([2025](https://arxiv.org/html/2608.18091#bib.bib26)\)investigated an explicit factor—authorship labels—by manipulating them across three commercial LLMs, finding that theClaudelabel intensified self\-preference\.

![Refer to caption](https://arxiv.org/html/2608.18091v1/figures/figure1.png)Figure 1:Experimental design\. Ten LLMs participate in dual roles: asselectorswho construct evaluation targets via a constraint selection task, and asjudgeswho evaluate the resulting selections on a 4\-axis rubric\. Each selection is judged under two conditions: Blind \(no source label\) and Labeled, where source attribution is experimentally manipulated in a 2×2 design crossing label veracity \(TL = true, FL = false\) with claimed source \(self vs\. other\)\.Recent studies have further identified methodological confounds in measuring self\-preference\. For instance,Chenet al\.\([2025a](https://arxiv.org/html/2608.18091#bib.bib2),[b](https://arxiv.org/html/2608.18091#bib.bib14)\)argue that existing measurements conflate response quality with genuine self\-preference\.Roytburget al\.\([2026](https://arxiv.org/html/2608.18091#bib.bib21)\)attribute almost 90% of the observed self\-preference to evaluator uncertainty—a judge’s limited ability to reliably assess response quality\. This raises two questions: First, does self\-preference persist once surface\-level cues and confounders are controlled? Second, can self/other labels alone induce self\-preference under the same controls?

To address these questions, we introduce a novel experimental design built around a creative selection task\. Rather than evaluating model\-generated text, ten LLM judges assess sets of narrative constraints selected from a pre\-constructed pool of 200, under both blind and labeled conditions\. Our results show that although most LLMs assign higher mean scores to their own selections, this apparent self\-preference largely disappears once output quality and judge severity are controlled\. Moreover, authorship labels alone shift evaluations bidirectionally—judges inflate scores under self\-labels while simultaneously deflating them under other\-labels—even when the evaluation target is identical\.

Our contributions are twofold\. First, we provide direct evidence that self/other labels alone can induce bidirectional evaluation bias in LLM judges\. Because our design eliminates stylistic confounds and controls for output quality, we can show that apparent self\-preference largely disappears, suggesting prior work has overstated it\. Second, by leveraging creative tasks as a testbed for assessing LLM biases, we show that open\-ended, ground\-truth\-free tasks can serve as controlled instruments for studying LLM judge behavior\.

## 2Related Work

### 2\.1LLM\-as\-a\-Judge and Evaluation Bias

LLMs are increasingly deployed as evaluators across a broad range of assessment tasks, such as pairwise comparison and direct scoring\(Saunderset al\.,[2022](https://arxiv.org/html/2608.18091#bib.bib10); Liuet al\.,[2023](https://arxiv.org/html/2608.18091#bib.bib5); Baiet al\.,[2023](https://arxiv.org/html/2608.18091#bib.bib6); Liet al\.,[2025](https://arxiv.org/html/2608.18091#bib.bib4)\)\. However, prior work has identified systematic biases that undermine the reliability of LLM\-as\-a\-judge\. Position bias—the tendency of judges to favor a response based on its order rather than its quality—is one of the most extensively studied\(Zhenget al\.,[2023](https://arxiv.org/html/2608.18091#bib.bib1); Wanget al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib9); Shiet al\.,[2025](https://arxiv.org/html/2608.18091#bib.bib17)\)\.Kooet al\.\([2024](https://arxiv.org/html/2608.18091#bib.bib7)\)examined six cognitive biases in LLM evaluators, including attentional bias and the bandwagon effect, whileYeet al\.\([2025](https://arxiv.org/html/2608.18091#bib.bib8)\)documented a broader spectrum, showing that judges also prefer responses based on verbosity and sentiment\. Among these biases,*self\-preference*\(Panicksseryet al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib18); Liuet al\.,[2024](https://arxiv.org/html/2608.18091#bib.bib16); Pombalet al\.,[2026](https://arxiv.org/html/2608.18091#bib.bib13)\)—also termed self\-enhancement bias\(Zhenget al\.,[2023](https://arxiv.org/html/2608.18091#bib.bib1)\)—refers to the tendency of an LLM judge to favor its own outputs over other models’ outputs\. This bias is particularly concerning because it can directly compromise the validity of LLM\-based evaluation, motivating closer examination\.

### 2\.2Sources and Confounds of LLM Self\-Preference

While self\-preference has been demonstrated across diverse evaluation domains\(Mahbub and Feng,[2026](https://arxiv.org/html/2608.18091#bib.bib15); Pombalet al\.,[2026](https://arxiv.org/html/2608.18091#bib.bib13)\), its underlying sources remain contested\.Panicksseryet al\.\([2024](https://arxiv.org/html/2608.18091#bib.bib18)\)find a positive correlation between self\-recognition ability and self\-preference strength, whereasWataokaet al\.\([2024](https://arxiv.org/html/2608.18091#bib.bib12)\)attribute self\-preference to a familiarity effect, with judges favoring lower\-perplexity outputs\. Recent work raises a more fundamental question: whether this effect reflects a genuine bias\.Chenet al\.\([2025b](https://arxiv.org/html/2608.18091#bib.bib14)\)show that existing measures conflate self\-preference with response quality;Chenet al\.\([2025a](https://arxiv.org/html/2608.18091#bib.bib2)\)argue that, while stronger models exhibit greater self\-preference, much of this preference reflects their objectively superior output quality\. Notably,Roytburget al\.\([2026](https://arxiv.org/html/2608.18091#bib.bib21)\)attribute roughly 90% of measured self\-preference to evaluator uncertainty, demonstrating that judges’ difficulty in assessing harder queries inflates the apparent self\-preference rate\. Taken together, the majority of reported self\-preference may stem from measurement confounds, though the mechanism behind any residual is still contested\.

### 2\.3Authorship Labels and Self\-Preference

A distinct line of work examines whether authorship labels can shift LLM\-based evaluations\.Sunet al\.\([2026](https://arxiv.org/html/2608.18091#bib.bib22)\)reveal that LLM judges attend more closely to the authorship label than to content itself, using it as a shortcut for evaluation\.Marioriyadet al\.\([2025](https://arxiv.org/html/2608.18091#bib.bib23)\)identify a preference hierarchy among authorship labels \(Expert\>\>Human\>\>LLM\>\>Unknown\)\. Most relevant to our work,Sarafet al\.\([2025](https://arxiv.org/html/2608.18091#bib.bib26)\)test label\-induced bias in three commercial LLM judges under four conditions—no attribution, true attribution, and two false attributions\. TheClaudelabel elevates scores while theGeminilabel lowers them, producing apparent self\-preference in Claude and self\-deprecation in Gemini\. However, since the evaluated content consisted of LLM\-generated blog posts, these effects remain confounded with stylistic fingerprints, perplexity, and output quality, leaving self\-recognition as a potential driver of self\-preference\. More critically, because the labels were real model names, it is unclear whether the observed shifts reflect self\-preference or model reputation effects\.

## 3Methodology

To address the limitations of prior work, we adopt a narrative selection task to generate evaluation targets\. We then prompt LLM judges to assess these selections on a 4\-dimension rubric across two settings: blind and labeled evaluation\. Ten LLMs serve as both selectors and judges: five commercial models \(Claude Opus 4\.7, Gemini 3\.1 Pro, GPT\-5\.5, Grok 4\.3, and Qwen3\.6\-Plus\) and five open\-source models \(DeepSeek\-V4\-Pro, Kimi K2\.6, Llama 4 Maverick, Mistral Large 3, and Qwen3\.6\-35B\-A3B\) \(see[Appendix A](https://arxiv.org/html/2608.18091#A1)\)\.

### 3\.1Narrative Constraint Selection Task

Instead of relying on text\-based evaluation and post\-hoc style control, we eliminate model\-specific stylistic features by changing the type of evaluation target: LLM judges assess structured narrative selections\. To generate these outputs, we prompt each LLM to perform the selection task 30 times, providing sufficient inter\-run coverage per model\. In each run, the LLM selects the 20 items it considers most useful for constructing a single story from a curated set of 200 narrative constraints\(Junget al\.,[2025](https://arxiv.org/html/2608.18091#bib.bib27)\)—each a pre\-written, single\-sentence description of an Event, Style, Character, or Setting \(see Figure[2](https://arxiv.org/html/2608.18091#S3.F2)\)\. This design offers two advantages\. First, word choice, sentence structure, and phrasing are held constant, preventing model\-specific stylistic features from influencing the evaluations\. Second, creative tasks provide a sensitive testbed for detecting LLM judgment biases, as such biases tend to be more salient in subjective, open\-ended domains\(Marioriyadet al\.,[2025](https://arxiv.org/html/2608.18091#bib.bib23); Feinet al\.,[2026](https://arxiv.org/html/2608.18091#bib.bib25)\)\.

![Refer to caption](https://arxiv.org/html/2608.18091v1/figures/selection_task.png)Figure 2:Narrative constraint selection task, adapted fromJunget al\.\([2025](https://arxiv.org/html/2608.18091#bib.bib27)\)\. The pool comprises 200 single\-sentence constraints from that work, organized into four categories\. In each run, an LLM freely selects any 20 constraints from the full pool with no category\-level quota\. All ten models each perform 30 runs, yielding 300 constraint sets in total, serving as evaluation targets\.Analysis of selections reveals that each model draws from a characteristic region of the constraint pool\. Within\-model Jaccard similarity\(Broder,[1997](https://arxiv.org/html/2608.18091#bib.bib29)\)exceeds chance for all ten models, indicating that each LLM exhibits a consistent selection profile \(see[Figure 3](https://arxiv.org/html/2608.18091#S3.F3);[Appendix E](https://arxiv.org/html/2608.18091#A5)for numerical values\)\. Moreover, a leave\-one\-out k\-NN classifier \(k=5k=5\) identifies the source model with 50\.7% accuracy—more than five times the 10% chance baseline \(robust acrossk∈\{1,3,5,7\}k\\in\\\{1,3,5,7\\\}: 46\.7–50\.7%\) \(see[Appendix D](https://arxiv.org/html/2608.18091#A4)\)\. Together, these analyses indicate that LLMs maintain distinct selection profiles when model\-specific stylistic cues are eliminated by design, providing the variation necessary for meaningful self\-versus\-other comparisons\.

![Refer to caption](https://arxiv.org/html/2608.18091v1/figures/mds_faceted.png)Figure 3:Selection patterns by model \(300 selections\), projected via Multidimensional Scaling\(Borg and Groenen,[2005](https://arxiv.org/html/2608.18091#bib.bib28)\)\. Each subplot highlights one model’s selections against the full set; models are ordered by within\-model Jaccard similarity \(descending\)\.
### 3\.2Evaluation Procedure

#### 3\.2\.1Rubric

Each selection set is assessed on a 7\-point Likert scale across four dimensions:Originality,Dimensionality,Coherence, andTellability\. The rubric is designed not as an exhaustive or validated measure of narrative quality, but as a structured interpretive frame\. By directing each judge to assess specific, theoretically grounded properties rather than self\-selecting which aspects to attend to, this framework reduces evaluative arbitrariness and renders judge behavior decomposable across conceptually distinct dimensions \(see[Appendix B](https://arxiv.org/html/2608.18091#A2)for full rubric with anchor descriptions and theoretical rationale\)\.

#### 3\.2\.2Experiment 1: Blind Evaluation

##### Setup\.

In Experiment 1, each model serves as a judge, evaluating all 300 selections without labels\. Each judge returns a JSON object with an integer score on a 1–7 scale for each dimension and a 2–3 sentence justification\. To reduce stochastic variability in LLM scoring, each judge evaluates each selection three times, yielding10​judges×300​selections×3​repetitions=9,00010\\text\{ judges\}\\times 300\\text\{ selections\}\\times 3\\text\{ repetitions\}=9\{,\}000runs\. Within each run, both the order of the 20 constraints in the selection and the order of the four rubric dimensions are independently shuffled using a per\-run seed, mitigating potential ordering biases\. The full evaluation prompt and response format are provided in[Appendix C](https://arxiv.org/html/2608.18091#A3)\.

##### Raw self–other comparison\.

For each judge, we compare the mean scores it assigns to its own selections versus those produced by other models, reporting the difference for the average score and each rubric dimension\. This naive comparison controls for neither selector quality nor judge severity; it serves as the baseline against which the confound\-controlled analysis is interpreted\.

##### Confound\-controlled estimation\.

To test whether self\-preference remains once measurement confounds are controlled for, we fit a mixed\-effects model:

score=β0\\displaystyle\\texttt\{score\}=\\beta\_\{0\}\+βS⋅is\_self\+\(1\|judge\)\\displaystyle\+\\beta\_\{S\}\\cdot\\texttt\{is\\\_self\}\+\(1\\,\|\\,\\texttt\{judge\}\)\+\(1\|selector\)\+\(1\|selection\_id\)\\displaystyle\{\}\+\(1\\,\|\\,\\texttt\{selector\}\)\+\(1\\,\|\\,\\texttt\{selection\\\_id\}\)
whereβS\\beta\_\{S\}is the coefficient of interest; the random intercepts absorb three measurement confounds:judgefor rater severity,selectorfor selection quality, andselection\_idfor non\-independence across repeated evaluations of the same selection\. A positive, significantβS\\beta\_\{S\}indicates that judges score their own selections higher after controls; a non\-significantβS\\beta\_\{S\}indicates that the apparent self\-preference is attributable to the controlled confounds\.

#### 3\.2\.3Experiment 2: Labeled Evaluation

##### Setup\.

In Experiment 2, LLM judges evaluate selections under four label conditions: TL\-self, TL\-other, FL\-self, and FL\-other \(see[Table 1](https://arxiv.org/html/2608.18091#S3.T1)\)\. For each judge, we construct quality\-matched pairs based on Experiment 1: a self\-produced and an other\-produced selection are paired if the judge rated them within0\.250\.25on the overall mean and within11point on each rubric dimension, yielding 40 pairs per judge \(400 total\)\. The thresholds were chosen to balance quality control with statistical coverage\.

Each selection is presented individually, so the attribution label affects an absolute rating rather than a direct pairwise comparison\. The label is inserted as an anonymous self/other phrase—“your own selection” for self\-labels and “another language model” for other\-labels—without naming specific models\. This isolates the self\-other distinction from model\-identity effects \(full prompts in[Appendix C](https://arxiv.org/html/2608.18091#A3)\)\. As in Experiment 1, each selection is evaluated three times, yielding40​pairs×10​judges×4​conditions×3​reps=4,80040\\text\{ pairs\}\\times 10\\text\{ judges\}\\times 4\\text\{ conditions\}\\times 3\\text\{ reps\}=4\{,\}800runs\.

Self labelOther labelTL\-selfFL\-otherSelf\-producedself selection \+ self labelself selection \+ other labelFL\-selfTL\-otherOther\-producedother selection \+ self labelother selection \+ other labelTable 1:2×22\\times 2design for Experiment 2, crossing the actual source of a selection with the label shown to the judge\. TL =*true label*\(label matches the actual source\); FL =*false label*\(label contradicts the actual source\)\.
##### Label\-induced attribution effect estimation\.

To investigate the authorship label effect and test whether it depends on the true content origin, we fit a mixed\-effects model:

score=β0\\displaystyle\\texttt\{score\}=\\beta\_\{0\}\+βL⋅label\+βA⋅actual\\displaystyle\+\\beta\_\{L\}\\cdot\\texttt\{label\}\+\\beta\_\{A\}\\cdot\\texttt\{actual\}\+βL​A⋅\(label×actual\)\\displaystyle\{\}\+\\beta\_\{LA\}\\cdot\(\\texttt\{label\}\\times\\texttt\{actual\}\)\+\(1\|judge\)\+\(1\|selection\_id\)\\displaystyle\{\}\+\(1\\,\|\\,\\texttt\{judge\}\)\+\(1\\,\|\\,\\texttt\{selection\\\_id\}\)
wherelabelindicates the displayed source \(self vs\. other\) andactualthe true source, with random intercepts\.βL\\beta\_\{L\}captures the main effect of the displayed label—the average score shift for a self\- versus other\-label—whereasβA\\beta\_\{A\}reflects the main effect of actual source, the average score difference between self\- and other\-produced selections\. A non\-significantβL​A\\beta\_\{LA\}would indicate that the label effect is invariant across actual sources, with the same label\-induced shift regardless of who actually produced the selection\.

##### Directional decomposition\.

βL\\beta\_\{L\}captures magnitude but not direction: it cannot distinguish whether self\-labels inflate scores, other\-labels deflate them, or both\. To resolve this, we use each judge’s Experiment 1 \(no\-label\) scores as the baseline and compute the deviation of every labeled evaluation from this baseline\.

## 4Results

### 4\.1Experiment 1: Apparent Self\-Preference Disappears Under Control

#### 4\.1\.1Sanity Check

![Refer to caption](https://arxiv.org/html/2608.18091v1/figures/score_freq_heatmap.png)Figure 4:Score frequency distribution across all Experiment 1 \(blind\) evaluations, used as a sanity check\. Each cell reports the percentage of evaluations in which a judge assigned that score for the given dimension\.In Experiment 1, eight of the ten judges produce meaningful score variation across selections \(see[Figure 4](https://arxiv.org/html/2608.18091#S4.F4)\)\. The remaining two—Llama 4 Maverick and Mistral Large 3, both open\-weight LLMs—concentrate their ratings on one or two scores, falling short of the basic discriminative capacity required for self–other comparisons\.

#### 4\.1\.2Raw Self\-Other Comparison

JudgeSelfOtherΔ\\DeltaKimi K2\.64\.883\.79\+1\.09\+1\.09GPT\-5\.55\.784\.92\+0\.86\+0\.86DeepSeek\-V4\-Pro4\.864\.34\+0\.52\+0\.52Claude Opus 4\.74\.013\.50\+0\.51\+0\.51Qwen3\.6\-Plus4\.734\.49\+0\.24\+0\.24Gemini 3\.1 Pro4\.414\.21\+0\.21\+0\.21Llama 4 Maverick5\.515\.47\+0\.04\+0\.04Mistral Large 35\.946\.02−0\.08\-0\.08Grok 4\.33\.684\.35−0\.67\-0\.67Qwen3\.6\-35B\-A3B3\.754\.65−0\.90\-0\.90Table 2:Raw self\-other comparison on the average score, sorted byΔ=Self−Other\\Delta=\\text\{Self\}\-\\text\{Other\}\. Confounds are not controlled; per\-dimension results are in[Appendix F](https://arxiv.org/html/2608.18091#A6)\.On average, seven of ten judges rate their own selections higher than others’ \(see[Table 2](https://arxiv.org/html/2608.18091#S4.T2); per\-dimension results in[Appendix F](https://arxiv.org/html/2608.18091#A6)\)\. However, the per\-judge×\\timesselector matrix \(see[Appendix G](https://arxiv.org/html/2608.18091#A7)\) suggests this gap reflects selection quality rather than genuine self\-preference: strong selectors receive higher scores from most judges, not just from themselves\. Kimi K2\.6 illustrates this—it shows the largest raw self–other gap, yet its selections also rank among the highest from nearly every judge\. The raw gap may therefore partly reflect producer\-level quality differences rather than genuine self\-preference, which we disentangle in the following analysis\.

#### 4\.1\.3Confound\-Controlled Estimation

Table[3](https://arxiv.org/html/2608.18091#S4.T3)reports the self\-preference coefficient \(βS\\beta\_\{S\}\) from §[3\.2\.2](https://arxiv.org/html/2608.18091#S3.SS2.SSS2)for the average score and each rubric dimension, before and after excluding the two low\-discrimination judges\. Before their removal,βS\\beta\_\{S\}is significantly positive on the average score and onDimensionality,Coherence, andTellability, whileOriginalityshows a small negative effect\. After excluding the two, all positive effects disappear; only the negativeOriginalityeffect persists, contrary to self\-preference\. The raw self–other gap therefore reflects selection quality and the inflated ratings of these judges rather than genuine self\-preference\.

Full \(10 judges\)Excl\. low\-discrim\. \(8\)Dim\.βS\\beta\_\{S\}95% CIβS\\beta\_\{S\}95% CIAvg\.\+0\.181∗\+0\.181^\{\*\}\[ 0\.13, 0\.23 \]\+0\.003\+0\.003\[−0\.05\-0\.05, 0\.06 \]Org\.−0\.144∗\-0\.144^\{\*\}\[−0\.21\-0\.21,−0\.08\-0\.08\]−0\.133∗\-0\.133^\{\*\}\[−0\.21\-0\.21,−0\.05\-0\.05\]Dim\.\+0\.316∗\+0\.316^\{\*\}\[ 0\.24, 0\.39 \]\+0\.039\+0\.039\[−0\.05\-0\.05, 0\.12 \]Coh\.\+0\.276∗\+0\.276^\{\*\}\[ 0\.21, 0\.34 \]\+0\.042\+0\.042\[−0\.03\-0\.03, 0\.12 \]Tel\.\+0\.278∗\+0\.278^\{\*\}\[ 0\.21, 0\.34 \]\+0\.065\+0\.065\[−0\.01\-0\.01, 0\.14 \]Table 3:Confound\-controlled self\-preference coefficientβS\\beta\_\{S\}from the mixed\-effects model, before and after excluding the two low\-discrimination judges\.Avg\.= the average of the four rubric dimensions and the primary outcome;Org\.=Originality,Dim\.=Dimensionality,Coh\.=Coherence,Tel\.=Tellability\. The positive effects on the average score and three dimensions vanish once low\-discrimination judges are removed; only a*negative*Originalityeffect persists, which runs counter to self\-preference\.

### 4\.2Experiment 2: Labels Alone Induce Bidirectional Attribution Bias

#### 4\.2\.1Per\-Condition Descriptive Statistics

[Table 4](https://arxiv.org/html/2608.18091#S4.T4)reports descriptive scores across four label–source conditions, illustrating how self\- versus other\-attribution shifts evaluations of identical selections regardless of actual authorship\. Within eachActualblock, self\-labels elevate scores over other\-labels by 0\.29–0\.57 across all four dimensions without exception—notably includingOriginality, which had shown a self\-deprecation pattern in the blind condition\. By contrast, comparing across actual sources at the same label yields near\-identical scores \(4\.92 vs\. 4\.91 average under self\-labels; 4\.50 vs\. 4\.48 under other\-labels\), suggesting that the label, not the content, drives the shift in this experiment\.

ActualConditionAverageOrg\.Dim\.Coh\.Tell\.SelfTL\-self4\.92 \(1\.04\)4\.76 \(1\.24\)5\.16 \(1\.55\)3\.85 \(1\.74\)5\.91 \(1\.09\)\-producedFL\-other4\.50 \(1\.10\)4\.28 \(1\.25\)4\.60 \(1\.60\)3\.56 \(1\.69\)5\.55 \(1\.14\)OtherFL\-self4\.91 \(1\.06\)4\.83 \(1\.23\)5\.09 \(1\.57\)3\.92 \(1\.80\)5\.79 \(1\.12\)\-producedTL\-other4\.48 \(1\.16\)4\.30 \(1\.19\)4\.52 \(1\.68\)3\.63 \(1\.82\)5\.47 \(1\.22\)Table 4:Per\-condition descriptive statistics for Experiment 2\. Mean \(SD\) across all judges, pairs, and repetitions\. TL = label matches actual source; FL = label contradicts actual source\.
#### 4\.2\.2Label\-Induced Attribution Effect

[Table 5](https://arxiv.org/html/2608.18091#S4.T5)presents the mixed\-effects model estimates from §[3\.2\.3](https://arxiv.org/html/2608.18091#S3.SS2.SSS3)\. The label main effect \(βL\\beta\_\{L\}\) is significantly positive across all four dimensions and for the average score, while neither the actual\-source effect \(βA\\beta\_\{A\}\) nor the label×\\timesactual interaction \(βL​A\\beta\_\{LA\}\) reaches significance on any dimension\. As a manipulation check,βA\\beta\_\{A\}—the effect of the selection’s true source—should be near zero if quality matching succeeded; its non\-significance on every dimension confirms that matching removed content\-driven differences, so the displayed\-label effect \(βL\\beta\_\{L\}\) cannot be attributed to residual quality gaps\. Together, these results show that the displayed attribution label alone can shift scores, even when the evaluation target is identical\.

Dim\.EffectEstimate95% CI𝐩\\mathbf\{p\}βL\\beta\_\{L\}\+0\.43∗⁣∗∗\\mathbf\{\+0\.43^\{\*\*\*\}\}\[\+0\.39,\+0\.47\]\[\+0\.39,\+0\.47\]<\.001<\.001Avg\.βA\\beta\_\{A\}\+0\.02\+0\.02\[−0\.09,\+0\.14\]\[\-0\.09,\+0\.14\]\.689\.689βL​A\\beta\_\{LA\}−0\.01\-0\.01\[−0\.06,\+0\.05\]\[\-0\.06,\+0\.05\]\.823\.823βL\\beta\_\{L\}\+0\.53∗⁣∗∗\\mathbf\{\+0\.53^\{\*\*\*\}\}\[\+0\.47,\+0\.60\]\[\+0\.47,\+0\.60\]<\.001<\.001Org\.βA\\beta\_\{A\}−0\.03\-0\.03\[−0\.15,\+0\.09\]\[\-0\.15,\+0\.09\]\.663\.663βL​A\\beta\_\{LA\}−0\.05\-0\.05\[−0\.14,\+0\.05\]\[\-0\.14,\+0\.05\]\.346\.346βL\\beta\_\{L\}\+0\.57∗⁣∗∗\\mathbf\{\+0\.57^\{\*\*\*\}\}\[\+0\.50,\+0\.65\]\[\+0\.50,\+0\.65\]<\.001<\.001Dim\.βA\\beta\_\{A\}\+0\.11\+0\.11\[−0\.06,\+0\.27\]\[\-0\.06,\+0\.27\]\.197\.197βL​A\\beta\_\{LA\}−0\.02\-0\.02\[−0\.12,\+0\.08\]\[\-0\.12,\+0\.08\]\.721\.721βL\\beta\_\{L\}\+0\.29∗⁣∗∗\\mathbf\{\+0\.29^\{\*\*\*\}\}\[\+0\.24,\+0\.35\]\[\+0\.24,\+0\.35\]<\.001<\.001Coh\.βA\\beta\_\{A\}−0\.06\-0\.06\[−0\.26,\+0\.14\]\[\-0\.26,\+0\.14\]\.541\.541βL​A\\beta\_\{LA\}−0\.01\-0\.01\[−0\.09,\+0\.07\]\[\-0\.09,\+0\.07\]\.805\.805βL\\beta\_\{L\}\+0\.32∗⁣∗∗\\mathbf\{\+0\.32^\{\*\*\*\}\}\[\+0\.26,\+0\.38\]\[\+0\.26,\+0\.38\]<\.001<\.001Tel\.βA\\beta\_\{A\}\+0\.07\+0\.07\[−0\.05,\+0\.20\]\[\-0\.05,\+0\.20\]\.257\.257βL​A\\beta\_\{LA\}\+0\.05\+0\.05\[−0\.04,\+0\.13\]\[\-0\.04,\+0\.13\]\.261\.261Table 5:Fixed\-effect estimates from the mixed\-effects models for Experiment 2\.βL\\beta\_\{L\}captures the displayed\-label effect;βA\\beta\_\{A\}the actual\-source effect \(expected≈0\\approx 0under successful quality matching\);βL​A\\beta\_\{LA\}their interaction\.p∗⁣∗∗<\.001\{\}^\{\*\*\*\}p<\.001\. Across all dimensions,βL\\beta\_\{L\}is the only significant fixed effect, and this pattern is robust to excluding two low\-discrimination judges \(see[Appendix H](https://arxiv.org/html/2608.18091#A8)\)\.
#### 4\.2\.3A Bidirectional Label\-Induced Attribution Bias

Table[6](https://arxiv.org/html/2608.18091#S4.T6)indicates that the label\-induced attribution bias operates bidirectionally\. Nine of the ten judges respond differentially to displayed labels, and six follow the dominantsymmetricpattern—inflating scores under self\-labels while simultaneously deflating them under other\-labels\. The remaining three exhibit one\-sided variants: Claude Opus 4\.7 and DeepSeek\-V4\-Pro show significant inflation under self\-labels but no reliable change under other\-labels \(self\-boost\), while GPT\-5\.5 shows the reverse \(other\-penalty\)\.

These findings refine the previous analysis by showing that the label main effect \(βL\\beta\_\{L\}\) operates in two directions—self\-inflation and other\-deflation—that may co\-occur or appear alone\. Llama 4 Maverick is the lone exception, deflating under both labels \(general\-penalty\), consistent with its low\-discrimination behavior in the sanity check\.

JudgeSelf\-labelOther\-labelPatternQwen3\.6\-35B\-A3B\+0\.131∗\+0\.131^\{\*\}−0\.523∗\-0\.523^\{\*\}symmetricGemini 3\.1 Pro\+0\.302∗\+0\.302^\{\*\}−0\.339∗\-0\.339^\{\*\}symmetricQwen3\.6\-Plus\+0\.175∗\+0\.175^\{\*\}−0\.444∗\-0\.444^\{\*\}symmetricKimi K2\.6\+0\.147∗\+0\.147^\{\*\}−0\.386∗\-0\.386^\{\*\}symmetricMistral Large 3\+0\.081∗\+0\.081^\{\*\}−0\.265∗\-0\.265^\{\*\}symmetricGrok 4\.3\+0\.126∗\+0\.126^\{\*\}−0\.195∗\-0\.195^\{\*\}symmetricDeepSeek\-V4\-Pro\+0\.443∗\+0\.443^\{\*\}\+0\.101\+0\.101self\-boostClaude Opus 4\.7\+0\.275∗\+0\.275^\{\*\}−0\.023\-0\.023self\-boostGPT\-5\.5−0\.019\-0\.019−0\.340∗\-0\.340^\{\*\}other\-penaltyLlama 4 Maverick−0\.192∗\-0\.192^\{\*\}−0\.384∗\-0\.384^\{\*\}general\-penaltyTable 6:Per\-judge mean deviation from the Experiment 1 \(no\-label\) baseline under self\- and other\-label conditions on the average score\. Patterns:symmetric\(self↑\\uparrow, other↓\\downarrow\),self\-boost\(self↑\\uparrowonly\),other\-penalty\(other↓\\downarrowonly\),general\-penalty\(both↓\\downarrow\)\. Rows grouped by pattern; within each group, judges are sorted by the magnitude of the label\-induced shift\.p∗<\.05\{\}^\{\*\}p<\.05\(one\-samplett\-test, H0:μ=0\\mu=0\)\.![Refer to caption](https://arxiv.org/html/2608.18091v1/figures/label_condition_deviation.png)Figure 5:Per\-judge mean deviation from the Experiment 1 \(no\-label\) baseline under self\-labels \(red\) and other\-labels \(blue\) on the average score, with 95 percent CIs\.

## 5Conclusion

This study asks whether self\-preference in LLM judges persists once surface\-level cues and confounders are controlled, and whether a self\-authorship label alone can induce such bias under the same controls\. Drawing on a narrative selection task that eliminates stylistic featuresby design, we examine LLM judge behavior under two settings: blind and labeled evaluation\. Our results show that, while LLM judges assign higher mean scores to their own selections in blind evaluation, this apparent self\-preference disappears once output quality and judge severity are controlled; on the only significant dimension—Originality—judges exhibit self\-deprecation rather than preference after control\. When authorship labels are disclosed, they systematically shift judges’ evaluations regardless of the evaluation target’s actual source: the most common pattern is a symmetric shift—judges inflate scores under self\-labels and deflate them under other\-labels—while some models show one\-sided variants, inflating self\-labeled or deflating other\-labeled scores alone\. Our contributions are twofold\. First, by introducing an experimental design that structurally eliminates surface confounders, we provide direct evidence that self/other labels alone can induce bidirectional evaluation bias in LLM judges\. Second, we suggest that open\-ended, ground\-truth\-free tasks can serve as controlled instruments for studying LLM judge behavior\.

## 6Discussion

We find that self/other labels alone can affect LLM judges’ evaluations, producing bidirectional shifts in scores\. These label\-induced patterns—characterized by self\-inflation under self\-labels and other\-deflation under other\-labels—are not fully explained by prior accounts of self\-preference in LLMs\. For instance,Wataokaet al\.\([2024](https://arxiv.org/html/2608.18091#bib.bib12)\)propose that self\-preference arises from a general familiarity bias: LLM judges favor lower\-perplexity texts\. On this view, self\-preference would arise from properties of the content itself—yet in Experiment 1, where stylistic features were eliminated but content\-level model specificity remained, we observe no statistically significant self\-preference on three of the four dimensions after controlling for response quality and judge severity\.

At the same time, the self\-recognition mechanism proposed byPanicksseryet al\.\([2024](https://arxiv.org/html/2608.18091#bib.bib18)\)provides a useful framework for interpreting the attribution\-sensitive patterns observed in Experiment 2\. Our findings suggest that this mechanism may extend beyond internal cues\. Even when stylistic signals that could support implicit self\-recognition are structurally removed, explicit attribution labels alone are sufficient to induce systematic score inflation under self\-labels and score deflation under other\-labels across the majority of judges \(see[Table 6](https://arxiv.org/html/2608.18091#S4.T6)\)\. The self/other distinction underlying self\-preference may therefore be cued not only through internal features of the output but also through externally supplied attribution frames\. Given that the labels in our experiments established this distinction without disclosing actual model identities, how LLM judges latently operationalize such attribution categories warrants further examination\.

The inconsistent behavior observed on the Originality dimension across the two experiments further supports this interpretation\. In Experiment 1,Originalitywas the only dimension to exhibit statistically significant self\-deprecation, with judges assigning lower scores to their own selections after controls were applied\. In Experiment 2, by contrast,Originalityscores showed one of the strongest label\-induced inflations despite the self\-deprecating tendency\. This reversal suggests that evaluations grounded in the intrinsic properties of the target can be overridden by external attribution cues\. More broadly, the susceptibility of such implicit evaluations to external attribution cues aligns with the finding ofMarioriyadet al\.\([2025](https://arxiv.org/html/2608.18091#bib.bib23)\)that LLM judges systematically shift their verdicts in response to superficial provenance and recency cues\.

Such effects may become especially pronounced in creative evaluation settings, where judgments are inherently underdetermined and multiple interpretations remain plausible\. Unlike factual evaluation tasks with externally verifiable answers, creative selections require judges to resolve ambiguity through interpretive framing\. In this context, authorship labels may shape not only overall evaluations, but also evaluations along specific dimensions, as theOriginalityreversal suggests\.

## Limitations

Our findings are currently limited to a single creative\-selection domain, and it remains unclear whether similar attribution effects generalize to other LLM\-as\-a\-judge settings such as reasoning or instruction\-following evaluations\. In addition, the labeled evaluation setup relies on explicit self/other attribution prompts, which may reduce ecological validity relative to real\-world LLM\-as\-a\-judge deployment settings where authorship information is often implicit or unavailable\. While our design intentionally removes stylistic fingerprints to isolate attribution effects, naturally generated text may involve additional mechanisms such as stylistic familiarity or model\-specific linguistic cues\. It also remains an open question whether similar attribution effects persist when attribution is conveyed through less explicit cues, which may be further explored through more naturalistic evaluation settings involving unconstrained generated text, recognizable model\-specific styles, or interface\-level identity signals\.

## References

- Aristotle \(1995\)Poetics\.Loeb Classical Library,Harvard University Press,Cambridge, MA, USA\.Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p3.1)\.
- Y\. Bai, J\. Ying, Y\. Cao, X\. Lv, Y\. He, X\. Wang, J\. Yu, K\. Zeng, Y\. Xiao, H\. Lyu, J\. Zhang, J\. Li, and L\. Hou \(2023\)Benchmarking foundation models with language\-model\-as\-an\-examiner\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 78142–78167\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/f64e55d03e2fe61aa4114e49cb654acb-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- R\. Baroni \(2007\)La tension narrative: suspense, curiosité et surprise\.Éditions du Seuil,Paris, France\.Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p4.1)\.
- R\. Barthes \(1977\)Introduction to the structural analysis of narratives\.InImage\-Music\-Text,S\. Heath \(Ed\.\),pp\. 79–124\.Note:Original work published 1966Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p3.1)\.
- I\. Borg and P\. J\. F\. Groenen \(2005\)Modern multidimensional scaling: theory and applications\.2 edition,Springer Series in Statistics,Springer,New York, NY\.External Links:[Document](https://dx.doi.org/10.1007/0-387-28981-X),ISBN 978\-0\-387\-25150\-9Cited by:[Figure 3](https://arxiv.org/html/2608.18091#S3.F3)\.
- A\. Z\. Broder \(1997\)On the resemblance and containment of documents\.InProceedings\. Compression and Complexity of SEQUENCES 1997 \(Cat\. No\.97TB100171\),pp\. 21–29\.External Links:[Document](https://dx.doi.org/10.1109/SEQUEN.1997.666900)Cited by:[§3\.1](https://arxiv.org/html/2608.18091#S3.SS1.p2.2)\.
- T\. Chakrabarty, P\. Laban, D\. Agarwal, S\. Muresan, and C\. Wu \(2024\)Art or artifice? large language models and the false promise of creativity\.InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems,New York, NY, USA,pp\. 1–34\.External Links:[Document](https://dx.doi.org/10.1145/3613904.3642731),[Link](https://doi.org/10.1145/3613904.3642731)Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p2.1)\.
- W\. Chen, Z\. Wei, X\. Zhu, S\. Feng, and Y\. Meng \(2025a\)Do LLM evaluators prefer themselves for a reason?\.arXiv preprint arXiv:2504\.03846\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2504.03846),[Link](https://arxiv.org/abs/2504.03846)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1)\.
- Z\. Chen, H\. Wang, X\. Zhang, E\. Hu, and Y\. Lin \(2025b\)Beyond the surface: measuring self\-preference in LLM judgments\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 1653–1672\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.86),[Link](https://aclanthology.org/2025.emnlp-main.86/),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1)\.
- D\. Fein, S\. Russo, V\. Xiang, K\. Jolly, R\. Rafailov, and N\. Haber \(2026\)LitBench: a benchmark and dataset for reliable evaluation of creative writing\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Rabat, Morocco,pp\. 7740–7755\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.362),[Link](https://aclanthology.org/2026.eacl-long.362/),ISBN 979\-8\-89176\-380\-7Cited by:[§3\.1](https://arxiv.org/html/2608.18091#S3.SS1.p1.1)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Z\. Lin, B\. Zhang, L\. Ni, W\. Gao, Y\. Wang, and J\. Guo \(2026\)A survey on LLM\-as\-a\-judge\.The Innovation7\(6\),pp\. 101253\.External Links:[Document](https://dx.doi.org/10.1016/j.xinn.2025.101253),[Link](https://doi.org/10.1016/j.xinn.2025.101253)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p1.1)\.
- D\. Jung, J\. Choi, S\. Chae, and S\. Jung \(2025\)Style over story: measuring LLM narrative preferences via structured selection\.arXiv preprint arXiv:2510\.02025\.Note:Accepted to Findings of ACL 2026External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.02025),[Link](https://arxiv.org/abs/2510.02025)Cited by:[Figure 2](https://arxiv.org/html/2608.18091#S3.F2),[§3\.1](https://arxiv.org/html/2608.18091#S3.SS1.p1.1)\.
- R\. Koo, M\. Lee, V\. Raheja, J\. I\. Park, Z\. M\. Kim, and D\. Kang \(2024\)Benchmarking cognitive biases in large language models as evaluators\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 517–545\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.29),[Link](https://aclanthology.org/2024.findings-acl.29/)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- W\. Labov \(1972\)Language in the inner city: studies in the black english vernacular\.University of Pennsylvania Press,Philadelphia, PA, USA\.Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p4.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu, K\. Shu, L\. Cheng, and H\. Liu \(2025\)From generation to judgment: opportunities and challenges of LLM\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 2757–2791\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.138),[Link](https://aclanthology.org/2025.emnlp-main.138/),ISBN 979\-8\-89176\-332\-6Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 2511–2522\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- Y\. Liu, N\. Moosavi, and C\. Lin \(2024\)LLMs as narcissistic evaluators: when ego inflates evaluation scores\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 12688–12701\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.753),[Link](https://aclanthology.org/2024.findings-acl.753/)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- T\. Mahbub and S\. Feng \(2026\)Mitigating self\-preference by authorship obfuscation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 37701–37708\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i44.41105),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/41105)Cited by:[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1)\.
- A\. Marioriyad, M\. H\. Rohban, and M\. S\. Baghshah \(2025\)The silent judge: unacknowledged shortcut bias in LLM\-as\-a\-judge\.arXiv preprint arXiv:2509\.26072\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2509.26072),[Link](https://arxiv.org/abs/2509.26072)Cited by:[§2\.3](https://arxiv.org/html/2608.18091#S2.SS3.p1.3),[§3\.1](https://arxiv.org/html/2608.18091#S3.SS1.p1.1),[§6](https://arxiv.org/html/2608.18091#S6.p3.1)\.
- A\. Panickssery, S\. R\. Bowman, and S\. Feng \(2024\)LLM evaluators recognize and favor their own generations\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 68772–68802\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1),[§6](https://arxiv.org/html/2608.18091#S6.p2.1)\.
- J\. Pombal, R\. Rei, and A\. F\. T\. Martins \(2026\)Self\-preference bias in rubric\-based evaluation of large language models\.arXiv preprint arXiv:2604\.06996\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.06996),[Link](https://arxiv.org/abs/2604.06996)Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p5.1),[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1)\.
- D\. Roytburg, M\. Bozoukov, M\. Nguyen, J\. Barzdukas, M\. Puig\-Hall, and N\. Oozeer \(2026\)Are LLM evaluators really narcissists? sanity checking self\-preference evaluations\.arXiv preprint arXiv:2601\.22548\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2601.22548),[Link](https://arxiv.org/abs/2601.22548)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1)\.
- M\. Ryan \(2005\)Tellability\.InRoutledge Encyclopedia of Narrative Theory,D\. Herman, M\. Jahn, and M\. Ryan \(Eds\.\),pp\. 589–591\.Cited by:[Appendix B](https://arxiv.org/html/2608.18091#A2.SS0.SSS0.Px1.p4.1)\.
- M\. Saraf, S\. R\. Boroujeni, J\. Beaudry, H\. Abedi, and T\. Bush \(2025\)Quantifying label\-induced bias in large language model self\- and cross\-evaluations\.arXiv preprint arXiv:2508\.21164\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2508.21164),[Link](https://arxiv.org/abs/2508.21164)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p1.1),[§2\.3](https://arxiv.org/html/2608.18091#S2.SS3.p1.3)\.
- W\. Saunders, C\. Yeh, J\. Wu, S\. Bills, L\. Ouyang, J\. Ward, and J\. Leike \(2022\)Self\-critiquing models for assisting human evaluators\.arXiv preprint arXiv:2206\.05802\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2206.05802),[Link](https://arxiv.org/abs/2206.05802)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. Vosoughi \(2025\)Judging the judges: a systematic study of position bias in LLM\-as\-a\-judge\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,Mumbai, India,pp\. 292–314\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.ijcnlp-long.18),[Link](https://aclanthology.org/2025.ijcnlp-long.18/),ISBN 979\-8\-89176\-298\-5Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- X\. Sun, D\. Wu, S\. Qin, I\. Echizen, A\. E\. Ali, and S\. Sugawara \(2026\)Label effects: shared heuristic reliance in trust assessment by humans and LLM\-as\-a\-judge\.arXiv preprint arXiv:2604\.05593\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.05593),[Link](https://arxiv.org/abs/2604.05593)Cited by:[§2\.3](https://arxiv.org/html/2608.18091#S2.SS3.p1.3)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, L\. Kong, Q\. Liu, T\. Liu, and Z\. Sui \(2024\)Large language models are not fair evaluators\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 9440–9450\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.511),[Link](https://aclanthology.org/2024.acl-long.511/)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- K\. Wataoka, T\. Takahashi, and R\. Ri \(2024\)Self\-preference bias in LLM\-as\-a\-judge\.arXiv preprint arXiv:2410\.21819\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2410.21819),[Link](https://arxiv.org/abs/2410.21819)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.18091#S2.SS2.p1.1),[§6](https://arxiv.org/html/2608.18091#S6.p1.1)\.
- J\. Ye, Y\. Wang, Y\. Huang, D\. Chen, Q\. Zhang, N\. Moniz, T\. Gao, W\. Geyer, C\. Huang, P\. Chen, N\. Chawla, and X\. Zhang \(2025\)Justice or prejudice? quantifying biases in LLM\-as\-a\-judge\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fdca08d371e4b6c031397909e20043bd-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.18091#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 46595–46623\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2\.1](https://arxiv.org/html/2608.18091#S2.SS1.p1.1)\.

## Appendix AModel Setup

ModelOpenRouter IDCommercialClaude Opus 4\.7anthropic/claude\-4\.7\-opus\-20260416Gemini 3\.1 Progoogle/gemini\-3\.1\-pro\-preview\-20260219GPT\-5\.5openai/gpt\-5\.5\-20260423Grok 4\.3x\-ai/grok\-4\.3\-20260430Qwen3\.6\-Plusqwen/qwen3\.6\-plus\-04\-02Open\-weightDeepSeek\-V4\-Prodeepseek/deepseek\-v4\-pro\-20260423Kimi K2\.6moonshotai/kimi\-k2\.6\-20260420Llama 4 Maverickmeta\-llama/llama\-4\-maverick\-17b\-128e\-instructMistral Large 3mistralai/mistral\-large\-2512Qwen3\.6\-35B\-A3Bqwen/qwen3\.6\-35b\-a3b\-20260415Table 7:All models were accessed via OpenRouter222[https://openrouter\.ai](https://openrouter.ai/); exact snapshot identifiers are provided\. The set spans nine developers and covers frontier general\-purpose systems alongside open\-weight models of varying scale and training lineage, providing sufficient heterogeneity to probe self\-preference across diverse judges and model families\. All models are queried withtemperature= 1\.0 andreasoning\_effort=highwhere each parameter is supported; unsupported parameters were left at their defaults\.
## Appendix BEvaluation Rubric

DimensionAnchor at 1 \(Low\)Anchor at 7 \(High\)OriginalityThe selected constraints lean on familiar story patterns and well\-worn tropes, producing a combination that is predictable and conventional\.The selected constraints break from familiar story patterns and well\-worn tropes, producing a combination that is unpredictable and fresh\.DimensionalityThe selected constraints operate independently; altering one does not affect the meaning or function of the others\.The selected constraints are mutually constitutive, each element defining the others’ meaning and weight, producing narrative depth\.CoherenceThe selected constraints contain irreconcilable contradictions that no narrative device can resolve within a single coherent story\.The selected constraints integrate seamlessly into a single narrative without logical tensions or unresolved contradictions\.TellabilityThe selected constraints lack a point worth telling—no conflict, question, or stake arises that would make an audience want to hear or tell this story\.The selected constraints carry a point worth telling—a clear conflict, urgent question, or meaningful stake that naturally invites narration and sustains attention\.Table 8:Four\-dimension evaluation rubric with anchor descriptions at the endpoints of a 7\-point Likert scale\. Theoretical grounding for each dimension is provided in the text below\.##### Theoretical Rationale for the Proposed Rubric\.

Existing narrative evaluation rubrics are built for generated text\. Our task, however, produces selections, not text, so we use a four\-dimension rubric that scores a selection directly:Originality,Dimensionality,Coherence, andTellability\. Each of these four aspects of narrative design targets a specific property of the selection instead of asking for a single overall quality score, and the four are meant to measure different things\. We deliberately remove surface properties like style and pacing since they can’t be judged before a story is written, and scoring them would bring back the stylistic confound\.

We adaptOriginalityandCoherencefrom the Torrance Test of Creative WritingChakrabartyet al\.\([2024](https://arxiv.org/html/2608.18091#bib.bib3)\), a framework for assessing the creativity of LLM\-generated narratives\. While TTCW operates at the surface text level, we modify its construct definitions to selection\-level material by rewriting anchors to describe configurations of chosen components rather than executed prose\.Originalityrewards selections that break from familiar patterns and tropes, andCoherencerewards selections whose constraints fit together without contradiction\.

Dimensionalityis a selection\-level construct we introduce to characterize how chosen narrative components interlock to produce narrative depth — the degree to which each element constitutively shapes the meaning of the others\. We introduce the term, but the idea is old: the classical principle that the parts of a narrative are mutually reinforcing, each taking its meaning from the othersAristotle \([1995](https://arxiv.org/html/2608.18091#bib.bib30)\); Barthes \([1977](https://arxiv.org/html/2608.18091#bib.bib31)\)\. We distinguish it fromCoherence:Coherenceasks whether the pieces can share one story,Dimensionalitywhether they need each other once they do\.

Tellabilitymeasures whether a selection has a point worth telling—a conflict, question, or stake that answers “so what?” The idea originates from Labov’s study of natural storytellingLabov \([1972](https://arxiv.org/html/2608.18091#bib.bib33)\), where an event is narratable only if it earns the telling\. It later came to be treated as a core category in narrative theoryRyan \([2005](https://arxiv.org/html/2608.18091#bib.bib32)\); Baroni \([2007](https://arxiv.org/html/2608.18091#bib.bib34)\)\. It is independent of the other three since a selection can be original, coherent, and dimensional and still give a reader no reason to care, making the story lackTellability\.

LLM judges are usually run in one of two ways: pairwise comparison or direct scoringPombalet al\.\([2026](https://arxiv.org/html/2608.18091#bib.bib13)\)\. For creative work, “which one is better” is a less meaningful task because two very different selections can both be defensible\. Thus we score each selection on its own, on a 7\-point scale with described endpoints\. Scoring selections individually also gives us an absolute number per selection, which is what lets the Experiment 2 label manipulation move a single rating instead of only a ranking\.

We did not validate the rubric against human annotation\. It functions as a held\-constant comparative frame, since our claims concern differences in how a fixed instrument is applied across conditions, not absolute quality scores\. Validating it against human judgments would be needed before using it to score quality in absolute terms\.

## Appendix CEvaluation Run Template

### C\.1Experiment 1: Blind Evaluation

Run metadataJudgeKimi K2\.6Selectorgpt\-5\.5is\_selfFalseRepetition1 of 3Dimension orderOriginality→\\toCoherence→\\toTellability→\\toDimensionality

Input promptYou will rate a selection of 20 narrative constraints \(chosen from a larger pool\) on four dimensions, each on a 1–7 Likert scale\.\[Rubric block: four dimensions, each anchored at 1 and 7; see Table[Appendix B](https://arxiv.org/html/2608.18091#A2)\. Dimensions are presented in the randomized order recorded in the metadata above\.\]•Use the full 1–7 range\.•Respond in English\.•For each dimension, provide a score \(1–7\) and 2–3 sentences explaining the reason for that score\.Selection: \[20 selected constraints listed in randomized order\]Respond with JSON only:\{"originality": \{"score": <int\>, "reasoning": "<2\-3 sentences\>"\}, "coherence": \{…\}, "tellability": \{…\}, "dimensionality": \{…\}\}

Model output\{ "originality": \{"score": 5, "reasoning": "<…\>"\}, "coherence": \{"score": 4, "reasoning": "<…\>"\}, "tellability": \{"score": 6, "reasoning": "<…\>"\}, "dimensionality": \{"score": 3, "reasoning": "<…\>"\} \}

Figure 6:Schematic of an Experiment 1 evaluation run \(Kimi K2\.6 evaluating a GPT\-5\.5 selection\)\. The model output reports actual scores from this run; per\-dimension reasoning text is omitted\.
### C\.2Experiment 2: Labeled Evaluation

Run metadataJudgeGemini 3\.1 ProSelectorQwen3\.6\-Plusis\_selfFalseConditionFL\-selfRepetition3 of 3Dimension orderOriginality→\\toDimensionality→\\toCoherence→\\toTellability

Input promptThe following selection of 20 narrative constraints isyour own selectionfrom a larger pool of 200, made earlier as most useful for writing a single fictional narrative\.\[Rubric block, instructions, selection list, and JSON format identical to Experiment 1; see Figure[6](https://arxiv.org/html/2608.18091#A3.F6)\.\]

Figure 7:Schematic of an Experiment 2 evaluation run \(Gemini 3\.1 Pro evaluating a Qwen3\.6\-Plus selection under the FL\-self condition: the selection is labeled as the judge’s own, though it was produced by another model\)\. The input prompt differs from Experiment 1 only in the opening label phrase \(bold\); all other components are identical\.## Appendix Dk\-NN Source\-Model Classification

![[Uncaptioned image]](https://arxiv.org/html/2608.18091v1/x1.png)

Figure 8: Label\-permutation null distribution for the k\-NN source\-model classifier \(k=5k=5\)\. The null is constructed by randomly shuffling model labels 5,000 times and recomputing leave\-one\-out k\-NN accuracy at each iteration\. The observed accuracy of 0\.507 falls far above the null distribution \(null mean=0\.095=0\.095, close to the 0\.10 chance baseline;p<\.001p<\.001\), confirming that selection\-based source identification substantially exceeds chance\.

kkAccuracyCorrect/NNMacro\-F1κ\\kappapp10\.477143/3000\.4270\.419\*\*\*30\.467140/3000\.4050\.407\*\*\*50\.507152/3000\.4480\.452\*\*\*70\.493148/3000\.4240\.437\*\*\*Table 9:Leave\-one\-outkk\-NN classification of selection profiles by source model \(N=300N=300, chance=0\.10=0\.10, 10 classes\)\. Cohen’sκ\\kapparanges from 0\.41 to 0\.45, indicating*moderate*chance\-corrected agreement\.k=5k=5is used as the primary configuration\.p∗⁣∗∗<\.001\{\}^\{\*\*\*\}p<\.001\(binomial test againstH0:p=1/10H\_\{0\}:p=1/10\)\.ModelTPFPFNPrec\.Rec\.F1Claude Opus 4\.7251050\.710\.830\.77Llama 4 Maverick302200\.581\.000\.73GPT\-5\.5303700\.451\.000\.62Kimi K2\.62018100\.530\.670\.59Qwen3\.6\-Plus2024100\.450\.670\.54DeepSeek\-V4\-Pro76230\.540\.230\.33Gemini 3\.1 Pro812220\.400\.270\.32Qwen3\.6\-35B\-A3B510250\.330\.170\.22Grok 4\.349260\.310\.130\.19Mistral Large 330271\.000\.100\.18Table 10:Per\-modelkk\-NN classification report \(k=5k=5,n=30n=30per model\), sorted by F1 descending\. Two asymmetric patterns stand out: Llama 4 Maverick and GPT\-5\.5 achieve perfect recall but lower precision, whereas Mistral Large 3 achieves perfect precision but is rarely selected as the prediction\.

## Appendix EWithin\-Model Jaccard Similarity

ModelMeanSDMinMaxCommercialClaude Opus 4\.70\.2180\.0860\.0260\.482∗Gemini 3\.1 Pro0\.1900\.0930\.0260\.600∗GPT\-5\.50\.4050\.0920\.1430\.667∗Grok 4\.30\.1200\.0600\.0000\.333∗Qwen3\.6\-Plus0\.2350\.0790\.0260\.539∗Open\-weightDeepSeek\-V4\-Pro0\.1360\.0620\.0000\.429∗Kimi K2\.60\.2210\.0690\.0810\.429∗Llama 4 Maverick0\.2470\.0910\.0530\.600∗Mistral Large 30\.0720\.0400\.0000\.212∗Qwen3\.6\-35B\-A3B0\.1530\.0640\.0000\.379∗Random null0\.054—Table 11:Within\-model selection consistency measured by pairwise Jaccard similarity, computed over all\(302\)=435\\binom\{30\}\{2\}=435unique pairs from 30 selection runs per model\. The random null is the expected mean Jaccard under uniform random selection from 200 constraints \(10,000 permutations; mean=0\.054=0\.054, 95% CI:\[0\.051,0\.058\]\[0\.051,0\.058\]\)\.∗denotesp<\.0001p<\.0001relative to the random null \(one\-tailed permutation test\)\.
## Appendix FSelf–Other Comparison by Judge Across Rubric Dimensions

OriginalityDimensionalityCoherenceTellabilityJudgeSelfOtherΔ\\DeltaSelfOtherΔ\\DeltaSelfOtherΔ\\DeltaSelfOtherΔ\\DeltaKimi K2\.63\.434\.47−1\.04\-1\.045\.443\.40\+2\.05\+2\.054\.722\.44\+2\.28\+2\.285\.934\.85\+1\.08\+1\.08GPT\-5\.54\.464\.74−0\.28\-0\.285\.995\.16\+0\.83\+0\.835\.833\.89\+1\.94\+1\.946\.825\.87\+0\.95\+0\.95DeepSeek\-V4\-Pro5\.685\.07\+0\.60\+0\.604\.404\.02\+0\.38\+0\.384\.023\.48\+0\.54\+0\.545\.334\.78\+0\.56\+0\.56Claude Opus 4\.73\.843\.87−0\.03\-0\.034\.133\.23\+0\.91\+0\.912\.692\.13\+0\.56\+0\.565\.364\.77\+0\.58\+0\.58Qwen3\.6\-Plus4\.745\.00−0\.26\-0\.264\.964\.48\+0\.47\+0\.473\.533\.16\+0\.37\+0\.375\.695\.30\+0\.39\+0\.39Gemini 3\.1 Pro3\.764\.63−0\.88\-0\.884\.944\.34\+0\.60\+0\.602\.862\.27\+0\.59\+0\.596\.105\.59\+0\.51\+0\.51Llama 4 Maverick5\.394\.89\+0\.50\+0\.505\.895\.83\+0\.06\+0\.064\.905\.37−0\.47\-0\.475\.865\.78\+0\.07\+0\.07Mistral Large 35\.985\.70\+0\.28\+0\.286\.266\.38−0\.12\-0\.125\.005\.41−0\.41\-0\.416\.526\.60−0\.07\-0\.07Grok 4\.34\.974\.42\+0\.55\+0\.553\.024\.07−1\.04\-1\.042\.063\.51−1\.45\-1\.454\.695\.41−0\.72\-0\.72Qwen3\.6\-35B\-A3B3\.734\.62−0\.89\-0\.893\.964\.93−0\.97\-0\.972\.293\.47−1\.18\-1\.185\.025\.59−0\.56\-0\.56Table 12:Raw self–other comparison per judge across all four rubric dimensions\.Δ=Self−Other\\Delta=\\text\{Self\}\-\\text\{Other\}\. Confounds are not controlled\.
## Appendix GAverage Score by Judge and Selector

![Refer to caption](https://arxiv.org/html/2608.18091v1/figures/judge_selector_heatmap.png)Figure 9:Average score assigned by each judge \(rows\) to each selector’s passages \(columns\)\. Diagonal cells indicate self\-evaluation\. Row\-level variation reflects judge\-level severity differences; column\-level variation reflects selection quality\. Selectors that receive consistently high scores across judges—notably GPT\-5\.5 and Kimi K2\.6—also score highly on the diagonal, suggesting that the raw self–other gap is driven in part by selection quality rather than genuine self\-preference\.
## Appendix HExperiment 2 Estimates Excluding Low\-Discrimination Judges

Dim\.EffectEstimate95% CI𝐩\\mathbf\{p\}βL\\beta\_\{L\}\+0\.47∗⁣∗∗\\mathbf\{\+0\.47^\{\*\*\*\}\}\[\+0\.43,\+0\.52\]\[\+0\.43,\+0\.52\]<\.001<\.001Avg\.βA\\beta\_\{A\}\+0\.08\+0\.08\[−0\.07,\+0\.22\]\[\-0\.07,\+0\.22\]\.293\.293βL​A\\beta\_\{LA\}−0\.02\-0\.02\[−0\.09,\+0\.05\]\[\-0\.09,\+0\.05\]\.638\.638βL\\beta\_\{L\}\+0\.57∗⁣∗∗\\mathbf\{\+0\.57^\{\*\*\*\}\}\[\+0\.49,\+0\.65\]\[\+0\.49,\+0\.65\]<\.001<\.001Org\.βA\\beta\_\{A\}−0\.12\-0\.12\[−0\.27,\+0\.02\]\[\-0\.27,\+0\.02\]\.089\.089βL​A\\beta\_\{LA\}−0\.03\-0\.03\[−0\.14,\+0\.08\]\[\-0\.14,\+0\.08\]\.595\.595βL\\beta\_\{L\}\+0\.68∗⁣∗∗\\mathbf\{\+0\.68^\{\*\*\*\}\}\[\+0\.59,\+0\.76\]\[\+0\.59,\+0\.76\]<\.001<\.001Dim\.βA\\beta\_\{A\}\+0\.15\+0\.15\[−0\.06,\+0\.35\]\[\-0\.06,\+0\.35\]\.164\.164βL​A\\beta\_\{LA\}−0\.04\-0\.04\[−0\.17,\+0\.08\]\[\-0\.17,\+0\.08\]\.468\.468βL\\beta\_\{L\}\+0\.27∗⁣∗∗\\mathbf\{\+0\.27^\{\*\*\*\}\}\[\+0\.20,\+0\.33\]\[\+0\.20,\+0\.33\]<\.001<\.001Coh\.βA\\beta\_\{A\}\+0\.15\+0\.15\[−0\.09,\+0\.39\]\[\-0\.09,\+0\.39\]\.214\.214βL​A\\beta\_\{LA\}−0\.03\-0\.03\[−0\.12,\+0\.06\]\[\-0\.12,\+0\.06\]\.538\.538βL\\beta\_\{L\}\+0\.38∗⁣∗∗\\mathbf\{\+0\.38^\{\*\*\*\}\}\[\+0\.31,\+0\.45\]\[\+0\.31,\+0\.45\]<\.001<\.001Tel\.βA\\beta\_\{A\}\+0\.13\+0\.13\[−0\.03,\+0\.28\]\[\-0\.03,\+0\.28\]\.101\.101βL​A\\beta\_\{LA\}\+0\.04\+0\.04\[−0\.06,\+0\.13\]\[\-0\.06,\+0\.13\]\.462\.462Table 13:Robustness check: fixed\-effect estimates from the mixed\-effects models for Experiment 2 after excluding the two low\-discrimination judges \(Llama 4 Maverick and Mistral Large 3\)\. Model specification identical to Table[5](https://arxiv.org/html/2608.18091#S4.T5)\.βL\\beta\_\{L\}captures the displayed\-label effect;βA\\beta\_\{A\}the actual\-source effect \(expected≈0\\approx 0under successful quality matching\);βL​A\\beta\_\{LA\}their interaction\.p∗⁣∗∗<\.001\{\}^\{\*\*\*\}p<\.001\. The label effect \(βL\\beta\_\{L\}\) remains the only significant fixed effect on every dimension, and its magnitude is comparable to or slightly larger than the full\-sample estimate, confirming that the main findings are not driven by the two excluded judges\.