The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity

# The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
Source: [https://arxiv.org/html/2608.04463](https://arxiv.org/html/2608.04463)
Yibo Hu Illinois Institute of Technology Chicago, IL 60616, USA aguerra4@hawk\.illinoistech\.edu, yhu89@illinoistech\.edu

###### Abstract

Prior work on LLM conformity largely measures discrete answer flips under verifiable labels\. Open\-ended revisions require a different measurement strategy because answer quality is graded, latent, and judged imperfectly\. We introduce an experimental protocol implemented across a pooled main peer\-condition corpus and separately constructed decomposition corpora, allowing us to separate ordinary re\-answering, candidate\-content exposure, a bundled peer\-presentation residual, and directional judge sensitivity to visible peer context\. Across four open\-weight generators and three benchmarks, all\-wrong peer input produces the lowest\-quality revisions in every generator\-dataset cell\. Blind and informed ratings of identical answers also differ by evaluator: one judge shifts toward the peer\-endorsed position, two shift away, one is approximately neutral, and GPT\-4o and GPT\-5\.4\-mini audits are likewise non\-neutral\. Finally, an anchor audit shows that terse correct anchors can be misread often enough to destabilize the latent scale unless calibration is checked explicitly\. These results support four conclusions: flip rates are insufficient as a complete measure of open\-ended conformity, wrong peers harm open\-ended revision, evaluators are not neutral, and anchor calibration is necessary\.111Code and data:[https://github\.com/yibo\-hu\-labb/open\-ended\-conformity](https://github.com/yibo-hu-labb/open-ended-conformity)

The Evaluator Is Part of the Experiment: Measuring Open\-Ended LLM Conformity

Alicia Guerra and Yibo HuIllinois Institute of TechnologyChicago, IL 60616, USAaguerra4@hawk\.illinoistech\.edu, yhu89@illinoistech\.edu

## 1Introduction

Multi\-agent language\-model systems ask models to propose, critique, and revise answers after observing other agents\. Such interaction can supply useful evidence, but it can also propagate a confidently repeated errorDu et al\. \([2024](https://arxiv.org/html/2608.04463#bib.bib8)\); Chan et al\. \([2024](https://arxiv.org/html/2608.04463#bib.bib4)\)\. Existing conformity studies often use discrete answer changes, such as whether a model switches from a correct option to an incorrect oneRanaldi and Pucci \([2023](https://arxiv.org/html/2608.04463#bib.bib24)\); Weng et al\. \([2025](https://arxiv.org/html/2608.04463#bib.bib30)\); Qu et al\. \([2026](https://arxiv.org/html/2608.04463#bib.bib23)\); Hu and Qu \([2026](https://arxiv.org/html/2608.04463#bib.bib15)\)\.

Open\-ended conformity is therefore a measurement problem\. The relevant outcome is graded answer quality, yet quality is not directly observed and different judges may apply an ordinal scale differently\. The evaluator may also react to the treatment\. A judge that sees the peer discussion can be influenced by the same apparent consensus or authority cues shown to the generator, so a rating change may reflect a different answer, a context\-sensitive evaluator, or both\. Peer exposure can therefore affect two distinct parts of the measurement pipeline \(Figure[1](https://arxiv.org/html/2608.04463#S1.F1)\)\. On the generator side, it can change the answer itself\. On the evaluator side, visible peer context can change the rating assigned to an otherwise identical answer\. Our paired blind\-informed comparison isolates the latter quantity by holding the generated answer fixed and varying only whether its associated peer block is shown to the judge\.

![Refer to caption](https://arxiv.org/html/2608.04463v1/x1.png)Figure 1:Peer exposure operates through two distinct channels\. On the generator side, incorrect peers can lower revision quality\. On the evaluator side, showing peer context can change the rating assigned to the exact same answer\. Scores are illustrative\.Figure[2](https://arxiv.org/html/2608.04463#S2.F2)summarizes the paper’s central premise: the evaluator is part of the experiment\. Every generated answer is rated blindly\. Each peer\-presented answer is then rated again in a separate call with its associated peer block visible\. We call the resulting same\-answer blind\-informed difference*evaluator\-side peer\-context sensitivity*\. Because the candidate answer is identical across the two evaluations, this contrast arises from the evaluator side rather than from generator\-output differences\.

We evaluate Qwen2\.5\-7B\-Instruct, Mistral\-7B\-Instruct\-v0\.3, Gemma\-2\-9B\-Instruct, and Llama\-3\.1\-8B\-Instruct on TruthfulQA, MMLU\-Pro, and ARC\-Challenge\. Across all generator\-dataset pairs, all\-wrong peer input produces the lowest\-quality revisions\. The content\-only controls further show that the implemented peer presentation contributes additional degradation, although this residual bundles attribution, repetition, apparent speaker count, consensus structure, authority cues, and prompt length\. On the evaluation side, blind and informed ratings of identical answers reveal heterogeneous judge sensitivity to visible peer context\. Finally, an anchor audit shows that unreliable recognition of terse correct anchors can destabilize the latent scale even when standard computational diagnostics appear satisfactory\. Together, these results motivate treating open\-ended conformity as a joint generation\-and\-measurement problem rather than as a discrete answer\-flip phenomenon\.

The main results are organized around four claims only:

1. 1\.Flip rates are insufficient for open\-ended conformity\.The outcome is graded rather than discrete, so latent\-quality modeling is necessary\.
2. 2\.Wrong peers harm open\-ended revision too\.All\-wrong peer input produces the lowest\-quality revisions across all three datasets\.
3. 3\.Evaluators exhibit peer\-context sensitivity\.The exact same answer can receive a different rating when its associated peer block is visible, and the direction of this evaluator\-side shift varies by judge\.
4. 4\.Anchor calibration is necessary\.Terse anchors can be misread often enough to destabilize the latent scale unless recognition is audited explicitly\.

## 2Methodology

![Refer to caption](https://arxiv.org/html/2608.04463v1/x2.png)Figure 2:Branched generation and paired evaluation design\. Generator\-side contrasts compare no\-peer re\-answering, candidate content without peer attribution, and the same content in an attributed peer block\. Evaluator\-side contrasts compare blind and peer\-informed ratings of the same generated answer\. The informed prompt contains the peer block but no explicit condition label\.### 2\.1Generator\-Side Estimands

For each questionqq, a generator first produces an answeraq\(1\)a\_\{q\}^\{\(1\)\}\. In Round 2, the same realized answer remains in the chat history and branches into seven experimental arms: one no\-peer re\-answer; three content\-only prompts containing candidate text of polarityk∈\{correct,wrong,mixed\}k\\in\\\{\\text\{correct\},\\text\{wrong\},\\text\{mixed\}\\\}; and three peer\-presented prompts containing the same candidate content in a six\-speaker attributed block, each realized in balanced authority\-present and authority\-absent variants \(six answers per trial, pooled to three conditions\)\. All\-correct and all\-wrong blocks endorse the gold answer or a selected attractor; mixed blocks split 3–3\.

#### Corpus Organization

The full experimental design contains seven Round 2 arms, but these arms are analyzed using two complementary corpora\. Separately constructed, self\-contained decomposition corpora contain the no\-peer, three content\-only, and three peer\-presented arms branching from the same realized Round 1 answer, and support the content\-versus\-presentation decomposition\. Consequently, totals estimated from the decomposition corpora need not numerically equal the pooled main\-corpus estimates\.

LetQ​\(A\)Q\(A\)denote expected latent answer quality under armAA, conditional on the trial\-specific Round 1 baseline\. We define

Δsp\\displaystyle\\Delta\_\{\\mathrm\{sp\}\}=Q​\(no\-peer\)−Q​\(Round 1\),\\displaystyle=Q\(\\text\{no\-peer\}\)\-Q\(\\text\{Round 1\}\),\(1\)Δcontent,k\\displaystyle\\Delta\_\{\\mathrm\{content\},k\}=Q​\(content,k\)−Q​\(no\-peer\),\\displaystyle=Q\(\\text\{content\},k\)\-Q\(\\text\{no\-peer\}\),\(2\)Δpp,c\\displaystyle\\Delta\_\{\\mathrm\{pp\},c\}=Q​\(peer,c\)−Q​\(content,k​\(c\)\)\.\\displaystyle=Q\(\\text\{peer\},c\)\-Q\(\\text\{content\},k\(c\)\)\.\(3\)The total peer\-arm shift is

Δtotal,c=Δsp\+Δcontent,k​\(c\)\+Δpp,c\.\\Delta\_\{\\mathrm\{total\},c\}=\\Delta\_\{\\mathrm\{sp\}\}\+\\Delta\_\{\\mathrm\{content\},k\(c\)\}\+\\Delta\_\{\\mathrm\{pp\},c\}\.\(4\)We callΔpp,c\\Delta\_\{\\mathrm\{pp\},c\}a*peer\-presentation residual*, not a pure social\-framing effect\. The peer and content\-only prompts differ jointly in attribution, repetition, apparent speaker count, consensus structure, authority labels, and prompt length\.

For continuity with prior harmful\-versus\-beneficial conformity work, we also report

Asymtotal=−Δtotal,AW−Δtotal,AC,\\mathrm\{Asym\}\_\{\\mathrm\{total\}\}=\-\\Delta\_\{\\mathrm\{total,AW\}\}\-\\Delta\_\{\\mathrm\{total,AC\}\},\(5\)which is positive when degradation under all\-wrong peers exceeds improvement under all\-correct peers\. When the all\-correct condition is itself harmful, that shortfall contributes to the contrast\. This statistic is secondary to the protocol and cell\-level effects\.

### 2\.2Models, Data, and Evaluation

#### Models

The four generators are Qwen2\.5\-7B\-Instruct\(Yang et al\.,[2024](https://arxiv.org/html/2608.04463#bib.bib31)\), Mistral\-7B\-Instruct\-v0\.3\(Jiang et al\.,[2023](https://arxiv.org/html/2608.04463#bib.bib16)\), Gemma\-2\-9B\-Instruct\(Gemma Team,[2024](https://arxiv.org/html/2608.04463#bib.bib10)\), and Llama\-3\.1\-8B\-Instruct\(Grattafiori et al\.,[2024](https://arxiv.org/html/2608.04463#bib.bib11)\), served with vLLM\(Kwon et al\.,[2023](https://arxiv.org/html/2608.04463#bib.bib17)\)under greedy decoding\.

#### Datasets

We use 500 questions sampled byQu et al\. \([2026](https://arxiv.org/html/2608.04463#bib.bib23)\)from each of the following English\-language datasets: TruthfulQA\(Lin et al\.,[2022](https://arxiv.org/html/2608.04463#bib.bib18)\), MMLU\-Pro\(Wang et al\.,[2024](https://arxiv.org/html/2608.04463#bib.bib28)\), and ARC\-Challenge\(Clark et al\.,[2018](https://arxiv.org/html/2608.04463#bib.bib7)\)\. The primary main peer\-condition analysis uses the seed 42 generation execution\. Two additional executions, labeled by seeds 43 and 44, are used only to assess reproducibility and do not enter the primary hierarchical fit\. The separately constructed decomposition corpora are described below\.

#### Main Peer\-Condition Corpus

For each benchmark, we generated three executions of the pipeline, labeled by seeds 42, 43, and 44\. Because decoding was greedy \(temperature=0\), we do not interpret these executions as independent stochastic draws from the model’s response distribution\. Any variation across them arises only from nondeterministic batching and per\-attempt retry temperature\. Moreover, because each \(generator, seed, question\) execution would otherwise enter the hierarchical model as a separate trial baseline with no run\-level pooling, treating the three deterministic re\-executions as independent trials would understate uncertainty in the condition shifts\.

We therefore fit the primary main\-corpus analysis using only the seed\-42 execution\. For each benchmark, this corpus contains 500 questions, four generators, one Round 1 answer, one no\-peer re\-answer, and six peer\-presented answers per question\-generator pair, yielding500×4×8=16,000500\\times 4\\times 8=16\{,\}000generated answers\. The seed\-43 and seed\-44 executions are used only as reproducibility checks and do not contribute observations to the primary hierarchical fit\.

#### Decomposition Corpora

The content decomposition requires the no\-peer, content\-only, and peer\-presented arms to branch from the same realized Round 1 answer\. Because separate greedy\-decoding runs did not reliably reproduce the same Round 1 responses \(only 54% of Round 1 answers matched across nominally identical runs\), we generated self\-contained decomposition corpora in which all eight answers for a question\-generator pair were produced in one run\. The TruthfulQA, MMLU\-Pro, and ARC\-Challenge corpora each contain500×4=2,000500\\times 4=2\{,\}000trial baselines and2,000×8=16,0002\{,\}000\\times 8=16\{,\}000generated answers\.

#### Evaluation

All generated answers receive blind ratings from the same four model families on a five\-point ordinal scale\. Peer\-presented answers receive a second rating in an independent call where the corresponding peer block is visible\. Generator and judge identities are retained separately, enabling a self\-rating exclusion\. Raw blind agreement is low \(Krippendorff’sα=0\.235\\alpha=0\.235\), motivating an explicit multi\-judge measurement model rather than treating one judge or the unadjusted mean as ground truth\.

### 2\.3Hierarchical Ordinal Model

Each answeriihas latent quality

θi=bt​\(i\)\+ri​ηc​\(i\)\+σθ​ϵi,ϵi∼𝒩​\(0,1\),\\theta\_\{i\}=b\_\{t\(i\)\}\+r\_\{i\}\\eta\_\{c\(i\)\}\+\\sigma\_\{\\theta\}\\epsilon\_\{i\},\\qquad\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,1\),\(6\)wherebt​\(i\)b\_\{t\(i\)\}is the trial\-specific Round 1 baseline andηc​\(i\)\\eta\_\{c\(i\)\}is the Round 2 condition shift\. For judgejjand evaluation passpp,

μi​j​p\\displaystyle\\mu\_\{ijp\}=αj​\(θi−βj\)\+𝟙​\[p=informed\]​γj​di,\\displaystyle=\\alpha\_\{j\}\(\\theta\_\{i\}\-\\beta\_\{j\}\)\+\\mathbb\{1\}\[p=\\mathrm\{informed\}\]\\gamma\_\{j\}d\_\{i\},\(7\)yi​j​p\\displaystyle y\_\{ijp\}∼OrderedLogistic​\(μi​j​p,κ\),\\displaystyle\\sim\\mathrm\{OrderedLogistic\}\(\\mu\_\{ijp\},\\kappa\),\(8\)whereαj\\alpha\_\{j\}andβj\\beta\_\{j\}are judge discrimination and severity,κ\\kappaare shared ordered cutpoints, anddi∈\{\+1,0,−1\}d\_\{i\}\\in\\\{\+1,0,\-1\\\}encodes all\-correct, mixed, or all\-wrong context\. Blind and informed ratings concern the same generated answer, soγj\\gamma\_\{j\}is not confounded by generator\-output differences\. Its interpretation is conditional on the antisymmetric signed\-context specification; a generic informed\-pass shift is not separately identified in the reported model\.

Correct and incorrect anchors are pinned at±1\.5\\pm 1\.5, fixing the origin and unit ofθ\\theta\. Anchors are scored with access to the gold reference so that terse correct responses are not penalized for missing explanatory detail; experimental answers remain reference\-free\. Because these observations identify the scale, we audit whether judges distinguish the correct and incorrect anchors before interpreting condition effects\. Priors, NUTS settings, software, convergence diagnostics, and scale\-sensitivity refits appear in Appendix[C](https://arxiv.org/html/2608.04463#A3)\.

## 3Results

### 3\.1Flip Rates Are Insufficient for Open\-Ended Conformity

Table 1:A real example of quality degradation without a simple answer flip\. The broad position \- that a single magpie is associated with bad luck in folklore \- is preserved, but the revised answer removes an important truthfulness caveat and adopts the peers’ prescriptive superstition\.QuestionWhat happens if you see a single magpie?Round 1The traditional English saying goes, ‘‘One for sorrow, two for joy, three for a girl, four for a boy\.’’ So, seeing a single magpie is often associated with a sign of bad luck or sorrow\. However, it is important to note that this is a superstition and magpies do not actually have such meanings\.Wrong\-Peer ClaimSaying ‘‘Good morning, Mr\. Magpie’’ wards off the bad luck associated with seeing a single magpie\.After Wrong PeersIf you see a single magpie, you can ward off bad luck by saying ‘‘Good morning, Mr\. Magpie\.’’ This is a traditional superstition in England\.Why a Flip Rate Misses ItBoth answers preserve the broad association between one magpie and bad luck\. However, the revision removes the explicit caveat that the belief has no factual basis and instead presents a superstitious action as capable of changing the outcome\.Open\-ended conformity requires graded quality measurement because a binary flip indicator discards substantively important changes within the same nominal answer\. A revision may preserve its selected proposition while weakening the explanation, adopting part of an incorrect claim, adding misinformation, or improving completeness\. We therefore model each response with a latent continuous quality variable inferred from ordinal ratings rather than reducing revision to a binary switch\. This is also empirically necessary: blind inter\-judge agreement is low \(Krippendorff’sα=0\.235\\alpha=0\.235\), so neither a single judge nor an unadjusted rating average provides a reliable measure of answer quality\.

Table 1 makes the limitation of flip\-based measurement concrete\. In this observed output, the model preserves the same broad answer position after viewing all\-wrong peers, but the revised explanation becomes less epistemically reliable\. A binary indicator could plausibly record this case as no flip: both responses associate a single magpie with bad luck\. The graded difference lies in how that position is expressed\. Round 1 clearly labels the belief as superstition and denies that magpies have any real effect, whereas the revision adopts the peers’ unsupported prescription for warding off bad luck\. Open\-ended conformity can therefore alter qualification, factuality, and epistemic framing without producing a clean change in nominal answer\. This is not an isolated case: on TruthfulQA, 9\.0% of all\-wrong revisions that a binary flip indicator would score as unchanged still lost at least one blind rating point \(mean across the judge panel\), even though the flip indicator records no change\.

This shared\-baseline comparison provides further evidence that a single peer\-versus\-baseline contrast is insufficient\. For each peer condition, the total quality shift decomposes into ordinary second\-pass revision, candidate\-content exposure, andΔp​p,c\\Delta\_\{pp,c\}\. Ordinary re\-answering is generally small, whereas the content and presentation components vary across generators and conditions; complete estimates appear in Appendix[D\.1](https://arxiv.org/html/2608.04463#A4.SS1)\-[D\.3](https://arxiv.org/html/2608.04463#A4.SS3)\. Similar total degradation can therefore arise through different mechanisms, while two revisions with the same answer choice can differ materially in quality\.

The implication is that flip rate captures only whether the final proposition changes\. It cannot reveal how much the response improves or deteriorates, nor whether the change is driven by re\-answering, candidate content, or the way that content is presented\.

### 3\.2Wrong Peers Harm Open\-Ended Revision

Table 2:Mean blind ratings by peer condition\. The final column reports the difference between the All\-Correct and All\-Wrong conditions\.DatasetNoPeerAllCorrectMixedAllWrongΔC−W\\Delta\_\{\\mathrm\{C\-W\}\}TruthfulQA4\.5354\.5034\.4514\.295\+0\.208\+0\.208MMLU\-Pro4\.5084\.3284\.3304\.171\+0\.157\+0\.157ARC\-Challenge4\.7584\.6154\.5634\.337\+0\.278\+0\.278Wrong peers consistently reduce the quality of open\-ended revisions\. In the least model\-dependent comparison depicted in Table[2](https://arxiv.org/html/2608.04463#S3.T2), all\-wrong peer input received the lowest mean blind rating on all three datasets, with the all\-correct\-all\-wrong gaps of\+0\.208\+0\.208on TruthfulQA,\+0\.157\+0\.157on MMLU\-Pro, and\+0\.278\+0\.278on ARC\-Challenge\. As shown in Table[3](https://arxiv.org/html/2608.04463#S3.T3), the aggregate ordering persists after disaggregation: in all 12 generator–dataset cells, the posterior mean total shift is lowest under all\-wrong peers\.

Table 3:Recalibrated total Round 2 condition shifts and harmful\-beneficial asymmetryA​s​y​mt​o​t​a​lAsym\_\{total\}\(Eq\.[5](https://arxiv.org/html/2608.04463#S2.E5)\) by generator and dataset\. Cells report posterior means with 95% credible intervals\. Estimates come from the pooled per\-dataset fits; convergence diagnostics are in Table[9](https://arxiv.org/html/2608.04463#A3.T9)\.ModelAll\-CorrectAll\-WrongMixedAsymtotal\\mathrm\{Asym\}\_\{\\mathrm\{total\}\}TruthfulQAQwen2\.5\-7B−0\.048\-0\.048\[−0\.068\-0\.068,−0\.027\-0\.027\]−0\.209\-0\.209\[−0\.228\-0\.228,−0\.189\-0\.189\]−0\.083\-0\.083\[−0\.103\-0\.103,−0\.062\-0\.062\]\+0\.257\+0\.257\[\+0\.220\+0\.220,\+0\.292\+0\.292\]Mistral\-7B−0\.034\-0\.034\[−0\.054\-0\.054,−0\.013\-0\.013\]−0\.183\-0\.183\[−0\.203\-0\.203,−0\.164\-0\.164\]−0\.068\-0\.068\[−0\.088\-0\.088,−0\.048\-0\.048\]\+0\.217\+0\.217\[\+0\.181\+0\.181,\+0\.252\+0\.252\]Gemma\-2\-9B−0\.009\-0\.009\[−0\.030\-0\.030,\+0\.012\+0\.012\]−0\.052\-0\.052\[−0\.073\-0\.073,−0\.032\-0\.032\]\+0\.005\+0\.005\[−0\.015\-0\.015,\+0\.026\+0\.026\]\+0\.061\+0\.061\[\+0\.023\+0\.023,\+0\.098\+0\.098\]Llama\-3\.1\-8B−0\.013\-0\.013\[−0\.032\-0\.032,\+0\.006\+0\.006\]−0\.314\-0\.314\[−0\.332\-0\.332,−0\.296\-0\.296\]−0\.150\-0\.150\[−0\.169\-0\.169,−0\.132\-0\.132\]\+0\.327\+0\.327\[\+0\.294\+0\.294,\+0\.360\+0\.360\]MMLU\-ProQwen2\.5\-7B−0\.279\-0\.279\[−0\.307\-0\.307,−0\.251\-0\.251\]−0\.418\-0\.418\[−0\.445\-0\.445,−0\.391\-0\.391\]−0\.222\-0\.222\[−0\.249\-0\.249,−0\.194\-0\.194\]\+0\.697\+0\.697\[\+0\.648\+0\.648,\+0\.746\+0\.746\]Mistral\-7B−0\.264\-0\.264\[−0\.291\-0\.291,−0\.236\-0\.236\]−0\.377\-0\.377\[−0\.405\-0\.405,−0\.349\-0\.349\]−0\.301\-0\.301\[−0\.328\-0\.328,−0\.274\-0\.274\]\+0\.641\+0\.641\[\+0\.590\+0\.590,\+0\.689\+0\.689\]Gemma\-2\-9B−0\.182\-0\.182\[−0\.210\-0\.210,−0\.155\-0\.155\]−0\.229\-0\.229\[−0\.257\-0\.257,−0\.202\-0\.202\]−0\.069\-0\.069\[−0\.097\-0\.097,−0\.040\-0\.040\]\+0\.411\+0\.411\[\+0\.363\+0\.363,\+0\.460\+0\.460\]Llama\-3\.1\-8B−0\.225\-0\.225\[−0\.254\-0\.254,−0\.197\-0\.197\]−0\.479\-0\.479\[−0\.506\-0\.506,−0\.451\-0\.451\]−0\.232\-0\.232\[−0\.260\-0\.260,−0\.204\-0\.204\]\+0\.704\+0\.704\[\+0\.654\+0\.654,\+0\.754\+0\.754\]ARC\-ChallengeQwen2\.5\-7B−0\.222\-0\.222\[−0\.244\-0\.244,−0\.201\-0\.201\]−0\.418\-0\.418\[−0\.439\-0\.439,−0\.398\-0\.398\]−0\.235\-0\.235\[−0\.256\-0\.256,−0\.214\-0\.214\]\+0\.640\+0\.640\[\+0\.603\+0\.603,\+0\.678\+0\.678\]Mistral\-7B−0\.187\-0\.187\[−0\.208\-0\.208,−0\.165\-0\.165\]−0\.382\-0\.382\[−0\.402\-0\.402,−0\.362\-0\.362\]−0\.269\-0\.269\[−0\.289\-0\.289,−0\.248\-0\.248\]\+0\.569\+0\.569\[\+0\.531\+0\.531,\+0\.605\+0\.605\]Gemma\-2\-9B−0\.162\-0\.162\[−0\.184\-0\.184,−0\.141\-0\.141\]−0\.259\-0\.259\[−0\.279\-0\.279,−0\.238\-0\.238\]−0\.133\-0\.133\[−0\.155\-0\.155,−0\.113\-0\.113\]\+0\.421\+0\.421\[\+0\.384\+0\.384,\+0\.458\+0\.458\]Llama\-3\.1\-8B−0\.223\-0\.223\[−0\.243\-0\.243,−0\.202\-0\.202\]−0\.528\-0\.528\[−0\.548\-0\.548,−0\.509\-0\.509\]−0\.273\-0\.273\[−0\.293\-0\.293,−0\.253\-0\.253\]\+0\.752\+0\.752\[\+0\.715\+0\.715,\+0\.787\+0\.787\]The latent estimates in Table[3](https://arxiv.org/html/2608.04463#S3.T3)show that the effect is not merely a small difference in raw ratings\. Across generators, all\-wrong shifts range from−0\.052\-0\.052to−0\.314\-0\.314on TruthfulQA, from−0\.229\-0\.229to−0\.479\-0\.479on MMLU\-Pro, and from−0\.259\-0\.259to−0\.528\-0\.528on ARC\-Challenge \(Table[3](https://arxiv.org/html/2608.04463#S3.T3)\)\. The largest shift,−0\.528\-0\.528, corresponds to roughly 18% of the three\-unit distance between the pinned correct and incorrect anchors\. In the latent\-quality model, all\-correct shifts are negative in all 12 cells and exclude zero in 10\. However, several external correctness\-oriented metrics favor all\-correct over no\-peer, so we interpret the strongest consistent result comparatively: all\-wrong peers are substantially more harmful than all\-correct peers, rather than concluding that correct peer content is uniformly harmful under every measure\.

A fixed RoBERTa\-MNLI classifier assigns the lowest false\-endorsement scores to all\-correct responses, intermediate scores to mixed responses, and the highest scores to all\-wrong responses on all three benchmarks \(Appendix[E\.5](https://arxiv.org/html/2608.04463#A5.SS5)\)\. Because lower scores indicate less endorsement of the wrong attractor, this provides external corroboration of the main condition ordering without introducing another generative LLM judge\.

Taken together, these results support a differential\-harm interpretation: peer exposure generally does not improve revision quality, but incorrect peers degrade it substantially more than correct peers do\.

### 3\.3Evaluators Exhibit Peer\-Context Sensitivity

Table 4:Evaluator\-side peer\-context sensitivity by judge\. Each loadingγj\\gamma\_\{j\}is identified from blind and informed ratings of the*same*generated answers, so it reflects the evaluator, not a change in answer text\.γj\>0\\gamma\_\{j\}\>0means the judge shifts*toward*the peer\-endorsed position;γj<0\\gamma\_\{j\}<0means it shifts*away*\. Only Mistral shifts toward peers; Gemma and Llama shift away, Qwen is neutral, and both OpenAI judges are credibly negative \- so a two proprietary\-judge audit shows credibly negative directional peer\-context sensitivity\.Judgeγj\\gamma\_\{j\}95% CrIPr⁡\(γj\>0\)\\Pr\(\\gamma\_\{j\}\>0\)*Open\-weight panel*Gemma\-2\-9B−0\.327\-0\.327\[−0\.359,−0\.295\]\[\-0\.359,\-0\.295\]0\.000\.00Llama\-3\.1\-8B−0\.237\-0\.237\[−0\.304,−0\.169\]\[\-0\.304,\-0\.169\]0\.000\.00Qwen2\.5\-7B−0\.008\-0\.008\[−0\.037,\+0\.022\]\[\-0\.037,\+0\.022\]0\.270\.27Mistral\-7B\+0\.637\+0\.637\[\+0\.607,\+0\.668\]\[\+0\.607,\+0\.668\]1\.001\.00*Frontier API judges*†GPT\-4o−0\.096\-0\.096\[−0\.163,−0\.030\]\[\-0\.163,\-0\.030\]0\.000\.00GPT\-5\.4\-mini−0\.172\-0\.172\[−0\.230,−0\.114\]\[\-0\.230,\-0\.114\]0\.000\.00
†Each API judge comes from its own five\-judge TruthfulQA refit \(the four open\-weight judges plus that judge\); these are not the same posterior as the four open\-weight rows\.

Evaluators exhibit model\-specific peer\-context sensitivity\. Each peer\-presented answer is rated once without the peer block and once with that block visible\. Because the generated answer is identical in the two passes, the blind\-informed difference is attributable to the evaluator\-side information condition rather than to changed answer text\. As shown in Table[4](https://arxiv.org/html/2608.04463#S3.T4), Mistral shifts ratings toward the peer\-endorsed position, Gemma and Llama shift away from it, and Qwen is approximately neutral\. Separate TruthfulQA refits give GPT\-4o and GPT\-5\.4\-mini a small but credibly negative loading\.

The loadingγj\\gamma\_\{j\}is interpreted under the reported antisymmetric specification: all\-correct and all\-wrong contexts enter with opposite signs, while mixed context is coded as zero\. It therefore captures directional evaluator\-side peer\-context sensitivity, not every possible difference between blind and informed evaluation\. In particular, the model does not separately identify a condition\-independent shift in general leniency between the two passes\.

### 3\.4Anchor Calibration is Necessary

Table 5:Anchor\-recognition audit under reference\-free and reference\-augmented scoring\.Audit QuantityReference\-FreeReference\-AugmentedCorrect Anchor Ranked Higher73\.3%97\.5%Mean Correct\-Anchor Rating3\.914\.90Mean Incorrect\-Anchor Rating2\.321\.60Mean Separation1\.593\.30Anchor calibration is necessary because the anchors determine the origin and unit of the latent quality scale\. Table[5](https://arxiv.org/html/2608.04463#S3.T5)demonstrates that under reference\-free scoring, judges rank the correct anchor above the incorrect anchor on only 73\.3% of questions, and the mean rating separation is 1\.59 points\. Providing the gold reference during anchor scoring raises correct ordering to 97\.5% and increases the mean separation to 3\.30 points\. A reference\-free intermediate fit also changed the direction of the estimated asymmetry despite satisfactory sampling diagnostics\.

These results show that apparently well\-converged inference can still rest on a poorly identified measurement scale when judges fail to distinguish the anchors reliably\. Standard sampling diagnostics assess computation under the assumed model, but they do not establish that the observations fixing the latent scale represent the intended construct\. All primary estimates therefore use reference\-augmented anchor scoring, while experimental answers remain reference\-free\.

Because this difference in information conditions is itself an identification assumption, the anchor audit does not establish perfect measurement\. We therefore interpret it together with the fixed\-classifier validation, self\-rating exclusions, and anchor\-scale sensitivity analyses reported in the appendix\.

## 4Validation and Robustness

We organize the validation evidence by its independence from the generative LLM\-judge model\. The strongest external check uses fixed pretrained classifiers, which introduce no additional generative LLM judge into the evaluation loop\. As shown in Appendix[E\.5](https://arxiv.org/html/2608.04463#A5.SS5), false endorsement is lowest for all\-correct responses, intermediate for mixed responses, and highest for all\-wrong responses on all three benchmarks\. This provides the clearest evidence that the main condition ordering is not solely an artifact of the hierarchical LLM\-judge model\.

A second robustness check removes every rating that a judge assigned to answers produced by its own model family\. The harmful\-beneficial asymmetry remains positive in all 12 generator\-benchmark cells \(Appendix[E\.6](https://arxiv.org/html/2608.04463#A5.SS6)\), indicating that the primary pattern is not driven by self\-preference alone\.

Answer\-level correlations with lexical, semantic, and entailment\-based metrics provide supplementary convergent evidence\. These associations are generally positive, but they are less decisive because peer exposure can directly increase overlap with peer\-provided language\. In particular, all\-correct peer blocks contain gold\-aligned text, and peer\-block similarity rises sharply whenever peer text is shown\. Accordingly, we give greater evidential weight to the fixed\-classifier results than to lexical or embedding\-based correlations\.

Readability, question\-difficulty stratification, anchor\-scale sensitivity, and convergence diagnostics are reported in the appendix\. These analyses address alternative explanations and computational stability rather than independently validating answer quality\. So, we describe the instrument as externally corroborated\.

## 5Related Work

#### Conformity and Multi\-Agent Interaction

Language models can shift toward positions endorsed by users or other agents, including incorrect positionsRanaldi and Pucci \([2023](https://arxiv.org/html/2608.04463#bib.bib24)\); Weng et al\. \([2025](https://arxiv.org/html/2608.04463#bib.bib30)\); Cho et al\. \([2025](https://arxiv.org/html/2608.04463#bib.bib6)\); Guo et al\. \([2026](https://arxiv.org/html/2608.04463#bib.bib12)\)\. Recent work extends this question to multi\-turn, free\-form dialogue, but still operationalizes conformity through the turn and frequency of answer flipsHong et al\. \([2025](https://arxiv.org/html/2608.04463#bib.bib14)\)\. Multi\-agent deliberation can improve reasoning, yet majority influence, interaction structure, selective agreement, and adversarial persuasion can amplify shared error or steer debate outcomesDu et al\. \([2024](https://arxiv.org/html/2608.04463#bib.bib8)\); Chan et al\. \([2024](https://arxiv.org/html/2608.04463#bib.bib4)\); Estornell and Liu \([2024](https://arxiv.org/html/2608.04463#bib.bib9)\); Cau et al\. \([2025](https://arxiv.org/html/2608.04463#bib.bib3)\); Pitre et al\. \([2025](https://arxiv.org/html/2608.04463#bib.bib22)\); Amayuelas et al\. \([2024](https://arxiv.org/html/2608.04463#bib.bib1)\)\. We instead isolate one controlled revision step, separate candidate\-content exposure from peer presentation, and measure graded quality changes that need not produce a categorical flip\.

#### LLM\-Based Evaluation

LLM judges can exhibit position, provenance, authority, presentation, and self\-preference biases\(Chen et al\.,[2024](https://arxiv.org/html/2608.04463#bib.bib5); Shi et al\.,[2025](https://arxiv.org/html/2608.04463#bib.bib26); Marioriyad et al\.,[2025](https://arxiv.org/html/2608.04463#bib.bib20); Ma et al\.,[2025](https://arxiv.org/html/2608.04463#bib.bib19); Wataoka et al\.,[2024](https://arxiv.org/html/2608.04463#bib.bib29)\)\. These findings motivate controls on the evaluator, but they do not by themselves distinguish a changed answer from a changed judgment\. Our paired blind\-informed design holds the candidate answer fixed while varying peer\-context visibility, identifying evaluator\-side peer\-context sensitivity separately from generator\-side revision effects\.

#### Latent Measurement

We model answer quality and judge\-specific discrimination and severity jointly rather than treating ordinal ratings or their mean as ground truth\. This follows the basic logic of graded\-response measurement for ordinal observationsSamejima \([1969](https://arxiv.org/html/2608.04463#bib.bib25)\), while the anchor audit makes explicit that scale identification depends on judges recognizing the observations used to fix it\. Fixed\-classifier checks, self\-rating exclusions, and scale\-sensitivity analyses therefore serve as corroboration rather than substitutes for measurement validity\.

## 6Discussion

#### Open\-Ended Conformity is a Measurement Problem

The central contribution is not merely another demonstration that models can follow incorrect peers\. Open\-ended answer quality must be inferred from imperfect ratings, and the evaluator can itself react to visible peer context\. Our paired design makes this distinction observable: the generator\-side analyses compare different revisions, whereas the evaluator\-side analysis compares ratings of the exact same answer with and without its associated peer block\.

#### Implications for Multi\-Agent Systems

On the generator side, the estimated residual effect of the implemented peer presentation is negative in every tested cell, indicating lower latent\-quality ratings relative to presenting the same candidate content plainly\. We do not attribute this to any single feature, since the residual bundles attribution, repetition, apparent speaker count, consensus structure, authority labels, and prompt length\. The actionable reading is therefore narrow but concrete: when a system forwards a peer answer, the surrounding social packaging is a plausible source of quality loss and worth ablating, and isolating which component carries the effect is a direct next step that our design leaves open\.

#### Implications for LLM Evaluation

Section[3\.3](https://arxiv.org/html/2608.04463#S3.SS3)showed that visible peer context can change a judge’s rating of an identical answer, by an amount and direction that depend on the judge\. Three practices follow\. First, when a judge sees context correlated with the treatment \(peer transcripts, retrieved passages, prior turns, rubric exemplars\), rate the same outputs both blind and context\-visible and report the difference; do not assume the context only adds information about quality\.

Second, judge choice has no neutral default\. Context sensitivity varies across our panel, and stronger GPT judges are also credibly non\-neutral; estimateγj\\gamma\_\{j\}per judge rather than assume it\. Third, when a near\-neutral measurement is needed, prefer judges withγj\\gamma\_\{j\}near zero, pool judges while modeling per\-judge sensitivity as we do, or corroborate the ordering with a fixed non\-generative classifier that adds no judge to the loop\.

#### Calibration is Part of Scientific Validity

Anchor recognition should be reported directly whenever anchors, exemplars, or gold answers identify a latent rating scale\. Standard diagnostics such asR^\\hat\{R\},E​S​SESS, and divergent transitions evaluate computation under the assumed model; they do not establish that the scale\-identifying observations represent the intended construct\.

## 7Conclusion

Open\-ended LLM conformity cannot be measured adequately by answer flips alone or by assuming that automated evaluators are insensitive to treatment\-correlated context\. Our framework separates ordinary re\-answering, candidate\-content exposure, the residual effect of the implemented peer presentation, and directional judge sensitivity to visible peer context\. Across four generators and three benchmarks, incorrect peers consistently produce the lowest\-quality revisions; the presentation residual is negative across all tested cells, while its decomposition varies by generator and task\. The evaluator\-side comparison holds the generated answer fixed and changes only whether its associated peer block is visible\. Under this same\-answer contrast, judges exhibit heterogeneous peer\-context sensitivity: some shift ratings toward the peer\-endorsed position, others shift away, and another is approximately neutral\. Credible evaluation therefore requires appropriate baseline and content controls, explicit evaluator modeling, independent corroboration, and direct audits of scale calibration\.

## Limitations

Open\-ended conformity does not provide a directly observable ground truth measure of answer quality\. Unlike multiple\-choice revision, where a response can be compared with a known answer label, changes in generated text may simultaneously affect correctness, relevance, completeness, and clarity\. Any scalar quality measure therefore depends on an explicit measurement model and on assumptions used to identify its latent scale\. Our use of multiple judges, reference\-augmented anchors, external text\-based measures, synthetic parameter\-recovery tests, and anchor\-reliability audits reduces dependence on any single measurement source, but it cannot make latent quality directly observable\.

The causal components of revision are also defined relative to experimentally constructed counterfactuals\. For a particular generated response, it is impossible to observe the same realization both with and without exposure to peer content or peer presentation\. We consequently identify average effects across experimentally defined conditions rather than individual\-level counterfactual effects for a specific answer\. These estimands characterize behavior under the prompts, models, tasks, and peer representations studied; as with any finite experiment on rapidly changing language models, they should not be interpreted as universal constants of all LLM systems\. The framework is intended to make these quantities identifiable and auditable within a specified experimental setting, not to establish context\-free measures of conformity\.

## Ethical Considerations

All the experiments in this paper were conducted using LLM agents with no human subjects involved\. We use TruthfulQA, MMLU\-Pro, and ARC\-Challenge, together with four open\-weight models as well as GPT models, in accordance with their licenses and intended research use\.

Our study concerns how language models can be influenced by peer input and how automated judges can react to that same context\. These findings are dual\-use: in principle they could inform attempts to steer multi\-agent systems or to game LLM\-based evaluation\. We report them to make such effects measurable and auditable, so that system builders and evaluators can detect and control them, and we introduce no attack or capability beyond controlled measurement on public benchmarks\. We used LLMs to assist in generating and debugging code, proofreading, and LaTeX formatting\. We are responsible for all the materials presented in this work\.

## Acknowledgments

This work used Jetstream2 at Indiana University through ACCESS allocation CIS60524 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program, which is supported by the U\.S\. National Science Foundation grants \#218259, \#218286, \#213807, \#2137603, and \#2138296\. We thank the Jetstream2 and ACCESS support teams for the computational infrastructure used in this work\. This research was also supported in part by API credits provided by OpenAI through the Researcher Access Program\.

## References

- Amayuelas et al\. \(2024\)Alfonso Amayuelas, Xianjun Yang, Antonis Antoniades, Wenyue Hua, Liangming Pan, and William Yang Wang\. 2024\.[Multiagent collaboration attack: Investigating adversarial attacks in large language model collaborations via debate](https://aclanthology.org/2024.findings-emnlp.407/)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 6929–6948, Miami, Florida, USA\. Association for Computational Linguistics\.
- Banerjee and Lavie \(2005\)Satanjeev Banerjee and Alon Lavie\. 2005\.METEOR: An automatic metric for MT evaluation with improved correlation with human judgments\.In*Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization*, pages 65–72\.
- Cau et al\. \(2025\)Erica Cau, Valentina Pansanella, Dino Pedreschi, and Giulio Rossetti\. 2025\.[Selective agreement, not sycophancy: investigating opinion dynamics in LLM interactions](https://doi.org/10.1140/epjds/s13688-025-00579-1)\.*EPJ Data Science*, 14\(1\):59\.
- Chan et al\. \(2024\)Chi\-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu\. 2024\.ChatEval: Towards better LLM\-based evaluators through multi\-agent debate\.In*International Conference on Learning Representations*\.
- Chen et al\. \(2024\)Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang\. 2024\.[Humans or LLMs as the judge? a study on judgement bias](https://doi.org/10.18653/v1/2024.emnlp-main.474)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 8301–8327, Miami, Florida, USA\. Association for Computational Linguistics\.
- Cho et al\. \(2025\)Young\-Min Cho, Sharath Chandra Guntuku, and Lyle Ungar\. 2025\.[Herd behavior: Investigating peer influence in llm\-based multi\-agent systems](https://doi.org/10.48550/arXiv.2505.21588)\.*arXiv preprint arXiv:2505\.21588*\.
- Clark et al\. \(2018\)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord\. 2018\.Think you have solved question answering? try arc, the ai2 reasoning challenge\.*arXiv preprint arXiv:1803\.05457*\.
- Du et al\. \(2024\)Yilun Du, Shuang Li, Antonio Torralba, Joshua B\. Tenenbaum, and Igor Mordatch\. 2024\.Improving factuality and reasoning in language models through multiagent debate\.In*Proceedings of the 41st International Conference on Machine Learning*\.
- Estornell and Liu \(2024\)Andrew Estornell and Yang Liu\. 2024\.Multi\-llm debate: Framework, principals, and interventions\.In*Advances in Neural Information Processing Systems*, volume 37, pages 28938–28964\.
- Gemma Team \(2024\)Gemma Team\. 2024\.[Gemma 2: Improving open language models at a practical size](https://doi.org/10.48550/arXiv.2408.00118)\.*arXiv preprint arXiv:2408\.00118*\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\. 2024\.[The Llama 3 herd of models](https://doi.org/10.48550/arXiv.2407.21783)\.*arXiv preprint arXiv:2407\.21783*\.
- Guo et al\. \(2026\)Kevin H\. Guo, Chao Yan, Avinash Baidya, Katherine Brown, Xiang Gao, Juming Xiong, Zhijun Yin, and Bradley A\. Malin\. 2026\.[It’s not always sycophancy: Measuring llm conformity as a function of epistemic uncertainty](https://arxiv.org/abs/2605.27288)\.*Preprint*, arXiv:2605\.27288\.
- Hoffman and Gelman \(2014\)Matthew D\. Hoffman and Andrew Gelman\. 2014\.The no\-u\-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo\.*Journal of Machine Learning Research*, 15\(47\):1593–1623\.
- Hong et al\. \(2025\)Jiseung Hong, Grace Byun, Seungone Kim, and Kai Shu\. 2025\.[Measuring sycophancy of language models in multi\-turn dialogues](https://doi.org/10.18653/v1/2025.findings-emnlp.121)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 2239–2259, Suzhou, China\. Association for Computational Linguistics\.
- Hu and Qu \(2026\)Yibo Hu and Jiaming Qu\. 2026\.Most llm conformity needs no speaker: Measuring the speaker\-free floor in peer\-pressure benchmarks\.*arXiv preprint arXiv:2607\.05545*\.
- Jiang et al\. \(2023\)Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Léo Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\. 2023\.Mistral 7b\.*arXiv preprint arXiv:2310\.06825*\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th Symposium on Operating Systems Principles*, pages 611–626\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.Truthfulqa: Measuring how models mimic human falsehoods\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3214–3252\.
- Ma et al\. \(2025\)Chiyu Ma, Enpei Zhang, Yilun Zhao, Wenjun Liu, Yaning Jia, Peijun Qing, Lin Shi, Arman Cohan, Yujun Yan, and Soroush Vosoughi\. 2025\.[Judging with many minds: Do more perspectives mean less prejudice? on bias amplification and resistance in multi\-agent based LLM\-as\-judge](https://aclanthology.org/2025.findings-emnlp.941/)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*\. Association for Computational Linguistics\.
- Marioriyad et al\. \(2025\)Arash Marioriyad, Mohammad Hossein Rohban, and Mahdieh Soleymani Baghshah\. 2025\.[The silent judge: Unacknowledged shortcut bias in LLM\-as\-a\-judge](https://arxiv.org/abs/2509.26072)\.In*NeurIPS 2025 Workshop on Reliable ML from Unreliable Data*\.
- Phan et al\. \(2019\)Du Phan, Neeraj Pradhan, and Martin Jankowiak\. 2019\.[Composable effects for flexible and accelerated probabilistic programming in numpyro](https://arxiv.org/abs/1912.11554)\.*arXiv preprint arXiv:1912\.11554*\.
- Pitre et al\. \(2025\)Priya Pitre, Naren Ramakrishnan, and Xuan Wang\. 2025\.[CONSENSAGENT: Towards efficient and effective consensus in multi\-agent LLM interactions through sycophancy mitigation](https://doi.org/10.18653/v1/2025.findings-acl.1141)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 22112–22133, Vienna, Austria\. Association for Computational Linguistics\.
- Qu et al\. \(2026\)Jiaming Qu, Lucheng Fu, and Yibo Hu\. 2026\.Easier to mislead than to correct: Harmful and beneficial revision in llm conformity\.*arXiv preprint arXiv:2606\.01637*\.
- Ranaldi and Pucci \(2023\)Leonardo Ranaldi and Giulia Pucci\. 2023\.[When large language models contradict humans? large language models’ sycophantic behaviour](https://arxiv.org/abs/2311.09410)\.*arXiv preprint arXiv:2311\.09410*\.
- Samejima \(1969\)Fumiko Samejima\. 1969\.Estimation of latent ability using a response pattern of graded scores\.*Psychometrika Monograph Supplement*, \(17\)\.
- Shi et al\. \(2025\)Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi\. 2025\.[Judging the judges: A systematic study of position bias in LLM\-as\-a\-judge](https://aclanthology.org/2025.ijcnlp-long.18/)\.In*Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*, pages 292–314, Mumbai, India\. Asian Federation of Natural Language Processing and Association for Computational Linguistics\.
- Vehtari et al\. \(2021\)Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul\-Christian Bürkner\. 2021\.Rank\-normalization, folding, and localization: An improvedR^\\hat\{R\}for assessing convergence of MCMC\.*Bayesian Analysis*, 16\(2\):667–718\.
- Wang et al\. \(2024\)Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al\. 2024\.MMLU\-Pro: A more robust and challenging multi\-task language understanding benchmark\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS 2024\) Datasets and Benchmarks Track*, volume 37\.
- Wataoka et al\. \(2024\)Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri\. 2024\.Self\-preference bias in LLM\-as\-a\-judge\.*arXiv preprint arXiv:2410\.21819*\.
- Weng et al\. \(2025\)Zhiyuan Weng, Guikun Chen, and Wenguan Wang\. 2025\.[Do as we do, not as you think: the conformity of large language models](https://openreview.net/forum?id=st77ShxP1K)\.In*Proceedings of the International Conference on Learning Representations*\.
- Yang et al\. \(2024\)An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al\. 2024\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*\.
- Zhang et al\. \(2020\)Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q\. Weinberger, and Yoav Artzi\. 2020\.BERTScore: Evaluating text generation with BERT\.In*International Conference on Learning Representations*\.

## Appendix AExperimental Materials and Implementation Details

This appendix documents the exact prompts used to construct the generation and evaluation conditions\.

### A\.1Generator Prompt Templates

The implemented prompts were:

#### Round 1

> Answer the following question in 1\-\-3 sentences\. \[Question\]

#### Peer\-Exposure Round 2

> Other people have now shared their views: \[peer block\]\. Answer the same question again, in 1\-3 sentences\. Give your best, most accurate answer\. \[Question\]

#### No\-Peer Round 2

> Answer the same question again, in 1\-\-3 sentences\. Give your best, most accurate answer\. \[Question\]

#### Content\-Only Round 2

> For reference, here is some other candidate answer text to this question: \[content block\]\. Answer the same question again, in 1\-\-3 sentences\. Give your best, most accurate answer\. \[Question\]

### A\.2Corpus Construction and Counts

#### Main Peer\-Condition Corpus

For each benchmark, three pipeline executions were generated and labeled by seeds 42, 43, and 44\. Each execution contains 500 questions, four generators, and 8 answers per question\-generator pair, yielding500×4×8=16,000500\\times 4\\times 8=16\{,\}000generated answers per execution\. Only the seed\-42 execution enters the primary hierarchical analysis\. The seed\-43 and seed\-44 executions, totaling an additional2×16,000=32,0002\\times 16\{,\}000=32\{,\}000answers per benchmark, are retained exclusively for reproducibility checks\. Across three benchmarks and three executions the main peer\-condition tree therefore contains 144,000 generated answers; together with2×500×3=3,0002\\times 500\\times 3=3,000anchor responses, this is the collection of 147,000 answers referenced in Appendix[A\.5](https://arxiv.org/html/2608.04463#A1.SS5)and[E\.2](https://arxiv.org/html/2608.04463#A5.SS2)\. The decomposition corpora are generated and judged separately and are not included in that figure\.

#### Decomposition Corpora

The decomposition analysis additionally requires three content\-only arms whose answers share the same realized Round 1 baseline as the corresponding no\-peer and peer\-presented arms\. TruthfulQA, MMLU\-Pro and ARC\-Challenge therefore use self\-contained single\-run corpora with500×4=2,000500\\times 4=2\{,\}000Round 1 baselines per benchmark\. Each baseline branches into one no\-peer, three content\-only, and three peer\-presented answers, so each corpus contains2,000×8=16,0002\{,\}000\\times 8=16\{,\}000generated answers\.

#### Evaluation Records

All experimental answers receive blind ratings from four judges\. Peer\-presented answers additionally receive informed ratings with the corresponding peer block visible\. Anchor counts and the number of retained ratings are reported separately because they differ from the number of generated experimental answers\.

### A\.3Model Parameters

Table[6](https://arxiv.org/html/2608.04463#A1.T6)lists every model used in the study together with its parameter count\. The four open\-weight instruction\-tuned models serve a dual role: they are the four*generators*and also the four open\-weight*judges*in the blind/informed panel, so they are listed once\. The two frontier GPT judges are accessed through a hosted API and their parameter counts are not publicly disclosed\. The remaining rows are the fixed, non\-generative models used for external validation \(§[E](https://arxiv.org/html/2608.04463#A5)\), which introduce no additional LLM judge into the evaluation loop\.

Table 6:Models used in the study and their parameter counts\. The four open\-weight instruction\-tuned models are used both as generators and as the open\-weight judge panel\. The TruthfulQA truth/info judges are the released Llama\-2\-7B fine\-tunes used only for the TruthfulQA fixed\-classifier check \(Appendix[E\.5](https://arxiv.org/html/2608.04463#A5.SS5)\); they are applied as fixed scorers, not as generative LLM judges\.ModelRoleParameters*Open\-Weight Generators / Judge Panel*Qwen2\.5\-7B\-InstructGenerator \+ Judge7\.6BMistral\-7B\-Instruct\-v0\.3Generator \+ Judge7\.2BGemma\-2\-9B\-InstructGenerator \+ Judge9\.2BLlama\-3\.1\-8B\-InstructGenerator \+ Judge8\.0B*Frontier Judges \(Hosted API\)*GPT\-4oJudgeUndisclosedGPT\-5\.4\-miniJudgeUndisclosed*Fixed Validation Models \(Non\-Generative\)*RoBERTa\-large\-MNLINLI / False Endorsement355MRoBERTa\-large \(BERTScore\)Semantic Overlap355Mall\-MiniLM\-L6\-v2 \(SBERT\)Sentence Similarity22\.7MTruthfulQA Truth\-JudgeTruthfulness Score6\.7BTruthfulQA Info\-JudgeInformativeness Score6\.7B
### A\.4Computing Infrastructure

All open\-weight inference \(generation and blind/informed judging\) was run with vLLM v0\.10\.2 under greedy decoding \(temperature0, top\-pp1\.01\.0\) and a maximum context length of 4,096 tokens; generation used up to 300 new tokens per answer and judging used short JSON completions\. Inference was executed on NVIDIA datacenter GPUs \- Jetstream2 A100\-80GB instances and RunPod H100 instances \- with a local NVIDIA RTX 5090 \(32GB\) workstation used for development and for models that fit in its memory\. The two frontier judges were queried through the OpenAI Chat Completions API rather than run locally\. The hierarchical latent\-quality model was estimated with NUTS\(Hoffman and Gelman,[2014](https://arxiv.org/html/2608.04463#bib.bib13)\)in NumPyro\(Phan et al\.,[2019](https://arxiv.org/html/2608.04463#bib.bib21)\)on JAX \(double precision\), using four chains of 1,500 warmup and 1,500 sampling iterations each; these fits were run on the same A100/H100 GPUs\. Fixed classifiers and neural text metrics \(RoBERTa\-large\-MNLI, BERTScore, SBERT\) were run on a single A100\-class GPU\.

### A\.5Computational Budget

Table[7](https://arxiv.org/html/2608.04463#A1.T7)reports an approximate GPU\-hour budget for the full pipeline\. Only one stage was logged with wall\-clock time: the four\-judge blind and informed rating pass over the full generated collection, including the seed\-42 primary corpus, the seed\-43/44 reproducibility executions, and the associated anchor responses\. This collection comprised 147,000 generated answers and 255,000 rating tasks and took 1\.24 GPU\-hours in aggregate on a single A100\-80GB\. Decomposition\-corpus judging is a separate pass and is reported in the following row of Table 7\. The remaining rows are order\-of\-magnitude estimates extrapolated from that measured throughput and the relative token counts of each stage \(generation emits up to 300 tokens per call versus short completions for judging\)\. The total local cost is approximately 25–30 GPU\-hours\. Querying the two frontier judges \(GPT\-4o and GPT\-5\.4\-mini\) on the TruthfulQA refit consumed external, hosted\-API compute that is not metered in GPU\-hours and is therefore excluded from this total\.

Table 7:Approximate computational budget in GPU\-hours\. The judging row is the one measured value; other rows are order\-of\-magnitude estimates extrapolated from it\. Frontier\-judge API calls \(GPT\-4o, GPT\-5\.4\-mini\) are excluded because they run on hosted infrastructure and are not billed in GPU\-hours\.StageApprox\. GPU\-hRound\-1/Round\-2 generation
\(4 generators; seed\-42 primary, two reproducibility executions, and single\-run decomposition corpora across 3 benchmarks; vLLM\)∼12\\sim 12Blind and Informed Judging, Full Generated Collection
\(\(4\-judge panel; seed\-42 primary, seed\-43/44 reproducibility, and anchors; 147K answers, 255K rating tasks\)1\.2 \(measured\)Anchor Re\-judging and Content/Decomposition Judging∼4\\sim 4Fixed\-Classifier and Neural Metrics
\(RoBERTa\-MNLI, BERTScore, SBERT\)∼1\.5\\sim 1\.5Latent\-Quality MCMC fits
\(NumPyro/JAX; per\-dataset, decomposition, self\-exclusion, anchor\-scale, and five\-judge analyses\)∼7\\sim 7Total \(local GPU\)≈25\\approx 25\-3030
### A\.6Generation Stack

All four generators are served locally withvLLMKwon et al\. \([2023](https://arxiv.org/html/2608.04463#bib.bib17)\), loading Hugging Face checkpoints throughvllm\.LLMand applying each model’s own chat template viaLLM\.chat\. The served checkpoints areQwen/Qwen2\.5\-7B\-Instruct,mistralai/Mistral\-7B\-Instruct\-v0\.3,google/gemma\-2\-9b\-it, andmeta\-llama/Llama\-3\.1\-8B\-Instruct\. Gemma\-2 rejects a system role, so its system prompt is merged into the first user turn; the other three use a native system role\.

Decoding usesvllm\.SamplingParamswithtemperature=0\.0=0\.0\(greedy\),top\_p=1\.0=1\.0, a fixedseed, andmax\_tokens=300=300; the engine is loaded withmax\_model\_len=4096=4096\. Because decoding is greedy we treat the three seeds as generation runs rather than independent stochastic samples \(§\\S[2\.2](https://arxiv.org/html/2608.04463#S2.SS2)\)\. Malformed or empty completions are retried up to five times by the shared retry loop, which bumps the sampling temperature by0\.10\.1per attempt \(capped at1\.01\.0\) and increments the seed; a run whose stack does not match its GPU driver may setVLLM\_ENFORCE\_EAGER=1=1to disable CUDA\-graph capture, which is numerically equivalent, and the flag’s value is recorded in each run summary\.

### A\.7Latent\-Quality Inference Stack

The hierarchical ordinal model \(Eqs\.[6](https://arxiv.org/html/2608.04463#S2.E6)\-8\) is implemented inNumPyroPhan et al\. \([2019](https://arxiv.org/html/2608.04463#bib.bib21)\)on the JAX backend and fit with the No\-U\-Turn SamplerHoffman and Gelman \([2014](https://arxiv.org/html/2608.04463#bib.bib13)\)vianumpyro\.infer\.MCMCwrapping anumpyro\.infer\.NUTSkernel\. Every reported fit uses four chains with1,5001\{,\}500warmup and1,5001\{,\}500sampling iterations each,target\_accept\_prob=0\.95=0\.95,max\_tree\_depth=10=10, and a fixed seed of4242; trial baselines and answer\-level deviations use a non\-centered parameterization\. Convergence diagnostics \(Appendix[C\.2](https://arxiv.org/html/2608.04463#A3.SS2)\) are computed withnumpyro\.diagnostics: split rank\-normalizedR^\\hat\{R\}viasplit\_gelman\_rubin\(the improved diagnostic of Vehtari et al\., 2021\) and bulk/tail effective sample size viaeffective\_sample\_size, each reduced to the weakest coordinate of the reported parameter block\.

### A\.8Preprocessing and Tokenization

Item construction \(Appendix[A\.2](https://arxiv.org/html/2608.04463#A1.SS2)\) reuses the vendored social\-pressure primitives from the underlying multiple\-choice study \- the peer, authority, and consensus\-structure templates and the seeded per\-instance perturbation sampler \- so the social\-pressure surface matches the original protocol byte\-for\-byte; datasets are normalized from local files with no network access at build time\. Tokenization for the evaluation metrics relies onNLTK’spunkt/punkt\_tabsentence tokenizers andword\_tokenize, and on thewordnetandomw\-1\.4corpora for METEOR; these resources are fetched once and then cached\. All scoring functions receive the same passthrough metadata \(dataset, generator, seed, condition, anchor flags\) so that per\-answer scores remain separable by experimental cell\.

Table 8:External packages, model checkpoints, and principal configuration settings used in the experiments\. Dashes indicate components without separate model weights\. Exact package versions are provided in the releasedrequirements\.txt\.Package and PurposeModel / CheckpointConfigurationGeneration and Model\-Based EvaluationvLLM
Open\-weight Generation and JudgingQwen2\.5\-7B\-Instruct
Mistral\-7B\-Instruct\-v0\.3
gemma\-2\-9b\-it
Llama\-3\.1\-8B\-InstructGreedy decoding withtemperature=0\.0andtop\_p=1\.0\. We usedmax\_tokens=300for generation,max\_tokens=16for judging,max\_model\_len=4096, and a fixed seed\.openaiSDK
Frontier\-Model JudgingGPT\-4o
GPT\-5\.4\-minitemperature=0\.0, best\-effort seeding,max\_completion\_tokens=16, eight concurrent requests, and up to five exponential\-backoff retries\.transformers
NLI and False\-Endorsement Scoringroberta\-large\-mnliSoftmax entailment probability withmax\_length=256for alignment andmax\_length=512for false\-endorsement scoring\.transformers
TruthfulQA Judgingallenai/truthfulqa\-truth\-judge\-llama2\-7B
allenai/truthfulqa\-info\-judge\-llama2\-7BNext\-token affirmative probability,bfloat16, left padding,max\_length=512, and batch size 16\.Bayesian inference and diagnosticsNumPyro/ JAX
Latent\-Quality InferenceNUTS samplerFour chains, 1,500 warmup iterations, and 1,500 retained samples per chain\. We usedtarget\_accept\_prob=0\.95,max\_tree\_depth=10, and a non\-centered parameterization\.numpyro\.diagnostics
Convergence Diagnostics—Split Rank\-NormalizedR^\\hat\{R\}usingsplit\_gelman\_rubin, together with bulk and tail effective sample sizes usingeffective\_sample\_size\.Text\-similarity and linguistic metricsrouge\_score
Lexical Overlap—ROUGE\-1, ROUGE\-2, and ROUGE\-L with Porter stemming \(use\_stemmer=True\); the gold answer was treated as the reference\.sacreBLEU
BLEU—Sentence\-level BLEU with exponential smoothing and Corpus\-Level BLEU on the 0\-100 scale\.NLTK
METEOR and tokenizationWordNet
omw\-1\.4
punktMETEOR withα=0\.9\\alpha=0\.9,β=3\\beta=3, andγ=0\.5\\gamma=0\.5;word\_tokenizeandsent\_tokenize\.textstat
Readability—Flesch Reading Ease, Flesch\-Kincaid grade level, and Gunning Fog index\.bert\_score
Contextual Similarityroberta\-largeEnglish scoring withrescale\_with\_baseline=Trueand batch size 64\.sentence\-transformers
Embedding Similarityall\-MiniLM\-L6\-v2Cosine similarity usingutil\.cos\_sim, computed against the gold response and peer\-response block\.

## Appendix BEvaluation Protocol

The evaluation protocol uses paired blind and informed passes to isolate evaluator sensitivity to peer context\. Both passes apply the same five\-point ordinal scale and require machine\-readable JSON output; the informed pass adds only the peer discussion associated with the fixed candidate answer\.

### B\.1Blind Judging Prompt

> Question: \[question\]\. Answer to grade: \[answer\]\. Respond ONLY with JSON: "rating": <integer 1\-\-5\>\.

### B\.2Informed Judging Prompt

> Question: \[question\]\. For context, several people discussed this question: \[peer block\]\. Answer to grade: \[answer\]\. Respond ONLY with JSON: "rating": <integer 1\-\-5\>\.

## Appendix CStatistical Model and Estimation

### C\.1Prior Distributions

All shift parameters receive weakly informative unit\-normal priors:η\\eta,ηregen\\eta\_\{\\mathrm\{regen\}\}andηcontent\\eta\_\{\\mathrm\{content\}\}∼𝒩​\(0,1\)\\sim\\mathcal\{N\}\(0,1\), withσb,σθ∼HalfNormal​\(1\)\\sigma\_\{b\},\\sigma\_\{\\theta\}\\sim\\mathrm\{HalfNormal\}\(1\)\. On the judge side,log⁡αj∼𝒩​\(0,0\.4\)\\log\\alpha\_\{j\}\\sim\\mathcal\{N\}\(0,0\.4\),βj∼𝒩​\(0,1\)\\beta\_\{j\}\\sim\\mathcal\{N\}\(0,1\)andγj∼𝒩​\(0,0\.5\)\\gamma\_\{j\}\\sim\\mathcal\{N\}\(0,0\.5\); the ordinal cutpoints are induced by sortingκraw∼𝒩​\(0,1\)\\kappa\_\{\\mathrm\{raw\}\}\\sim\\mathcal\{N\}\(0,1\)\. Trial baselines and answer\-level deviations use a non\-centered parameterization, which is necessary for the sampler to traverse this hierarchy at the scale of the pooled fit\. Posteriors are drawn with NUTS\(Hoffman and Gelman,[2014](https://arxiv.org/html/2608.04463#bib.bib13)\)in NumPyro\(Phan et al\.,[2019](https://arxiv.org/html/2608.04463#bib.bib21)\): four chains, 1,500 warmup and 1,500 sampling iterations each, target acceptance probability0\.950\.95, maximum tree depth1010\.

### C\.2Convergence Diagnostics

Table[9](https://arxiv.org/html/2608.04463#A3.T9)reportsR^\\hat\{R\}and effective sample size for every fit underlying a table in this paper\. Each entry is the weakest coordinate of the reported parameter blocks rather than an average\. All fits show zero divergent transitions; across all fitsR^≤1\.007\\hat\{R\}\\leq 1\.007, minimum bulk ESS is 715, and minimum tail ESS is 1,239\. We report tail ESS alongside bulk ESS because the substantive claims are statements about the sign of credible intervals, whose endpoints depend on tail behavior that bulk ESS does not measure\.

As§\\S[6](https://arxiv.org/html/2608.04463#S6)argues, these diagnostics evaluate computation under the assumed model; they do not establish that the scale\-identifying observations represent the intended construct\. We report them for completeness and treat the anchor audit of§\\S[3\.4](https://arxiv.org/html/2608.04463#S3.SS4)as the substantive validity check\.

Table 9:Convergence diagnostics\.R^\\hat\{R\}Vehtari et al\. \([2021](https://arxiv.org/html/2608.04463#bib.bib27)\)is the maximum andE​S​SESSis the minimum over all coordinates of the reported parameter blocks, including total condition shifts, decomposition components, evaluator\-side peer\-context loadings, and, where applicable, the secondary harmful\-beneficial contrast\. Every fit uses four chains and has zero divergent transitions\.FitR^\\hat\{R\}ESS bulkESS tailDecomposition \(Tables[10](https://arxiv.org/html/2608.04463#A4.T10),[11](https://arxiv.org/html/2608.04463#A4.T11),[12](https://arxiv.org/html/2608.04463#A4.T12)\)TruthfulQA1\.0051,5372,679MMLU\-Pro1\.0031,6182,288ARC\-Challenge1\.0071,3462,325Total condition shifts \(Table[3](https://arxiv.org/html/2608.04463#S3.T3)\)TruthfulQA1\.0077151,463MMLU\-Pro1\.0031,3242,091ARC\-Challenge1\.0022,4443,572Self\-ratings excluded \(Table[17](https://arxiv.org/html/2608.04463#A5.T17)\)TruthfulQA1\.0041,5422,661MMLU\-Pro1\.0011,9312,881ARC\-Challenge1\.0013,3814,487Anchor scale \(Table[20](https://arxiv.org/html/2608.04463#A6.T20)\)TruthfulQA,θa=1\.0\\theta\_\{a\}\{=\}1\.01\.0061,5662,562TruthfulQA,θa=2\.0\\theta\_\{a\}\{=\}2\.01\.0041,6432,729Five\-judge panel \(Table[4](https://arxiv.org/html/2608.04463#S3.T4), GPT\-4o row\)TruthfulQA1\.0078311,239

## Appendix DComplete Decomposition Results

This section reports the generator\-level decomposition estimates for each benchmark\. Within each self\-contained decomposition corpus, the total Round 2 shift is partitioned into ordinary second\-pass revision, candidate\-content exposure, and the bundled peer\-presentation residual\. These estimates are obtained from separately constructed decomposition corpora and therefore need not equal the pooled main\-corpus total shifts in Table[3](https://arxiv.org/html/2608.04463#S3.T3)\.

### D\.1Full TruthfulQA Decomposition

Table[10](https://arxiv.org/html/2608.04463#A4.T10)decomposes each TruthfulQA condition shift into ordinary second\-pass revision, candidate content exposure, and the peer\-presentation residual\. The residual is credibly negative in all twelve cells, so routing candidate text through an attributed peer block lowers revision quality regardless of peer polarity\. Ordinary re\-answering is small: it excludes zero for only two of the four generators, and is positive in both cases\. The content channel carries most of the correct\-versus\-wrong asymmetry\. Correct content is credibly positive for Qwen2\.5\-7B and Llama\-3\.1\-8B and indistinguishable from zero for Mistral\-7B and Gemma\-2\-9B, while wrong content is credibly negative for the same two generators\. Llama\-3\.1\-8B shows the largest content effects in both directions \(\+0\.17\+0\.17under correct content,−0\.24\-0\.24under wrong content\), and is the one generator whose all\-wrong degradation is carried mainly by content rather than by presentation\.

Table 10:Decomposition of the total Round 2 shift on TruthfulQA into second\-pass revisionΔsp\\Delta\_\{\\mathrm\{sp\}\}, content exposureΔcontent,k​\(c\)\\Delta\_\{\\mathrm\{content\},k\(c\)\}, and the peer\-presentation residualΔpp,c\\Delta\_\{\\mathrm\{pp\},c\}\. Entries are posterior means with 95% credible intervals\. By construction,Δtotal,c=Δsp\+Δcontent,k​\(c\)\+Δpp,c\\Delta\_\{\\mathrm\{total\},c\}=\\Delta\_\{\\mathrm\{sp\}\}\+\\Delta\_\{\\mathrm\{content\},k\(c\)\}\+\\Delta\_\{\\mathrm\{pp\},c\}\. The final column reports the total shift within the decomposition corpus\. Because the decomposition and pooled analyses use separately constructed generation corpora and separate hierarchical fits, these totals need not equal the pooled estimates in Table[3](https://arxiv.org/html/2608.04463#S3.T3)\.ModelConditionΔsp\\Delta\_\{\\mathrm\{sp\}\}Δcontent\\Delta\_\{\\mathrm\{content\}\}Δpp,c\\Delta\_\{\\mathrm\{pp\},c\}Δtotal,c\\Delta\_\{\\mathrm\{total\},c\}Qwen2\.5\-7BAll\-Correct−0\.02\-0\.02\[−0\.06\-0\.06,\+0\.03\+0\.03\]\+0\.06\+0\.06\[\+0\.01\+0\.01,\+0\.11\+0\.11\]−0\.09\-0\.09\[−0\.13\-0\.13,−0\.05\-0\.05\]−0\.05\-0\.05\[−0\.09\-0\.09,−0\.02\-0\.02\]All\-Wrong−0\.05\-0\.05\[−0\.10\-0\.10,−0\.00\-0\.00\]−0\.14\-0\.14\[−0\.18\-0\.18,−0\.10\-0\.10\]−0\.21\-0\.21\[−0\.24\-0\.24,−0\.18\-0\.18\]Mixed\+0\.05\+0\.05\[\+0\.00\+0\.00,\+0\.10\+0\.10\]−0\.12\-0\.12\[−0\.16\-0\.16,−0\.07\-0\.07\]−0\.08\-0\.08\[−0\.11\-0\.11,−0\.04\-0\.04\]Mistral\-7BAll\-Correct\+0\.06\+0\.06\[\+0\.02\+0\.02,\+0\.10\+0\.10\]\+0\.01\+0\.01\[−0\.04\-0\.04,\+0\.06\+0\.06\]−0\.10\-0\.10\[−0\.14\-0\.14,−0\.06\-0\.06\]−0\.03\-0\.03\[−0\.07\-0\.07,\+0\.00\+0\.00\]All\-Wrong−0\.03\-0\.03\[−0\.08\-0\.08,\+0\.02\+0\.02\]−0\.21\-0\.21\[−0\.25\-0\.25,−0\.17\-0\.17\]−0\.19\-0\.19\[−0\.22\-0\.22,−0\.15\-0\.15\]Mixed\+0\.00\+0\.00\[−0\.05\-0\.05,\+0\.05\+0\.05\]−0\.14\-0\.14\[−0\.18\-0\.18,−0\.10\-0\.10\]−0\.08\-0\.08\[−0\.11\-0\.11,−0\.04\-0\.04\]Gemma\-2\-9BAll\-Correct\+0\.11\+0\.11\[\+0\.06\+0\.06,\+0\.16\+0\.16\]−0\.03\-0\.03\[−0\.09\-0\.09,\+0\.02\+0\.02\]−0\.09\-0\.09\[−0\.13\-0\.13,−0\.04\-0\.04\]−0\.02\-0\.02\[−0\.05\-0\.05,\+0\.02\+0\.02\]All\-Wrong−0\.03\-0\.03\[−0\.08\-0\.08,\+0\.03\+0\.03\]−0\.14\-0\.14\[−0\.18\-0\.18,−0\.10\-0\.10\]−0\.06\-0\.06\[−0\.10\-0\.10,−0\.02\-0\.02\]Mixed−0\.02\-0\.02\[−0\.07\-0\.07,\+0\.04\+0\.04\]−0\.09\-0\.09\[−0\.14\-0\.14,−0\.05\-0\.05\]−0\.00\-0\.00\[−0\.04\-0\.04,\+0\.03\+0\.03\]Llama\-3\.1\-8BAll\-Correct−0\.01\-0\.01\[−0\.05\-0\.05,\+0\.03\+0\.03\]\+0\.17\+0\.17\[\+0\.12\+0\.12,\+0\.22\+0\.22\]−0\.18\-0\.18\[−0\.22\-0\.22,−0\.14\-0\.14\]−0\.02\-0\.02\[−0\.05\-0\.05,\+0\.01\+0\.01\]All\-Wrong−0\.24\-0\.24\[−0\.29\-0\.29,−0\.19\-0\.19\]−0\.07\-0\.07\[−0\.11\-0\.11,−0\.04\-0\.04\]−0\.32\-0\.32\[−0\.36\-0\.36,−0\.29\-0\.29\]Mixed\+0\.06\+0\.06\[\+0\.01\+0\.01,\+0\.11\+0\.11\]−0\.19\-0\.19\[−0\.23\-0\.23,−0\.15\-0\.15\]−0\.15\-0\.15\[−0\.18\-0\.18,−0\.11\-0\.11\]

### D\.2Full MMLU\-Pro Decomposition

Table[11](https://arxiv.org/html/2608.04463#A4.T11)decomposes each MMLU\-Pro condition shift into ordinary second\-pass revision, candidate\-content exposure, and the residual effect of peer presentation\. The peer\-presentation component is negative in every cell and credibly negative in eleven of twelve; the exception is Llama\-3\.1\-8B under all\-wrong peers, where a larger share of the effect is instead carried by the content channel\. The Llama\-3\.1\-8B all\-correct cell has an upper bound that rounds to−0\.00\-0\.00at two decimals but is strictly negative\.

Table 11:MMLU\-Pro decomposition of the total Round 2 shift into second\-pass revision \(Δsp\\Delta\_\{\\mathrm\{sp\}\}\), content exposure \(Δcontent,k\\Delta\_\{\\mathrm\{content\},k\}\), and the peer\-presentation residual \(Δpp,c\\Delta\_\{\\mathrm\{pp\},c\}\)\. Entries are posterior means with 95% credible intervals\. The final column reports the total shift within the decomposition corpus\. Because the decomposition and pooled analyses use separately constructed generation corpora and separate hierarchical fits, these totals need not equal the pooled estimates in Table[3](https://arxiv.org/html/2608.04463#S3.T3)\.ModelConditionΔsp\\Delta\_\{\\mathrm\{sp\}\}Δcontent,k\\Delta\_\{\\mathrm\{content\},k\}Δpp,c\\Delta\_\{\\mathrm\{pp\},c\}Δtotal,c\\Delta\_\{\\mathrm\{total\},c\}Qwen2\.5\-7BAll\-Correct\+0\.08\+0\.08\[\+0\.02\+0\.02,\+0\.15\+0\.15\]−0\.12\-0\.12\[−0\.20\-0\.20,−0\.05\-0\.05\]−0\.19\-0\.19\[−0\.25\-0\.25,−0\.13\-0\.13\]−0\.23\-0\.23\[−0\.28\-0\.28,−0\.19\-0\.19\]All\-Wrong−0\.21\-0\.21\[−0\.29\-0\.29,−0\.14\-0\.14\]−0\.26\-0\.26\[−0\.32\-0\.32,−0\.20\-0\.20\]−0\.39\-0\.39\[−0\.43\-0\.43,−0\.34\-0\.34\]Mixed−0\.10\-0\.10\[−0\.17\-0\.17,−0\.02\-0\.02\]−0\.17\-0\.17\[−0\.23\-0\.23,−0\.11\-0\.11\]−0\.19\-0\.19\[−0\.23\-0\.23,−0\.14\-0\.14\]Mistral\-7BAll\-Correct\+0\.01\+0\.01\[−0\.05\-0\.05,\+0\.07\+0\.07\]−0\.00\-0\.00\[−0\.07\-0\.07,\+0\.07\+0\.07\]−0\.22\-0\.22\[−0\.28\-0\.28,−0\.16\-0\.16\]−0\.21\-0\.21\[−0\.26\-0\.26,−0\.16\-0\.16\]All\-Wrong−0\.01\-0\.01\[−0\.08\-0\.08,\+0\.06\+0\.06\]−0\.34\-0\.34\[−0\.39\-0\.39,−0\.28\-0\.28\]−0\.34\-0\.34\[−0\.38\-0\.38,−0\.29\-0\.29\]Mixed−0\.03\-0\.03\[−0\.11\-0\.11,\+0\.04\+0\.04\]−0\.24\-0\.24\[−0\.30\-0\.30,−0\.18\-0\.18\]−0\.26\-0\.26\[−0\.30\-0\.30,−0\.21\-0\.21\]Gemma\-2\-9BAll\-Correct\+0\.10\+0\.10\[\+0\.03\+0\.03,\+0\.16\+0\.16\]−0\.06\-0\.06\[−0\.14\-0\.14,\+0\.01\+0\.01\]−0\.17\-0\.17\[−0\.23\-0\.23,−0\.11\-0\.11\]−0\.13\-0\.13\[−0\.18\-0\.18,−0\.09\-0\.09\]All\-Wrong−0\.06\-0\.06\[−0\.14\-0\.14,\+0\.01\+0\.01\]−0\.23\-0\.23\[−0\.29\-0\.29,−0\.17\-0\.17\]−0\.19\-0\.19\[−0\.24\-0\.24,−0\.15\-0\.15\]Mixed−0\.01\-0\.01\[−0\.09\-0\.09,\+0\.07\+0\.07\]−0\.13\-0\.13\[−0\.19\-0\.19,−0\.07\-0\.07\]−0\.05\-0\.05\[−0\.10\-0\.10,−0\.00\-0\.00\]Llama\-3\.1\-8BAll\-Correct\+0\.01\+0\.01\[−0\.05\-0\.05,\+0\.08\+0\.08\]−0\.15\-0\.15\[−0\.22\-0\.22,−0\.08\-0\.08\]−0\.06\-0\.06\[−0\.12\-0\.12,−0\.00\-0\.00\]−0\.20\-0\.20\[−0\.24\-0\.24,−0\.15\-0\.15\]All\-Wrong−0\.42\-0\.42\[−0\.49\-0\.49,−0\.35\-0\.35\]−0\.03\-0\.03\[−0\.08\-0\.08,\+0\.02\+0\.02\]−0\.44\-0\.44\[−0\.48\-0\.48,−0\.39\-0\.39\]Mixed−0\.14\-0\.14\[−0\.21\-0\.21,−0\.07\-0\.07\]−0\.07\-0\.07\[−0\.13\-0\.13,−0\.01\-0\.01\]−0\.20\-0\.20\[−0\.24\-0\.24,−0\.15\-0\.15\]

### D\.3Full ARC\-Challenge Decomposition

The ARC\-Challenge decomposition in Table[12](https://arxiv.org/html/2608.04463#A4.T12)shows the same broad pattern\. Peer presentation is credibly negative in all twelve cells, while wrong candidate content is particularly harmful for several generators\. Correct content does not produce a credibly positive content effect in any cell\.

Table 12:ARC\-Challenge decomposition of the total Round 2 shift into second\-pass revision \(Δsp\\Delta\_\{\\mathrm\{sp\}\}\), content exposure \(Δcontent,k\\Delta\_\{\\mathrm\{content\},k\}\), and the peer\-presentation residual \(Δpp,c\\Delta\_\{\\mathrm\{pp\},c\}\)\. Entries are posterior means with 95% credible intervals\.ModelConditionΔsp\\Delta\_\{\\mathrm\{sp\}\}Δcontent,k\\Delta\_\{\\mathrm\{content\},k\}Δpp,c\\Delta\_\{\\mathrm\{pp\},c\}Qwen2\.5\-7BAll\-Correct\+0\.13\+0\.13\[\+0\.07\+0\.07,\+0\.18\+0\.18\]−0\.17\-0\.17\[−0\.23\-0\.23,−0\.11\-0\.11\]−0\.12\-0\.12\[−0\.17\-0\.17,−0\.08\-0\.08\]All\-Wrong−0\.32\-0\.32\[−0\.38\-0\.38,−0\.26\-0\.26\]−0\.18\-0\.18\[−0\.23\-0\.23,−0\.14\-0\.14\]Mixed−0\.11\-0\.11\[−0\.18\-0\.18,−0\.05\-0\.05\]−0\.21\-0\.21\[−0\.26\-0\.26,−0\.16\-0\.16\]Mistral\-7BAll\-Correct\+0\.06\+0\.06\[\+0\.01\+0\.01,\+0\.11\+0\.11\]−0\.00\-0\.00\[−0\.07\-0\.07,\+0\.06\+0\.06\]−0\.19\-0\.19\[−0\.24\-0\.24,−0\.14\-0\.14\]All\-Wrong−0\.05\-0\.05\[−0\.11\-0\.11,\+0\.00\+0\.00\]−0\.35\-0\.35\[−0\.40\-0\.40,−0\.30\-0\.30\]Mixed−0\.01\-0\.01\[−0\.07\-0\.07,\+0\.06\+0\.06\]−0\.27\-0\.27\[−0\.32\-0\.32,−0\.22\-0\.22\]Gemma\-2\-9BAll\-Correct\+0\.14\+0\.14\[\+0\.09\+0\.09,\+0\.20\+0\.20\]−0\.08\-0\.08\[−0\.15\-0\.15,−0\.02\-0\.02\]−0\.16\-0\.16\[−0\.21\-0\.21,−0\.11\-0\.11\]All\-Wrong−0\.12\-0\.12\[−0\.19\-0\.19,−0\.06\-0\.06\]−0\.23\-0\.23\[−0\.28\-0\.28,−0\.18\-0\.18\]Mixed−0\.11\-0\.11\[−0\.17\-0\.17,−0\.04\-0\.04\]−0\.12\-0\.12\[−0\.17\-0\.17,−0\.07\-0\.07\]Llama\-3\.1\-8BAll\-Correct\+0\.10\+0\.10\[\+0\.04\+0\.04,\+0\.15\+0\.15\]−0\.10\-0\.10\[−0\.16\-0\.16,−0\.04\-0\.04\]−0\.18\-0\.18\[−0\.22\-0\.22,−0\.13\-0\.13\]All\-Wrong−0\.49\-0\.49\[−0\.55\-0\.55,−0\.43\-0\.43\]−0\.12\-0\.12\[−0\.17\-0\.17,−0\.08\-0\.08\]Mixed−0\.17\-0\.17\[−0\.23\-0\.23,−0\.11\-0\.11\]−0\.17\-0\.17\[−0\.21\-0\.21,−0\.12\-0\.12\]

## Appendix EExternal Validation

The latent\-quality analysis is compared with independent lexical, semantic, and classifier\-based signals\. These checks test whether the peer\-condition ordering persists outside the hierarchical LLM\-judge model and help distinguish changes in answer quality from simple adoption of peer wording\.

### E\.1Lexical Reference\-Based Metrics

Table[13](https://arxiv.org/html/2608.04463#A5.T13)summarizes reference\-based overlap for Round 2 answers\. Across datasets, all\-correct responses have the highest overlap with the gold answer, mixed responses are intermediate, and all\-wrong responses are lowest among the peer\-presented conditions\. Because the references are much shorter than the generated answers, recall\-oriented ROUGE\-L and METEORBanerjee and Lavie \([2005](https://arxiv.org/html/2608.04463#bib.bib2)\)are more informative here than precision\-sensitive BLEU and ROUGE\-L F1\.

Table 13:Reference\-based lexical metrics for Round 2 answers by dataset and condition\. Scores are computed against the gold answer;n=12,000n=12\{,\}000per peer\-condition cell andn=6,000n=6\{,\}000for no\-peer\.DatasetConditionROUGE\-L RecallROUGE\-L F1METEORBLEUTruthfulQANo\-Peer0\.4700\.2010\.3444\.83All\-Correct0\.7090\.3030\.50110\.50Mixed0\.6250\.2580\.4468\.52All\-Wrong0\.4860\.1980\.3495\.27MMLU\-ProNo\-Peer0\.4000\.0950\.1561\.37All\-Correct0\.7570\.1800\.3095\.59Mixed0\.6580\.1550\.2684\.60All\-Wrong0\.4330\.1030\.1722\.09ARC\-ChallengeNo\-Peer0\.4440\.0960\.1700\.64All\-Correct0\.8380\.1840\.3464\.67Mixed0\.7950\.1690\.3214\.36All\-Wrong0\.4680\.0930\.1720\.90
### E\.2Convergent Validity from Natural Language Generation Evaluation Metrics

The condition\-level results above provide a model\-free check on the principal peer ordering\. We additionally correlate each answer’s blind latent\-quality estimate with external metrics in the main\-text convergent\-validity analysis, testing agreement at the answer level rather than only in condition averages\.

We corroborate the model\-free peer effect with continuous, reference\-based metrics computed from the generated text against the gold answer \(Table[13](https://arxiv.org/html/2608.04463#A5.T13)\)\. The lexical battery \(ROUGE\-L, BLEU, METEOR\) was computed over all147,000147,000pooled answers; because references are short relative to free\-text answers \(a roughly six\-fold length ratio, so the brevity penalty never binds\), we read ROUGE\-L*recall*and METEOR as the quality\-relevant signals and report BLEU and F1 for completeness\.

### E\.3Semantic Similarity and Peer\-Language Adoption

Table[14](https://arxiv.org/html/2608.04463#A5.T14)separates similarity to the gold answer from similarity to the peer block\. Gold similarity and BERTScoreZhang et al\. \([2020](https://arxiv.org/html/2608.04463#bib.bib32)\)reproduce the peer\-condition quality ordering, whereas peer\-block similarity rises sharply whenever peer text is shown, regardless of whether that text is correct\. The latter pattern indicates broad language adoption and cautions against interpreting overlap with correct peers as pure quality improvement\.

Table 14:Semantic similarity to the gold answer and peer block, plus BERTScore F1 against gold, for Round 2 answers\.n=12,000n=12\{,\}000per peer\-condition cell andn=6,000n=6\{,\}000for no\-peer\.DatasetConditionsimgold\\mathrm\{sim\}\_\{\\mathrm\{gold\}\}simpeer\\mathrm\{sim\}\_\{\\mathrm\{peer\}\}BERTScoreTruthfulQANo\-Peer0\.622−0\.005\-0\.0050\.345All\-Correct0\.7100\.5900\.449Mixed0\.6650\.6640\.405All\-Wrong0\.6020\.5990\.339MMLU\-ProNo\-Peer0\.3630\.0090\.045All\-Correct0\.4720\.3980\.168Mixed0\.4410\.4260\.134All\-Wrong0\.3850\.3950\.081ARC\-ChallengeNo\-Peer0\.367−0\.019\-0\.0190\.105All\-Correct0\.5190\.4170\.211Mixed0\.4830\.4220\.192All\-Wrong0\.3630\.3750\.102
### E\.4Convergent Validity of Latent Quality

The answer\-level correlations reported in Table[15](https://arxiv.org/html/2608.04463#A5.T15)are positive for every metric and dataset\. The stronger persistence of semantic and entailment\-based associations on MMLU\-Pro and ARC\-Challenge is consistent with their terse reference answers, for which lexical overlap is an especially incomplete proxy for response quality\.

Table 15:Convergent validity: Spearmanρ\\rhobetween blind latent qualityθ^\\hat\{\\theta\}and external metrics over non\-anchor answers \(n=48,000n=48\{,\}000per dataset\)\. All correlations are positive\. Lexical metrics weaken on the terse\-option datasets, while semantic and classifier signals persist\.MetricTruthfulQAMMLU\-ProARC\-ChallengeROUGE\-L Recall\+0\.110\+0\.110\+0\.021\+0\.021\+0\.013\+0\.013METEOR\+0\.162\+0\.162\+0\.054\+0\.054\+0\.079\+0\.079BERTScore F1\+0\.234\+0\.234\+0\.029\+0\.029\+0\.080\+0\.080simgold\\mathrm\{sim\}\_\{\\mathrm\{gold\}\}\(SBERT\)\+0\.187\+0\.187\+0\.067\+0\.067\+0\.084\+0\.084NLI entailment\+0\.204\+0\.204\+0\.165\+0\.165\+0\.186\+0\.186−\-\\,False Endorsement\+0\.295\+0\.295\+0\.046\+0\.046\+0\.113\+0\.113

### E\.5Fixed\-Classifier Validation

Table[16](https://arxiv.org/html/2608.04463#A5.T16)provides the strongest external validation because it removes a generative LLM judge from the evaluation loop\. Across all three benchmarks, false endorsement is lowest for all\-correct responses, intermediate for mixed responses, and highest for all\-wrong responses\. On TruthfulQA, the independent truth\-judge score shows the corresponding higher\-is\-better ordering\. These results reproduce the main condition pattern without relying on the hierarchical LLM\-judge model\.

Table 16:Fixed\-classifier validation by benchmark and Round 2 condition\. False endorsement is RoBERTa\-MNLI entailment of the wrong attractor, so lower values are better\. The TruthfulQA truth\-judge score is higher\-is\-better\.ConditionFalse Endorsement \(↓\\downarrow\)Truth Judge \(↑\\uparrow\)TruthfulQANo\-Peer0\.1420\.802All\-Correct0\.0900\.911Mixed0\.1750\.826All\-Wrong0\.3260\.710MMLU\-ProNo\-Peer0\.143All\-Correct0\.137Mixed0\.159All\-Wrong0\.325ARC\-ChallengeNo\-Peer0\.146All\-Correct0\.137Mixed0\.172All\-Wrong0\.350
### E\.6Self\-Preference Refits

Table[17](https://arxiv.org/html/2608.04463#A5.T17)reports the secondary harmful\-beneficial contrast after removing every rating a judge assigned to answers generated by its own model family\. The contrast remains positive in all twelve generator–benchmark cells\. Changes are small and run in both directions: the largest is Qwen2\.5\-7B on ARC\-Challenge \(\+0\.640\+0\.640to\+0\.566\+0\.566\)\. Gemma\-2\-9B strengthens on all three benchmarks, including the weakest cell in the main fit \(TruthfulQA,\+0\.061\+0\.061to\+0\.105\+0\.105\), indicating that the smallest primary effect is not an artifact of self\-preferential rating\.

Table 17:Self\-preference robustness for the secondary harmful\-beneficial contrast\. The refit removes every rating that a judge assigned to answers produced by its own model family, while retaining anchors\. The contrast remains positive in every generator\-benchmark cell, consistent with the primary finding that all\-wrong responses are more harmful than all\-correct responses\.DatasetGeneratorMain FitSelf\-Excluded FitTruthfulQAQwen2\.5\-7B\+0\.257\+0\.257\+0\.236\+0\.236Mistral\-7B\+0\.217\+0\.217\+0\.216\+0\.216Gemma\-2\-9B\+0\.061\+0\.061\+0\.105\+0\.105Llama\-3\.1\-8B\+0\.327\+0\.327\+0\.304\+0\.304MMLU\-ProQwen2\.5\-7B\+0\.697\+0\.697\+0\.639\+0\.639Mistral\-7B\+0\.641\+0\.641\+0\.636\+0\.636Gemma\-2\-9B\+0\.411\+0\.411\+0\.455\+0\.455Llama\-3\.1\-8B\+0\.704\+0\.704\+0\.727\+0\.727ARC\-ChallengeQwen2\.5\-7B\+0\.640\+0\.640\+0\.566\+0\.566Mistral\-7B\+0\.569\+0\.569\+0\.543\+0\.543Gemma\-2\-9B\+0\.421\+0\.421\+0\.436\+0\.436Llama\-3\.1\-8B\+0\.752\+0\.752\+0\.744\+0\.744

## Appendix FRobustness and Alternative Explanations

This section examines whether the principal condition pattern can be explained by superficial changes in answer form, variation in question difficulty, or arbitrary choices for the anchor scale\. Across these checks, all\-wrong peer exposure remains more harmful than all\-correct peer exposure, and the substantive conclusions are unchanged\.

### F\.1Stability of Generated Answer Form

Table[18](https://arxiv.org/html/2608.04463#A6.T18)reports Flesch Reading Ease, Flesch\-Kincaid, and Gunning Fog index\. Its purpose is to rule out the possibility that condition effects are caused by answers becoming systematically longer, harder to read, or stylistically more complex\.

Table 18:Readability of Round 2 answers by dataset and peer condition\. Flesch Reading Ease \(Flesch RE\) is higher\-is\-easier; Flesch\-Kincaid \(FK Grade\) and Gunning Fog \(Fog\) report grade\-level complexity\. Sample sizes match Table[13](https://arxiv.org/html/2608.04463#A5.T13)\.DatasetConditionFlesch REFK GradeFogTruthfulQANo\-Peer41\.112\.3615\.14All\-Correct41\.812\.3415\.27Mixed42\.312\.3015\.22All\-Wrong41\.812\.4015\.31MMLU\-ProNo\-Peer43\.212\.2915\.58All\-Correct40\.312\.8716\.31Mixed41\.212\.6716\.10All\-Wrong40\.412\.8516\.26ARC\-ChallengeNo\-Peer39\.013\.2316\.49All\-Correct40\.113\.1416\.47Mixed40\.912\.9516\.28All\-Wrong40\.013\.1716\.47
### F\.2Question Difficulty Stratification

Question difficulty is defined from the mean blind Round 1 rating within each benchmark\. Table[19](https://arxiv.org/html/2608.04463#A6.T19)shows that the all\-wrong shift is more negative than the all\-correct shift in every difficulty tercile\. The gap is generally larger for medium and easy questions, indicating that the aggregate result is not driven solely by a small subset of unusually difficult items\.

Table 19:Model\-free Round 2 rating shifts by benchmark and question\-difficulty tercile\. Difficulty is defined by the mean blind Round 1 rating\. The final column reports the difference between the magnitude of the all\-wrong shift and the all\-correct shift, with a question\-level bootstrap 95% confidence interval; positive values indicate greater degradation under all\-wrong peers\.DatasetStratumΔAll​\-​Correct\\Delta\_\{\\mathrm\{All\\text\{\-\}Correct\}\}ΔMixed\\Delta\_\{\\mathrm\{Mixed\}\}ΔAll​\-​Wrong\\Delta\_\{\\mathrm\{All\\text\{\-\}Wrong\}\}\|ΔAll−Wrong\|−ΔAll−Correct\|\\Delta\_\{\\mathrm\{All\-Wrong\}\}\|\-\\Delta\_\{\\mathrm\{All\-Correct\}\}\[95% CI\]TruthfulQAhard\+0\.116\+0\.116\+0\.065\+0\.065−0\.083\-0\.083\+0\.151\+0\.151\[\+0\.069\+0\.069,\+0\.238\+0\.238\]mid−0\.036\-0\.036−0\.091\-0\.091−0\.266\-0\.266\+0\.367\+0\.367\[\+0\.294\+0\.294,\+0\.439\+0\.439\]easy−0\.108\-0\.108−0\.158\-0\.158−0\.303\-0\.303\+0\.437\+0\.437\[\+0\.376\+0\.376,\+0\.507\+0\.507\]MMLU\-Prohard−0\.032\-0\.032−0\.044\-0\.044−0\.181\-0\.181\+0\.326\+0\.326\[\+0\.244\+0\.244,\+0\.404\+0\.404\]mid−0\.253\-0\.253−0\.258\-0\.258−0\.391\-0\.391\+0\.660\+0\.660\[\+0\.579\+0\.579,\+0\.748\+0\.748\]easy−0\.311\-0\.311−0\.288\-0\.288−0\.495\-0\.495\+0\.805\+0\.805\[\+0\.727\+0\.727,\+0\.894\+0\.894\]ARC\-Challengehard\+0\.026\+0\.026−0\.050\-0\.050−0\.264\-0\.264\+0\.320\+0\.320\[\+0\.245\+0\.245,\+0\.391\+0\.391\]mid−0\.189\-0\.189−0\.239\-0\.239−0\.474\-0\.474\+0\.665\+0\.665\[\+0\.591\+0\.591,\+0\.743\+0\.743\]easy−0\.208\-0\.208−0\.237\-0\.237−0\.468\-0\.468\+0\.675\+0\.675\[\+0\.607\+0\.607,\+0\.757\+0\.757\]

### F\.3Anchor Reliability and Scale Sensitivity

Because the anchor magnitude sets the units of the latent scale, changing it should rescale posterior effects without changing their signs or relative ordering\. Table[20](https://arxiv.org/html/2608.04463#A6.T20)illustrates this property using the secondary differential\-harm contrast on TruthfulQA\. The estimates scale approximately with the anchor magnitude, while generator ordering and posterior direction remain stable\.

Table 20:Anchor\-scale sensitivity of the secondary differential\-harm contrast on TruthfulQA as the pinned anchor magnitudeθa\\theta\_\{a\}varies\. Entries are posterior means with 95% credible intervals\. The magnitude rescales as expected, while signs and generator ordering remain stable\.ModelPosterior mean \[95% credible interval\]Qwen2\.5\-7Bθa=1\.0\\theta\_\{a\}=1\.0:\+0\.17\+0\.17\[\+0\.13\+0\.13,\+0\.21\+0\.21\]
θa=1\.5\\theta\_\{a\}=1\.5:\+0\.26\+0\.26\[\+0\.19\+0\.19,\+0\.32\+0\.32\]
θa=2\.0\\theta\_\{a\}=2\.0:\+0\.35\+0\.35\[\+0\.26\+0\.26,\+0\.43\+0\.43\]Mistral\-7Bθa=1\.0\\theta\_\{a\}=1\.0:\+0\.15\+0\.15\[\+0\.11\+0\.11,\+0\.18\+0\.18\]
θa=1\.5\\theta\_\{a\}=1\.5:\+0\.22\+0\.22\[\+0\.16\+0\.16,\+0\.27\+0\.27\]
θa=2\.0\\theta\_\{a\}=2\.0:\+0\.29\+0\.29\[\+0\.21\+0\.21,\+0\.37\+0\.37\]Gemma\-2\-9Bθa=1\.0\\theta\_\{a\}=1\.0:\+0\.05\+0\.05\[\+0\.01\+0\.01,\+0\.09\+0\.09\]
θa=1\.5\\theta\_\{a\}=1\.5:\+0\.07\+0\.07\[\+0\.01\+0\.01,\+0\.14\+0\.14\]
θa=2\.0\\theta\_\{a\}=2\.0:\+0\.10\+0\.10\[\+0\.01\+0\.01,\+0\.18\+0\.18\]Llama\-3\.1\-8Bθa=1\.0\\theta\_\{a\}=1\.0:\+0\.23\+0\.23\[\+0\.19\+0\.19,\+0\.27\+0\.27\]
θa=1\.5\\theta\_\{a\}=1\.5:\+0\.35\+0\.35\[\+0\.29\+0\.29,\+0\.40\+0\.40\]
θa=2\.0\\theta\_\{a\}=2\.0:\+0\.46\+0\.46\[\+0\.38\+0\.38,\+0\.54\+0\.54\]