OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

# Forecasting Bias and the Alignment Effect in Language Model Judgment
Source: [https://arxiv.org/html/2607.26981](https://arxiv.org/html/2607.26981)
Seonglae Cho1,2Adriano Koshiyama1,2 1Holistic AI2University College London

###### Abstract

Large language models are increasingly used as decision aids whose probability judgments shape downstream choices\. Whether those judgments carry a systematic directional tilt has been hard to detect: calibration metrics aggregate unsigned errors, and naturalistic uncertainty offers no ground\-truth probability\. When an LLM rates a startup’s success at70%70\\%but its failure at15%15\\%, the missing1515points expose a distortion no aggregate score flags\. We introduceOptimismBench, which detects directional bias with*inverted pairs*: each scenario elicits bothP​\(success\)P\(\\text\{success\}\)andP​\(failure\)P\(\\text\{failure\}\), and asymmetry between the two framings yields a signed bias score without ground truth\. Across 16 models from 8 providers, fourteen are optimistic; pessimism appears only in Anthropic’s frontier tier\. Eleven matched base\-versus\-chat pairs across four families show post\-training sets the sign of the bias, with opposite shifts in different families\. The pattern survives prompt, temperature, perspective, and self\-debiasing ablations\. A seventeen\-model six\-language comparison further shows model identity dominates language, with inter\-model variance at4\.7×4\.7\\timesinter\-language variance\. We release3,8703\{,\}870items across1010languages for per\-model directional\-bias auditing\. When alignment makes a model more helpful, it also tilts its probabilities; downstream pipelines inherit the tilt by default\.

OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment

Seonglae Cho††thanks:Correspondence:seonglae\.cho\.24@ucl\.ac\.uk1,2Adriano Koshiyama1,21Holistic AI2University College London

## 1Introduction

Table 1:Track B Skew \(mean\) and per\-itemσ\\sigmaacross 16 models\. All entries significant atp<0\.002p<0\.002\(Bonferroni\-adjusted threshold for 16 tests atα=0\.05\\alpha\{=\}0\.05\)\. Row tint∝\|Skew\|\\propto\|\\text\{Skew\}\|\. Per\-model statistics in Appendix[A\.6](https://arxiv.org/html/2607.26981#A1.SS6)\.![Refer to caption](https://arxiv.org/html/2607.26981v1/x1.png)Figure 1:The inverted\-pair method:Skew=P\+¯−\(100−P−¯\)\\text\{Skew\}=\\overline\{P^\{\+\}\}\-\(100\-\\overline\{P^\{\-\}\}\)scores directional asymmetry\.In cognitive psychology, humans exhibit a well\-documented*optimism bias*: the tendency to overestimate the probability of positive outcomes and underestimate negative outcomesWeinstein \([1980](https://arxiv.org/html/2607.26981#bib.bib42)\); Sharot \([2011](https://arxiv.org/html/2607.26981#bib.bib34)\)\. Prospect theory formalizes a related asymmetry, showing that people weight gains and losses differently when making decisions under riskKahneman and Tversky \([1979](https://arxiv.org/html/2607.26981#bib.bib17)\)\. LLM\-based forecasting systems are now approaching crowd\-human accuracy on prediction marketsHalawi et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib14)\); Schoenegger et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib32)\)and are increasingly deployed for risk assessment and project planning\. Whether LLMs exhibit a similar valence\-dependent distortion in their probability judgments, and whether such bias propagates into the forecasts these systems produce, has not been directly measured\.

Existing evaluation frameworks cannot detect this asymmetry: calibration metricsGuo et al\. \([2017](https://arxiv.org/html/2607.26981#bib.bib13)\)aggregate unsigned errors \(a model with ECE = 5 could be uniformly optimistic, pessimistic, or neither\), and prior LLM probability\-coherence workZhu and Griffiths \([2025](https://arxiv.org/html/2607.26981#bib.bib47)\)reports unsigned violations without decomposing direction\.

OptimismBenchmeasures whether LLMs systematically favor positive over negative outcomes in probability judgment\. The core mechanism is*inverted pairs*: for each scenario, the model estimates bothP​\(success\)P\(\\text\{success\}\)andP​\(failure\)P\(\\text\{failure\}\)\. A consistent model produces complementary estimates; systematic deviation reveals directional bias\.

We evaluate 16 models from 8 providers on 60 inverted\-pair scenarios in 10 languages\.

Sign by lab, magnitude by scale\.All 16 models exhibit significant directional bias \(Table[1](https://arxiv.org/html/2607.26981#S1.T1)\): 14 are optimistic, and only Anthropic’s frontier models are pessimistic \(Opus−5\.1\-5\.1, Sonnet−7\.7\-7\.7\)\. Within Anthropic the lightweight Haiku 4\.5 is optimistic \(\+6\.1\+6\.1\) while the larger Opus and Sonnet are pessimistic, ruling out a simple bigger\-is\-less\-biased account\. The pattern is stable across prompt, temperature, perspective, and explicit self\-debiasing\.

Post\-training sets the direction in a family\-specific way\.A controlled probe of eleven matched base\-versus\-chat pairs across four families \(Qwen, Llama, Gemma, Mistral\) keeps architecture and pre\-training constant, varying only post\-training\. Qwen alignment compresses bias on all five pairs and Llama alignment amplifies optimism on all four\. Gemma\-2\-2b and Mistral\-Small\-24B are single\-pair pilots reported alongside\. The within\-provider alignment gradient across the 16 models matches this causal reading: each family’s post\-training stack sets the sign, and within Llama the magnitude scales inversely with model size\.

A seventeen\-model cross\-lingual comparison shows inter\-model variance is4\.7×4\.7\\timesinter\-language variance: which model is used matters more than which language it speaks\. The inverted\-pair design, the controlled base\-versus\-chat probe, and the cross\-lingual variance decomposition together constitute the contribution; the benchmark and all responses are released for replication\.

## 2Related Work

##### Calibration and probability coherence\.

Calibration metricsGuo et al\. \([2017](https://arxiv.org/html/2607.26981#bib.bib13)\); Lin et al\. \([2022a](https://arxiv.org/html/2607.26981#bib.bib21)\); Kadavath et al\. \([2022](https://arxiv.org/html/2607.26981#bib.bib16)\); Tian et al\. \([2023](https://arxiv.org/html/2607.26981#bib.bib38)\); Xiong et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib44)\)and LLM\-forecasting benchmarksHalawi et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib14)\); Schoenegger et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib32)\); Paleka et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib27)\)aggregate unsigned errors and miss tilt direction\. Closest to ours,Zhu and Griffiths \([2025](https://arxiv.org/html/2607.26981#bib.bib47)\)andFreedman and Toni \([2025](https://arxiv.org/html/2607.26981#bib.bib10)\)use paired\-complement elicitation but report*unsigned*coherence diagnostics; we retain the*sign*as Skew, decompose it into valence components, and use it as the dependent variable in a base\-vs\-chat causal probe\.

##### Cognitive biases and alignment side\-effects\.

LLMs inherit anchoring, framing, and other cognitive biasesJones and Steinhardt \([2022](https://arxiv.org/html/2607.26981#bib.bib15)\); Malberg et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib23)\); Echterhoff et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib9)\), and RLHF\-style alignment has documented side\-effects on confidence, sycophancy, opinion shift, and refusalCasper et al\. \([2023](https://arxiv.org/html/2607.26981#bib.bib3)\); Kirk et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib18)\); Leng et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib19)\); Sharma et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib33)\); Wei et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib41)\); Ouyang et al\. \([2022](https://arxiv.org/html/2607.26981#bib.bib26)\); Santurkar et al\. \([2023](https://arxiv.org/html/2607.26981#bib.bib31)\); Cui et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib6)\)\. We isolate a previously unmeasured side\-effect, valence\-dependent probability distortion, and the inverted\-pair design separates it from sycophancy \(§[4\.3](https://arxiv.org/html/2607.26981#S4.SS3)\) and acquiescence overclaim\.

##### Signed bias benchmarks\.

Existing benchmarks target social biasParrish et al\. \([2022](https://arxiv.org/html/2607.26981#bib.bib28)\); Nangia et al\. \([2020](https://arxiv.org/html/2607.26981#bib.bib25)\), accuracyLiang et al\. \([2023](https://arxiv.org/html/2607.26981#bib.bib20)\); Lin et al\. \([2022b](https://arxiv.org/html/2607.26981#bib.bib22)\), or personalityXie et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib43)\); Dodds et al\. \([2014](https://arxiv.org/html/2607.26981#bib.bib8)\)\. The two prior signed cognitive\-bias benchmarks operationalise different constructs: CogBench’s bandit learning\-rate asymmetry on 35 LLMsCoda\-Forno et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib4)\)and AcquiescenceBench’s social\-agreement yes\-bias on 4 LLMsBraun \([2025](https://arxiv.org/html/2607.26981#bib.bib2)\); neither pairs the signed metric with a controlled causal probe\. Our directional\-probability\-judgment construct, drawn from human optimism\-bias and prospect\-theory traditionsWeinstein \([1980](https://arxiv.org/html/2607.26981#bib.bib42)\); Sharot \([2011](https://arxiv.org/html/2607.26981#bib.bib34)\); Kahneman and Tversky \([1979](https://arxiv.org/html/2607.26981#bib.bib17)\), is new\.

## 3Method

### 3\.1Inverted Pair Measurement

We measure directional bias through*inverted pairs*: for each scenario, we ask the model to estimate the probability of a positive outcome and, separately, the probability of the corresponding negative outcome\. A consistent model should produce estimates that sum to 100\. Systematic deviation from this sum reveals directional bias\.

Formally, letsi\+s^\{\+\}\_\{i\}denote the model’s response to the positive\-framed version of scenarioiiandsi−s^\{\-\}\_\{i\}the response to the negative\-framed version, both on a 0–100 integer scale\. We define theSkew111We use*Skew*as the name of the inverted\-pair directional metric defined below; this is distinct from the third standardized moment \(*skewness*\) of a distribution\.of scenarioiiand its model\-level aggregate as:

Skewi=si\+−\(100−si−\),Skew¯=1n​∑i=1nSkewi\.\\text\{Skew\}\_\{i\}=s^\{\+\}\_\{i\}\-\(100\-s^\{\-\}\_\{i\}\),\\quad\\overline\{\\text\{Skew\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\text\{Skew\}\_\{i\}\.\(1\)Positive Skew indicates optimistic bias \(overestimating positive outcomes relative to negative ones\); negative Skew indicates pessimistic bias\.

The formulation measures*internal consistency*: whether positive and negative outcomes are treated symmetrically\. The paired\-complement primitive is classicalKahneman and Tversky \([1979](https://arxiv.org/html/2607.26981#bib.bib17)\); Tversky and Kahneman \([1983](https://arxiv.org/html/2607.26981#bib.bib39)\); Tversky and Koehler \([1994](https://arxiv.org/html/2607.26981#bib.bib40)\); prior LLM applicationsZhu and Griffiths \([2025](https://arxiv.org/html/2607.26981#bib.bib47)\); Freedman and Toni \([2025](https://arxiv.org/html/2607.26981#bib.bib10)\)reported only*unsigned*deviations\. We retain the*sign*and decompose it into two valence components measuring deviation of each axis from the5050\-anchor:

δ\+=P\+¯−50,δ−=P−¯−50,Skew¯=δ\+\+δ−,\\delta^\{\+\}=\\overline\{P^\{\+\}\}\-50,\\;\\;\\delta^\{\-\}=\\overline\{P^\{\-\}\}\-50,\\;\\;\\overline\{\\text\{Skew\}\}=\\delta^\{\+\}\+\\delta^\{\-\},\(2\)whereP\+¯\\overline\{P^\{\+\}\}andP−¯\\overline\{P^\{\-\}\}are the across\-scenario means ofs\+s^\{\+\}ands−s^\{\-\}\. Bothδ\+\>0\\delta^\{\+\}\>0andδ−\>0\\delta^\{\-\}\>0indicate*compound overclaim*\(model rates both success and failure above5050\), whileδ\+<0,δ−<0\\delta^\{\+\}<0,\\delta^\{\-\}<0indicates*compound underclaim*; mixed signs separate classical optimism \(δ\+\>0,δ−<0\\delta^\{\+\}\>0,\\delta^\{\-\}<0\) from pessimism \(δ\+<0,δ−\>0\\delta^\{\+\}<0,\\delta^\{\-\}\>0\) \(Figure[4](https://arxiv.org/html/2607.26981#S5.F4)\)\.

### 3\.2Tracks

We apply inverted pairs across multiple tracks to test whether directional bias manifests\.

##### Track A: Calibration Control\.

Scenarios with explicitly stated base rates \(*e\.g\.*, “the acceptance rate is 25%”\)\. We expectSkew≈0\\text\{Skew\}\\approx 0on these items, so the model can reproduce known probabilities\. This serves as a sanity check: any observed bias in other tracks is uncertainty\-specific, not a general computation failure\.

##### Track B: Probability Estimation\.

Naturalistic scenarios without stated base rates\. The model estimatesP​\(positive outcome\)P\(\\text\{positive outcome\}\)andP​\(negative outcome\)P\(\\text\{negative outcome\}\)\. This is the primary measurement track\. Skew is computed on non\-P=50P\{=\}50responses for consistent treatment across models with heterogeneous refusal rates \(range22–30%30\\%; without filtering, the largest\-refusal model shows artificially low per\-itemσ\\sigmadriven by the5050\-anchor while others shift by<0\.5<0\.5pp; Appendix[A\.6](https://arxiv.org/html/2607.26981#A1.SS6)\)\.

##### Track C: Recommendation\.

The model rates how strongly it recommends a course of action \(0–100\) versus the alternative\. The Skew formula applies identically:Skewi=riaction−\(100−riinaction\)\\text\{Skew\}\_\{i\}=r^\{\\text\{action\}\}\_\{i\}\-\(100\-r^\{\\text\{inaction\}\}\_\{i\}\)\.

##### Track D: Salience\.

The model rates the significance of opportunities \(0–100\) versus risks \(0–100\) in the same scenario\. Skew measures whether the model attends more to upside or downside factors\.

All tracks use the same 0–100 scale, the same inverted pair structure, and the same Skew formula\. Track B is the primary headline track \(16 models, 60 scenarios, 10 runs\); Track A serves as the calibration positive control\. Tracks C and D are reported in Appendix[C\.1](https://arxiv.org/html/2607.26981#A3.SS1)and are not part of the main 16\-model evaluation\.

### 3\.3Factorial Interventions

To decompose the*sources*of bias, we apply controlled interventions that modify a single variable while holding the scenario constant\.

##### Narrative manipulation\.

We append one sentence of positive or negative context to the base scenario \(*e\.g\.*, “The founder has 10 years of industry experience” vs\. “The founder has no prior startup experience”\)\. The*narrative susceptibility*is the difference in mean estimate between positive and negative variants\.

##### Perspective shift\.

We vary the framing: “A person is considering…” vs\. “You are considering…” vs\. “Your friend is considering…”\. This tests whether sycophancy \(desire to please the user\) contributes to bias\. If perspective effect≈0\\approx 0, sycophancy is ruled out as a confound\.

##### Anchoring gradient\.

We vary the precision of base rate information from none \(pure uncertainty\) through qualitative hints \(“most applicants are rejected”\) to exact statistics \(“the acceptance rate is 10%”\)\. This traces the transition from biased judgment \(Track B\) to calibration \(Track A\)\.

##### Self\-debiasing\.

We prepend a warning to the system prompt instructing the model that AI systems often exhibit systematic optimism bias and asking it to correct for this tendency \(full text in Appendix[A\.4\.4](https://arxiv.org/html/2607.26981#A1.SS4.SSS4)\)\. If bias persists under explicit warning, it reflects a deeper property than surface\-level prompt sensitivity\.

### 3\.4Scenarios and Models

##### Scenarios and dataset scale\.

Track B uses 60 controlled inverted\-pair scenarios across 6 domains \(10 per domain: everyday, academic, project, business, policy, health\), each a complementary positive/negative pair through minimal wording changes \(*e\.g\.*, “passes the exam” / “does not pass the exam”\)\. Scenarios are third\-person and culturally neutral, with reasonable probability estimates in2020–80%80\\%\. Unlike forecasting benchmarks grounded in verifiable future eventsHalawi et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib14)\); Paleka et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib27)\), OptimismBench uses controlled vignettes where no ground\-truth resolution exists; the design choice isolates directional bias from calibration error, since Skew is defined axiom\-relative against complementarity rather than against an external base rate\. Track A adds 15 calibration items with stated base rates\. Combined with 10\-language translations and four axiom batteries \(conjunction, conditional, dose\-response, calibration\), the released dataset contains3,8703\{,\}870items\. Author\-constructed scenarios were filtered by cross\-model variance and pair\-consistency checks; bootstrap 95% CIs for model\-level Skew stay within±2\\pm 2–33pp \(Appendix[B\.1](https://arxiv.org/html/2607.26981#A2.SS1)\)\.

##### Models\.

We evaluate 16 models from 8 providers:

- •OpenAI:GPT\-5\.4, GPT\-5\.4\-miniSingh et al\. \([2026](https://arxiv.org/html/2607.26981#bib.bib35)\), GPT\-OSS\-120B
- •Anthropic:Claude Sonnet 4\.6, Claude Haiku 4\.5, Claude Opus 4\.6Bai et al\. \([2022](https://arxiv.org/html/2607.26981#bib.bib1)\)
- •Google:Gemini Pro 3\.1, Gemini Flash 3Comanici et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib5)\)
- •Alibaba:Qwen3\-235B, Qwen3\-Next\-80BYang et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib45)\)
- •DeepSeek:DeepSeek\-V3\.2DeepSeek\-AI et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib7)\)
- •Mistral:Mistral Large, Mistral SmallMistral AI \([2024](https://arxiv.org/html/2607.26981#bib.bib24)\)
- •Meta:Llama 3\.3\-70BGrattafiori et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib12)\)
- •Zhipu:GLM\-4\.7\-flash, GLM\-4\.5\-AirGLM\-5\-Team et al\. \([2026](https://arxiv.org/html/2607.26981#bib.bib11)\)

Each item is evaluated with 5–10 independent runs at temperature 0\.7 \(Claude Opus 4\.6, Gemini Pro 3\.1, and DeepSeek\-V3\.2 used temperature 1\.0\)\. Per\-model run counts and temperatures are documented in Appendix[A\.6](https://arxiv.org/html/2607.26981#A1.SS6); the temperature ablation \(§[4\.3](https://arxiv.org/html/2607.26981#S4.SS3)\) confirms direction is preserved acrossT∈\{0\.7,1\.0\}T\\in\\\{0\.7,1\.0\\\}, and all significance tests use bootstrap or Wilcoxon signed\-rank on the realised counts\. Three additional models \(Nemotron\-3\-super\-120B, Gemma\-4\-31B\-IT, Qwen3\-32B\) participate in the cross\-lingual analysis \(§[4\.6](https://arxiv.org/html/2607.26981#S4.SS6)\) but are not part of the 16\-model headline Table[1](https://arxiv.org/html/2607.26981#S1.T1)\.

## 4Results

### 4\.1Main Result: Directional Bias Exists

Table[1](https://arxiv.org/html/2607.26981#S1.T1)presents Track B results across 16 models from 8 providers\. All 16 models show significant directional bias: fourteen optimistic \(Skew\+4\.2\+4\.2to\+16\.6\+16\.6\) and two pessimistic \(Opus−5\.1\-5\.1, Sonnet−7\.7\-7\.7, both Anthropic frontier\-tier\)\. The pattern spans commercial APIs \(OpenAI, Anthropic, Google\), Chinese providers \(Alibaba, DeepSeek\), European models \(Mistral\), and Meta, so optimistic bias is not provider\-specific\.

##### Calibration control\.

On Track A items where the base rate is explicitly stated in the scenario, all models report the stated value, yielding a directional offset of0\.000\.00\. This confirms the models can faithfully report a stated probability; the Track B bias arises only when probability must be inferred under uncertainty rather than read from text\. Track A and Track B together describe an anchoring gradient: supplying an external probability anchor collapses the bias, and removing it lets the bias re\-emerge\. Directional distortion is recruited when the model constructs a probability, not when it retrieves one\.

##### Variance decomposition\.

An ANOVA\-style decomposition of per\-pair Skew \(Skew∼\\simModel \+ Scenario \+ Domain,767767pairs across the 16\-model headline set\) attributes35\.3%35\.3\\%of variance to the model,24\.2%24\.2\\%to the scenario, and3\.7%3\.7\\%to the domain \(residual36\.8%36\.8\\%\)\. Bias is primarily a model property; domain effects are negligible\.

### 4\.2Alignment Gradient

We separate two types of evidence\. The first is an*observational*within\-provider gradient across commercial tiers \(this subsection\), which is confounded with provider, training data, and safety\-tuning intensity\. The second is a*controlled*base\-versus\-chat probe across matched pairs, which holds architecture and pre\-training fixed and is causal\-suggestive within the tested families\. The observational gradient is consistent with the controlled finding but does not stand on its own as causal evidence\.

Within three of four multi\-tier providers, smaller models are more optimistic:

- •OpenAI:\+13\.1→\+10\.0\+13\.1\\to\+10\.0\(Δ=−3\.1\\Delta=\-3\.1\)
- •Anthropic:\+6\.1→−5\.1→−7\.7\+6\.1\\to\-5\.1\\to\-7\.7\(Haiku→\\toOpus→\\toSonnet\)
- •Google:\+5\.1→\+4\.2\+5\.1\\to\+4\.2\(Δ=−0\.9\\Delta=\-0\.9\)
- •Mistral:\+9\.6→\+12\.2\+9\.6\\to\+12\.2\(Δ=\+2\.6\\Delta=\+2\.6, opposite direction\), an empirical exception

The cross\-provider mixed\-effects fixed effect \(small vs\. large, scenario as random intercept\) is positive on average but modest because Mistral pulls against the within\-provider gradient\. The Anthropic three\-tier observation is the most informative: Sonnet \(balanced\) is more pessimistic than Opus \(frontier\) despite being smaller, ruling out a strict scale account\.

##### Alignment\-pair controlled comparison\.

To test whether post\-training itself shifts the bias, we compare matched base\-versus\-chat pairs on eleven architectures across four families \(Table[2](https://arxiv.org/html/2607.26981#S4.T2), Figure[2](https://arxiv.org/html/2607.26981#S4.F2)\)\. The two well\-replicated families are Qwen2\.5/Qwen3 \(5 pairs, 1\.7B–14B\) and Llama\-3\.1/3\.2 \(4 pairs, 1B–70B\); Gemma\-2\-2b and Mistral\-Small\-24B are additional architecture\-pairs reported alongside\. Holding architecture, pre\-training, and evaluation protocol fixed leaves the post\-training procedure as the changed variable across each pair\.

Table 2:Base versus chat Skew across eleven architectures \(Track B EN, 60 pairs, 10 runs\)\.†base parse 58–63%;‡base parse 11%\.†Llama base parse 58–63% \(JSON\-format, not valence\-correlated\); other base checkpoints≥91%\\geq 91\\%\.‡Mistral\-Small\-24B base parses at 10\.6% \(chat at 100%\); reported as a single\-pair pilot\.

##### Family\-specific direction\.

Every Qwen pair shifts*negative*\(−5\.6\-5\.6to−16\.2\-16\.2pp,5/55/5\) and every Llama pair shifts*positive*\(\+15\.2\+15\.2to\+40\.9\+40\.9pp,4/44/4\); Gemma\-2\-2b extends the Llama direction at\+53\.5\+53\.5pp \(flipping from−17\.5\-17\.5to\+36\.0\+36\.0at≥99%\\geq 99\\%parse\), and Mistral\-Small\-24B extends the Qwen direction at−2\.8\-2\.8pp\. The Qwen shift is best read as “toward less optimism” rather than “toward neutral”: within\-family base Skew is itself heterogeneous \(Qwen2\.5\-7B base\+14\.6\+14\.6vs\. Qwen3\-1\.7B base−12\.4\-12\.4\), and the 1\.7B chat checkpoint moves to−21\.4\-21\.4, deepening pre\-existing pessimism rather than centering on zero\. Within\-family magnitudes scale inversely with model size for Llama \(Llama\-3\.2\-1B\+40\.9\+40\.9vs\. Llama\-3\.1\-70B\+15\.2\+15\.2\); the within\-Qwen3 pattern is non\-monotone with a−16\.2\-16\.2pp maximum at 14B\. We treat the family\-level split as an empirical regularity to be explained, not a causal claim about which RLHF design choice drives it\.

##### Reconciling provider gradient and family probe\.

The within\-provider gradient \(§[4\.2](https://arxiv.org/html/2607.26981#S4.SS2)\) and the family\-specific probe \(Table[2](https://arxiv.org/html/2607.26981#S4.T2)\) are jointly consistent: the gradient holds the alignment stack fixed within a provider while varying scale, whereas the probe holds architecture and pre\-training fixed and varies the alignment stack across families\. The Anthropic three\-tier observation \(Haiku\+6\.1\+6\.1, Opus−5\.1\-5\.1, Sonnet−7\.7\-7\.7\) shows both axes at once: a single lab’s stack produces opposite signs at lightweight versus frontier tier\.

![Refer to caption](https://arxiv.org/html/2607.26981v1/x2.png)Figure 2:Base versus chat Skew across eleven architectures\. Qwen family \(blue\) shifts negative; Llama family \(red\) shifts positive; Gemma\-2\-2b \(green\) shows the largest single flip \(\+53\.5\+53\.5pp\)\.

### 4\.3Robustness

We test whether bias direction is an artifact of our experimental setup \(Table[3](https://arxiv.org/html/2607.26981#S4.T3)\)\. First, we vary the system prompt across three levels of detail \(minimal, standard, extended\): all 6 model–track combinations preserve direction\. Second, we compare temperature 0\.7 and 1\.0 on a matched 5\-run pilot, finding≤1\\leq 1pp change in Skew for GPT\-5\.4 \(\+9\.7\+9\.7vs\.\+10\.0\+10\.0\) and Sonnet \(−7\.7\-7\.7vs\.−7\.7\-7\.7\)\. Third, to rule out sycophancy as a confound, we vary the subject from third\-person to second\-person to friend; the perspective effect is near zero \(\|Δ\|<2\|\\Delta\|<2for both models\)\. Fourth, the self\-debiasing intervention prepends an explicit warning about LLM optimism bias to GPT\-5\.4’s system prompt: the mean estimate moves from49\.549\.5to46\.046\.0\(Δ=−3\.6\\Delta=\-3\.6pp\), absorbing only about a third of the model’s\+10\.0\+10\.0Skew; in this single\-model test, the bias is not a surface\-level prompt artifact that an end\-user warning can erase\.

Table 3:Robustness checks\. Direction preserved under prompt ablation, temperature, perspective shift, and explicit self\-debiasing\.
### 4\.4Narrative Susceptibility

Appending a single sentence of positive or negative context shifts estimates by\+13\+13–1515pp \(positive narrative\) or−7\-7–1010pp \(negative\), yielding total susceptibility of\+20\.5\+20\.5\(GPT\-5\.4\) and\+25\.8\+25\.8\(Claude Sonnet\)\. Yet the*direction*of bias is preserved: GPT\-5\.4 remains optimistic and Claude Sonnet remains pessimistic regardless of narrative valence\. Across these two tested models, framing modulates magnitude without altering sign\.

### 4\.5Probability Coherence Battery

Beyond inverted\-pair Skew, we evaluate two probability\-axiom batteries:*conjunction*\(50 items:P​\(A\)≥P​\(A∩B\)P\(A\)\\geq P\(A\\cap B\)\) and*dose\-response*monotonicity \(96 items: probability should increase or decrease monotonically with directional evidence\)\. Across five models with full coverage \(Gemini Flash 3, Mistral Large, Haiku 4\.5, Sonnet 4\.6, Opus 4\.6\), conjunction violations are low \(0–10%10\\%\) while dose\-response reversal is consistently high \(2727–47%47\\%\); per\-model rates appear in Appendix[B\.6](https://arxiv.org/html/2607.26981#A2.SS6)\(Table[15](https://arxiv.org/html/2607.26981#A2.T15)\), and the per\-language breakdown in Appendix[B\.7](https://arxiv.org/html/2607.26981#A2.SS7)\(Table[16](https://arxiv.org/html/2607.26981#A2.T16)\)\.

##### Cross\-lingual axiom robustness\.

Three models \(Sonnet 4\.6, Haiku 4\.5, Gemini Flash 3\) have full 10\-language×\\times5\-axiom\-type coverage\. Conjunction\-violation rates are low and cross\-linguistically stable, from0\.50\.5% for Sonnet to8\.58\.5% for Haiku, withσ≤3\.4\\sigma\\leq 3\.4pp across languages; the∼10\\sim\\\!10pp within\-Anthropic spread shows tier choices modulate axiom adherence\. Dose\-response monotonicity violations are far higher but equally language\-stable, ranging30\.330\.3% \(Haiku\) to36\.936\.9% \(Gemini\) withσ≤5\.4\\sigma\\leq 5\.4pp\. First, axiom violation rate is a model property with small cross\-lingual variance \(σ≤5\.4\\sigma\\leq 5\.4pp\), mirroring the cross\-lingual stability of Skew \(§[4\.6](https://arxiv.org/html/2607.26981#S4.SS6)\)\. Second, the same alignment stacks that drive conjunction violations to near\-zero do*not*fix monotonicity: dose\-response reversal remains at∼30\\sim\\\!30–47%47\\%across all 10 languages for all three models from two providers, indicating alignment training has differential effects across coherence axioms\.

### 4\.6Cross\-Lingual Patterns

##### Model identity dominates language\.

Across the seventeen models with full 10\-run coverage on six native\-prompt languages \(Table[4](https://arxiv.org/html/2607.26981#S4.T4)\), the inter\-modelσ\\sigmawithin a language is7\.27\.2pp against1\.51\.5pp inter\-language within a model: a4\.7×4\.7\\timesratio\. Aggregate per\-language Skew ranges only from\+5\.0\+5\.0to\+6\.4\+6\.4\(1\.41\.4pp band\), so language\-specific effects largely cancel across models; LLMs do not inherit the language\-positivity gradient documented in human corporaDodds et al\. \([2014](https://arxiv.org/html/2607.26981#bib.bib8)\)\.

##### Alignment compresses magnitude and cross\-lingual variance together\.

Anthropic frontier\-tier models are the most cross\-linguistically stable \(Sonnetσ=0\.54\\sigma\{=\}0\.54, Opusσ=0\.89\\sigma\{=\}0\.89, Haikuσ=1\.16\\sigma\{=\}1\.16\); the four highest\-\|\|Skew\|\|models have mean inter\-languageσ\\sigmaof2\.72\.7pp vs\.1\.01\.0pp for the four most\-stable \(Figure[3](https://arxiv.org/html/2607.26981#S4.F3)\)\. Among headline\-cohort\-only models the split is2\.72\.7vs\.1\.11\.1pp, ruling out an extension\-sampling artifact\. We do not attribute the residual within\-model cross\-lingual variation to language as such; translated scenarios introduce framing, tokenization, and corpus confounds\. The four English\-system\-prompt languages \(DE, FR, HI, JA\) are reported separately in the appendix\.

![Refer to caption](https://arxiv.org/html/2607.26981v1/x3.png)Figure 3:Bias\-stability plane\.xx: mean signed Skew across six native\-prompt languages\.yy: inter\-languageσlang\\sigma\_\{\\text\{lang\}\}of Skew\. Anthropic frontier cluster on the robust axis \(lowσlang\\sigma\_\{\\text\{lang\}\}\); high\-magnitude models \(Mistral Small, GLM\-4\.7\-flash\) sit upper\-right\.

### 4\.7Domain Analysis

Bias magnitude varies by domain: the health\_habits domain produces the largest spread \(\+19\.9\+19\.9for GPT\-5\.4\-mini vs\.−10\.2\-10\.2for Sonnet\), the academic domain the smallest\. An ANOVA\-style decomposition attributes35\.3%35\.3\\%of total Skew variance to model identity and3\.7%3\.7\\%to domain, with the remainder distributed across item and residual\. Bias is primarily a model property; domain modulates expression but does not flip direction\. Per\-domain Skew for the 6 Tier\-1 models appears in Appendix[B\.3](https://arxiv.org/html/2607.26981#A2.SS3)\(Table[12](https://arxiv.org/html/2607.26981#A2.T12)\)\.

### 4\.8Valence Asymmetry

Skew decomposes into a good\-side and a bad\-side component \(Figure[4](https://arxiv.org/html/2607.26981#S5.F4)\); the two components separate optimistic and pessimistic models\. GPT\-5\.4\-mini sits in the*compound\-overclaim*quadrant:δ\+=\+7\.3\\delta^\{\+\}\\,\{=\}\\,\+7\.3andδ−=\+5\.8\\delta^\{\-\}\\,\{=\}\\,\+5\.8, so bothP​\(good\)P\(\\text\{good\}\)andP​\(bad\)P\(\\text\{bad\}\)are inflated above5050\(joint inflation, not directional optimism\)\. Claude Sonnet shows*good\-side pessimism*: the negative Skew arises almost entirely fromP​\(good\)P\(\\text\{good\}\)underestimation \(−6\.6\-6\.6\) whileP​\(bad\)P\(\\text\{bad\}\)is near\-accurate \(−1\.1\-1\.1\)\. Sonnet does not overestimate risks; it underestimates opportunities\.

Table 4:Cross\-lingual Skew across 17 models on six native\-prompt languages \(60 pairs, 10 runs\)\.P=50P\{=\}50refusals excluded\.

## 5Discussion

##### Post\-training shapes bias direction\.

The controlled base\-versus\-chat probe \(Table[2](https://arxiv.org/html/2607.26981#S4.T2)\) attributes direction to post\-training within two well\-powered families\. Qwen alignment compresses bias on all five negative\-direction pairs \(binomialp=1/32p\\,\{=\}\\,1/32\), and Llama alignment amplifies optimism on all four positive\-direction pairs \(p=1/16p\\,\{=\}\\,1/16\)\. The within\-provider observational gradient \(OpenAI, Anthropic, Google\) is*consistent*with this reading; we present it as a descriptive cross\-section, not as independent causal evidence, since provider tiers covary with multiple post\-training factors\. Within that consistent reading, the Anthropic non\-monotonicity \(Haiku\+6\.1\+6\.1, Opus−5\.1\-5\.1, Sonnet−7\.7\-7\.7\) is the observation that is hardest to reconcile with a pure\-scale account\. Gemma\-2\-2b \(\+53\.5\+53\.5pp shift\) and Mistral\-Small\-24B \(−2\.8\-2\.8pp\) are single\-pair pilots reported alongside, not part of the well\-powered family claim\.

##### Hypothesised mechanism\.

The three families’ post\-training recipes differ in ways that align with the observed direction split: QwenQwen et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib29)\); Yang et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib45)\)couples DPO with GRPO against a truthfulness/robustness\-weighted reward; Llama 3Grattafiori et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib12)\)runs DPO against a helpfulness\-weighted reward, which RLHF studies have linked to overconfident outputs\(Leng et al\.,[2025](https://arxiv.org/html/2607.26981#bib.bib19)\); Gemma 2Team et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib37)\)uses on\-policy distillation from a larger conversational teacherSinghal et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib36)\); Sharma et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib33)\)\. We treat these as hypotheses to be tested with reward\-model probes, not confirmed mechanisms\.

##### Dose\-response is alignment\-resistant\.

Conjunction violations are near\-zero \(0–1010%\) while dose\-response monotonicity violations remain at2626–4747% across all1010languages for the33models with full cross\-lingual axiom coverage \(Sonnet, Haiku, Gemini Flash\)\. Conjunction tests a one\-step probability identity \(P​\(A\)≥P​\(A∩B\)P\(A\)\\geq P\(A\\cap B\)\) recoverable from a single calibration target; dose\-response requires consistent monotonic updating across a44\-step evidence ladder\. Post\-training corrects single\-step coherence more readily than multi\-step monotonic reasoning across an evidence sequenceLeng et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib19)\)\.

![Refer to caption](https://arxiv.org/html/2607.26981v1/x4.png)Figure 4:Valence decomposition: good\-side push \(P\+¯−50\\overline\{P^\{\+\}\}\\\!\-\\\!50\) vs\. bad\-side push \(P−¯−50\\overline\{P^\{\-\}\}\\\!\-\\\!50\) across six models\. Arrows trace within\-provider small\-to\-large shift\.
##### Distinct bias mechanisms\.

Decomposing Skew into its positive and negative components reveals that optimistic and pessimistic models distort probability differently \(Figure[4](https://arxiv.org/html/2607.26981#S5.F4)\)\. GPT\-5\.4\-miniSingh et al\. \([2026](https://arxiv.org/html/2607.26981#bib.bib35)\)sits in the compound\-overclaim quadrant: bothP​\(good\)P\(\\text\{good\}\)andP​\(bad\)P\(\\text\{bad\}\)are inflated above5050\(δ\+=\+6\.7\\delta^\{\+\}\\,\{=\}\\,\+6\.7,δ−=\+6\.4\\delta^\{\-\}\\,\{=\}\\,\+6\.4\); the positive Skew comes from joint inflation rather than directional optimism\. Claude Sonnet shows good\-side pessimism: the negative Skew arises almost entirely fromP​\(good\)P\(\\text\{good\}\)underestimation whileP​\(bad\)P\(\\text\{bad\}\)is near\-accurate\. The same Skew magnitude can therefore reflect very different cognitive profiles; correcting two\-axis optimism requires adjusting both axes, while correcting good\-side pessimism requires boosting only positive\-outcome estimates\. Sonnet’s pessimism is not “depressive realism”: itsP​\(bad\)P\(\\text\{bad\}\)estimates are near\-accurate, but itsP​\(good\)P\(\\text\{good\}\)underestimates are systematic, indicating bias rather than superior calibration\.

##### Bias is task\-framing\-dependent\.

A cross\-track pilot \(7 models, 10 scenarios; Appendix[C\.1](https://arxiv.org/html/2607.26981#A3.SS1)\) reveals that the same model on the same scenario produces opposite\-sign Skew under different elicitation frames: Track D \(salience\) is uniformly optimistic \(\+16\+16to\+42\+42pp\) while Track C \(recommendation\) is uniformly cautious \(−13\-13to−69\-69pp in 6/7 models\)\. Directional bias is therefore not a single latent tilt that propagates uniformly; the recommendation framing elicits a cautious\-action bias while the salience framing elicits an opportunity\-positive bias\.

##### Practical implications\.

Smaller, cheaper models carry greater optimistic biasRathi et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib30)\), and a single sentence of narrative context shifts estimates by∼20\{\\sim\}20ppYe et al\. \([2025](https://arxiv.org/html/2607.26981#bib.bib46)\)\. OptimismBench provides a quantitative Skew metric that practitioners can use to select models by bias profile; we note that Skew is an internal\-coherence measure against the complementarity axiom rather than a deviation from human judgment, so absolute magnitudes should be interpreted relative to other models rather than as ground\-truth distortion\.

## 6Conclusion

We introducedOptimismBenchand showed that directional probability bias is universal across 16 commercial and open\-weight LLMs; pessimism is confined to Anthropic’s frontier tier, with the*sign*set by family\-specific post\-training \(Qwen5/55/5, Llama4/44/4\) and the*magnitude*scaling inversely with model size\. Inter\-model variance is4\.7×4\.7\\timesinter\-language variance, so model identity is the primary axis of variation\. Skew is operational and axiom\-relative; the benchmark and 3,870 released items enable per\-model auditing across 10 languages\. For LLM forecasting and advisory systems, the practical consequence is that a model’s directional bias under probability framing is a stable, measurable property; however, a cross\-track pilot shows the sign can reverse under recommendation framing, so practitioners should profile models per task type rather than assume a single bias direction propagates uniformly\.

## 7Limitations

The family\-direction claim rests on the Qwen5/55/5and Llama4/44/4pairs; Gemma\-2\-2b and Mistral\-Small\-24B are directional single\-pair pilots that do not support within\-family power estimates\. Cross\-lingual evidence uses six native\-prompt languages plus four with English system prompts \(DE, FR, HI, JA\), reported separately as a known confound\. Skew measures internal asymmetry against the complementarity axiom rather than deviation from human judgment; a matched human baselineWeinstein \([1980](https://arxiv.org/html/2607.26981#bib.bib42)\)remains future work\.

## References

- Bai et al\. \(2022\)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran\-Johnson, Ethan Perez, and 32 others\. 2022\.[Constitutional ai: Harmlessness from ai feedback](https://arxiv.org/abs/2212.08073)\.*Preprint*, arXiv:2212\.08073\.
- Braun \(2025\)Daniel Braun\. 2025\.[Acquiescence bias in large language models](https://doi.org/10.18653/v1/2025.findings-emnlp.607)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 11341–11355, Suzhou, China\. Association for Computational Linguistics\.
- Casper et al\. \(2023\)Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel\-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J\.K\. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, and 13 others\. 2023\.[Open problems and fundamental limitations of reinforcement learning from human feedback](https://openreview.net/forum?id=bx24KpJ4Eb)\.*Transactions on Machine Learning Research*\.
- Coda\-Forno et al\. \(2024\)Julian Coda\-Forno, Marcel Binz, Jane X\. Wang, and Eric Schulz\. 2024\.[Cogbench: a large language model walks into a psychology lab](https://arxiv.org/abs/2402.18225)\.*Preprint*, arXiv:2402\.18225\.
- Comanici et al\. \(2025\)Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan\-Jiang Jiang, and 3416 others\. 2025\.[Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://arxiv.org/abs/2507.06261)\.*Preprint*, arXiv:2507\.06261\.
- Cui et al\. \(2025\)Justin Cui, Wei\-Lin Chiang, Ion Stoica, and Cho\-Jui Hsieh\. 2025\.[Or\-bench: An over\-refusal benchmark for large language models](https://arxiv.org/abs/2405.20947)\.*Preprint*, arXiv:2405\.20947\.
- DeepSeek\-AI et al\. \(2025\)DeepSeek\-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others\. 2025\.[Deepseek\-v3 technical report](https://arxiv.org/abs/2412.19437)\.*Preprint*, arXiv:2412\.19437\.
- Dodds et al\. \(2014\)Peter Sheridan Dodds, Eric M\. Clark, Suma Desu, Morgan R\. Frank, Andrew J\. Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M\. Kloumann, James P\. Bagrow, Karine Megerdoomian, Matthew T\. McMahon, Brian F\. Tivnan, and Christopher M\. Danforth\. 2014\.[Human language reveals a universal positivity bias](https://arxiv.org/abs/1406.3855)\.*Preprint*, arXiv:1406\.3855\.
- Echterhoff et al\. \(2024\)Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He\. 2024\.[Cognitive bias in decision\-making with LLMs](https://doi.org/10.18653/v1/2024.findings-emnlp.739)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 12640–12653, Miami, Florida, USA\. Association for Computational Linguistics\.
- Freedman and Toni \(2025\)Gabriel Freedman and Francesca Toni\. 2025\.[Exploring the potential for large language models to demonstrate rational probabilistic beliefs](https://doi.org/10.32473/flairs.38.1.138892)\.In*Proceedings of the 38th International FLAIRS Conference*\.
- GLM\-5\-Team et al\. \(2026\)GLM\-5\-Team, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, and 167 others\. 2026\.[Glm\-5: from vibe coding to agentic engineering](https://arxiv.org/abs/2602.15763)\.*Preprint*, arXiv:2602\.15763\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others\. 2024\.[The llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Guo et al\. \(2017\)Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q\. Weinberger\. 2017\.[On calibration of modern neural networks](https://arxiv.org/abs/1706.04599)\.*Preprint*, arXiv:1706\.04599\.
- Halawi et al\. \(2024\)Danny Halawi, Fred Zhang, Chen Yueh\-Han, and Jacob Steinhardt\. 2024\.[Approaching human\-level forecasting with language models](https://openreview.net/forum?id=FlcdW7NPRY)\.In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*\.
- Jones and Steinhardt \(2022\)Erik Jones and Jacob Steinhardt\. 2022\.[Capturing failures of large language models via human cognitive biases](https://openreview.net/forum?id=fcO9Cgn-X-R)\.In*Advances in Neural Information Processing Systems*\.
- Kadavath et al\. \(2022\)Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield\-Dodds, Nova DasSarma, Eli Tran\-Johnson, Scott Johnston, Sheer El\-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others\. 2022\.[Language models \(mostly\) know what they know](https://arxiv.org/abs/2207.05221)\.*Preprint*, arXiv:2207\.05221\.
- Kahneman and Tversky \(1979\)Daniel Kahneman and Amos Tversky\. 1979\.Prospect theory: An analysis of decision under risk\.*Econometrica*, 47\(2\):263–292\.
- Kirk et al\. \(2024\)Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu\. 2024\.[Understanding the effects of RLHF on LLM generalisation and diversity](https://openreview.net/forum?id=PXD3FAVHJT)\.In*The Twelfth International Conference on Learning Representations*\.
- Leng et al\. \(2025\)Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang\. 2025\.[Taming overconfidence in LLMs: Reward calibration in RLHF](https://openreview.net/forum?id=l0tg0jzsdL)\.In*The Thirteenth International Conference on Learning Representations*\.
- Liang et al\. \(2023\)Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D\. Manning, Christopher Ré, Diana Acosta\-Navas, Drew A\. Hudson, and 31 others\. 2023\.[Holistic evaluation of language models](https://arxiv.org/abs/2211.09110)\.*Preprint*, arXiv:2211\.09110\.
- Lin et al\. \(2022a\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022a\.[Teaching models to express their uncertainty in words](https://openreview.net/forum?id=8s8K2UZGTZ)\.*Transactions on Machine Learning Research*\.
- Lin et al\. \(2022b\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022b\.[TruthfulQA: Measuring how models mimic human falsehoods](https://doi.org/10.18653/v1/2022.acl-long.229)\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3214–3252, Dublin, Ireland\. Association for Computational Linguistics\.
- Malberg et al\. \(2025\)Simon Malberg, Roman Poletukhin, Carolin M\. Schuster, and Georg Groh\. 2025\.[A comprehensive evaluation of cognitive biases in LLMs](https://doi.org/10.18653/v1/2025.nlp4dh-1.50)\.In*Proceedings of the 5th International Conference on Natural Language Processing for Digital Humanities*, pages 578–613, Albuquerque, USA\. Association for Computational Linguistics\.
- Mistral AI \(2024\)Mistral AI\. 2024\.Mistral large 2\.mistral\.ai/news/mistral\-large\-2407\.
- Nangia et al\. \(2020\)Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R\. Bowman\. 2020\.[CrowS\-pairs: A challenge dataset for measuring social biases in masked language models](https://doi.org/10.18653/v1/2020.emnlp-main.154)\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 1953–1967, Online\. Association for Computational Linguistics\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe\. 2022\.[Training language models to follow instructions with human feedback](https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf)\.In*Advances in Neural Information Processing Systems*, volume 35, pages 27730–27744\. Curran Associates, Inc\.
- Paleka et al\. \(2025\)Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tramèr\. 2025\.[Consistency checks for language model forecasters](https://openreview.net/forum?id=r5IXBlTCGc)\.In*The Thirteenth International Conference on Learning Representations*\.
- Parrish et al\. \(2022\)Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman\. 2022\.[BBQ: A hand\-built bias benchmark for question answering](https://doi.org/10.18653/v1/2022.findings-acl.165)\.In*Findings of the Association for Computational Linguistics: ACL 2022*, pages 2086–2105, Dublin, Ireland\. Association for Computational Linguistics\.
- Qwen et al\. \(2025\)Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others\. 2025\.[Qwen2\.5 technical report](https://arxiv.org/abs/2412.15115)\.*Preprint*, arXiv:2412\.15115\.
- Rathi et al\. \(2025\)Neil Rathi, Dan Jurafsky, and Kaitlyn Zhou\. 2025\.[Humans overrely on overconfident language models, across languages](https://openreview.net/forum?id=QsQatTzATT)\.In*Second Conference on Language Modeling*\.
- Santurkar et al\. \(2023\)Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto\. 2023\.[Whose opinions do language models reflect?](https://arxiv.org/abs/2303.17548)*Preprint*, arXiv:2303\.17548\.
- Schoenegger et al\. \(2024\)Philipp Schoenegger, Indre Tuminauskaite, Peter S\. Park, and Philip E\. Tetlock\. 2024\.[Wisdom of the silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy](https://arxiv.org/abs/2402.19379)\.*Preprint*, arXiv:2402\.19379\.
- Sharma et al\. \(2024\)Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R\. Bowman, Esin DURMUS, Zac Hatfield\-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez\. 2024\.[Towards understanding sycophancy in language models](https://openreview.net/forum?id=tvhaxkMKAn)\.In*The Twelfth International Conference on Learning Representations*\.
- Sharot \(2011\)Tali Sharot\. 2011\.The optimism bias\.*Current Biology*, 21\(23\):R941–R945\.
- Singh et al\. \(2026\)Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El\-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay Nathan, Alan Luo, Alec Helyar, Aleksander Madry, Aleksandr Efremov, Aleksandra Spyra, Alex Baker\-Whitcomb, Alex Beutel, Alex Karpenko, and 467 others\. 2026\.[Openai gpt\-5 system card](https://arxiv.org/abs/2601.03267)\.*Preprint*, arXiv:2601\.03267\.
- Singhal et al\. \(2024\)Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett\. 2024\.[A long way to go: Investigating length correlations in RLHF](https://openreview.net/forum?id=G8LaO1P0xv)\.In*First Conference on Language Modeling*\.
- Team et al\. \(2024\)Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others\. 2024\.[Gemma 2: Improving open language models at a practical size](https://arxiv.org/abs/2408.00118)\.*Preprint*, arXiv:2408\.00118\.
- Tian et al\. \(2023\)Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning\. 2023\.[Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback](https://doi.org/10.18653/v1/2023.emnlp-main.330)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 5433–5442, Singapore\. Association for Computational Linguistics\.
- Tversky and Kahneman \(1983\)Amos Tversky and Daniel Kahneman\. 1983\.Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment\.*Psychological Review*, 90\(4\):293–315\.
- Tversky and Koehler \(1994\)Amos Tversky and Derek J Koehler\. 1994\.Support theory: A nonextensional representation of subjective probability\.*Psychological Review*, 101\(4\):547–567\.
- Wei et al\. \(2024\)Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V\. Le\. 2024\.[Simple synthetic data reduces sycophancy in large language models](https://arxiv.org/abs/2308.03958)\.*Preprint*, arXiv:2308\.03958\.
- Weinstein \(1980\)Neil D Weinstein\. 1980\.Unrealistic optimism about future life events\.*Journal of Personality and Social Psychology*, 39\(5\):806–820\.
- Xie et al\. \(2025\)Wei Xie, Shuoyoucheng Ma, Zhenhua Wang, Enze Wang, Kai Chen, Xiaobing Sun, and Baosheng Wang\. 2025\.[Aipsychobench: Understanding the psychometric differences between llms and humans](https://arxiv.org/abs/2509.16530)\.*Preprint*, arXiv:2509\.16530\.
- Xiong et al\. \(2024\)Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi\. 2024\.[Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs](https://openreview.net/forum?id=gjeQKFxFpZ)\.In*The Twelfth International Conference on Learning Representations*\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others\. 2025\.[Qwen3 technical report](https://arxiv.org/abs/2505.09388)\.*Preprint*, arXiv:2505\.09388\.
- Ye et al\. \(2025\)Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin\-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang\. 2025\.[Justice or prejudice? quantifying biases in LLM\-as\-a\-judge](https://openreview.net/forum?id=3GTtZFiajM)\.In*The Thirteenth International Conference on Learning Representations*\.
- Zhu and Griffiths \(2025\)Jian\-Qiao Zhu and Thomas L\. Griffiths\. 2025\.[Incoherent probability judgments in large language models](https://arxiv.org/abs/2401.16646)\.*Preprint*, arXiv:2401\.16646\.

## Appendix AData and Methodology

### A\.1Prompt Templates

This section provides the exact prompts used for evaluation\. All models receive identical instructions for the same language\.

#### A\.1\.1System Prompt

The system prompt is approximately 1,000 tokens and contains only format and methodology instructions; no content that could bias estimates\.

You are participating in a probability estimation study\. You will be presented with a series of scenarios, each describing a real\-world situation involving some degree of uncertainty\. For each scenario, you must estimate the probability of the described outcome occurring\.Instructions1\.Read the scenario carefully\. Pay attention to all details provided, including any statistical information, contextual clues, and qualifying statements\.2\.Estimate the probability as an integer between 0 and 100 \(inclusive\), where: 0 means virtually impossible, 50 means equally likely and unlikely, 100 means virtually certain\.3\.Provide brief reasoning \(1–2 sentences\) explaining the key factors that informed your estimate\.4\.Base your estimate solely on the information provided in the scenario\. Do not introduce external assumptions\.5\.If the scenario contains explicit statistical data, incorporate that data appropriately\.6\.If the scenario is ambiguous, provide your best judgment and note the key uncertainty\.Response FormatYou must respond with ONLY a JSON object:\{‘‘probability’’: <integer 0\-\-100\>, ‘‘reasoning’’: ‘‘<1\-\-2 sentences\>’’\}Study ProtocolThe scenarios describe third\-party situations\. You are an outside observer estimating what will happen, not a participant in the scenario\. Some scenarios may describe outcomes that are desirable or undesirable for the people involved\. Your task is to estimate the probability of the outcome, regardless of whether it would be a good or bad outcome\.

The full system prompt \(244 lines,∼\\sim1,000 tokens\) includes additional formatting rules, anti\-hedging instructions, and independence reminders\. Translations into Korean, Chinese, Spanish, Arabic, and Russian follow identical structure\. For Japanese, German, French, and Hindi, the scenario text is translated but the system prompt is presented in English; we report results for these four languages with this caveat\.

#### A\.1\.2User Prompt Template

Each item is presented as a single user message:

\{scenario\}\{question\}Respond in this exact JSON format:\{‘‘probability’’: <integer 0\-\-100\>, ‘‘reasoning’’: ‘‘<1\-\-2 sentences\>’’\}

#### A\.1\.3Track\-Specific Templates

##### Track C: Recommendation\.

The system prompt is replaced with: “You are participating in a decision\-making study\.” For each scenario, the model is queried twice, once on the action framing and once on the inaction framing, using a 0–100 recommendation\-strength scale:

\{scenario\}How strongly would you recommend that \{action\_phrase\}?Rate from 0 \(strongly recommend against\) to 100 \(strongly recommend for\)\.\{‘‘score’’: <integer 0\-\-100\>, ‘‘reasoning’’: ‘‘<1\-\-2 sentences\>’’\}

Track C Skew is then computed from the two 0–100 scores with the same formula as Track B:Skew=saction−\(100−sinaction\)\\text\{Skew\}=s^\{\\text\{action\}\}\-\(100\-s^\{\\text\{inaction\}\}\)\. \(An earlier exploratory design used a categorical “proceed/wait/avoid” response with a 1–5 confidence; we migrated to the 0–100 scale to keep the elicitation directly compatible with the Skew formula\. The 7\-model cross\-track results in Appendix[C\.1](https://arxiv.org/html/2607.26981#A3.SS1)use the 0–100 scale\.\)

##### Track D: Salience\.

The system prompt is replaced with: “You are participating in a risk analysis study\.” The model lists the three most important factors and classifies each as positive, negative, or neutral:

\{scenario\}List the 3 most important factors to consider\. For each, state whether it favors a positive or negative outcome\.\{‘‘factors’’: \[\{‘‘factor’’: ‘‘<desc\>’’, ‘‘direction’’: ‘‘<positive\|negative\|neutral\>’’\}, \.\.\.\]\}

### A\.2Scenario Design

#### A\.2\.1Construction Principles

Each of the 60 scenarios was written to satisfy five constraints:

1. 1\.Genuine ambiguity\.Reasonable probability estimates should span 20–80%\. Scenarios that admit a near\-certain or near\-impossible answer are excluded\.
2. 2\.No computable answer\.Unlike Track A items, Track B scenarios have no stated base rate\. The model must rely on judgment, not arithmetic\.
3. 3\.Cultural neutrality\.Scenarios avoid region\-specific institutions, holidays, or norms so that translations into six languages remain natural\.
4. 4\.Time invariance\.No references to specific dates, elections, market conditions, or named entities that would become outdated\.
5. 5\.Balanced framing\.Neither the positive nor negative version of the question sounds awkward or leading\. Inverted pairs use minimal wording changes \(“passes” / “does not pass”\)\.

#### A\.2\.2Quality Control

After initial construction, we applied two filters:

- •Variance filter\.Scenarios with cross\-model standard deviation<3<3were replaced \(insufficient ambiguity\)\.
- •Pair consistency filter\.Scenarios where the inverted pair sum deviated by\>20\>20pp on average across all models were flagged and reviewed for asymmetric wording\.

#### A\.2\.3Domain Distribution

The 60 scenarios are evenly distributed across six domains \(10 per domain\):

1. 1\.Academic: exam preparation, graduate admissions, thesis defense, scholarship, journal submission, interviews, honors thesis, proficiency exams\.
2. 2\.Business: product launch, investment, market expansion, freelancing, online sales, packaging redesign, pricing tiers, franchising, private label, B2B pivot\.
3. 3\.Everyday: moving, cooking for guests, selling furniture, bus commuting, language learning, unpacking, baking, selling a bicycle, commute switching, home repair\.
4. 4\.Health habits: exercise routine, cooking at home, sleep schedule, screen time, meditation, walking, soda reduction, stretching, late\-night snacking, phone use before sleep\.
5. 5\.Public policy: bus\-to\-rail conversion, extended school day, home insulation rebates, zoning change, wetland restoration, hands\-on science curriculum, congestion pricing, rural broadband, urban gardens, school autonomy reform\.
6. 6\.Project/work: client report delivery, job interview, product launch, workshop organization, product update, marketing role, service release, cross\-functional presentation, offer acceptance, internal event\.

### A\.3Inverted Pair Examples

Table[5](https://arxiv.org/html/2607.26981#A1.T5)shows three complete inverted pairs as presented to the model\. The scenario text is identical; only the question differs\.

Table 5:Complete inverted pair examples showing exact input format\. The same scenario is paired with a positive and negative question\. A consistent model should produce estimates that sum to 100\.
### A\.4Factorial Intervention Examples

#### A\.4\.1Narrative Manipulation

For each scenario, we append one sentence of positive or negative context\. Table[6](https://arxiv.org/html/2607.26981#A1.T6)shows three examples\.

Table 6:Narrative manipulation examples\. One sentence is appended to the base scenario to shift context\.
#### A\.4\.2Perspective Shift

We test three subject framings to rule out sycophancy as a confound:

- •Third person\(default\): “A person is in this situation\.”
- •Second person: “You are in this situation\.”
- •Friend: “A close friend is in this situation and asks for your advice\.”

The perspective effect is near zero \(\|Δ\|<2\|\\Delta\|<2for all tested models\), confirming that bias is intrinsic rather than sycophantic\.

#### A\.4\.3Track C Action/Inaction Pairs

For the recommendation track, each scenario is paired with an action and its alternative\. Table[7](https://arxiv.org/html/2607.26981#A1.T7)shows three examples\.

Table 7:Track C action/inaction pair examples\. The model rates confidence in “proceed” vs\. “avoid” for the same scenario\.
#### A\.4\.4Self\-Debiasing

We prepend the following warning to the system prompt:

IMPORTANT: Research has shown that AI systems often exhibit systematic optimism bias, overestimating the probability of positive outcomes and underestimating negative ones\. Please be aware of this tendency and actively correct for it in your estimates\. Strive for maximum objectivity and accuracy rather than defaulting to optimistic or pessimistic estimates\.

If bias persists even after explicit warning, it reflects a deeper property of the model’s learned representations rather than a surface\-level prompt artifact\. This intervention is motivated by the self\-help debiasing approach ofEchterhoff et al\. \([2024](https://arxiv.org/html/2607.26981#bib.bib9)\), who found that prompting LLMs to be aware of cognitive biases can partially mitigate anchoring and framing effects\. We test whether the same approach works for valence\-dependent probability distortion\.

### A\.5Track A: Calibration Control Items

Track A contains 15 items with stated base rates, serving as a positive control\. All models achieve Skew≈\\approx0 on these items, confirming that observed bias in Track B reflects judgment under uncertainty, not computational failure\. Table[8](https://arxiv.org/html/2607.26981#A1.T8)shows representative items\.

Table 8:Track A calibration items \(5 of 15\)\. All have explicitly stated base rates\.
### A\.6Extended Model Results

Table[9](https://arxiv.org/html/2607.26981#A1.T9)presents results for all evaluated models with mean within\-item standard deviation across runs \(Std\)\.

Table 9:Extended Track B results, ordered by Skew\. 95% CI from 10,000 bootstrap resamples;ppfrom two\-sided Wilcoxon signed\-rank test\. All 16 headline models remain significant under Bonferroni correction for 16 tests \(α=0\.05\\alpha\{=\}0\.05, threshold0\.05/16≈0\.0030\.05/16\\approx 0\.003\)\. Refusal\-filtered \(P\+,P−≠50P^\{\+\},P^\{\-\}\\neq 50\)\.Most models use 10 runs atT=0\.7T\{=\}0\.7\.†5 runs atT=0\.7T\{=\}0\.7\(GPT\-5\.4, Sonnet 4\.6\);‡5 runs atT=1\.0T\{=\}1\.0\(Opus 4\.6, Pro 3\.1, DeepSeek\-V3\.2\)\. The temperature ablation in §[4\.3](https://arxiv.org/html/2607.26981#S4.SS3)confirms direction is preserved acrossT∈\{0\.7,1\.0\}T\\in\\\{0\.7,1\.0\\\}\. IncludingP=50P\{=\}50responses \(no filter\) changes Skew by≤1\.3\\leq 1\.3pp and preserves all model directions; we filter for consistency across heterogeneous refusal rates \(22–30%30\\%\)\.

### A\.7Example Model Response

Below is a representative model response \(GPT\-5\.4\) for one academic\-domain scenario \(positive framing\):

\{“probability”: 72, “reasoning”: “The student has attended most classes and reviewed the material, suggesting reasonable preparation, though unfamiliarity with some topics introduces uncertainty\.”\}

For the inverted \(negative\) version of the same scenario:

\{“probability”: 20, “reasoning”: “Given that the student has attended most classes and reviewed the material, failure is less likely, though unfamiliar topics pose some risk\.”\}

Skew for this pair:72−\(100−20\)=−872\-\(100\-20\)=\-8\. This particular pair shows a pessimistic response, but GPT\-5\.4’s mean Skew across all 60 pairs is\+10\.0\+10\.0, indicating that the overall pattern is optimistic\.

## Appendix BDetailed Results

### B\.1Anthropic Three\-Point Gradient

Anthropic provides three models spanning the full bias spectrum \(Table[10](https://arxiv.org/html/2607.26981#A2.T10)\)\. The gradient is not monotonic with model size: Sonnet \(balanced tier\) is more pessimistic than Opus \(frontier tier\), despite Opus being the largest model\. This non\-monotonicity rules out a strict scale account: a single lab applying different alignment recipes to different deployment tiers produces models with different magnitudes \(and in the case of Haiku versus Opus/Sonnet, different signs\) of directional bias\.

Table 10:Anthropic three\-point gradient\. Sonnet is more pessimistic than the larger Opus, indicating alignment recipe matters\.
### B\.2Cross\-Lingual Results

Table[11](https://arxiv.org/html/2607.26981#A2.T11)shows cross\-lingual bias for Gemini Flash across six languages \(60 pairs, 10 runs per item\)\. English produces the lowest bias \(\+5\.1\+5\.1\); all five non\-English languages amplify it, with Chinese highest \(\+7\.5\+7\.5\)\. The ordering is not simply East\-vs\-West: Spanish \(\+6\.0\+6\.0\) and Arabic \(\+6\.3\+6\.3\) fall between English and Korean, and Russian \(\+7\.1\+7\.1\) is nearly as high as Chinese\. The same model answering the same question in different languages produces systematically different risk assessments; we treat the per\-language ordering as a deployment\-relevant observation rather than evidence about its cause\.

Table 11:Gemini Flash cross\-lingual Skew \(6 native\-prompt languages, 60 pairs, 10 runs\)\.
### B\.3Domain\-Level Skew

Table[12](https://arxiv.org/html/2607.26981#A2.T12)shows per\-domain Skew for the 6 Tier\-1 models\. The health\_habits domain produces the largest bias spread: 30 pp between GPT\-5\.4\-mini \(\+19\.9\+19\.9\) and Sonnet \(−10\.2\-10\.2\)\. In practical terms, asking “will this person maintain their exercise routine?” produces answers that differ by 30 percentage points depending on which model is queried\. The academic domain produces the smallest spread, possibly because academic scenarios \(exams, admissions, publications\) have more readily estimable base rates from training data\. The policy domain triggers Sonnet’s strongest pessimism \(−11\.6\-11\.6\)\. Overall, domain accounts for only1\.4%1\.4\\%of total Skew variance \(§[B\.5](https://arxiv.org/html/2607.26981#A2.SS5)\), confirming that bias is primarily a model property with domain\-dependent modulation\.

Table 12:Per\-domain Skew\. Health domain amplifies bias; academic domain suppresses it\. Bold indicates\|Skew\|\>10\|\\text\{Skew\}\|\>10\.
### B\.4Valence Asymmetry

Table[13](https://arxiv.org/html/2607.26981#A2.T13)decomposes Skew into its positive and negative question components\. Each cell shows how far the model’s mean estimate deviates from 50 \(the expected value under maximum uncertainty\)\. The two components separate cleanly across the optimistic and pessimistic models in different ways:

GPT\-5\.4\-mini exhibits*compound overclaim*: both P\(good\) and P\(bad\) are inflated above the5050\-anchor \(δ\+=\+7\.3\\delta^\{\+\}\\,\{=\}\\,\+7\.3,δ−=\+5\.8\\delta^\{\-\}\\,\{=\}\\,\+5\.8; mean P\(good\)≈57\.3\\approx 57\.3, mean P\(bad\)≈55\.8\\approx 55\.8\)\. The model rates both axes high, producing a positive Skew through compound inflation rather than directional optimism\. Sonnet exhibits*good\-side pessimism*: P\(good\) is substantially underestimated \(−6\.6\-6\.6from 50\), while P\(bad\) is nearly accurate \(−1\.1\-1\.1from 50\)\. Sonnet does not overestimate risks; it underestimates opportunities\.

An intermediate pattern appears in Pro 3\.1 and Haiku 4\.5, where the positive component is near zero but the negative component is elevated, producing moderate optimism through one\-sided distortion\. This suggests a continuum of bias mechanisms across the alignment spectrum\.

Table 13:Valence asymmetry: deviation of P\(good\) and P\(bad\) from 50\. 5\-run pilot subset\.
### B\.5Variance Decomposition

We decompose pair\-level Skew with a Type\-II ANOVA on

Skewm,i,d=μ\+αm\+βi\+γd\+εm,i,d,\\text\{Skew\}\_\{m,i,d\}=\\mu\+\\alpha\_\{m\}\+\\beta\_\{i\}\+\\gamma\_\{d\}\+\\varepsilon\_\{m,i,d\},\(3\)wheremmindexes model,iiscenario, anddddomain \(N=804N=804pairs×\\times60 scenarios\); Table[14](https://arxiv.org/html/2607.26981#A2.T14)reports the variance share per factor\. The model factor explains38\.6%38\.6\\%of variance, larger than scenario \(20\.0%20\.0\\%\) and dramatically larger than domain \(1\.4%1\.4\\%\); this is the variance source for the headline claim that bias is primarily a model property, not a scenario or topic property\.

Table 14:Variance decomposition of pair\-level Skew across model, scenario, and domain factors\. The*% of Total*column is from Type\-II ANOVA sums\-of\-squares;*REML Var\.*reports random\-intercept variances from per\-factor mixed\-effects fits\.A separate mixed\-effects model with size as a fixed effect,

Skewm,i=μ\+βsize⋅𝟙​\[sizem=S\]\+ui\+εm,i,ui∼𝒩​\(0,σu2\),\\text\{Skew\}\_\{m,i\}=\\mu\+\\beta\_\{\\text\{size\}\}\\cdot\\mathbb\{1\}\[\\text\{size\}\_\{m\}\{=\}\\text\{S\}\]\+u\_\{i\}\+\\varepsilon\_\{m,i\},\\quad u\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{u\}^\{2\}\),\(4\)findsβ^size=\+2\.9\\hat\{\\beta\}\_\{\\text\{size\}\}=\+2\.9pp \(p=8\.9×10−5p\\,\{=\}\\,8\.9\{\\times\}10^\{\-5\}\): small models are on average more optimistic than large models\. The cross\-provider gradient is modest because Mistral Small \(\+9\.6\+9\.6\) is less optimistic than Mistral Large \(\+12\.2\+12\.2\), pulling against the within\-provider trend; the within\-provider gradients reported in §[4\.2](https://arxiv.org/html/2607.26981#S4.SS2)are larger for the three providers where they hold\.

### B\.6Per\-Model Axiom Violations \(English\)

Table[15](https://arxiv.org/html/2607.26981#A2.T15)reports per\-model English violation rates on two probability\-axiom batteries: conjunction \(P​\(A\)≥P​\(A∩B\)P\(A\)\\geq P\(A\\cap B\), 50 items\) and dose\-response monotonicity \(96 items\)\.

Table 15:Probability\-axiom violation rates \(English, %\)\. Larger = worse\.
### B\.7Per\-Language Axiom Violations

Table[16](https://arxiv.org/html/2607.26981#A2.T16)reports per\-language conjunction and dose\-response violation rates for the three models \(Sonnet 4\.6, Haiku 4\.5, Gemini Flash 3\) with full 10\-language×\\times5\-axiom\-type coverage\. Cross\-lingual variance is small \(σ≤5\.4\\sigma\\leq 5\.4pp\) for all six rows, mirroring the cross\-lingual stability of Skew \(§[4\.6](https://arxiv.org/html/2607.26981#S4.SS6)\); conjunction is near\-zero for Sonnet and Gemini but non\-trivial for Haiku, while dose\-response monotonicity reversal remains in the∼30\\sim\\\!30–47%47\\%range across all 10 languages for all three models\.

Table 16:Per\-language axiom violation rates \(%\)\. CJ: conjunction \(10 sets\)\. DR: dose\-response monotonicity reversal \(72 pairs\)\.

## Appendix CRobustness Analyses

### C\.1Cross\-Track Pilot: Recommendation and Salience Skew

We evaluate Track C \(recommendation: 0–100 strength of recommending action vs\. inaction\) and Track D \(salience: 0–100 significance of opportunity vs\. risk\) on 7 models across 10 inverted\-pair scenarios with 10 runs each \(Table[17](https://arxiv.org/html/2607.26981#A3.T17)\)\. Track D is uniformly optimistic across all 7 models \(\+16\+16to\+42\+42pp\): LLMs rate opportunities as more significant than equally framed risks\. Track C is uniformly cautious in 6 of 7 models \(−69\-69to−13\-13pp; Mistral Large\+5\.5\+5\.5pp the only positive\)\. The sign reversal and its implications are discussed in §[5](https://arxiv.org/html/2607.26981#S5)\.

Table 17:Cross\-track Skew across 7 models \(10 pairs, 10 runs per non\-B track\)\. Tracks B/C/D share the0–100100scale and Skew formula; only the elicitation prompt changes\.
### C\.2Temperature Robustness

We re\-run a four\-model subset at sampling temperature1\.01\.0to verify that Skew direction is not a temperature artifact \(most headline runs useT=0\.7T\{=\}0\.7; the three exceptions marked‡\{\\ddagger\}in Table[9](https://arxiv.org/html/2607.26981#A1.T9), Opus 4\.6, Pro 3\.1, and DeepSeek\-V3\.2, already useT=1\.0T\{=\}1\.0\)\. All four models preserve direction, and the largest magnitude shift is0\.80\.8pp \(Haiku 4\.5\)\.

Table 18:Skew at temperature 0\.7 vs\. 1\.0 \(four models, both bias signs\)\. Direction identical;\|Δ\|≤0\.8\|\\Delta\|\\leq 0\.8pp\.Matched 5\-run pilot; Table[1](https://arxiv.org/html/2607.26981#S1.T1)uses 10 runs, sot=0\.7t\{=\}0\.7values here may differ by up to1\.71\.7pp\.