Directional Hallucinations: Ideological Drift in News-Grounded LLM Question Answering

# Directional Hallucinations: Ideological Drift in News-Grounded LLM Question Answering
Source: [https://arxiv.org/html/2607.20487](https://arxiv.org/html/2607.20487)
Liam CunninghamTom YishayJieying Chen Vrije Universiteit Amsterdam chendi\.wang@vu\.nl, l\.e\.cunningham@student\.vu\.nl, t\.yishay3@student\.vu\.nl, j\.y\.chen@vu\.nl

###### Abstract

Large language models \(LLMs\) are increasingly used to answer questions about political information, including in election\-adjacent information settings where factual errors and ideological distortions are high\-stakes\. We present a reproducible measurement framework that treats hallucinations, unsupported statements in document\-grounded QA, as diagnostic signals of ideological drift\. Using 21,727 expert\-labeled U\.S\. political news articles from QBias spanning left, center, and right sources, we \(i\) generate an article\-specific question, \(ii\) elicit document\-grounded answers from three open\-weight LLMs and one proprietary model, \(iii\) detect sentence\-level hallucinations via reference\-based comparison, \(iv\) classify the ideological valence of hallucinated sentences with a fine\-tuned stance classifier, and \(v\) probe output logits to relate token\-level uncertainty to hallucination and drift\. Hallucination rates vary substantially across models and concentrate in contentious topics, while source\-ideology differences in hallucination frequency are modest\. In contrast, hallucination content exhibits robust leftward drift: a majority of hallucinated sentences are classified as left\-leaning, including among hallucinations generated from right\-leaning sources\. Logit\-level analysis shows hallucinations arise in high\-entropy generation contexts, and in some models uncertainty also predicts leftward drift, consistent with an “uncertainty→\\rightarrowguessing” mechanism\. We discuss implications for auditing AI\-mediated political information and for designing safeguards in election\-relevant deployments\.

## 1Societal Problem and Real\-World Impact

Societal problem\.As LLMs are integrated into search, chat assistants, and news interfaces, they increasingly mediate political information access\. In civic and electoral contexts, hallucinated or ideologically skewed answers can misinform voters, distort issue salience, and erode trust in institutions and platformsWeidingeret al\.\([2021](https://arxiv.org/html/2607.20487#bib.bib103)\); Benderet al\.\([2021](https://arxiv.org/html/2607.20487#bib.bib104)\); Allcott and Gentzkow \([2017](https://arxiv.org/html/2607.20487#bib.bib57)\); Vosoughiet al\.\([2018](https://arxiv.org/html/2607.20487#bib.bib58)\)\. Recent evidence shows conversational AI can be politically persuasive in controlled settingsHackenburget al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib65)\); Linet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib62)\); Hackenburg and Margetts \([2024](https://arxiv.org/html/2607.20487#bib.bib69)\), raising concerns about downstream effects when AI systems generate incorrect or ideologically slanted claims\.

Real\-world impact\.Our output is an auditable, model\-agnostic evaluation workflow that can be used by platforms and public institutions to stress\-test LLM deployments for civic information\. The pipeline produces: \(i\) hallucination prevalence by source ideology and topic, \(ii\) directional drift metrics with uncertainty intervals, and \(iii\) uncertainty\-linked risk indicators usable for abstention or warning policies \(e\.g\., “do not answer when uncertainty exceeds threshold”\)Kalaiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib105)\)\.

## 2Introduction

LLMs are increasingly deployed as interfaces to political and civic information, often in*grounded*settings where a system is asked to answer a question using a provided document \(e\.g\., a news article\) or retrieved sources\. Even in such settings, models can produce hallucinations, i\.e\. unsupported statements that appear confident and coherentHuanget al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib64)\); Jiet al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib92)\); Kalaiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib105)\)\. Separately, a growing literature documents political bias and value misalignment in LLM outputs, frequently showing systematic skews that vary by topic and evaluation designRettenbergeret al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib27)\); Banget al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib86)\); Chenet al\.\([2026](https://arxiv.org/html/2607.20487#bib.bib89)\); Navigliet al\.\([2023](https://arxiv.org/html/2607.20487#bib.bib91)\); Motokiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib23)\); Gallegoset al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib87)\)\.

Core question\.What happens when these two failure modes intersect? When an LLM loses grounding and begins to “fill in” missing information, is the resulting hallucinated content ideologically*directional*? If so, that would imply hallucinations are not merely reliability failures, but also*bias\-revealing events*that surface latent priors\.

Why this matters for social good\.In civic contexts \(e\.g\., election information access\), an LLM that hallucinates could introduce ideologically slanted misinformation even when the underlying source is ideologically opposed or neutral, undermining fairness, trust, and informed decision\-makingWeidingeret al\.\([2021](https://arxiv.org/html/2607.20487#bib.bib103)\); Benderet al\.\([2021](https://arxiv.org/html/2607.20487#bib.bib104)\); Allcott and Gentzkow \([2017](https://arxiv.org/html/2607.20487#bib.bib57)\)\.

Contributions\.Our contributions are fourfold:

- •Measurement framework:a transparent, reproducible pipeline for auditing*ideological directionality*of hallucinated content in news\-grounded QA\.
- •Empirical finding:hallucination rates vary by model and are modestly higher for left sources, but hallucination content shows robust leftward drift even from right sources\.
- •Topical risk profiling:drift and hallucination vulnerability are amplified on contested topics \(including election\-relevant issues\)\.
- •Mechanistic evidence:logit\-level uncertainty signals predict hallucination events; uncertainty analyses help distinguish “guessing” from source\-conditioned generation, and clarify whether drift reflects uncertainty or priors\.

## 3Related Work

#### Political bias in language models\.

Multiple approaches quantify political bias in LLMs, including ideology tests, policy questionnaires, and comparisons to human polling distributionsBanget al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib86)\); Santurkaret al\.\([2023](https://arxiv.org/html/2607.20487#bib.bib31)\); Motokiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib23)\); Rozado \([2025](https://arxiv.org/html/2607.20487#bib.bib29)\)\. Bias is not monolithic: it varies by topic and evaluation protocolMotokiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib23)\); Navigliet al\.\([2023](https://arxiv.org/html/2607.20487#bib.bib91)\); Gallegoset al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib87)\)\. These findings motivate bias auditing in*realistic tasks*rather than only prompt\-based questionnaires\.

#### Hallucinations and grounded generation\.

Hallucinations in QA and summarization are widely documentedHuanget al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib64)\); Jiet al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib92)\); Zhanget al\.\([2023](https://arxiv.org/html/2607.20487#bib.bib38)\)\. Recent work argues hallucinations arise partly because training and evaluation reward guessing over abstentionKalaiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib105)\)\. Detection methods include human evaluation, entailment\-style scoring, and learned annotatorsGuet al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib93)\)\. Our work uses a reference\-based hallucination detector to identify unsupported sentences in grounded QA outputs\.

#### Uncertainty and abstention\.

Confidence estimates in neural models are often miscalibratedGuoet al\.\([2017](https://arxiv.org/html/2607.20487#bib.bib106)\)\. Selective prediction and abstention policies can mitigate harm when uncertainty is highGeifman and El\-Yaniv \([2017](https://arxiv.org/html/2607.20487#bib.bib107)\)\. In LLM contexts, logit\-derived metrics \(entropy, margins\) provide interpretable uncertainty signals that can be connected to hallucination riskKalaiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib105)\); Jianget al\.\([2021](https://arxiv.org/html/2607.20487#bib.bib108)\)\.

#### Democracy, elections, and conversational AI\.

AI systems can influence political attitudes and persuasion dynamicsHackenburget al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib65)\); Linet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib62)\); Baiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib66)\)\. These results heighten the importance of auditing not only overt persuasion, but also*misinformation pathways*produced through hallucinated civic and political information\. This question carries significant societal implications\. If hallucinations systematically inject political bias into news\-grounded QA, users assuming AI neutrality may unknowingly consume ideologically skewed misinformation\. Such content could reinforce echo chambersCinelliet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib9)\), distort public perception, and erode trust in both AI systems and the information they convey, particularly concerning given LLMs’ growing role in search and content moderationNoelset al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib41)\)\. The problem is compounded by the fact that hallucinated content often appears fluent and confident, making it difficult for users to distinguish fabricated claims from accurately summarized information\.

## 4Data and Task

#### Dataset\.

We use the QBias corpusHaak and Schaer \([2023](https://arxiv.org/html/2607.20487#bib.bib14)\), which contains 21,747 U\.S\. political news articles collected from AllSides balanced news headline roundups \(June 2012–November 2022\) and labeled by expert annotators as*left*,*center*, or*right*\. After cleaning \(duplicate removal, missing text\), our final set containsN=21727N=\\text\{21727\}articles with counts: left=10261, right=7214, center=4252\. We retain topical tags provided by QBias \(e\.g\., Elections, Immigration, Gun Control\), enabling topic\-level risk analysis\. This imbalance reflects the underlying distribution of partisan sources and is accounted for through normalization in our analysis\. The temporal scope \(2012–2022\) captures a particularly relevant period for studying political discourse, encompassing three presidential administrations, multiple election cycles, and the emergence of “fake news” as a prominent concern\.

#### Task: news\-grounded question answering\.

We evaluate*grounded QA*rather than generic summarization to reduce degrees of freedom: each model must answer a question using only the provided article as evidence\. For each article:

1. 1\.Question generation:generate a single neutral question capturing the core claim of the article\.
2. 2\.Grounded answering:provide a short answer in sentence form using only the article content\.

This yields a controlled setting where sentence\-level hallucination detection is well\-defined\.

## 5Methodology

Figure[1](https://arxiv.org/html/2607.20487#S5.F1)illustrates the end\-to\-end pipeline of our methodology, from question generation to hallucination analysis and ideological assessment\.

![Refer to caption](https://arxiv.org/html/2607.20487v1/x1.png)Figure 1:Pipeline overview: \(1\) question generation, \(2\) news\-grounded answer generation, \(3\) sentence\-level hallucination detection, \(4\) ideology classification of hallucinated sentences \(binary left/right\), and \(5\) logit\-based uncertainty probing\.### 5\.1Models and generation

We evaluate three instruction\-tuned open\-source LLMs and one proprietary LLM:Llama 3 8B Instruct,Mistral 7B v0\.3 Instruct,Deepseek 7B ChatandGPT\-4o\-Mini\. For each article, we generate a single neutral question using Llama 3 and then have each of the four models answer the same question using the full article as context \(Appendix: prompts and decoding parameters\)\. Using a single shared question per article ensures comparability across models and mitigates question\-content confounds\. We segment each model answer into sentences; all hallucination and ideology analyses are conducted at the sentence level\.

### 5\.2Hallucination detection \(sentence\-level\)

We apply ANAH\-v2Guet al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib93)\)to compare each answer sentence against the corresponding source article and assigning one of three labels:ok\(supported\),unverifiable\(not supported\), orcontradictory\(conflicts with the article\)\. In all subsequent analyses, we collapseunverifiableandcontradictoryinto a single hallucination indicator\. We define a binary hallucination indicatorℍ∈\{0,1\}\\mathbb\{H\}\\in\\\{0,1\\\}at the sentence level as

ℍ=𝕀​\(ℓ∈\{unverifiable,contradictory\}\),\\mathbb\{H\}\\;=\\;\\mathbb\{I\}\\\!\\left\(\\ell\\in\\\{\\textit\{unverifiable\},\\,\\textit\{contradictory\}\\\}\\right\),where𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function andℓ\\elldenotes the ANAH\-v2 sentence label\.

### 5\.3Ideology classification of hallucinated content

To label the ideology of hallucinated content, we fine\-tune DeBERTa\-v3Heet al\.\([2021](https://arxiv.org/html/2607.20487#bib.bib40)\)on QBias article text forbinaryleft/right classification\. We adopt the binary setup because ternary \(left/center/right\) classification is substantially less reliable on QBias \(F1 = 0\.62 vs\. 0\.74\), which would introduce additional noise and reduce the robustness of drift estimation\. The exclusion of the “center” articles is therefore a methodological choice driven by classifier reliability, rather than an assumption about the underlying ideological structure\. We downsample the larger class so that the left and right training sets are balanced atNmin=min⁡\(Nleft,Nright\)N\_\{\\min\}=\\min\(N\_\{\\text\{left\}\},N\_\{\\text\{right\}\}\)articles per class \(details in Appendix[C\.2](https://arxiv.org/html/2607.20487#A3.SS2)\)\. The resulting classifier achieves test macro\-F1 = 0\.74\.

We apply this classifier to hallucinated sentences and interpret its outputs as*hallucination\-content ideology*\. Because the classifier is binary, alldirectional drift analyses \(RQ2\)are restricted to: source articles labeled left or right \(center excluded\), and hallucinated sentences classified as left or right \(binary output\)\. We nevertheless includecentersources inRQ1hallucination\-rate and topic\-risk profiling\.

### 5\.4Directional drift metrics and hypothesis tests

Lets∈\{L,R\}s\\in\\\{L,R\\\}denote the ideology of source\-article ideology \(restricted to left/right for drift analyses\) andy^∈\{L,R\}\\hat\{y\}\\in\\\{L,R\\\}denote the predicted ideology of a hallucinated sentence\.

We report the following metrics:

- •Alignment rate\.Pr⁡\(y^=s∣hallucination,s∈\{L,R\}\)\\Pr\(\\hat\{y\}=s\\mid\\text\{hallucination\},s\\in\\\{L,R\\\}\)\.
- •Leftward drift from right sources\.Pr⁡\(y^=L∣hallucination,s=R\)\\Pr\(\\hat\{y\}=L\\mid\\text\{hallucination\},s=R\)\.
- •Overall left\-share among hallucinations\.Pr⁡\(y^=L∣hallucination,s∈\{L,R\}\)\\Pr\(\\hat\{y\}=L\\mid\\text\{hallucination\},s\\in\\\{L,R\\\}\)\.

To test directionality, we conduct one\-sided binomial tests ofPr⁡\(y^=L\)\\Pr\(\\hat\{y\}=L\)against a symmetric nullp0=0\.5p\_\{0\}=0\.5and report exact 95% confidence intervals and Cohen’shheffect sizes\.

### 5\.5Logit\-level uncertainty probing \(RQ3\)

To investigate underlying generation mechanisms, we extract final\-layer token logits during decoding and compute per\-sentence uncertainty measures\. For each token with predictive distributionp​\(⋅\)p\(\\cdot\), entropy is defined as:

H​\(p\)=−∑vp​\(v\)​log⁡p​\(v\)\.H\(p\)=\-\\sum\_\{v\}p\(v\)\\log p\(v\)\.We aggregate uncertainty at the sentence level by computing the mean token entropy and include token count as a control for sentence length\. We restrict the logit\-level analysis to the three open\-weight models: for these we compute entropy over the full output distribution, whereas GPT\-4o\-Mini exposes only its top\-5 token log\-probabilities through the API, yielding entropy that is not comparable in scale; we report a separate probe of its logits in Section[6\.3](https://arxiv.org/html/2607.20487#S6.SS3)\.

T1 \(uncertainty→\\rightarrowhallucination\)\.We fit logistic regression models predicting sentence\-level hallucination from mean token entropy, while controlling for sentence length \(token count\)\. We report odds ratios, 95% confidence intervals\(CIs\), and ROC–AUC\.

T2 \(uncertainty→\\rightarrowdrift, within hallucinations\)\.Restricting to hallucinated sentences, we fit logistic regressions predicting whether hallucinated content is classified as left\-leaning \(vs right\-leaning\) from mean entropy and token count\. We estimate models separately by LLM to characterize heterogeneity in the uncertainty–drift relationship\. Table[1](https://arxiv.org/html/2607.20487#S5.T1)summarizes our experimental setup in the paper\.

Table 1:Experimental setup summary\.

## 6Results

### 6\.1RQ1: Do hallucination rates vary by source ideology and topic?

Overall hallucination rate\.To quantify model behaviour under grounding failure, we first examine the overall volume and type of hallucinations each system produces\. Understanding baseline hallucination rates is essential because it determines how much ungrounded content contributes to downstream ideological drift\.

Figure[2](https://arxiv.org/html/2607.20487#S6.F2)summarizes these distributions across models\. Substantial differences emerge across models despite similar parameter counts\. Deepseek produced the most hallucinations \(∼\\sim7,000 sentences\), nearly doubling Llama’s count \(∼\\sim3,500\), while GPT\-4o\-Mini generated the least \(∼\\sim2,000\)\. This variation suggests that hallucination propensity depends not only on parameter count but also heavily on model architecture and training procedures\.

The substantial variation in hallucination rates across models with similar parameter counts warrants further examination\. In percentage terms, hallucination rates also differ significantly across models \(χ2=2082\.20\\chi^\{2\}=2082\.20,p<0\.001p<0\.001\)\. Deepseek exhibits the highest hallucination rate \(21\.3%\), substantially exceeding Llama \(14\.1%\), Mistral \(14\.2%\) and GPT\-4o\-Mini \(7\.8%\)\. Pairwise comparisons reveal that Deepseek differs significantly from both Llama and Mistral \(p<0\.001p<0\.001\), while Llama and Mistral show identical rates \(p=0\.72p=0\.72\)\. GPT\-4o\-Mini also differs significantly from all three open\-weight models \(allp<0\.001p<0\.001\), exhibiting the lowest hallucination rate\. Llama 3’s relatively low hallucination rate within the open\-weight camp may partially reflect its use for question generation—questions it formulated may align better with its internal knowledge organization, providing a slight advantage\. However, even accounting for this potential confound, Deepseek’s hallucination rate represents a substantial increase over Llama, suggesting meaningful architectural or training differences beyond our experimental design\. Moreover, GPT\-4o\-Mini’s superior performance despite not benefiting from the same question\-generation advantage suggests that its lower hallucination rate reflects genuine differences in model capabilities rather than an artifact of the evaluation setup\. This pattern suggests that Deepseek’s elevated hallucination propensity reflects a model\-specific property, potentially related to training data composition or architecture, rather than task difficulty, since Llama and Mistral perform equivalently on the same inputs\. Notably, all three open\-weight models were instruction\-tuned, yet their hallucination profiles differ markedly\. This suggests that instruction tuning, while improving task compliance, does not uniformly address factual grounding\.

![Refer to caption](https://arxiv.org/html/2607.20487v1/x2.png)Figure 2:Hallucination counts by model\. Deepseek exhibits highest hallucination rates\.By source ideology\.A natural next question is whether models’ hallucination frequency varies by one ideological side\. After normalizing for dataset imbalance, the three open\-weight models exhibit hallucinations at substantially higher rates for left\-leaning source articles: Deepseek \(22\.5% left vs\. 19\.7% center vs\. 20\.7% right\), Llama \(15\.0% vs\. 12\.8% vs\. 13\.4%\), and Mistral \(15\.1% vs\. 12\.4% vs\. 14\.0%\)\. This statistically significant pattern \(p<0\.001p<0\.001\) is remarkably consistent across the three models, suggesting that left\-leaning content may be systematically more challenging for them to ground accurately\. In contrast, GPT\-4o\-Mini shows virtually no variation across source ideology \(7\.9% vs\. 7\.7% vs\. 7\.8%;p=0\.927p=0\.927\)\. This suggests that the ideology\-linked variation in hallucination rates is concentrated among the open\-weight models rather than being a universal property of the task\. Potential explanations include differences in linguistic complexity, topic distribution, gaps in training data representation, or model\-specific factual grounding behavior\.

![Refer to caption](https://arxiv.org/html/2607.20487v1/x3.png)Figure 3:Normalized hallucination proportions by source article political leaning\.By topic\.Beyond overall frequency, hallucination propensity may vary by topical domain\. Political news spans diverse issue areas that differ in polarity, factual stability, and narrative complexity\. To illuminate where models struggle most, we analyze hallucination rates by topic in Figure[4](https://arxiv.org/html/2607.20487#S6.F4)\.

Figure[4](https://arxiv.org/html/2607.20487#S6.F4)shows hallucination rates by topic, normalized to baseline topic distribution in the dataset\. Hallucinations cluster around contested political topics\. Across all models, the top hallucination\-prone topics include presidential politics, presidential elections, healthcare, immigration and gun control\. These topics share common characteristics: they feature rapidly evolving information, contested factual claims, and strong partisan framing in media coverage\. This suggests that training data composition plays a role—topics with abundant but conflicting information may leave models with uncertain associations, increasing hallucination likelihood\.

![Refer to caption](https://arxiv.org/html/2607.20487v1/x4.png)Figure 4:Top hallucinated topics by model \(normalized by baseline topic frequency\)\.
### 6\.2RQ2: Is hallucinated content ideologically directional?

Alignment vs\. misalignment\.After understanding where hallucinations occur, we next assess whether hallucinated content stays consistent with the ideological direction of the source article\. This alignment analysis reveals how often hallucinations reinforce versus diverge from the original narrative\. Because our stance classifier is binary for reliability, RQ2 drift analyses are restricted to source articles labeled left/right and hallucinated sentences classified as left/right\.

To quantify ideological alignment, we compare ownBERT’s predicted leaning for each hallucinated sentence with the expert\-annotated leaning of its source article\. A sentence is considered*aligned*if the labels match \(left–left or right–right\) and*misaligned*otherwise\. Figure[5](https://arxiv.org/html/2607.20487#S6.F5)reports the proportion of hallucinated sentences whose predicted political leaning matches the orientation of the source article\. Across all models, only 56–58% of hallucinations preserve the ideological direction of their source—barely above chance\. Approximately 43% of fabricated content introduces perspectives*absent*from the original article\. Differences across models are minimal \(56\.0% for LLaMA, 56\.8% for Mistral, 57\.7% for DeepSeek and 57\.0% for GPT\), indicating that while hallucination*frequency*varies substantially across models, ideological alignment remains largely invariant\.

![Refer to caption](https://arxiv.org/html/2607.20487v1/x5.png)Figure 5:Ideological alignment of hallucinations: proportion matching source orientation \(Aligned\) vs\. diverging \(Not Aligned\)\.Directional drift\.The alignment analysis reveals that roughly 43% of hallucinations diverge from their source orientation, but this alone does not indicate whether drift is random or directional\. The critical question is: in which direction do misaligned hallucinations drift? If errors were symmetric, we would expect roughly equal proportions of left/right misclassifications\. Systematic asymmetry, by contrast, would indicate an underlying ideological bias in ungrounded generation\. To measure the directionality of misaligned hallucinations, we compare the predicted ideological leaning of each hallucination against its source article\.

Figure[6](https://arxiv.org/html/2607.20487#S6.F6)shows the confusion matrices comparing source article orientation \(rows\) with predicted hallucination orientation \(columns\)\. All four models exhibit systematic leftward drift: regardless of source ideology, hallucinated content is predominantly classified as left\-leaning\. Examining the overall proportion of left\-leaning predictions among all predictions: Deepseek shows the strongest skew \(69\.8% of all hallucinations classified as left\), followed by Mistral \(67\.0%\), GPT\-4o\-Mini \(66\.9%\) and Llama \(64\.2%\)\.

\(a\)Llama ![Refer to caption](https://arxiv.org/html/2607.20487v1/x6.png)

\(b\)Mistral ![Refer to caption](https://arxiv.org/html/2607.20487v1/x7.png)

\(c\)Deepseek ![Refer to caption](https://arxiv.org/html/2607.20487v1/x8.png)

\(d\)GPT\-4o\-Mini ![Refer to caption](https://arxiv.org/html/2607.20487v1/x9.png)

Figure 6:Confusion matrices: source article orientation \(rows\) vs\. predicted hallucination orientation \(columns\)\. All models show systematic leftward drift regardless of source ideology\.Critically, this leftward skew persists even when doing news\-grounded QA of right\-leaning articles\. For right\-leaning sources, the proportion of hallucinations classified as left\-leaning is: Deepseek 65\.5%, Mistral 62\.8%, GPT\-4o\-Mini \(61\.9%\) and Llama 60\.3%\. This asymmetry confirms that hallucinations do not introduce random noise, as they systematically inject left\-leaning content regardless of source orientation\. The consistency of this pattern across four independently developed models suggests a common underlying cause, likely related to the composition of English\-language training corpora\.

Given the frequency asymmetry above, one might hypothesize that hallucination political bias simply reflects the higher volume of hallucinations from left\-leaning sources\. However, this explanation fails: even when models hallucinate on right\-leaning source articles, the hallucinated content still drifts leftward\. This demonstrates that leftward content bias is independent of source ideology, and the models inject left\-leaning content regardless of what they are reading\. Table[2](https://arxiv.org/html/2607.20487#S6.T2)presents binomial test results\.

Table 2:Leftward drift in hallucinated outputs\. Entries report the proportion of left\-leaning predictions among hallucinated responses classified as either left or right\. 95% confidence intervals are exact binomial intervals\.pp\-values are from one\-sided binomial tests against a null of equal left/right probability \(p0=0\.5p\_\{0\}=0\.5\)\. All models show significant drift \(p<0\.001p<0\.001\), with medium effect sizes \(Cohen’sh\>0\.25h\>0\.25\)\.If question generation confounded results by biasing content toward Llama’s priors, Llama should exhibit the strongest leftward drift\. Instead, Llama shows theweakestdrift \(64\.2% vs\. Deepseek’s 69\.8%;χ2=30\.51\\chi^\{2\}=30\.51,p<0\.001p<0\.001\)\. Furthermore, Llama and Mistral show statistically identical hallucination rates \(14\.1% vs\. 14\.2%,p=0\.72p=0\.72\), suggesting no unique Llama advantage\. These patterns argue against the QG confound driving our findings\.

### 6\.3RQ3: Do uncertainty patterns explain hallucination and drift?

T1: Uncertainty predicts hallucination\.Logit\-level analysis indicates that hallucinations arise in high\-uncertainty generation contexts\. Mean token entropy is substantially higher for hallucinated sentences \(M=0\.639M=0\.639\) than for correct sentences \(M=0\.323M=0\.323\), yielding a difference that is both statistically significant \(Welch’st=100\.3t=100\.3,p<0\.001p<0\.001\) and practically large \(Cohen’sd=1\.03d=1\.03\)\.

To quantify the predictive relationship, we fit a logistic regression predicting hallucination from mean entropy while controlling for response length \(number of tokens\)\. Each unit increase in mean entropy multiplies hallucination odds by 40\.4 \(95% CI: \[37\.6, 43\.3\],p<0\.001p<0\.001\)\. The model achieves ROC\-AUC = 0\.78, indicating good discriminative ability\. This supports an “uncertainty→\\rightarrowguessing” mechanism consistent withKalaiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib105)\)\.

T2: Uncertainty\-Ideology Relationship\.We also examined whether higher uncertainty predicts leftward drift within hallucinations, using logistic regression to control for response length\. Models differ markedly:

Deepseek shows a strong uncertainty\-ideology relationship: each unit increase in mean entropy multiplies the odds of left\-leaning hallucination content by 2\.32 \(95% CI: \[2\.02, 2\.67\],p<0\.001p<0\.001\)\. Leftward bias emerges specifically in high\-uncertainty contexts\. Llama shows a moderate relationship in the same direction \(OR = 1\.47, 95% CI: \[1\.14, 1\.89\],p=0\.003p=0\.003\)\. Mistral shows no significant uncertainty\-ideology relationship \(OR = 0\.71, 95% CI: \[0\.50, 1\.02\],p=0\.065p=0\.065\), indicating that Mistral’s leftward bias operates independently of model confidence, i\.e\. the bias is present regardless of uncertainty level\.

This heterogeneity suggests different bias mechanisms: Deepseek and Llama’s biases activate under uncertainty, while Mistral’s bias appears more uniformly embedded\. This indicates that while all models share leftward\-biased priors, the mechanism differs across models\.

Mechanistic Interpretation\.Our results adjudicate between two hypotheses about how leftward bias manifests\. Under H1 \(uncertainty\-gated priors\), leftward drift becomes more likely as uncertainty increases\. Under H2 \(uncertainty\-invariant priors\), drift is present regardless of uncertainty level\. Our results support H1 for Deepseek and Llama: within hallucinations, higher entropy significantly increases the odds that hallucinated content is classified as left\-leaning\. In contrast, Mistral shows no positive uncertainty–drift relationship, consistent with H2 or with mechanisms not captured by entropy alone, suggesting a more uniformly embedded prior that neither hypothesis fully explains\. We therefore interpret uncertainty as a robust trigger for hallucination, and \(for some models\) a trigger for directional drift; identifying the source of leftward priors \(e\.g\., training data composition or alignment\) remains outside the scope of these analyses\.

This heterogeneity implies that bias mitigation strategies may need to be model\-specific: uncertainty detection may help for Deepseek and Llama, while Mistral may require interventions at the training or architectural level\. These findings also explain existing observations that AI produces more errors when advocating right\-leaning positionsLinet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib62)\)generating right\-leaning content requires overcoming the model’s latent leftward prior, increasing error susceptibility\. While our results illuminate thetrigger mechanism, they do not identify thesourceof leftward priors, which likely reflects training data composition\.

Closed\-model probe\.We also applied the logit analysis to GPT\-4o\-Mini, but its API exposes only the top\-5 token log\-probabilities\. The resulting entropy is sharply compressed with max sentence\-level entropy 0\.70 nats versus 3\.3–3\.4 for the open\-weight models, and the separation between hallucinated and correct sentences nearly vanishes \(0\.04 nats vs\. 0\.33–0\.38\), leaving its uncertainty–drift estimate uninformative \(OR = 1\.58, 95% CI: \[0\.62, 4\.03\],p=0\.343p=0\.343\)\. We therefore exclude GPT\-4o\-Mini from the RQ3 estimates above\. This limitation is itself relevant to auditing: a deployed proprietary model’s uncertainty mechanism cannot be probed this way without fuller access to its output distribution\.

## 7Discussion

Directional hallucinations as structured errors\.Our results separate two phenomena that are often conflated:

- •Reliability \(how often models hallucinate\):models hallucinate more frequently on left\-leaning sources, but the difference is not large across left/center/right source material in our grounded QA setup\.
- •Ideological structure of errors \(what they hallucinate\):systematically left\-skewed hallucination content, even when the source is right\-leaning\. This finding aligns with documented left\-leaning tendencies in LLM outputsRozado \([2025](https://arxiv.org/html/2607.20487#bib.bib29)\); Linet al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib18)\); Chenet al\.\([2026](https://arxiv.org/html/2607.20487#bib.bib89)\)\.

This dissociation matters for social\-good deployments: even if overall hallucination rates are reduced, the remaining errors may still be directionally biased\.

Mechanisms: uncertainty, guessing incentives, and priors\.Logit\-level evidence supports that hallucinations concentrate in high\-uncertainty contexts, aligning with the argument that current training/evaluation regimes reward “guessing”Kalaiet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib105)\)\. Yet leftward drift persists even when uncertainty patterns differ by model, consistent with the view that hallucinations expose asymmetric priors embedded during pretraining and alignment \(e\.g\., corpus composition, RLHF preferences, or safety tuning\)Navigliet al\.\([2023](https://arxiv.org/html/2607.20487#bib.bib91)\); Gallegoset al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib87)\); Santurkaret al\.\([2023](https://arxiv.org/html/2607.20487#bib.bib31)\)\.

Election\-specific implications\.In election\-adjacent contexts, both*factual error*and*perceived partisan skew*can reduce trust, even when systems are intended to be neutral\. Our topic\-level profiling provides a practical pathway: models can be audited and constrained more aggressively on election\-relevant topics \(e\.g\., impose abstention at high uncertainty, require citations, enforce retrieval grounding\)Geifman and El\-Yaniv \([2017](https://arxiv.org/html/2607.20487#bib.bib107)\); Guoet al\.\([2017](https://arxiv.org/html/2607.20487#bib.bib106)\); Lewiset al\.\([2020](https://arxiv.org/html/2607.20487#bib.bib109)\)\.

## 8Limitations

Classifier Limitations\.Our classifier \(F1 = 0\.74\) may introduce noise, but random errors bias toward null, making our significant findings conservative\. We cannot fully rule out differential accuracy across ideologies; future work should examine classifier calibration by ideology\.

Data Contamination\.QBias articles \(2012–2022\) may appear in training corpora\. However, memorization wouldreducehallucination, not bias direction\. Consistent leftward drift across models with different training data \(including Deepseek\) suggests contamination is not the primary driver\.

Detection Validity\.ANAH\-v2 was not validated specifically on political news\. While ANAH\-v2 was validated on data containing substantial political content \(37% Politics/Military\), our U\.S\. partisan news sources may differ in style from the validation corpus\.

Question Generation Confound\.Using one model for question generation may advantage that model in answering\. If Llama\-generated questions biased content leftward, Llama should show the strongest effect\. Instead, Llama shows theweakestdrift \(64\.2%\), arguing against this confound\.

Logit access for closed models\.Our logit\-level uncertainty analysis \(RQ3\) requires access to the full output distribution, which closed APIs do not expose; as quantified by the closed\-model probe in Section[6\.3](https://arxiv.org/html/2607.20487#S6.SS3), this precludes a comparable analysis for GPT\-4o\-Mini, so RQ3 is restricted to the open\-weight models\. GPT\-4o\-Mini’s RQ1 and RQ2 results, which do not rely on logits, are unaffected\.

## 9Implications for AI\-Mediated Information Systems

These findings have broad implications for the design, deployment, and governance of AI systems that mediate political information:

For users and society: As LLMs become integrated into news consumption workflows, users may unconsciously internalize politically biased misinformation\. Even when source material is balanced or right\-leaning, hallucinations can inject left\-leaning perspectives, potentially reinforcing echo chambers and contributing to societal polarization—particularly concerning given assumptions of AI neutrality\. Users who rely on AI summaries without consulting original sources may develop systematically skewed understandings of political issues\.

For platform design: AI\-powered QA systems should implement hallucination\-aware content warnings and provide transparency about model tendencies\. Factual grounding through retrieval\-augmented generationNiuet al\.\([2024](https://arxiv.org/html/2607.20487#bib.bib24)\)may help, but our results suggest that when grounding fails, systematic biases emerge\. Platforms might consider displaying confidence indicators or providing links to source material for verification\.

For AI governance: Policymakers should consider establishing accuracy and neutrality standards for AI systems used in politically sensitive contexts\. Our methodology provides a template for auditing political bias in AI\-generated content that could be applied to new models as they are released\. Recent work has also revealed linguistic bias in governmental documentsde Swartet al\.\([2025](https://arxiv.org/html/2607.20487#bib.bib2)\), suggesting that bias can emerge in both source texts and AI\-generated content\.

For model development: Bias mitigation efforts should be topic\-aware, as our analysis reveals that contested issues amplify partisan biases\. Simply reducing overall hallucination rates may not address the systematic ideological skew in remaining hallucinations\. Developers might consider balanced training data curation or targeted fine\-tuning on politically sensitive topics\.

## Ethics Statement

This research relies exclusively on publicly available, annotated news records\. The dataset contains no personally identifiable information and does not involve private individuals\. No human participants were recruited or directly engaged in the study; consequently, informed consent was not required\.

## Acknowledgements

This work was supported by the Dutch Research Council \(NWO\) through the project PoliBiasEU: A Scalable and Multilingual Benchmark for Political Bias Detection in Large Language Models \(project no\. ZZKNS44563\) and by the Dutch Ministry of Education, Culture and Science \(OCW\) under the Starter Grant, in the scope of the project NEST – Navigating the Storm: European Political Contestation in Geopolitical Transformation, and partially supported by the Network Institute, Vrije Universiteit Amsterdam, through the Academy Assistants Program\.

## References

- H\. Allcott and M\. Gentzkow \(2017\)Social media and fake news in the 2016 election\.Journal of economic perspectives31\(2\),pp\. 211–236\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1),[§2](https://arxiv.org/html/2607.20487#S2.p3.1)\.
- H\. Bai, J\. G\. Voelkel, S\. Muldowney, J\. C\. Eichstaedt, and R\. Willer \(2025\)LLM\-generated messages can persuade humans on policy issues\.Nature Communications16\(1\),pp\. 6037\.Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px4.p1.1)\.
- Y\. Bang, D\. Chen, N\. Lee, and P\. Fung \(2024\)Measuring political bias in large language models: what is said and how it is said\.arXiv preprint arXiv:2403\.18932\.Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px1.p1.1)\.
- E\. M\. Bender, T\. Gebru, A\. McMillan\-Major, and S\. Shmitchell \(2021\)On the dangers of stochastic parrots: can language models be too big?\.InProceedings of the 2021 ACM conference on fairness, accountability, and transparency,pp\. 610–623\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1),[§2](https://arxiv.org/html/2607.20487#S2.p3.1)\.
- J\. Chen, K\. de Jong, A\. Poole, J\. Burakowski, E\. E\. Nosti, J\. Windt, and C\. Wang \(2026\)Uncovering political bias in large language models using parliamentary voting records\.arXiv preprint arXiv:2601\.08785\.Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[2nd item](https://arxiv.org/html/2607.20487#S7.I1.i2.p1.1)\.
- Cinelli, G\. D\. F\. Morales, A\. Galeazzi, W\. Quattrociocchi, and M\. Starnini \(2025\)The echo chamber effect on social media\.ResearchGate\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2023301118)Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px4.p1.1)\.
- M\. de Swart, F\. den Hengst, and J\. Chen \(2025\)Detecting linguistic bias in government documents using large language models\.InProceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025\- 2 May 2025,G\. Long, M\. Blumestein, Y\. Chang, L\. Lewin\-Eytan, Z\. H\. Huang, and E\. Yom\-Tov \(Eds\.\),pp\. 5034–5044\.External Links:[Link](https://doi.org/10.1145/3696410.3714526),[Document](https://dx.doi.org/10.1145/3696410.3714526)Cited by:[§9](https://arxiv.org/html/2607.20487#S9.p4.1)\.
- I\. O\. Gallegos, R\. A\. Rossi, J\. Barrow, M\. M\. Tanjim, S\. Kim, F\. Dernoncourt, T\. Yu, R\. Zhang, and N\. K\. Ahmed \(2024\)Bias and fairness in large language models: a survey\.Computational Linguistics50\(3\),pp\. 1097–1179\.Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2607.20487#S7.p2.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.Advances in neural information processing systems30\.Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2607.20487#S7.p3.1)\.
- Y\. Gu, Z\. Ji, W\. Zhang, C\. Lyu, D\. Lin, and K\. Chen \(2024\)Anah\-v2: scaling analytical hallucination annotation of large language models\.Advances in Neural Information Processing Systems37,pp\. 60012–60039\.Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2607.20487#S5.SS2.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2607.20487#S7.p3.1)\.
- F\. Haak and P\. Schaer \(2023\)Qbias \- A Dataset on Media Bias in Search Queries and Query Suggestions\.InProceedings of the 15th ACM Web Science Conference 2023,pp\. 239–244\.External Links:2311\.17780,[Document](https://dx.doi.org/10.1145/3578503.3583628)Cited by:[§4](https://arxiv.org/html/2607.20487#S4.SS0.SSS0.Px1.p1.1)\.
- K\. Hackenburg and H\. Margetts \(2024\)Evaluating the persuasive influence of political microtargeting with large language models\.Proceedings of the National Academy of Sciences121\(24\),pp\. e2403116121\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1)\.
- K\. Hackenburg, B\. M\. Tappin, L\. Hewitt, E\. Saunders, S\. Black, H\. Lin, C\. Fist, H\. Margetts, D\. G\. Rand, and C\. Summerfield \(2025\)The levers of political persuasion with conversational artificial intelligence\.Science390\(6777\),pp\. eaea3884\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px4.p1.1)\.
- P\. He, J\. Gao, and W\. Chen \(2021\)DeBERTaV3: improving deberta using electra\-style pre\-training with gradient\-disentangled embedding sharing\.External Links:2111\.09543Cited by:[§5\.3](https://arxiv.org/html/2607.20487#S5.SS3.p1.1)\.
- L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px2.p1.1)\.
- Z\. Ji, Y\. Gu, W\. Zhang, C\. Lyu, D\. Lin, and K\. Chen \(2024\)Anah: analytical annotation of hallucinations in large language models\.arXiv preprint arXiv:2405\.20315\.Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px2.p1.1)\.
- Z\. Jiang, J\. Araki, H\. Ding, and G\. Neubig \(2021\)How can we know when language models know? on the calibration of language models for question answering\.Transactions of the Association for Computational Linguistics9,pp\. 962–977\.Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px3.p1.1)\.
- A\. T\. Kalai, O\. Nachum, S\. S\. Vempala, and E\. Zhang \(2025\)Why language models hallucinate\.arXiv preprint arXiv:2509\.04664\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p2.1),[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px3.p1.1),[§6\.3](https://arxiv.org/html/2607.20487#S6.SS3.p2.2),[§7](https://arxiv.org/html/2607.20487#S7.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§7](https://arxiv.org/html/2607.20487#S7.p3.1)\.
- H\. Lin, G\. Czarnek, B\. Lewis, J\. P\. White, A\. J\. Berinsky, T\. Costello, G\. Pennycook, and D\. G\. Rand \(2025\)Persuading voters using human–artificial intelligence dialogues\.Nature,pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px4.p1.1),[§6\.3](https://arxiv.org/html/2607.20487#S6.SS3.p7.1)\.
- Z\. Lin, S\. Guan, W\. Zhang, H\. Zhang, Y\. Li, and H\. Zhang \(2024\)Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models\.Artificial Intelligence Review57\(9\),pp\. 243\.External Links:ISSN 1573\-7462,[Document](https://dx.doi.org/10.1007/s10462-024-10896-y)Cited by:[2nd item](https://arxiv.org/html/2607.20487#S7.I1.i2.p1.1)\.
- L\. Madaan, A\. K\. Singh, R\. Schaeffer, A\. Poulton, S\. Koyejo, P\. Stenetorp, S\. Narang, and D\. Hupkes \(2024\)Quantifying Variance in Evaluation Benchmarks\.arXiv\.External Links:2406\.10229,[Document](https://dx.doi.org/10.48550/arXiv.2406.10229)Cited by:[Appendix B](https://arxiv.org/html/2607.20487#A2.p1.1)\.
- F\. Y\. S\. Motoki, V\. Pinho Neto, and V\. Rangel \(2025\)Assessing political bias and value misalignment in generative artificial intelligence\.Journal of Economic Behavior & Organization,pp\. 106904\.External Links:ISSN 0167\-2681,[Document](https://dx.doi.org/10.1016/j.jebo.2025.106904)Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px1.p1.1)\.
- R\. Navigli, S\. Conia, and B\. Ross \(2023\)Biases in large language models: origins, inventory, and discussion\.ACM Journal of Data and Information Quality15\(2\),pp\. 1–21\.Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1),[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2607.20487#S7.p2.1)\.
- C\. Niu, Y\. Wu, J\. Zhu, S\. Xu, K\. Shum, R\. Zhong, J\. Song, and T\. Zhang \(2024\)RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval\-Augmented Language Models\.arXiv\.External Links:2401\.00396,[Document](https://dx.doi.org/10.48550/arXiv.2401.00396)Cited by:[§9](https://arxiv.org/html/2607.20487#S9.p3.1)\.
- S\. Noels, G\. Bied, M\. Buyl, A\. Rogiers, Y\. Fettach, J\. Lijffijt, and T\. De Bie \(2025\)What large language models do not talk about: an empirical study of moderation and censorship practices\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2504.03803),[Link](https://arxiv.org/abs/2504.03803)Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px4.p1.1)\.
- OpenLLM \(2025\)Open LLM Leaderboard\.Note:https://huggingface\.co/open\-llm\-leaderboardCited by:[Appendix B](https://arxiv.org/html/2607.20487#A2.p2.1)\.
- L\. Rettenberger, M\. Reischl, and M\. Schutera \(2025\)Assessing political bias in large language models\.Journal of Computational Social Science8\(2\),pp\. 42\.External Links:ISSN 2432\-2717, 2432\-2725,[Document](https://dx.doi.org/10.1007/s42001-025-00376-w)Cited by:[§2](https://arxiv.org/html/2607.20487#S2.p1.1)\.
- D\. Rozado \(2025\)Measuring Political Preferences in AI Systems: An Integrative Approach\.arXiv\.External Links:2503\.10649,[Document](https://dx.doi.org/10.48550/arXiv.2503.10649)Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2607.20487#S7.I1.i2.p1.1)\.
- S\. Santurkar, E\. Durmus, F\. Ladhak, C\. Lee, P\. Liang, and T\. Hashimoto \(2023\)Whose Opinions Do Language Models Reflect?\.arXiv\.External Links:2303\.17548,[Document](https://dx.doi.org/10.48550/arXiv.2303.17548)Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px1.p1.1),[§7](https://arxiv.org/html/2607.20487#S7.p2.1)\.
- S\. Vosoughi, D\. Roy, and S\. Aral \(2018\)The spread of true and false news online\.science359\(6380\),pp\. 1146–1151\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1)\.
- L\. Weidinger, J\. Mellor, M\. Rauh, C\. Griffin, J\. Uesato, P\. Huang, M\. Cheng, M\. Glaese, B\. Balle, A\. Kasirzadeh,et al\.\(2021\)Ethical and social risks of harm from language models\.arXiv preprint arXiv:2112\.04359\.Cited by:[§1](https://arxiv.org/html/2607.20487#S1.p1.1),[§2](https://arxiv.org/html/2607.20487#S2.p3.1)\.
- Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen, L\. Wang, A\. T\. Luu, W\. Bi, F\. Shi, and S\. Shi \(2023\)Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models\.arXiv\.External Links:2309\.01219,[Document](https://dx.doi.org/10.48550/arXiv.2309.01219)Cited by:[§3](https://arxiv.org/html/2607.20487#S3.SS0.SSS0.Px2.p1.1)\.

## Appendix

## Appendix AData example

Table 3:Example Row From the QBias Dataset
## Appendix BModels and Generation

We evaluate three instruction\-tuned open\-weight LLMs—Llama 3 8B Instruct, Mistral 7B v0\.3 Instruct, and Deepseek 7B Chat—together with one proprietary model, GPT\-4o\-Mini, included as a closed\-source reference point\. The open\-weight models were selected based on three criteria: \(1\) open accessibility enabling reproducible research, \(2\) comparable parameter counts \(7–8B\) allowing fair comparison, and \(3\) demonstrated instruction\-following capabilities\. They represent current capabilities among freely available systemsMadaanet al\.\[[2024](https://arxiv.org/html/2607.20487#bib.bib22)\]while remaining computationally feasible\. All four models are used in the behavioral analyses \(RQ1 and RQ2\); the logit\-level analysis \(RQ3\) is restricted to the three open\-weight models, since GPT\-4o\-Mini exposes only its top\-5 token log\-probabilities through the API\.

Our generation procedure follows a two\-step process\. First, we generate a focused question for each article using Llama 3, selected for its top ranking on the Open LLM LeaderboardOpenLLM \[[2025](https://arxiv.org/html/2607.20487#bib.bib25)\]\. The question\-generation prompt instructs the model to formulate a single question that captures the article’s core argument\. The exact prompt used is provided below\.

> You are an expert at distilling complex news articles into concise, insightful questions\. Your goal is to generate a \*single\*, focused question that captures the absolute core point or main argument of the provided news article\. Here are the key rules to follow: 1\. \*\*Be Singular:\*\* Generate only one question\. 2\. \*\*Be Focused:\*\* The question must directly address the central theme or most important takeaway of the article, not a minor detail\. 3\. \*\*Be Comprehensive \(of the core\):\*\* The question should encapsulate the primary subject and key action/event/implication discussed\. 4\. \*\*Avoid Personal Opinions/Bias:\*\* Do not inject your own opinions or biases into the question\. 5\. \*\*Avoid Leading Questions:\*\* The question should be neutral and not presuppose a specific answer\. 6\. \*\*Be Concise:\*\* Aim for clarity and brevity\. Avoid overly long or convoluted phrasing\. 7\. \*\*No Explanations or Context:\*\* Do not provide any introductory phrases, explanations, or additional context outside of the question itself\. Just the question\. 8\. \*\*Directly Actionable:\*\* The question should prompt a discussion or reflection on the article’s core message\. Bad Example of a question \(too broad, not specific enough to \*this\* article\): \- ”What is happening in the world?” Good Example of a question \(specific to an article about a new climate policy\): \- ”How will the proposed carbon tax impact the nation’s energy sector and consumer costs?”

Second, each of the four LLMs answers these questions using the full article as context\. The news\-grounded QA prompt instructs the models to provide factual, sentence\-level answers grounded exclusively in the source text\. This setup creates a controlled environment in which hallucinations—content not supported by the article—can be clearly detected\. The exact prompt used is provided below\.

> ”You are an expert at extracting specific information from news articles to answer questions directly and concisely\. ” ”For each question, you MUST provide a factual answer based on the accompanying article\. ” ”Provide your answer in sentence format\. Do NOT use bullet points, numbered lists, or any other list format\. ” ”DO NOT elaborate, ask clarifying questions, explain your reasoning, or list steps\. ” ”Provide ONLY the answer\.”

## Appendix CIdeological Classifier Fine\-Tuning and Validation

To ensure that the observed ideological skew in hallucinated outputs is not an artifact of the ideological classifier itself, we fine\-tune and validate an independent left–right text classifier based on a pretrained transformer model\.

### C\.1Training Data and Preprocessing

We construct a balanced binary classification dataset consisting of news articles labeled asleft\- orright\-leaning\. To eliminate class imbalance as a potential source of bias, we downsample the majority class such that the final dataset contains an equal number of left\- and right\-labeled observations \(7,226 per class\)\. The dataset is split into training \(80%\), validation \(10%\), and test \(10%\) sets using stratified sampling, preserving class balance across all splits\. The held\-out test set contains 1,446 observations \(723 per class\)\.

Each input consists of the article headline concatenated with the article body text\. Texts are tokenized using the DeBERTa\-v3 tokenizer and truncated to a maximum sequence length of 256 tokens\.

### C\.2Model Architecture and Training Procedure

We fine\-tunemicrosoft/deberta\-v3\-basefor binary sequence classification\. The classification head is randomly initialized and trained end\-to\-end\. Model training is performed for five epochs using the AdamW optimizer with a learning rate of2×10−52\\times 10^\{\-5\}and weight decay of 0\.01\. Training uses a batch size of 16 and evaluation uses a batch size of 32\. Mixed\-precision training \(FP16\) is enabled when supported by the hardware\.

To further guard against residual imbalance, we employ inverse\-frequency class weighting in the cross\-entropy loss function\. Model selection is based on macro\-averaged F1 score on the validation set, and the best\-performing checkpoint is retained for evaluation\.

### C\.3Classifier Performance

On the held\-out test set, the classifier achieves an overall accuracy of 0\.74 and a macro\-averaged F1 score of 0\.74, indicating balanced performance across ideological classes\. Figure[7](https://arxiv.org/html/2607.20487#A3.F7)reports the confusion matrix\. This implies recall of 0\.77 for left\-labeled articles and 0\.70 for right\-labeled articles\.

![Refer to caption](https://arxiv.org/html/2607.20487v1/Figures/confusionMatrix.png)Figure 7:Confusion matrix for the binary stance classifier on the held\-out test set\.Across the entire test set, the classifier predictsleft\-leaning labels in 53\.4% of cases, reflecting a mild asymmetry that is substantially smaller than the 64–70% leftward skew observed in hallucinated outputs produced by large language models\. This gap indicates that the classifier itself cannot account for the magnitude of ideological drift identified in the main analysis\.

### C\.4Implications for Bias Measurement

These results confirm that the ideological classifier is not the primary driver of the leftward skew observed in hallucinated content\. The classifier is trained on balanced data, exhibits comparable performance across ideological classes, and shows only a modest tendency toward left predictions\. Consequently, the pronounced leftward drift documented in hallucinated outputs reflects properties of the generative models rather than artifacts of downstream classification\.