99% 的准确度衡量了什么？针对广泛使用的假新闻语料库中捷径学习的可复现审计

# What Does 99% Accuracy Measure? A Reproducible Audit of Shortcut Learning in a Widely Used Fake News Corpus
Source: [https://arxiv.org/html/2609.25006](https://arxiv.org/html/2609.25006)
###### Abstract

Text classifiers trained on theISOT/Kaggle“Fake and Real News” corpus routinely report accuracy andF1F\_\{1\}above 0\.98, a level of performance that sits uneasily beside the difficulty of assessing veracity\. We ask what those numbers actually measure\. Treating a conventional TF–IDF and linear\-classifier pipeline as a measurement instrument rather than a contribution, we audit the corpus along three leakage channels and two distribution\-shift protocols, with all code, seeds, and derived numbers released\. Three findings emerge\. First, the benchmark is partly degenerate: a classifier that discards the article text entirely and observes only thesubjectmetadata field attainsF1F\_\{1\}=1\.000=1\.000, because the topic labels of the two classes do not overlap\. Second, removing all three leakage channels—metadata, a newswire source tag present in 99\.2% of real articles, and 6,251 duplicate documents that contaminate 19\.4% of a naive test split—lowersF1F\_\{1\}by only 1\.21 points \(0\.9935 to 0\.9814\), because the residual signal is diffuse editorial style rather than a small set of giveaway tokens: nine representation variants span just 0\.6F1F\_\{1\}points, 543 labelled documents already reachF1F\_\{1\}=0\.933=0\.933, and deleting the 1,000 highest\-weight unigrams still leaves 0\.926\. Third, that style signal does not transfer\. Under a topic\-disjoint protocol, average precision falls from 0\.9995 to 0\.9475 and deployedF1F\_\{1\}from 0\.9905 to 0\.8067; a prior\-matched resampling analysis attributes most of the latter collapse to class\-prior shift and threshold miscalibration \(F1F\_\{1\}=0\.9427=0\.9427\) while confirming a genuine 5\.2\-point loss of discrimination\. Temporal transfer within the corpus is by contrast nearly lossless\. A fine\-tuned DistilBERT sharpens rather than resolves the problem: it is stronger in\-distribution \(F1F\_\{1\}=0\.9993=0\.9993\) yet degrades far more under topic shift, losing 12\.9 points of average precision against the linear model’s 5\.2 and reaching prior\-matchedF1F\_\{1\}of only 0\.689, while its in\-domain validationF1F\_\{1\}of 0\.9995 gives no warning of that collapse\. The decisive test is cross\-corpus: transferred to the independently built LIAR benchmark, all three models fall to near\-chance ranking \(ROC\-AUC0\.540\.54–0\.570\.57\) and none beats a majority\-class baseline\. We conclude that within\-corpus scores on this benchmark quantify source and topic separability rather than veracity assessment, that added capacity exploits the shortcut more efficiently instead of avoiding it, and we recommend metadata\-only, small\-sample, and topic\-disjoint baselines as inexpensive diagnostics that any future study on this corpus can report\.

## IIntroduction

False news propagates measurably faster and more widely than true news on social platforms\[[60](https://arxiv.org/html/2609.25006#bib.bib10)\], and its documented reach during the 2016 United States election\[[3](https://arxiv.org/html/2609.25006#bib.bib11)\]motivated sustained interest in automated detection\. The task, however, is difficult in a way that resists purely lexical solutions: deciding whether a claim is true generally requires evidence external to the text\[[56](https://arxiv.org/html/2609.25006#bib.bib6),[65](https://arxiv.org/html/2609.25006#bib.bib8)\]\. It is therefore notable that a large body of applied work reports near\-perfect performance on the task, with accuracies above 0\.99 on public corpora\[[1](https://arxiv.org/html/2609.25006#bib.bib1),[2](https://arxiv.org/html/2609.25006#bib.bib2),[29](https://arxiv.org/html/2609.25006#bib.bib15)\]\. Either the problem is easier than its formulation suggests, or the reported numbers measure something other than veracity assessment\. This paper investigates the second possibility for one specific, heavily used corpus\.

Our object of study is the “Fake and Real News” dataset\[[4](https://arxiv.org/html/2609.25006#bib.bib3)\], a redistribution of the corpus introduced by Ahmed et al\.\[[1](https://arxiv.org/html/2609.25006#bib.bib1),[2](https://arxiv.org/html/2609.25006#bib.bib2)\]and among the most frequently used public resources for this task\[[29](https://arxiv.org/html/2609.25006#bib.bib15)\]\. It pairs 21,417 genuine articles drawn from a single newswire against 23,481 articles from outlets flagged by fact\-checking organisations\. That construction is convenient and, as we show, consequential: the two classes differ systematically in provenance and house style, not merely in truthfulness\.

The phenomenon we investigate is*shortcut learning*: a model attaining high benchmark scores by exploiting surface regularities that correlate with the label in the dataset but not in the underlying task\[[16](https://arxiv.org/html/2609.25006#bib.bib19)\]\. Shortcut learning is well documented in natural language inference, where hypothesis\-only baselines and annotation artifacts account for much of apparent model competence\[[19](https://arxiv.org/html/2609.25006#bib.bib20),[45](https://arxiv.org/html/2609.25006#bib.bib21),[32](https://arxiv.org/html/2609.25006#bib.bib22),[37](https://arxiv.org/html/2609.25006#bib.bib23)\], and in computer vision, where dataset identity is itself predictable\[[57](https://arxiv.org/html/2609.25006#bib.bib24)\]\. For fake news specifically, Bozarth and Budak\[[5](https://arxiv.org/html/2609.25006#bib.bib18)\]document how evaluation choices drive reported performance, and Schuster et al\.\[[53](https://arxiv.org/html/2609.25006#bib.bib17)\]show that stylometric cues fail when provenance and veracity are decoupled\. What has been missing, to the best of our knowledge, is a per\-channel, fully reproducible decomposition for this particular corpus, combined with a shift analysis that separates calibration effects from genuine degradation\.

We deliberately do not propose a new architecture\. A stronger model would confound the question: if the benchmark is degenerate, a higher score is evidence about the benchmark, not the model\. We instead fix a transparent, fully interpretable pipeline \(TF–IDF features with four linear classifiers\) and vary the*data*and the*evaluation protocol*, using the classifier as a probe\. Linear models make this strategy viable because every decision decomposes into per\-token weights that can be inspected and ablated directly\.

Contributions\.

1. 1\.A reproducible audit protocol \(Section[IV](https://arxiv.org/html/2609.25006#S4)\) that isolates three leakage channels in theISOT/Kagglecorpus and quantifies each independently, released as executable code with fixed seeds \(Section[VI](https://arxiv.org/html/2609.25006#S6)\)\.
2. 2\.Evidence that the benchmark is partly degenerate: a metadata\-only classifier that never sees the article text attainsF1F\_\{1\}=1\.000=1\.000\(Section[VII\-B](https://arxiv.org/html/2609.25006#S7.SS2)\)\.
3. 3\.A demonstration that the residual post\-mitigation signal is*diffuse*rather than localised, established through representation ablations, a learning curve, and a top\-KKfeature\-deletion probe \(Section[VII\-C](https://arxiv.org/html/2609.25006#S7.SS3)\)\.
4. 4\.A prior\-controlled distribution\-shift analysis \(Section[VII\-D](https://arxiv.org/html/2609.25006#S7.SS4)\) that decomposes an 18\.4\-point deployedF1F\_\{1\}collapse into a class\-prior/calibration component and a genuine 5\.2\-point loss of average precision\.
5. 5\.A capacity control \(Section[VII\-E](https://arxiv.org/html/2609.25006#S7.SS5)\) in which a fine\-tuned DistilBERT, trained on identical documents and splits, is stronger in\-distribution but loses 2\.5 times more average precision under topic shift, evidence that the limitation is a property of the data rather than of model expressiveness\.
6. 6\.A cross\-corpus test \(Section[VII\-F](https://arxiv.org/html/2609.25006#S7.SS6)\) showing that all three ISOT\-trained models transfer to the LIAR benchmark at near\-chance ranking, the decisive external check that in\-corpus scores do not reflect a transferable notion of veracity\.
7. 7\.Concrete, low\-cost diagnostics we recommend as standard practice for future work on this corpus \(Section[VIII](https://arxiv.org/html/2609.25006#S8)\)\.

## IIRelated Work

### II\-AFake News Detection and Its Benchmarks

Surveys of the area distinguish content\-based approaches from those exploiting social context and external evidence\[[54](https://arxiv.org/html/2609.25006#bib.bib7),[65](https://arxiv.org/html/2609.25006#bib.bib8),[39](https://arxiv.org/html/2609.25006#bib.bib9)\]\. Purely content\-based detection is attractive operationally, since it needs only the article, but it is theoretically limited, since textual form underdetermines factual accuracy\. Within that paradigm, Pérez\-Rosas et al\.\[[42](https://arxiv.org/html/2609.25006#bib.bib12)\]report linguistic\-feature classifiers across several domains, while Horne and Adalı\[[22](https://arxiv.org/html/2609.25006#bib.bib13)\]find that headlines alone carry substantial discriminative signal—an observation our input\-field ablation revisits in Section[VII\-C](https://arxiv.org/html/2609.25006#S7.SS3)\. Zellers et al\.\[[63](https://arxiv.org/html/2609.25006#bib.bib16)\]extend the problem to machine\-generated articles, where provenance and veracity come apart by construction\. Benchmarks reflect this tension\. LIAR\[[62](https://arxiv.org/html/2609.25006#bib.bib4)\]supplies short PolitiFact\-labelled claims with fine\-grained veracity labels; FEVER\[[56](https://arxiv.org/html/2609.25006#bib.bib6)\]pairs claims with Wikipedia evidence, making the evidence\-retrieval step explicit; NELA\-GT\[[38](https://arxiv.org/html/2609.25006#bib.bib5)\]provides large multi\-labelled article collections with source\-level annotations\. TheISOT/Kagglecorpus\[[1](https://arxiv.org/html/2609.25006#bib.bib1),[2](https://arxiv.org/html/2609.25006#bib.bib2),[4](https://arxiv.org/html/2609.25006#bib.bib3)\]differs in an important respect: its two classes were collected from disjoint sets of publishers\. Labels therefore carry provenance information, a property central to our analysis\.

Reported results on the corpus are uniformly high\. Ahmed et al\. obtained accuracies in the high nineties usingnn\-gram features with linear models\[[1](https://arxiv.org/html/2609.25006#bib.bib1),[2](https://arxiv.org/html/2609.25006#bib.bib2)\]; the benchmark study of Khan et al\.\[[29](https://arxiv.org/html/2609.25006#bib.bib15)\]finds that simple models remain competitive with neural alternatives across several fake\-news corpora, including this one, and that scores cluster near the ceiling\. Transformer\-based systems such as FakeBERT\[[26](https://arxiv.org/html/2609.25006#bib.bib14)\]report comparable figures\. Our reading of this pattern is that near\-ceiling agreement across model families of very different capacity is itself diagnostic: when a linear bag\-of\-words model and a pretrained transformer perform indistinguishably, the discriminative information is likely to be shallow\.

### II\-BShortcut Learning and Dataset Artifacts

Geirhos et al\.\[[16](https://arxiv.org/html/2609.25006#bib.bib19)\]formalise shortcut learning as reliance on decision rules that succeed on benchmark data but fail under distribution shift\. In NLI, Gururangan et al\.\[[19](https://arxiv.org/html/2609.25006#bib.bib20)\]and Poliak et al\.\[[45](https://arxiv.org/html/2609.25006#bib.bib21)\]show that hypothesis\-only models substantially outperform chance, revealing annotation artifacts; McCoy et al\.\[[32](https://arxiv.org/html/2609.25006#bib.bib22)\]and Niven and Kao\[[37](https://arxiv.org/html/2609.25006#bib.bib23)\]show that apparently competent models rely on syntactic heuristics and spurious statistical cues\. Diagnostic methodologies developed in response include contrast sets\[[15](https://arxiv.org/html/2609.25006#bib.bib29)\]and behavioural testing\[[50](https://arxiv.org/html/2609.25006#bib.bib30)\]\. Our metadata\-only and small\-sample probes are direct analogues of the hypothesis\-only baseline: cheap experiments whose success indicates that the benchmark, not the model, deserves scrutiny\.

### II\-CLeakage and Evaluation Protocol

Kaufman et al\.\[[28](https://arxiv.org/html/2609.25006#bib.bib25)\]give the canonical taxonomy of leakage; Kapoor and Narayanan\[[27](https://arxiv.org/html/2609.25006#bib.bib26)\]document its prevalence and its role in irreproducible machine\-learning claims across scientific fields\. Duplicate records spanning train and test are a recognised instance\. Gorman and Bedrick\[[17](https://arxiv.org/html/2609.25006#bib.bib31)\]show that conclusions drawn from a single standard split can reverse under resampling, motivating our use of cross\-validation and bootstrap intervals alongside a fixed split\. On protocol design specifically, Bozarth and Budak\[[5](https://arxiv.org/html/2609.25006#bib.bib18)\]show that performance rankings for fake\-news classifiers depend strongly on evaluation choices, and argue for source\-aware splits\. Schuster et al\.\[[53](https://arxiv.org/html/2609.25006#bib.bib17)\]demonstrate that stylometric detectors degrade sharply once provenance is decoupled from veracity\. Our topic\-disjoint protocol operationalises that concern for this corpus, and our prior\-matched analysis adds a component their setting does not isolate: how much of an observed collapse is attributable to class\-prior shift rather than to loss of discriminative power\.

### II\-DPositioning

Relative to prior work, this paper contributes neither a new model nor a new dataset\. Its contribution is measurement: a per\-channel quantification of leakage in a specific widely used corpus, an explicit demonstration that the surviving signal is distributed rather than concentrated, and a shift analysis that separates two mechanisms usually reported as one number\. Every figure in the paper is regenerated by a released script from the raw corpus\.

## IIIProblem Statement

Let𝒟=\{\(di,yi\)\}i=1N\\mathcal\{D\}=\\\{\(d\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}be a corpus of news documentsdi∈𝒳d\_\{i\}\\in\\mathcal\{X\}with labelsyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}, wherey=1y=1denotes*fake*\. The nominal task is to learnf:𝒳→\{0,1\}f:\\mathcal\{X\}\\rightarrow\\\{0,1\\\}minimising expected risk under the deployment distribution𝒫dep\\mathcal\{P\}\_\{\\text\{dep\}\}:

Rdep​\(f\)=𝔼\(d,y\)∼𝒫dep​\[𝕀​\[f​\(d\)≠y\]\]\.R\_\{\\text\{dep\}\}\(f\)\\;=\\;\\mathbb\{E\}\_\{\(d,y\)\\sim\\mathcal\{P\}\_\{\\text\{dep\}\}\}\\big\[\\mathbb\{I\}\[f\(d\)\\neq y\]\\big\]\.\(1\)Benchmark practice instead reports empirical risk on a held\-out split of the same corpus, i\.e\. an estimate ofRbench​\(f\)R\_\{\\text\{bench\}\}\(f\)under the corpus distribution𝒫bench\\mathcal\{P\}\_\{\\text\{bench\}\}\. The two coincide only if𝒫bench≈𝒫dep\\mathcal\{P\}\_\{\\text\{bench\}\}\\approx\\mathcal\{P\}\_\{\\text\{dep\}\}\.

We formalise the failure mode of interest as follows\. Write each document as a paird=\(s,c\)d=\(s,c\)whereccis veracity\-bearing content andssis a*provenance signature*: stylistic, formatting, and editorial regularities determined by the publishing organisation\. In𝒫bench\\mathcal\{P\}\_\{\\text\{bench\}\}the two classes were sampled from disjoint publisher sets, sossandyyare strongly dependent:

I​\(s;y\)\|𝒫bench≫I​\(s;y\)\|𝒫dep,I\(s;y\)\\Big\|\_\{\\mathcal\{P\}\_\{\\text\{bench\}\}\}\\;\\gg\\;I\(s;y\)\\Big\|\_\{\\mathcal\{P\}\_\{\\text\{dep\}\}\},\(2\)whereI​\(⋅;⋅\)I\(\\cdot;\\cdot\)denotes mutual information\. A learner minimising empirical risk has no incentive to preferccoverss; ifssis more easily extracted, the learned rule will exploit it\. Benchmark risk is then a biased estimate of deployment risk, and the bias is not removed by increasing model capacity or dataset size—only by changing the data or the protocol\.

This yields three empirical questions, which organise our experiments:

Q1*Explicit leakage\.*How much of the benchmark score is attributable to identifiable artifacts \(metadata fields, source tags, and duplicate documents\) that can be removed by preprocessing?

Q2*Residual signal structure\.*After removing those artifacts, is the remaining signal concentrated in a few tokens \(hence removable\) or diffusely distributed across editorial style \(hence not\)?

Q3*Transfer\.*Does the surviving decision rule remain valid when the topic mix or time period changes, and how much of any observed degradation reflects loss of discrimination as opposed to class\-prior shift?

Q4*Capacity\.*Is the limitation a property of the instrument or of the data? If a pretrained contextual model, which can represent veracity\-bearing content far better than a bag of words, nonetheless degrades at least as much under shift, the constraint lies in𝒫bench\\mathcal\{P\}\_\{\\text\{bench\}\}rather than in model expressiveness\.

Q5*External validity\.*Does anything learned on this corpus transfer to an independently constructed one? A model trained here and tested on a different benchmark isolates whatever veracity signal, if any, is not tied to this corpus’s provenance structure\.

## IVProposed Methodology

### IV\-ADesign Rationale

The instrument must be transparent, cheap, and deterministic, since it is applied across 20 experimental conditions\. We therefore use sparse TF–IDF features with linear classifiers\. This choice is deliberate on three grounds\. First, interpretability: a linear model assigns one scalar weight per token, so the top\-KKdeletion probe of Section[VII\-C](https://arxiv.org/html/2609.25006#S7.SS3)has a well\-defined meaning\[[49](https://arxiv.org/html/2609.25006#bib.bib55)\]\. Second, determinism: with a fixed seed and a coordinate\-descent solver the pipeline is bit\-reproducible, which matters when differences of interest are on the order of10−210^\{\-2\}\. Third, capacity control: if a low\-capacity model already saturates the benchmark, high scores cannot be attributed to sophisticated inference\. The trade\-off is accepted knowingly: no modelling of word order beyond shortnn\-grams, and no lexical generalisation\. and Section[IX](https://arxiv.org/html/2609.25006#S9)states where it constrains our conclusions\.

Distributed word representations\[[35](https://arxiv.org/html/2609.25006#bib.bib43),[41](https://arxiv.org/html/2609.25006#bib.bib44)\], subword linear models\[[25](https://arxiv.org/html/2609.25006#bib.bib41)\], character\-level convolutional networks\[[64](https://arxiv.org/html/2609.25006#bib.bib42)\], and pretrained transformers\[[58](https://arxiv.org/html/2609.25006#bib.bib45),[9](https://arxiv.org/html/2609.25006#bib.bib46)\]are all defensible alternatives, and we return to them in Section[X](https://arxiv.org/html/2609.25006#S10)\. None is preferable*as an instrument*for the present question, because each replaces inspectable per\-token weights with distributed parameters that the deletion probe of Section[VII\-C](https://arxiv.org/html/2609.25006#S7.SS3)could not target\. Sparse lexical features with linear models also remain strong baselines for topical text classification\[[24](https://arxiv.org/html/2609.25006#bib.bib34),[61](https://arxiv.org/html/2609.25006#bib.bib40)\], so little discriminative power is sacrificed\.

### IV\-BCorpus Construction and Leakage Control

Algorithm[1](https://arxiv.org/html/2609.25006#alg1)specifies corpus construction\. Three controls are applied, each independently switchable so that its contribution can be measured:

\(C1\) Metadata exclusion\.Thesubjectanddatefields are discarded\. In this corpus the real class carries only the subjectspoliticsNewsandworldnewswhile the fake class carries six entirely different values \(Table[I](https://arxiv.org/html/2609.25006#S6.T1)\); the supports are disjoint, sosubjectalone determines the label\. Thedatefields are also not exchangeable: 2,926 fake articles fall outside the date range spanned by the real class\.

\(C2\) Source\-tag removal\.Real articles are newswire copy that typically opens with a dateline of the formCITY \(Reuters\) \-\-; the agency name appears in 99\.21% of real articles and 0\.04% of fake ones\. We strip the leading dateline and every remaining occurrence of the agency name\. The transformation is applied symmetrically to both classes, so it introduces no new asymmetry\.

\(C3\) De\-duplication\.Exact duplicates of the cleaned document string are collapsed to a single instance*before*splitting\. Under a naive protocol that omits this step, 19\.37% of test documents also occur verbatim in the training set \(Section[VII\-B](https://arxiv.org/html/2609.25006#S7.SS2)\), so test performance partly measures memorisation\.

Remaining normalisation is deliberately conservative and purely regular: the title and body are concatenated, case is folded, URLs and e\-mail addresses are removed, characters outside\[a\-\-z’\]are mapped to whitespace, runs of whitespace are collapsed, and documents shorter than 20 characters are dropped\. We do not stem or lemmatise\. Stemming would add a dependency on an external resource without addressing any of Q1–Q3, and with bigrams and stop\-word removal already in place its effect on this corpus is marginal\.

Algorithm 1Leakage\-controlled corpus construction1:Raw tables

Treal,TfakeT\_\{\\text\{real\}\},T\_\{\\text\{fake\}\}; flags

c1,c2,c3\\textsc\{c1\},\\textsc\{c2\},\\textsc\{c3\}; seed

σ\\sigma
2:Disjoint splits

𝒮tr,𝒮va,𝒮te\\mathcal\{S\}\_\{\\text\{tr\}\},\\mathcal\{S\}\_\{\\text\{va\}\},\\mathcal\{S\}\_\{\\text\{te\}\}
3:

𝒟←\\mathcal\{D\}\\leftarrowconcat

\(Treal×\{0\},Tfake×\{1\}\)\(T\_\{\\text\{real\}\}\\\!\\times\\\!\\\{0\\\},\\;T\_\{\\text\{fake\}\}\\\!\\times\\\!\\\{1\\\}\)
4:for all

\(d,y\)∈𝒟\(d,y\)\\in\\mathcal\{D\}do

5:

u←title​\(d\)​‖“ ”‖​body​\(d\)u\\leftarrow\\textsc\{title\}\(d\)\\\|\\text\{\`\` ''\}\\\|\\textsc\{body\}\(d\)
6:if

¬c1\\neg\\textsc\{c1\}then⊳\\trianglerightleakage condition only

7:

u←subject​\(d\)​‖“ ”‖​uu\\leftarrow\\textsc\{subject\}\(d\)\\\|\\text\{\`\` ''\}\\\|u
8:endif

9:

u←lowercase​\(u\)u\\leftarrow\\textsc\{lowercase\}\(u\)
10:ifc2then

11:

u←stripDateline​\(u\)u\\leftarrow\\textsc\{stripDateline\}\(u\);

u←removeAgencyName​\(u\)u\\leftarrow\\textsc\{removeAgencyName\}\(u\)
12:endif

13:

u←removeUrlsEmails​\(u\)u\\leftarrow\\textsc\{removeUrlsEmails\}\(u\);

u←keepAlpha​\(u\)u\\leftarrow\\textsc\{keepAlpha\}\(u\)
14:

u←collapseSpace​\(u\)u\\leftarrow\\textsc\{collapseSpace\}\(u\)
15:endfor

16:

𝒟←\{\(u,y\)∈𝒟:\|u\|≥20\}\\mathcal\{D\}\\leftarrow\\\{\(u,y\)\\in\\mathcal\{D\}:\|u\|\\geq 20\\\}
17:ifc3then

18:

𝒟←dropDuplicates​\(𝒟,key=u\)\\mathcal\{D\}\\leftarrow\\textsc\{dropDuplicates\}\(\\mathcal\{D\},\\text\{key\}=u\)
19:endif

20:

\(𝒟′,𝒮te\)←stratifiedSplit​\(𝒟,0\.20,σ\)\(\\mathcal\{D\}^\{\\prime\},\\mathcal\{S\}\_\{\\text\{te\}\}\)\\leftarrow\\textsc\{stratifiedSplit\}\(\\mathcal\{D\},0\.20,\\sigma\)
21:

\(𝒮tr,𝒮va\)←stratifiedSplit​\(𝒟′,0\.125,σ\)\(\\mathcal\{S\}\_\{\\text\{tr\}\},\\mathcal\{S\}\_\{\\text\{va\}\}\)\\leftarrow\\textsc\{stratifiedSplit\}\(\\mathcal\{D\}^\{\\prime\},0\.125,\\sigma\)⊳\\triangleright0\.10/0\.800\.10/0\.80

22:return

𝒮tr,𝒮va,𝒮te\\mathcal\{S\}\_\{\\text\{tr\}\},\\mathcal\{S\}\_\{\\text\{va\}\},\\mathcal\{S\}\_\{\\text\{te\}\}

### IV\-CFeature Representation

Documents are mapped to sparse TF–IDF vectors\[[55](https://arxiv.org/html/2609.25006#bib.bib32),[51](https://arxiv.org/html/2609.25006#bib.bib33)\]over word unigrams and bigrams\. WithNNtraining documents anddf​\(t\)\\mathrm\{df\}\(t\)the document frequency of termtt, we use sublinear term frequency and smoothed inverse document frequency,

tf′​\(t,d\)\\displaystyle\\mathrm\{tf\}^\{\\prime\}\(t,d\)=1\+ln⁡tf​\(t,d\),\\displaystyle=1\+\\ln\\mathrm\{tf\}\(t,d\),\(3\)idf​\(t\)\\displaystyle\\mathrm\{idf\}\(t\)=ln⁡1\+N1\+df​\(t\)\+1,\\displaystyle=\\ln\\\!\\frac\{1\+N\}\{1\+\\mathrm\{df\}\(t\)\}\+1,\(4\)followed byℓ2\\ell\_\{2\}normalisation of each document vector,

xd,t=tf′​\(t,d\)​idf​\(t\)∥tf′​\(⋅,d\)⊙idf​\(⋅\)∥2\.x\_\{d,t\}\\;=\\;\\frac\{\\mathrm\{tf\}^\{\\prime\}\(t,d\)\\,\\mathrm\{idf\}\(t\)\}\{\\big\\lVert\\mathrm\{tf\}^\{\\prime\}\(\\cdot,d\)\\odot\\mathrm\{idf\}\(\\cdot\)\\big\\rVert\_\{2\}\}\.\(5\)Sublinear scaling prevents a term repeated many times in one document from dominating;ℓ2\\ell\_\{2\}normalisation removes document\-length effects, which matters here because the classes differ in mean length \(436\.7 versus 395\.8 tokens, Table[I](https://arxiv.org/html/2609.25006#S6.T1)\)\. Terms occurring in fewer than five documents or more than 90% of documents are discarded, English stop words are removed, and the vocabulary is capped at50,00050\{,\}000features by corpus frequency\.

Critically, the vectoriser is fitted on the training split only and applied unchanged to validation and test\. Fitting on the full corpus would leak document\-frequency statistics of the evaluation data into the representation—a subtle but real instance of the leakage this paper studies\.

### IV\-DClassifiers

Four linear classifiers are trained on identical features\. All four are standard; we summarise them to fix notation and to make the interpretability argument precise\. Letx∈ℝVx\\in\\mathbb\{R\}^\{V\}be a document vector,w∈ℝVw\\in\\mathbb\{R\}^\{V\}a weight vector, andbba bias\.

*Logistic regression*modelsP​\(y=1∣x\)=σ​\(w⊤​x\+b\)P\(y\{=\}1\\mid x\)=\\sigma\(w^\{\\\!\\top\}x\+b\)withσ​\(z\)=\(1\+e−z\)−1\\sigma\(z\)=\(1\+e^\{\-z\}\)^\{\-1\}, fitted byℓ2\\ell\_\{2\}\-regularised maximum likelihood\. With labelsy~∈\{−1,\+1\}\\tilde\{y\}\\in\\\{\-1,\+1\\\}the objective solved by the coordinate\-descent solver\[[14](https://arxiv.org/html/2609.25006#bib.bib61)\]is

minw,b⁡12​∥w∥22\+C​∑i=1nln⁡\(1\+e−y~i​\(w⊤​xi\+b\)\)\.\\min\_\{w,b\}\\;\\tfrac\{1\}\{2\}\\lVert w\\rVert\_\{2\}^\{2\}\+C\\sum\_\{i=1\}^\{n\}\\ln\\\!\\big\(1\+e^\{\-\\tilde\{y\}\_\{i\}\(w^\{\\\!\\top\}x\_\{i\}\+b\)\}\\big\)\.\(6\)It is our reference probe becausewwis directly interpretable and becauseσ​\(⋅\)\\sigma\(\\cdot\)yields calibrated scores\.

*Multinomial naive Bayes*\[[31](https://arxiv.org/html/2609.25006#bib.bib37)\]estimates per\-class term distributions with additive smoothingα\\alpha,

P^​\(t∣y\)=Ny,t\+αNy\+α​\|V\|,y^=arg​maxy⁡\[ln⁡P^​\(y\)\+∑txt​ln⁡P^​\(t∣y\)\],\\begin\{split\}\\hat\{P\}\(t\\mid y\)&=\\frac\{N\_\{y,t\}\+\\alpha\}\{N\_\{y\}\+\\alpha\|V\|\},\\\\\[2\.0pt\] \\hat\{y\}&=\\operatorname\*\{arg\\,max\}\_\{y\}\\Big\[\\ln\\hat\{P\}\(y\)\+\\sum\_\{t\}x\_\{t\}\\ln\\hat\{P\}\(t\\mid y\)\\Big\],\\end\{split\}\(7\)under the assumption that terms are conditionally independent given the class\. The assumption is violated by construction in text—overlapping unigrams and bigrams are strongly dependent—and we use the model as a deliberately mis\-specified reference point\[[11](https://arxiv.org/html/2609.25006#bib.bib39),[48](https://arxiv.org/html/2609.25006#bib.bib38)\]\.

*Linear support vector classification*\[[6](https://arxiv.org/html/2609.25006#bib.bib35),[24](https://arxiv.org/html/2609.25006#bib.bib34)\]maximises the margin under squared hinge loss,

minw,b12∥w∥22\+C∑i=1nmax\(0,1−y~i\(w⊤xi\+b\)\)2\.\\min\_\{w,b\}\\;\\tfrac\{1\}\{2\}\\lVert w\\rVert\_\{2\}^\{2\}\+C\\sum\_\{i=1\}^\{n\}\\max\\\!\\big\(0,\\,1\-\\tilde\{y\}\_\{i\}\(w^\{\\\!\\top\}x\_\{i\}\+b\)\\big\)^\{2\}\.\(8\)
*The passive–aggressive classifier*\[[7](https://arxiv.org/html/2609.25006#bib.bib36)\]is an online large\-margin learner\. On example\(xt,y~t\)\(x\_\{t\},\\tilde\{y\}\_\{t\}\)with hinge lossℓt=max⁡\(0,1−y~t​wt⊤​xt\)\\ell\_\{t\}=\\max\(0,1\-\\tilde\{y\}\_\{t\}\\,w\_\{t\}^\{\\\!\\top\}x\_\{t\}\), the PA\-I update is

τt=min⁡\(C,ℓt∥xt∥22\),wt\+1=wt\+τt​y~t​xt,\\tau\_\{t\}=\\min\\\!\\Big\(C,\\;\\frac\{\\ell\_\{t\}\}\{\\lVert x\_\{t\}\\rVert\_\{2\}^\{2\}\}\\Big\),\\qquad w\_\{t\+1\}=w\_\{t\}\+\\tau\_\{t\}\\,\\tilde\{y\}\_\{t\}\\,x\_\{t\},\(9\)i\.e\. the weights are left unchanged when the margin is already satisfied \(ℓt=0\\ell\_\{t\}=0\) and otherwise moved by the smallest step that corrects the current example, withCCbounding the step\. In all four modelsCCis an inverse regularisation strength: largerCCpenalises weight magnitude less\.

### IV\-EEvaluation Protocols

Deliberately shifting the test distribution is standard practice for exposing brittle decision rules\[[46](https://arxiv.org/html/2609.25006#bib.bib27),[30](https://arxiv.org/html/2609.25006#bib.bib28)\]\. Three protocols partition the same cleaned corpus\.

Random \(standard\)\.Stratified70/10/2070/10/20train/validation/test\. This reproduces conventional practice and provides the reference score\.

Topic\-disjoint\.Training and test pools are drawn from disjointsubjectvalues:politicsNews\(real\) andNews,politics,left\-news\(fake\) for training;worldnews\(real\) andGovernment News,US\_News,Middle\-east\(fake\) for test\. Validation is carved from the training pool only, so no target\-domain labels are available at model\-selection time—the realistic deployment condition\. Because subject values also proxy for topic and sub\-publication, this protocol perturbs provenance and topic jointly\.

Temporal\.Documents are ordered by publication date and split chronologically70/10/2070/10/20, restricted to the window in which both classes are present so that the test period is not trivially single\-class\.

Both shift protocols alter the class prior as a side effect \(Table[II](https://arxiv.org/html/2609.25006#S6.T2)\)\. SinceF1F\_\{1\}depends on the prior, a rawF1F\_\{1\}comparison across protocols conflates prior shift with loss of discrimination\. Algorithm[2](https://arxiv.org/html/2609.25006#alg2)therefore reports four complementary quantities:F1F\_\{1\}at the default threshold \(what a deployed system would achieve\), threshold\-oracleF1F\_\{1\}\(the best any cutoff on the model’s own scores could achieve, isolating miscalibration\), average precision \(threshold\-free ranking quality\), andF1F\_\{1\}on prior\-matched subsamples in which negatives are resampled so that the fake ratio equals that of the random split\. Only the last two support a like\-for\-like comparison of discriminative power\.

Algorithm 2Prior\-controlled shift evaluation1:Fitted model

ffwith scores

s​\(⋅\)s\(\\cdot\); test set

𝒮te\\mathcal\{S\}\_\{\\text\{te\}\}; reference prior

π⋆\\pi^\{\\star\}; repeats

MM; seed

σ\\sigma
2:

y^←f​\(𝒮te\)\\hat\{y\}\\leftarrow f\(\\mathcal\{S\}\_\{\\text\{te\}\}\);

s←s​\(𝒮te\)s\\leftarrow s\(\\mathcal\{S\}\_\{\\text\{te\}\}\)
3:

F1def←F1​\(y,y^\)F\_\{1\}^\{\\text\{def\}\}\\leftarrow F\_\{1\}\(y,\\hat\{y\}\)⊳\\trianglerightdeployed operating point

4:

F1orc←maxθ⁡F1​\(y,𝕀​\[s\>θ\]\)F\_\{1\}^\{\\text\{orc\}\}\\leftarrow\\max\_\{\\theta\}F\_\{1\}\\big\(y,\\mathbb\{I\}\[s\>\\theta\]\\big\)⊳\\trianglerightisolates calibration

5:

AP←averagePrecision​\(y,s\)\\mathrm\{AP\}\\leftarrow\\textsc\{averagePrecision\}\(y,s\)⊳\\trianglerightthreshold\-free

6:

𝒫←\{i:yi=1\}\\mathcal\{P\}\\leftarrow\\\{i:y\_\{i\}\{=\}1\\\};

𝒩←\{i:yi=0\}\\mathcal\{N\}\\leftarrow\\\{i:y\_\{i\}\{=\}0\\\}
7:

m←round​\(\|𝒫\|​\(1−π⋆\)/π⋆\)m\\leftarrow\\mathrm\{round\}\\big\(\|\\mathcal\{P\}\|\(1\-\\pi^\{\\star\}\)/\\pi^\{\\star\}\\big\)
8:for

j←1j\\leftarrow 1to

MMdo

9:

𝒩j∼sampleWithoutReplacement​\(𝒩,m;σ\)\\mathcal\{N\}\_\{j\}\\sim\\textsc\{sampleWithoutReplacement\}\(\\mathcal\{N\},m;\\sigma\)
10:

ϕj←F1​\(y𝒫∪𝒩j,y^𝒫∪𝒩j\)\\phi\_\{j\}\\leftarrow F\_\{1\}\\big\(y\_\{\\mathcal\{P\}\\cup\\mathcal\{N\}\_\{j\}\},\\hat\{y\}\_\{\\mathcal\{P\}\\cup\\mathcal\{N\}\_\{j\}\}\\big\)
11:endfor

12:return

F1def,F1orc,AP,mean​\(ϕ\),sd​\(ϕ\)F\_\{1\}^\{\\text\{def\}\},\\,F\_\{1\}^\{\\text\{orc\}\},\\,\\mathrm\{AP\},\\,\\mathrm\{mean\}\(\\phi\),\\,\\mathrm\{sd\}\(\\phi\)

### IV\-FCapacity Control: Fine\-Tuned DistilBERT

Answering Q4 requires a model of substantially greater capacity evaluated under identical conditions\. We fine\-tune DistilBERT\[[52](https://arxiv.org/html/2609.25006#bib.bib47)\], a six\-layer distillation of BERT\[[9](https://arxiv.org/html/2609.25006#bib.bib46)\]with roughly 66M parameters, whose subword tokenisation and contextual self\-attention\[[58](https://arxiv.org/html/2609.25006#bib.bib45)\]can in principle represent the semantic content that a bag of words cannot\.

Comparability is the whole point of the experiment, so two things are held fixed and one is deliberately allowed to differ\. The document set and the split assignment are*identical*: we rebuild the primary frame preserving the original row index, replicate each protocol exactly, and assert that the resulting split cardinalities match those in Table[II](https://arxiv.org/html/2609.25006#S6.T2)before training proceeds\. The leakage controls C1–C3 are also identical\.

What differs is surface normalisation\. The linear pipeline maps every character outside\[a\-\-z’\]to whitespace, which suits a bag\-of\-words model but discards punctuation and casing that a subword transformer can exploit\. Applying that destructive normalisation would handicap the transformer for reasons unrelated to Q4, so the transformer receives a minimally normalised variant of the*same*documents: the source tag, URLs, and e\-mail addresses are removed and whitespace is collapsed, but punctuation and digits are retained\. This asymmetry favours the transformer, which is the conservative direction for our argument—we are testing whether extra capacity*helps*, so giving it the better input strengthens any negative result\.

Training uses the recipe standard for this model class: batch size 16, learning rate2×10−52\\times 10^\{\-5\}with linear warmup over the first 10% of steps, two epochs, mixed\-precision arithmetic, and a maximum sequence length of 256 subword tokens\. No hyperparameter search was performed, for the same reason it was omitted for the linear models\. Model selection follows the same rule as the linear models—the per\-epoch checkpoint with the best*validation*F1F\_\{1\}is the one evaluated on test\. Because the corpus mean of 415 words exceeds 256 subword tokens, we additionally repeat the random protocol at a 512\-token limit to confirm that truncation is not what determines the outcome\.

### IV\-GMetrics

We report accuracy, and precision, recall, andF1F\_\{1\}for the positive \(fake\) class:

P=TPTP\+FP,R=TPTP\+FN,F1=2​P​RP\+R\.P=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FP\}\},\\quad R=\\frac\{\\mathrm\{TP\}\}\{\\mathrm\{TP\}\+\\mathrm\{FN\}\},\\quad F\_\{1\}=\\frac\{2PR\}\{P\+R\}\.\(10\)Taking*fake*as positive makes the error types operationally meaningful: a false positive suppresses a legitimate article, a false negative lets misinformation through\. We additionally report ROC\-AUC\[[20](https://arxiv.org/html/2609.25006#bib.bib48)\], which equals the probability that a randomly chosen fake article receives a higher score than a randomly chosen real one and is invariant to the decision threshold, and average precision, which is more informative than ROC\-AUC when the positive class is rare—as it becomes under both shift protocols\. Model selection uses validationF1F\_\{1\}throughout, a rule fixed before any test set was examined\.

## VSystem Architecture

Raw corpusTrue\.csv21,417Fake\.csv23,481Leakage controlsC1 metadataC2 source tagC3 duplicatesNormaliselowercase, URLs,alpha\-only, collapseProtocol splitrandom /topic\-disjoint /temporalTF–IDF1–2 gramsV=50,000V=50\{,\}000*fit on train*Linear modelsLR, MNB,SVC, PASelect onvalidationF1F\_\{1\}Single testevaluationAnalysesablation, shift,deletion probeReportedtables andfiguresprotocol variedFigure 1:Audit architecture\. The measurement pipeline \(solid, blue\) is held fixed while the leakage controls C1–C3 \(dashed, red\) and the evaluation protocol are varied\. The validation split governs model selection; the test split is evaluated exactly once per condition\. Every reported figure is regenerated from the raw corpus by the released scripts\.Figure[1](https://arxiv.org/html/2609.25006#S5.F1)shows the system\. It is organised so that exactly one factor varies per experimental condition: the preprocessing controls C1–C3 of Section[IV\-B](https://arxiv.org/html/2609.25006#S4.SS2), the representation hyperparameters of Equations \([3](https://arxiv.org/html/2609.25006#S4.E3)\)–\([5](https://arxiv.org/html/2609.25006#S4.E5)\), or the splitting protocol of Section[IV\-E](https://arxiv.org/html/2609.25006#S4.SS5)\. Configuration is centralised in a single module and serialised into the results file with every run, so each reported number carries the exact settings that produced it\.

### V\-AComputational Complexity

LetNNbe the number of training documents,L¯\\bar\{L\}the mean token count,VVthe vocabulary size, andzzthe mean number of non\-zero features per document\. Vectorisation isO​\(N​L¯\)O\(N\\bar\{L\}\)time, dominated by tokenisation andnn\-gram enumeration, with a vocabulary\-construction pass overO​\(N​L¯\)O\(N\\bar\{L\}\)candidate terms before frequency pruning\. Training the linear models costsO​\(κ​N​z\)O\(\\kappa Nz\)forκ\\kappapasses: coordinate descent for logistic regression and the SVM\[[14](https://arxiv.org/html/2609.25006#bib.bib61)\], a single closed\-form counting pass for naive Bayes, andκ\\kappaonline epochs for passive–aggressive\. Inference isO​\(z\)O\(z\)per document—one sparse dot product\.

The measured quantities corroborate these rates and explain why the audit is cheap enough to repeat across 20 conditions\. On the training splitz=190\.1z=190\.1ofV=50,000V=50\{,\}000features, i\.e\. a density of0\.38%0\.38\\%; the sparse matrix holds5\.17×1065\.17\\times 10^\{6\}non\-zeros in≈62\\approx 62MB at 12 bytes per stored value, whereas a dense equivalent would require≈10\.9\\approx 10\.9GB\. Sparse storage is thus not an optimisation but a precondition\. Section[VII\-G](https://arxiv.org/html/2609.25006#S7.SS7)reports wall\-clock figures\.

## VIExperimental Setup

Data\.The corpus is obtained from its public distribution\[[4](https://arxiv.org/html/2609.25006#bib.bib3)\]and is not redistributed with our code\. Table[I](https://arxiv.org/html/2609.25006#S6.T1)reports its composition together with the leakage indicators discussed in Section[IV\-B](https://arxiv.org/html/2609.25006#S4.SS2); all values are computed by the released script rather than quoted from prior work\. The cross\-corpus test \(Section[VII\-F](https://arxiv.org/html/2609.25006#S7.SS6)\) additionally uses the LIAR test split\[[62](https://arxiv.org/html/2609.25006#bib.bib4)\]:1,2671\{,\}267short political statements \(mean18\.418\.4tokens, against ISOT’s415415\) fact\-checked by PolitiFact\. LIAR’s six\-way labels are binarised with the conventional cut—\{pants\-fire, false, barely\-true\}→\\rightarrowfake and \{half\-true, mostly\-true, true\}→\\rightarrowreal—which yields a test prior of0\.4360\.436fake\. The barely\-true/half\-true boundary is the one debatable choice; moving it shifts the prior but not the near\-chance result of Section[VII\-F](https://arxiv.org/html/2609.25006#S7.SS6)\.

Hardware and software\.All experiments run on a single machine: Intel Core i7\-14700HX \(28 logical cores\), 15\.7 GB RAM, Windows 10 \(AMD64\)\. The linear pipeline uses the CPU only; the DistilBERT control uses one NVIDIA GeForce RTX 4050 Laptop GPU \(6 GB\), peaking at 3\.9 GB of device memory\. The stack is Python 3\.10\.11 with scikit\-learn 1\.7\.2\[[40](https://arxiv.org/html/2609.25006#bib.bib60)\], NumPy 1\.24\.3\[[21](https://arxiv.org/html/2609.25006#bib.bib62)\], SciPy 1\.15\.3\[[59](https://arxiv.org/html/2609.25006#bib.bib63)\], pandas 2\.3\.3\[[33](https://arxiv.org/html/2609.25006#bib.bib64)\], and Matplotlib 3\.10\.8\[[23](https://arxiv.org/html/2609.25006#bib.bib65)\]; the transformer control adds PyTorch 2\.6\.0 \(CUDA 12\.4\) and Transformers 4\.56\.1\. Exact versions are pinned in the releasedrequirements\.txt\.

Reproducibility\.A single seed \(σ=42\\sigma=42\) governs the splits and all stochastic model components\. Text normalisation is purely regular; the vocabulary is frequency\-ordered and deterministic; logistic regression uses the deterministicliblinearsolver\. Two independent executions of the released pipeline produced bit\-identical metrics\. This guarantee covers the linear pipeline; the DistilBERT control is seeded identically but runs mixed\-precision CUDA kernels whose reduction order is not guaranteed, so its figures should be expected to reproduce to roughly three decimal places rather than exactly\. Following community recommendations on reporting practice\[[43](https://arxiv.org/html/2609.25006#bib.bib59),[47](https://arxiv.org/html/2609.25006#bib.bib54)\]we release code, seeds, configuration snapshots, and the machine\-readable results file from which every table and figure in this paper is generated\. Appendix[D](https://arxiv.org/html/2609.25006#A4)contains the reproducibility checklist and Appendix[A](https://arxiv.org/html/2609.25006#A1)the full hyperparameter listing\.

Statistical procedure\.Point estimates on the fixed test split are accompanied by percentile bootstrap 95% confidence intervals over20002000resamples of the test set\[[13](https://arxiv.org/html/2609.25006#bib.bib51)\]\. Because conclusions from a single split can be fragile\[[17](https://arxiv.org/html/2609.25006#bib.bib31)\], we additionally report five\-fold stratified cross\-validationF1F\_\{1\}on the union of the training and validation splits, refitting the vectoriser within each fold\. Pairwise model comparisons use McNemar’s exact test on discordant predictions\[[34](https://arxiv.org/html/2609.25006#bib.bib49),[10](https://arxiv.org/html/2609.25006#bib.bib50)\], the appropriate paired test when two classifiers are evaluated on the same test set\[[12](https://arxiv.org/html/2609.25006#bib.bib53)\]; omnibus procedures designed for comparisons across many datasets\[[8](https://arxiv.org/html/2609.25006#bib.bib52)\]do not apply to a single\-corpus design\. Prior\-matchedF1F\_\{1\}is averaged overM=20M=20resamples \(Algorithm[2](https://arxiv.org/html/2609.25006#alg2)\)\.

Sanity control\.As a check that the harness itself introduces no leakage, we ran the complete primary pipeline with the labels randomly permuted\. Test ROC\-AUC was0\.49740\.4974, statistically indistinguishable from the chance value of0\.50\.5, confirming that the reported performance originates in the data rather than in an implementation defect\.

TABLE I:Corpus composition and leakage indicators\. All quantities are computed from the raw distribution by the released script\.TABLE II:Evaluation protocols\. Both shift protocols alter the class prior as a side effect, which motivates the prior\-controlled analysis of Algorithm[2](https://arxiv.org/html/2609.25006#alg2)\.
## VIIResults

### VII\-AReference Performance

Table[III](https://arxiv.org/html/2609.25006#S7.T3)reports the four classifiers under the standard random protocol\. Passive–aggressive attains the best validationF1F\_\{1\}\(0\.99240\.9924\) and is selected; on the test split it reaches accuracy0\.99120\.9912, precision0\.99330\.9933, recall0\.98770\.9877,F1F\_\{1\}0\.99050\.9905, and ROC\-AUC0\.99950\.9995, reproducing the range reported in the literature for this corpus\[[2](https://arxiv.org/html/2609.25006#bib.bib2),[29](https://arxiv.org/html/2609.25006#bib.bib15)\]\. Its confusion matrix \(Figure[2](https://arxiv.org/html/2609.25006#S7.F2)\) contains 24 false positives and 44 false negatives out of7,7667\{,\}766test documents\. Bootstrap intervals are tight—F1F\_\{1\}0\.99050\.9905, 95% CI\[0\.9883,0\.9928\]\[0\.9883,0\.9928\]—and five\-fold cross\-validation agrees closely \(0\.9916±0\.00150\.9916\\pm 0\.0015\), so the reference figure is not an artefact of the particular split\.

Two observations already qualify the headline number\. First, the three margin\- and likelihood\-based models are separated by very little: McNemar’s exact test finds no significant difference between passive–aggressive and LinearSVC \(n01=8n\_\{01\}\{=\}8,n10=11n\_\{10\}\{=\}11,p=0\.65p=0\.65\), so the selected “winner” is statistically indistinguishable from the runner\-up, and reporting it as the best model would overstate what the data support\. Differences against logistic regression \(p=3\.2×10−12p=3\.2\\times 10^\{\-12\}\) and naive Bayes \(p=3\.5×10−55p=3\.5\\times 10^\{\-55\}\) are significant\. Second, the mis\-specified naive Bayes baseline still reachesF1F\_\{1\}0\.95580\.9558\. When a model whose independence assumption is violated by construction lands within four points of the best system, the task as posed is unlikely to require sophisticated inference\.

TABLE III:Reference performance under the random protocol\. Selection uses validationF1F\_\{1\}; test columns are computed once for the selected model and, for completeness, for the remaining models\. CV is five\-fold stratifiedF1F\_\{1\}on train∪\{\\,\\cup\\,\}validation\. The final column gives McNemarpp\-values against the selected model\.Val\.TestModelF1F\_\{1\}Acc\.PPRRF1F\_\{1\}AUCCVF1F\_\{1\}Logistic regression0\.98400\.98300\.99060\.97230\.98140\.99870\.98290\.9829±\.0030\\pm\.0030Multinomial NB0\.95690\.95920\.95380\.95780\.95580\.99110\.95700\.9570±\.0010\\pm\.0010LinearSVC0\.99220\.99090\.99330\.98690\.99010\.99950\.99060\.9906±\.0015\\pm\.0015Passive–aggressive†0\.99240\.99120\.99330\.98770\.99050\.99950\.9916\\mathbf\{0\.9916\}±\.0015\\pm\.0015†selected\. McNemar vs\. selected: LinearSVCp=0\.65p\{=\}0\.65\(n\.s\.\);logistic regressionp=3\.2×10−12p\{=\}3\.2\{\\times\}10^\{\-12\}; naive Bayesp=3\.5×10−55p\{=\}3\.5\{\\times\}10^\{\-55\}\.Bootstrap 95% CI \(selected\): Acc\.\[\.9892,\.9933\]\[\.9892,\.9933\],F1F\_\{1\}\[\.9883,\.9928\]\[\.9883,\.9928\],PP\[\.9905,\.9958\]\[\.9905,\.9958\],RR\[\.9839,\.9913\]\[\.9839,\.9913\], AUC\[\.9992,\.9997\]\[\.9992,\.9997\]\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x1.png)Figure 2:Confusion matrix of the selected passive–aggressive model on the random\-protocol test split \(n=7,766n=7\{,\}766\):3,5363\{,\}536true positives,4,1624\{,\}162true negatives, 24 false positives, 44 false negatives\. The 24 false positives are legitimate articles that would be suppressed; the 44 false negatives are fake articles that would pass\.
### VII\-BQ1: Explicit Leakage Channels

Table[IV](https://arxiv.org/html/2609.25006#S7.T4)and Figure[3](https://arxiv.org/html/2609.25006#S7.F3)isolate the three channels using logistic regression as a fixed probe\.

The most consequential result requires no classifier of the article at all\. A model whose only input is thesubjectfield, with the article text discarded entirely, attainsF1F\_\{1\}=1\.000=1\.000and accuracy=1\.000=1\.000on a held\-out split, with a vocabulary of nine tokens\. This is not a subtle correlation but a deterministic mapping: the two classes have disjoint subject supports \(Table[I](https://arxiv.org/html/2609.25006#S6.T1)\), so any model given this field can recover the label exactly\. Any experiment on this corpus that retainssubjectamong its features is measuring nothing about text\.

Restricting inputs to title and body, the naive configuration reachesF1F\_\{1\}0\.99350\.9935\. De\-duplication alone costs0\.700\.70points and source\-tag removal alone0\.490\.49points; applied together they cost1\.211\.21points, yielding the primary figure of0\.98140\.9814\. The duplicate channel has a clear mechanism: under the naive protocol1,7391\{,\}739of8,9788\{,\}978test documents \(19\.37%19\.37\\%\) occur verbatim in the training set, so roughly a fifth of the test score reflects retrieval of memorised strings rather than generalisation\.

The honest reading of Table[IV](https://arxiv.org/html/2609.25006#S7.T4)is two\-sided, and we state both directions\. Each channel is real, measurable, and worth removing; a study that ignores all three overstates its result by more than a point and, if it retains metadata, by far more\. Yet the mitigations do*not*bring performance down to a plausible level for veracity assessment—0\.98140\.9814remains close to the ceiling\. Explicit artifacts are therefore not the principal explanation for the benchmark’s easiness, which motivates Q2\.

TABLE IV:Leakage ablation with a fixed logistic\-regression probe\.Δ\\Deltais the change in testF1F\_\{1\}relative to the naive configuration\. The metadata\-only row uses thesubjectfield alone, with the article text discarded\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x2.png)Figure 3:Effect of the leakage mitigations on testF1F\_\{1\}\(logistic\-regression probe\)\. Removing all three channels costs1\.211\.21points, leaving performance close to the ceiling\. The dashed line marks the metadata\-only baseline, which solves the benchmark exactly without reading any article text\.
### VII\-CQ2: The Residual Signal Is Diffuse

Three independent probes indicate that what survives mitigation is not a small set of removable cues but a pervasive stylistic difference\.

Representation is nearly irrelevant\.Table[V](https://arxiv.org/html/2609.25006#S7.T5)varies the feature space over nine configurations spanningnn\-gram order, vocabulary size across a 35\-fold range, term weighting, and stop\-word handling\. TestF1F\_\{1\}lies between0\.97730\.9773and0\.98350\.9835—a spread of0\.620\.62points\. A5,0005\{,\}000\-term unigram\-and\-bigram vocabulary \(0\.98350\.9835\) slightly*outperforms*the50,00050\{,\}000\-term default and the175,406175\{,\}406\-term unrestricted vocabulary \(0\.98000\.9800\)\. When a tenfold reduction in capacity does not degrade performance, the discriminative information is heavily redundant\.

Very little data suffices\.Figure[4](https://arxiv.org/html/2609.25006#S7.F4)shows the learning curve\. With543543labelled documents,2%2\\%of the training split, the probe reachesF1F\_\{1\}0\.93340\.9334, i\.e\.95\.1%95\.1\\%of the full\-data figure;135135documents already yield0\.68020\.6802\. Tasks requiring genuine inference do not typically saturate on a few hundred examples\.

Deleting the strongest cues is not enough\.Ranking unigrams by\|w\|\|w\|under the probe and removing the topKKfrom the vocabulary entirely \(so bigrams containing them are also destroyed\) yields Figure[5](https://arxiv.org/html/2609.25006#S7.F5):F1F\_\{1\}declines gracefully from0\.98140\.9814atK=0K=0to0\.96000\.9600atK=100K=100and0\.92630\.9263atK=1000K=1000\. Removing the thousand most discriminative unigrams \(among them*said*,*video*,*washington*,*image*,*featured*,*getty*, and the weekday names\) still leaves a model within5\.55\.5points of the original\. The shortcut is not a token; it is a register\.

Figure[6](https://arxiv.org/html/2609.25006#S7.F6)makes that register legible\. The strongest evidence for*real*consists of newswire conventions \(*said*, attribution phrasing, weekday datelines,*factbox*\); the strongest evidence for*fake*consists of web\-publishing conventions \(*video*,*featured image*,*getty images*,*watch*\)\. These are properties of editorial workflow and content\-management systems, not of factual accuracy—exactly the provenance signaturessof Section[III](https://arxiv.org/html/2609.25006#S3)\.

TABLE V:Representation and input\-field ablation \(logistic\-regression probe, primary cleaning, random protocol\)\. Performance is insensitive to the feature space across a 35\-fold range of vocabulary size\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x3.png)Figure 4:Learning curve of the logistic\-regression probe under the primary configuration\. Two per cent of the training data recovers95\.1%95\.1\\%of the full\-dataF1F\_\{1\}, indicating a signal that is easy to acquire\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x4.png)Figure 5:Shortcut concentration\. Deleting theKKhighest\-weight unigrams from the vocabulary degradesF1F\_\{1\}only gradually; atK=1000K=1000the model remains within5\.55\.5points of the unmodified baseline\. The discriminative signal is distributed across many stylistic features rather than concentrated in a few\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x5.png)Figure 6:Highest\-magnitude logistic\-regression coefficients under the primary configuration\. Real\-class evidence reflects newswire house style; fake\-class evidence reflects web\-publishing conventions such as embedded media and image credits\. Neither group encodes veracity\.
### VII\-DQ3: Transfer Under Distribution Shift

Table[VI](https://arxiv.org/html/2609.25006#S7.T6)and Figure[7](https://arxiv.org/html/2609.25006#S7.F7)report the prior\-controlled analysis; Figure[8](https://arxiv.org/html/2609.25006#S7.F8)contrasts ROC behaviour between protocols\.

Under the topic\-disjoint protocol the deployed operating point degrades severely:F1F\_\{1\}for the selected model falls from0\.99050\.9905to0\.80670\.8067\(−18\.4\-18\.4points\), and for logistic regression from0\.98140\.9814to0\.69960\.6996\(−28\.2\-28\.2points\)\. Precision drives the collapse \(passive–aggressive:0\.9933→0\.70110\.9933\\rightarrow 0\.7011\) while recall is largely preserved \(0\.9877→0\.94980\.9877\\rightarrow 0\.9498\), the signature of a decision threshold that is no longer appropriate for the target class prior, which falls from0\.4610\.461to0\.1190\.119\.

Decomposing that collapse is essential for an honest claim, and it cuts both ways\. Matching the prior by resampling negatives recovers most of the loss \(F1F\_\{1\}0\.9427±0\.00320\.9427\\pm 0\.0032\), and the threshold\-oracle value \(0\.88870\.8887\) shows that a substantial part of the remainder is miscalibration rather than an inability to rank\. We therefore do*not*claim an 18\-point loss of discriminative power\. However, average precision—which is threshold\-free and so immune to both effects—falls from0\.99950\.9995to0\.94750\.9475, a genuine5\.25\.2\-point degradation, and balanced accuracy falls from0\.99100\.9910to0\.94750\.9475\. Logistic regression degrades further \(AP0\.9985→0\.90180\.9985\\rightarrow 0\.9018,−9\.7\-9\.7points\), indicating that the more heavily regularised probe leans harder on cues that do not transfer\. The correct statement is thus: under topic shift the model retains much of its ranking ability but loses a real and measurable fraction of it, and its deployed decision rule becomes unusable without recalibration on target data that a real deployment would not possess\.

Temporal shift, by contrast, is nearly benign: average precision moves only from0\.99950\.9995to0\.99640\.9964and prior\-matchedF1F\_\{1\}is statistically indistinguishable from the random\-protocol value \(0\.9914±0\.00130\.9914\\pm 0\.0013versus0\.99050\.9905\)\. The asymmetry is informative\. Publisher house style is stable over the 2016–2017 window, so a chronological split, the protocol often recommended as a realism improvement, does*not*expose the shortcut\. Only perturbing provenance and topic does\. Practitioners seeking a stress test for this corpus should therefore prefer topic\- or source\-disjoint splits over temporal ones\.

TABLE VI:Prior\-controlled distribution\-shift analysis\.F1defF\_\{1\}^\{\\text\{def\}\}is the deployed operating point;F1orcF\_\{1\}^\{\\text\{orc\}\}is the bestF1F\_\{1\}attainable by any threshold on the model’s own scores; AP is average precision;F1matchF\_\{1\}^\{\\text\{match\}\}isF1F\_\{1\}on subsamples whose class prior is resampled to that of the random protocol \(mean±\\pms\.d\. overM=20M\{=\}20draws\)\. ComparingF1defF\_\{1\}^\{\\text\{def\}\}across protocols conflates prior shift with loss of discrimination; AP andF1matchF\_\{1\}^\{\\text\{match\}\}do not\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x6.png)Figure 7:Selected model across protocols\. The gap between the twoF1F\_\{1\}bars under topic\-disjoint evaluation is the contribution of class\-prior shift and threshold miscalibration; the drop in average precision is the genuine loss of discriminative power\. Temporal shift leaves both essentially intact\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x7.png)Figure 8:ROC curves under the random \(left\) and topic\-disjoint \(right\) protocols, with ROC\-AUC in parentheses\. ROC\-AUC remains high under topic shift even where deployedF1F\_\{1\}collapses, illustrating why threshold\-free ranking metrics alone can mask an unusable operating point when the positive class becomes rare\.
### VII\-EQ4: Pretrained Capacity Amplifies the Shortcut

Table[VIII](https://arxiv.org/html/2609.25006#S7.T8)and Figure[9](https://arxiv.org/html/2609.25006#S7.F9)report the DistilBERT control against the selected linear model on identical documents and splits\.

In\-distribution the transformer is clearly the stronger classifier, as expected:F1F\_\{1\}rises from0\.99050\.9905to0\.99930\.9993and average precision from0\.99950\.9995to1\.00001\.0000\(to four decimals; the unrounded value is0\.9999990\.999999\)\. The random\-split benchmark therefore rewards capacity, and a study reporting only this row would conclude that a pretrained transformer is the better system\.

Under topic shift the ordering reverses sharply, and it does so on precisely the metrics that are robust to the prior\. Average precision falls from1\.00001\.0000to0\.87110\.8711, a loss of12\.912\.9points against the linear model’s5\.25\.2; prior\-matchedF1F\_\{1\}falls to0\.6886±0\.00270\.6886\\pm 0\.0027against the linear model’s0\.94270\.9427; and balanced accuracy falls to0\.61310\.6131against0\.94750\.9475\. At its actual operating point the transformer collapses toF1F\_\{1\}=0\.2592=0\.2592with precision0\.14890\.1489at recall0\.99920\.9992: it labels almost everything fake, and since the shifted test prior is0\.1190\.119, a precision of0\.14890\.1489is only marginally better than a degenerate always\-fake rule\. Two qualifications keep this honest\. The ranking is not destroyed—an average precision of0\.87110\.8711remains far above the0\.1190\.119expected of a random ranker—and the threshold\-oracle value of0\.80230\.8023shows that a well\-chosen cutoff would recover much of the deployed loss\. But every prior\-robust comparison places the transformer well below the linear model off\-distribution, so the reversal is not an artefact of calibration alone\.

Temporal shift again shows the opposite pattern: the transformer holds atF1F\_\{1\}=0\.9922=0\.9922and average precision0\.99990\.9999, marginally ahead of the linear model\. This reinforces the conclusion of Section[VII\-D](https://arxiv.org/html/2609.25006#S7.SS4)that a chronological split is not a meaningful stress test for this corpus, for either model class\.

Truncation does not explain the result\. Repeating the random protocol with a 512\-token limit changesF1F\_\{1\}only from0\.99930\.9993to0\.99940\.9994\(Table[VIII](https://arxiv.org/html/2609.25006#S7.T8)\), so the 256\-token window is not the binding constraint on what the model can see\.

The most practically alarming number in the table is not a test score but a validation score\. Under the topic\-disjoint protocol the selected checkpoint had validationF1F\_\{1\}=0\.9995=0\.9995, higher than under the random protocol, while its testF1F\_\{1\}was0\.25920\.2592\. Because the validation split must be drawn from the training pool when no target\-domain labels exist, standard model selection returned a confident, well\-validated model that fails almost completely on the target distribution\. No amount of care with the validation set detects this; only a shifted evaluation does\.

*Robustness across seeds\.*Table[VIII](https://arxiv.org/html/2609.25006#S7.T8)reports a single representative run \(seed 42\) for a like\-for\-like comparison with the deterministic linear models\. To confirm that its topic\-shift collapse is not a lucky draw, we repeat both protocols across five seeds \(Table[VII](https://arxiv.org/html/2609.25006#S7.T7)\)\. In\-distribution performance is essentially constant \(average precision1\.0000±0\.00001\.0000\\pm 0\.0000\)\. The topic\-disjoint result is stable in direction and moderate in spread: average precision is0\.8610±0\.05670\.8610\\pm 0\.0567and prior\-matchedF1F\_\{1\}is0\.6858±0\.02510\.6858\\pm 0\.0251, and every one of the five seeds lands below the linear model’s average precision of0\.94750\.9475—the best seed reaches only0\.92320\.9232\. The model\-selection trap is the most stable feature of all: validationF1F\_\{1\}is0\.9999±0\.00010\.9999\\pm 0\.0001across seeds while deployed testF1F\_\{1\}averages0\.2575±0\.02350\.2575\\pm 0\.0235\. The single\-seed conclusion therefore holds under reseeding\.

TABLE VII:Five\-seed distribution \(seeds\{42,0,1,2,3\}\\\{42,0,1,2,3\\\}\) of the DistilBERT control, as mean±\\pmstandard deviation\. In\-distribution scores are constant; the topic\-disjoint collapse is robust, and its validationF1F\_\{1\}stays near11in every seed even as testF1F\_\{1\}averages0\.260\.26\.Taken together, Q4 answers in the direction that the data, not the instrument, is the binding constraint\. Capacity sufficient to represent veracity\-bearing content did not cause the model to use it\. Instead the transformer fit the provenance signature more sharply, reaching an essentially perfect in\-distribution ranking, and paid for it with a steeper loss when that signature stopped being predictive\.

TABLE VIII:Capacity control: fine\-tuned DistilBERT against the selected linear model on identical documents and splits\. Columns follow Table[VI](https://arxiv.org/html/2609.25006#S7.T6)\. The transformer wins in\-distribution on every metric and loses off\-distribution on every prior\-robust metric\. The final row repeats the random protocol at a 512\-token limit, confirming that truncation is not the operative factor\. ValidationF1F\_\{1\}is shown to expose the model\-selection trap: under topic shift it is at its highest exactly where test performance is at its worst\. DistilBERT rows are the seed\-42 run; the five\-seed distribution is in Table[VII](https://arxiv.org/html/2609.25006#S7.T7)\.ProtocolModelVal\.F1F\_\{1\}PPRRF1defF\_\{1\}^\{\\text\{def\}\}F1orcF\_\{1\}^\{\\text\{orc\}\}AUCAPBal\. acc\.F1matchF\_\{1\}^\{\\text\{match\}\}Random \(standard\)TF–IDF\+\+PA0\.99240\.99330\.98770\.99050\.99180\.99950\.99950\.99100\.99050\.9905DistilBERT0\.99920\.99970\.99890\.99930\.99971\.00001\.00000\.99930\.9993\\mathbf\{0\.9993\}Topic\-disjointTF–IDF\+\+PA0\.99280\.70110\.94980\.80670\.88870\.98720\.94750\.94750\.9427\\mathbf\{0\.9427\}±\.0032\\pm\.0032DistilBERT0\.99950\.14890\.99920\.25920\.80230\.96540\.87110\.61310\.68860\.6886±\.0027\\pm\.0027TemporalTF–IDF\+\+PA0\.96840\.96700\.98800\.97740\.98090\.99930\.99640\.99180\.99140\.9914±\.0013\\pm\.0013DistilBERT0\.99560\.98570\.99880\.99220\.99461\.00000\.99990\.99850\.9985\\mathbf\{0\.9985\}Random, 512 tokensDistilBERT0\.99970\.99940\.99940\.99940\.99961\.00001\.00000\.99950\.99940\.9994![Refer to caption](https://arxiv.org/html/2609.25006v1/x8.png)Figure 9:Prior\-robust comparison across protocols\. DistilBERT matches or exceeds the linear model in\-distribution and under temporal shift, but loses substantially more under topic shift on both average precision and prior\-matchedF1F\_\{1\}\. Greater capacity fits the provenance signature more sharply rather than replacing it with veracity\-bearing evidence\.
### VII\-FQ5: Cross\-Corpus Transfer Collapses to Chance

The within\-corpus protocols hold the corpus fixed\. The decisive test of external validity is to change the corpus entirely\. We train each model on all38,82638\{,\}826ISOT documents and evaluate on the LIAR test split, whose short single\-source claims share almost none of ISOT’s provenance structure\. Table[IX](https://arxiv.org/html/2609.25006#S7.T9)reports the result, and Figure[10](https://arxiv.org/html/2609.25006#S7.F10)places it beside the in\-corpus and topic\-shift figures\.

The transfer is essentially complete failure\. ROC\-AUC falls to0\.53980\.5398for logistic regression,0\.56170\.5617for passive–aggressive, and0\.56910\.5691for DistilBERT, all within seven points of the chance value of0\.50\.5; balanced accuracy lies between0\.52440\.5244and0\.54870\.5487against a chance value of0\.50\.5; and average precision \(0\.48590\.4859to0\.50600\.5060\) barely exceeds the0\.4360\.436expected of a random ranker\. None of the three models reaches the accuracy of a majority\-class predictor \(0\.56350\.5635\): trained to near\-perfection on ISOT, they are worse than a constant “always real” rule on LIAR\. The ordering among them is preserved but meaningless at this scale—DistilBERT ranks marginally best, as it did in\-distribution, but an AUC of0\.56910\.5691is not a usable classifier\.

Two controls rule out trivial explanations\. Every one of the1,2671\{,\}267LIAR documents contains at least one term in the ISOT vocabulary, so the collapse is not an artefact of empty feature vectors; the shared words simply carry no veracity signal the ISOT model can use\. And the effect holds for the transformer as well as the linear models, so it is not a limitation of bag\-of\-words representation\. The contrast with the within\-corpus AUC values, which remain near1\.01\.0even under topic shift \(Table[VI](https://arxiv.org/html/2609.25006#S7.T6)\), is the sharpest statement of the paper’s thesis: what the models learn ranks ISOT documents almost perfectly and ranks LIAR documents no better than a coin\.

This is the external check that Q1–Q4 could only approach from inside the corpus\. A model can attainF1F\_\{1\}=0\.99=0\.99on this benchmark and carry essentially no transferable capacity to judge whether a claim is true\.

TABLE IX:Cross\-corpus transfer: trained on all of ISOT, tested on the LIAR test split \(n=1,267n=1\{,\}267, fake prior0\.4360\.436\)\. All models fall to near\-chance ranking \(AUC and balanced accuracy near0\.50\.5; average precision near the prior0\.4360\.436\) and none beats the majority\-class accuracy of0\.56350\.5635\. Compare the within\-corpus AUC values near1\.01\.0in Table[VI](https://arxiv.org/html/2609.25006#S7.T6)\.![Refer to caption](https://arxiv.org/html/2609.25006v1/x9.png)Figure 10:ROC\-AUC of each model across three evaluation regimes: in\-corpus random split, in\-corpus topic\-disjoint split, and cross\-corpus transfer to LIAR\. Ranking quality survives topic shift within ISOT but collapses to near\-chance on an independently constructed corpus\. The dashed line marks chance \(0\.50\.5\)\.
### VII\-GComputational Cost

Table[X](https://arxiv.org/html/2609.25006#S7.T10)reports measured cost on the hardware of Section[VI](https://arxiv.org/html/2609.25006#S6)\. Vectorising27,17727\{,\}177documents takes12\.1912\.19s and dominates training, which needs0\.440\.44s for logistic regression and under0\.20\.2s for the other three; the full four\-model pipeline completes in well under a minute on a single CPU\. The serialised pipeline occupies2\.152\.15MB and sustains3,0023\{,\}002documents per second in batch mode, with0\.660\.66ms single\-document latency\. Two consequences follow\. First, the audit is inexpensive: the complete linear study, comprising 20 fitted conditions, runs in roughly twelve minutes on one CPU, and the DistilBERT control adds about 32 minutes on one laptop GPU\. The diagnostics we recommend therefore impose no practical barrier\. Second, on cost grounds the linear pipeline would be an attractive deployment candidate—it needs no accelerator, and it holds the better off\-distribution operating point of the two arms—which is precisely why its failure under topic shift matters\. Scalability is favourable in the regime that matters here: memory grows asO​\(N​z\)O\(Nz\)rather thanO​\(N​V\)O\(NV\), and inference is independent ofNN, so throughput would be maintained on substantially larger corpora, with vocabulary construction the first component to require distribution\.

TABLE X:Measured computational cost of the linear pipeline \(single CPU, no GPU;N=27,177N=27\{,\}177training documents,V=50,000V=50\{,\}000\)\. The DistilBERT control, for comparison, requires5\.85\.8–6\.56\.5minutes of GPU time per protocol\.

## VIIIDiscussion

### VIII\-AWhat the Benchmark Score Measures

Consider the evidence together\. The corpus can be solved exactly from a metadata field \(Section[VII\-B](https://arxiv.org/html/2609.25006#S7.SS2)\), and removing that field along with two further leakage channels costs only1\.211\.21points\. What survives is insensitive to representation, acquirable from a few hundred examples, and robust to deleting a thousand of its strongest features \(Section[VII\-C](https://arxiv.org/html/2609.25006#S7.SS3)\); the features it does rely on are recognisably editorial conventions \(Figure[6](https://arxiv.org/html/2609.25006#S7.F6)\)\. Finally, it degrades measurably when provenance and topic change, but not when only time changes \(Section[VII\-D](https://arxiv.org/html/2609.25006#S7.SS4)\)\. The consistent explanation is that high scores on this benchmark quantify*source and topic separability*\. In the notation of Section[III](https://arxiv.org/html/2609.25006#S3), the pipeline is an efficient estimator ofss, and becauseI​\(s;y\)I\(s;y\)is near\-maximal in𝒫bench\\mathcal\{P\}\_\{\\text\{bench\}\}by the asymmetry of Equation \([2](https://arxiv.org/html/2609.25006#S3.E2)\), an estimator ofssis nearly an estimator ofyyin\-distribution\. Nothing in our results indicates that veracity\-bearing contentccis being used\.

This is a claim about the benchmark, not about the feasibility of the task\. It also does not imply prior work is invalid: reported numbers are reproducible under the protocols used\. What does not follow is the inference from those numbers to deployment readiness\.

### VIII\-BWhy Model Capacity Does Not Help

The capacity control of Section[VII\-E](https://arxiv.org/html/2609.25006#S7.SS5)lets us make this argument from our own evidence rather than by inference from the literature’s near\-ceiling clustering\[[29](https://arxiv.org/html/2609.25006#bib.bib15),[26](https://arxiv.org/html/2609.25006#bib.bib14)\]\. A pretrained transformer, able in principle to represent the veracity\-bearing contentccthat a bag of words cannot, did not use it\. It fit the provenance signaturessmore sharply instead, achieving an essentially perfect in\-distribution ranking, and then lost2\.52\.5times more average precision than the linear model whenssceased to be predictive\. Capacity was spent on the shortcut, not on an alternative to it\.

The explanation is that the limitation is an identifiability problem in the data rather than a modelling deficiency\.𝒫bench\\mathcal\{P\}\_\{\\text\{bench\}\}contains no counterfactual pairs \(same publisher, different veracity\), so no objective computed on it can distinguish a rule keyed onccfrom a rule keyed onss\. Both achieve identical empirical risk\. Gradient descent then selects whichever is easier to extract, and provenance style is trivially easier than factual verification\. A higher\-capacity model does not resolve this ambiguity; it merely resolves the easy side of it more precisely, which is why its in\-distribution advantage and its out\-of\-distribution disadvantage appear together\. This is the mechanism Schuster et al\.\[[53](https://arxiv.org/html/2609.25006#bib.bib17)\]describe for machine\-generated text, and our metadata\-only and deletion probes are the analogue of hypothesis\-only diagnostics in NLI\[[19](https://arxiv.org/html/2609.25006#bib.bib20),[45](https://arxiv.org/html/2609.25006#bib.bib21)\]\.

There is a corollary worth stating plainly, because it inverts a common assumption\. On a benchmark whose labels are confounded with provenance, a*higher*reported score is weak evidence of a better system and can be evidence of a worse one\. Our transformer would be selected over the linear model by any standard random\-split comparison, and it is the worse choice for deployment on unseen sources\.

### VIII\-CMetric Choice and Deployment Risk

The shift analysis carries a practical lesson beyond this corpus\. Under topic\-disjoint evaluation, ROC\-AUC remains at0\.98720\.9872for the selected model while deployedF1F\_\{1\}sits at0\.80670\.8067\(Figure[8](https://arxiv.org/html/2609.25006#S7.F8)\)\. A study reporting only ROC\-AUC would therefore describe a system that is, at its actual operating point, wrong about30%30\\%of its positive predictions\. Because misinformation is rare in realistic streams, precision as defined in Equation \([10](https://arxiv.org/html/2609.25006#S4.E10)\) is the quantity that governs harm, since false positives suppress legitimate journalism, and it is exactly the quantity most sensitive to the prior shift that accompanies deployment\. We therefore recommend reporting average precision together with the operating point actually used, and treating calibration as a first\-class concern rather than a post\-hoc adjustment\[[44](https://arxiv.org/html/2609.25006#bib.bib56),[36](https://arxiv.org/html/2609.25006#bib.bib57),[18](https://arxiv.org/html/2609.25006#bib.bib58)\]\.

### VIII\-DRecommended Diagnostics

The probes used here are cheap, a few minutes of CPU time each, and we suggest them as routine reporting for any study using this corpus or one built by pairing publisher sets:

1. 1\.Metadata\-only baseline\.Train on non\-textual fields alone\. If it solves the task, textual results on the same feature set are uninterpretable\.
2. 2\.Small\-sample baseline\.Report performance at11–2%2\\%of the training data\. Near\-ceiling performance there indicates a shallow signal\.
3. 3\.Duplicate audit\.Report the fraction of test documents occurring verbatim in training, before de\-duplication\.
4. 4\.Topic\- or source\-disjoint split\.Report alongside the random split\. Our results indicate this is a far more sensitive stress test than a temporal split for this corpus\.
5. 5\.Prior\-controlled reporting\.When protocols change the class prior, report average precision and prior\-matchedF1F\_\{1\}so that calibration effects are not mistaken for lost discrimination\.
6. 6\.Do not trust in\-domain validation as a shift warning\.Our transformer’s validationF1F\_\{1\}was0\.99950\.9995—its highest across protocols—in the very condition where testF1F\_\{1\}was0\.25920\.2592\. A validation split drawn from the training pool, which is all that is available without target labels, provides no signal about this failure\. Report a shifted evaluation or report nothing about generalisation\.

## IXLimitations

We state the boundaries of these conclusions explicitly\.

Training corpus and label mapping\.All models are trained on one corpus, and our conclusions concern ISOT and, by extension, corpora built by pairing disjoint publisher sets\. The cross\-corpus test \(Section[VII\-F](https://arxiv.org/html/2609.25006#S7.SS6)\) uses a single external benchmark, LIAR, with one binarisation of its six\-way labels; a different cut, or a different external corpus such as NELA\-GT\[[38](https://arxiv.org/html/2609.25006#bib.bib5)\], could shift the exact numbers, though the near\-chance transfer is a wide margin that a boundary choice is unlikely to close\. LIAR’s claim\-level format also differs from ISOT’s article\-level format, so the transfer failure conflates a change of source with a change of granularity; disentangling the two would require an article\-level external corpus\.

Instrument capacity\.The primary probe is a bag\-of\-words linear model that cannot represent word order beyond bigrams, negation, or discourse structure\. The DistilBERT control \(Section[VII\-E](https://arxiv.org/html/2609.25006#S7.SS5)\) addresses the obvious objection, but it does not license an unlimited claim\. We show that one widely used pretrained model, under a standard fine\-tuning recipe, exploits the shortcut more rather than less; we do not show that no model or training procedure could learn veracity\-bearing features from this corpus\. Our claim remains asymmetric in that direction\.

Scope of the capacity control\.The transformer result rests on a single architecture, two epochs, and no hyperparameter search, though it is averaged over five seeds \(Table[VII](https://arxiv.org/html/2609.25006#S7.T7)\)\. The topic\-disjoint average precision carries real spread \(0\.8610±0\.05670\.8610\\pm 0\.0567\), so individual magnitudes should be read with that variance in mind; the direction, however, is consistent across all five seeds and across three prior\-robust metrics, and every seed falls below the linear model\. A larger model and a search over learning rate and epoch count would sharpen the estimate further, and a single architecture cannot speak for all transformers\.

Asymmetric preprocessing in the control\.The transformer receives minimally normalised text while the linear model receives alpha\-only text \(Section[IV\-F](https://arxiv.org/html/2609.25006#S4.SS6)\)\. Document sets, splits, and leakage controls are identical, but this surface difference means the two arms are not preprocessed identically\. It favours the transformer, so it cannot explain the transformer’s worse off\-distribution result; it does inflate its in\-distribution advantage by an amount we have not measured\.

GPU nondeterminism\.Unlike the linear pipeline, the transformer runs mixed\-precision CUDA kernels and is therefore reproducible only to approximately three decimal places\.

Confounded shift protocol\.Subject values proxy simultaneously for topic, sub\-publication, and time\-of\-collection, so the topic\-disjoint protocol perturbs several factors at once\. Our prior\-matched analysis removes the class\-prior confound but not this one; the resulting AP drop should be read as the effect of a bundle of provenance\-related shifts rather than of topic alone\. A source\-level split with publisher metadata—absent from this distribution—would be cleaner\.

Exact\-duplicate detection only\.De\-duplication matches identical cleaned strings\. Near\-duplicates \(syndicated rewrites, partial overlaps\) are not detected, so the19\.37%19\.37\\%contamination figure is a lower bound and the0\.98140\.9814primary score may still be mildly optimistic\.

Residual artifacts\.We remove one agency name\. Other provenance traces \(wire\-service phrasing, content\-management boilerplate\) remain and are, by the argument of Section[VII\-C](https://arxiv.org/html/2609.25006#S7.SS3), precisely what the model uses\. We do not claim to have produced an artifact\-free version of this corpus; we claim the opposite, namely that doing so by token filtering is not feasible\.

English, single period, no external evidence\.The corpus is English and concentrated in 2016–2017 US politics, and the pipeline consults no knowledge source, so it cannot in principle verify a claim\.

Statistical scope\.Bootstrap intervals and McNemar tests quantify uncertainty on a fixed test split; cross\-validation quantifies split sensitivity\. Neither accounts for variation across corpus construction choices, which our ablations address descriptively rather than inferentially\.

## XFuture Work

Article\-level cross\-corpus transfer\.Our LIAR result \(Section[VII\-F](https://arxiv.org/html/2609.25006#S7.SS6)\) changes both source and granularity at once, since LIAR is claim\-level\. Repeating the transfer test on an article\-level external corpus such as NELA\-GT\[[38](https://arxiv.org/html/2609.25006#bib.bib5)\]would isolate the effect of changing source alone, and would test whether the near\-chance collapse we observe is specific to LIAR’s short claims or general to out\-of\-corpus text\.

Source\-controlled data construction\.The identifiability problem motivates corpora containing veracity variation*within*publisher, so thatssis uninformative aboutyyby design\. Retracted or corrected articles from a single outlet are one route; matched\-pair sampling on topic and outlet is another\.

Scaling the capacity control\.Our DistilBERT result \(Section[VII\-E](https://arxiv.org/html/2609.25006#S7.SS5)\) is a single architecture at a single seed\. Extending it to a multi\-seed protocol, to full\-size encoders, and to instruction\-tuned decoder models would establish whether the amplification we observe grows or diminishes with scale—the question that matters for whether current large models are safe to trust on provenance\-confounded benchmarks\.

Debiasing and invariance\.Adversarial removal of source\-predictive directions, reweighting, or invariant\-risk objectives could be evaluated by whether they improve topic\-disjoint AP without degrading in\-distribution performance—the trade\-off our protocol is designed to expose\.

Calibration under shift\.Given that most of the deployedF1F\_\{1\}collapse was calibration, methods that maintain calibration without target labels\[[18](https://arxiv.org/html/2609.25006#bib.bib58)\]deserve evaluation in this setting\.

Beyond exact duplicates\.Near\-duplicate detection \(shingling, MinHash\) would tighten the contamination estimate and yield a cleaner corpus version\.

## XIConclusion

We audited a widely used fake news corpus by holding a transparent TF–IDF and linear\-classifier pipeline fixed and varying the data and the evaluation protocol\. The pipeline reproduces the literature’s near\-ceiling result \(F1F\_\{1\}=0\.9905=0\.9905, 95% CI\[0\.9883,0\.9928\]\[0\.9883,0\.9928\]\), and we then established what that number rests on\. A metadata\-only classifier that never reads the article text solves the benchmark exactly \(F1F\_\{1\}=1\.000=1\.000\)\. Removing that field, a newswire source tag present in99\.21%99\.21\\%of real articles, and duplicates that contaminate19\.37%19\.37\\%of a naive test split costs only1\.211\.21F1F\_\{1\}points, because the residual signal is diffuse editorial style: nine representation variants span0\.620\.62points,543543documents recover95\.1%95\.1\\%of full\-data performance, and deleting the1,0001\{,\}000highest\-weight unigrams leavesF1F\_\{1\}at0\.92630\.9263\. That signal does not transfer\. Under a topic\-disjoint protocol average precision falls from0\.99950\.9995to0\.94750\.9475and deployedF1F\_\{1\}to0\.80670\.8067; a prior\-matched analysis attributes most of the latter to class\-prior shift and miscalibration \(0\.94270\.9427\) while confirming a genuine5\.25\.2\-point loss of discrimination\. Temporal transfer is by contrast nearly lossless, which identifies topic\- and source\-disjoint splits—not chronological ones—as the informative stress test for this corpus\.

A capacity control settles where the limitation lies\. Fine\-tuned DistilBERT, trained on identical documents and splits, is the stronger classifier in\-distribution \(F1F\_\{1\}=0\.9993=0\.9993, average precision1\.00001\.0000\) and the weaker one under topic shift on every prior\-robust metric, losing12\.912\.9points of average precision against the linear model’s5\.25\.2and reaching prior\-matchedF1F\_\{1\}of0\.68860\.6886against0\.94270\.9427\. Capacity sufficient to represent veracity\-bearing content was instead spent fitting the provenance signature more sharply\. Notably, its validationF1F\_\{1\}peaked at0\.99950\.9995in precisely the condition where testF1F\_\{1\}fell to0\.25920\.2592, so standard in\-domain model selection gives no warning of the failure\. The cross\-corpus test is the final word: transferred to the LIAR benchmark, all three ISOT\-trained models rank at near\-chance \(ROC\-AUC0\.540\.54–0\.570\.57\) and none beats a majority\-class predictor, even though every model exceeds0\.990\.99AUC within ISOT\. A benchmark score of0\.990\.99on this corpus therefore certifies almost no transferable ability to judge veracity\.

Our contribution is measurement rather than modelling, and its limits are correspondingly specific: one corpus, a shift protocol that perturbs several provenance factors together, and a capacity control at one architecture and one seed \(Section[IX](https://arxiv.org/html/2609.25006#S9)\)\. Within those limits the practical implication is concrete\. Reported accuracy on this benchmark should be read as a measurement of source and topic separability, and studies using it should accompany their headline number with the metadata\-only, small\-sample, duplicate\-audit, and topic\-disjoint diagnostics of Section[VIII](https://arxiv.org/html/2609.25006#S8), each of which costs minutes of CPU time\. For practitioners, the operative finding is that a system scoring0\.990\.99in\-distribution can mispredict roughly30%30\\%of its positive decisions once the topic mix changes—a failure mode invisible to threshold\-free metrics and to random\-split evaluation alike\. All code, seeds, and derived results are released to make each of these checks a matter of re\-execution rather than reimplementation\.

## Appendix AHyperparameters

Table[XI](https://arxiv.org/html/2609.25006#A1.T11)lists every setting required to reproduce the reported numbers\. Values are not tuned: defaults were fixed before the experiments to avoid an implicit search that the single fixed test split could not support\. A consequence is that reported figures are not upper bounds; the ablations of Table[V](https://arxiv.org/html/2609.25006#S7.T5)show the achievable range is narrow in any case\.

TABLE XI:Complete hyperparameter listing\.ComponentParameterValueTF–IDFngram\_range\(1,2\)\(1,2\)min\_df5max\_df0\.9max\_features50,000sublinear\_tftruestop\_wordsEnglish \(built\-in list\)strip\_accentsunicodeLogistic regressionCC, solver,max\_iter1\.01\.0, liblinear, 1000Multinomial NBα\\alpha0\.10\.1LinearSVCCC, loss1\.01\.0, squared hingePassive–aggressiveCC,max\_iter1\.01\.0, 1000DistilBERTcheckpointdistilbert\-base\-uncasedmax sequence length256 \(512 in thesensitivity run\)batch size, epochs16, 2learning rate, warmup2×10−52\\times 10^\{\-5\}, 10% linearprecisionfp16 autocastselectionbest per\-epoch validationF1F\_\{1\}Protocolsplit70/10/2070/10/20, stratifiedseedσ\\sigma42min\. document length20 charactersselection metricvalidationF1F\_\{1\}Statisticsbootstrap resamples2000prior\-match repeatsMM20
## Appendix BLearning\-Curve and Deletion\-Probe Values

Numerical values underlying Figure[4](https://arxiv.org/html/2609.25006#S7.F4)\(logistic\-regression probe, primary configuration, random protocol\), as training documents→\\rightarrowtestF1F\_\{1\}:135→0\.6802135\\rightarrow 0\.6802;271→0\.8704271\\rightarrow 0\.8704;543→0\.9334543\\rightarrow 0\.9334;1,358→0\.93701\{,\}358\\rightarrow 0\.9370;2,717→0\.95792\{,\}717\\rightarrow 0\.9579;6,794→0\.97146\{,\}794\\rightarrow 0\.9714;13,588→0\.976513\{,\}588\\rightarrow 0\.9765;27,177→0\.981427\{,\}177\\rightarrow 0\.9814\.

Values underlying Figure[5](https://arxiv.org/html/2609.25006#S7.F5), as unigrams removedKK→\\rightarrowtestF1F\_\{1\}:0→0\.98140\\rightarrow 0\.9814;10→0\.975810\\rightarrow 0\.9758;50→0\.966250\\rightarrow 0\.9662;100→0\.9600100\\rightarrow 0\.9600;500→0\.9383500\\rightarrow 0\.9383;1000→0\.92631000\\rightarrow 0\.9263\. The twenty highest\-weight unigrams, removed first, are:*said, video, washington, image, just, featured, read, gop, wednesday, watch, tuesday, obama, america, thursday, hillary, getty, like, friday, republican, minister*\.

## Appendix CCorpus Composition

Figure[11](https://arxiv.org/html/2609.25006#A3.F11)shows class counts in the raw distribution and the token\-length distribution after normalisation\. The length distributions overlap substantially, so length alone is a weak cue; the bimodality visible in the real class reflects the mixture of short wire briefs and full reports\.

![Refer to caption](https://arxiv.org/html/2609.25006v1/x10.png)Figure 11:Raw class counts \(left\) and token\-length distributions after normalisation, clipped at the 98th percentile for legibility \(right\)\.
## Appendix DReproducibility Checklist

- •
- •Data\.Not redistributed; obtained from\[[4](https://arxiv.org/html/2609.25006#bib.bib3)\]\. Construction from the raw files is fully specified by Algorithm[1](https://arxiv.org/html/2609.25006#alg1)\.
- •Environment\.Exact versions pinned \(Section[VI](https://arxiv.org/html/2609.25006#S6)\); no GPU required\.
- •Randomness\.Single seedσ=42\\sigma=42for splits and models; deterministic solver and frequency\-ordered vocabulary\. Two independent runs of the linear pipeline produced bit\-identical metrics; the DistilBERT control is seeded identically but, running mixed\-precision CUDA kernels, reproduces to approximately three decimal places\.
- •Split alignment\.The transformer script asserts that its split cardinalities equal those of the linear runs before training, so the capacity comparison cannot silently drift out of alignment\.
- •Protocol integrity\.Model selection on validation only; the test split is evaluated once per condition; the vectoriser is fitted on training data only\.
- •Negative control\.Label\-permutation run yields test ROC\-AUC=0\.4974=0\.4974\.
- •Uncertainty\.Bootstrap 95% intervals \(B=2000B\{=\}2000\), five\-fold cross\-validation, McNemar exact tests, andM=20M\{=\}20prior\-matched resamples\.
- •Artifacts\.Machine\-readable results \(results\.json,shift\_analysis\.json,transformer\.json,transformer\_multiseed\.json,crosscorpus\.json\) accompany the code, and all reported values are drawn from them\.
- •Compute\.Linear study≈12\\approx 12minutes on one CPU; transformer control≈32\\approx 32minutes on one RTX 4050 \(three protocols at6\.56\.5,5\.85\.8and6\.26\.2minutes, plus13\.013\.0minutes for the 512\-token replication\); five\-seed protocol a further≈60\\approx 60minutes; cross\-corpus evaluation under one minute plus the LIAR download\.

## Appendix EUse of AI Tools

Consistent with arXiv policy, we disclose tool use\. Modelling, data analysis, and figures use the standard scientific\-Python stack cited in Section[VI](https://arxiv.org/html/2609.25006#S6); no generative model is part of the method, the data pipeline, or the results\. During manuscript preparation, an AI coding assistant was used to help implement the experiment scripts and to help draft and edit prose\. All experiments were executed by the author on the author’s hardware; every reported number is produced by the released code and independently checked against the machine\-readable result files by a verification script also included in the repository\. The author reviewed all code and text and takes full responsibility for the contents of this paper, including any errors\.

## References

- \[1\]H\. Ahmed, I\. Traore, and S\. Saad\(2017\)Detection of online fake news using N\-gram analysis and machine learning techniques\.InIntelligent, Secure, and Dependable Systems in Distributed and Cloud Environments \(ISDDC\),Lecture Notes in Computer Science, Vol\.10618,pp\. 127–138\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1),[§I](https://arxiv.org/html/2609.25006#S1.p2.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p2.1)\.
- \[2\]H\. Ahmed, I\. Traore, and S\. Saad\(2018\)Detecting opinion spams and fake news using text classification\.Security and Privacy1\(1\),pp\. e9\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1),[§I](https://arxiv.org/html/2609.25006#S1.p2.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p2.1),[§VII\-A](https://arxiv.org/html/2609.25006#S7.SS1.p1.13)\.
- \[3\]H\. Allcott and M\. Gentzkow\(2017\)Social media and fake news in the 2016 election\.Journal of Economic Perspectives31\(2\),pp\. 211–236\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1)\.
- \[4\]C\. Bisaillon\(2020\)Fake and real news dataset\.Note:Kaggle[https://www\.kaggle\.com/datasets/clmentbisaillon/fake\-and\-real\-news\-dataset](https://www.kaggle.com/datasets/clmentbisaillon/fake-and-real-news-dataset)Cited by:[2nd item](https://arxiv.org/html/2609.25006#A4.I1.i2.p1.1),[§I](https://arxiv.org/html/2609.25006#S1.p2.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1),[§VI](https://arxiv.org/html/2609.25006#S6.p1.6)\.
- \[5\]L\. Bozarth and C\. Budak\(2020\)Toward a better performance evaluation framework for fake news classification\.InProceedings of the International AAAI Conference on Web and Social Media \(ICWSM\),pp\. 60–71\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-C](https://arxiv.org/html/2609.25006#S2.SS3.p1.1)\.
- \[6\]C\. Cortes and V\. Vapnik\(1995\)Support\-vector networks\.Machine Learning20\(3\),pp\. 273–297\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p4.1)\.
- \[7\]K\. Crammer, O\. Dekel, J\. Keshet, S\. Shalev\-Shwartz, and Y\. Singer\(2006\)Online passive\-aggressive algorithms\.Journal of Machine Learning Research7,pp\. 551–585\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p5.2)\.
- \[8\]J\. Demšar\(2006\)Statistical comparisons of classifiers over multiple data sets\.Journal of Machine Learning Research7,pp\. 1–30\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p4.4)\.
- \[9\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\-HLT\),pp\. 4171–4186\.Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1),[§IV\-F](https://arxiv.org/html/2609.25006#S4.SS6.p1.1)\.
- \[10\]T\. G\. Dietterich\(1998\)Approximate statistical tests for comparing supervised classification learning algorithms\.Neural Computation10\(7\),pp\. 1895–1923\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p4.4)\.
- \[11\]P\. Domingos and M\. Pazzani\(1997\)On the optimality of the simple Bayesian classifier under zero\-one loss\.Machine Learning29\(2–3\),pp\. 103–130\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p3.2)\.
- \[12\]R\. Dror, G\. Baumer, S\. Shlomov, and R\. Reichart\(2018\)The hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 1383–1392\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p4.4)\.
- \[13\]B\. Efron and R\. J\. Tibshirani\(1993\)An introduction to the bootstrap\.Chapman and Hall\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p4.4)\.
- \[14\]R\. Fan, K\. Chang, C\. Hsieh, X\. Wang, and C\. Lin\(2008\)LIBLINEAR: a library for large linear classification\.Journal of Machine Learning Research9,pp\. 1871–1874\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p2.4),[§V\-A](https://arxiv.org/html/2609.25006#S5.SS1.p1.11)\.
- \[15\]M\. Gardner, Y\. Artzi, V\. Basmov, J\. Berant, B\. Bogin, S\. Chen, P\. Dasigi, D\. Dua, Y\. Elazar, A\. Gottumukkala, N\. Gupta, H\. Hajishirzi, G\. Ilharco, D\. Khashabi, K\. Lin, J\. Liu, N\. F\. Liu, P\. Mulcaire, Q\. Ning, S\. Singh, N\. A\. Smith, S\. Subramanian, R\. Tsarfaty, E\. Wallace, A\. Zhang, and B\. Zhou\(2020\)Evaluating models’ local decision boundaries via contrast sets\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1307–1323\.Cited by:[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1)\.
- \[16\]R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. Wichmann\(2020\)Shortcut learning in deep neural networks\.Nature Machine Intelligence2\(11\),pp\. 665–673\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1)\.
- \[17\]K\. Gorman and S\. Bedrick\(2019\)We need to talk about standard splits\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 2786–2791\.Cited by:[§II\-C](https://arxiv.org/html/2609.25006#S2.SS3.p1.1),[§VI](https://arxiv.org/html/2609.25006#S6.p4.4)\.
- \[18\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \(ICML\),pp\. 1321–1330\.Cited by:[§X](https://arxiv.org/html/2609.25006#S10.p5.1),[§VIII\-C](https://arxiv.org/html/2609.25006#S8.SS3.p1.4)\.
- \[19\]S\. Gururangan, S\. Swayamdipta, O\. Levy, R\. Schwartz, S\. R\. Bowman, and N\. A\. Smith\(2018\)Annotation artifacts in natural language inference data\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\-HLT\),pp\. 107–112\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1),[§VIII\-B](https://arxiv.org/html/2609.25006#S8.SS2.p2.3)\.
- \[20\]J\. A\. Hanley and B\. J\. McNeil\(1982\)The meaning and use of the area under a receiver operating characteristic \(ROC\) curve\.Radiology143\(1\),pp\. 29–36\.Cited by:[§IV\-G](https://arxiv.org/html/2609.25006#S4.SS7.p1.2)\.
- \[21\]C\. R\. Harris, K\. J\. Millman, S\. J\. van der Walt, R\. Gommers, P\. Virtanen, D\. Cournapeau, E\. Wieser, J\. Taylor, S\. Berg, N\. J\. Smith, R\. Kern, M\. Picus, S\. Hoyer, M\. H\. van Kerkwijk, M\. Brett, A\. Haldane, J\. F\. del Río, M\. Wiebe, P\. Peterson, P\. Gérard\-Marchant, K\. Sheppard, T\. Reddy, W\. Weckesser, H\. Abbasi, C\. Gohlke, and T\. E\. Oliphant\(2020\)Array programming with NumPy\.Nature585,pp\. 357–362\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p2.1)\.
- \[22\]B\. D\. Horne and S\. Adalı\(2017\)This just in: fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news\.InProceedings of the International AAAI Conference on Web and Social Media \(ICWSM\), Workshop on News and Public Opinion,Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.
- \[23\]J\. D\. Hunter\(2007\)Matplotlib: a 2D graphics environment\.Computing in Science and Engineering9\(3\),pp\. 90–95\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p2.1)\.
- \[24\]T\. Joachims\(1998\)Text categorization with support vector machines: learning with many relevant features\.InProceedings of the 10th European Conference on Machine Learning \(ECML\),pp\. 137–142\.Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1),[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p4.1)\.
- \[25\]A\. Joulin, E\. Grave, P\. Bojanowski, and T\. Mikolov\(2017\)Bag of tricks for efficient text classification\.InProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),pp\. 427–431\.Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1)\.
- \[26\]R\. K\. Kaliyar, A\. Goswami, and P\. Narang\(2021\)FakeBERT: fake news detection in social media with a BERT\-based deep learning approach\.Multimedia Tools and Applications80\(8\),pp\. 11765–11788\.Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p2.1),[§VIII\-B](https://arxiv.org/html/2609.25006#S8.SS2.p1.4)\.
- \[27\]S\. Kapoor and A\. Narayanan\(2023\)Leakage and the reproducibility crisis in machine\-learning\-based science\.Patterns4\(9\),pp\. 100804\.Cited by:[§II\-C](https://arxiv.org/html/2609.25006#S2.SS3.p1.1)\.
- \[28\]S\. Kaufman, S\. Rosset, C\. Perlich, and O\. Stitelman\(2012\)Leakage in data mining: formulation, detection, and avoidance\.ACM Transactions on Knowledge Discovery from Data6\(4\),pp\. 1–21\.Cited by:[§II\-C](https://arxiv.org/html/2609.25006#S2.SS3.p1.1)\.
- \[29\]J\. Y\. Khan, Md\. T\. I\. Khondaker, S\. Afroz, G\. Uddin, and A\. Iqbal\(2021\)A benchmark study of machine learning models for online fake news detection\.Machine Learning with Applications4,pp\. 100032\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1),[§I](https://arxiv.org/html/2609.25006#S1.p2.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p2.1),[§VII\-A](https://arxiv.org/html/2609.25006#S7.SS1.p1.13),[§VIII\-B](https://arxiv.org/html/2609.25006#S8.SS2.p1.4)\.
- \[30\]P\. W\. Koh, S\. Sagawa, H\. Marklund, S\. M\. Xie, M\. Zhang, A\. Balsubramani, W\. Hu, M\. Yasunaga, R\. L\. Phillips, I\. Gao, T\. Lee, E\. David, I\. Stavness, W\. Guo, B\. A\. Earnshaw, I\. S\. Haque, S\. Beery, J\. Leskovec, A\. Kundaje, E\. Pierson, S\. Levine, C\. Finn, and P\. Liang\(2021\)WILDS: a benchmark of in\-the\-wild distribution shifts\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),pp\. 5637–5664\.Cited by:[§IV\-E](https://arxiv.org/html/2609.25006#S4.SS5.p1.1)\.
- \[31\]A\. McCallum and K\. Nigam\(1998\)A comparison of event models for naive Bayes text classification\.InAAAI\-98 Workshop on Learning for Text Categorization,pp\. 41–48\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p3.1)\.
- \[32\]T\. McCoy, E\. Pavlick, and T\. Linzen\(2019\)Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 3428–3448\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1)\.
- \[33\]W\. McKinney\(2010\)Data structures for statistical computing in Python\.InProceedings of the 9th Python in Science Conference \(SciPy\),pp\. 56–61\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p2.1)\.
- \[34\]Q\. McNemar\(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p4.4)\.
- \[35\]T\. Mikolov, K\. Chen, G\. Corrado, and J\. Dean\(2013\)Efficient estimation of word representations in vector space\.Note:arXiv:1301\.3781Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1)\.
- \[36\]A\. Niculescu\-Mizil and R\. Caruana\(2005\)Predicting good probabilities with supervised learning\.InProceedings of the 22nd International Conference on Machine Learning \(ICML\),pp\. 625–632\.Cited by:[§VIII\-C](https://arxiv.org/html/2609.25006#S8.SS3.p1.4)\.
- \[37\]T\. Niven and H\. Kao\(2019\)Probing neural network comprehension of natural language arguments\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4658–4664\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1)\.
- \[38\]J\. Nørregaard, B\. D\. Horne, and S\. Adalı\(2019\)NELA\-GT\-2018: a large multi\-labelled news dataset for the study of misinformation in news articles\.InProceedings of the International AAAI Conference on Web and Social Media \(ICWSM\),pp\. 630–638\.Cited by:[§X](https://arxiv.org/html/2609.25006#S10.p1.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1),[§IX](https://arxiv.org/html/2609.25006#S9.p2.1)\.
- \[39\]R\. Oshikawa, J\. Qian, and W\. Y\. Wang\(2020\)A survey on natural language processing for fake news detection\.InProceedings of the 12th Language Resources and Evaluation Conference \(LREC\),pp\. 6086–6093\.Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.
- \[40\]F\. Pedregosa, G\. Varoquaux, A\. Gramfort, V\. Michel, B\. Thirion, O\. Grisel, M\. Blondel, P\. Prettenhofer, R\. Weiss, V\. Dubourg, J\. Vanderplas, A\. Passos, D\. Cournapeau, M\. Brucher, M\. Perrot, and É\. Duchesnay\(2011\)Scikit\-learn: machine learning in Python\.Journal of Machine Learning Research12,pp\. 2825–2830\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p2.1)\.
- \[41\]J\. Pennington, R\. Socher, and C\. D\. Manning\(2014\)GloVe: global vectors for word representation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1532–1543\.Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1)\.
- \[42\]V\. Pérez\-Rosas, B\. Kleinberg, A\. Lefevre, and R\. Mihalcea\(2018\)Automatic detection of fake news\.InProceedings of the 27th International Conference on Computational Linguistics \(COLING\),pp\. 3391–3401\.Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.
- \[43\]J\. Pineau, P\. Vincent\-Lamarre, K\. Sinha, V\. Larivière, A\. Beygelzimer, F\. d’Alché\-Buc, E\. Fox, and H\. Larochelle\(2021\)Improving reproducibility in machine learning research: a report from the NeurIPS 2019 reproducibility program\.Journal of Machine Learning Research22,pp\. 1–20\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p3.1)\.
- \[44\]J\. C\. Platt\(1999\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.InAdvances in Large Margin Classifiers,pp\. 61–74\.Cited by:[§VIII\-C](https://arxiv.org/html/2609.25006#S8.SS3.p1.4)\.
- \[45\]A\. Poliak, J\. Naradowsky, A\. Haldar, R\. Rudinger, and B\. Van Durme\(2018\)Hypothesis only baselines in natural language inference\.InProceedings of the Seventh Joint Conference on Lexical and Computational Semantics \(\*SEM\),pp\. 180–191\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1),[§VIII\-B](https://arxiv.org/html/2609.25006#S8.SS2.p2.3)\.
- \[46\]J\. Quiñonero\-Candela, M\. Sugiyama, A\. Schwaighofer, and N\. D\. Lawrence \(Eds\.\)\(2009\)Dataset shift in machine learning\.MIT Press\.Cited by:[§IV\-E](https://arxiv.org/html/2609.25006#S4.SS5.p1.1)\.
- \[47\]N\. Reimers and I\. Gurevych\(2017\)Reporting score distributions makes a difference: performance study of LSTM\-networks for sequence tagging\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 338–348\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p3.1)\.
- \[48\]J\. D\. Rennie, L\. Shih, J\. Teevan, and D\. R\. Karger\(2003\)Tackling the poor assumptions of naive Bayes text classifiers\.InProceedings of the 20th International Conference on Machine Learning \(ICML\),pp\. 616–623\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25006#S4.SS4.p3.2)\.
- \[49\]M\. T\. Ribeiro, S\. Singh, and C\. Guestrin\(2016\)“Why should I trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining \(KDD\),pp\. 1135–1144\.Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p1.3)\.
- \[50\]M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh\(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 4902–4912\.Cited by:[§II\-B](https://arxiv.org/html/2609.25006#S2.SS2.p1.1)\.
- \[51\]G\. Salton and C\. Buckley\(1988\)Term\-weighting approaches in automatic text retrieval\.Information Processing and Management24\(5\),pp\. 513–523\.Cited by:[§IV\-C](https://arxiv.org/html/2609.25006#S4.SS3.p1.3)\.
- \[52\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.Note:arXiv:1910\.01108Cited by:[§IV\-F](https://arxiv.org/html/2609.25006#S4.SS6.p1.1)\.
- \[53\]T\. Schuster, R\. Schuster, D\. J\. Shah, and R\. Barzilay\(2020\)The limitations of stylometry for detecting machine\-generated fake news\.Computational Linguistics46\(2\),pp\. 499–510\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1),[§II\-C](https://arxiv.org/html/2609.25006#S2.SS3.p1.1),[§VIII\-B](https://arxiv.org/html/2609.25006#S8.SS2.p2.3)\.
- \[54\]K\. Shu, A\. Sliva, S\. Wang, J\. Tang, and H\. Liu\(2017\)Fake news detection on social media: a data mining perspective\.ACM SIGKDD Explorations Newsletter19\(1\),pp\. 22–36\.Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.
- \[55\]K\. Spärck Jones\(1972\)A statistical interpretation of term specificity and its application in retrieval\.Journal of Documentation28\(1\),pp\. 11–21\.Cited by:[§IV\-C](https://arxiv.org/html/2609.25006#S4.SS3.p1.3)\.
- \[56\]J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal\(2018\)FEVER: a large\-scale dataset for fact extraction and VERification\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\-HLT\),pp\. 809–819\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.
- \[57\]A\. Torralba and A\. A\. Efros\(2011\)Unbiased look at dataset bias\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 1521–1528\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p3.1)\.
- \[58\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1),[§IV\-F](https://arxiv.org/html/2609.25006#S4.SS6.p1.1)\.
- \[59\]P\. Virtanen, R\. Gommers, T\. E\. Oliphant, M\. Haberland, T\. Reddy, D\. Cournapeau, E\. Burovski, P\. Peterson, W\. Weckesser, J\. Bright, S\. J\. van der Walt, M\. Brett, J\. Wilson, K\. J\. Millman, N\. Mayorov, A\. R\. J\. Nelson, E\. Jones, R\. Kern, E\. Larson, C\. J\. Carey, İ\. Polat, Y\. Feng, E\. W\. Moore, J\. VanderPlas, D\. Laxalde, J\. Perktold, R\. Cimrman, I\. Henriksen, E\. A\. Quintero, C\. R\. Harris, A\. M\. Archibald, A\. H\. Ribeiro, F\. Pedregosa, and P\. van Mulbregt\(2020\)SciPy 1\.0: fundamental algorithms for scientific computing in Python\.Nature Methods17,pp\. 261–272\.Cited by:[§VI](https://arxiv.org/html/2609.25006#S6.p2.1)\.
- \[60\]S\. Vosoughi, D\. Roy, and S\. Aral\(2018\)The spread of true and false news online\.Science359\(6380\),pp\. 1146–1151\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1)\.
- \[61\]S\. Wang and C\. D\. Manning\(2012\)Baselines and bigrams: simple, good sentiment and topic classification\.InProceedings of the 50th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 90–94\.Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1)\.
- \[62\]W\. Y\. Wang\(2017\)“Liar, liar pants on fire”: a new benchmark dataset for fake news detection\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 422–426\.Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1),[§VI](https://arxiv.org/html/2609.25006#S6.p1.6)\.
- \[63\]R\. Zellers, A\. Holtzman, H\. Rashkin, Y\. Bisk, A\. Farhadi, F\. Roesner, and Y\. Choi\(2019\)Defending against neural fake news\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.
- \[64\]X\. Zhang, J\. Zhao, and Y\. LeCun\(2015\)Character\-level convolutional networks for text classification\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§IV\-A](https://arxiv.org/html/2609.25006#S4.SS1.p2.1)\.
- \[65\]X\. Zhou and R\. Zafarani\(2020\)A survey of fake news: fundamental theories, detection methods, and opportunities\.ACM Computing Surveys53\(5\),pp\. 1–40\.Cited by:[§I](https://arxiv.org/html/2609.25006#S1.p1.1),[§II\-A](https://arxiv.org/html/2609.25006#S2.SS1.p1.1)\.