Benchmarking LLM Competence on Logical Inference over Probability Operators

# Benchmarking LLM Competence on Logical Inference over Probability Operators
Source: [https://arxiv.org/html/2607.27405](https://arxiv.org/html/2607.27405)
Jack Greff Haverford College Alvin Grissom II Haverford College

###### Abstract

Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural\-language expressions of uncertainty are necessary for not only everyday conversations but also for high\-stakes domains such as medicine and law\. While large language models are increasingly evaluated on logical reasoning tasks, disentangling principled, symbolic reasoning from clever surface\-level pattern matching is fraught with difficulty\. We introduce a benchmark for reasoning over probability operators—inference over sentences with gradable epistemic modals \(e\.g\.,probably,might,must\) containing 14,320 procedurally\-generated English prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content\. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No\. We summarize this with a competence floor: the worse of a model’s accuracy on Yes\-correct and No\-correct items\. Only 9 of 29 models exceed random chance\. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the prompts, finding biases across every axis\.

## 1Introduction

Epistemic modals \(EMs\) express a state of knowledge, uncertainty, credence, or belief, and such modals are ubiquitous in human language\(Teller,[1972](https://arxiv.org/html/2607.27405#bib.bib42)\)\. In English, epistemic modality \(EM\) is facilitated largely by modal verbs such asmust,might,may, and adverbs such asprobably, and it pervades both everyday and specialist language\. The sentencesBob might be home earlyandBob will possibly be home early, for example, express that there is some chance, not necessarily a high chance, that Bob will arrive home early, while the epistemic mood of the sentenceBob must be home earlyexpresses certainty that he will be\. While modal logics address the subset of cases expressing necessity and possibility—e\.g\.,mustandmight—gradable epistemic modality also includes cases expressingdegreesof certainty, known asprobability operators\(Yalcin,[2010](https://arxiv.org/html/2607.27405#bib.bib3)\)\.

In our work, we build upon theoretical work that studies inference rules over natural\-language sentences containing probability operatorsYalcin \([2010](https://arxiv.org/html/2607.27405#bib.bib3)\)to benchmark and analyze the capabilities of LLMs in correctly performing zero\-shot logicalinferencewith a large set of procedurally\-generated EM sentences\. EM has received little attention in modern computational linguistics, though earlier work artificial intelligence explored modal and epistemic logics for the integration of uncertain informationMoore \([1981](https://arxiv.org/html/2607.27405#bib.bib39)\), which is a fundamental problem of AI, broadly defined\.

While there exists philosophy and linguistics literature on the semantics of epistemic modals and inference over them \(e\.g\.,Yalcin \([2010](https://arxiv.org/html/2607.27405#bib.bib3)\); Kratzer \([2012](https://arxiv.org/html/2607.27405#bib.bib5)\), there is less work on their analysis in NLP\.Hollidayet al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib6)\)take an important step in this direction, probing LLMs on modal reasoning withmightandmust, representable by modal logic\. They find basic errors and logically inconsistent judgments across related inference types, echoing concerns in other areas about the plausible\-but\-invalid inferences to which LLMs are prone\.

This echoes similar work which finds that LLMs’ accuracy on syllogisms depends heavily on recognizing specific token patterns, with changes to names, entities, or quantifiers causing predictable performance shiftsJianget al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib9)\), which we also examine\. EM reasoning competence is particularly relevant given that EM inference is central to nearly all high\-stakes, natural\-language scenarios in which LLMs—in these contexts, often colloquially called AI—are likely to be deployed: a clinical decision\-support system reading a note that a treatment “will probably work” or a person “probably has a disease” must synthesize premises under uncertainty before ranking options; a system triaging legal documents must distinguish between “might have known” and “probably knew”, since the two carry different evidentiary weight; and a financial pipeline that treats “the market will probably recover” as equivalent to “the market will certainly recover” mischaracterizes risk\. As LLMs are embedded in decision loops in medical, legal, and financial domains, their ability to correctly handle inferences withprobably,might, andmustbecomes a practical concern, not only a theoretical one\.

Our benchmark covers fifteen templates, encoding thirteen distinct inference patterns \(10 valid, 3 invalid\) with 14,320 prompts\. We vary the wording for logically equivalent inference patterns to check whether the model is changing its inference based on tokens instead of the underlying logic: we use five question forms \(Is it correct that…?,Is it true that…?,Does it follow that…?, etc\.\), multiple negation strategies \(prefix,not, proposition\-level\), and surface content \(names from seven nationality groups, five activity descriptions\)\. A genuine reasoner will produce the same answer across all of these, irrespective of lexical variation\. We evaluate 29 models; most show a fixed bias toward one answer, a systematic preference forYesorNothat holds across inference types rather than tracking what each inference licenses\.

Our primary contributions are: \(1\) a dataset and benchmark, tested across a variety of models, for inference over probability operators that varies question form, polarity, and content \(verb and noun phrases\) for fixed logical forms, disentangling models’ answer bias and pattern\-matching from principled reasoning \(§[4](https://arxiv.org/html/2607.27405#S4)\); \(2\) strong evidence that models answer from a fixed answer bias rather than the logical inference \(only 9/29 models exceed a random baseline\), and that this finding is negation\-independent \(§[4\.1](https://arxiv.org/html/2607.27405#S4.SS1)\); and \(3\) a comparison of negated sentences and non\-negated sentences, revealing accuracy gaps of up to 64 percentage points on semantically identical questions, identifying negation strategy as a confound in benchmark design \(§[5\.1\.2](https://arxiv.org/html/2607.27405#S5.SS1.SSS2)\)\.

## 2Related Work

To provide theoretical context for the task and situate our work within prior literature on evaluating LLM reasoning, we review related work\.

### 2\.1Modal and Probabilistic Reasoning in LLMs

Prior work on the semantics of epistemic modals provides the theoretical grounding for the inference templates on which we evaluate models\.Kratzer \([1991](https://arxiv.org/html/2607.27405#bib.bib4),[2012](https://arxiv.org/html/2607.27405#bib.bib5)\)establishes the standard quantificational framework;Lassiter \([2011](https://arxiv.org/html/2607.27405#bib.bib1),[2017](https://arxiv.org/html/2607.27405#bib.bib2)\)proposes the probabilistic alternative in whichprobablyϕ\\phientails thatp​\(ϕ\)p\(\\phi\)is high,mightϕ\\phithatp​\(ϕ\)p\(\\phi\)is non\-trivial, andmustϕ\\phithatp​\(ϕ\)p\(\\phi\)is near\-maximal; andYalcin \([2010](https://arxiv.org/html/2607.27405#bib.bib3)\)’s work on probability operators catalogues valid and invalid inference patterns involving epistemic comparatives\.

Hollidayet al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib6)\)test 29 LLMs on 20 inference patterns using conditionals and the modal auxiliariesmightandmust\. They find that all models commit basic fallacies and that even the strongest model in their study \(GPT\-4, closed source\) exhibits logically inconsistent judgments across related patterns \(e\.g\., accepting Modus Tollens withmustbut rejecting it withmight, which is logically contradictory\)\. Their templates focus on modal auxiliaries \(might,must\) with conditionals, while ours, following semantics work byYalcin \([2010](https://arxiv.org/html/2607.27405#bib.bib3)\), focuses on the adverbprobably, examining LLM inference patterns specific to graded epistemic reasoning \(e\.g\., distribution over conjunction, conditional\-to\-comparative, and conjunction fallacy\)\.

Liet al\.\([2025](https://arxiv.org/html/2607.27405#bib.bib44)\)furthermore provide LLMs with short narratives containing factual information and queries the LLMs with prompts containing modal auxiliaries may/might vs\. must/have to as well as with attitude verbs \(know,believe,doubt\)\. They note that accuracy is affected by prompt format\. Our work is more reductive in its focus on simple templates rather than complex narratives\.

Imannezhadet al\.\([2026](https://arxiv.org/html/2607.27405#bib.bib7)\)examine GPT\-5’s probabilistic reasoning on conjunction and disjunction fallacies and binary complementarity violations using matched human participant profiles, arguing that GPT\-5 appears to have a consistent internal probability model \(which our results contravene\) that align with quantum\-probabilistic models and exceed that of humans\.

They elicit numeric probability ratings under a single fixed prompt format and find the judgments internally coherent\. In contrast, we hold the inference fixed and vary the question format, finding that binary judgments shift according to the formulation\. Our panel is predominantly open\-weight for replicability, with a small number of frontier closed models included as reference points; we do not evaluate the commercial GPT\-5 model examined by Imannezhad et al\., the closest model in our panel being GPT\-5\.4\-mini\. Additionally, rather than compare with human judgments, our work focuses on the inherent correctness of LLM inferences\.

Macmillan\-Scott and Musolesi \([2024](https://arxiv.org/html/2607.27405#bib.bib8)\)tests seven LLMs on cognitive bias tasks from the psychology literature, finding that LLMs are irrational, but that their irrationality does not reflect human patterns: when models give incorrect answers, they are incorrect in different ways than humans\. They also found significant inconsistency in responses, consistent with our finding that much of what looks like reasoning on these tasks is surface\-driven answer behavior rather than proposition\-tracking\.

### 2\.2Surface Sensitivity and Token Bias

LLM performance shifts with surface\-form changes that should not affect the answer\.Jianget al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib9)\)describe this as “token bias” and introduce a hypothesis\-testing framework using McNemar’s test\(McNemar,[1947](https://arxiv.org/html/2607.27405#bib.bib28)\)on matched pairs\. They show that on conjunction fallacies and syllogistic problems, changing names, entities, or quantifiers while holding the logic constant produces predictable performance shifts, indicating reliance on superficial patterns rather than reasoning\. Similarly,Binz and Schulz \([2023](https://arxiv.org/html/2607.27405#bib.bib10)\)find that GPT\-3 \(which, being closed source, is no longer available\) solved canonical cognitive psychology vignettes “similarly or better than human subjects” but that “small perturbations to vignette\-based tasks can lead GPT\-3 vastly astray\.” Other work highlights that even meaning\-preservingformattingchanges \(separator characters, label styles\) produce accuracy swings of up to 76 percentage points\(Sclaret al\.,[2024](https://arxiv.org/html/2607.27405#bib.bib11)\), echoing similar issues documented in machine translation\(Shiet al\.,[2022](https://arxiv.org/html/2607.27405#bib.bib41)\)\. Furthermore, LLMs have a bias toward accepting false presuppositions, making them prone to inferences based on implicit misinformation\(Ermakovaet al\.,[2026](https://arxiv.org/html/2607.27405#bib.bib12)\)\.Sharmaet al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib30)\)directly challenges claims of LLM and large reasoning model “reasoning”, demonstrating that the former outperform the latter at low problem complexity but that both collapse at higher problem complexities\.

In terms of perturbations, our work—which focuses on a particular, but fundamental, corner of syllogistic reasoning—differs from prior work in several respects\. First, perturbations in prior work change what the problem talks about \(names, entities, quantifiers\) or how the text looks on the page \(separators, label styles\)\. We also do this but also change how the question is asked, swapping the metalinguistic wrapper \(Is it true that…,Does it follow that…\) and the way a question’s negation is expressed, a dimension of surface sensitivity closer to the logical structure itself\. We also decompose accuracy into orthogonal components \(the answer bias and polarity sensitivity\), going beyond binary detection of sensitivity to measurement of how much each factor explains\.

### 2\.3Response Bias and Acquiescence

Tjuatjaet al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib13)\)investigate whether LLMs exhibit human\-like response biases \(acquiescence, response order, opinion floating\)\. Testing nine models on 2,578 question pairs, they find that LLMs generallydo notreflect human\-like bias patterns and that RLHF reduces sensitivity to bias\-inducing modifications while increasing sensitivity to non\-bias perturbations\.Braun \([2025](https://arxiv.org/html/2607.27405#bib.bib14)\)extended this to 37,975 question variations phrased survey\-style \(“do you agree…”, “don’t you agree…”\) over classification tasks on legal\-domain texts in English, German, and Polish, finding that LLMs display a bias toward answeringNoin English, the opposite of human acquiescence bias\. This aligns with our results The direction of the answer bias \(yes vs\. no\) is model\-dependent rather than universal, with some models strongly yes\-biased and others strongly no\-biased\.

Our baseline takes the minimum of the two answer\-conditioned accuracies rather than their mean because the mean alone obscures the bias: mean accuracy can be high when one answer class is near ceiling and the other near floor\. The benchmark’s validity\-by\-negation structure is what makes this minimum informative, providing an absolute baseline under which a maximally biased LLM scores 0 regardless of the valid/invalid mix\. We defer the full construction to the discussion of accuracy metrics \(§[4\.1](https://arxiv.org/html/2607.27405#S4.SS1)\)\.

A third evaluation framework treats answer\-label priors as a calibration problem\.Zhaoet al\.\([2021](https://arxiv.org/html/2607.27405#bib.bib23)\)show that few\-shot classification with language models is skewed by majority\-label, recency, and common\-token biases, estimated the model’s prior from a content\-free input;Zhaoet al\.\([2021](https://arxiv.org/html/2607.27405#bib.bib23)\)divide it out at inference, with large accuracy gains;Feiet al\.\([2023](https://arxiv.org/html/2607.27405#bib.bib25)\)refined the prior estimate with in\-domain words, andZhouet al\.\([2024](https://arxiv.org/html/2607.27405#bib.bib26)\)with the mean prediction over a test batch\.Holtzmanet al\.\([2021](https://arxiv.org/html/2607.27405#bib.bib24)\)traced related distortions to probability mass splitting across surface forms of the same answer\. These results were established on models from 2021 to 2024; our panel suggests the phenomenon has not aged out, with current instruction\-tuned and reasoning models still carrying extreme answer priors\. Our answer bias is a label bias on a Yes/No space, and this literature suggests such priors are often removable by calibration\. Where that work treats the prior as a nuisance to be removed so that accuracy improves, we instead report it directly through the competence floor, which remains diagnostic whether or not the prior could be calibrated away\.

### 2\.4Negation Processing

Truonget al\.\([2023](https://arxiv.org/html/2607.27405#bib.bib15)\)evaluate GPT\-neo, GPT\-3, and InstructGPT on six negation benchmarks, finding three key limitations: insensitivity to the presence of negation, inability to capture its lexical semantics, and failure to reason under negation\. They also find that negation shows flat or inverse scaling \(larger models aremoreinsensitive\), a pattern alleviated only by instruction fine\-tuning\.García\-Ferreroet al\.\([2023](https://arxiv.org/html/2607.27405#bib.bib17)\)introduced a large negation benchmark of approximately 400,000 sentences, confirming that LLMs rely on superficial cues and that fine\-tuning improves but does not fully resolve the problem\.Soet al\.\([2025](https://arxiv.org/html/2607.27405#bib.bib18)\)establish a typology distinguishing morphological negation \(un\-,in\-\) from syntactic negation \(not\), which directly corresponds to the contrast we test\.Elkins and Chun \([2026](https://arxiv.org/html/2607.27405#bib.bib19)\)audit negation sensitivity in ethical decisions, finding that open\-source models endorse prohibited actions 77% of the time under simple negation and 100% under compound negation\.

Our benchmark contains prompts with no negation at all, some expecting Yes and some expecting No, so the core contrast does not depend on negation\. On top of that, we implement negation in several ways, asking whether a model that accepts a valid inference \(or rejects an invalid one\) in the affirmative form can still do so when the question is negated\. Implementing it became a source of findings in its own right: writing the same negation as a prefix or with the wordnotproduces dramatically different accuracy on semantically identical content \(§[5\.1\.2](https://arxiv.org/html/2607.27405#S5.SS1.SSS2)\)\.

## 3Benchmark Design

In this section, we describe the overall design of our benchmarks, including the inference templates and their logical forms\.

### 3\.1Inference Templates

Our work is theoretically motivated by prior work on EM semantics which characterizes valid inferences\. In particular, we create prompts based on probability operators analyzed byYalcin \([2010](https://arxiv.org/html/2607.27405#bib.bib3)\), crafted as binary questions with a known correct response\.

Thirteen inference templates compose the benchmark of EM expressions; for exposition, we group the templates into three groups based on validity and complexity\. Eleven inference rules have a single template; two \(distribution over conjunction and conjunctivitis; §[5\.2](https://arxiv.org/html/2607.27405#S5.SS2)\) each have two surface\-form templates\. Altogether, we have fifteen\.

Table[1](https://arxiv.org/html/2607.27405#S3.T1)shows each pattern with a representative prompt from the dataset with the logical form described in terms of probability and modal operators for necessity \(□\\square\) and possibility \(◆\\lozenge\)\. We use⪰\\succeqto denote “is at least as likely as“, i\.e\.,A⪰B⇔p​\(A\)≥p​\(B\)A\\succeq B\\iff p\(A\)\\geq p\(B\)\. For brevity, we also useP​r​\(A\)Pr\(A\)to denote “probably A” andp​\(A\)p\(A\)for a numerical probability, i\.e\.,P​r​\(A\)⇔p​\(A\)≥0\.5Pr\(A\)\\iff p\(A\)\\geq 0\.5under typical probability assumptions\.

#### 3\.1\.1Template Descriptions

##### Simple valid inferences \(5 templates\)

These test straightforward relationships between epistemic operators\.

- ProbablytoMight: ifϕ\\phiis probable, thenϕ\\phiis possible\.
- MusttoProbably: ifϕ\\phiis necessary, thenϕ\\phiis probable\.
- ProbablytoNot Probably Not: ifϕ\\phiis probable, then it is not probable that¬ϕ\\neg\\phi\.
- Positive Form Transfer\(PFT\) andComplement Transfer\(CT\) test probability preservation under reformulation\. Positive Form Transferψ\\psiis at least as likely asϕ\\phiprobablyϕ\\phi\.probablyψ\\psi∴\\therefore Complement Transferψ\\psiis at least as likely asϕ\\phiϕ\\phiis at least as likely as¬ϕ\\lnot\\phi\.ψ\\psiis at least as likely as¬ϕ\\lnot\\phi∴\\therefore

##### Multi\-step valid inferences \(5 templates\)

Chancy Modus Ponens and its contrapositive variant Chancy Modus Tollens require combining premises or comparing probabilities\.

- Chancy Modus Ponensifϕ\\phithenψ\\psiprobablyϕ\\phi\.probablyψ\\psi∴\\therefore Chancy Modus Tollensifϕ\\phithenψ\\psi¬\\lnotprobablyψ\\psi\.¬\\lnotprobablyϕ\\phi∴\\therefore
- Distribution over conjunction: if probably \(ϕ∧ψ\\phi\\land\\psi\), then probablyϕ\\phiand probablyψ\\psi\.
- Conditional to Comparative: if probably \(ϕ→ψ\\phi\\to\\psi\), thenψ\\psiis at least as probable asϕ\\phi\.
- Chancy Disjunction Introduction: if probablyϕ\\phi, then probably \(ϕ∨ψ\\phi\\lor\\psi\)\.

##### Invalid inferences \(3 templates\)

These measure correct rejection of fallacious reasoning\.

- ConjunctivitisKyburg Jr \([1970](https://arxiv.org/html/2607.27405#bib.bib27)\)is a fascinating conjunction fallacy discussed in the philosophical literature\. The typical logical assumption of closure under conjunction does not hold for probability operators\. Closure under conjunction holds when the truth ofψ\\psiandϕ\\phiindependently implies the truth ofψ∧ϕ\\psi\\land\\phi\. It holds for non EM predicates but not for the probability operator\. I\.e\.,probably​\(ψ\),probably​\(ϕ\)／⟹probably​\(ψ∧ϕ\)\\text\{probably\}\(\\psi\),\\text\{probably\}\(\\phi\)\\mathchoice\{\\mathrel\{\\hbox to0\.0pt\{\\kern 3\.75pt\\kern\-5\.27776pt$\\displaystyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 3\.75pt\\kern\-5\.27776pt$\\textstyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 2\.625pt\\kern\-4\.45831pt$\\scriptstyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 1\.875pt\\kern\-3\.95834pt$\\scriptscriptstyle\\not$\\hss\}\{\\implies\}\}\}\\text\{probably\}\(\\psi\\land\\phi\), a fact that can be illustrated by the lottery paradox, whereby we have a collection of propositions of probability at least0\.50\.5but less than11, whose conjunction probability necessarily decreases with each new proposition\. Since this is a known fallacy discussed in the literature, one might expect that LLMs would be less prone to fall for it\.
- MighttoProbablyinference: possibility does not entail probability\.◇​ϕ／⟹□​ϕ\\Diamond\\phi\\mathchoice\{\\mathrel\{\\hbox to0\.0pt\{\\kern 3\.75pt\\kern\-5\.27776pt$\\displaystyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 3\.75pt\\kern\-5\.27776pt$\\textstyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 2\.625pt\\kern\-4\.45831pt$\\scriptstyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 1\.875pt\\kern\-3\.95834pt$\\scriptscriptstyle\\not$\\hss\}\{\\implies\}\}\}\\Box\\phi\.
- ProbablytoCertain: probability does not entail certainty\. probablyϕ／⟹□​ϕ\\phi\\mathchoice\{\\mathrel\{\\hbox to0\.0pt\{\\kern 3\.75pt\\kern\-5\.27776pt$\\displaystyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 3\.75pt\\kern\-5\.27776pt$\\textstyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 2\.625pt\\kern\-4\.45831pt$\\scriptstyle\\not$\\hss\}\{\\implies\}\}\}\{\\mathrel\{\\hbox to0\.0pt\{\\kern 1\.875pt\\kern\-3\.95834pt$\\scriptscriptstyle\\not$\\hss\}\{\\implies\}\}\}\\Box\\phi\.

Table 1:Inference templates with formal schema and example prompts \(truthquestion form, affirmative; the negated condition replacesIs it true thatwithIs it false thatand flips the correct answer\)\. In the affirmative examples shown the correct answer isYesfor the valid templates andNofor the invalid ones, and negation reverses both\.P​r​\(ϕ\)Pr\(\\phi\):probable;M​\(ϕ\)M\(\\phi\):possible;□​ϕ\\Box\\phi:certain;ϕ≽ψ\\phi\\succcurlyeq\\psi:at least as likely\. Names and activities vary\.

### 3\.2Question Form

Table 2:The five English question forms and their negated versions, plus the twonot\-prefixed forms, shown onMusttoProbably\(premise: “Savir must be at the party”\)\. The repeated conclusion “it is probable that Savir will be at the party” is abbreviated as …\. Correct answers flip with polarity: Yes affirmative, No negated\.Question formAffirmative \(Yes\)Negated \(No\)truthIs it true that …?Is it false that …?correctIs it correct that …?Is it incorrect that …?validIs it valid to conclude that …?Is it invalid to conclude that …?followDoes it follow that …?Does it not follow that …?directIs it probable that Savir will be at the party?Is it not probable that …not\-correct—Is it not correct that …?not\-valid—Is it not valid to conclude that …?
directcarries no metalinguistic wrapper, negating the proposition itself \(probabletonot probable\)\.

We present questions in bothaffirmativeandnegatedforms to test consistency under negation\. In the affirmative condition, we ask whether the conclusion holds \(correct answer:Yesfor valid;Nofor invalid\)\. In the negated condition, we ask whether the same conclusion does not hold \(correct answer:Nofor valid,Yesfor invalid\)\.

Five metalinguistic question forms vary how the question is asked\.correct,truth, andvalidnegate by switching to the negative adjective \(correct to incorrect, true to false, valid to invalid\)\.followinserts the wordnotchanging, for example,does it followtodoes it not follow\)\.directnegates the proposition itself\.

An additional pair of question forms,Is it not correct that …?andIs it not valid to conclude that …?, replaces the negative adjective with the wordnoton the same adjective, enabling direct comparison of negation strategies on identical semantic content \(§[5\.1\.2](https://arxiv.org/html/2607.27405#S5.SS1.SSS2)\)\. In our analysis, we discuss the perlocutionary force of such questions in light of known issues of LLM sycophancy \(§[5\.1\.2](https://arxiv.org/html/2607.27405#S5.SS1.SSS2)\)\. Specifically, for our question templates, introducing negation can pragmatically encourage the model to agree with the questioner\. Asking,Is it not probable thatϕ\\phi?can, under a common interpretation, sound like the answershouldbe theYes\. We introduce clauses such asIs it true thatfor affirmative andIs it false thatfor the negation to enforce some neutral symmetry in the question surface form without such pragmatic force skewing the results\.

### 3\.3Demographic Bias Controls

Natural language models are known to be prone to spurious biases\. To control for this, names are drawn from seven nationality groups \(Indian, Russian, Japanese, African, German, French, American\) plus abstract letter variables, balanced by gender \(common names for men or women\)\. Activities include five semantically similar scenarios\. These function as robustness checks: a model that reasons about the logic should show minimal accuracy variation across these controls\.

### 3\.4Models Evaluated

We evaluate 29 models in English spanning five size\-graded families \(Gemma 3: 270M–27B; Gemma 4: E2B–31B; Qwen 3: 0\.6B–32B; DeepSeek\-R1: 7B–32B; Llama 3: 3B, 8B\) plus frontier models \(Claude Opus 4\.7, Claude Sonnet 4\.6, GPT\-oss 120B, GPT\-oss 20, GPT\-5\.4\-mini\), Granite 3\.2:8B, Mistral:7b, and the Arabic\-oriented Aya\-Expanse 8B and 32B\.

We show all models the same 80 surface prompts per template, and we run each model once per prompt at temperatureτ=0\\tau=0All main analyses pool across templates and question forms\.

##### Prompting setup

We query open\-weight models zero\-shot, and the first answer token is parsed asYes,No, or classified asUncertainotherwise\. Two choices bear on interpretation\. First, to keep the evaluation strictly zero\-shot, we disable chain\-of\-thought wherever the model permits it: Qwen 3 and DeepSeek\-R1 are run with thinking turned off, and GPT\-oss is set to its lowest available reasoning level\. The reported numbers therefore reflect direct\-answer behavior\. Second, Qwen 3:4B is queried with a constrained Yes/No output format, because unconstrained generation produced a non\-compliance rate as high as 96%\. Constraining the output still lets the model choose either answer, so it does not manufacture the answer bias we report\.

### 3\.5Metrics

We score each model on a 2×\\times2 grid, crossing inference validity \(valid/invalid\) with question negation \(affirmative/negated\); we refer to its four cells throughout\. This crossing determines the correct answer in each cell\. We use abbreviations to refer to the accuracy for each intersection of conditions\. V\.AFF, for example, refers to avalidtemplate with a question phrased in the affirmative, while I\.NEG refers to aninvalidtemplate with a negated question\. The correct answer isYesin two cells \(V\.AffandI\.Neg\) andNoin the other two \(V\.NegandI\.Aff\)\.

Overall accuracy is the fraction of correct responses \(with uncertain counted as incorrect\), computed over whatever set of prompts a table covers\. Since valid templates outnumber invalid ones, weighting the four validity×\\timesnegation cells equally instead pulls every strong model toward 0\.5, by up to about 9 points; supplementary Table[14](https://arxiv.org/html/2607.27405#A1.T14)reports accuracy under five weighting schemes so the sensitivity to this choice is visible\.

To separate the model’s surface\-level bias from principled proposition\-tracking, we measure the yes/no bias\. We do this by averaging the model accuracy, respectively, on questions for which the answers should be affirmative \(Yes\) and on questions for which the answers should be negative \(No\)\. For each, we have two cases, the valid and invalid conditions\.111These are true positive rates, but we avoid using the term since “’positive” is confusing when discussing yes/no questions that can be negated\.LetAcc\(Yes\)=12​\(V\.Aff\+I\.Neg\)\\text\{Acc\(Yes\)\}=\\frac\{1\}\{2\}\(\\text\{\{V\.Aff\}\}\+\\text\{\{I\.Neg\}\}\)andAcc\(No\)=12​\(V\.Neg\+I\.Aff\)\\text\{Acc\(No\)\}=\\frac\{1\}\{2\}\(\\text\{\{V\.Neg\}\}\+\\text\{\{I\.Aff\}\}\)\.

The signed gap is the bias towardYesorNos:

Bias=Acc\(Yes\)−Acc\(No\),\\text\{Bias\}=\\text\{Acc\(Yes\)\}\-\\text\{Acc\(No\)\},\(1\)If a model always guessesYes, the bias will be 1; if it always guessesNo, it will be \-1; and a random baseline has a bias of 0222We have more valid templates than invalid ones; averaging them prevents in this way prevents the valid category from dominating the bias and associated metrics\. However, the average still provides useful information\.\.

We also track polarity sensitivity \(PS\), the model’s accuracy drop under negation:

PS=\(V\.Aff\+I\.Aff\)−\(V\.Neg\+I\.Neg\)2\\displaystyle=\\frac\{\(\\text\{\{V\.Aff\}\}\+\\text\{\{I\.Aff\}\}\)\-\(\\text\{\{V\.Neg\}\}\+\\text\{\{I\.Neg\}\}\)\}\{2\}\(2\)=\[Acc\(non\-negated questions\)−Acc\(negated questions\)\]2\\displaystyle=\\frac\{\\left\[\\text\{Acc\(non\-negated questions\)\}\-\\text\{Acc\(negated questions\)\}\\right\]\}\{2\}\(3\)Bias is positive when the model favorsYes; PS is positive when negation hurts\. Bias is an indirect way of measuring a model’scompetence, i\.e\., a model’s actual reasoning ability\. To measure competence, we take the worse of the two accuracy measurements, the*floor*:

Floor=min⁡\(Acc\(Yes\),Acc\(No\)\)\.\\text\{Floor\}=\\min\\\!\\big\(\\text\{Acc\(Yes\)\},\\,\\text\{Acc\(No\)\}\\big\)\.\(4\)Intuitively, this related metric concisely describes how poorly the model performs on its worst\-performing question polarity \(those with a correct answer of eitherYesorNo\)\. This is motivated by our observation that models tend to be extremely biased for one answer or the other, irrespective of the internal logic\. A constant responder scores 0 on the floor, a coin\-flipper 0\.5, and a competent reasoner near 1, so the floor has an absolute baseline that overall accuracy lacks\.

## 4Results

We now describe the EM benchmarking results\. While some of the differences in accuracy across conditions may seem small \(but many are not\), given the query rate of LLMs, seemingly small differences in the number of errors can have huge impacts in the aggregate\. If, for example, an LLM is queried 1 billion times, a0\.3%0\.3\\%increase in errors is equivalent to33million more\. Currently, ChatGPT alone receives approximately 2\.5 billion queries per day, and GPT\-4o received approximately 774 billion queries in 2025, which is still only a fraction of Google’s queries, which now integrate LLMsSingh \([2026](https://arxiv.org/html/2607.27405#bib.bib43)\)\.

### 4\.1Overall Accuracy

First, we report the aggregate accuracy across all EM prompts for a bird’s eye view of respective model performance as a prelude to more specific analysis\. Table[4\.1](https://arxiv.org/html/2607.27405#S4.SS1)presents the four cells for all 29 models, using the metrics defined in §[3\.5](https://arxiv.org/html/2607.27405#S3.SS5)\.

Overall accuracy ranges from 20\.2% \(DeepSeek\-R1:7B\) to 81\.8% \(Gemma4:31B\), but this obscures the strategies \(such as they are\) being used\. Qwen3:0\.6B scores 54\.8% overall by nearly always respondingYes\(V\.Aff=0\.997\{\}=0\.997,V\.Neg=0\.160\{\}=0\.160,I\.Aff=0\.030\{\}=0\.030,I\.Neg=0\.878\{\}=0\.878\), while Llama3\.1:8B is balanced \(bias only\+\+0\.03\), yet every cell performs near the random baseline, suggesting that the predictions are themselves arbitrary and disconnected from the prompt\.

Uncertain responses are rare for most models, below 2%, but high for a handful \(e\.g\., DeepSeek\-R1:7b is uncertain 61\.5% of the time\)\. Uncertain rates are relatively flat across conditions \(expected\-Yes 3\.2% vs\. expected\-No 3\.4%\), so counting them as incorrect does not unduly affect the competence floor\. We report rates in Table[4\.1](https://arxiv.org/html/2607.27405#S4.SS1)\.

Table 3:English models, the four cells, sorted by floor; the faded rule separates the nine models that clear the 0\.5 floor from those that do not\. The four columnsV\.Aff,V\.Neg,I\.Aff,I\.Negare per\-cell accuracies, so e\.g\.V\.Aff=\{\}=\{\}Acc\(V\.Aff\)\.Accis overall accuracy\.Uncertain responses count as incorrect\. Bias=Acc\(Yes\)−Acc\(No\)\{\}=\\text\{Acc\(Yes\)\}\-\\text\{Acc\(No\)\}\(signed; positive=\{\}=\{\}yes\-biased\)\. Floor=min⁡\(Acc\(Yes\),Acc\(No\)\)\{\}=\\min\(\\text\{Acc\(Yes\)\},\\text\{Acc\(No\)\}\)and has a random baseline of 0\.5; floors above random chance are inbold\. Per\-condition accuracies below chance areunderlined\.ModelAccUncV\.AffV\.NegI\.AffI\.NegBiasPSFloorgemma4:31b0\.8180\.0%0\.9770\.7610\.9800\.372−\-0\.20\+\+0\.410\.675Sonnet 4\.60\.7510\.0%0\.9940\.6360\.6480\.491\+\+0\.10\+\+0\.260\.642gemma4:12b0\.7640\.0%0\.9670\.7540\.7350\.261−\-0\.13\+\+0\.340\.614qwen3:32b0\.7180\.0%0\.8130\.7080\.8270\.374−\-0\.17\+\+0\.280\.594gemma3:27b0\.6481\.0%0\.9130\.4700\.7120\.334\+\+0\.03\+\+0\.410\.591gpt\-oss:200\.7370\.0%0\.8150\.7400\.8940\.356−\-0\.23\+\+0\.310\.586Opus 4\.70\.7560\.0%0\.8920\.7620\.8700\.248−\-0\.25\+\+0\.380\.570gpt\-oss:120b0\.7240\.0%0\.7270\.8080\.8890\.326−\-0\.32\+\+0\.240\.526gemma3:12b0\.6150\.0%0\.8120\.6200\.4490\.224−\-0\.02\+\+0\.210\.518\\arrayrulecolorblack\!30\\arrayrulecolorblack gemma4:e4b0\.6080\.0%0\.6520\.6550\.6740\.294−\-0\.19\+\+0\.190\.473llama3\.1:8b0\.46314\.3%0\.4450\.4350\.5040\.544\+\+0\.03−\-0\.010\.469gpt\-5\.4\-mini0\.6560\.0%0\.6710\.7100\.8870\.239−\-0\.34\+\+0\.300\.455aya\-expanse:32b0\.6240\.1%0\.9050\.5220\.3110\.440\+\+0\.26\+\+0\.130\.416gemma3:270m0\.5000\.0%0\.6400\.3690\.4560\.512\+\+0\.16\+\+0\.110\.413gemma3:4b0\.5290\.0%0\.6030\.5240\.2830\.589\+\+0\.19−\-0\.110\.403qwen3:8b0\.5990\.0%0\.9250\.4100\.3860\.424\+\+0\.28\+\+0\.240\.398qwen3:14b0\.6050\.0%0\.4720\.8130\.7290\.281−\-0\.39\+\+0\.050\.377aya\-expanse:8b0\.5860\.6%0\.8590\.4670\.2240\.517\+\+0\.34\+\+0\.050\.346mistral:7b0\.5350\.0%0\.8360\.2870\.3100\.604\+\+0\.42\+\+0\.130\.298granite3\.2:8b0\.6070\.0%0\.4060\.8630\.9090\.169−\-0\.60\+\+0\.140\.287qwen3:4b0\.5690\.0%0\.4670\.6990\.9860\.081−\-0\.57\+\+0\.340\.274llama3\.2:3b0\.4671\.5%0\.2340\.6050\.9310\.275−\-0\.51\+\+0\.140\.254gemma4:e2b0\.5901\.7%0\.4500\.7970\.9930\.016−\-0\.66\+\+0\.320\.233deepseek\-r1:32b0\.5562\.4%0\.9530\.0870\.3490\.942\+\+0\.73\+\+0\.140\.218qwen3:1\.7b0\.5171\.5%0\.8350\.2860\.1380\.649\+\+0\.53\+\+0\.020\.212gemma3:1b0\.5370\.2%0\.9670\.1660\.1010\.790\+\+0\.74\+\+0\.060\.133qwen3:0\.6b0\.5480\.0%0\.9970\.1600\.0300\.878\+\+0\.84−\-0\.010\.095deepseek\-r1:14b0\.47710\.6%0\.9050\.0140\.1340\.889\+\+0\.82\+\+0\.070\.074deepseek\-r1:7b0\.20261\.5%0\.4150\.0020\.0000\.357\+\+0\.39\+\+0\.030\.001
![Refer to caption](https://arxiv.org/html/2607.27405v1/x1.png)Figure 1:Competence Floor against answer bias \(\|Bias\|\|\\text\{Bias\}\|\) for all 29 models\. The two are strongly inversely correlated \(r=−0\.82r=\-0\.82,R2=0\.68R^\{2\}=0\.68\): the more a model is biased toward one answer, the lower its floor\. Only nine models clear the 0\.5 random baseline \(dashed\), and the most biased models \(e\.g\. Qwen3:0\.6B\) sit near the floor’s zero\.
##### Few models clear the floor\.

Only 9 of 29 models exceed the random Floor baseline of 0\.5: Gemma4:31B \(0\.675\), Sonnet 4\.6 \(0\.642\), Gemma4:12B \(0\.614\), Qwen3:32B \(0\.594\), Gemma3:27B \(0\.591\), GPT\-oss:20 \(0\.586\), Opus 4\.7 \(0\.570\), GPT\-oss:120B \(0\.526\), and Gemma3:12B \(0\.518\)\. Even these nine clear it only modestly, by about 2 to 18 points above the 0\.5 baseline\. Eleven of the 29 models fall below a floor of 0\.30\. Qwen3:0\.6B is the textbook constant Yes responder: near\-ceiling where Yes is correct \(V\.Aff0\.997,I\.Neg0\.878\), near\-zero where No is \(V\.Neg0\.160,I\.Aff0\.030\), with 54\.8% overall accuracy the average of these extremes\.

##### Competence differences are not only due to negation

It is instructive to contrast the competence floor in affirmative vs\. negated questions\. Recomputing the competence floor from the affirmative cells withmin⁡\(V\.Aff,I\.Aff\)\\min\(\\text\{\{V\.Aff\}\},\\text\{\{I\.Aff\}\}\)achieves this\. This affirmative\-only floor correlatesr=0\.90r=0\.90with the full floor, still only yielding 10 of 29 models above chance, and still leaving the biased models with the worst measures of competence \(Qwen3:0\.6B at 0\.030\)\. Importantly, the answer bias is visible in the affirmative validity contrast alone, so negation is one way to expose it, not necessarily its cause\.

## 5Analysis

### 5\.1Robustness to Surface Variation

A genuine reasoner would return the same answer when only the surface of a prompt changes and the logic is held fixed\. We examine several such surface variations: the question form and how its negation is written \(prefix vs\. the wordnot\), the activity scenario, and the name’s nationality and gender\. The tables here show the models that move most; full per\-model tables are in Appendix[A](https://arxiv.org/html/2607.27405#A1)\.

#### 5\.1\.1Question Form

Among affirmative questions, the five question forms land within about four points of each other \(67\.6–71\.4%\), a much smaller spread than what we see with negation \(29\.2–58\.4%; Table[4](https://arxiv.org/html/2607.27405#S5.T4)\)\. The collapse is confined to the negated wrapper of thefollowform,Does it not follow that …?, which falls to 29\.2%, below the 50% chance level and nearly 20 points under the next\-lowest wrapper \(48\.8%\)\. The other four negated wrappers cluster together: Varying adjectives withIs it false that …?\(58\.4%\),Is it incorrect that …?\(56\.0%\) andIs it invalid to conclude that …?\(48\.8%\), together with the proposition\-levelIs it not probable that …?\(56\.5%\), all sit between 49 and 58%\. Measured within each form as the affirmative−\-negated drop \(polarity sensitivity\), the gap is smallest for thetruthform \(Is it true thatvs\.Is it false that,\+0\.125\+0\.125\) and largest for thefollowform \(Does it follow thatvs\.Does it not follow that,\+0\.384\+0\.384\); thefollowform is the only one whose negated wrapper leaves the shared range\.

The collapse is broad but not uniform: 21 of 29 models score at least 10 points lower ondoes it not followthan on the other negated wrappers, and the strongest models that fall furthest \(Gemma4:31B, Sonnet 4\.6 and Opus 4\.7 each drop 60 or more points\), while the handful that do not show it are near\-random constant responders already poor in all cases\. Tellingly, thedirectform’s negated wrapperIs it not probable that …?also contains the wordnotyet holds at 56\.5%, inside that range, so the difficulty is specific to writing the negation asnotinside the entailment wrapper \(does it*not*follow\), not the wordnotitself\.

Table 4:Per\-question\-form results \(all English models pooled\)\.Aff/Negare affirmative/negated accuracy \(micro\-averaged over all templates and models, uncertain counts as incorrect\);PSis polarity sensitivity \(Aff−\{\}\-\{\}Neg\)\. Sorted by PS\. Under negation only thefollowform’s wrapper,Does it not follow that …?, collapses to near chance; the other four stay in a tight band, includingdirect\(Is it not probable that …?\), which also uses the wordnot\.Per model, the same divergence shows up as a wide swing across forms; the nine floor\-clearing models have question\-form ranges from about 0\.19 to 0\.50 \(Table[5](https://arxiv.org/html/2607.27405#S5.T5)\), each concentrated in thedoes it not followform\.

Table 5:Question\-form sensitivity \(models that move most\)\. Overall accuracy in each of the five question forms \(fraction correct, micro\-averaged over all templates and both the affirmative and negated conditions; uncertain counts as incorrect\);Rangeis the best−\-worst spread\. The remaining 23 models have Range0\.020\.02–0\.320\.32; the smallest belong to committed responders \(Qwen3:1\.7B0\.020\.02, DeepSeek\-R1:7B0\.060\.06\)\. Full per\-model table in Table[A](https://arxiv.org/html/2607.27405#A1)\.To test whether the negation strategy itself drives this collapse, we addednot\-word versions of thecorrectandvalidforms \(Is it not correct that …?,Is it not valid to conclude that …?\), creating matched pairs where the only difference is whether the negation is written as a prefix \(incorrect/invalid\) or with the wordnot\.

#### 5\.1\.2Prefix Negation versusnot\-Negation

Since thenotnegation is pragmatically ambiguous—it has a possible interpretation as pining for agreement with a rhetorical question—while prefix negation is unambiguous, these results provide information about the effect of this pragmatic ambiguity versus an unambiguous control\. Comparing these matched pairs \(prefixincorrect/invalidagainstnot correct/not valid\), the prefix form yields 52\.3% accuracy versus 33\.6% for thenotform, an 18\.7\-point gap, suggesting that the ambiguous form confuses the model\. Table[6](https://arxiv.org/html/2607.27405#S5.T6)presents per\-model results\. Each model contributes 2,400 paired observations per negation style\. The per\-model gaps are far larger: Sonnet drops 61\.1 points, Opus 59\.1, Qwen3:32B 40\.3\. The direction is not uniform: 25 of 29 models perform better with the prefix form, but 4 \(Qwen3:0\.6B, Gemma3:1B, Qwen3:1\.7B, Llama3\.2:3B\) show the reverse\.

Table 6:Negation written as a prefix vs\. with the wordnot, for all 29 models\. Prefix=\{\}=\{\}incorrect/invalid\(negation fused into the adjective\);not=\{\}=\{\}not correct/not valid\(a separate “not”\)\. Both sides are negated questions with the same proposition and gold answer; only the negation surface form differs\.Δ=\\Delta=\{\}Prefix accuracy−\{\}\-\{\}notaccuracy\. The Yes\-rate columns give the model’s percentage of Yes answers under each form\.
#### 5\.1\.3Activity

Replacing the scenario verb phrases \(“party”, “event”, “gathering”, “meeting”, or “celebration”\) attenuates accuracy\. The average per\-model range across the five activities is 3\.4 pts, relatively minor compared to negation and question\-form effects\. Only DeepSeek\-R1:7b moves substantially \(13\.2 pts\)\. As with nationality, this is a refusal pattern, its refusal rate ranging from 46% to 72% across the five activities while accuracy on the items it answers stays near 52%; so the model is sensitive to the verb but expresses that as refusal to answer rather than as different reasoning \(Table[7](https://arxiv.org/html/2607.27405#S5.T7)\)\.

Table 7:Accuracy by activity scenario \(models that move most; uncertain counts as incorrect\)\.Rangeis the spread across the five activities\. The remaining 24 models have Range≤0\.04\{\}\\leq 0\.04\(mean Range=0\.034=0\.034\)\. Full per\-model table in Table[17](https://arxiv.org/html/2607.27405#A1.T17)\.
#### 5\.1\.4Nationality

We find biases are introduced by varying gender, nationality, and activity with every model\. As a control, we also use variablesXXandYYinstead of people’s names to compare against\. Names are drawn from seven nationality groups, and Table[8](https://arxiv.org/html/2607.27405#S5.T8)reports each nationality’s accuracy*relative to*the abstract variable \(X,YX,Y\) baseline\. \(A positive value means the model does better on that nationality than on the variables\.\) For 23 of 29 the largest per\-nationality gap is at most 4\.7 points\. The clear exception is DeepSeek\-R1:7b, which sits 13 to 34 points below variables on every nationality, being worst on Japanese\. It frequently refuses to answer the question, but on the items it actually answers, its accuracy across nationalities spans about 6 points \(51–57%\), against roughly 21 points on the headline accuracy; the gap comes instead from how often it refuses, and that refusal rate itself swings sharply with nationality, from 46% on Indian names to 87% on Japanese\.

Table 8:Accuracy on each nationality group*minus*accuracy on the abstract letter\-variable \(X,YX,Y\) baseline, in percentage points \(models that move most; uncertain counts as incorrect\)\. Positive=\{\}=\{\}higher on that nationality than on XY\. Ind=Indian, Rus=Russian, Jpn=Japanese, Afr=African, Ger=German, Fre=French, Amr=American\.max⁡\|Δ\|\\max\|\\Delta\|is the largest absolute gap\. The remaining 23 models havemax⁡\|Δ\|≤4\.7\\max\|\\Delta\|\\leq 4\.7points\. Full per\-model table in Table[18](https://arxiv.org/html/2607.27405#A1.T18)\.
#### 5\.1\.5Gender

Names are balanced by gender, and Table[9](https://arxiv.org/html/2607.27405#S5.T9)reports each gender’s accuracy relative to the variable \(X,YX,Y\) control\. Women’s and men’s names score within about 3 points of variables for every model except DeepSeek\-R1:7b, which exhibits the same refusal behavior seen for nationality\. The man\-woman name gap itself average 0\.6 percentage points; the largest being DeepSeek\-R1:32b \(\+2\.3\+2\.3toward men’s names\) and GPT\-5\.4\-mini \(\+2\.2\+2\.2toward women’s names\)\.

Table 9:Accuracy on female and male names*minus*accuracy on the variable \(X,YX,Y\) control, in percentage points \(models that move most; uncertain counts as incorrect\)\. Positive=\{\}=\{\}higher thanX,YX,Y; the male−\-female gap is the difference between the two columns\.max⁡\|Δ\|\\max\|\\Delta\|is the larger absolute gap\. The remaining 23 models havemax⁡\|Δ\|≤3\.0\\max\|\\Delta\|\\leq 3\.0points\. Full per\-model table in Table[16](https://arxiv.org/html/2607.27405#A1.T16)\.

### 5\.2Conjunction Variants

We test two conjunction inferences, each written two ways that keep the logic fixed \(Table[10](https://arxiv.org/html/2607.27405#S5.T10)\), with per\-model accuracies in Table[11](https://arxiv.org/html/2607.27405#S5.T11)\. Recall that distribution over conjunction \(DOC\) is valid while conjunctivitis is invalid \(§[3\.1](https://arxiv.org/html/2607.27405#S3.SS1)\)\. By DOC,P​r​\(A∧B\)⟹P​r​\(A\)∧P​r​\(B\)Pr\(A\\land B\)\\implies Pr\(A\)\\land Pr\(B\)\. To test for consistent reasoning, we vary whether we ask the model about the truth ofP​r​\(A\)Pr\(A\)orP​\(B\)P\(B\), since logically both are true\.

On DOC examples, affirmative questions differ by about 5 points on average, and 25 of 29 models stay within 10 points, a large gap for such a simple variation\. For conjunctivitis examples how the two premises are worded matters even more\. We see this by testing two conditions: one with two people engaging in the same activity and one with two people engaging in different activities\. On affirmative items, where the model has to reject the inference, the same\-activity wording \(two people at one activity\) is easier to reject than the dual\-activity wording \(one person at two activities\) by a mean of 12\.8 points \(up to 96 points for Qwen3:14B, and Sonnet drops from 96\.5% to 56\.5%\)\. This gap resides entirely in the reject condition\. On negated items, where the answer isYes, the two wordings yield very close results \(0\.3 points\), so pooling the two conditions dilutes the effect to 6\.6 points, which is why we report it on the affirmative items\.

Table 10:The two conjunction inference types, each shown in two surface realizations \(affirmative condition; the negated condition swapsIs it true thatforIs it false thatand flips the gold answer\)\. The distribution\-over\-conjunction rows differ only in which name is queried; the conjunctivitis rows differ only in whether the two premises describe two people at one activity or one person at two activities\.Table 11:Per\-model accuracy on the two conjunction templates under a wording change that leaves the logic fixed \(affirmative items; uncertain counts as incorrect\)\.Conjunctivitisis invalid \(correct answer No\), so its accuracy is the rejection rate;Samestates the two premises as two people at one activity andDualas one person at two activities\.Distribution over conjunctionis valid \(correct answer Yes\);Name 1andName 2ask about the first or second name in the shared premise\.Δ\\Deltais the accuracy difference between the two wordings\. Rewriting the conjunctivitis premises moves accuracy by a mean of 12\.8 points \(up to 96\), while which name distribution over conjunction asks about moves it by only 4\.9 on average\. Sorted by the conjunctivitisΔ\\Delta\.
### 5\.3Per\-Template Breakdown

The 13 templates greatly vary in accuracy \(42\.8% for Conjunctivitis to 74\.3% for Probably to Might\)\. The key finding is a dissociation \(Table[12](https://arxiv.org/html/2607.27405#S5.T12)\): the answer bias varies less across templates than polarity sensitivity does \(\|Bias\|\|\\text\{Bias\}\|std=0\.056=0\.056versus PS std=0\.126=0\.126, about twice as much\), and it stays high on every template \(0\.42 to 0\.62\)\. The bias is thus a model\-level property carried across all templates, while negation sensitivity is template\-dependent\. Model biases also change direction based on template: on the easiest template \(Probably to Might\) 28 of 29 models are biased towardYes, so nearly all give the same answer\. On the two hardest templates the panel splits roughly in half \(Conjunctivitis 13 yes\-biased vs\. 16 no\-biased; Might to Probably 12 vs\. 17\): the template exerts no shared pull, so which answer a model gives is set mostly by its own bias rather than by the content of the inference\.

Table 12:Per\-template analysis \(all English models pooled\), grouped by validity\.Accis overall accuracy on the template, a micro\-average with uncertain counted as incorrect\.\|Bias\|\|\\textbf\{Bias\}\|is the mean absolute answer bias \(from the yes\-rate\),PSthe mean polarity sensitivity, andAgree\.the fraction of models sharing the majority bias direction\.TypeTemplateAcc\|Bias\|\|\\textbf\{Bias\}\|PSAgree\.ValidChancy Disjunction Introduction\.574\.539\+\+\.09862\.1%Chancy Modus Ponens\.695\.453\+\+\.27672\.4%Chancy Modus Tollens\.547\.526\+\+\.09558\.6%Complement Transfer\.604\.496\+\+\.18365\.5%Conditional to Comparative\.540\.497\+\+\.13462\.1%Distribution over Conjunction\.695\.418\+\+\.35189\.7%Must to Probably\.633\.509\+\+\.17262\.1%Positive Form Transfer\.674\.488\+\+\.02365\.5%Probably to Might\.743\.463\+\+\.41796\.6%Probably to not probably not\.578\.548\+\+\.33579\.3%InvalidConjunctivitis\.428\.619\+\+\.04755\.2%Might to Probably\.447\.608\+\+\.09758\.6%Probably to Certain\.683\.462\+\+\.34279\.3%
### 5\.4Model Scale

Four families provide size variants \(Table[13](https://arxiv.org/html/2607.27405#S5.T13)\)\. The answer bias tends to fall as models grow, most clearly in Qwen 3 \(Spearmanρ=−0\.89\\rho=\-0\.89between\|\|Bias\|\|and size\) and more weakly in Gemma 3 \(ρ=−0\.60\\rho=\-0\.60\) and Gemma 4 \(ρ=−0\.40\\rho=\-0\.40\), but no family is monotonic \(Gemma 3, for instance, spikes at 1B before falling again\)\. DeepSeek\-R1 runs the other way, its bias rising with size \(ρ=\+0\.50\\rho=\+0\.50\)\. Thus scaling lowers the bias on average in three of four families, never cleanly, and not at all in the fourth\. Thefollowquestion form \(the wordnot\) shows little scaling improvement across families, suggesting the difficulty withnotis not resolved by additional parameters\.

Table 13:Scaling within size\-graded model families \(English\)\.Accis overall accuracy with uncertain responses counted as incorrect\.\|Bias\|\|\\textbf\{Bias\}\|is the magnitude of the signed answer bias andPSthe polarity sensitivity, both defined in §[3\.5](https://arxiv.org/html/2607.27405#S3.SS5)\(Bias=Acc\(Yes\)−Acc\(No\)\{\}=\\text\{Acc\(Yes\)\}\-\\text\{Acc\(No\)\}; PS=Acc\(affirmative\)−Acc\(negated\)\{\}=\\text\{Acc\(affirmative\)\}\-\\text\{Acc\(negated\)\}, so positive=\{\}=\{\}better on affirmative\)\.Unc%is the uncertain rate\. Sizes are listed smallest to largest within each family\.![Refer to caption](https://arxiv.org/html/2607.27405v1/x2.png)Figure 2:Effect of scaling on accuracy by answer cell \(valid/invalid×\\timesnegated/non\-negated\) across model families\. Dashed line marks 50% chance\. Scaling does not consistently improve accuracy\.fig:scaling## 6Discussion and Conclusion

Overall, we find widespread biases across multiple dimensions, including name origin, gender, and various perturbations to question formation\. More generally, we find definitive evidence that while some models manage to perform above chance, many do not even reach this low bar, and LLMs do not, in general, make basic EM inferences consistently, making them inappropriate for use in question\-answering settings that require synthesizing epistemic statements to make valid conclusions\.

##### What accuracy can hide

Most models show a fixed answer bias forYesorNobelied by aggregate accuracy\. \(Valid\) reasoning requires both accepting valid inferences and rejecting invalid ones, but we demonstrate conclusively that state of the art models fail to do this\. We suggest that future benchmarks using binary questions to evaluate reasoning also report the competence floor, which is cheap to compute and is not inflated by an answer bias\. While precision and recall also attempt to address this, they are more difficult to interpret in this specific scenario\.

##### Negation is not the main source of answer bias

Two of our probes use negation, and answer biases do not depend on it: the affirmative\-only competence floor correlatesr=0\.90r=0\.90with the full \(affirmative\+negation\) floor, and the most biased models \(Qwen3:0\.6B at 0\.030\) score the worst at both\.

##### What the floor does and does not say

The floor measures whether a model performs above chance on both answer classes\. The invalid templates are uncontroversial: the conjunction fallacy, possibility not entailing probability, and probability not entailing certainty hold under any reasonable semantics\. However, some valid templates rest on stronger commitments\. Conditional\-to\-comparative and probably\-to\-not\-probably\-not depend on specific axioms a competent reasoner could contest, though this is not responsible for the results: recomputing the floor with those two templates leaves the number of models with accuracy above 0\.5 unchanged at 9 of 29 and moves every above\-floor model by at most 0\.03 \(Sonnet 0\.642 to 0\.665, Opus 0\.570 to 0\.567, Qwen3:32B 0\.594 to 0\.606\)\. Restricting further to a conservative core of only uncontestable inferences \(the three invalid templates plus the four most basic valid ones, whereprobablyϕ\\phientailsmightϕ\\phi,mustϕ\\phientailsprobablyϕ\\phi, distribution over conjunction, and chancy disjunction introduction\) tells the same story \(10 of 29 above 0\.5\), and in both restrictions the most biased models remain at the bottom \(DeepSeek\-R1:7B at 0\.00\)\. Biased responders fail the uncontroversial, invalid templates regardless of semantics, so their low floor cannot be attributed to legitimately contested axioms\. An expert or human baseline calibrating how much residual sub\-floor behavior is theory disagreement is the natural next step\.

##### Limitations

Our evaluation is zero\-shot and English\-only, and parses the first answer token rather than a free\-form justification; whether these patterns persist under longer generation or in other languages is left to future work\.

## 7Conclusion

We have introduced a benchmark for inference over probability operators that separates a model’s answer bias from proposition\-tracking, demonstrating conclusively that models consistently fail at zero\-shot reasoning under trivial prompts with an unambiguous correct response\. Most models show a fixed bias towardYesorNo: only 9 of 29 models exceed a competence floor with an absolute random baseline, and the result reproduces from the affirmative validity contrast alone, so it does not appear to depend on negation\. Negation variation exposes a strong bias, where prefix andnotphrasing of the same content differ by up to 64 points\. Taken together, these results strongly suggest that part of what is commonly reported as “reasoning” ability may reflect answer bias and is not the result of a coherent internal inference model\. A competence floor with an absolute baseline is a useful and easy\-to\-calculate complement to accuracy as a summary of logical competence\. Our prompts are simple and formulaic, and all information necessary to answer correctly is provided in the prompts, suggesting that more subtle ways of expressing the same premises would not lead to successful inferences\. Since most inference work limited to English, future work can extend this to other languages\.

## Acknowledgments

Grissom is supported by U\.S\. National Science Foundation grant 2403439\.

## References

- M\. Binz and E\. Schulz \(2023\)Using cognitive psychology to understand GPT\-3\.Proceedings of the National Academy of Sciences120\(6\),pp\. e2218523120\.Cited by:[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- D\. Braun \(2025\)Acquiescence bias in large language models\.arXiv preprint arXiv:2509\.08480\.Cited by:[§2\.3](https://arxiv.org/html/2607.27405#S2.SS3.p1.1)\.
- K\. Elkins and J\. Chun \(2026\)When prohibitions become permissions: auditing negation sensitivity in language models\.arXiv preprint arXiv:2601\.21433\.Cited by:[§2\.4](https://arxiv.org/html/2607.27405#S2.SS4.p1.1)\.
- L\. Ermakova, A\. Firsov, and J\. Kamps \(2026\)Confirmation, framing, and position biases in LLM responses\.InProceedings of the 2026 Conference on Human Information Interaction and Retrieval \(CHIIR ’26\),pp\. 480–484\.Cited by:[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- Y\. Fei, Y\. Hou, Z\. Chen, and A\. Bosselut \(2023\)Mitigating label biases for in\-context learning\.InProceedings of ACL 2023,pp\. 14014–14031\.Cited by:[§2\.3](https://arxiv.org/html/2607.27405#S2.SS3.p3.1)\.
- I\. García\-Ferrero, B\. Altuna, J\. Alvez, I\. Gonzalez\-Dios, and G\. Rigau \(2023\)This is not a dataset: a large negation benchmark to challenge large language models\.InProceedings of EMNLP 2023,pp\. 8596–8615\.Cited by:[§2\.4](https://arxiv.org/html/2607.27405#S2.SS4.p1.1)\.
- W\. H\. Holliday, M\. Mandelkern, and C\. E\. Zhang \(2024\)Conditional and modal reasoning in large language models\.InProceedings of EMNLP 2024,pp\. 3800–3821\.Cited by:[§1](https://arxiv.org/html/2607.27405#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p2.1)\.
- A\. Holtzman, P\. West, V\. Shwartz, Y\. Choi, and L\. Zettlemoyer \(2021\)Surface form competition: why the highest probability answer isn’t always right\.InProceedings of EMNLP 2021,pp\. 7038–7051\.Cited by:[§2\.3](https://arxiv.org/html/2607.27405#S2.SS3.p3.1)\.
- P\. Imannezhad, E\. M\. Pothos, and A\. J\. Wills \(2026\)Divergent patterns of probabilistic reasoning in humans and GPT\-5\.Frontiers in Psychology17,pp\. 1782184\.Cited by:[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p4.1)\.
- B\. Jiang, Y\. Xie, Z\. Hao, X\. Wang, T\. Mallick, W\. J\. Su, C\. J\. Taylor, and D\. Roth \(2024\)A peek into token bias: large language models are not yet genuine reasoners\.InProceedings of EMNLP 2024,pp\. 4722–4756\.Cited by:[§1](https://arxiv.org/html/2607.27405#S1.p4.1),[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- A\. Kratzer \(1991\)Modality\.InSemantics: An International Handbook of Contemporary Research,pp\. 639–650\.Cited by:[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p1.6)\.
- A\. Kratzer \(2012\)Modals and conditionals\.Oxford University Press\.Cited by:[§1](https://arxiv.org/html/2607.27405#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p1.6)\.
- H\. E\. Kyburg Jr \(1970\)Conjunctivitis\.InInduction, acceptance and rational belief,pp\. 55–82\.Cited by:[§3\.1\.1](https://arxiv.org/html/2607.27405#S3.I3.ix1.p1.6)\.
- D\. Lassiter \(2011\)Measurement and modality: the scalar basis of modal semantics\.Ph\.D\. Thesis,New York University\.Cited by:[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p1.6)\.
- D\. Lassiter \(2017\)Graded modality: qualitative and quantitative perspectives\.Oxford University Press\.Cited by:[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p1.6)\.
- M\. Li, M\. Vrazitulis, and D\. Schlangen \(2025\)Representations of fact, fiction and forecast in large language models: epistemics and attitudes\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 27734–27757\.Cited by:[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p3.1)\.
- O\. Macmillan\-Scott and M\. Musolesi \(2024\)\(Ir\)rationality and cognitive biases in large language models\.Royal Society Open Science11\(6\),pp\. 240255\.Cited by:[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p6.1)\.
- Q\. McNemar \(1947\)Note on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- R\. C\. Moore \(1981\)Reasoning about knowledge and action\.InReadings in artificial intelligence,pp\. 473–477\.External Links:[Link](https://www.sciencedirect.com/science/chapter/edited-volume/abs/pii/B9780934613033500374)Cited by:[§1](https://arxiv.org/html/2607.27405#S1.p2.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design\.InProceedings of ICLR 2024,Cited by:[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- M\. Sharma, M\. Tong, T\. Korbak, D\. Duvenaud, A\. Askell, S\. R\. Bowman, N\. Cheng, E\. Durmus, Z\. Hatfield\-Dodds, S\. R\. Johnston, S\. Kravec, T\. Maxwell, S\. McCandlish, K\. Ndousse, O\. Rausch, N\. Schiefer, D\. Yan, M\. Zhang, and E\. Perez \(2024\)Towards understanding sycophancy in language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- R\. Shi, A\. Grissom II, and D\. M\. Trinh \(2022\)Rare but severe neural machine translation errors induced by minimal deletion: an empirical study on Chinese and English\.InProceedings of the 29th International Conference on Computational Linguistics,N\. Calzolari, C\. Huang, H\. Kim, J\. Pustejovsky, L\. Wanner, K\. Choi, P\. Ryu, H\. Chen, L\. Donatelli, H\. Ji, S\. Kurohashi, P\. Paggio, N\. Xue, S\. Kim, Y\. Hahm, Z\. He, T\. K\. Lee, E\. Santus, F\. Bond, and S\. Na \(Eds\.\),Gyeongju, Republic of Korea,pp\. 5175–5180\.External Links:[Link](https://aclanthology.org/2022.coling-1.459/)Cited by:[§2\.2](https://arxiv.org/html/2607.27405#S2.SS2.p1.1)\.
- S\. Singh \(2026\)DamenSage: chatgpt statistics \(july 2026\) – latest active users data\.Note:[https://www\.demandsage\.com/chatgpt\-statistics/](https://www.demandsage.com/chatgpt-statistics/)\[Accessed 13\-07\-2026\]Cited by:[§4](https://arxiv.org/html/2607.27405#S4.p1.2)\.
- Y\. So, G\. Lee, S\. Jung, J\. Lee, J\. Kang, S\. Kim, and J\. Lee \(2025\)Thunder\-NUBench: a benchmark for LLMs’ sentence\-level negation understanding\.arXiv preprint arXiv:2506\.14397\.Cited by:[§2\.4](https://arxiv.org/html/2607.27405#S2.SS4.p1.1)\.
- P\. Teller \(1972\)Epistemic possibility\.Philosophia2\(4\),pp\. 303–320\.Cited by:[§1](https://arxiv.org/html/2607.27405#S1.p1.1)\.
- L\. Tjuatja, V\. Chen, T\. Wu, A\. Talwalkar, and G\. Neubig \(2024\)Do LLMs exhibit human\-like response biases? a case study in survey design\.Transactions of the Association for Computational Linguistics12,pp\. 1011–1026\.Cited by:[§2\.3](https://arxiv.org/html/2607.27405#S2.SS3.p1.1)\.
- T\. H\. Truong, T\. Baldwin, K\. Verspoor, and T\. Cohn \(2023\)Language models are not naysayers: an analysis of language models on negation benchmarks\.InProceedings of \*SEM 2023,pp\. 101–114\.Cited by:[§2\.4](https://arxiv.org/html/2607.27405#S2.SS4.p1.1)\.
- S\. Yalcin \(2010\)Probability operators\.Philosophy Compass5\(11\),pp\. 916–937\.Cited by:[§1](https://arxiv.org/html/2607.27405#S1.p1.1),[§1](https://arxiv.org/html/2607.27405#S1.p2.1),[§1](https://arxiv.org/html/2607.27405#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p1.6),[§2\.1](https://arxiv.org/html/2607.27405#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2607.27405#S3.SS1.p1.1)\.
- Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.InProceedings of ICML 2021,pp\. 12697–12706\.Cited by:[§2\.3](https://arxiv.org/html/2607.27405#S2.SS3.p3.1)\.
- H\. Zhou, X\. Wan, L\. Proleev, D\. Mincu, J\. Chen, K\. Heller, and S\. Roy \(2024\)Batch calibration: rethinking calibration for in\-context learning and prompt engineering\.InProceedings of ICLR 2024,Cited by:[§2\.3](https://arxiv.org/html/2607.27405#S2.SS3.p3.1)\.

## Appendix AAdditional Results

Table 14:Overall accuracy under five pooling schemes \(framesnot correct/not valid/conclude excluded, templates folded to 13, uncertain counts as incorrect\)\. All five score the same responses and differ only in weighting\.microis the headline flat mean over every response\.prompt\-bal,tmpl\-balandframe\-balweight each prompt, template or question form equally\.cat\-avgweights the four validity×\\timesnegation cells equally,\(V\.Aff\+V\.Neg\+I\.Aff\+I\.Neg\)/4\(\\text\{V\.Aff\}\+\\text\{V\.Neg\}\+\\text\{I\.Aff\}\+\\text\{I\.Neg\}\)/4\. Frame\-balancing equals micro by construction and prompt\-balancing is within a fraction of a point of it; only cat\-avg moves the number appreciably, always toward 0\.5, because it stops the abundant easy valid cells from dominating the rare invalid\-negation cell\.Spreadis the max−\-min across the five schemes for each model\. The mean weighting spread is 4\.7 points, and averaged over models cat\-avg sits 3\.2 points below micro\. Sorted by micro\.Table 15:Per\-model question\-form sensitivity, full 29\-model table \(uncertain counts as incorrect\)\. Overall accuracy in each of the five question forms;Rangeis the best−\-worst spread\. Ordered by competence floor; the faded rule separates the nine models that clear the 0\.5 floor from those that do not \(§[5\.1\.1](https://arxiv.org/html/2607.27405#S5.SS1.SSS1)\)\.Table 16:Accuracy on female and male names*minus*accuracy on the letter\-variable \(XY\) baseline, all 29 models, in percentage points \(uncertain counts as incorrect\)\. Positive=\{\}=\{\}higher than XY\.max⁡\|Δ\|\\max\|\\Delta\|is the larger absolute gap\. Sorted bymax⁡\|Δ\|\\max\|\\Delta\|\.
Table 17:Accuracy by activity scenario, all 29 models \(uncertain counts as incorrect\)\.Rangeis the spread across the five activities\. Sorted by Range\.
Table 18:Accuracy on each nationality group*minus*accuracy on the abstract letter\-variable \(XY\) baseline, all 29 models, in percentage points \(uncertain counts as incorrect\)\. Positive=\{\}=\{\}higher on that nationality than on XY\. Ind=Indian, Rus=Russian, Jpn=Japanese, Afr=African, Ger=German, Fre=French, Amr=American\.max⁡\|Δ\|\\max\|\\Delta\|is the largest absolute gap\. Sorted bymax⁡\|Δ\|\\max\|\\Delta\|\.

![[Uncaptioned image]](https://arxiv.org/html/2607.27405v1/x3.png)![[Uncaptioned image]](https://arxiv.org/html/2607.27405v1/x4.png)![[Uncaptioned image]](https://arxiv.org/html/2607.27405v1/x5.png)

Figure 3:Accuracy \(%\) for every \(template×\\timesnegation\) cell by model, shown separately for each of the five question forms\. The same logical content swings from near\-zero to near\-ceiling across question forms, visualizing the question\-form instability summarized in Table[5](https://arxiv.org/html/2607.27405#S5.T5)\.