Hey, guess what: CUES lead to inferences about typicality

# Hey, guess what: CUES lead to inferences about typicality
Source: [https://arxiv.org/html/2610.33239](https://arxiv.org/html/2610.33239)
Sanghyeok ChoiHannah RohdeEmily AllawayAffiliation:University of EdinburghAffiliation:\*Corresponding author:a\.h\.chi@sms\.ed\.ac\.uk

###### Abstract

Throughout life, we accumulate knowledge of how typical certain events are\. We use this knowledge in daily conversation, often interpreting meaning beyond what has been said explicitly\. In this work, we take steps to better understand this meaning recovery, investigating both atypicality and typicality inferences\. First, we construct two corpora,EventTypand the human\-validatedEventTypGold\. We conduct large\-scale human and LLM experiments and find that for humans, threeCUESpredict inferences about typicality:Commonsense Knowledge,UtteranceEmphasis, andSnark\. However, in most experimental settings, we find that LLMs lack the prerequisiteCommonsense Knowledge necessary to accurately assess an event’s general typicality, and therefore struggle to follow these CUES\.

## 1Introduction

In our daily lives, we make decisions based on what we think will and will not happen\. This knowledge about event typicality \(i\.e\., which events are likely or unlikely to happen\) helps us navigate the intricacies of human conversation, interpreting meaning beyond what speakers have literally said[Grice \(1975\)](https://arxiv.org/html/2610.33239#bib.bib11);[Bergen et al\. \(2016\)](https://arxiv.org/html/2610.33239#bib.bib1)\.

In this work, we investigate two types ofevent\-typicality inferences\. These rely on our commonsense knowledge about which events are typical \(andunlikelyto be mentioned when they occur\) and which are atypical \(andlikelyto be mentioned when they occur\)\. Consider[Fig\.1](https://arxiv.org/html/2610.33239#S1.F1)\. We might expect that when someone thanks their boss, they typically mean it \(there is no need to explicitly state this\)\. But we also expect that when a speaker says something, it is informative\. So, when a typical eventisexplicitly mentioned \(left panel\), our expectations of the speaker conflict with our commonsense knowledge \(i\.e\., the speaker appears to be stating something uninformative\)\. To resolve this conflict, we make anatypicality inferencethat, in this context, this event that is generally typical for others is in factatypicalfor Anahita\. Conversely, when an atypical event is explicitly mentioned \(right panel\), this is straightforwardly informative and so we may simply make atypicality inferencethat, in this context, this generally atypical event ismore typicalfor Anahita than we previously believed\.111Although we call both of these phenomenainferences, we only claim that atypicality inferences require pragmatic reasoning\. We consider a typicality inference to be a belief update according to semantics \(see[§A\.3](https://arxiv.org/html/2610.33239#A1.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2610.33239v1/atypicality_example_fig.png)Figure 1:Examples of what can cause an atypicality or typicality inference\. Someone making anatypicalityinference would answer the question on the left withunlikely, whereas if they made atypicalityinference, they would answer the question on the right withlikely\.Previous studies have carried out only limited human experiments on event\-typicality inferences[Kravtchenko and Demberg \(2015\)](https://arxiv.org/html/2610.33239#bib.bib25);[Kravtchenko and Demberg \(2022\)](https://arxiv.org/html/2610.33239#bib.bib26);[Ryzhova et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib39), and even less work has been done on LLMs[Kurch et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib27)\. However, event\-typicality inferences pose important challenges because they rely on both commonsense knowledge and reasoning ability\. For LLMs, investigating such inferences is particularly critical since recent works have shown that LLMs often exhibit shallow and inconsistent commonsense event\-knowledge[Chen et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib3);[Bian et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib2);[Kim and Hwang \(2025\)](https://arxiv.org/html/2610.33239#bib.bib23)\. Therefore, in this paper we aim to expand our understanding of event\-typicality inferences in both humans and LLMs\. We ask:

- •RQ1: Do humans and LLMsshare Commonsense Knowledgeabout event typicality?
- •RQ2: When dohumansmake inferences about event typicality?
- •RQ3: DoLLMsmake inferences about event typicality in the contexts where humans do?

To address these, we createEventTyp, a corpus of 2,354 labeled event tuples, andEventTypGold, a subset of 858 tuples with additional human validation\. Using our corpora, we conduct large\-scale experiments with both humans \(∼900\{\\sim\}900participants\) and LLMs \(10 models and 3 inference methods\)\.

First, our results show that LLMs mostlydo notagree with humans in their commonsense knowledge about event typicality \(RQ1\)\. This has broad implications for LLM usage, since many downstream tasks, such as task planning[Zhao et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib49)and knowledge graph completion[Ji et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib19), rely on LLMs’ commonsense knowledge\.

Second, when examining event\-typicality inferences in humans \(RQ2\), we find that threeCUESinvite them: 1\)Commonsense Knowledge, 2\)UtteranceEmphasis \(e\.g\., “And hey, guess what”\), and 3\)Snark \(i\.e\., when an atypicality inference helps the listener learn something negative about someone\)\.

Finally, when examining LLM behaviour \(RQ3\) we find that while some LLMs make event\-typicality inferences following similar CUES as humans, this is limited primarily to large models with Chain\-of\-Thought prompting\([Wei et al\., 2022](https://arxiv.org/html/2610.33239#bib.bib45)\)\.

Our contributions are:\(i\)We create two new large\-scale, high\-quality corpora for studying event typicality,EventTypandEventTypGold;\(ii\)We conduct large\-scale human experiments and find evidence for three CUES that can lead to inferences about event typicality; and\(iii\)We conduct extensive experiments with LLMs and show that LLMs often lack event commonsense knowledge, which undermines their ability to make event\-typicality inferences\.222[https://github\.com/alisonhc/heygw\-eventtyp](https://github.com/alisonhc/heygw-eventtyp)

## 2Related Work

##### LLMs & commonsense knowledge about events

LLMs have been shown to struggle to reason about relationships between events, whether they are rare/newsworthy[Koupaee et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib24);[Islakoglu and Kalo \(2025\)](https://arxiv.org/html/2610.33239#bib.bib17)or daily life[Sabour et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib40);[Wang et al\. \(2026\)](https://arxiv.org/html/2610.33239#bib.bib44);[Toroghi et al\. \(2026\)](https://arxiv.org/html/2610.33239#bib.bib43)\. Prior work has examined LLMs’ ability to differentiate event arguments\([Pedinotti et al\., 2021](https://arxiv.org/html/2610.33239#bib.bib35)\), as well as how probable events are on different levels of granularity: e\.g\. plausible vs\. implausible[Jain et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib18);[Joshi et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib21)or a spectrum from probable to inconceivable[Kauf et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib22);[Hu et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib15);[Lepori et al\. \(2026\)](https://arxiv.org/html/2610.33239#bib.bib29)\. To our knowledge, no work has investigated the more fine\-grained task of distinguishing the typicality of two events that areboth highly plausiblebutstill different in typicality\.

##### Human pragmatic reasoning about typicality

Pragmatic reasoning can be characterized in terms of comprehenders’ expectations for what cooperative speakers are likely to do, including the expectation for speakers to provide an appropriate Quantity of information[Grice \(1975\)](https://arxiv.org/html/2610.33239#bib.bib11)\. Apparent violations of this expectation \(saying something ambiguous or otherwise underinformative\) can invite inferences\.

With atypicality inferences, a speaker’s apparent uninformativity \(the mention of some typical event given the context\) contrasts with the listener’s belief in the speaker’s cooperativity\. But the speakerisinformative if the reported event is actually*atypical*given the context \(see[AppendixA](https://arxiv.org/html/2610.33239#A1)for more details\)\. Existing experiments on such inferences[Kravtchenko and Demberg \(2015\)](https://arxiv.org/html/2610.33239#bib.bib25);[Kravtchenko and Demberg \(2022\)](https://arxiv.org/html/2610.33239#bib.bib26)have asked participants to read dialogue snippets and rate the mentioned events’ typicality, revealing atypicality inferences for typical events but no inferences for atypical events\. Subsequent work confirms the inferences reflect Quantity\-driven reasoning[Ryzhova et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib39)\.

##### LLMs & pragmatic reasoning

The limited work on LLMs’ pragmatic reasoning has shown that a wide gap persists between humans and LLMs[Hu et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib14);[Yerukola et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib48);[Ma et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib31)\. For event\-typicality inferences in particular,[Kurch et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib27)conducted prompting experiments with two LLM families on a set of 24 utterance tuples, which is relatively limited in scale compared to our experiments, and observed no atypicality inferences\.

DefinitionExampleWritten byValidated byLLMHumanLLMHumanContextThe first event utteranceAnahita thanks her bossTypicFollowUpEvent utterance stating something thatAnahita means it2,0622921,2261,128is generallytypicalgiven Context\.AtypicFollowUpEvent utterance stating something thatAnahita is in tears1,67967502,354is generallyatypicalgiven Context\.

Table 1:Example \(Context, TypicFollowUp, AtypicFollowUp\) tuple fromEventTyp, along with the number of items per\-type written and validated by an LLM and a human\. Note that we also count the human\-written and human\-validated data from[Kravtchenko and Demberg \(2015\)](https://arxiv.org/html/2610.33239#bib.bib25)\. The total number of tuples inEventTypis2,354\.

## 3TheEventTypCorpus

This section describes the process of creating theEventTypcorpus\. The corpus comprisesevent tuples\(Context, TypicFollowUp, AtypicFollowUp\): Context is an event extracted from a commonsense dataset \([§3\.1](https://arxiv.org/html/2610.33239#S3.SS1)\), TypicFollowUp is a typical follow\-up event to Context and AtypicFollowUp is an atypical follow\-up event to Context\. See examples in[Table1](https://arxiv.org/html/2610.33239#S2.T1)\. Both the TypicFollowUps and AtypicFollowUps are generated by an LLM \(GPT\-4o,[OpenAI et al\., 2024](https://arxiv.org/html/2610.33239#bib.bib33)\)\. Both the TypicFollowUp and the AtypicFollowUp depend on the Context, but we generate each AtypicFollowUp according to both its Context and the TypicFollowUp\. Therefore, after generating the TypicFollowUps \([§3\.2](https://arxiv.org/html/2610.33239#S3.SS2)\), we filter them to remove those that could not plausibly lead to an atypicality inference \([§3\.3](https://arxiv.org/html/2610.33239#S3.SS3)\) and then only generate AtypicFollowUps for the remaining examples \([§3\.4](https://arxiv.org/html/2610.33239#S3.SS4)\)\.Our finalEventTypcorpus has 2,354 event tuples\.

### 3\.1Context Event Extraction and Filtering

We source our Contexts fromATOMIC[Sap et al\. \(2019\)](https://arxiv.org/html/2610.33239#bib.bib41);[Hwang et al\. \(2021\)](https://arxiv.org/html/2610.33239#bib.bib16), a commonsense knowledge base consisting of if\-then event pairs \(e\.g\., ‘‘PersonX takes PersonY’s advice’’ isAfter ‘‘PersonY gives advice to PersonX’’\)\. We extract all event pairs for two relation types \(isBefore and isAfter\) and limit utterances to those with up to two entities\. Combining and deduplicating them yields 33,924 utterances\.333See[§B\.1](https://arxiv.org/html/2610.33239#A2.SS1)for why we usedATOMICfor Context extraction but not for follow\-ups\.We then apply rule\-based and LLM\-based filtering to remove noisy or unsuitable events \(e\.g\., idioms, passive voice; see full details in[§B\.2](https://arxiv.org/html/2610.33239#A2.SS2)\)\. This leaves 12,978 Context events\.

### 3\.2Typical Follow\-ups \(TypicFollowUps\)

For each Context, we use an LLM to generate up to three TypicFollowUps in order to increase the diversity of our corpus \(see[§B\.3](https://arxiv.org/html/2610.33239#A2.SS3)for details\)\. This yields 19,827 \(Context, TypicFollowUp\) candidates\.

### 3\.3Atypicality Inference Event Criteria

Despite prior work on atypicality inferences, no precise attributes have been identified as triggering them\.444[Kravtchenko and Demberg \(2015\)](https://arxiv.org/html/2610.33239#bib.bib25)combined events from[Raisig et al\. \(2009\)](https://arxiv.org/html/2610.33239#bib.bib36)’s ‘scripts’ with some of their own events to construct their corpus, using heuristics like ‘semantically uninformative’ to characterize TypicFollowUps\. But these heuristics are too vague to apply in large\-scale annotation\.To createEventTyp, we develop a set of specific triggering criteria through manual annotation\. We then apply these criteria through LLM\-prompting to our candidate \(Context, TypicFollowUp\) pairs\.

##### Criteria Development\.

One of the authors manually annotated a subset \(N = 9,188\) of candidate event pairs for whether or not it was possible to make an atypicality inference from them, and if not, why it was not possible\. The annotations yielded a set of 2,026 \(Context, TypicFollowUp\) pairs labeled as triggers for atypicality inferences\. From these we determine two primary rules \(see Appendix[Table5](https://arxiv.org/html/2610.33239#A2.T5)for rule details and examples\):

1. 1\.The follow\-ups should be natural for dialogue \(e\.g\. grammatical; ‘Mary takes Linda’s hand’ is but ‘Mary take Linda hand’ is not\)\.
2. 2\.The TypicFollowUp should have ‘natural’ typicality, e\.g\. very typical given the Context but still*not entailed*by the Context\. So, \(“Mary gives Linda advice”, “Linda listens”\) is acceptable, but \(“Mary arrives at Linda’s party”, “Linda has a party”\) is not\.

##### LLM\-based Criteria Filtering\.

We operationalize our criteria by prompting an LLM with 27 true\-false questions \(see[§B\.3](https://arxiv.org/html/2610.33239#A2.SS3)for details\) We reject candidate event pairs where even one criteria rule is violated\. Due to inconsistency in the LLM responses, we split some criteria into multiple overlapping questions in order to increase accuracy\. We use few\-shot prompting with manually crafted examples and iteratively develop the prompts, favoring precision over recall\. That is, we prefer the LLM rejects more good follow\-ups if that ensures fewer bad ones\.

To evaluate the quality of the LLM\-filtering, we create a set of human labels by having one human annotator label 39 randomly sampled \(Context, TypicFollowUp\) pairs from our dataset using the same 27\-question setup as the LLM\. We then calculate precision \(78\.11\), recall \(69\.93\), and inter\-annotator agreement in the form of Cohen’s Kappa \(κ=\.4532\\kappa=\.4532\)[Cohen \(1960\)](https://arxiv.org/html/2610.33239#bib.bib5)between the LLM and human annotator on this set\. While agreement is modest, this is primarily because the LLM rejects more follow\-ups than the human annotator\. On the annotated sample, the human annotator rejects 16\.48%, while the LLM annotator rejects 26\.37%\.

We run the LLM\-filtering on all 19,827 candidate \(Context, TypicFollowUp\) event pairs, resulting in 4,155 that meet our criteria\.

### 3\.4Atypical Follow\-ups \(AtypicFollowUps\)

For each \(Context, TypicFollowUp\) pair that passes our filtering in §[3\.3](https://arxiv.org/html/2610.33239#S3.SS3), we use an LLM to generate a candidate AtypicFollowUp\. We instruct the model to generate a less typical, more specific alternative to the original TypicFollowUp \([§B\.4](https://arxiv.org/html/2610.33239#A2.SS4)\)\. The number of generated AtypicFollowUps \(4,155\) is much lower than the number of TypicFollowUps generated in[§3\.2](https://arxiv.org/html/2610.33239#S3.SS2)\(19,827\), so we manually validate and correct all candidate AtypicFollowUps\.555Some tuples which were found to have errors were removed during this stage or retroactively, see[§B\.4](https://arxiv.org/html/2610.33239#A2.SS4)\.We are left with 4,113 \(Context, TypicFollowUp, AtypicFollowUp\) tuples, from which we subsample 2,330 so our data has only one TypicFollowUp and one AtypicFollowUp per Context \(note that there are up to three followups per Context from §[3\.2](https://arxiv.org/html/2610.33239#S3.SS2), and see[§B\.5](https://arxiv.org/html/2610.33239#A2.SS5)for details\)\.

We add the 24 tuples from[Kravtchenko and Demberg \(2015\)](https://arxiv.org/html/2610.33239#bib.bib25)\(K&D\) to our data, rewritten to have a consistent format with our tuples\. In total, ourEventTypcorpus has 2,354\(Context, TypicFollowUp, AtypicFollowUp\) event tuples\. See[Table1](https://arxiv.org/html/2610.33239#S2.T1)for the distribution of follow\-ups written and validated by a human or an LLM\.

## 4Human and LLM Experiment Details

Here we describe our collection of human and LLM event\-likelihood ratings, which applies to both our Commonsense Knowledge experiment \([§5](https://arxiv.org/html/2610.33239#S5)\) and our Event\-Typicality Inference experiment \([§6](https://arxiv.org/html/2610.33239#S6)\)\. We collect event\-typicality ratings by asking how ‘likely’ certain follow\-up events are; hence, we use the termlikelihood ratingsthroughout\.

### 4\.1Human Participant Details

We recruit 330 participants for the Commonsense Knowledge experiment \([§5](https://arxiv.org/html/2610.33239#S5)\) and 575 for the Event\-Typicality Inference experiment \([§6](https://arxiv.org/html/2610.33239#S6)\);666We use Prolific \([https://www\.prolific\.com/](https://www.prolific.com/)\) to recruit participants\. SinceEventTyp’s source data was created in the United States and contains US\-specific references \(e\.g\., Home Depot, Thanksgiving\), we only recruit participants based in the United States\.see[§C\.1](https://arxiv.org/html/2610.33239#A3.SS1)for more details\. For both experiments, each participant rated the likelihood of a batch \(Context, Follow\-up\) pairs, where Follow\-up was either TypicFollowUp or AtypicFollowUp, and no Context event appeared more than once in each batch\. Each batch contained 6 attention check questions that were randomly mixed with the other items\. For each event pair, we collected likelihood ratings from multiple participants: 3 for the Commonsense Knowledge experiment and 5 for the Event\-Typicality Inference experiment\.

### 4\.2LLM Details

We test 28 LLM configurations covering three inference methods \(surprisal, two prompting approaches\) and five model families with two sizes each: GPT\-5\.4,[OpenAI \(2026\)](https://arxiv.org/html/2610.33239#bib.bib34); Llama\-3\.1,[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib10); Olmo\-3,[Olmo et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib32); Qwen3,[Yang et al\., 2025](https://arxiv.org/html/2610.33239#bib.bib47); Gemma\-3,[Gemma Team \(2025\)](https://arxiv.org/html/2610.33239#bib.bib9)\. See[§C\.2](https://arxiv.org/html/2610.33239#A3.SS2)for details\.

Figure 2:Setup for our surprisal experiments with examples using theUnlikelylikelihood rating option\. Surprisal is computed over the Completion, given the Input\.##### Surprisal

We compute surprisal \(negative\-log\-likelihood\), which can be used to probe LLMs for underlying language understanding\. We adapt the method from[Cilleruelo et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib4), computing surprisalSp⁡\(Completion\)\\mathrm\{Sp\}\(\\text\{Completion\}\)over tokenstit\_\{i\}:

Sp\(Completion\)=∑i=1N−logp\(ti∣t1,…,ti−1\)\\mathrm\{Sp\}\(\\text\{Completion\}\)=\\sum\_\{i=1\}^\{N\}\-\\log p\\\!\\left\(t\_\{i\}\\mid t\_\{1\},\\ldots,t\_\{i\-1\}\\right\)To simulate multiple\-choice question\-answering, we insert each likelihood option into our input \(shown in[Fig\.2](https://arxiv.org/html/2610.33239#S4.F2)\) and then compute surprisal on the Completion \(Follow\-up\)\. Since we are only comparing surprisals of the same Follow\-up, there is no need to normalize the surprisal values by length\. We consider the LLM’s chosen answer to be the Input likelihood option where the Completion’s summed surprisal is lowest\. For this approach, we use only non\-instruction\-tuned \(base\) LLMs\.

##### Prompting

To evaluate LLMs in a way that more closely mirrors how we obtain human results, we also prompt them for a likelihood rating in the same way as we do humans\. We try both direct prompting \(Prompting\) and Chain\-of\-Thought prompting \(CoT\-Prompting\)\.777We call this CoT for all LLMs, but forGPT\-5\.4, we instead set the reasoning parameter to ‘medium\.’For this approach, we use the instruction\-tuned versions of the models used to compute Surprisal\.

Table 2:Example of the questions we ask in the Commonsense Knowledge Experiment\. Two questions are asked per Context, one for the likelihood of the TypicFollowUp and one for that of the AtypicFollowUp\.

## 5Commonsense Knowledge Experiment

To answerRQ1\(whether humans and LLMs share Commonsense Knowledge about event typicality\), we elicit likelihood ratings from both humans and LLMs for every \(Context, Follow\-up\) pair inEventTypand then compare their responses\.

We then createEventTypGold\(N = 858\) from only theEventTyptuples where the humans voted that the TypicFollowUp was typical and the AtypicFollowUp was atypical \([Section5\.3](https://arxiv.org/html/2610.33239#S5.SS3)\)\. We useEventTypGoldfor our Event\-Typicality Inference experiment \([§6](https://arxiv.org/html/2610.33239#S6)\)\.

Chedeline and Kathem are having a conversation about their friend Anahita\.Chedeline: “Anahita thanked her boss\.”Follow\-up SettingFollow\-upNONEPlainHeyGWQuestionTypicFollowUp–“She meant it\.”“And hey, guess what,In general, when Anahita thanks her boss, how likely is it thatshe meant it\!”she means it?\(Very Unlikely, Unlikely, Likely, Very Likely\)AtypicFollowUp–“She was in tears\.”“And hey, guess what,In general, In general, when Anahita thanks her boss, how likely is it thatshe was in tears\!”she is in tears?\(Very Unlikely, Unlikely, Likely, Very Likely\)

Table 3:Setup for Event\-Typicality Inference experiments \([§6](https://arxiv.org/html/2610.33239#S6)\)\. A participant sees the top dialogue \(containing a Context\), and ONE Follow\-up setting \(containing either a TypicFollowUp or a AtypicFollowUp\), along with its corresponding question\. This means that for each tuple, there are 6 possible experiment conditions\.### 5\.1Experimental Setup

We collect likelihood ratings for all \(Context, Follow\-up\) pairs inEventTypfollowing the procedures in[§4](https://arxiv.org/html/2610.33239#S4)\.[Table2](https://arxiv.org/html/2610.33239#S4.T2)shows an example of the question format we use\. For each tuple inEventTyp,888Excluding the 24 items from K&D, for which we manually write the typicality questions\.we generate two questions: one for the \(Context, TypicFollowUp\) pair and the other for \(Context, AtypicFollowUp\) pair\. We useGPT\-4o\([§D\.1](https://arxiv.org/html/2610.33239#A4.SS1)\) to generate the questions, manually validating them and correcting any errors \(e\.g\., pronoun ambiguity\)\. For each question, the possible likelihood ratings are \{Very Unlikely, Unlikely, Likely, Very Likely\}\.

For each LLM configuration, we calculate inter\-annotator agreement \(more details in[AppendixG](https://arxiv.org/html/2610.33239#A7)\) using Cohen’s Kappa[Cohen \(1960\)](https://arxiv.org/html/2610.33239#bib.bib5)for the LLM and human responses \(each response is 1 majority\-vote\)\. We collapse the four likelihood options into the binary categories of UNLIKELY \(Very Unlikely/Unlikely\) and LIKELY \(Very Likely/Likely\) and calculateκTypicFollowUp\\kappa^\{\\text\{TypicFollowUp \}\}for TypicFollowUps andκAtypicFollowUp\\kappa^\{\\text\{AtypicFollowUp \}\}for AtypicFollowUps\.

Figure 3:Human\-LLM agreement for the Commonsense Knowledge experiment, withκAtypicFollowUp\\kappa^\{\\text\{AtypicFollowUp \}\}on the X\-axis andκTypicFollowUp\\kappa^\{\\text\{TypicFollowUp \}\}on the Y\-axis\.
### 5\.2RQ1: Do humans and LLMs share Commonsense Knowledge about event typicality?

Mostly no: humans and LLMs generally do not agree on which events are typical versus atypical\.

[Fig\.3](https://arxiv.org/html/2610.33239#S5.F3)shows our agreement results\. Out of our 28 LLM configurations, onlyGPT\-5\.4with CoT has higher than0\.40\.4agreement for both Follow\-ups \(κTypicFollowUp=0\.4336\\kappa^\{\\text\{TypicFollowUp \}\}=0\.4336andκAtypicFollowUp=0\.5059\\kappa^\{\\text\{AtypicFollowUp \}\}=0\.5059\)\. OnlyGPT\-5\.4andLlama\-3\.1\-70B\-Instructachievedκ\>\.3\\kappa\>\.3for both follow\-ups with both prompting settings, but the other three model families do not achieve this even with CoT\-Prompting\. This low agreement illustrates the lack of shared commonsense knowledge between our human participants and LLMs\.

### 5\.3TheEventTypGoldCorpus

We are interested in whetherEventTyp’s Follow\-up categories \(Typic/Atypic\) match our human annotators’ actual likelihood ratings\. We first compute, for each Context, the percentage of tuples where the average rating of the TypicFollowUp was higher than that of the AtypicFollowUp \(i\.e\., the TypicFollowUp was judged more likely than the AtypicFollowUp\)\. This condition holds for80\.04%of our generated tuples and 95\.83% of the K&D tuples\. This aligns with how we prompted the LLM \([§3\.4](https://arxiv.org/html/2610.33239#S3.SS4)\), asking it to generate a Follow\-up that was less likely than the TypicFollowUp but not necessarily unlikely\.

However, for a given Context we want not only the correctrelativerelation between Follow\-ups but also the stricter condition that the TypicFollowUp is LIKELY and the AtypicFollowUp UNLIKELY\. We compute the percentage of tuples where this occurs with at least 2/3 annotators agreeing\. From the tuples fromEventTypthat we generated, this condition holds for 36\.09% \(841\), while it does for 70\.83% \(17\) tuples from K&D, despite the fact that the tuples were manually crafted by their original authors to have pairs of likely\-unlikely events\.The 858 tuples that meet the strict condition are our finalEventTypGoldcorpus\.

## 6Event\-Typicality Inference Experiment

To answerRQ2andRQ3, we collect human and LLM likelihood ratings on a sample \(N = 450\) ofEventTypGoldwithdialogue scenarios\. We subsample due to budget constraints \(see[§E\.1](https://arxiv.org/html/2610.33239#A5.SS1)\)\.

### 6\.1Experimental Setup

##### Dialogue Scenarios

To create an environment where pragmatic inferences are possible, we insert events into dialogue scenarios and then ask questions about those scenarios\. Unlike[Kravtchenko and Demberg \(2022\)](https://arxiv.org/html/2610.33239#bib.bib26), we opt to use minimal dialogue framing in order to avoid confounding factors\. We create two example instances from each \(Context, TypicFollowUp, AtypicFollowUp\) tuple using randomly selected rare names \(see[§E\.2](https://arxiv.org/html/2610.33239#A5.SS2)for details\)\. Then we useGPT\-4oto generate a typicality question for the TypicFollowUp and AtypicFollowUp events\. We manually validate and correct all questions\. Unlike in the Commonsense Knowledge experiment \([§5](https://arxiv.org/html/2610.33239#S5)\), the questions are use names specific to the dialogue scenarios\.

##### Experimental Conditions

We embed the Context in a dialogue scenario and then investigate how the likelihood ratings for its Follow\-up \(TypicFollowUp/AtypicFollowUp\) change when the Follow\-up is mentioned \(PlainandHeyGW\) or unmentioned \(NONE\)\.[Table3](https://arxiv.org/html/2610.33239#S5.T3)explains this setup\. In thePlainsetting, the Follow\-up is mentioned without Utterance Emphasis, while in theHeyGWsetting the Follow\-up is preceded by Utterance Emphasis \(the phrase “And hey, guess what”\)\. We consider a participant’s response in the NONE setting to be a control \(i\.e\., the Follow\-up’s baseline likelihood given the Context\)\. Our analyses compare responses from the NONE setting to those from Plain or HeyGW\.

##### Inference score \(InfScore\)

To compute whether an atypicality or typicality inference was made, we define InfScore \(see[AppendixG](https://arxiv.org/html/2610.33239#A7)\)\. For each individual event, InfScore measures the difference in average responses between the NONE and the mentioned \(Plain or HeyGW\) settings\. A positive InfScore indicates an atypicality inference and a negative InfScore a typicality inference\. The magnitude of InfScore represents inference strength\.

##### Snark Annotation

We hypothesize that one possible reason why speakers produce utterances that lead toatypicalityinferences is that they wish to convey negative information about someone without it beingexplicitlynegative\. We define these scenarios asSnarky: if the atypicality inference helps the lister learn that the person being talked about has some negative trait, and the listener can therefore interpret the speaker as having a snarky attitude\. For example, on the TypicFollowUp side of[Fig\.1](https://arxiv.org/html/2610.33239#S1.F1), if an atypicality inference is made \(i\.e\., that Anahita usually does not mean it when she thanks her boss\), this may mean Anahita is fake or dishonest\. Thus, we label this scenario assnarky\. For all \(Context, TypicFollowUp\) pairs inEventTypGold, we manually annotate whether they form a snarky scenario or not\. See[§E\.3](https://arxiv.org/html/2610.33239#A5.SS3)for a deeper discussion on how snark relates to commonsense knowledge and irony\.

Figure 4:Human results for TypicFollowUps vs\. AtypicFollowUps, where a point’s size denotes the number of InfScores equal to its X value\. InfScore \> 0 =atypicalityinference, and InfScore < 0 =typicalityinference\.Figure 5:Human distributions of responses in the NONE vs\. HeyGW settings for TypicFollowUps, with non\-snarky scenarios \(left\) and snarky scenarios \(right\)\.

### 6\.2RQ2: When do humans make inferences about event typicality?

##### Atypicality inferences

We find that humans only consistently make atypicality inferences on TypicFollowUps when Utterance Emphasis is present; Snark increases these inferences\.

[Fig\.4](https://arxiv.org/html/2610.33239#S6.F4)shows the InfScores for the Plain vs\. HeyGW settings\. We can see that for TypicFollowUps,InfScore\>0\\text\{InfScore \}\>0appears often for HeyGW \(p<<0\.001, see[§E\.5](https://arxiv.org/html/2610.33239#A5.SS5)\), indicating atypicality inferences are made when Utterance Emphasis is present\. But for Plain, the opposite,InfScore<0\\text\{InfScore \}<0, is more frequent \(p<<0\.001\), indicatingtypicalityinferences\. This contrasts with[Kravtchenko and Demberg \(2022\)](https://arxiv.org/html/2610.33239#bib.bib26)’s work, which found that participants madeatypicality inferences in the Plain setting\. A possible reason for this is that their events were presented within verbose backstories explicitly mentioning typicality, which may have primed participants to make atypicality inferences\. Our items, in contrast, use only a short dialogue\.

To check Snark’s effect on atypicality inferences, we focus on TypicFollowUp events in the HeyGW setting, the condition for which participants made consistent atypicality inferences\.[Fig\.5](https://arxiv.org/html/2610.33239#S6.F5)shows that the difference between ratings in the NONE and HeyGW settings is higher when Snark is present \(p<<0\.001, see[§E\.5](https://arxiv.org/html/2610.33239#A5.SS5)\), confirming that participants make more atypicality inferences for snarky scenarios\. This finding is consistent with the claim that people use pragmatics to communicate about socially sensitive situations\. But for a listener to recognize ‘snark’ in a specific scenario and make an atypicality inference on it, they must not only have the commonsense knowledge that the TypicFollowUp is generally typical, but also that itshouldhappen according to social conventions or expectations of competence \(see[§E\.3](https://arxiv.org/html/2610.33239#A5.SS3)\)\. So, it is interesting that these scenarios are the ones with higher rates of atypicality inferences\.

##### Typicality inferences

We find that humans make typicality inferences for AtypicFollowUps regardless of whether Utterance Emphasis is present; humans also make them for TypicFollowUps when Utterance Emphasis is not present\.

For the majority of AtypicFollowUps,InfScore<0\\text\{InfScore \}<0\([Fig\.4](https://arxiv.org/html/2610.33239#S6.F4)\), showing typicality inferences for both the Plain and HeyGW settings \(p<<0\.001\)\. This not consistent with[Kravtchenko and Demberg \(2022\)](https://arxiv.org/html/2610.33239#bib.bib26)’s findings, which showed no significant difference between likelihood ratings for unmentioned vs\. mentioned Follow\-up settings\. However, our findingisconsistent with the intuition that if a Follow\-up event is initially unexpected \(i\.e\., has a low likelihood rating in the NONE setting\), but a listener learns that it happened once \(i\.e\., mentioned in Plain/HeyGW settings\), the listener may assume that it actually happens more often than they initially expected \. In other words, they simply update their beliefs based on what happened without making a pragmatic inference\.

Figure 6:Percentage of Follow\-ups in the HeyGW setting where an LLM made a typicality \(X\-axis\) vs\. atypicality inference \(Y\-axis\)\. Points in the gray halo are LLM configurations where the atypicality and typicality inferences were both statistically significant \(p<<0\.01\)\.

### 6\.3RQ3: Do LLMs make inferences about event typicality in the contexts where humans do?

Only large models with CoT\-Prompting do this\.

We focus our analyses on the HeyGW setting, as this is where humans made both typicality and atypicality inferences in the expected event contexts \(see[§E\.5](https://arxiv.org/html/2610.33239#A5.SS5)\)\.[Fig\.6](https://arxiv.org/html/2610.33239#S6.F6)shows that very few LLM configurations achieve a similar balance to humans between typicality and atypicality inferences\. We confirm this with statistical tests[§E\.5](https://arxiv.org/html/2610.33239#A5.SS5): only 6 out of the 28 configurations produce statistically significant inferences \(i\.e\., thementionedstatus of the Follow\-up in the dialogue, e\.g\., NONE vs\. HeyGW, predicts the Follow\-up’s likelihood rating\) for both inference types, and 4 out of those are with CoT prompting on larger models\. For most larger LLM configurations which showed atypicality inferences, we also see an increase in these inferences for Snarky scenarios \([§E\.5](https://arxiv.org/html/2610.33239#A5.SS5)and[Fig\.12](https://arxiv.org/html/2610.33239#A5.F12)\)\.

Figure 7:LLM consistency; i\.e\. the percentage of Follow\-ups where responses from the Event\-Typicality Inference experiment NONE setting matched those from the Commonsense Knowledge experiment, \(X\-axis for AtypicFollowUps and Y\-Axis for TypicFollowUps\)\.

## 7Discussion

While some LLMs appear to follow the same CUES as humans in making event\-typicality inferences, the story may be more complicated due to their often\-inconsistent displays of commonsense knowledge\. With LLMs, we find a significant \(p<<0\.01\) correlation between higher commonsense knowledge and more event\-typicality inferences \(see[§E\.6](https://arxiv.org/html/2610.33239#A5.SS6)\), which indicates that it is an essential component of these inferences\. However, LLMs also fail to exhibit consistent event likelihood ratings, calling into question their commonsense knowledge\.[Fig\.7](https://arxiv.org/html/2610.33239#S6.F7)shows the percentage of Follow\-ups where each LLM configuration rated the likelihood of the that Follow\-upthe samein both the Commonsense Knowledge Experiment \([§5](https://arxiv.org/html/2610.33239#S5)\) and the Event\-Typicality Inference Experiment \(in the comparable NONE\-setting;[§6](https://arxiv.org/html/2610.33239#S6)\)\. The LLMs’ consistency is very low compared to humans: no LLM configuration apart fromGPT\-5\.4with CoT scored higher than 80% consistency on both TypicFollowUps and AtypicFollowUps\.

Note that we would hope for much higher scores because in each case, the questions are aboutexact sameFollow\-up events; the only difference is the latter’s addition of dialogue and named entities\. The fact that LLMs show such inconsistency when asked about the exact same events suggests that they are more sensitive to theframing of the promptthan to theactual typicalityof an event \(see[AppendixF](https://arxiv.org/html/2610.33239#A6)\)\. This is in line with prior work on both commonsense knowledge tasks and more general tasks, which has found that when the wording of the prompt is slightly changed, LLMs give inconsistent, and sometimes even opposite, answers to the same exact questions[Chen et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib3);[Bian et al\. \(2024\)](https://arxiv.org/html/2610.33239#bib.bib2);[Kim and Hwang \(2025\)](https://arxiv.org/html/2610.33239#bib.bib23);[Liu et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib30)\.

## 8Conclusion

In this paper, we investigated event\-related typicality inferences on a much larger scale than any previous work has before\. We createdEventTypandEventTypGold, recounting the corpora creation process in detail to enable reproducibility\. We conducted a human study that gave us three CUES that predict inferences:Commonsense Knowledge,UtteranceEmphasis, andSnark\. We asked whether LLMs are also sensitive to these CUES and found that although they sometimes are, having the Commonsense Knowledge to predict an event’s general typicality is a much bigger hurdle than expected\.

Our findings leave many open questions for future work\. For example, other features aside from textual emphasis and snark, like verb aspect or prosodic cues, might predict these inferences in humans\. And for LLMs, further exploration is necessary to answer why models fail to consistently exhibit commonsense knowledge and to find solutions to mitigate this\.

## Limitations

In our experiments, we tested only one form of utterance emphasis \(the HeyGW setting\) in eliciting pragmatic inferences\. Similarly, we used a single, minimal form of dialogue context for our experiments\. It remains to be seen whether other interventions, including more minimal ones tested in prior works \(e\.g\., adding ‘\!’;[Kravtchenko and Demberg, 2022](https://arxiv.org/html/2610.33239#bib.bib26)\), and other forms of context, also trigger atypicality and typicality inferences at scale for humans or LLMs\. Furthermore, because we use textual data, our experiments cannot account for the role that prosody may play in triggering pragmatic inferences\.

The events tested in our experiments are constrained to those can that happen relatively often in an average \(US\) person’s life, as well as relatively quickly \(i\.e\., the event is complete within a day of starting\)\. It remains to be seen whether other types of events can trigger typicality\-related inferences in the same patterns\.

For our human experiments, we specifically tested people based in the US, due to the US\-centric nature of the original data\. As such, the underlying participant pool may not reflect the full diversity of English speakers\. Additionally, our criteria for Context and Follow\-up events were developed by English speakers\. Event\-typicality inferences may be triggered differently for speakers of languages other than English\.

In our LLM experiments, we only tested five LLM families\. Since we saw varied behavior across these models, we cannot predict how LLMs from other model families \(e\.g\.,Mistral[Jiang et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib20),DeepSeek\([Guo et al\., 2025](https://arxiv.org/html/2610.33239#bib.bib12)\)\) will behave\. In particular, we only tested one closed source model \(GPT\-5\.4\)\. Testing with other such models could provide more insights\.

## References

- Bergen et al\. \(2016\)Leon Bergen, Roger Levy, and Noah Goodman\. 2016\.Pragmatic reasoning through semantic inference\.*Semantics and Pragmatics*, 9:20–1\.
- Bian et al\. \(2024\)Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, Ben He, Shanshan Jiang, and Bin Dong\. 2024\.[ChatGPT is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models](https://aclanthology.org/2024.lrec-main.276/)\.In*Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\)*, pages 3098–3110, Torino, Italia\. ELRA and ICCL\.
- Chen et al\. \(2023\)Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao\. 2023\.[Say what you mean\! large language models speak too positively about negative commonsense knowledge](https://doi.org/10.18653/v1/2023.acl-long.550)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 9890–9908, Toronto, Canada\. Association for Computational Linguistics\.
- Cilleruelo et al\. \(2025\)Gustavo Cilleruelo, Emily Allaway, Barry Haddow, and Alexandra Birch\. 2025\.[Generics are puzzling\. can language models find the missing piece?](https://aclanthology.org/2025.coling-main.438/)In*Proceedings of the 31st International Conference on Computational Linguistics*, pages 6571–6588, Abu Dhabi, UAE\. Association for Computational Linguistics\.
- Cohen \(1960\)Jacob Cohen\. 1960\.A coefficient of agreement for nominal scales\.*Educational and psychological measurement*, 20\(1\):37–46\.
- Degen et al\. \(2020\)Judith Degen, Robert D Hawkins, Caroline Graf, Elisa Kreiss, and Noah D Goodman\. 2020\.When redundancy is useful: A bayesian approach to “overinformative” referring expressions\.*Psychological review*, 127\(4\):591\.
- Floyd et al\. \(2025\)Sammy Floyd, Olessia Jouravlev, Moshe Poliak, Zachary Mineroff, Edward Gibson, and Evelina Fedorenko\. 2025\.Three distinct components of pragmatic language use: Social conventions, intonation, and world knowledge–based causal reasoning\.*Proceedings of the National Academy of Sciences*, 122\(50\):e2424400122\.
- Franke \(2011\)Michael Franke\. 2011\.[Quantity implicatures, exhaustive interpretation, and rational conversation](https://doi.org/10.3765/sp.4.1)\.*Semantics and Pragmatics*, 4\(1\):1–82\.
- Gemma Team \(2025\)Gemma Team\. 2025\.[Gemma 3 technical report](https://arxiv.org/abs/2503.19786)\.*Preprint*, arXiv:2503\.19786\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others\. 2024\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*\.
- Grice \(1975\)HP Grice\. 1975\.Logic and conversation\.*Syntax and semantics*, 3\.
- Guo et al\. \(2025\)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z\. F\. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others\. 2025\.[Deepseek\-r1 incentivizes reasoning in llms through reinforcement learning](https://doi.org/10.1038/s41586-025-09422-z)\.*Nature*, 645\(8081\)\.
- Haverkate \(1990\)Henk Haverkate\. 1990\.A speech act analysis of irony\.*Journal of pragmatics*, 14\(1\):77–109\.
- Hu et al\. \(2023\)Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson\. 2023\.[A fine\-grained comparison of pragmatic language understanding in humans and language models](https://doi.org/10.18653/v1/2023.acl-long.230)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 4194–4213, Toronto, Canada\. Association for Computational Linguistics\.
- Hu et al\. \(2025\)Jennifer Hu, Felix Sosa, and Tomer Ullman\. 2025\.[Shades of zero: Distinguishing impossibility from inconceivability](https://doi.org/10.1016/j.jml.2025.104640)\.*Journal of Memory and Language*, 143:104640\.
- Hwang et al\. \(2021\)Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi\. 2021\.\(comet\-\) atomic 2020: On symbolic and neural commonsense knowledge graphs\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 35, pages 6384–6392\.
- Islakoglu and Kalo \(2025\)Duygu Sezen Islakoglu and Jan\-Christoph Kalo\. 2025\.[ChronoSense: Exploring temporal understanding in large language models with time intervals of events](https://doi.org/10.18653/v1/2025.acl-short.46)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 590–602, Vienna, Austria\. Association for Computational Linguistics\.
- Jain et al\. \(2023\)Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat\. 2023\.[Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models](https://doi.org/10.18653/v1/2023.emnlp-main.418)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 6750–6774, Singapore\. Association for Computational Linguistics\.
- Ji et al\. \(2024\)Shengwei Ji, Longfei Liu, Jizhong Xi, Xiaoxue Zhang, and Xinlu Li\. 2024\.Klr\-kgc: Knowledge\-guided llm reasoning for knowledge graph completion\.*Electronics*, 13\(24\):5037\.
- Jiang et al\. \(2023\)Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\. 2023\.[Mistral 7b](https://arxiv.org/abs/2310.06825)\.*Preprint*, arXiv:2310\.06825\.
- Joshi et al\. \(2024\)Abhinav Joshi, Areeb Ahmad, and Ashutosh Modi\. 2024\.Cold: Causal reasoning in closed daily activities\.*Advances in Neural Information Processing Systems*, 37:5145–5187\.
- Kauf et al\. \(2023\)Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci\. 2023\.Event knowledge in large language models: the gap between the impossible and the unlikely\.*Cognitive Science*, 47\(11\):e13386\.
- Kim and Hwang \(2025\)Jongho Kim and Seung\-won Hwang\. 2025\.[Counterfactual\-consistency prompting for relative temporal understanding in large language models](https://doi.org/10.18653/v1/2025.acl-short.97)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 1210–1225, Vienna, Austria\. Association for Computational Linguistics\.
- Koupaee et al\. \(2025\)Mahnaz Koupaee, Xueying Bai, Mudan Chen, Greg Durrett, Nathanael Chambers, and Niranjan Balasubramanian\. 2025\.[Causal graph based event reasoning using semantic relation experts](https://doi.org/10.18653/v1/2025.acl-long.1269)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 26169–26199, Vienna, Austria\. Association for Computational Linguistics\.
- Kravtchenko and Demberg \(2015\)Ekaterina Kravtchenko and Vera Demberg\. 2015\.Semantically underinformative utterances trigger pragmatic inferences\.In*Proceedings of the annual meeting of the cognitive science society*, volume 37\.
- Kravtchenko and Demberg \(2022\)Ekaterina Kravtchenko and Vera Demberg\. 2022\.Informationally redundant utterances elicit pragmatic inferences\.*Cognition*, 225:105159\.
- Kurch et al\. \(2024\)Charlotte Kurch, Margarita Ryzhova, and Vera Demberg\. 2024\.[Large language models fail to derive atypicality inferences in a human\-like manner](https://doi.org/10.18653/v1/2024.cmcl-1.8)\.In*Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 86–100, Bangkok, Thailand\. Association for Computational Linguistics\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E\. Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.Efficient memory management for large language model serving with pagedattention\.In*Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*\.
- Lepori et al\. \(2026\)Michael Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, and Ellie Pavlick\. 2026\.Is this just fantasy? language model representations reflect human judgments of event plausibility\.In*International Conference on Learning Representations*, volume 2026, pages 113814–113831\.
- Liu et al\. \(2025\)Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, and Nigel Collier\. 2025\.[Aligning with logic: Measuring, evaluating and improving logical preference consistency in large language models](https://arxiv.org/abs/2410.02205)\.*Preprint*, arXiv:2410\.02205\.
- Ma et al\. \(2025\)Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, and Barbara Plank\. 2025\.[Pragmatics in the era of large language models: A survey on datasets, evaluation, opportunities and challenges](https://doi.org/10.18653/v1/2025.acl-long.425)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 8679–8696, Vienna, Austria\. Association for Computational Linguistics\.
- Olmo et al\. \(2025\)Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and 1 others\. 2025\.Olmo 3\.*arXiv preprint arXiv:2512\.13961*\.
- OpenAI et al\. \(2024\)OpenAI, :, Aaron Hurst, Adam Lerer, Adam P\. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker\-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others\. 2024\.[Gpt\-4o system card](https://arxiv.org/abs/2410.21276)\.*Preprint*, arXiv:2410\.21276\.
- OpenAI \(2026\)OpenAI\. 2026\.[GPT\-5\.4 Thinking System Card](https://openai.com/index/gpt-5-4-thinking-system-card/)\.Technical report, OpenAI\.
- Pedinotti et al\. \(2021\)Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache\. 2021\.[Did the cat drink the coffee? challenging transformers with generalized event knowledge](https://doi.org/10.18653/v1/2021.starsem-1.1)\.In*Proceedings of \*SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics*, pages 1–11, Online\. Association for Computational Linguistics\.
- Raisig et al\. \(2009\)Susanne Raisig, Tinka Welke, Herbert Hagendorf, and Elke Van der Meer\. 2009\.Insights into knowledge representation: The influence of amodal and perceptual variables on event knowledge retrieval from memory\.*Cognitive Science*, 33\(7\):1252–1266\.
- Rees et al\. \(2026\)Alice Rees, Vilde Reksnes, and Hannah Rohde\. 2026\.Why are you telling me this? The availability and timing of triviality\-driven inferences\.*Journal of Memory and Language*, 148:104741\.
- Rohde and Rubio\-Fernandez \(2022\)Hannah Rohde and Paula Rubio\-Fernandez\. 2022\.Color interpretation is guided by informativity expectations, not by world knowledge about colors\.*Journal of Memory and Language*, 127:104371\.
- Ryzhova et al\. \(2023\)Margarita Ryzhova, Alexandra Mayn, and Vera Demberg\. 2023\.What inferences do people actually make upon encountering informationally redundant utterances? an individual differences study\.In*Proceedings of the Annual Meeting of the Cognitive Science Society*, volume 45\.
- Sabour et al\. \(2024\)Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang\. 2024\.[EmoBench: Evaluating the emotional intelligence of large language models](https://doi.org/10.18653/v1/2024.acl-long.326)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 5986–6004, Bangkok, Thailand\. Association for Computational Linguistics\.
- Sap et al\. \(2019\)Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi\. 2019\.Atomic: An atlas of machine commonsense for if\-then reasoning\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 33, pages 3027–3035\.
- Sperber et al\. \(1986\)Dan Sperber, Deirdre Wilson, and Deirdre Wilson\. 1986\.*Relevance: Communication and cognition*, volume 142\.Harvard University Press Cambridge, MA\.
- Toroghi et al\. \(2026\)Armin Toroghi, Faeze Moradi Kalarde, and Scott Sanner\. 2026\.[Commonwhy: A dataset for evaluating entity\-based causal commonsense reasoning in large language models](https://arxiv.org/abs/2605.12918)\.*Preprint*, arXiv:2605\.12918\.
- Wang et al\. \(2026\)Shaofei Wang, Yunan Liu, Xiaolan Tang, and Wenlong Chen\. 2026\.[Collaborative enhancement of large and small models for question answering via dual knowledge transfer](https://doi.org/10.1609/aaai.v40i40.40652)\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 40:33630–33638\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others\. 2022\.Chain\-of\-thought prompting elicits reasoning in large language models\.*Advances in neural information processing systems*, 35:24824–24837\.
- Wilson \(2006\)Deirdre Wilson\. 2006\.The pragmatics of verbal irony: Echo or pretence?*Lingua*, 116\(10\):1722–1743\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others\. 2025\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*\.
- Yerukola et al\. \(2024\)Akhila Yerukola, Saujas Vaduguru, Daniel Fried, and Maarten Sap\. 2024\.[Is the pope catholic? yes, the pope is catholic\. generative evaluation of non\-literal intent resolution in LLMs](https://doi.org/10.18653/v1/2024.acl-short.26)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 265–275, Bangkok, Thailand\. Association for Computational Linguistics\.
- Zhao et al\. \(2023\)Zirui Zhao, Wee Sun Lee, and David Hsu\. 2023\.Large language models as commonsense knowledge for large\-scale task planning\.*Advances in neural information processing systems*, 36:31967–31987\.

## Appendix APragmatics Background

### A\.1Cooperativity & Quantity

Pragmatics is the branch of Linguistics that targets questions about how speakers package information and how listeners draw inferences and recover intended meaning\. In assessing why a speaker produces a particular utterance, listeners bring to bear various expectations about the ways in which the speaker’s contribution will be relevant and, crucially for our purposes, newsworthy\.

These expectations reflect a set of principles that are posited to underlie cooperative communication[Grice \(1975\)](https://arxiv.org/html/2610.33239#bib.bib11): A speaker’s contribution is expected to be \(i\) appropriate linked to the discourse topic and goals \(via some Relation to prior information\), \(ii\) sufficiently informative \(as measured in the Quantity of new information\), and also \(iii\) truthful \(to uphold the Quality of the contribution\)\. Given these expectations for cooperativity, a speaker who makes an apparent violation of any of these can invite subsequent inferences, i\.e\., how does the discourse contribution uphold a commitment to cooperativity despite the apparent violation of Quality/Relation/Quantity? For example, an apparent violation of Quality may be interpreted as sarcastic, or an apparent violation of relation may encourage the listener to seek additional pathways to relevance\. When a speaker’s contribution appears to violate Quantity, the listener may potentially infer speaker ignorance \(if the speaker has provided unexpectedly little information\) or they may seek to establish why the contribution is newsworthy at all \(if the speaker has been unexpectedly redundant\)\.

Quantity\-driven reasoning includes the treatment of scalar implicatures, whereby the use of an ambiguous scalar \(e\.g\., ‘warm’\) generates an implicature from its seeming underinformativity\. A comprehender reasons that if the speaker is cooperative, their choice to not use a more informative alternative \(‘hot’\) implies the stronger meaning is not true \(→\\rightarrowwarm but not hot\)\. That comprehenders seek to establish a cooperative interpretation is what drives a range of Quantity inferences about apparent redundancy[Degen et al\. \(2020\)](https://arxiv.org/html/2610.33239#bib.bib6);[Rohde and Rubio\-Fernandez \(2022\)](https://arxiv.org/html/2610.33239#bib.bib38), underspecification[Franke \(2011\)](https://arxiv.org/html/2610.33239#bib.bib8), or triviality[Rees et al\. \(2026\)](https://arxiv.org/html/2610.33239#bib.bib37), including atypicality inferences\.

### A\.2Atypicality Inferences

With atypicality inferences, a speaker’s apparent uninformativity \(the mention of some individual’s TypicFollowUp act\), combines with the listener’s belief that the speaker is cooperative: The speakerisinformative if the reported act is actually*atypical*for that individual\.

For the example in Figure[1](https://arxiv.org/html/2610.33239#S1.F1), a speaker asserts content that is typically inferable from shared knowledge of day\-to\-day events: When thanking one’s boss, it is typical for an individual to mean it\. A speaker who states that information explicitly \("XX thanked their boss\. \[And hey, guess what,\] she meant it \[\!\]"\) appears to violate expectations about the appropriate Quantity of information that should be present in a cooperative contribution to a discourse\. If the listener trusts that the speaker intends to be cooperative, they may look for ways in which the contribution is indeed informative\. A such, the listener may ‘rescue’ the speaker’s seemingly redundant contribution by drawing an atypicality inference: For this individual, the mentioned eventisnewsworthy and the speaker intends to convey that the individual’s behavior is atypicalfor that individual\(whose usual behavior is at odds with accepted norms for such situations\)\.

Atypicality inferences are thus an example of non\-trivial pragmatic reasoning\. They require the listener to draw a conclusion \(that the individual doesn’t typically do X\) from a statement whose literal meaning provides evidence of the opposite \(that the individual \*has\* done X\)\. In this way, they depend on subtle reasoning that reflects broader expectations of speaker cooperativity alongside an awareness that what a speaker intends to convey may go beyond what has been explicitly said\.

### A\.3Typicality inferences

Unlike in atypicality inferences, typicality inferences do not appear to be pragmatic in nature\. There is no apparent redundancy when a listener hears "Anahita thanked her boss\. And hey, guess what, she was in tears \[\!\]" Being in tears while thanking one’s boss is generally atypical and therefore aligns with listeners’ expectations to contribute the appropriate Quantity of information to the discourse\. Upon learning this new information about Anahita, the listener may update their beliefs and think that for Anahita, crying is actually more typical than it is for others\. As such, we do not consider typicality inferences to be pragmatics; but rather a straightforward belief update\.

## Appendix BTheEventTypCorpus

Note that for all LLM generation and filtering steps, we useGPT\-4o, specifically ‘gpt\-4o\-2024\-08\-06,’ with a temperature of 1\. We selected this model specifically because it was the best\-performing model that was within our budget\.

### B\.1Context Event Extraction

##### Source data\.

We source our context events fromATOMIC[Sap et al\. \(2019\)](https://arxiv.org/html/2610.33239#bib.bib41);[Hwang et al\. \(2021\)](https://arxiv.org/html/2610.33239#bib.bib16), a commonsense reasoning dataset consisting of if\-then event pairs\. We limit events to those with up to two named Person entities \(PersonX and PersonY\)\.

##### We cannot directly use event relations fromATOMIC\.

We originally wanted to see if we could directly pull \(Context, TypicFollowUp\) and \(Context, AtypicFollowUp\) pairs fromATOMIC, but this does not work because:

- •After manually looking at some of the events, we determine that the proportion of TypicFollowUps that would meet our criteria \(see[§3\.3](https://arxiv.org/html/2610.33239#S3.SS3)and[§B\.3](https://arxiv.org/html/2610.33239#A2.SS3)\) is too low\. For example, highly atypical events like “PersonX murders PersonY” are quite common, as well as directly entailed events like “PersonX takes PersonY’s advice” isAfter “PersonY gives advice to PersonX”\. Although our LLM\-generated follow\-ups still have errors, the filtering did not need to eliminate as much\.
- •We wanted one TypicFollowUp and one AtypicFollowUp for every Context, but that does not follow the structure ofATOMIC, where one Context might appear any number of times, possibly only mapping to one follow\-up type \(TypicFollowUp or AtypicFollowUp\) each of those times\.
- •Aggregating all unique events from the isBefore/isAfter relation type, regardless of position, enabled us to start with the largest possible pool of Contexts from which to generate follow\-ups\.

So, regardless of whether an event was originally considered the ‘before’ or the ‘after’ event in a pair, we put all events into the same pool of Contexts which we later use to generate the TypicFollowUps and AtypicFollowUps\.

TypicFollowUp EventExample followup toCategoryMary thanks her bossStateMary is at the officeSubactionMary means itNext\-actionMary’s boss says you’re welcomeTable 4:Example TypicFollowUp events for the three utterance categories\.

### B\.2Context Event Filtering

TheATOMICdataset is noisy, so we use both rules\-based and LLM\-based methods to filter it\. We replace all instances of PersonX with Mary and PersonY with Linda, replacing all pronouns with she/her/hers\. We perform LLM\-filtering in two passes usingGPT\-4o: the easy pass and the hard pass\. Theeasy passconsists of filtering out utterances with incomplete or otherwise malformed clauses, idioms, passive voice, implicit verbs, non\-present tense, pronouns with no antecedents, qualitative statements, and statements explicitly about habituality\. Thehard passattempts to remove items that are too vague, make absolutely no sense without context, or describe extreme situations like murder\. The result is 12,978 Contexts\.

#### B\.2\.1Easy pass prompt

1Youareanexpertannotatorofnaturallanguage\.Youwillbepresentedwithaclause\.

2Yourtaskistodetermineifanyofthefollowingistrue:

3

4\-Theclauseisincomplete

5\-Theclauseismalformed

6\-Theclauseisabstract\.

7\-Theclauseisidiomaticornotliteral\.

8\-Theclausecontainsapassive\.

9\-Theclausecontainsanimplicitverb\.

10\-Theclausedoesnottakesimplepresenttenseform\.

11\-Theclausecontainsapronounwithnoantecedent\.

12\-Theclausecontainsaqualitativeorevaluativeaction\.

13\-Theclausecontainsahabitualorstativeverbform,oritsmeaningcanbeinterpretedashabitual/stative\.

14

15Ifanyoftheabovearetrue,respondwiththetuple\(True,\[aboveitemwhichmostclearlyistrue\]\)\.Otherwise,respondwiththetuple\(False,None\)\.

16

17CLAUSE:MarylosesLindaarm

18RESPONSE:\(True,’malformed’\)

19

20CLAUSE:Marylosesherkeys

21RESPONSE:\(False,None\)

22

23CLAUSE:Maryreinventsthewheel

24RESPONSE:\(True,’idiomatic’\)

25

26CLAUSE:Marywantsacar\.

27RESPONSE:\(True,’stative’\)

28

29CLAUSE:Maryknowswoulddie

30RESPONSE:\(True,’malformed’\)

31

32CLAUSE:Maryexposesoneself

33RESPONSE:\(True,’pronoun’\)

34

35CLAUSE:Marysingstoher

36RESPONSE:\(True,’pronoun’\)

37

38CLAUSE:Marysingsatthecompetition

39RESPONSE:\(False,None\)

40

41CLAUSE:Maryspendsaday

42RESPONSE:\(True,’incomplete’\)

43

44CLAUSE:Marybuyssomeapples

45RESPONSE:\(False,None\)

46

47CLAUSE:Marysingstome

48RESPONSE:\(True,’pronoun’\)

49

50CLAUSE:Maryexposesthem

51RESPONSE:\(True,’pronoun’\)

52

53CLAUSE:MarymeansnothingtoLinda

54RESPONSE:\(True,’abstract’\)

55

56CLAUSE:Lindaforgetshercoat

57RESPONSE:\(True,’implicitverb’\)

58

59CLAUSE:MarygetsinsultedbyLinda

60RESPONSE:\(True,’passive’\)

61

62CLAUSE:LindainsultsMary

63RESPONSE:\(False,None\)

64

65CLAUSE:Maryinsultshim

66RESPONSE:\(True,’pronoun’\)

67

68CLAUSE:Maryiscalminstressfulsituations

69RESPONSE:\(True,’habitual’\)

70

71CLAUSE:Marydoeswellatthecompetition

72RESPONSE:\(True,’evaluative’\)

73

74CLAUSE:MarycompetesagainstLinda

75RESPONSE:\(False,None\)

76

77CLAUSE:Marygetsonthebus

78RESPONSE:\(False,None\)

79

80CLAUSE:Maryremembershercoat

81RESPONSE:\(True,’implicitverb’\)

82

83CLAUSE:Marywantstoeatlunch

84RESPONSE:\(True,’notsimplepresent’\)

85

86CLAUSE:Maryeatslunch

87RESPONSE:\(False,None\)

88

89CLAUSE:\{\}

90RESPONSE:

Listing 1:classify\-easy
#### B\.2\.2Hard pass prompt

1Youareanexpertannotatorofnaturallanguage\.Youwillbepresentedwithaclause\.

2Yourtaskistodetermineifanyofthefollowingistrue:

3

4\-Theclausedescribesavagueactionthatwouldtypicallybeseenwithmorecontext,e\.g\.withamodifierorwithamoredetailedpredicate\.\-Theclauseappearstobegroundedinaspecificcontextthatisnotspecified\.

5\-Theclausedescribesasituationwhichisextremelyunusualindailylife\.Perhapsthesituationiscommononfictionalstories,butnotreallife\.

6\-Theclausedescribesanactionwhichisabnormalornonsensicalwithoutcontext\.

7Ifanyoftheabovearetrue,respondwiththetuple\(True,\[aboveitemwhichmostclearlyistrue\]\)\.Otherwise,respondwiththetuple\(False,None\)\.

8

9CLAUSE:Marylosesherkeys

10RESPONSE:\(False,None\)

11

12CLAUSE:Marybuyssomeapples

13RESPONSE:\(False,None\)

14

15CLAUSE:Maryleavestheapple

16RESPONSE:\(True,’needscontext’\)

17

18CLAUSE:LindakillsMary

19RESPONSE:\(True,’unusual’\)

20

21CLAUSE:Marysingsatthecompetition

22RESPONSE:\(False,None\)

23

24CLAUSE:Marycomesover

25RESPONSE:\(True,’vague’\)

26

27CLAUSE:MarycomesovertoLinda’shouse

28RESPONSE:\(False,None\)

29

30CLAUSE:Marysmellsthenail

31RESPONSE:\(False,’needscontext’\)

32

33CLAUSE:Marysmellstheflower

34RESPONSE:\(True,None\)

35

36CLAUSE:MarytellsLinda

37RESPONSE:\(True,’vague’\)

38

39CLAUSE:MarystealsLinda’schild

40RESPONSE:\(True,’unusual’\)

41

42CLAUSE:MarycompetesagainstLinda

43RESPONSE:\(False,None\)

44

45CLAUSE:Marybeatsherswordsintoploughshares

46RESPONSE:\(True,’abnormal’\)

47

48CLAUSE:Lindahitstheapple

49RESPONSE:\(True,’abnormal’\)

50

51CLAUSE:Maryeatslunch

52RESPONSE:\(False,None\)

53

54CLAUSE:Marygetsonthebus

55RESPONSE:\(False,None\)

56

57CLAUSE:Maryfindsoutherbiologicaldadisnothers

58RESPONSE:\(True,’unusual’\)

59

60CLAUSE:\{\}

61RESPONSE:

Listing 2:classify\-hard

### B\.3Typical Follow\-ups \(TypicFollowUps\)

We promptGPT\-4oto generate up to three TypicFollowUps: a state of being, subaction, and next\-action \(see[Table4](https://arxiv.org/html/2610.33239#A2.T4)for examples\)\. This approach increases the diversity of our dataset compared to[Kravtchenko and Demberg](https://arxiv.org/html/2610.33239#bib.bib25)’s \([2015](https://arxiv.org/html/2610.33239#bib.bib25)\) corpus, which consists of mainly subaction or next\-action tuples\. Although some state\-of\-being utterances may not be considered ‘events,’ for simplicity, we use the term ‘event’ throughout\. We allow the model to return None at its discretion \(\)\.

1Youareanexpertannotatorofnaturallanguage\.Youwillbepresentedwithaclause,whichispresumablyabouttakinganactionorbeingataspecificplaceorevent\.

2Generatethreepotentialnextclauses:Astateofbeing,asub\-action,andasubsequentaction\.

3

4Criteriaforallclauses:

5\-Theclausemustbeaboutanaturalassumptiongiventheoriginalclausewhichfeelsobviousandredundant\.

6\-Theclausemustnotbeadirectentailmentoftheoriginalclause

7\-Theclausemustnotbeaparaphraseoftheoriginalclause

8\-Theclausecanbeaboutthesameentityasthatintheoriginal,oraboutanotherentityassociatedwiththeoriginal\.

9\-Theclauseyougenerateshouldbebrief\.Donotaddextramodifiersordetailstoit\.

10

11Criteriaforthestateofbeingclause:

12\-Thestateofbeingmustnotbeapresuppositionoftheoriginalclause\.

13\-Itmustnotbedirectlyinferablefromtheoriginalclause\.Inotherwords,therecannotbealogicalentailmentrelationship\.

14

15Criteriaforthesub\-actionclause:

16\-Itmustnotbeadirectentailmentoftheoriginalclause\.

17\-Itmustnotbeaparaphraseofthesameactionintheoriginalclause\.

18\-Itmightnotbepossibletogenerateagoodsub\-actionclause\.Forexample,whentheoriginalclausecontainsnoaction,orcontainsanactionthatistooshorttocontainasub\-action,itmaynotbepossible\.

19Criteriaforthesubsequentactionclause:

20\-Itmustbeaveryobviousassumptiongiventheoriginalclause,tothepointthatitseemsredundanttoexplicitlymentioninnaturalspeech\.

21\-Itshouldbeasgeneralorvagueaspossible,withnodetailsthatwouldmakeituniqueorimportanttomentioninreallife\.

22Yourresponseformatshouldbeathree\-itemtupleof\(stateofbeing,sub\-action,subsequentaction\)\.Eachiteminthetupleisoptional\.Whenthereisnogoodclauseforthatcategory,respondwithNoneinthatindexofthetuple\.Iftherearenogoodclausestogenerateatall,respondwith\(None,None,None\)\.CLAUSE:Marygoestotherestaurant

23RESPONSE:\("Maryishungry",None,"Maryeats"\)

24CLAUSE:Marygoestoapicnic

25RESPONSE:\("Marywantsto","Marybringsablanket","Maryeats"\)

26CLAUSE:Marysingsatthecontest

27RESPONSE:\(None,"Thecrowdnotices","Maryreceivesapplause"\)

28CLAUSE:Lindabuyscandyatthestore

29RESPONSE:\("Lindahasmoneyonher","Lindapaysthecashier","Lindadoesnotthrowthecandyaway"\)

30CLAUSE:Lindaattendsaconcert

31RESPONSE:\("Lindaknowstheband","Lindasingsalong","Lindaclapsattheend"\)

32CLAUSE:Maryattendsthetalk

33RESPONSE:\("Maryknowswhothespeakeris","Marypaysattention","Marydoesnotregretit"\)

34CLAUSE:Marydriveshomeafteralongdayofwork

35RESPONSE:\(None,"Maryhasbothhandsonthewheel","Maryparkswithouttrouble"\)

36CLAUSE:MarywavesherhandsinLinda’sface

37RESPONSE:\("MaryknowsLinda",None,"Lindanotices"\)

38CLAUSE:Marydoesthereadingfornextweek’smathclass

39RESPONSE:\("Maryisenrolledintheclass","Maryabsorbssomeofit","Maryattendsthenextclass"\)

40CLAUSE:\{\}

41RESPONSE:

Listing 3:gen3clause##### LLM\-based Criteria Filtering

Table 5:27 LLM\-filtering prompts and their classes \(Unnatural for dialogue vs\. Wrong typicality\)[Table5](https://arxiv.org/html/2610.33239#A2.T5)describes each prompt we use and links to our specific prompts in[AppendixH](https://arxiv.org/html/2610.33239#A8)\. Note that even if, during the prompting process, an LLM’s answer warrants rejection of a follow\-up, we still prompt it with all 27 prompts\. This is to a\) allow us to go back and change individual prompts if we want to and b\) remove pairs of answers when they contradict each other\.

Specifically, a logical LLM should never answer entailment \(too likely,[§H\.2](https://arxiv.org/html/2610.33239#A8.SS2)\) prompts and Expectedness \(not likely enough\) \([§H\.3](https://arxiv.org/html/2610.33239#A8.SS3)\) prompts the same way\. So, when this happens, we remov the LLMs’ answers to both of these sets of prompts\.

Similarly, we had originally tried Explanation \(\) and Result \(,\) prompts\. However, we find that the LLM is especially bad at assessing causality, and answers the three prompts extremely inconsistently\. Since almost every set of answers contradicts each other, the prompts have no effect in the end\.

##### Generating typicality questions used for TypicFollowUp filtering

For the prompt we used to filter out \(Context, TypicFollowUp\) pairs which had no typicality \(\), we input a question about typicality to the LLM and ask it if the question makes sense \(\)\. In order to do this, we first separately prompt the LLM to generate the typicality question in the form of: “When \{Context\}, how often \{TypicFollowUp\}?” We find that this question format forces a frequency interpretation unlike the “In general, when…how likely” format used in our later experiments, and this frequency interpretation makes the question sound infelicitous when the event has no typicality\. So, that is why we choose this question format\.

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadSTATEMENT\_ONEandSTATEMENT\_TWO\.

3YourtaskistoproduceaQUESTIONintheformatof"When\[STATEMENT\_ONE\],howoften\[STATEMENT\_TWO\]?

4WhenSTATEMENT\_ONEuses’the’tomodifyitsobject,pleaseswitchtousing’a’inthefirstclauseofyourgeneratedQUESTION\.

5ButifSTATEMENT\_TWOuses’the’tomodifyitsobject,pleaseuse’the’inthesecondclauseoftheQUESTION\.

6

7STATEMENT\_ONE:WhenMaryfindsthestraydog

8STATEMENT\_TWO:Marypetsit

9QUESTION:WhenMaryfindsastraydog,howoftendoesshepetit?

10STATEMENT\_ONE:WhenMaryfindsastraydog

11STATEMENT\_TWO:Maryisananimallover

12QUESTION:WhenMaryfindsastraydog,howoftenissheananimallover?

13STATEMENT\_ONE:Marybuildsthehouse

14STATEMENT\_TWO:Marytakesawholeyear

15QUESTION:WhenMarybuildsahouse,howoftendoesshetakeawholeyear?

16STATEMENT\_ONE:MarybuildsLinda’shouse

17STATEMENT\_TWO:Lindamovesin

18QUESTION:WhenMarybuildsLinda’shouse,howoftendoesLindamovein?

19STATEMENT\_ONE:Marymakescookies

20STATEMENT\_TWO:Marytakesthecookiestotheoffice

21QUESTION:WhenMarymakescookies,howoftendoesshetakethecookiestotheoffice?

22STATEMENT\_ONE:Marymakesfriendshipbracelets

23STATEMENT\_TWO:Maryusesmorethanonebeadshape

24QUESTION:WhenMarymakesfriendshipbracelets,howoftendoessheusemorethanonebeadshape?

25STATEMENT\_ONE:Marymakesfriendshipbracelets

26STATEMENT\_TWO:Maryenjoyscraftingasahobby

27QUESTION:WhenMarymakesfriendshipbracelets,howoftendoessheenjoycraftingasahobby?

28STATEMENT\_ONE:Maryseesshoesforsale

29STATEMENT\_TWO:Marybuysthem

30QUESTION:WhenMaryseesshoesforsale,howoftendoesshebuythem?

31STATEMENT\_ONE:Maryseesshoesforsale

32STATEMENT\_TWO:Maryisinterestedinfashion

33QUESTION:WhenMaryseesshoesforsale,howoftenissheinterestedinfashion?

34STATEMENT\_ONE:\{\}

35STATEMENT\_TWO:\{\}

36QUESTION:

Listing 4:Prompt used to generate the “When…how often” typicality question for the LLM\-filtering\.

### B\.4Atypical Follow\-ups \(AtypicFollowUps\)

##### Generation

Below are the prompts \(for states,for subactions, andfor next\-actions\) used for AtypicFollowUp generation\. Both the state and subaction prompts originally had errors where a few of the examples were not joined properly with the newline character, and this could have degraded LLM performance due to tokenization problems\. However, since all generated AtypicFollowUps were manually filtered, this should not have caused any downstream problems\.

1Youwillbepresentedwithtwoutterances,ORIGINALandSTATE\.Theyappearinasequence,whereORIGINALiswhatapersondoes,andSTATEdescribesacorrespondingstateofbeing\.

2YourtaskistogenerateyourversionofaSTATEutterance\(STATE\_YOURS\)\.

3STATE\_YOURSshouldbetemporary:thatis,itcannotbesomethingthatusuallylastsforweeks,months,oryears\.

4Examplesofsomenon\-temporarystatestoavoidgenerating:Maryisawoman,Maryhasacar,Maryhasadisability\.

5Themostnotableexceptiontothisruleisthestateofknowing\.Forexample,in’Maryknowstheband,’theconceptofknowingabandcanbeappliedtomanyotherbands,soitcanapplytomultiplesituationsinaperson’slife\.

6Inadditiontobeingtemporaryinaperson’slife,makesureSTATE\_YOURSismorespecificthanSTATE\.ItalsomustbeasclosetotheoriginalSTATElengthaspossible\.Itisacceptabletomakeitjustonewordlonger,onewordshorter,orthesamelength\.

7

8ORIGINAL:Marygoestotherestaurant

9STATE:Maryishungry

10STATE\_YOURS:Maryisravenous

11ORIGINAL:Marygoestoapicnic

12STATE:Marywantsto

13NEXT\_YOURS:Maryisstarving

14ORIGINAL:Marysingsatthecontest

15STATE:Marysoundsgood

16STATE\_YOURS:Marysoundsmagnificent

17ORIGINAL:Lindabuyscandyatthestore

18STATE:Lindahasmoneyonher

19STATE\_YOURS:Lindahasacraving

20ORIGINAL:Lindaattendsaconcert

21STATE:Lindaknowstheband

22STATE\_YOURS:Lindaknowseverysong

23ORIGINAL:Maryattendsthetalk

24STATE:Maryknowswhothespeakeris

25STATE\_YOURS:Maryknowsthespeaker’smentor

26ORIGINAL:Marydriveshomeafteralongdayofwork

27STATE:Maryhasherhandsonthewheel

28STATE\_YOURS:Maryhasplanstowatchamovie

29ORIGINAL:Marydoesthereadingfornextweek’smathclass

30STATE:Maryisenrolledintheclass

31STATE\_YOURS:Maryisboredwithit

32ORIGINAL:\{\}

33STATE:\{\}

34STATE\_YOURS:

Listing 5:Prompt to generate the AtypicFollowUp state events\.1Youwillbepresentedwithtwoutterances,ORIGINALandSUBACTION\.Theyappearinasequence,whereORIGINALiswhatapersondoes,andSUBACTIONdescribesacorrespondingsubaction\.

2YourtaskistogenerateyourversionofaSUBACTIONutterance\(SUBACTION\_YOURS\)\.MakeitmoreinterestingthanSUBACTION\.

3ButtrytokeepitasclosetotheoriginalSUBACTIONlengthaspossible\.Itisacceptableifyoumakeitjustonewordlonger,onewordshorter,orthesamelength\.

4

5ORIGINAL:Marygoestotherestaurant

6SUBACTION:Marydrivesthere

7SUBACTION\_YOURS:Maryrunslate

8ORIGINAL:Marygoestoapicnic

9SUBACTION:Marybringsablanket

10SUBACTION\_YOURS:Marybringsmayonnaise

11ORIGINAL:Marysingsatthecontest

12SUBACTION:Thecrowdnotices

13SUBACTION\_YOURS:Thecrowdgoeswild

14ORIGINAL:Lindabuyscandyatthestore

15SUBACTION:Lindapaysthecashier

16SUBACTION\_YOURS:Lindausescoupons

17ORIGINAL:Lindaattendsaconcert

18SUBACTION:Lindasingsalong

19SUBACTION\_YOURS:Lindascreamsalong

20ORIGINAL:Maryattendsthetalk

21SUBACTION:Marypaysattention

22SUBACTION\_YOURS:Maryfallsasleep

23ORIGINAL:Marydriveshomeafteralongdayofwork

24SUBACTION:Marykeepsherhandsonthewheel

25SUBACTION\_YOURS:Marygetscaughtintraffic

26ORIGINAL:MarywavesherhandsinLinda’sface

27SUBACTION:Lindanoticesit

28SUBACTION\_YOURS:Lindaignoresit

29ORIGINAL:Marydoesthereadingfornextweek’smathclass

30SUBACTION:Maryabsorbssomeofit

31SUBACTION\_YOURS:Maryabsorbsnothing

32ORIGINAL:\{\}

33SUBACTION:\{\}

34SUBACTION\_YOURS:

Listing 6:Prompt to generate the AtypicFollowUp subaction events\.1Youwillbepresentedwithtwoutterances,ORIGINALandNEXT\.Theyappearinasequence,whereORIGINALiswhatapersondoes,andNEXTiswhathappensnext\.

2YourtaskistogenerateyourversionofaNEXTutterance\(NEXT\_YOURS\)\.MakeitmorespecificthanNEXT\.

3ButtrytokeepitasclosetotheoriginalNEXTlengthaspossible\.It’sbestifyoudon’tmakeitanylonger,butitisacceptableifyoumakeitjustonewordlonger\.

4

5ORIGINAL:Marygoestotherestaurant

6NEXT:Maryeatsfood

7NEXT\_YOURS:Maryeatspizza

8ORIGINAL:Marygoestoapicnic

9NEXT:Maryeatsfood

10NEXT\_YOURS:Maryeatstwosandwiches

11ORIGINAL:Marysingsatthecontest

12NEXT:Theaudienceclapsattheend

13NEXT\_YOURS:Marygetsthirdplace

14ORIGINAL:Lindabuyscandyatthestore

15NEXT:Lindapaysthecashier

16NEXT\_YOURS:LindagivessometoMary

17ORIGINAL:Lindaattendsaconcert

18NEXT:Lindaleaveswhenitends

19NEXT\_YOURS:Lindaleaveslate

20ORIGINAL:Maryattendsthetalk

21NEXT:Marydoesnotregretit

22NEXT\_YOURS:Marytalkstothespeaker

23ORIGINAL:Marydriveshomeafteralongdayofwork

24NEXT:Maryparkswithouttrouble

25NEXT\_YOURS:Maryheatsupleftovers

26ORIGINAL:MarywavesherhandsinLinda’sface

27NEXT:Lindanoticesit

28NEXT\_YOURS:Lindaisannoyed

29ORIGINAL:Marydoesthereadingfornextweek’smathclass

30NEXT:Maryattendsthenextmathclass

31NEXT\_YOURS:Marybarelyunderstandsit

32ORIGINAL:\{\}

33NEXT:\{\}

34NEXT\_YOURS:

Listing 7:Prompt to generate the AtypicFollowUp next\-events\.
##### Manual validation, correction, & removal

We choose to manually validate our generated AtypicFollowUps in order to minimize error propagation \(as the AtypicFollowUps were generated from already LLM\-filtered Context\-TypicFollowUp pairs\)\. Since we have close to 4,000 generated events \(as opposed to over 12,000 or TypicFollowUps\), one of the paper authors had enough time to do the validation\.

During the AtypicFollowUp validation, we find that some of the TypicFollowUps that had passed earlier filtering had errors\. So, in those cases, we also correct the TypicFollowUps when possible\.

But sometimes, it isn’t possible to correct the TypicFollowUp, because theContextit depends on was problematic \(e\.g\. the filtering from[§B\.2](https://arxiv.org/html/2610.33239#A2.SS2)had not filtered it out\)\. In those cases, we just remove the tuple entirely\. We find some problematic cases during the later manual validation stage \(see[§D\.1](https://arxiv.org/html/2610.33239#A4.SS1)\) and remove them retroactively\. In our released data, we trace each generation and filtering stage and provide a list of which tuples got retroactively removed\.

### B\.5Data Sample

After the AtypicFollowUp validation and filtering, we are left with 4,113 tuples\. Since we had originally generated up to 3 follow\-ups per Context \(state/subaction/next\-action\), there are some duplicate Contexts\. We sample so that each Context is unique, oversampling states and subactions to try to alleviate imbalances in the original data, but the resulting sample still has many more next\-actions than the other two categories\. We also useGPT\-4oto extract the main ‘action’ from each Context utterance and further deduplicate those, only keeping one Context per action\. Nonetheless, this action extraction was not strict, so some Context inEventTypare still about the same topic, e\.g\. “camp with friends” and “go camping” might both be kept\. The result is 2,330 tuples, upon which we add the 24 from[Kravtchenko and Demberg \(2015\)](https://arxiv.org/html/2610.33239#bib.bib25)\.

## Appendix CHuman and LLM Details

### C\.1Human Participant Details

Table 6:Attention check questions\. These qualify as ‘nonsensical items’ per Prolific’s attention check policy\.Below are more details about our human participants\. All participants were US\-based, had at least a 98% approval rating on Prolific, had at least 100 previous submissions, and stated that their first and primary language were both English\. Our aim was to pay equivalent to $12\.50/hr, but since we paid per task and not per hour, the estimation was not perfectly accurate, and we averaged out to paying more\.

[Table6](https://arxiv.org/html/2610.33239#A3.T6)shows our exact attention check questions for both the Commonsense Knowledge and the Event\-Typicality Inference experiments\. Before deciding on them, we informally piloted each one in order to ensure that any participant who was paying attention could answer them correctly\. We format the questions to blend in with the rest of the items in their corresponding study\. In each study, all participants see the exact same attention checks\.

##### Commonsense Knowledge Experiment

We recruited 354 participants in total, and we required 330 complete submissions \(3 \* 2 pilot batches of 24, 3 \* 108 larger batches of 44\.\) Each batch also included 6 attention checks \(see[§C\.1](https://arxiv.org/html/2610.33239#A3.SS1)\), so we had 30 questions for the pilot batches and 50 for the rest\.

340 were approved, 12 returned, 1 timed out, and 1 was rejected\. Out of the 340 approved, 332 had usable submissions \(i\.e\., passed the attention checks\); however, due to a an error in our processing, we overwrote data from 2 participants and had to recruit 2 replacements\. For this study, we had auto\-approve enabled, and therefore were only able to reject one participant who failed attention checks but was flagged by Prolific as having completed the study too quickly\. This means 9 people total failed the attention checks, but 8 were approved due to their responses being completed in a normal amount of time and therefore not being flagged by the Prolific system\.

##### Event\-Typicality Inference Experiment

For this experiment, we only recruited participants who had passed a pragmatics pre\-test \(see[§E\.4](https://arxiv.org/html/2610.33239#A5.SS4)for details\)\. Due to batching, each participant answered between 30 and 32 questions, consisting of about 24 questions fromEventTypGoldand 6 attention checks\. All participants see the same attention checks\. We randomized and divide the data so that each question is answered by 5 different participants and no participant sees any Context utterance more than once\.

For our data collection to be complete, we needed responses from 540 participants who passed our attention checks\. So, our policy was to recruit replacements for any participants who failed the attention checks\. Out of the 676 possible participants, 634 attempted the main experiment\. Of these attempts, 575 resulted in approval,999Includes the 35 who got 4/6 of the attention checks correct, a failing score but not enough for us to reject\.39 resulted in a return, 19 resulted in rejection, and 1 timed out\. Our pay averaged $21\.50 over all rounds of the experiment\.

### C\.2LLM Details

Table 7:LLMs used\. For those where prompting was available, we tried prompting both with and without CoT \(see[§C\.2\.2](https://arxiv.org/html/2610.33239#A3.SS2.SSS2)\. For details on model version selection, see[§C\.2\.1](https://arxiv.org/html/2610.33239#A3.SS2.SSS1)\. Since OpenAI has not made GPT\-5\.4’s parameter number publicly available, we just put the timestamp model IDs\.[Table7](https://arxiv.org/html/2610.33239#A3.T7)describes which LLMs we choose to use, as well as their sizes and whether we used the base versions, instruct versions, or both\. We do not change the model parameters\. Since our inference is performed on so little data \(N = 4,708\) for the Commonsenes Experiment and \(N = 2,700\) for the Event\-Typicality Inference experiment, it takes about 5 minutes for us to run Prompting inference on NVIDIA A100 GPUs and about 30 minutes for CoT\-Prompting\. For Surprisal, it usually took between 5\-30 minutes, as we did not optimize with vLLM and directly used the transformers library\.

#### C\.2\.1LLM version selection

Our goal is to keep each LLM configuration as consistent as possible across open\-weight LLMs\. This means our goal is to select models where both a base and instruction\-tuned version exist, and for the instruction\-tuned version, we wanted to avoid using Thinking so we could use the CoT prompting approach across models\. Therefore, we made two decisions for Olmo\-3 and Qwen3 specifically\.

- •Olmo\-3[Olmo et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib32): Olmo\-3 has anInstructmodel for its smaller version, but its larger model \(32B\) uses Thinking which cannot be switched off\. So, for the larger model with prompting, we used Olmo\-3\.1\-Instruct, whose base version is the same as for Olmo\-3\.
- •Qwen3[Yang et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib47): For our instruction\-tuned versions, we chose Qwen3\-30B\-A3B and Qwen3\-8B, but we disabled thinking for both our CoT and non\-CoT prompting configurations\. This is because we wanted a fair comparison with other models with our own CoT prompt \(see[§C\.2\.2](https://arxiv.org/html/2610.33239#A3.SS2.SSS2)andfor an example\)\.

#### C\.2\.2Detailed Setup of Prompting

To create our prompts, we append the text “Choose one of the following options: ‘Very Unlikely’, ‘Unlikely’, ‘Likely’, ‘Very Likely’ to the end of each question inEventTypGold\. We find this to be necessary in order to prevent unwanted bias toward ‘Likely’ that is introduced by the prompt \(“…how*likely*is it …”\)\.

We sample 20 answers per prompt with temperature 1\.0 for all model families exceptGPT\-5\.4, for which we sample 10\. We reverse the response\-option order for half of the samples to minimize order bias\.

We consider with or without CoT for the prompting experiment\. Here, we provide details for each setting, along with an example prompt and the LLM’s response\.

- •Without CoT\.For open\-source models, we force the LLM to directly output the answer by constraining the output space to the four answer options\. ForGPT\-5\.4, we set ‘reasoning\_effort’ to none to disable reasoning\.101010[https://developers\.openai\.com/api/docs/guides/reasoning](https://developers.openai.com/api/docs/guides/reasoning)
- •With CoT\.For open\-source models, we force the LLM to generate reasoning about its response first and then give its final answer\. We achieve this by using “structured output” generation from vLLM\([Kwon et al\., 2023](https://arxiv.org/html/2610.33239#bib.bib28)\)\. ForGPT\-5\.4, we set ‘reasoning\_effort’ to medium\.

1\[Prompt\]ChedelineandKathemarehavingaconversationabouttheirfriendAnahita\.Chedeline:"Anahitathankedherboss\.Andhey,guesswhat,shewasintears\!"Ingeneral,whenAnahitathanksherboss,howlikelyisitthatsheisintears?\\nChooseoneofthefollowingoptions:VeryUnlikely,Unlikely,Likely,VeryLikely\.

2

3\[Response\]"Unlikely"

Listing 8:An example prompt used for the prompting experimentwithoutCoT and an example response \(from Llama\-3\.1\-70B\)\.1\[Prompt\]ChedelineandKathemarehavingaconversationabouttheirfriendAnahita\.Chedeline:"Anahitathankedherboss\.Andhey,guesswhat,shemeantit\!"Ingeneral,whenAnahitathanksherboss,howlikelyisitthatshemeansit?\\nAnswerthequestionwiththefollowingschema:\{"type":"object","properties":\{"reasoning":\{"type":"string","description":"Thestep\-by\-stepreasoningprocessfortheanswer\."\},"answer":\{"type":"string","enum":\["VeryUnlikely","Unlikely","Likely","VeryLikely"\]\}\},"required":\["reasoning","answer"\]\}

2

3

4\[Response\]"\{\\"reasoning\\":\\"ThestatementisthatAnahitathankedherbossandactuallymeantit\.Thetoneoftheconversationsuggestssurprise,implyingthatitisnottypicalforAnahitatogenuinelythankherboss\.\\",\\"answer\\":\\"VeryUnlikely\\"\}"

Listing 9:An example prompt used for the prompting experimentwithCoT and an example response \(from Llama\-3\.1\-70B\)\.

## Appendix DCommonsense Knowledge Experiment

### D\.1Experimental Setup

[Fig\.8](https://arxiv.org/html/2610.33239#A4.F8)shows an example of a likelihood rating question in the user interface format our human participants saw\. We use the prompt into generate the likelihood questions\. Note that the actual raw output is a question in the format of:

> When \{Context\}, how likely \{Follow\-up\}?

We then use rules\-based methods to pre\-pend ‘In general, ’ to the question and lowercase the ‘When\.’ For each resulting question, each human answer validates it and corrects an errors \(e\.g\., pronoun ambiguity\)\.

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadSTATEMENT\_ONEandSTATEMENT\_TWO\.

3YourtaskistoproduceaQUESTIONintheformatof"When\[STATEMENT\_ONE\],howlikelyisitthat\[STATEMENT\_TWO\]?

4WhenSTATEMENT\_ONEuses’the’tomodifyitsobject,pleaseswitchtousing’a’inthefirstclauseofyourgeneratedQUESTION\.

5ButifSTATEMENT\_TWOuses’the’tomodifyitsobject,pleaseuse’the’inthesecondclauseoftheQUESTION\.

6Ifonlyonepersonexistsinthestatements,usetheword’someone’insteadoftheirname\.

7Iftwopeopleexist,usethephrase’oneperson’insteadofthefirstnameand’secondperson’forthesecondname\.

8

9STATEMENT\_ONE:Maryfindsthestraydog

10STATEMENT\_TWO:Marypetsit

11QUESTION:Whensomeonefindsastraydog,howlikelyisitthattheypetit?

12STATEMENT\_ONE:Maryfindsastraydog

13STATEMENT\_TWO:Maryishappy

14QUESTION:Whensomeonefindsastraydog,howlikelyisitthattheyarehappy?

15STATEMENT\_ONE:Marybuildsthehouse

16STATEMENT\_TWO:Marytakesawholeyear

17QUESTION:Whensomeonebuildsahouse,howlikelyisitthattheytakeawholeyear?

18STATEMENT\_ONE:MarybuildsLinda’shouse

19STATEMENT\_TWO:Lindamovesin

20QUESTION:Whenonepersonbuildsasecondperson’shouse,howlikelyisitthatthesecondpersonmovesin?

21STATEMENT\_ONE:Marymakescookies

22STATEMENT\_TWO:Marytakesthecookiestotheoffice

23QUESTION:Whensomeonemakescookies,howlikelyisitthattheytakethecookiestotheoffice?

24STATEMENT\_ONE:Marymakesfriendshipbracelets

25STATEMENT\_TWO:Maryusesmorethanonebeadshape

26QUESTION:Whensomeonemakesfriendshipbracelets,howlikelyisitthattheyusemorethanonebeadshape?

27STATEMENT\_ONE:MarycookspastaforLinda

28STATEMENT\_TWO:Lindaeatsit

29QUESTION:Whenonepersoncookspastaforasecondperson,howlikelyisitthatthesecondpersoneatsit?

30STATEMENT\_ONE:\{\}

31STATEMENT\_TWO:\{\}

32QUESTION:

Listing 10:Prompt used to generate the typicality questions used in the norming study\.![Refer to caption](https://arxiv.org/html/2610.33239v1/figs/norming_screenshot.png)Figure 8:Example of what human participants saw in our Commonsense Knowledge experiment\.![Refer to caption](https://arxiv.org/html/2610.33239v1/figs/mainexp_screenshot.png)Figure 9:Example of what human participants saw during our Event\-Typicality Inference experiment\. Each participant answered between 30 and 32 questions\.

## Appendix EEvent\-Typicality Inference Experiment

[Fig\.9](https://arxiv.org/html/2610.33239#A4.F9)shows an example of a likelihood rating question fromEventTypGoldin the format that our human participants saw\.

### E\.1Data Sample

To create our 450\-tuple set of experimental materials, we sampled fromEventTypGold\. To help narrow down the data, we decided to limit our experiments to context events that happen relatively quickly \(take a full day or less to complete\) and frequently in a person’s life \(at least once a year\)\. So, via manual labeling, we set aside tuples that did not meet these criteria\. However, we still consider these tuples to be high quality and hope that followup experiments can include them all\.

### E\.2Dialogue Scenarios: Name selection

For both the dialogue names and the Mary/Linda name replacements, we choose to use names from non\-English speaking countries that are rare in the United States\. This is to fit with both our human and LLM experiments: we want to minimize the chance the LLMs have seen the exact sentences appear in their training data\. We source names from a rare names list,111111[https://github\.com/sigpwned/popular\-names\-by\-country\-dataset](https://github.com/sigpwned/popular-names-by-country-dataset)dropping names from English\-speaking countries and removing those which were in the top 10,000 names from a US census dataset\.121212[https://github\.com/dxdc/babynames/blob/main/data/all\-names\.csv](https://github.com/dxdc/babynames/blob/main/data/all-names.csv)We balance names by gender and sample from 198 of them\. Note that we use the name ‘Anahita’ in[Fig\.1](https://arxiv.org/html/2610.33239#S1.F1), but the actual nameEventTypis ‘Anohito\.’ Both names are valid spellings of a popular Tajik name; however, ‘Anohito’ can easily be mixed up with a Japanese pronoun\. So, to prevent confusion, we use ‘Anahita\.’

### E\.3More explanation of the snarky label

When a speaker wants to communicate something negative about another person, they may communicate via inference so their on\-record utterance is positive \(“Hey, guess what, she meant it”\) instead of saying something negative directly \(“She usually doesn’t mean it when she says thank you\.”\)\. As an example of a non\-snarky dialogue, consider “Anahita thanked her boss\. And hey, guess what, she was in her boss’s office\!”\. The atypicality inference \(=Anahita usually does not thank her boss while in her boss’s office\) does not imply anything negative about Anahita – maybe her and her boss usually meet outside of that office, but this does not make either of them look bad in any way\. Hence, we would label this utterance asnot snarky\.

##### Relationship with commonsense knowledge & irony

In the context of our research questions, we can define snark as a specific way atypicality inferences can interact with commonsense knowledge to produce irony\.

Commonsense knowledge: In the snark case, the TypicFollowUp event is not only generally typical, but is also an event thatshouldhappen according to some social convention or standard of competence\.

Irony: Verbal irony can be defined as a speaker expressing something by using language that would usually convey the opposite[Haverkate \(1990\)](https://arxiv.org/html/2610.33239#bib.bib13), where the speaker expects the listener to recognize the “critical or mocking attitude” behind their words[Wilson \(2006\)](https://arxiv.org/html/2610.33239#bib.bib46)\.[Sperber et al\. \(1986\)](https://arxiv.org/html/2610.33239#bib.bib42)have connected this “critical or mocking attitude” to utterances which violate listeners’ expectations of informativity, e\.g\. in our case with a speaker mentioning a \(Context, TypicFollowUp\) event pair\. In this paper, we call this “critical or mocking attitude”Snark\. After making an atypicality inference, a listener would only interpret the speaker as being snarky if the speaker’s utterance was about someone fulfilling a social convention or meeting a minimal standard\. e\.g\., the atypicality inference from “Anahita thanked her boss\. And hey, guess what, she meant it\!” conveys that Anahita is usually the opposite of thankful\. In other words, if the atypicality inference can be interpreted as ironic, we label it as snarky\.

### E\.4Pragmatics Pre\-test for Human Participants

![Refer to caption](https://arxiv.org/html/2610.33239v1/figs/pretest_impreq_screenshot.png)Figure 10:Example of an implicit request question participants saw in the pragmatics pre\-test\. The expected answer here is d\)\.![Refer to caption](https://arxiv.org/html/2610.33239v1/figs/pretest_coh_screenshot.png)Figure 11:Example of a coherence question participants saw in the pragmatics pre\-test\. The expected answer here is ‘Incoherent\.’##### Justification and Source data

We choose to focus on participants who we know are capable of making pragmatic inferences\. Therefore, before recruiting participants for this experiment, we conduct an 18\-question pragmatics pre\-test using data fromPragMega, a human\-written multiple\-choice dataset which includes questions regarding multiple pragmatic phenomena[Hu et al\. \(2023\)](https://arxiv.org/html/2610.33239#bib.bib14);[Floyd et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib7)\.

[Floyd et al\. \(2025\)](https://arxiv.org/html/2610.33239#bib.bib7)conclude that pragmatic reasoning can be broken into the understanding of 1\) social conventions \(SocCon\), 2\) intonation, and 3\) world\-knowledge based causal reasoning\. Since the task of understanding atypicality inferences from text\-only data requires both 1\) and 3\), we choose questions from those two categories: 12 SocCon \(half implicatures, half implicit requests\) and 6 commonsense\-knowledge based coherence questions \(half coherent, half incoherent\)\. Half of our SocCon questions have literal answers as the correct answer, and the other half nonliteral\. We release our pre\-test questions to the public\.131313Withheld due to anonymity\.

##### Usage and survey setup

[Fig\.10](https://arxiv.org/html/2610.33239#A5.F10)and[Fig\.11](https://arxiv.org/html/2610.33239#A5.F11)contain examples for an implicit request and coherence questions respectively\. We set up the pre\-test on Prolific and conduct three separate rounds, where the pay is $18\.00 an hour averaged over the rounds\. We give a passing score if a participant gets at least 5 out of 6 answers correct for each group of 6 questions \(implicatures, requests, coherence\) and finishes the test in 7 minutes or less\. We do NOT inform participants that this is a pre\-test, and we pay all participants regardless of their score\. Out of 1,255 total participants, 1,196 are approved, 53 return, and 6 time out\. Among the approved participants, 676 pass \(56\.52%\) and are subsequently invited to the main experiment\.

### E\.5Ordinal Regressions

We use mixed\-effects ordinal regressions to model the rating category \(very unlikely to very likely\) with fixed effects for Follow\-up type \(TypicFollowUp/AtypicFollowUp\) and setting \(NONE/Plain/HeyGW\) and their interaction, along with random effects for Context utterance, and, in the human data case, participant\. The mixed\-effects ordinal regressions were built in R, usingclmmfrom the libraryordinal\. For all regressions, fixed effects were centered and maximal random effect structure was included, as permitted by the data\.

We predict opposite effects of Follow\-up setting for TypicFollowUp \(where an atypicality inference should yield lower ratings for Plain or HeyGW compared to NONE\) and AtypicFollowUp \(where a typicality inference should yield lower ratings for NONE\)\. We test the Snark effect in a furtherFollow\-up×setting×snark\\text\{Follow\-up\}\\times\\text\{setting\}\\times\\text\{snark\}model, where the atypicality inference pattern is predicted to be stronger for snarky scenarios\.

Table 8:Regression output for the main mixed effects model fit on human data \(Follow\-up×setting\\text\{Follow\-up\}\\times\\text\{setting\}\)\.Table 9:Regression output for the snark mixed effects model fit on human data,Follow\-up×setting×snark\\text\{Follow\-up\}\\times\\text\{setting\}\\times\\text\{snark\}\.β\\betazpTypicFollowUpPlain\.NONE0\.6610\.06<<0\.001NONE\.HeyGW0\.648\.77<<0\.001AtypicFollowUpPlain\.NONE1\.8626\.50<<0\.001NONE\.HeyGW\-1\.34\-17\.98<<0\.001TypicFollowUp\-SnarkPlain\.NONE0\.657\.06<<0\.001NONE\.HeyGW0\.817\.17<<0\.001TypicFollowUp\-NoSnarkPlain\.NONE0\.657\.08<<0\.001NONE\.HeyGW0\.545\.64<<0\.001AtypicFollowUp\-SnarkPlain\.NONE1\.8717\.01<<0\.001NONE\.HeyGW\-1\.24\-11\.13<<0\.001AtypicFollowUp\-NoSnarkPlain\.NONE1\.8520\.51<<0\.001NONE\.HeyGW\-1\.43\-14\.15<<0\.001

Table 10:Regression output for the subset mixed effects models on human data\.##### On human responses

We collect 13,500 rating responses\.[Table8](https://arxiv.org/html/2610.33239#A5.T8)reports the regression output for the main model \(Follow\-up×setting\\text\{Follow\-up\}\\times\\text\{setting\}\), and[Table9](https://arxiv.org/html/2610.33239#A5.T9)reports the output for the larger model with snark \(Follow\-up×setting×snark\\text\{Follow\-up\}\\times\\text\{setting\}\\times\\text\{snark\}\)\. The table uses abbreviations for T\(ypicality\) and S\(nark\)\. All interactions in both tables are significant, inviting the critical subset comparisons listed in[Table10](https://arxiv.org/html/2610.33239#A5.T10)\.

[Table10](https://arxiv.org/html/2610.33239#A5.T10)shows the subsequent analyses that test for a main effect ofsettingin each of the relevant subsets\. Regular font indicates a typicality inference, and bold font indicates an atypicality inference\.

Table 11:Summary of mixed effects models fit on LLM output data\. Atypicality and typicality inferences abbreviated to AtypInf and TypInf\. Note that the snark effect is only present if atypicality inferences are made\.
##### On LLM responses

In the same way as for the human data but without the participant ID random effect, we fit mixed effects models for each of the sets of LLM results\. Like for humans, we fit models on the full LLM data\.[Table11](https://arxiv.org/html/2610.33239#A5.T11)summarizes the results\. Only CoT\-Prompting model configurations consistently follow human\-like patterns, 4 out of the 5 making atypicality inferences, typicality inferences, and exhibiting the snark effect\.

Figure 12:GPT\-5\.4comparison of the distribution of responses to NONE vs\. HeyGW settings for TypicFollowUps, with non\-snarky dialogues on the left and snarky dialogues on the right\.Figure 13:Correlation \(p<0\.01\) between Commonsense Knowledge \(CKnowledge\) human\-LLM agreement \(κCombined\\kappa^\{\\text\{Combined\}\},[AppendixG](https://arxiv.org/html/2610.33239#A7)\) and the percentage of Event\-Typicality Inference Experiment HeyGW setting tuples where the LLM made both an atypicality inference \(on the TypicFollowUp\)anda typicality inference \(on the AtypicFollowUp\)\.

### E\.6Additional Figures

As an example,[Fig\.12](https://arxiv.org/html/2610.33239#A5.F12)shows the Snark effect forGPT\-5\.4\(with CoT\), but full results are in[Table11](https://arxiv.org/html/2610.33239#A5.T11)\.

[Fig\.13](https://arxiv.org/html/2610.33239#A5.F13)shows a significant correlation \(R2=0\.235,p<0\.01R^\{2\}=0\.235,p<0\.01\) between higher commonsense knowledge and more event\-typicality inferences, which is in line with our belief that event\-typicality inferences rely on commonsense knowledge\. To get the correlation values, we fit an ordinary least\-squares linear model in R with the commonsense knowledge agreement as a predictor of InfScore \(InfScore∼κ\\text\{InfScore\}\{\\sim\}\\kappa\)\.

Figure 14:Percentage of the time a typicality inference was made on events voted as AtypicFollowUp by LLM \(X\-axis\) vs\. humans \(Y\-Axis\)Figure 15:Percentage of the time an atypicality inference was made on events voted as TypicFollowUp by LLM \(X\-axis\) vs\. humans \(Y\-Axis\)

## Appendix FAdditional Discussion

Here, we further discuss LLMs’ oversensitivity to prompt wording instead of actual event typicality\. Because we found that LLMs and humans do not share the same Commonsense Knowledge, we wondered whether flipping the labels of each event inEventTypGoldto match the LLMs’ voted response typicality \(flipping from TypicFollowUp to AtypicFollowUp and vice versa\) would increase their number of “correct” inferences \(i\.e\. typicality inferences for AtypicFollowUp events and atypicality inferences for TypicFollowUp events\)\. However,[Fig\.14](https://arxiv.org/html/2610.33239#A5.F14)and[Fig\.15](https://arxiv.org/html/2610.33239#A5.F15)show that this does not work: the scores remain about the same whether we use the human knowledge labels or the LLM ones\.

This is not surprising given the lack of consistency illustrated in[Fig\.7](https://arxiv.org/html/2610.33239#S6.F7)\. If the LLMs are responding to each question based on how it’s asked \(with dialogue context vs\. without\) and not based on any underlying knowledge about the events, then changing the event labels would not make a difference\.

## Appendix GMetric definitions

We denote the dataset as𝒟\\mathcal\{D\}with\|𝒟\|=2,354\|\\mathcal\{D\}\|=2,354forEventTypand\|𝒟\|=450\|\\mathcal\{D\}\|=450forEventTypGold\. Letℱ∈\{‘TypF’,‘AtypF’\}\\mathcal\{F\}\\in\\\{\\text\{\`TypF'\},\\text\{\`AtypF'\}\\\}be the Follow\-up \(abbreviated from TypicFollowUp and AtypicFollowUp in order to save space\)\. We collectKKresponses for each datapointiiand Follow\-upℱ\\mathcal\{F\}\. We useRiℱ​\[k\]R^\{\\mathcal\{F\}\}\_\{i\}\[k\]to indicate thekk\-th response \(k=1,…,Kk=1,\\ldots,K\) to the question fromii\-th datapoint \(i=1,…,\|𝒟\|i=1,\\ldots,\|\\mathcal\{D\}\|\)\.

Each responseRiℱ​\[k\]R^\{\\mathcal\{F\}\}\_\{i\}\[k\]takes one of the integer values from\{1,2,3,4\}\\\{1,2,3,4\\\}, which map to the labels ‘Very Unlikely’, ‘Unlikely’, ‘Likely’, and ‘Very Likely’, respectively\. For example,RiAtypF​\[k\]=3R^\{\\text\{AtypF\}\}\_\{i\}\[k\]=3means that thekk\-th response to theii\-th datapoint with a Follow\-up of AtypF is ‘Likely’\. We use two different ways of aggregating responses, which are detailed below\.

Average\-responseμiℱ∈\[1,4\]\\mu^\{\\mathcal\{F\}\}\_\{i\}\\in\[1,4\]is average ofKKresponse values,i\.e\.,

μiℱ,=1K​∑k=1KRiℱ​\[k\]\.\\mu^\{\\mathcal\{F\},\}\_\{i\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}R^\{\\mathcal\{F\}\}\_\{i\}\[k\]\.
Voted\-responseviℱ∈\{−1,1\}v^\{\\mathcal\{F\}\}\_\{i\}\\in\\\{\-1,1\\\}is result of majority\-voting fromKKresponses, specifically,

viℱ=\{1if​voteiℱ\>121if​\(voteiℱ=12\)​and​\(μiℱ≥2\.5\)−1otherwisev^\{\\mathcal\{F\}\}\_\{i\}=\\begin\{cases\}1&\\\!\\\!\\\!\\text\{if \}\\\!\\operatorname\{vote\}^\{\\mathcal\{F\}\}\_\{i\}\>\\frac\{1\}\{2\}\\\\ 1&\\\!\\\!\\\!\\text\{if \}\\\!\\left\(\\operatorname\{vote\}^\{\\mathcal\{F\}\}\_\{i\}=\\frac\{1\}\{2\}\\right\)\\\!\\text\{ and \}\\\!\\left\(\\mu^\{\\mathcal\{F\}\}\_\{i\}\\geq 2\.5\\right\)\\\\ \-1&\\\!\\\!\\\!\\text\{otherwise\}\\end\{cases\}wherevoteiℱ=1K​∑k=1K𝕀⁡\(Riℱ​\[k\]∈\{3,4\}\)\\operatorname\{vote\}^\{\\mathcal\{F\}\}\_\{i\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{I\}\\left\(R^\{\\mathcal\{F\}\}\_\{i\}\[k\]\\in\\\{3,4\\\}\\right\)\. In words,viℱv^\{\\mathcal\{F\}\}\_\{i\}becomes11\(‘Likely’\) if the majority of responses are positive \(3 or 4\), and−1\-1\(‘Unlikely’\) if the majority are negative \(1 or 2\)\. When a 50:50 tie occurs \(i\.e\.,voteiℱ=12\\operatorname\{vote\}^\{\\mathcal\{F\}\}\_\{i\}=\\frac\{1\}\{2\}\), we break the tie using the average response\. Let\{viTypF\}i=1\|𝒟\|\\\{v\_\{i\}^\{\\text\{TypF\}\}\\\}\_\{i=1\}^\{\|\\mathcal\{D\}\|\}be the set of all votes for TypicFollowUps and\{viAtypF\}i=1\|𝒟\|\\\{v\_\{i\}^\{\\text\{AtypF\}\}\\\}\_\{i=1\}^\{\|\\mathcal\{D\}\|\}be the set of all votes for AtypicFollowUps\.

Agreementκ\\kappa: We can now define the following three agreement scores, whereκ\\kappais Cohen’s Kappa[Cohen \(1960\)](https://arxiv.org/html/2610.33239#bib.bib5)\. Note we abbreviateCombinedtoCombhere for space\.

κTypF\\displaystyle\\kappa^\{\\text\{TypF\}\}=κ⁡\(\{viTypF\}i=1\|𝒟\|\),\\displaystyle=\\kappa\\\!\\left\(\\\{v\_\{i\}^\{\\text\{TypF\}\}\\\}\_\{i=1\}^\{\\lvert\\mathcal\{D\}\\rvert\}\\right\),κAtypF\\displaystyle\\kappa^\{\\text\{AtypF\}\}=κ⁡\(\{viAtypF\}i=1\|𝒟\|\),\\displaystyle=\\kappa\\\!\\left\(\\\{v\_\{i\}^\{\\text\{AtypF\}\}\\\}\_\{i=1\}^\{\\lvert\\mathcal\{D\}\\rvert\}\\right\),κComb\\displaystyle\\kappa^\{\\text\{Comb\}\}=κ\(\{\[viTypF=1∧viAtypF=−1\]\}i=1\|𝒟\|\)\\displaystyle=\\kappa\\\!\\left\(\\left\\\{\\mathbf\{1\}\\\!\\left\[v\_\{i\}^\{\\text\{TypF\}\}=1\\wedge v\_\{i\}^\{\\text\{AtypF\}\}=\-1\\right\]\\right\\\}\_\{i=1\}^\{\\lvert\\mathcal\{D\}\\rvert\}\\right\)
Now, let𝒮∈\{‘NONE’, ‘Plain’, ‘HeyGW’\}\\mathcal\{S\}\\in\\\{\\text\{\`NONE', \`Plain', \`HeyGW'\}\\\}be the Follow\-upsettingfrom our Event\-Typicality Inference experiment \(see[Table3](https://arxiv.org/html/2610.33239#S5.T3)\)\. All the definitions from above stay the same, but we additionally useRiℱ,𝒮​\[k\]R^\{\\mathcal\{F\},\\mathcal\{S\}\}\_\{i\}\[k\]\(e\.g\.RiAtypF,Plain​\[k\]=3R^\{\\text\{AtypF\},\\text\{Plain\}\}\_\{i\}\[k\]=3means thekk\-th response to theii\-th datapoint with \(AtypF, Plain\)\), along withμiℱ,𝒮\\mu^\{\\mathcal\{F\},\\mathcal\{S\}\}\_\{i\}andviℱ,𝒮v^\{\\mathcal\{F\},\\mathcal\{S\}\}\_\{i\}\.

Consistency scoreCℱ∈\[0,1\]C^\{\\mathcal\{F\}\}\\in\[0,1\], which is defined for eachℱ\\mathcal\{F\}, is the proportion of datapoints from𝒟\\mathcal\{D\}where the voted\-response in the Commonsense Knowledge experimentviℱv^\{\\mathcal\{F\}\}\_\{i\}matches the voted\-response with NONE setting in the Event\-Typicality Inference experimentviℱ,NONEv^\{\\mathcal\{F\},\\text\{NONE \}\}\_\{i\}\.i\.e\.,

CTypF\\displaystyle C^\{\\text\{TypF\}\}=1\|𝒟\|​∑i=1\|𝒟\|𝕀⁡\(viTypF=viTypF,NONE\),\\displaystyle=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\mathbb\{I\}\\left\(v^\{\\text\{TypF\}\}\_\{i\}=v^\{\\text\{TypF\},\\text\{NONE \}\}\_\{i\}\\right\),CAtypF\\displaystyle C^\{\\text\{AtypF\}\}=1\|𝒟\|​∑i=1\|𝒟\|𝕀⁡\(viAtypF=viAtypF,NONE\)\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{D\}\|\}\\mathbb\{I\}\\left\(v^\{\\text\{AtypF\}\}\_\{i\}=v^\{\\text\{AtypF\},\\text\{NONE \}\}\_\{i\}\\right\)\.
Inference score \(InfScore\)Iiℱ,𝒮Mentioned∈\[−3,3\]I^\{\\mathcal\{F\},\\mathcal\{S\}\_\{\\text\{Mentioned\}\}\}\_\{i\}\\in\[\-3,3\], which is defined for each datapointiiwith someℱ\\mathcal\{F\}, measures the change in average\-response when a MENTIONED setting \(Plain or HeyGW\) is introduced, compared to the baseline setting without followup \(NONE\),i\.e\.,

Iiℱ,𝒮Mentioned=μiℱ,NONE−μiℱ,𝒮Mentioned,I^\{\\mathcal\{F\},\\mathcal\{S\}\_\{\\text\{Mentioned\}\}\}\_\{i\}=\\mu^\{\\mathcal\{F\},\\text\{NONE \}\}\_\{i\}\-\\mu^\{\\mathcal\{F\},\\mathcal\{S\}\_\{\\text\{Mentioned\}\}\}\_\{i\},where𝒮Mentioned∈\{‘Plain’,‘HeyGW’\}\\mathcal\{S\}\_\{\\text\{Mentioned\}\}\\in\\\{\\text\{\`Plain'\},\\text\{\`HeyGW'\}\\\}\. A positive InfScore indicates an atypicality inference, a negative InfScore indicates a typicality inference, and zero indicates that no inference occurred\.

## Appendix HAll LLM\-filtering prompts

### H\.1Abstract

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isthisstatementaboutanabstractaction?Definitionofanabstractaction:anactionthatmayencompassaseriesofundefined,highlycontextual,andpotentiallyabstractsteps\.

6STATEMENT:Maryloseshope

7ANSWER:Yes

8STATEMENT:Marylosesthebet

9ANSWER:No

10STATEMENT:Maryassertsherindependence

11ANSWER:Yes

12STATEMENT:MarygetsLindaajob

13ANSWER:Yes

14STATEMENT:MarygetsLindaaglassofwater

15ANSWER:No

16STATEMENT:MarydrivesLindacrazy

17ANSWER:Yes

18STATEMENT:Marydriveshome

19ANSWER:No

20STATEMENT:Maryappliesherself

21ANSWER:Yes

22STATEMENT:Maryappliesforajob

23ANSWER:No

24STATEMENT:\{Context\}

25ANSWER:

Listing 11:Abstract\-Direct: Applies to only Context utterances\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5

6QUESTION:Doesthisstatementcontainanyidioms?

7STATEMENT:Maryreinventsthewheel

8ANSWER:Yes

9STATEMENT:Maryeatsanapple

10ANSWER:No

11STATEMENT:\{Context\}

12ANSWER:

Listing 12:Abstract\-Idiom: Applies to only Context utterances\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Doeswhatisexpressedinthisstatementhappensolelyinsomeone’shead?Asin,it’stheirthoughtortheirfeeling?

6STATEMENT:Maryfeelsasharpsting

7ANSWER:Yes

8STATEMENT:MaryrealizesLinda’serror

9ANSWER:Yes

10STATEMENT:Maryconsidersbuyingthem

11ANSWER:Yes

12STATEMENT:Maryjumps

13ANSWER:No

14STATEMENT:\{Context\}

15ANSWER:

Listing 13:Abstract\-Imagine: Applies to only Context utterances\.
### H\.2Entailment or near\-entailment

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:DoesstatementAentailstatementB?

6STATEMENT\_A:Marymakesareservation

7STATEMENT\_B:Maryprovidesatime

8ANSWER:Yes

9STATEMENT\_A:Marymakesareservation

10STATEMENT\_B:Maryplanstogo

11ANSWER:No

12STATEMENT\_A:MaryinvitesLindatocometoherparty

13STATEMENT\_B:Maryishavingaparty

14ANSWER:Yes

15STATEMENT\_A:MaryinvitesLindatocometoherparty

16STATEMENT\_B:Lindaacceptstheinvitation

17ANSWER:No

18STATEMENT\_A:\{Context\}

19STATEMENT\_B:\{Event\}

20ANSWER:

Listing 14:Entail\-Direct: Applies to all Context\-Event pairs\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:IsdeathortheapocalypsetheonlywayyoucanimaginestatementBbeingfalse?\-Fallingasleep\\rightarrowWakingup,\-Goingsomewhere\\rightarrowLeaving,\-Doinganactionthatalwayshassomekindofend\\rightarrowItending

6STATEMENT\_A:MaryvisitsLinda’sfather

7STATEMENT\_B:Maryleaves

8ANSWER:Yes

9STATEMENT\_A:MaryvisitsLinda’sfather

10STATEMENT\_B:Linda’sfatherrecognizesher

11ANSWER:No

12STATEMENT\_A:Marytakesahike

13STATEMENT\_B:Maryreturnshome

14ANSWER:Yes

15STATEMENT\_A:Marytakesahike

16STATEMENT\_B:Maryfollowsthetrail

17ANSWER:No

18STATEMENT\_A:Marygoestosleep

19STATEMENT\_B:Marywakesup

20ANSWER:Yes

21STATEMENT\_A:Marycoversherhead

22STATEMENT\_B:Maryuncoversherheadlater

23ANSWER:Yes

24STATEMENT\_A:\{Context\}

25STATEMENT\_B:\{Event\}

26ANSWER:

Listing 15:Entail\-Death: Applies to all Context\-Event pairs\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:CanthemainpointofstatementBbeseenasapartofthedefinitionofanactioninstatementA?Notethatif,accordingtoitsdefinition,statementAcanhappenwithoutstatementBoccurring,youranswershouldbeNo\.

6STATEMENT\_A:Maryplayschess

7STATEMENT\_B:Marymakesamove

8ANSWER:Yes

9STATEMENT\_A:Maryplayschess

10STATEMENT\_B:Marywins

11ANSWER:No

12STATEMENT\_A:Marybuysgroceries

13STATEMENT\_B:Maryselectsfooditems

14ANSWER:Yes

15STATEMENT\_A:Marybuysgroceries

16STATEMENT\_B:Maryselectsapples

17ANSWER:No

18STATEMENT\_A:MaryfollowsLinda’ssteps

19STATEMENT\_B:MaryreachesthesamedestinationasLinda

20ANSWER:Yes

21STATEMENT\_A:\{Context\}

22STATEMENT\_B:\{Event\}

23ANSWER:

Listing 16:Entail\-Definition: Applies to all Context\-Event pairs\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:CanstatementBbeseenasaparaphraseofstatementA?

6STATEMENT\_A:Marygetsaticket

7STATEMENT\_B:Marytakestheticket

8ANSWER:Yes

9STATEMENT\_A:Marygetsaticket

10STATEMENT\_B:Marygoestotheevent

11ANSWER:No

12STATEMENT\_A:\{Context\}

13STATEMENT\_B:\{Event\}

14ANSWER:

Listing 17:Entail\-Para: Applies to all Context\-Event pairs\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:IsitextremelyimplausibleforstatementBtobefalsegivenstatementA?Inotherwords,doesittakementalgymnasticstoimaginestatementBbeingfalse?

6STATEMENT\_A:MaryinvitesLindatojoin

7STATEMENT\_B:Maryhasplans

8ANSWER:Yes

9STATEMENT\_A:MaryinvitesLindatojoin

10STATEMENT\_B:Lindaacceptstheinvitation

11ANSWER:No

12STATEMENT\_A:Marymakeschocolatechipcookies

13STATEMENT\_B:Marymixestheingredients

14ANSWER:Yes

15STATEMENT\_A:Marymakeschocolatechipcookies

16STATEMENT\_B:Marydoublestherecipe

17ANSWER:No

18STATEMENT\_A:MaryaccusesLindaofcheating

19STATEMENT\_B:Lindaresponds

20ANSWER:Yes

21STATEMENT\_A:MaryaccusesLindaofcheating

22STATEMENT\_B:Lindadeniesit

23ANSWER:No

24STATEMENT\_A:Marywatchesthefilm

25STATEMENT\_B:Thecreditsroll

26ANSWER:Yes

27STATEMENT\_A:\{Context\}

28STATEMENT\_B:\{Event\}

29ANSWER:

Listing 18:Entail\-Plausible: Applies to all Context\-Event pairs\.
### H\.3Not sufficiently likely

Note that the prompt may look strange for\. This is the result of iterative prompt engineering to get the best result\. Previously, more intuitive prompts resulted in the LLM rejecting almost everything or almost nothing\.

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:GivenstatementA,howsurprisedareyouthatstatementBistrue?Onascaleof0to10,where10isthemostsurprised,ifyoursurprisalis4orless,answerYes\.Otherwise,answerNo\.

6STATEMENT\_A:MarygivesLindaasmallsmile

7STATEMENT\_B:Lindaacknowledgesit

8ANSWER:Yes

9STATEMENT\_A:MarygivesLindaasmallsmile

10STATEMENT\_B:Maryisinajoyfulmood

11ANSWER:No

12STATEMENT\_A:Marypicksberries

13STATEMENT\_B:Marytakesthemhome

14ANSWER:Yes

15STATEMENT\_A:Marypicksberries

16STATEMENT\_B:Maryisinaforest

17ANSWER:No

18STATEMENT\_A:Marydecidestotryoutfortheteam

19STATEMENT\_B:Maryplanstoattendpractice

20ANSWER:Yes

21STATEMENT\_A:Marydecidestotryoutfortheteam

22STATEMENT\_B:Maryfillsoutanapplication

23ANSWER:No

24STATEMENT\_A:MarygetsLindaajob

25STATEMENT\_B:MarythanksLinda

26ANSWER:Yes

27STATEMENT\_A:MarygetsLindaajob

28STATEMENT\_B:MarylovesLinda

29ANSWER:No

30STATEMENT\_A:Marycatchesthedog

31STATEMENT\_B:Marymakessureitgetshome

32ANSWER:Yes

33STATEMENT\_A:Marycatchesthedog

34STATEMENT\_B:Maryclingstothedog

35ANSWER:No

36STATEMENT\_A:\{Context\}

37STATEMENT\_B:\{Event\}

38ANSWER:

Listing 19:Expect\-Direct: Applies to all Context\-Event pairs\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:DoesstatementBaddunexpectedinformationthatisnotcontainedinstatementA?Themostcommonwayofaddingunexpectedinformationistheadditionofextraentitiesorcharactersintothesituation\.Intheprizeexample,youcansee’judges’beingadded,wheretheydidnotexistandwerenotautomaticallyinferablefromstatementA\.Inthedrinkexample,somevaguepersonwhoneedsanapologyhasbeenadded\.

6STATEMENT\_A:Marywinstheprize

7STATEMENT\_B:Marythanksthejudges

8ANSWER:Yes

9STATEMENT\_A:Marywinstheprize

10STATEMENT\_B:Maryishappy

11ANSWER:No

12STATEMENT\_A:Maryspillsadrink

13STATEMENT\_B:Maryapologizes

14ANSWER:Yes

15STATEMENT\_A:Maryspillsadrink

16STATEMENT\_B:Marycleansitup

17ANSWER:No

18STATEMENT\_A:\{Context\}

19STATEMENT\_B:\{Event\}

20ANSWER:

Listing 20:Expect\-Extra: Applies to all Context\-Event pairs\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:DoesstatementBinterpretstatementAinanon\-defaultway?Themostcommonwaytotakeaphotoistouseasmartphone,inwhichcasestoringthephotoandadjustingthefocusshouldhappenautomatically\.So,statementBseemstobeinterpretingstatementAasbeingaboutarealcamera,whichisnon\-default\.

6STATEMENT\_A:Marytakesaphoto

7STATEMENT\_B:Marystoresthephoto

8ANSWER:Yes

9STATEMENT\_A:Marytakesaphoto

10STATEMENT\_B:Maryadjuststhefocus

11ANSWER:Yes

12STATEMENT\_A:Marytakesaphoto

13STATEMENT\_B:Maryusesflash

14ANSWER:No

15STATEMENT\_A:\{Context\}

16STATEMENT\_B:\{Event\}

17ANSWER:

Listing 21:Expect\-Nondefault: Applies to all Context\-Event pairs\.
### H\.4Malformed context

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isthisstatementungrammatical,misspelled,malformed,orotherwisenonsensical?

6STATEMENT:MarylosesLindaarm

7ANSWER:Yes

8STATEMENT:MarylosesLinda’skeys

9ANSWER:No

10STATEMENT:Themallshopsback

11ANSWER:Yes

12STATEMENT:Marybuysmanyitems

13ANSWER:No

14STATEMENT:Marygoeshoppingatthemall

15ANSWER:Yes

16STATEMENT:Marygoesshoppingatthemall

17ANSWER:No

18STATEMENT:\{Context\}

19ANSWER:

Listing 22:Malformed\-A\-Base: Applies to all Context utterances\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isthisstatementveryawkwardtosaybecauseitgivesunnecessaryinformation?Forexample,shoesalwaysgoesonfeet,soitisunnecessarytomentionfeetifsomeoneisputtingshoeson\.Similarly,lunchisalwaysfood,soitisunnecessarytomentionthatsomeonemadefoodforlunch\.Peoplegrabthingswiththeirhands,soitisunnecessarytomentiontheuseofhandsforgrabbingsomething\.

6STATEMENT:Maryputsshoesonherfeet

7ANSWER:Yes

8STATEMENT:Maryputsshoeson

9ANSWER:No

10STATEMENT:Marymakesfoodforlunch

11ANSWER:Yes

12STATEMENT:Marymakeslunch

13ANSWER:No

14STATEMENT:Marygrabsthecupwithherhand

15ANSWER:Yes

16STATEMENT:Marygrabsthecup

17ANSWER:No

18STATEMENT:\{Context\}

19ANSWER:

Listing 23:Malformed\-A\-Redundant: Applies to all Context utterances
### H\.5Malformed event utterance

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isanythingineitherstatementungrammatical,misspelled,malformed,orotherwisenonsensical?

6STATEMENT\_A:MarylosesLindakeys

7STATEMENT\_B:Lindaisupset

8ANSWER:Yes

9STATEMENT\_A:MarylosesLinda’skeys

10STATEMENT\_B:Lindaisupset

11ANSWER:No

12STATEMENT\_A:MarylosesLinda’sarm

13STATEMENT\_B:Lindaisupset

14ANSWER:Yes

15STATEMENT\_A:Marygoeshoppingatthemall

16STATEMENT\_B:Marybuysmanyittems

17ANSWER:Yes

18STATEMENT\_A:Marygoesshoppingatthemall

19STATEMENT\_B:Marybuysmanyitems

20ANSWER:No

21STATEMENT\_A:\{Context\}

22STATEMENT\_B:\{Event\}

23ANSWER:

Listing 24:Malformed\-AB\-Base: Applies to all Context\-Event pairs1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:ArestatementsAandBbothinsimplepresenttense?Notethatnoothertensesareincluded,andpresentprogressiveisNOTsimplepresenttense\.

6STATEMENT\_A:Marygoesshoppingatthemall

7STATEMENT\_B:Marywasexcited

8ANSWER:No

9STATEMENT\_A:Marygoesshoppingatthemall

10STATEMENT\_B:Maryisexcited

11ANSWER:Yes

12STATEMENT\_A:Marygoesshoppingatthemall

13STATEMENT\_B:Marybuysmanyitems

14ANSWER:Yes

15STATEMENT\_A:Marywentshoppingatthemall

16STATEMENT\_B:Marybuysmanyitems

17ANSWER:No

18STATEMENT\_A:Marywentshoppingatthemall

19STATEMENT\_B:Maryisbuyingmanyitems

20ANSWER:No

21STATEMENT\_A:\{Context\}

22STATEMENT\_B:\{Event\}

23ANSWER:

Listing 25:Malformed\-AB\-Tense: Applies to all Context\-Event pairs1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Puttogether,arestatementsAandBcoherent?Inotherwords,ifstatementsAandBmakesenseandarerelatedtoeachother,answerYes\.Otherwise,answerNo\.

6STATEMENT\_A:Marygoesshoppingatthemall

7STATEMENT\_B:Marybuysmanyitems

8ANSWER:Yes

9STATEMENT\_A:Marygoesshoppingatthemall

10STATEMENT\_B:Themallshopsback

11ANSWER:No

12STATEMENT\_A:Marygoesshoppingatthemall

13STATEMENT\_B:Theelfsmilesback

14ANSWER:No

15STATEMENT\_A:\{Context\}

16STATEMENT\_B:\{Event\}

17ANSWER:

Listing 26:Malformed\-AB\-Coherent: Applies to all Context\-Event pairs
### H\.6Utterance not brief

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:CouldstatementBbeediteddowntobemorebrief,strippedofdescriptivemodifiers,whilestillretainingitsmeaning?Ifminormodifiersordetailscanbedeleted,answerYes\.Otherwise,answerNo\.Pleasebeverycareful\.IfyouthinktheentiretyofstatementBcanbedeleted,answerNo\.

6STATEMENT\_A:Marypurchasesthebike

7STATEMENT\_B:Maryinspectsitbriefly

8ANSWER:Yes

9STATEMENT\_A:Marypurchasesthebike

10STATEMENT\_B:Maryinspectsit

11ANSWER:No

12STATEMENT\_A:Marywatchestelevision

13STATEMENT\_B:Maryturnsitofflater

14ANSWER:Yes

15STATEMENT\_A:Marywatchestelevision

16STATEMENT\_B:Maryturnsitoff

17ANSWER:No

18STATEMENT\_A:Marytakesashoppingcarttothestore

19STATEMENT\_B:Maryfillsthecartwithproducts

20ANSWER:Yes

21STATEMENT\_A:Marytakesashoppingcarttothestore

22STATEMENT\_B:Maryfillsthecartwithcheese

23ANSWER:No

24STATEMENT\_A:\{Context\}

25STATEMENT\_B:\{Event\}

26ANSWER:STATEMENT\_A:Context

27STATEMENT\_B:Event

28ANSWER:

Listing 27:Brevity: Applies to all Context\-Event pairs
### H\.7Strange use of definites

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:IfstatementBhasanobjectwith’a’,’an’,orapossessiveatprecedingit,canitbereplacedwith’the’oroneof’her/his/their/its’andstillmakesense?IfstatementBalreadyhasanobjectmodifiedby’the’,answerNo\.IfstatementBnoarticleorpossessive,ornoobject,answerNoOnlyansweryesifstatementBhas’a’,’an’,orapossessivemodifyingtheobject,ANDitcanbereplacedwiththewhilestillmakingsense\.

6STATEMENT\_A:Marybuysnewsheets

7STATEMENT\_B:Maryselectsacolor

8ANSWER:Yes

9STATEMENT\_A:Marybuysnewsheets

10STATEMENT\_B:Maryselectsthecolor

11ANSWER:No

12STATEMENT\_A:MarygoesbowlingwithLinda

13STATEMENT\_B:Marypicksabowlingball

14ANSWER:Yes

15STATEMENT\_A:MarygoesbowlingwithLinda

16STATEMENT\_B:Marypicksherbowlingball

17ANSWER:No

18STATEMENT\_A:Marymakesfriendshipbracelets

19STATEMENT\_B:Marytiesknots

20ANSWER:Yes

21STATEMENT\_A:Marymakesfriendshipbracelets

22STATEMENT\_B:Marytiestheknots

23ANSWER:No

24STATEMENT\_A:\{Context\}

25STATEMENT\_B:\{Event\}

26ANSWER:

Listing 28:Definite: Applies to only subaction Events\.
### H\.8Narration

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Pairedtogether,dostatementsAandBseemliketheywouldbefoundinanovelasopposedtoatextmessageoranonlineforum?Cluestolookfor:\-Verbsthatareloaded,colorful,orliterary:’tries’tocleanupaspillisloadedbecauseitimpliessomefailuretocompletethetask\.’Rushes’insteadof’goes’or’clutches’insteadof’uses’aremorecolorful/literarythantheiralternatives\.\-Descriptionsofminuteactions:e\.g\.’Maryturnsherhead’\-Information,likeaninternalmonologue,thataregularfriendwouldnotbeprivytoo,butthatanauthorofastorymightwrite\.

6STATEMENT\_A:Marybuysnewsheets

7STATEMENT\_B:Marythinksaboutherbed

8ANSWER:Yes

9STATEMENT\_A:Marybuysnewsheets

10STATEMENT\_B:Marywashesthem

11ANSWER:No

12STATEMENT\_A:Maryspillsadrink

13STATEMENT\_B:Marytriestocleanitup

14ANSWER:Yes

15STATEMENT\_A:Maryspillsadrink

16STATEMENT\_B:Marycleansitup

17ANSWER:No

18STATEMENT\_A:Maryhearsathud

19STATEMENT\_B:Maryturnsherhead

20ANSWER:Yes

21STATEMENT\_A:Maryhearsathud

22STATEMENT\_B:Marygetsup

23ANSWER:No

24STATEMENT\_A:MarycutsLinda’shairshort

25STATEMENT\_B:Lindalooksinthemirror

26ANSWER:Yes

27STATEMENT\_A:MarycutsLinda’shairshort

28STATEMENT\_B:Thereisamirror

29ANSWER:No

30STATEMENT\_A:Maryroastsmarshmallows

31STATEMENT\_B:Maryclutchesastick

32ANSWER:Yes

33STATEMENT\_A:Maryroastsmarshmallows

34STATEMENT\_B:Maryusesastick

35ANSWER:No

36STATEMENT\_A:\{Context\}

37STATEMENT\_B:\{Event\}

38ANSWER:

Listing 29:Narration: Applies to all Context\-Event pairs
### H\.9Context event not sufficiently frequent

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Doesthisstatementdescribeasituationoractivitywhichcanbeconsideredabnormalorhyperspecific?Forexample,fishingasageneralactivityisnormal,butaddingcertaindetailstothestatementcanmakeiteitherabnormalorhyperspecific\.Murderorotherkindsofnon\-pettyviolencedohappenintheworlddaily,butthesearenotconsideredapartofnormallife\.

6STATEMENT:Marygoesfishing

7ANSWER:No

8STATEMENT:Marygoesfishingfortreasure

9ANSWER:Yes

10STATEMENT:Marybringsfourrodstothelaketofish

11ANSWER:Yes

12STATEMENT:MarymakesdinnerforLinda

13ANSWER:No

14STATEMENT:MarygivesLindaapresent

15ANSWER:No

16STATEMENT:MarygivesLindapoison

17ANSWER:Yes

18STATEMENT:Marymeetswithherdad

19ANSWER:No

20STATEMENT:Marymeetswithherdad’sfortuneteller

21ANSWER:Yes

22STATEMENT:Maryfindsoutherdadisnotherbiologicaldad

23ANSWER:Yes

24STATEMENT:\{Context\}

25ANSWER:

Listing 30:Life\-Daily\-Life: Applies to all Context utterances\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isthisstatementaboutsomethingthathappensasamilestoneinaperson’slifetime?Notethatifitcanbeamilestone,butitcanalsobearegularoccurrenceforcertainpeople’sjobsorhobbies,youranswershouldbeno\.Anexampleofthisiswinningacompetition:thiscanhappenregularlyforathletesandothertypesofpeople\.Butsomethinglikegraduatinghighschool,bydefinition,canonlyhappenonceinalifetime\.Andgivingbirthcanhappenafewtimes,butisalwaysasignificantevent\.

6STATEMENT:Maryattendsherclass

7ANSWER:No

8STATEMENT:Marygraduateshighschool

9ANSWER:Yes

10STATEMENT:MarygivesLindaapresent

11ANSWER:No

12STATEMENT:Marygivesbirth

13ANSWER:Yes

14STATEMENT:\{Context\}

15ANSWER:

Listing 31:Life\-Lifetime: Applies to all Context utterances\.
### H\.10Time\-gap

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isthereaverywidetime\-gaporconceptualgapbetweenstatementsAandB?

6STATEMENT\_A:MarywatersLinda’splants

7STATEMENT\_B:Linda’splantsstayhealthy

8ANSWER:Yes

9STATEMENT\_A:MarywatersLinda’splants

10STATEMENT\_B:Lindathanksher

11ANSWER:No

12STATEMENT\_A:Maryfindsawife

13STATEMENT\_B:Maryplansawedding

14ANSWER:Yes

15STATEMENT\_A:Marygetsengaged

16STATEMENT\_B:Maryplansawedding

17ANSWER:No

18STATEMENT\_A:Marywalkstothelibrary

19STATEMENT\_B:Maryreadsabook

20ANSWER:Yes

21STATEMENT\_A:Marywalkstothelibrary

22STATEMENT\_B:Marychecksoutabook

23ANSWER:No

24STATEMENT\_A:\{Context\}

25STATEMENT\_B:\{Event\}

26ANSWER:

Listing 32:Gap: Applies to all next\-action Events\.
### H\.11Time\-scope

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:CanstatementAbeconsideredmuchwiderinscopethanstatementB?

6STATEMENT\_A:Marytakesmanyphotos

7STATEMENT\_B:Maryadjuststhelens

8ANSWER:Yes

9STATEMENT\_A:Marytakesaphoto

10STATEMENT\_B:Maryadjuststhelens

11ANSWER:No

12STATEMENT\_A:Maryplayschess

13STATEMENT\_B:Maryplanshernextmove

14ANSWER:Yes

15STATEMENT\_A:Marymovesherchesspiece

16STATEMENT\_B:Maryplanshernextmove

17ANSWER:No

18STATEMENT\_A:Marysailsalongthecoast

19STATEMENT\_B:Maryadjuststhesail

20ANSWER:Yes

21STATEMENT\_A:Marysailsalongthecoast

22STATEMENT\_B:Marydocumentsit

23ANSWER:No

24STATEMENT\_A:Maryspendsthedayatthebeach

25STATEMENT\_B:Marysetsupatowel

26ANSWER:Yes

27STATEMENT\_A:Maryspendsthedayatthebeach

28STATEMENT\_B:Maryswims

29ANSWER:No

30STATEMENT\_A:\{Context\}

31STATEMENT\_B:\{Event\}

32ANSWER:

Listing 33:Scope: Applies to just subaction Events\.
### H\.12Passive

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Doesthisstatementcontainpassivevoice?

6STATEMENT:MarygetstoldoffbyLinda

7ANSWER:Yes

8STATEMENT:LindatellsMaryoff

9ANSWER:No

10STATEMENT:\{Context\}

11ANSWER:

Listing 34:Passive: Applies to all Context utterances\.
### H\.13Not standalone

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Canthisstatementstandalonewithoutextracontext?AnswerYesifthestatementmakesnoreferencestoapreviouscontextanditmakessensewithoutextracontext\.

6STATEMENT:Maryleavestheshop

7ANSWER:Yes

8STATEMENT:Marygoesbacktotheshop

9ANSWER:No

10STATEMENT:Marycrashesintoacar

11ANSWER:Yes

12STATEMENT:Marycrashesintoanothercar

13ANSWER:No

14STATEMENT:MarygivesLindamoney

15ANSWER:Yes

16STATEMENT:MarygetsLinda’smoneyback

17ANSWER:No

18STATEMENT:MarygetsLinda’smoney

19ANSWER:Yes

20STATEMENT:Marylosesherkeys

21ANSWER:Yes

22STATEMENT:Maryfindsthekeysathome

23ANSWER:No

24STATEMENT:\{Context\}

25ANSWER:

Listing 35:Standalone: Applies to all Context utterances\.
### H\.14Too vague

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Puttogether,arestatementsAandBextremelyunderspecified?Notethatifthestatementsaregeneral,butnotextremelyunderspecified,youranswershouldbeNo\.YouranswershouldonlybeYesifthegeneralnessexceedsathresholdwhereacorepartofthemeaningofthetextseemstobemissing\.

6STATEMENT\_A:MaryseesthelookonLinda’sface

7STATEMENT\_B:Maryreacts

8ANSWER:Yes

9STATEMENT\_A:MaryseestheangrylookonLinda’sface

10STATEMENT\_B:Maryreacts

11ANSWER:No

12STATEMENT\_A:MaryseesthelookonLinda’sface

13STATEMENT\_B:Marycallsitout

14ANSWER:No

15STATEMENT\_A:Marystartsaclub

16STATEMENT\_B:Maryisinterestedinatopic

17ANSWER:Yes

18STATEMENT\_A:Marystartsachessclub

19STATEMENT\_B:Maryisinterestedinchess

20ANSWER:No

21STATEMENT\_A:\{Context\}

22STATEMENT\_B:\{Event\}

23ANSWER:

Listing 36:Vague: Applies to all Context\-Event pairs
### H\.15No typicality

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aTEXT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Doesthefollowingquestion"When\{\},howoftenis\{\}?"makesensegrammaticallyandmeaning\-wise?Forexample,ifstatementBisaboutaninherenttraitofsomeone,thequestiondoesnotmakesenseandyouranswershouldbeNo\.Generally,answerNoifitdoesnotmakesensetothinkaboutasituationalrelationshipbetweenstatementsAandB\.Otherwise,answerYes\.

6TEXT:WhenMaryfindsastraydog,howoftendoesshepetit?

7ANSWER:Yes

8TEXT:WhenMaryfindsastraydog,howoftenissheananimallover?

9ANSWER:No

10TEXT:WhenMarybuildsahouse,howoftendoesshetakeawholeyear?

11ANSWER:Yes

12TEXT:WhenMarybuildsLinda’shouse,howoftendoesLindamovein?

13ANSWER:No

14TEXT:WhenMarymakesfriendshipbracelets,howoftendoessheusemorethanonebeadshape?

15ANSWER:Yes

16TEXT:WhenMarymakesfriendshipbracelets,howoftendoessheenjoycraftingasahobby?

17ANSWER:No

18TEXT:WhenMaryseesshoesforsale,howoftendoesshebuythem?

19ANSWER:Yes

20TEXT:WhenMaryseesshoesforsale,howoftenissheinterestedinfashion?

21ANSWER:No

22TEXT:When\{\},howoftenis\{\}?

23ANSWER:

Listing 37:No\-Typicality: Applies to all Context\-Event pairs
### H\.16Prompts that had no effect in the end

1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:IsstatementBtheanswerforwhystatementAhappened?Inotherwords,canyoueasilyinferthewordBECAUSEbetweenstatementsAandB?"ContextBECAUSEEvent"Pleaseconsideryouranswerintermsofcoherencerelations\.Inthiscase,isBECAUSEmorelikelythanASARESULT?IfASARESULTismorelikelythanorequallylikelytoBECAUSE,answerNo\.IfstatementBisjustbackgroundinformationaboutthescenario,answerNo\.

6STATEMENT\_A:Marytakesashoppingcarttothestore

7STATEMENT\_B:Maryplanstobuymultipleitems

8ANSWER:Yes

9STATEMENT\_A:Marytakesashoppingcarttothestore

10STATEMENT\_B:Maryhasalist

11ANSWER:No

12STATEMENT\_A:Marygoesbacktosleep

13STATEMENT\_B:Maryistired

14ANSWER:Yes

15STATEMENT\_A:Marygoesbacktosleep

16STATEMENT\_B:Maryisonabed

17ANSWER:No

18STATEMENT\_A:Marycoversherhead

19STATEMENT\_B:Maryfeelscold

20ANSWER:Yes

21STATEMENT\_A:Marycoversherhead

22STATEMENT\_B:Maryisoutside

23ANSWER:No

24STATEMENT\_A:\{Context\}

25STATEMENT\_B:\{Event\}

26ANSWER:

Listing 38:Explain: Applies to only state Events\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:IsstatementBthedirectresultofstatementAhappening?Inotherwords,isthetextmeaningequivalentto"Event,ANDASARESULTContext"?

6STATEMENT\_A:Marysmellssmoke

7STATEMENT\_B:Maryisconcerned

8ANSWER:Yes

9STATEMENT\_A:Marysmellssmoke

10STATEMENT\_B:Maryhassomethingintheoven

11ANSWER:No

12STATEMENT\_A:Marywinstheprize

13STATEMENT\_B:Maryisoverjoyed

14ANSWER:Yes

15STATEMENT\_A:Marywinstheprize

16STATEMENT\_B:Maryisfaraheadofeveryoneelse

17ANSWER:No

18STATEMENT\_A:\{Context\}

19STATEMENT\_B:\{Event\}

20ANSWER:

Listing 39:Result\-AB: Applies to only state Events\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,STATEMENT\_A,STATEMENT\_B,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:IsstatementAthedirectresultofstatementBhappening?Inotherwords,isthetextmeaningequivalentto"Context,ANDASARESULTEvent"?

6STATEMENT\_A:Maryisconcerned

7STATEMENT\_B:Marysmellssmoke

8ANSWER:Yes

9STATEMENT\_A:Maryhassomethingintheoven

10STATEMENT\_B:Marysmellssmoke

11ANSWER:No

12STATEMENT\_A:Maryisoverjoyed

13STATEMENT\_B:Marywinstheprize

14ANSWER:Yes

15STATEMENT\_A:Maryisfaraheadofeveryoneelse

16STATEMENT\_B:Marywinstheprize

17ANSWER:No

18STATEMENT\_A:\{Context\}

19STATEMENT\_B:\{Event\}

20ANSWER:

Listing 40:Result\-BA: Applies to only state Events\.1Youareanexpertannotatorofnaturallanguagedata\.

2YouwillreadaQUESTION,aSTATEMENT,andsomeEXAMPLES\.

3Tothebestofyourknowledge,andaccordingtotheexamples,answertheQUESTIONwithyesorno\.

4

5QUESTION:Isthisastatementexplicitlyaboutsomebody’shabits?

6STATEMENT:Maryplaysfootballweekly

7ANSWER:Yes

8STATEMENT:Maryplaysinthefootballgame

9ANSWER:No

10STATEMENT:\{Context\}

11ANSWER:

Listing 41:Habitual: Applies to all Context utterances\.