Hindsight Bias in Clinical Temporal Reasoning: How Future Data Exposure Affects Large Language Model Judgment

# Hindsight Bias in Clinical Temporal Reasoning: How Future Data Exposure Affects Large Language Model Judgment
Source: [https://arxiv.org/html/2609.13454](https://arxiv.org/html/2609.13454)
\\mlhtrack

proceedings \[3pt\]

Sayantan Kumar2Ojas Kadam3Jeremy C\. Weiss2Affiliation:Affiliation:Affiliation:1Case School of Engineering, Case Western Reserve University, USA 2National Library of Medicine, National Institutes of Health, USA 3Rice University University, USA

###### Abstract

Clinical decisions are prospective, but clinical language models are often evaluated on retrospective records that reveal the final diagnosis, treatment response, and outcome\. Such evaluations may reward the use of future information rather than reasoning under the uncertainty present at the decision point\. We introduce a paired benchmark for measuring outcome\-conditioned shifts consistent with hindsight bias in clinical temporal reasoning\. It contains 171 case reports from the PubMed Central Open Access Subset—40 sepsis and 131 GLP\-1/diabetes cases—represented as both textual narratives and human\-annotated and LLM\-generated textual time series \(TTS\)\. For each case, questions are tied to a clinically meaningful cutoff and paired with a prospective reference answer and an outcome\-consistent*hindsight trap*\. Models answer each question using either a TTS truncated at the cutoff or the complete timeline; additional conditions vary the narrative source \(original or synthetic\) and TTS annotation source \(human or LLM\)\. We evaluate accuracy \(Acc\), hindsight trap rate \(HTR\), answer instability rate \(AIR\), and hindsight bias rate \(HBR\), each of which captures different signals of hindsight bias\. Across GPT 5\.6 Sol, Gemma 4, GLM 5\.2, and Opus 5, full timeline exposure produces consistent hindsight\-sensitive shifts, while temporal masking reduces bias without lowering accuracy\.

††workshop:Machine Learning for Health \(ML4H\) 2026††proceedings:: Submitted to ML4H 2026:\\mlhtrackname††workshop:Machine Learning for Health \(ML4H\) 2026###### keywords

hindsight bias; clinical temporal reasoning; large language models; clinical question answering, benchmark evaluation

![Refer to caption](https://arxiv.org/html/2609.13454v1/images/pipeline_figure.png)Figure 1:Benchmark pipeline for measuring hindsight\-sensitive clinical reasoning\. Models answer the same question using masked and complete case trajectories, and response changes are evaluated for correctness, instability, and movement toward an outcome\-consistent hindsight trap\.## 1Introduction

Clinical decision\-making is inherently prospective\. A clinician deciding whether to admit, treat, escalate, or discharge a patient must act using only the evidence available at that moment, before the subsequent trajectory and outcome are known\. Clinical artificial intelligence systems inherit the same information constraint: a model intended to support a decision at timettshould be evaluated using information available by timett, rather than information documented later in the episode\.

This evaluation setting creates the conditions for hindsight bias, in which knowledge of an outcome changes judgments about what was knowable or predictable beforehand\([Fischhoff, 1975](https://arxiv.org/html/2609.13454#bib.bib9)\)\. Hindsight bias has been observed in physicians estimating diagnostic probabilities\([Arkes et al\., 1981](https://arxiv.org/html/2609.13454#bib.bib1);[Dawson et al\., 1988](https://arxiv.org/html/2609.13454#bib.bib6)\)and evaluating earlier care\. In a randomized vignette study, clinicians judged identical antecedent care to recovery or mortality outcomes more harshly after a death outcome in two of three cases, and greater seniority did not eliminate the effect\([Banham\-Hall and Stevens, 2019](https://arxiv.org/html/2609.13454#bib.bib2)\)\.

However, not every answer change after additional evidence is a bias\. If the task asks for the final diagnosis, later evidence should appropriately update the answer\. The narrower failure mode studied here arises when later information changes a judgment whose intended target is what was supportable at an earlier reference point\. Measuring this failure requires holding the clinical question and antecedent evidence fixed while varying access to post\-reference information\. Prior paired clinical vignette studies have shown that general and reasoning\-focused LLMs can exhibit hindsight bias when outcome cues are manipulated, but these evaluations are based on individual vignettes rather than a corpus of longitudinal clinical cases\([Wang and Redelmeier, 2024](https://arxiv.org/html/2609.13454#bib.bib19);[Degany et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib7)\)\. ExAnte evaluates temporal leakage in general domain QA by imposing explicit temporal cutoffs\([Liu et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib14)\), but its objective differs from ours\. It evaluates whether answers are supported by information available before a specified time, whereas we ask whether exposure to a patient’s subsequent clinical trajectory changes a model’s judgment about what was supportable at an earlier decision point\. This requires paired full–masked evaluation with a fixed prospective reference answer and an outcome\-consistent hindsight trap, allowing us to measure not only correctness but the direction of within\-question answer changes\. To the best of our knowledge, no existing clinical benchmark applies this design across a corpus of real longitudinal case trajectories\.

##### Contributions\.

We introduce \(1\) a paired benchmark for clinical hindsight bias that compares masked and complete trajectories using prospective reference answers and predefined hindsight traps; \(2\) complementary metrics \(Δ\\DeltaHTR, AIR, and HBR\) for measuring directional hindsight\-sensitive changes; and \(3\) a cross\-model evaluation across two clinical cohorts and multiple input representations showing that temporal masking reduces hindsight bias without reducing accuracy\.

## 2Related Work

Here, we present an overview of related works\. A detailed version can be found in Appendix[A](https://arxiv.org/html/2609.13454#A1)\.

##### General temporal reasoning\.

TRAM, TimeBench, Test of Time, and TimE evaluate event ordering, temporal arithmetic, duration, and multistep reasoning in synthetic or general domain settings\([Wang and Zhao, 2024](https://arxiv.org/html/2609.13454#bib.bib20);[Chu et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib4);[Fatemi et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib8);[Wei et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib21)\)\. ExAnte is closest to our temporal setup\. It imposes explicit cutoffs and measures leakage from post\-cutoff knowledge, but its tasks are non\-clinical\([Liu et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib14)\)\. Our benchmark instead tests whether later events in a patient trajectory shift a judgment about an earlier clinical point toward the hindsight trap\.

##### Clinical question answering over longitudinal records\.

MedAlign, MIMIC\-Instr, EHRNoteQA, and EHRSQL evaluate instruction following or question answering over longitudinal notes and structured EHR data\([Fleming et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib10);[Wu et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib23);[Kweon et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib12);[Lee et al\., 2022](https://arxiv.org/html/2609.13454#bib.bib13)\)\. More recent benchmarks target temporal reasoning directly\. TIMER grounds instructions to patient\-specific timestamps, ASCENT evaluates diagnosis as evidence accumulates, and RealICU evaluates windowed ICU decisions using hindsight\-informed labels\([Cui et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib5);[Choi et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib3);[Shen et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib18)\)\. These tasks measure temporal grounding or stepwise reasoning\. We pair the same question and pre\-cutoff evidence with complete and masked timelines, allowing measurement of whether the answer changes and whether it moves in an outcome\-consistent direction\.

##### Reasoning from clinical case reports\.

MedCaseReasoning evaluates diagnostic answers and reasoning derived from open access case reports, while MedR\-Bench covers examination, diagnosis, and treatment planning\([Wu et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib22);[Qiu et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib17)\)\. Prior work also reconstructs time\-localized sepsis trajectories from PubMed Central case reports as textual time series\([Noroozizadeh and Weiss, 2026](https://arxiv.org/html/2609.13454#bib.bib16)\)\. We build on this representation but study a different question: not only whether a model reaches the final conclusion, but whether that conclusion was supportable before later events were known\.

## 3Methods

The overall pipeline is illustrated in Figure[1](https://arxiv.org/html/2609.13454#S0.F1), and all prompts used are listed in Appendix[C](https://arxiv.org/html/2609.13454#A3)\.

### 3\.1Study design and data source

Case reports were drawn from the PubMed Central Open Access \(PMC OA\) Subset provided by[National Library of Medicine \(2025\)](https://arxiv.org/html/2609.13454#bib.bib15)\. We used two cohorts comprising 171 case reports: 40 sepsis cases and 131 cases of type 2 diabetes patients taking Glucagon\-like Peptide Receptor Agonists \(GLP1\-RA\) medications\. The sepsis and GLP1\-RA cohorts build on previously published works\([Noroozizadeh and Weiss, 2026](https://arxiv.org/html/2609.13454#bib.bib16);[Kumar and Weiss, 2026](https://arxiv.org/html/2609.13454#bib.bib11)\), which provide independently constructed timelines from clinically trained annotators and LLM extractors\. The two cohorts were selected to evaluate the benchmark across contrasting clinical trajectories: comparatively dense, acute care episodes in sepsis and more heterogeneous longitudinal trajectories in GLP\-1/diabetes\.

### 3\.2Clinical representations

Each caseiiwas represented by its original case report narrativeNiN\_\{i\}and by two textual time series \(TTS\): a human\-annotated TTSTiHT\_\{i\}^\{\\mathrm\{H\}\}and an independently generated LLM TTSTiLT\_\{i\}^\{\\mathrm\{L\}\}\. A TTS is a sequenceTis=\{\(ei​js,ti​js\)\}j=1nis,s∈\{H,L\}T\_\{i\}^\{s\}=\\\{\(e\_\{ij\}^\{s\},t\_\{ij\}^\{s\}\)\\\}\_\{j=1\}^\{n\_\{i\}^\{s\}\},\\qquad s\\in\\\{\\mathrm\{H\},\\mathrm\{L\}\\\}, whereei​jse\_\{ij\}^\{s\}is a clinical event andti​jst\_\{ij\}^\{s\}is its relative time in hours\. This representation preserves the distinction between narrative order and clinical event order, which may diverge in retrospectively written case reports\.

### 3\.3Question construction

Questions were generated with GPT 5\.6 Sol with the default medium reasoning effort from the original narrativeNiN\_\{i\}and human TTSTiHT\_\{i\}^\{\\mathrm\{H\}\}\. For each case, the generator returned ten questions as a schema\-constrained JSON array; outputs that failed validation received one repair attempt\. Across 171 cases, this yielded 1,710 questions\. Figure[2](https://arxiv.org/html/2609.13454#S3.F2)shows a representative item, including the fields visible to the answering model and the hidden metadata used for evaluation\.

Case:58\-year\-old male admitted with septic shock secondary to pneumonia\. \[…\]Cutoff time:24 hours post\-admissionVisible to answering model:Question:What probability range best represents the likelihood of requiring mechanical ventilation within 48 hours, given the patient’s current respiratory status and haemodynamic profile?Format:Multiple choice \(ordinal\)Options:\[A\.\]<<10%, \[B\.\] 10–30%, \[C\.\] 31–60%, \[D\.\]\>\>60%Hidden evaluation metadata:a∗a^\{\*\}\(reference answer\):A\.<<10%atrapa^\{\\mathrm\{trap\}\}\(hindsight trap\):D\.\>\>60%Trap explanation:The patient was intubated at 36 hours; a model with access to the full trajectory may inflate the prospective probability estimate in light of this outcome\.Eval target:bothCutoff event:Arterial blood gas showing mild hypoxaemia; vasopressor requirement stable\.

Figure 2:Representative benchmark question\. The visible component \(above the divider\) is shown to the answering model; the hidden metadata \(below\) is used only for evaluation\. The reference answera∗a^\{\*\}is what was defensible at the cutoff, while the hindsight trap answeratrapa^\{\\mathrm\{trap\}\}is what may seem more compelling once the eventual outcome is known\.Questions were assigned toaccuracy,hindsight\_bias, orboth\.accuracyitems targeted verifiable answers with low hindsight sensitivity;hindsight\_biasitems targeted subjective judgments such as likelihood and treatment appropriateness that could shift after later evidence; andbothitems required a prospectively supported answer and a plausible trap\. Each case targeted a 2/3/5 split across these categories\.

The benchmark included free response, true/false, and four\-option multiple choice questions, with multiple choice items labeled categorical or ordinal\. Ordinal options used explicit ranges, withaq∗a\_\{q\}^\{\*\}andaqtrapa\_\{q\}^\{\\mathrm\{trap\}\}placed at maximally separated positions\. The prompt limited each reasoning type to two questions per case to maximize diversity\. It also omitted explicit temporal references from visible question text so that prospective reasoning depended on the supplied context rather than an overt instruction\.

### 3\.4Temporal masking and cutoff position

For each questionqq, the generator supplied a numeric cutoffcqc\_\{q\}in the time coordinate of the human TTS\. Given a TTSTisT\_\{i\}^\{s\}, we retained events at or before the cutoff:

Ti​q,maskeds=\{\(ei​js,ti​js\)∈Tis:ti​js≤cq\}T\_\{iq,\\mathrm\{masked\}\}^\{s\}=\\\{\(e\_\{ij\}^\{s\},t\_\{ij\}^\{s\}\)\\in T\_\{i\}^\{s\}:t\_\{ij\}^\{s\}\\leq c\_\{q\}\\\}\(1\)whileTi,fulls=TisT\_\{i,\\mathrm\{full\}\}^\{s\}=T\_\{i\}^\{s\}retained the complete trajectory\. The same numeric cutoff was applied to the human and LLM TTS\.

After sorting valid rows by numeric time and preserving source order within ties, letkq∈\{1,…,ni\}k\_\{q\}\\in\\\{1,\\ldots,n\_\{i\}\\\}denote the chronological index matching the cutoff\. We defineRelPos⁡\(q\)=kq−1ni−1,\\operatorname\{RelPos\}\(q\)=\\frac\{k\_\{q\}\-1\}\{n\_\{i\}\-1\},where 0 and 1 denote the first and last recorded events, respectively\.

##### Constructing synthetic case reports\.

To test whether our results depended on the original case report prose or persisted under an independently reconstructed narrative representation, we generated synthetic narratives from the TTS\. A separate narrative was generated for every questionqq, cutoffci​qc\_\{iq\}, and annotation sources∈\{H,L\}s\\in\\\{\\mathrm\{H\},\\mathrm\{L\}\\\}: each was derived from the masked TTSTi​qs,maskT\_\{iq\}^\{s,\\mathrm\{mask\}\}rather than the full TTS, so no post\-cutoff information could leak into the narrative seen by the answering model\. A single case reportiiwithQiQ\_\{i\}questions and two annotation sources therefore yields up to2​Qi2Q\_\{i\}distinct synthetic narratives, guaranteeing strict temporal alignment between the narrative and the TTS in each paired condition\. For each condition, narratives were generated with the same model used for answering\.

### 3\.5Answer generation

We evaluated GPT\-5\.6 Sol, Gemma\-4\-31B, GLM\-5\.2\-FP8, and Opus 5\. Each model answered every question independently and received only the visible question fields—question,format, andoptions—together with the context specified by Table[1](https://arxiv.org/html/2609.13454#S3.T1)\. The rest were withheld; see Figure[2](https://arxiv.org/html/2609.13454#S3.F2)\. Responses contained ananswerand brief supportingevidence\. No model had access to external retrieval, tools, or the web\.

##### Hindsight warning sensitivity analysis\.

We repeated the Sepsis experiments for GPT and GLM after appending explicit instructions to use only contemporaneously available information and disregard the final outcome\. All other procedures were unchanged\.

### 3\.6Experimental conditions

We evaluated seven input conditions \(Table[1](https://arxiv.org/html/2609.13454#S3.T1), Appendix[B](https://arxiv.org/html/2609.13454#A2)\) using the same question set throughout\. Condition 0 provided the complete original narrative as an unpaired retrospective baseline\. Conditions 1\.x combined that narrative with full or masked human\- or LLM\-generated TTS; only the TTS differed within each pair, so the narrative could still reveal post\-cutoff information\. Conditions 2\.x used TTS alone and therefore directly isolated the effect of temporal masking\. Conditions 3\.x paired each TTS with a synthetic narrative generated from the corresponding masked or full information scope\.

Condition 0 is unpaired and therefore supports only Acc and HTR\. Conditions 1\.x, 2\.x, and 3\.x provide matched masked and full responses and additionally supportΔ\\DeltaAcc,Δ\\DeltaHTR, AIR, and HBR\.

Table 1:Input conditions\. H/L\-TTS denote human/LLM\-generated textual time series; F/M denotes full/masked variants\.
### 3\.7Evaluation

Letaq,za\_\{q,z\}denote the model answer to questionqqunder visibility conditionz∈\{full,masked\}z\\in\\\{\\mathrm\{full\},\\mathrm\{masked\}\\\}\. We compare each answer with the prospective referenceaq∗a\_\{q\}^\{\*\}and the hindsight trapaqtrapa\_\{q\}^\{\\mathrm\{trap\}\}\. Metrics were computed per question and then averaged\. Structured responses were scored by exact match\. Free responses were scored by a GPT\-5\.6 Sol judge against the prospective reference and hindsight\-trap answers using\{0,0\.5,1\}\\\{0,0\.5,1\\\}semantic\-agreement scores\.

#### 3\.7\.1Individual metrics

Accuracy \(Acc\)\.Acc measures agreement with the prospective reference answeraq∗a\_\{q\}^\{\*\}and is defined foraccuracyandbothquestions, which comprise 70% of the dataset\. For true/false and multiple choice questions,Accq\(a\)=𝟏\[a=aq∗\]∈\{0,1\}\\mathrm\{Acc\}\_\{q\}\(a\)=\\mathbf\{1\}\[a=a\_\{q\}^\{\*\}\]\\in\\\{0,1\\\}, whereas free response questions receiveAccq​\(a\)∈\{0,0\.5,1\}\\mathrm\{Acc\}\_\{q\}\(a\)\\in\\\{0,0\.5,1\\\}from the judge, with 0\.5 denoting partial correctness\.

Hindsight trap rate \(HTR\)\.HTR measures agreement with the outcome\-consistent hindsight trapaqtrapa\_\{q\}^\{\\mathrm\{trap\}\}and is defined forhindsight\_biasandbothquestions, which comprise 80% of the dataset\. For true/false and categorical multiple choice questions,

HTRq\(a\)=𝟏\[a=aqtrap\]∈\{0,1\},\\mathrm\{HTR\}\_\{q\}\(a\)=\\mathbf\{1\}\[a=a\_\{q\}^\{\\mathrm\{trap\}\}\]\\in\\\{0,1\\\},\(2\)whereas free response questions receiveHTRq​\(a\)∈\{0,0\.5,1\}\\mathrm\{HTR\}\_\{q\}\(a\)\\in\\\{0,0\.5,1\\\}from the judge\. For ordinal multiple choice questions, HTR is graded by movement fromaq∗a\_\{q\}^\{\*\}towardaqtrapa\_\{q\}^\{\\mathrm\{trap\}\}\. In Figure[2](https://arxiv.org/html/2609.13454#S3.F2), for example, answers B, C, and D receive13\\tfrac\{1\}\{3\},23\\tfrac\{2\}\{3\}, and 1, respectively, yieldingHTRq​\(a\)∈\{0,13,23,1\}\\mathrm\{HTR\}\_\{q\}\(a\)\\in\\\{0,\\tfrac\{1\}\{3\},\\tfrac\{2\}\{3\},1\\\}\.

#### 3\.7\.2Paired metrics

Delta accuracy \(Δ\\DeltaAcc\)\.

Δ​Accq=Accq​\(aq,masked\)−Accq​\(aq,full\)\.\\Delta\\mathrm\{Acc\}\_\{q\}=\\mathrm\{Acc\}\_\{q\}\(a\_\{q,\\mathrm\{masked\}\}\)\-\\mathrm\{Acc\}\_\{q\}\(a\_\{q,\\mathrm\{full\}\}\)\.\(3\)A positiveΔ​Accq\\Delta\\mathrm\{Acc\}\_\{q\}indicates greater correctness under masking, meaning post\-cutoff information reduces average accuracy\.

Delta hindsight trap rate \(Δ\\DeltaHTR\)\.

Δ​HTRq=HTRq​\(aq,full\)−HTRq​\(aq,masked\)\.\\Delta\\mathrm\{HTR\}\_\{q\}=\\mathrm\{HTR\}\_\{q\}\(a\_\{q,\\mathrm\{full\}\}\)\-\\mathrm\{HTR\}\_\{q\}\(a\_\{q,\\mathrm\{masked\}\}\)\.\(4\)A positiveΔ​HTRq\\Delta\\mathrm\{HTR\}\_\{q\}indicates greater trap attraction under full exposure, meaning post\-cutoff information increases hindsight bias\.

Answer instability rate \(AIR\)\.AIR records any answer change, regardless of direction:

AIRq=𝟏\[aq,full≢aq,masked\]∈\{0,1\}\.\\mathrm\{AIR\}\_\{q\}=\\mathbf\{1\}\[a\_\{q,\\mathrm\{full\}\}\\not\\equiv a\_\{q,\\mathrm\{masked\}\}\]\\in\\\{0,1\\\}\.\(5\)AIR upper\-bounds any bias effect because a model answer must change to exhibit hindsight bias\.

Hindsight bias rate \(HBR\)\.HBR captures the joint event that a model is fully correct under masking and moves toward the trap under full exposure\. It is defined only forbothquestions, for which bothaq∗a\_\{q\}^\{\*\}andaqtrapa\_\{q\}^\{\\mathrm\{trap\}\}are available\. LetIqM=𝟏\[Accq\(aq,masked\)=1\]I\_\{q\}^\{\\mathrm\{M\}\}=\\mathbf\{1\}\[\\mathrm\{Acc\}\_\{q\}\(a\_\{q,\\mathrm\{masked\}\}\)=1\]\. Then

HBRq=IqM⋅HTRq​\(aq,full\),\\mathrm\{HBR\}\_\{q\}=I\_\{q\}^\{\\mathrm\{M\}\}\\cdot\\mathrm\{HTR\}\_\{q\}\(a\_\{q,\\mathrm\{full\}\}\),\(6\)so HBR is nonzero only when the masked answer is correct\.

#### 3\.7\.3Statistical analyses

We estimated 95% confidence intervals using a case\-clustered bootstrap with 2,000 replicates\. Cases were sampled with replacement, and all questions and paired responses from each sampled case were retained\. Sepsis and GLP\-1/diabetes were analyzed separately\. For metrics evaluated relative to zero, effects were considered statistically significant when the 95% bootstrap confidence interval excluded zero\.

### 3\.8Human validation

Four clinical annotators \(one physician and three medical or physician assistant trainees\) reviewed an error\-enriched 10% sample of cases, split into two non\-overlapping subsets with two annotators each\. Annotators assessed whether each question was clinically reasonable and whether they agreed with the prospective reference and hindsight\-trap answers\. Sampling and validation details are provided in Appendix[G](https://arxiv.org/html/2609.13454#A7)\.

## 4Results

Figure[3](https://arxiv.org/html/2609.13454#S4.F3)summarizes Accuracy and Hindsight Trap Rate \(HTR\) for GPT\-5\.6 Sol on the GLP and Sepsis datasets\. Condition 0 serves as the full information baseline, while Conditions 1\.1–3\.2 allow direct comparison between the full and masked settings\. Results for Gemma 4, GLM 5\.2, and Opus 5 are provided in Appendix[E](https://arxiv.org/html/2609.13454#A5)\.

Figure 3:Accuracy and Hindsight Trap Rate \(HTR\) across experimental conditions for GPT for the GLP and sepsis datasets\. Condition 0 represents the full information setting\. For Conditions 1\.1–3\.2, results are shown for both the full and masked settings\. Points indicate observed values and error bars indicate 95% bootstrap confidence intervals\.Figure 4:Hindsight bias metrics across paired experimental conditions for GPT for the GLP and sepsis datasets\. We reportΔ\\DeltaHTR, AIR, and HBR for Conditions 1\.1–3\.2\. Points indicate observed values and error bars indicate 95% bootstrap confidence intervals\.##### Naive narrative answering already shows substantial hindsight bias\.

Before considering the paired masking experiments, condition 0 establishes that hindsight bias is already a meaningful problem when the model answers questions directly from the clinical narrative\. As shown in Figure[3](https://arxiv.org/html/2609.13454#S4.F3)C and Figure[3](https://arxiv.org/html/2609.13454#S4.F3)D, in both datasets, the hindsight trap rate \(HTR\) is significantly greater than zero\. 14–15% of responses are hindsight\-biased when naively answering questions based on narratives\.

##### Paired masking reveals robust hindsight bias effects\.

The paired conditions \(1\.x–3\.x\) allow hindsight bias to be quantified directly using three complementary metrics:Δ\\DeltaHTR, AIR, and HBR\. As seen in Figure[4](https://arxiv.org/html/2609.13454#S4.F4), across both datasets, AIR and HBR are significantly greater than zero in every paired condition, indicating that access to future information systematically changes model responses and, more stringently, can shift initially correct prospective answers toward predefined hindsight traps\. This is particularly notable for HBR, which is a strict subset of AIR and therefore distinguishes hindsight\-biased shifts from general answer instability\. The evidence is strongest under strict temporal masking \(Conditions 2\.x and 3\.x\), where all three metrics are significantly positive across both datasets\. For example, AIR reaches 16\.0% in Sepsis Condition 2\.2, while HBR reaches 5\.5% in the same condition, demonstrating that a meaningful subset of responses not only changed after future information was revealed but shifted specifically from a prospectively correct answer toward the hindsight trap\. Overall, the consistency ofΔ\\DeltaHTR, AIR, and HBR under strict masking shows that the observed effects are not merely general response instability: access to future clinical information systematically shifts model judgments toward hindsight\-consistent answers\.

##### Condition 1 provides weaker temporal isolation\.

In conditions 1\.1 and 1\.2,Δ\\DeltaHTR remains small and non\-significant despite significant AIR and HBR \(Figure[4](https://arxiv.org/html/2609.13454#S4.F4)\)\. Unlike conditions 2\.x and 3\.x, the masked variant in condition 1 retains the complete original narrative, which may contain post\-cutoff or outcome\-relevant information\. Consequently, the full–masked comparison isolates only the effect of masking the TTS, not the effect of removing future information from the model’s entire input\. The significant AIR and HBR nevertheless show that changing TTS access can alter responses, including shifts from prospectively correct answers toward hindsight traps, even when the retrospective narrative remains available\.

##### Human versus LLM TTS\.

Across both datasets, the human\-TTS variants generally achieve slightly higher absolute accuracy than the corresponding LLM\-TTS variants\. This comparison should be interpreted in light of the benchmark construction: the human TTS was used as the reference representation during question generation\. Discrepancies in the LLM\-reconstructed TTS—including differences in event content or timing—may therefore omit information needed to answer a question or place information on the wrong side of its masking cutoff\.

##### The synthetic narrative conditions support the masking interpretation\.

Conditions 3\.x are especially informative because they retain a narrative style prompt while restoring temporal control through a synthetic narrative\. Their results closely track the strict masking pattern seen in 2\.x: all hindsight bias metrics are significantly positive, and their magnitudes are much closer to 2\.x than to the attenuated 1\.x conditions\. This suggests that the weaker effects in 1\.x are not due to narrative prompting per se, but to leakage from the original unmasked narrative\. The synthetic narrative therefore appears to preserve the benefits of a narrative format without undermining the masking manipulation\.

##### Bias reduction does not come at the cost of accuracy\.

In contrast to the consistent hindsight bias signal, Figure[3](https://arxiv.org/html/2609.13454#S4.F3)A and Figure[3](https://arxiv.org/html/2609.13454#S4.F3)B highlight thatΔ\\DeltaAcc is not significant in any paired condition in either dataset\. Although some point estimates are positive and others negative, there is no reliable evidence that access to the full case improves accuracy relative to the masked version\. This is a central result that masking reduces hindsight bias without imposing a detectable accuracy penalty\. Put differently, future information is systematically biasing the model’s answers, but not meaningfully helping it answer correctly\. This strengthens the case for temporally masked evaluation as a way to reduce hindsight bias while preserving task performance\.

##### Explicit hindsight warning attenuates but does not eliminate the effect\.

As a sensitivity analysis, we repeated the Sepsis evaluation for GPT and GLM with explicit instructions to avoid hindsight\. The warning generally reduced hindsight\-related metrics, particularly under strict masking, while accuracy remained broadly similar\. However, AIR and HBR remained positive across paired conditions, indicating that hindsight\-sensitive response changes persisted despite the explicit instruction\. Full results are reported in Appendix[F](https://arxiv.org/html/2609.13454#A6)\.

##### Human validation supports benchmark label validity\.

Human review of the error\-enriched validation sample found that 95\.9% of questions were judged clinically reasonable, with agreement rates of 98\.7% for the prospective reference answers and 97\.1% for the hindsight\-trap answers\. Inter\-annotator agreement was 91\.8% for question reasonableness, 99\.2% for the prospective answer, and 94\.1% for the hindsight trap\. Detailed results are reported in Appendix[G](https://arxiv.org/html/2609.13454#A7)\.

## 5Discussion

##### Principal findings\.

Across both cohorts, exposure to the complete timeline shifted model responses toward outcome\-consistent alternatives\. In the unpaired narrative only baseline, the hindsight trap rate was 14–15%; because this condition has no masked counterpart, it is best interpreted as a descriptive baseline rate rather than a causal estimate of hindsight bias\. The paired conditions provide stronger evidence\. AIR and HBR were positive across all paired input settings, and strict temporal masking \(conditions 2\.x and 3\.x\) produced the largest changes in trap attraction, withΔ\\DeltaHTR ranging from 5\.8% to 9\.6% and HBR from 3\.3% to 5\.5%\. By contrast,Δ\\DeltaAcc was statistically insignificant\. Thus, later information changed a subset of answers in the outcome\-consistent direction without a corresponding improvement in overall accuracy\.

The condition comparisons clarify the role of information control\. In conditions 1\.x, the original narrative remained complete in both variants, so future information was shared across the pair and the masked–full contrast was attenuated\. Conditions 2\.x removed the narrative, while conditions 3\.x paired each TTS with a synthetic narrative generated from the corresponding information scope\. The larger effects in these strictly controlled settings suggest that the information boundary, rather than narrative format alone, drives the measured difference\. Human\- and LLM\-generated TTS produced the same qualitative pattern, although effect sizes varied by cohort and condition\. While our experiments do not identify the source of the observed bias, its persistence after explicit hindsight warning prompting suggests that instructions alone may be insufficient to eliminate it\. This result suggests that input level temporal control may be more reliable than instruction alone when evaluating judgments about an earlier clinical state\.

##### Validity of benchmark labels\.

Human review provides additional support for the validity of the GPT\-generated questions, prospective reference answers, and hindsight traps \(Section[3\.8](https://arxiv.org/html/2609.13454#S3.SS8); Appendix[G](https://arxiv.org/html/2609.13454#A7)\)\. Importantly, the validation set was intentionally enriched for cases in which models made errors or exhibited hindsight\-sensitive behavior—the same cases that contribute most directly to our evidence of hindsight bias\. Positive human validation on this challenging subset therefore strengthens confidence that the benchmark’s central findings are not driven by invalid questions or labels\.

Furthermore, our various metrics provide robustness to annotation error by examining hindsight bias from different angles that rely on different annotation components: AIR requires only an answer change, Acc depends on the prospective reference answer, HTR depends on the hindsight\-trap answer, and HBR combines both\. Convergence across these complementary measures, particularly under strict masking, makes it less likely that the observed hindsight\-bias signal is an artifact of any single annotation label or metric definition\.

##### Limitations\.

Our studies have several limitations\.First, the reference time was hidden and questions were intentionally unanchored, so some full–masked differences may reflect changes in task interpretation rather than hindsight bias\. Explicitly specifying the reference time and allowable evidence could reduce this ambiguity, but such tightly structured questions may not reflect real clinical deployments, where models may need to infer the relevant temporal context from available information\.Second, free\-response scoring relies on an LLM judge and may be sensitive to paraphrase or judge\-specific variation\. However, the judge only compares responses with predefined reference and hindsight\-trap answers, and most questions use structured formats, reducing reliance on LLM\-based scoring\.Third, published case reports overrepresent unusual and educational cases, limiting generalizability to routine clinical practice\. We use these datasets because they are publicly available and, importantly, provide manually annotated TTS, enabling controlled evaluation of temporal reasoning with high\-quality representations of the clinical trajectories\.Fourth, temporal masking relies on recorded timestamps, which may be imprecise or misplace information relative to a cutoff\. Moreover, observation time may not reflect the true onset or duration of a clinical state\.

##### Future directions\.

Evaluations of earlier clinical decisions should restrict model inputs to information available at the decision point and report directional measures of outcome\-sensitive change alongside accuracy\. Future work should extend the benchmark to routine longitudinal EHRs, larger expert\-reviewed samples, and counterfactual outcome controls, while investigating training strategies and model architectures that better enforce prospective information boundaries\.

## 6Conclusion

We introduced a paired benchmark for testing whether later clinical information changes LLM judgments about an earlier decision point\. Across two clinical cohorts and four models, complete trajectories increased movement toward outcome\-consistent hindsight traps, whereas temporal masking reduced these shifts without a detectable loss in accuracy\. These findings show that retrospective clinical evaluation can overstate prospective reasoning performance and motivate evaluation protocols that restrict model inputs to information available at the target decision time\.

## Acknowledgements

This research was supported by the Intramural Research Program of the National Institutes of Health \(NIH\) and utilized the computational resources of the[NIH HPC Biowulf cluster](http://hpc.nih.gov/)\. The contributions of the NIH author\(s\) are considered Works of the United States Government\. The findings and conclusions presented in this paper are those of the author\(s\) and do not necessarily reflect the views of the NIH or the U\.S\. Department of Health and Human Services\.

## References

- Arkes et al\. \(1981\)H\. R\. Arkes, R\. L\. Wortmann, P\. D\. Saville, and A\. R\. Harkness\.Hindsight bias among physicians weighing the likelihood of diagnoses\.*Journal of Applied Psychology*, 66\(2\):252–254, 1981\.[10\.1037/0021\-9010\.66\.2\.252](https://doi.org/10.1037/0021-9010.66.2.252)\.
- Banham\-Hall and Stevens \(2019\)Edward Banham\-Hall and Sian Stevens\.Hindsight bias critically impacts on clinicians’ assessment of care quality in retrospective case note review\.*Clinical Medicine*, 19\(1\):16–21, 2019\.[10\.7861/clinmedicine\.19\-1\-16](https://doi.org/10.7861/clinmedicine.19-1-16)\.
- Choi et al\. \(2026\)Yera Choi, Yeong Hwa Kim, JaeDeok Lee, Taekang Kim, Sangdoo Yun, and Seong\-Eun Moon\.ASCENT: A benchmark for evaluating and advancing stepwise diagnostic reasoning in large language models on common clinical scenarios\.In*Proceedings of the 7th Conference on Health, Inference, and Learning*, volume 333 of*Proceedings of Machine Learning Research*, pages 404–426\. PMLR, 2026\.URL[https://proceedings\.mlr\.press/v333/choi26a\.html](https://proceedings.mlr.press/v333/choi26a.html)\.
- Chu et al\. \(2024\)Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin\.TimeBench: A comprehensive evaluation of temporal reasoning abilities in large language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1204–1228, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.[10\.18653/v1/2024\.acl\-long\.66](https://doi.org/10.18653/v1/2024.acl-long.66)\.URL[https://aclanthology\.org/2024\.acl\-long\.66/](https://aclanthology.org/2024.acl-long.66/)\.
- Cui et al\. \(2025\)Hejie Cui, Alyssa Unell, Bowen Chen, Jason Alan Fries, Emily Alsentzer, Sanmi Koyejo, and Nigam H\. Shah\.TIMER: Temporal instruction modeling and evaluation for longitudinal clinical records\.*npj Digital Medicine*, 8:577, 2025\.[10\.1038/s41746\-025\-01965\-9](https://doi.org/10.1038/s41746-025-01965-9)\.URL[https://www\.nature\.com/articles/s41746\-025\-01965\-9](https://www.nature.com/articles/s41746-025-01965-9)\.
- Dawson et al\. \(1988\)Neal V\. Dawson, Hal R\. Arkes, Carl Siciliano, Richard Blinkhorn, Mark Lakshmanan, and Mary Petrelli\.Hindsight bias: An impediment to accurate probability estimation in clinicopathologic conferences\.*Medical Decision Making*, 8\(4\):259–264, 1988\.[10\.1177/0272989X8800800406](https://doi.org/10.1177/0272989X8800800406)\.
- Degany et al\. \(2025\)Or Degany, Sahar Laros, Daphna Idan, and Sharon Einav\.Evaluating the o1 reasoning large language model for cognitive bias: a vignette study\.*Critical Care*, 29\(1\):376, 2025\.[10\.1186/s13054\-025\-05591\-5](https://doi.org/10.1186/s13054-025-05591-5)\.
- Fatemi et al\. \(2025\)Bahare Fatemi, Seyed Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi\.Test of time: A benchmark for evaluating LLMs on temporal reasoning\.In*International Conference on Learning Representations*, 2025\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/eb7295a8bc613b375726659c2ecd6f14\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/eb7295a8bc613b375726659c2ecd6f14-Abstract-Conference.html)\.
- Fischhoff \(1975\)Baruch Fischhoff\.Hindsight is not equal to foresight: The effect of outcome knowledge on judgment under uncertainty\.*Journal of Experimental Psychology: Human Perception and Performance*, 1\(3\):288–299, 1975\.[10\.1037/0096\-1523\.1\.3\.288](https://doi.org/10.1037/0096-1523.1.3.288)\.
- Fleming et al\. \(2024\)Scott L\. Fleming, Alejandro Lozano, William J\. Haberkorn, Jenelle A\. Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z\. Genkins, Ethan Steinberg, Ashwin Nayak, Birju Patel, Chia\-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott Adams, Oluseyi Fayanju, Shreya J\. Shah, Thomas Savage, Ethan Goh, Akshay S\. Chaudhari, Nima Aghaeepour, Christopher Sharp, Michael A\. Pfeffer, Percy Liang, Jonathan H\. Chen, Keith E\. Morse, Emma P\. Brunskill, Jason A\. Fries, and Nigam H\. Shah\.MedAlign: A clinician\-generated dataset for instruction following with electronic medical records\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 22021–22030, 2024\.[10\.1609/aaai\.v38i20\.30205](https://doi.org/10.1609/aaai.v38i20.30205)\.URL[https://ojs\.aaai\.org/index\.php/AAAI/article/view/30205](https://ojs.aaai.org/index.php/AAAI/article/view/30205)\.
- Kumar and Weiss \(2026\)Sayantan Kumar and Jeremy C\. Weiss\.Temporally phenotyping GLP\-1RA case reports with large language models: A textual time series corpus and risk modeling\.*medRxiv*, 2026\.[10\.64898/2026\.04\.05\.26350197](https://doi.org/10.64898/2026.04.05.26350197)\.Preprint\.
- Kweon et al\. \(2024\)Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwanghyun Kim, Jeewon Yang, Seunghyun Won, and Edward Choi\.EHRNoteQA: An LLM benchmark for real\-world clinical practice using discharge summaries\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.[10\.52202/079017\-3958](https://doi.org/10.52202/079017-3958)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/hash/e15c4afff22f12c4986c1fcb4e941e03\-Abstract\-Datasets\_and\_Benchmarks\_Track\.html](https://proceedings.neurips.cc/paper_files/paper/2024/hash/e15c4afff22f12c4986c1fcb4e941e03-Abstract-Datasets_and_Benchmarks_Track.html)\.Datasets and Benchmarks Track\.
- Lee et al\. \(2022\)Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong\-Yeup Kim, and Edward Choi\.EHRSQL: A practical text\-to\-SQL benchmark for electronic health records\.In*Advances in Neural Information Processing Systems*, volume 35, 2022\.[10\.52202/068431\-1134](https://doi.org/10.52202/068431-1134)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/hash/643e347250cf9289e5a2a6c1ed5ee42e\-Abstract\-Datasets\_and\_Benchmarks\.html](https://proceedings.neurips.cc/paper_files/paper/2022/hash/643e347250cf9289e5a2a6c1ed5ee42e-Abstract-Datasets_and_Benchmarks.html)\.Datasets and Benchmarks Track\.
- Liu et al\. \(2026\)Yachuan Liu, Xiaochun Wei, Lin Shi, Xinnuo Li, Bohan Zhang, Paramveer Dhillon, and Qiaozhu Mei\.ExAnte: A benchmark for ex\-ante inference in large language models\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1551–1571, Rabat, Morocco, March 2026\. Association for Computational Linguistics\.[10\.18653/v1/2026\.eacl\-long\.72](https://doi.org/10.18653/v1/2026.eacl-long.72)\.URL[https://aclanthology\.org/2026\.eacl\-long\.72/](https://aclanthology.org/2026.eacl-long.72/)\.
- National Library of Medicine \(2025\)National Library of Medicine\.Pubmed central open access subset \(PMC OA\)\.National Library of Medicine Open Data Portal, 2025\.URL[https://datadiscovery\.nlm\.nih\.gov/Literature/PubMed\-Central\-Open\-Access\-Subset\-PMC\-OA\-/qbit\-hxvw](https://datadiscovery.nlm.nih.gov/Literature/PubMed-Central-Open-Access-Subset-PMC-OA-/qbit-hxvw)\.Last updated June 18, 2025; accessed August 8, 2026\.
- Noroozizadeh and Weiss \(2026\)Shahriar Noroozizadeh and Jeremy Weiss\.Reconstructing sepsis trajectories from clinical case reports using LLMs: The textual time series corpus for sepsis\.In*Proceedings of the 7th Conference on Health, Inference, and Learning*, volume 333 of*Proceedings of Machine Learning Research*, pages 10–51\. PMLR, 2026\.URL[https://proceedings\.mlr\.press/v333/noroozizadeh26a\.html](https://proceedings.mlr.press/v333/noroozizadeh26a.html)\.
- Qiu et al\. \(2025\)Pengcheng Qiu, Chaoyi Wu, Shuyu Liu, Yanjie Fan, Weike Zhao, Zhuoxia Chen, Hongfei Gu, Chuanjin Peng, Ya Zhang, Yanfeng Wang, and Weidi Xie\.Quantifying the reasoning abilities of LLMs on clinical cases\.*Nature Communications*, 16:9799, 2025\.[10\.1038/s41467\-025\-64769\-1](https://doi.org/10.1038/s41467-025-64769-1)\.URL[https://www\.nature\.com/articles/s41467\-025\-64769\-1](https://www.nature.com/articles/s41467-025-64769-1)\.
- Shen et al\. \(2026\)Chengzhi Shen, Weixiang Shen, Tobias Susetzky, Chen Chen, Jun Li, Yuyuan Liu, Xuepeng Zhang, Zhenyu Gong, Daniel Rueckert, and Jiazhen Pan\.RealICU: Do LLM agents understand long\-context ICU data? a benchmark beyond behavior imitation\.*arXiv preprint arXiv:2605\.13542*, 2026\.[10\.48550/arXiv\.2605\.13542](https://doi.org/10.48550/arXiv.2605.13542)\.URL[https://arxiv\.org/abs/2605\.13542](https://arxiv.org/abs/2605.13542)\.
- Wang and Redelmeier \(2024\)Jonathan Wang and Donald A\. Redelmeier\.Cognitive biases and artificial intelligence\.*NEJM AI*, 1\(12\):AIcs2400639, 2024\.[10\.1056/AIcs2400639](https://doi.org/10.1056/AIcs2400639)\.
- Wang and Zhao \(2024\)Yuqing Wang and Yun Zhao\.TRAM: Benchmarking temporal reasoning for large language models\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 6389–6415, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.[10\.18653/v1/2024\.findings\-acl\.382](https://doi.org/10.18653/v1/2024.findings-acl.382)\.URL[https://aclanthology\.org/2024\.findings\-acl\.382/](https://aclanthology.org/2024.findings-acl.382/)\.
- Wei et al\. \(2025\)Shaohang Wei, Wei Li, Feifan Song, Wen Luo, Tianyi Zhuang, Haochen Tan, Zhijiang Guo, and Houfeng Wang\.TimE: A multi\-level benchmark for temporal reasoning of LLMs in real\-world scenarios\.In*Advances in Neural Information Processing Systems*, volume 38, 2025\.[10\.52202/085713\-3082](https://doi.org/10.52202/085713-3082)\.URL[https://papers\.neurips\.cc/paper\_files/paper/2025/hash/84f1e188c2be52f89f6e206bc37d092d\-Abstract\-Datasets\_and\_Benchmarks\_Track\.html](https://papers.neurips.cc/paper_files/paper/2025/hash/84f1e188c2be52f89f6e206bc37d092d-Abstract-Datasets_and_Benchmarks_Track.html)\.Datasets and Benchmarks Track\.
- Wu et al\. \(2025\)Kevin Wu, Eric Wu, Rahul Thapa, Kevin Wei, Angela Zhang, Arvind Suresh, Jacqueline J\. Tao, Min Woo Sun, Alejandro Lozano, and James Zou\.MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports\.*arXiv preprint arXiv:2505\.11733*, 2025\.[10\.48550/arXiv\.2505\.11733](https://doi.org/10.48550/arXiv.2505.11733)\.URL[https://arxiv\.org/abs/2505\.11733](https://arxiv.org/abs/2505.11733)\.
- Wu et al\. \(2024\)Zhenbang Wu, Anant Dadu, Mike Nalls, Faraz Faghri, and Jimeng Sun\.Instruction tuning large language models to understand electronic health records\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.[10\.52202/079017\-1737](https://doi.org/10.52202/079017-1737)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/hash/62986e0a78780fe5f17b495aeded5bab\-Abstract\-Datasets\_and\_Benchmarks\_Track\.html](https://proceedings.neurips.cc/paper_files/paper/2024/hash/62986e0a78780fe5f17b495aeded5bab-Abstract-Datasets_and_Benchmarks_Track.html)\.Datasets and Benchmarks Track\.

Supplementary Material

This supplementary material provides detailed information organized as follows:

- •Appendix[A](https://arxiv.org/html/2609.13454#A1):Detailed literature review\.
- •Appendix[B](https://arxiv.org/html/2609.13454#A2):Experimental conditions and input configurations\.
- •Appendix[C](https://arxiv.org/html/2609.13454#A3):Prompts used for question construction, answer generation, evaluation, and synthetic narrative generation\.
- •Appendix[D](https://arxiv.org/html/2609.13454#A4):Distribution of temporal masking cutoffs across the clinical trajectories\.
- •Appendix[E](https://arxiv.org/html/2609.13454#A5):Complete results across other models \(Gemma, GLM, Opus\)\.
- •Appendix[F](https://arxiv.org/html/2609.13454#A6):Sensitivity analysis with explicit hindsight warning\.
- •Appendix[G](https://arxiv.org/html/2609.13454#A7):Human validation of benchmark questions, prospective answers, and hindsight traps\.

## Appendix ADetailed Literature Review

##### General temporal reasoning

Several benchmarks evaluate temporal reasoning in general domain LLMs\. TRAM covers event order, arithmetic, frequency, and duration\([Wang and Zhao, 2024](https://arxiv.org/html/2609.13454#bib.bib20)\); TimeBench provides a hierarchical evaluation across a broad range of temporal phenomena\([Chu et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib4)\); Test of Time uses controlled synthetic tasks to study temporal logic and sensitivity to problem structure\([Fatemi et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib8)\); and TimE evaluates multi\-level reasoning over Wikipedia, news, and dialogue scenarios\([Wei et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib21)\)\. Most closely related, ExAnte evaluates inference under explicit temporal cutoffs and measures leakage from post\-cutoff knowledge across several non\-clinical tasks\([Liu et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib14)\)\. These benchmarks establish that temporal constraints remain difficult for LLMs, but they do not study clinical judgments or quantify whether exposing the later course of the same patient trajectory moves an answer toward a clinically defined hindsight trap\.

##### Clinical question answering over longitudinal records\.

Clinical benchmarks increasingly test LLMs on realistic or longitudinal EHR tasks\. MedAlign contains clinician\-generated instructions grounded in longitudinal records\([Fleming et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib10)\); MIMIC\-Instr provides large\-scale instruction\-following examples derived from MIMIC\-IV\([Wu et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib23)\); and EHRNoteQA evaluates questions spanning one or more discharge summaries\([Kweon et al\., 2024](https://arxiv.org/html/2609.13454#bib.bib12)\)\. EHRSQL tests text\-to\-SQL reasoning over structured EHR databases, including time\-sensitive questions and temporal expressions\([Lee et al\., 2022](https://arxiv.org/html/2609.13454#bib.bib13)\)\. TIMER explicitly links instructions to timestamps and evaluates temporal boundary adherence, trend detection, and chronological precision in longitudinal EHRs\([Cui et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib5)\)\. ASCENT evaluates stepwise diagnostic reasoning as evidence accumulates under incomplete information\([Choi et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib3)\)\. RealICU evaluates windowed ICU decision support using prefix\-only observations and labels produced after physicians review the full trajectory\([Shen et al\., 2026](https://arxiv.org/html/2609.13454#bib.bib18)\); unlike our paired design, it uses hindsight to construct reference labels rather than measuring how a model’s own answer changes after outcome exposure\. These works evaluate retrieval, temporal synthesis, instruction following, evolving diagnosis, or prospective performance against retrospectively informed labels\. We instead hold the question and prospective reference target fixed and compare masked with complete\-record exposure to measure the directional effect of later information on the model’s response\.

##### Reasoning from clinical case reports\.

Clinical case reports have also been used to construct challenging reasoning benchmarks\. MedCaseReasoning pairs diagnostic cases with detailed reasoning statements derived from open\-access case reports and evaluates both final diagnosis and reasoning recall\([Wu et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib22)\)\. MedR\-Bench structures case reports into examination recommendation, diagnostic decision\-making, and treatment\-planning tasks\([Qiu et al\., 2025](https://arxiv.org/html/2609.13454#bib.bib17)\)\. Related work has reconstructed time\-localized sepsis trajectories from PubMed Open Access case reports as textual time series\([Noroozizadeh and Weiss, 2026](https://arxiv.org/html/2609.13454#bib.bib16)\)\. These resources improve the realism and granularity of clinical reasoning evaluation, but primarily assess whether a model reaches or explains a final clinical conclusion\. Our benchmark instead asks when a conclusion was supportable and measures how the same answer changes when post\-cutoff events become visible\.

## Appendix BExperimental conditions

We evaluated seven input conditions \(Table[1](https://arxiv.org/html/2609.13454#S3.T1)\), each defining what information the answering model has access to when responding to a question\. The question set itself was identical across all conditions: all questions were generated from the original narrativeNiN\_\{i\}paired with the full human TTSTiHT\_\{i\}^\{\\mathrm\{H\}\}, and the answering model never sees the generation inputs or the hidden evaluation metadata\.

##### Condition 0

Condition 0 serves as a naive baseline representing the setting most common in current clinical QA systems, where the model is given a complete retrospective case report and asked to answer questions without any temporal masking\. Because case reports are written retrospectively and may reference diagnoses, treatments, and outcomes in any order regardless of when they occurred, there is no reliable way to mask post\-cutoff information from a narrative\. Condition 0 therefore reflects the upper bound of potential hindsight exposure under narrative\-only input\.

##### Condition 1\.x

Conditions 1\.x address this limitation partially by pairing the original narrative with a temporally masked TTS\. The narrative component is held fixed and complete across the full and masked variants; only the TTS is truncated at the cutoff\. While masking the TTS removes explicit post\-cutoff temporal data, the original narrative may still implicitly contain outcome information, meaning that conditions 1\.x achieve only partial information control\.

##### Condition 2\.x

Conditions 2\.x remove the narrative entirely, providing the TTS alone\. This yields the cleanest paired comparison between prefix\-only and complete temporal information, as the TTS is the sole information source and can be precisely truncated atcqc\_\{q\}\.

##### Condition 3\.x

Conditions 3\.x also achieve rigorous temporal control\. Conditions 3\.x are motivated by the complementary limitations of conditions 1\.x and 2\.x: while condition 1\.x retains a narrative but cannot fully mask it, and condition 2\.x achieves strict masking but strips away the clinical context that a narrative provides, conditions 3\.x combine both desiderata by supplying a narrative that is simultaneously present and strictly temporally controlled\. The synthetic narrative presented to the answering model is derived from the same masked TTS used in that condition variant, as described in Section[3\.2](https://arxiv.org/html/2609.13454#S3.SS2)\. Because the synthetic narrative is generated from pre\-cutoff input, it cannot contain implicit references to subsequent events\. Paired with a masked TTS, conditions 3\.x ensure that all information available to the answering model — both narrative and temporal — is strictly limited to what was observable before the cutoff, a guarantee that conditions 1\.x cannot provide\.

## Appendix CPrompts

Question generation promptYou are generating difficult clinical reasoning questions from case reports to build a benchmark evaluating accuracy and hindsight bias in language models\. Questions should be genuinely hard—the answering model should make errors, especially by using outcome knowledge to revise past assessments it should not revise\.Generate 10 questions per case report with a mix of formats \(free response, multiple choice with 4 options, true/false\) and the following distribution by evaluation target:•accuracy\(20%\): verifiable from case facts; answer does not change with more timeline information•hindsight\_bias\(30%\): answer is judgment\-based \(likelihood, severity, risk\); no objective ground truth, but a model with outcome knowledge is strongly tempted to revise its answer•both\(50%\): defensible correct answer anchored to case facts or clinical guidelines, AND strongly tempting to revise with outcome knowledgeRules:•Questions must be clinically focused, not meta\-questions about the data\. Never phrase a question in terms of what the TTS “establishes,” “shows,” “records,” or “contains\.” Ask about the clinical facts themselves \(e\.g\., “Did angiography occur before embolization?” not “Does the TTS establish a strict chronological sequence between angiography and embolization?”\)\.•Do not include any time reference in the question text\. Questions must be atemporal—e\.g\., “What was the probability of a serious infectious complication?” not “At readmission, what was the probability…”\. The answering model’s timeline is controlled externally through masking; an explicit time reference tips off prospective reasoning\.•Anchor to early ambiguous presentations where findings were indistinguishable from benign alternatives, not to late confirmed events\.•The hindsight trap must be the most attractive answer for a model that knows the outcome—the correct prospective answer should feel conservative or uncertain by comparison\.•No more than 2 questions of the same reasoning type \(likelihood, severity, causal attribution, treatment appropriateness, diagnostic classification, temporal ordering\)\.•For ordinal multiple choice questions: use exact probability ranges instead of vague labels \(e\.g\., “A\.<<10%”, “B\. 10–30%”, “C\. 30–60%”, “D\.\>\>60%”\)\. The correct answer and hindsight trap must be at opposite ends of the scale—if the correct answer is the lowest option, the hindsight trap must be the highest, and vice versa\.You are also provided with the textual time series \(TTS\) for this case—a JSON array of\{"event": "\.\.\.", "time": <hours\>\}tuples ordered chronologically by time\.Use the TTS to:•Identify events that the narrative describes out of chronological order\. These are especially valuable for hindsight bias questions—the narrative’s non\-chronological framing may lead a model reading only the narrative to assume a different temporal order than what actually occurred\. Design questions that target these moments, where a model relying on narrative framing would answer differently than one reasoning from the true chronological sequence\.•Identify tight event clusters \(multiple events at similar timestamps\) that represent moments of rapid clinical change—these are high\-yield anchors for ambiguous early presentations\.Return a valid JSON array only\. Each element:\{"question": "\.\.\.", "format": "free response \| multiple choice \| true/false", "options": \["A\. \.\.\.", "B\. \.\.\.", "C\. \.\.\.", "D\. \.\.\."\] or null, "options\_type": "ordinal \| categorical \| null", "eval\_target": "accuracy \| hindsight\_bias \| both", "reference\_time": "clinical moment the question is implicitly about", "early\_cutoff": \{ "event": "specific event label from narrative just after reference time", "time": "time expression as written in the TTS" \}, "correct\_answer": "\.\.\.", "hindsight\_trap": "the most tempting wrong answer for a model with full\-case access; null if eval\_target is accuracy", "hindsight\_trap\_explanation": "one sentence naming the post\-cutoff fact that makes the trap so tempting; null if eval\_target is accuracy" \}

Answer generation prompt: Condition 0 \(Narrative only\)You answer clinical reasoning questions using only the provided case report\.For each request, you will receive:1\.A case report\.2\.One JSON question object containing only these fields:"question","format", and"options"\.Answer only from the visible case report and the visible question object\. Use general medical knowledge only to interpret clinical language, not to invent patient\-specific facts\.Rules:•If"format"is"true/false", answer exactly"true"or"false"\.•If"format"is"multiple choice", answer with exactly one option string from"options", including the option letter and text\.•If"format"is"free response", answer concisely in one to three sentences\.•Always include an"evidence"field explaining why you gave the answer\.•The evidence should be a concise case\-grounded rationale, ideally naming the relevant event, symptom, test, treatment, timing, or absence of needed information from the case report\.•Do not include reasoning, citations, markdown, or extra text outside the JSON\.Return only valid JSON in this exact shape:``` {"answer": "...", "evidence": "..."} ```

Answer generation prompt: Conditions 1 and 3 \(Narrative \+ TTS\)You answer clinical reasoning questions using the provided case report and its textual time series \(TTS\)\.For each request, you will receive:1\.A case report\.2\.A TTS—a JSON array of\{"event": "\.\.\.", "time": <hours\>\}tuples ordered chronologically by time relative to an arbitrary anchor event\.3\.One JSON question object containing only these fields:"question","format", and"options"\.Answer using both the case report and the TTS\. When they conflict, trust the TTS for event ordering and timing, and the narrative for clinical detail\.Use general medical knowledge only to interpret clinical language, not to invent patient\-specific facts\.Rules:•If"format"is"true/false", answer exactly"true"or"false"\.•If"format"is"multiple choice", answer with exactly one option string from"options", including the option letter and text\.•If"format"is"free response", answer concisely in one to three sentences\.•Always include an"evidence"field explaining why you gave the answer\.•The evidence should be a concise rationale grounded in the case report or TTS, naming the relevant event, timestamp, symptom, test, treatment, or absence of needed information\.•Do not include reasoning, citations, markdown, or extra text outside the JSON\.Return only valid JSON in this exact shape:``` {"answer": "...", "evidence": "..."} ```

Answer generation prompt: Condition 2 \(TTS only\)You answer clinical reasoning questions using only the provided textual time series \(TTS\)\.For each request, you will receive:1\.A TTS—a JSON array of\{"event": "\.\.\.", "time": <hours\>\}tuples ordered chronologically by time relative to an arbitrary anchor event\.2\.One JSON question object containing only these fields:"question","format", and"options"\.Answer using only the TTS\. Use general medical knowledge only to interpret clinical language, not to invent patient\-specific facts\.Rules:•If"format"is"true/false", answer exactly"true"or"false"\.•If"format"is"multiple choice", answer with exactly one option string from"options", including the option letter and text\.•If"format"is"free response", answer concisely in one to three sentences\.•Always include an"evidence"field explaining why you gave the answer\.•The evidence should be a concise rationale grounded in the TTS, naming the relevant event, timestamp, or absence of needed information\.•Do not include reasoning, citations, markdown, or extra text outside the JSON\.Return only valid JSON in this exact shape:``` {"answer": "...", "evidence": "..."} ```

Hindsight warning promptFor the sensitivity analysis experiments, the answer\-generation prompts for Conditions 0–3 were identical to the corresponding prompts above, except that the following two sentences were appended to the beginning \(for GPT\) or end \(for GLM\) of each prompt:Use your medical expertise for this reasoning task; do not let hindsight cloud your judgment\. Use only knowledge available at the time; do not let the final outcome of the case affect your answer\.

Synthetic case generation prompt: Condition 3You are a medical expert tasked with reconstructing a synthetic clinical case report from a patient timeline presented in textual time\-series \(TTS\) format\. The input contains two columns: event and time in hours, where the timeline conventions are as follows: Time 0 represents the patient’s hospital admission; events occurring before admission have negative time values and events occurring after admission have positive time values\.Guidelines:•Use clinical and natural language to convert fragmented timeline entries into a readable narrative\.•Follow the order in which events appear in the TTS file as the narrative sequence\.•Central events include: symptom onset or major clinical changes; hospital admission, transfer, discharge, or death; diagnostic tests and clinically significant results; diagnoses; procedures and interventions; medication initiation, discontinuation, or major adjustment; complications and treatment responses\.•Express the timing of central events relative to nearby central events \(e\.g\., “two weeks after that,” “one month after presentation,” “shortly after admission”\) using any unit of time \(hours, days, weeks, months, years\) that is the most natural to read\.•Use a standalone or absolute timepoint only when there is no clear anchoring event or when it is clearer than a relative phrase\.•Include only information explicitly supported by the input timeline\.•Do not infer, invent, or add diagnoses, symptoms, treatments, results, outcomes, demographic details, or causal relationships that are not documented\.•When the timeline is ambiguous or incomplete, report the information conservatively without any speculation\.Output Requirements:Return only the synthetic clinical case report\. Do not include headings about the task, explanations, notes, disclaimers, or commentary outside the report\.Example input:42\-year\-old \| 0 female \| 0 no pathological history \| 0 consulted the rheumatology clinic \| 0 initiated subcutaneous semaglutide for weight loss \| \-1008 starts complaining of myalgia \| \-672 edema in the four limbs \| \-672 review of symptoms was negative \| \-672 new contrast MRI took place \| \+730 notable decrease in inflammatory changes in the limb muscles \| \+730Example output:A 42\-year\-old female patient with no pathological history consulted the rheumatology clinic\. Six weeks before the consultation, she had initiated subcutaneous semaglutide for weight loss\. Two weeks after that, she starts complaining of myalgia and edema in the four limbs\. The review of symptoms by systems was negative\. One month after presentation, a new contrast MRI showed a notable decrease in the extent of inflammatory changes in the limb muscles\.

GPT judge prompt: Individual metrics for free\-response questionsYou are evaluating an answer to a clinical temporal\-QA question\.Return only JSON\. Decide clinical meaning, not exact wording\.•accuracy\_scoremust be 1 for fully correct, 0\.5 for partially correct, and 0 for incorrect\.•htr\_scoremust be 1 if the answer fully matches thehindsight\_trap, 0\.5 if it partially reflects the hindsight trap, and 0 if it does not reflect the trap\.•Ifhindsight\_trapis null,htr\_scoremust be 0\.

GPT judge prompt: Paired metrics for free\-response questionsYou are evaluating paired answers to a clinical temporal\-QA question\.Return only JSON\. Decide clinical meaning, not exact wording\.•full\_equals\_maskedis true if the full\-condition and masked\-condition answers mean the same thing clinically\.•Forfull\_accuracy\_scoreandmasked\_accuracy\_score, use 1 for fully correct, 0\.5 for partially correct, and 0 for incorrect\.•Forfull\_htr\_scoreandmasked\_htr\_score, use 1 if the answer fully matches thehindsight\_trap, 0\.5 if it partially reflects the hindsight trap, and 0 if it does not reflect the trap\.•Ifhindsight\_trapis null, both HTR scores andhbr\_scoremust be 0\.•Forhbr\_score, give hindsight\-bias credit only to the extent that the masked answer is correct and the full answer reflects the trap; use 1, 0\.5, or 0\.

## Appendix DDistribution of Masking Cutoffs

To characterize where masking occurs within the clinical timelines, we define the relative masking cutoff position as the chronological index of the cutoff event within each TTS, normalized to the interval\[0,1\]\[0,1\], where 0 and 1 correspond to the beginning and end of the timeline, respectively\. This measure reflects position within the sequence of recorded events rather than elapsed clinical time\.

Table[D\.1](https://arxiv.org/html/2609.13454#A4.T1)summarizes the distribution of relative cutoff positions,RelPos⁡\(q\)\\operatorname\{RelPos\}\(q\), across the two datasets\. In both cases, the mean and median were near the midpoint of the timeline, and the standard deviations indicate substantial spread across temporal positions\. The observed ranges were also broad, showing that questions span much of the clinical trajectory rather than clustering within a narrow temporal window\.

Table D\.1:Distribution of relative cutoff positionRelPos⁡\(q\)\\operatorname\{RelPos\}\(q\)across questions in each dataset\.Figure[D\.1](https://arxiv.org/html/2609.13454#A4.F1)shows that masking points are distributed broadly across the clinical timeline rather than being concentrated at a single position\. Sepsis cutoffs tend to occur somewhat earlier than GLP cutoffs, particularly in the middle of the timeline, although both datasets span nearly the full range of relative positions\. This distribution indicates that the prospective evaluation is conducted at diverse stages of the underlying clinical trajectories rather than at a fixed or narrowly defined point in each case\.

Figure D\.1:Empirical cumulative distribution of relative masking cutoff positions for GLP and Sepsis\. Relative position is defined using the chronological index of the cutoff event within the TTS and normalized to\[0,1\]\[0,1\], with 0 representing the beginning and 1 the end of the recorded event sequence\.
## Appendix EResults for Other Models

Figures[E\.1](https://arxiv.org/html/2609.13454#A5.F1)–[E\.6](https://arxiv.org/html/2609.13454#A5.F6)report the case\-level 95% bootstrap confidence intervals for the individual and paired evaluation metrics for Gemma\-4, GLM\-5\.2, and Opus\-5\.

Figure E\.1:Accuracy and HTR results for Gemma with case\-level 95% bootstrap confidence intervals\.Figure E\.2:Hindsight bias metrics for Gemma with case\-level 95% bootstrap confidence intervals\.Figure E\.3:Accuracy and HTR results for GLM with case\-level 95% bootstrap confidence intervals\.Figure E\.4:Hindsight bias metrics for GLM with case\-level 95% bootstrap confidence intervals\.Figure E\.5:Accuracy and HTR results for Opus\-5 with case\-level 95% bootstrap confidence intervals\.Figure E\.6:Hindsight bias metrics for Opus\-5 with case\-level 95% bootstrap confidence intervals\.##### Comparison across models\.

Across GPT\-5\.6 Sol, Gemma\-4, GLM\-5\.2, and Opus\-5, accuracy remained generally high, with relatively modest full–masked differences for most paired conditions \(Figure[E\.7](https://arxiv.org/html/2609.13454#A5.F7)\)\. Condition 0 is an unpaired original\-narrative baseline and is therefore shown only under full information\. In contrast to accuracy, HTR showed a clearer masking effect: full and masked HTR were similar in the 1\.x conditions, whereas masked HTR was consistently lower across models in the strict 2\.x and 3\.x conditions \(Figure[E\.8](https://arxiv.org/html/2609.13454#A5.F8)\)\.

The paired metrics reinforce this pattern \(Figure[E\.9](https://arxiv.org/html/2609.13454#A5.F9)\)\.Δ\\DeltaHTR was near zero in the 1\.x conditions but consistently positive under strict masking, while AIR also increased substantially in the 2\.x and 3\.x conditions\. Positive HBR further indicates that some answer changes were directional, shifting from the prospective reference under masking toward the hindsight trap when post\-cutoff information was restored\. Although effect sizes varied across models and datasets, the same qualitative pattern appeared across all four model families\.

Cross\-model differences should be interpreted cautiously\. GPT was the only answerer from the same family as the question generator, and GPT was also used to judge free\-response answers; however, the similar strict\-masking pattern among Gemma, GLM, and Opus suggests that the overall effect is not specific to GPT\. The plots also suggest that lower\-performing model–condition combinations can exhibit greater degradation under masking, but this relationship is not monotonic across models or datasets\. Thus, overall model quality may contribute to robustness, but accuracy and susceptibility to hindsight information appear to capture distinct aspects of model behavior\.

Figure E\.7:Accuracy across models and input conditions\.Accuracy for GPT\-5\.6 Sol, Gemma\-4, GLM\-5\.2, and Opus\-5 on GLP and Sepsis under full \(top\) and masked \(bottom\) information\. Asterisks denote strict masking conditions\.Figure E\.8:Hindsight Trap Rate across models and input conditions\.HTR for GPT\-5\.6 Sol, Gemma\-4, GLM\-5\.2, and Opus\-5 on GLP and Sepsis under full \(top\) and masked \(bottom\) information\. Asterisks denote strict masking conditions\.Figure E\.9:Paired hindsight\-bias metrics across models\.Δ\\DeltaHTR, Answer Instability Rate \(AIR\), and Hindsight Bias Rate \(HBR\) for GPT, Gemma, GLM, and Opus on GLP and Sepsis\. Asterisks denote strict masking conditions\.

## Appendix FSensitivity Analysis with Explicit Hindsight Warning

Figures[F\.1](https://arxiv.org/html/2609.13454#A6.F1)–[F\.2](https://arxiv.org/html/2609.13454#A6.F2)and[F\.3](https://arxiv.org/html/2609.13454#A6.F3)–[F\.4](https://arxiv.org/html/2609.13454#A6.F4)report the sensitivity analysis using explicit hindsight warning for GPT and GLM, respectively\. Error bars indicate case\-level 95% bootstrap confidence intervals\. The modification in prompts can be found in Appendix[C](https://arxiv.org/html/2609.13454#A3)\.

Figure F\.1:Accuracy and hindsight trap rate \(HTR\) across conditions for GPT under both prompt versions\. Error bars show case\-level 95% bootstrap confidence intervals\.Figure F\.2:Hindsight bias metrics across paired conditions for GPT under both prompt versions\. Metrics include the change in hindsight trap rate \(Δ\\DeltaHTR\), AIR, and HBR\. Error bars show case\-level 95% bootstrap confidence intervals\.Figure F\.3:Accuracy and hindsight trap rate \(HTR\) across conditions for GLM under both prompt versions\. Error bars show case\-level 95% bootstrap confidence intervals\.Figure F\.4:Hindsight bias metrics across paired conditions for GLM under both prompt versions\. Metrics include the change in hindsight trap rate \(Δ\\DeltaHTR\), AIR, and HBR\. Error bars show case\-level 95% bootstrap confidence intervals\.Across both models, explicit hindsight warning attenuated hindsight\-biased responding but did not eliminate it\. The reduction was most apparent in Conditions 2\.1–3\.2, where full\-condition HTR,Δ\\DeltaHTR, and HBR were generally lower under the modified prompt\. AIR was also reduced, particularly for GPT, but remained positive across the paired conditions, as did HBR, indicating persistent hindsight\-sensitive answer changes despite the explicit warning\. Accuracy remained broadly similar between prompt versions for both models, suggesting that the reductions in hindsight\-related metrics were not driven by a general deterioration in task performance\. The consistency of this pattern across GLM and GPT indicates that models can respond to explicit instructions against hindsight bias, but that simple prompt\-level mitigation is insufficient to eliminate the effect\.

## Appendix GHuman Validation

This appendix provides additional details on the human validation described in Section[3\.8](https://arxiv.org/html/2609.13454#S3.SS8), including the error\-enriched sampling procedure and validation results\.

##### Annotators and review procedure\.

Four clinical annotators participated: one physician and three medical or physician assistant trainees\. We sampled 10% of cases for review and divided them into two non\-overlapping 5% subsets, each independently evaluated by two annotators\. For each question, annotators assessed \(1\) whether the question was clinically reasonable, \(2\) whether they agreed with the prospective reference answer, and \(3\) whether they agreed with the hindsight\-trap answer\. All judgments were binary \(yes/no\)\.

##### Error\-enriched sampling\.

Rather than sampling uniformly, we preferentially selected cases in which the evaluated model exhibited errors or hindsight\-sensitive behavior\. For each case, we computed an error\-proneness score \(EPS\) from four model\-derived quantities:1−Accuracy1\-\\mathrm\{Accuracy\}and HTR under Condition 0, and AIR and HBR under Condition 2\.1\. Each quantity was aggregated to the case level and min–max normalized within cohort:

EPSi=\\displaystyle\\mathrm\{EPS\}\_\{i\}=\{\}0\.4​\(1−Accuracy\)~i\+0\.2​HTR~i\\displaystyle 0\.4\\,\\widetilde\{\(1\-\\mathrm\{Accuracy\}\)\}\_\{i\}\+0\.2\\,\\widetilde\{\\mathrm\{HTR\}\}\_\{i\}\(G\.1\)\+0\.2​AIR~i\+0\.2​HBR~i\.\\displaystyle\+0\.2\\,\\widetilde\{\\mathrm\{AIR\}\}\_\{i\}\+0\.2\\,\\widetilde\{\\mathrm\{HBR\}\}\_\{i\}\.
Cases were sampled without replacement with weights

wi∝exp⁡\(EPSiτ\),τ=0\.3\.w\_\{i\}\\propto\\exp\\left\(\\frac\{\\mathrm\{EPS\}\_\{i\}\}\{\\tau\}\\right\),\\qquad\\tau=0\.3\.Thus, a 0\.3\-unit increase in EPS corresponds to an approximatelye≈2\.72e\\approx 2\.72\-fold increase in relative sampling weight\. This design provides a targeted stress test of benchmark annotations in cases most relevant to the observed model errors and hindsight effects\.

##### Human validation results\.

The validation sample comprised 170 questions, each reviewed by two annotators\. Agreement with the benchmark annotations was high across question reasonableness, prospective reference answers, and hindsight\-trap answers, with sample agreement ranging from 95\.9% to 98\.7% \(Table[G\.1](https://arxiv.org/html/2609.13454#A7.T1)\)\. Inter\-annotator agreement was similarly high \(91\.8–99\.2%\)\.

Because sampling favored cases with higher EPS, we additionally report inverse\-probability\-weighted estimates based on the cohort\-specific sampling design\. These closely matched the corresponding sample estimates \(Table[G\.1](https://arxiv.org/html/2609.13454#A7.T1)\) and should be interpreted as design\-weighted point estimates; uncertainty intervals would require case\-level resampling\.

Table G\.1:Human validation results\. Sample estimates are calculated from the error\-enriched validation set; weighted estimates use inverse\-probability weights from the cohort\-specific sampling design\.Note\.Sample agreement is calculated from individual annotator assessments\. Inter\-annotator agreement is the proportion of applicable questions for which both annotators assigned to the subset gave the same judgment\. Not\-applicable answer assessments are excluded from the corresponding denominators\.