CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives

# CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives
Source: [https://arxiv.org/html/2608.12779](https://arxiv.org/html/2608.12779)
Chengyang HeThanks:Stevens Institute of Technology, Hoboken, NJ, 07030, USA \([che14@stevens\.edu](mailto:[email protected]),[tarif1@stevens\.edu](mailto:[email protected]),[yue\.ning@stevens\.edu](mailto:[email protected]),[pwang44@stevens\.edu](mailto:[email protected])\)Marko ZivkovicThanks:Genesis Research Group, Hoboken, NJ, 07030,USA \([marko\.zivkovic@genesisrg\.com](mailto:[email protected])\)Lijing WangThanks:New Jersey Institute of Technology, Newark, NJ, 07102, USA \([lijing\.wang@njit\.edu](mailto:[email protected])Yue Ning11footnotemark:1Ping Wang11footnotemark:1

###### Abstract

Understanding the temporal progression of symptoms in clinical narratives is critical for disease monitoring, safety surveillance, and causality assessment\. Clinical narratives, however, rarely provide explicit temporal anchors\. Current approaches to temporal information reasoning focus predominantly on pairwise relation classification across multi\-visit and timestamp\-rich records, leaving the reconstruction of structured symptom trajectories from individual anchor\-sparse reports largely unaddressed\. We propose CRAFT, an LLM framework that pairs a generator with a constraint\-based verifier to iteratively produce and refine stage\-wise symptom timelines through targeted feedback\. We conduct evaluation on MedTempo, a new benchmark of 5,347 vaccine adverse\-event narratives spanning three COVID\-19 vaccine types, with expert\-validated temporal stage annotations for 3,166 reports\. Experiments across four LLM backbones demonstrate that CRAFT consistently improves temporal ordering accuracy, with ablation analysis isolating the contribution of generator and verifier components across model capability levels\.

## 1Introduction\.

Clinical temporal reasoning, recovering a stage\-wise trajectory of clinical events from narrative text, is central to disease progression modeling, treatment outcome monitoring, and safety signal detection\[[10](https://arxiv.org/html/2608.12779#bib.bib24),[20](https://arxiv.org/html/2608.12779#bib.bib19)\]\. However, constructing such orderings from free text remains labor\-intensive\. Temporal cues in these narratives are frequently implicit or expressed relative to other events rather than anchored to fixed dates, and co\-mentions, restatements, and status updates further obscure the true chronological order of symptoms\[[37](https://arxiv.org/html/2608.12779#bib.bib21),[1](https://arxiv.org/html/2608.12779#bib.bib20)\]\.

A large body of work on temporal information reasoning has focused on identifying events and temporal expressions and predicting pairwise relations to derive a global ordering\[[16](https://arxiv.org/html/2608.12779#bib.bib16),[25](https://arxiv.org/html/2608.12779#bib.bib37)\], with recent efforts expanding to exhaustive relation coverage and complex temporal fact extraction\[[2](https://arxiv.org/html/2608.12779#bib.bib17),[8](https://arxiv.org/html/2608.12779#bib.bib18)\]\. In the clinical domain, however, progress has been bottlenecked by limited benchmark diversity, with most work concentrated on a small number of corpora and relation inventories\[[1](https://arxiv.org/html/2608.12779#bib.bib20),[28](https://arxiv.org/html/2608.12779#bib.bib25)\]\. Recent LLM\-based approaches to clinical temporal reasoning further assume multi\-visit timelines, timestamp\-linked supervision, or constrained data settings\[[3](https://arxiv.org/html/2608.12779#bib.bib23),[10](https://arxiv.org/html/2608.12779#bib.bib24),[35](https://arxiv.org/html/2608.12779#bib.bib22)\]\. As a result, standardized methods and benchmarks for ordering symptom progressions within single\-report, anchor\-sparse clinical narratives remain underexplored\.

To address this gap, we proposeCRAFT\(ClinicalRefinement withAdaptiveFeedback forTemporal ordering\), a generator–verifier framework that models temporal trajectory reconstruction as an iterative structured prediction task under weak temporal anchoring, where candidate trajectories are refined via constraint\-based feedback\. Inspired by iterative refinement paradigms such as Self\-Refine\[[17](https://arxiv.org/html/2608.12779#bib.bib33)\], CRAFT introduces a structured trajectory representation and a task\-specific verification mechanism tailored to temporal ordering, and can be instantiated with different generator and verifier configurations\. In this work, we instantiate CRAFT asCRAFT\-Full, which pairs a full\-regeneration generator with a multi\-criterion additive verifier; we additionally define two baselines \(PIVOT,GUIDE\) and two ablations \(CRAFT\-G,CRAFT w/o V\) to isolate the contribution of the generator and verifier components respectively\.

To enable rigorous evaluation, we introduceMedTempo\(MedicalTemporalOrdering Benchmark\), a benchmark for temporal progression reconstruction from medical free text\. MedTempo contains5,347narrative reports spanning three vaccine types, each consisting of a single report per patient with no explicit absolute time anchor and paired with a provided symptom list\. Our benchmark task focuses on the3,166reports that exhibit temporal evidence of distinct symptom progression, for which we provide expert\-validated stage\-wise ordering annotations\. The remaining reports contain no temporal progression and are retained in the dataset release to support future work on temporal\-evidence identification, but fall outside the scope of the current benchmark evaluation\. Our primary contributions are as follows:

- •We propose CRAFT, a generator–verifier framework for iterative temporal reasoning refinement under weak temporal anchoring, with controlled baselines and ablations that isolate generator and verifier contributions\.
- •We introduce MedTempo, an expert\-annotated benchmark for evaluating structured temporal trajectories over anchor\-sparse clinical narratives\.
- •We conduct extensive experiments across four LLMs, revealing distinct refinement behaviors tied to model capability and verifier calibration\.

Figure[4\.1](https://arxiv.org/html/2608.12779#S4.F1)provides a schematic overview of the full pipeline, from dataset construction through iterative refinement and post\-hoc evaluation\.

## 2Related Work\.

Temporal information extraction has been shaped by the TimeML annotation framework\[[23](https://arxiv.org/html/2608.12779#bib.bib6)\], which introduced a markup language for events, temporal expressions, and their relations\. The TempEval shared tasks\[[31](https://arxiv.org/html/2608.12779#bib.bib7)\]established standardized evaluation for pairwise temporal relation classification, with systems progressing from rule\-based and CRF/SVM approaches to neural methods\. A common paradigm is to predict pairwise relations and then induce a globally consistent ordering\[[16](https://arxiv.org/html/2608.12779#bib.bib16),[9](https://arxiv.org/html/2608.12779#bib.bib35),[33](https://arxiv.org/html/2608.12779#bib.bib34)\], while more recent benchmarks emphasize richer annotation for event ordering\[[2](https://arxiv.org/html/2608.12779#bib.bib17)\]and LLM\-based strategies for temporally grounded fact extraction\[[8](https://arxiv.org/html/2608.12779#bib.bib18)\]\. Transformer\-based methods have become the dominant paradigm, as surveyed in\[[28](https://arxiv.org/html/2608.12779#bib.bib25)\]\. However, these efforts typically center on local relation correctness rather than the end\-to\-end reconstruction of an ordered, grouped trajectory under weak anchoring\.

In the clinical domain, the i2b2 2012 challenge\[[29](https://arxiv.org/html/2608.12779#bib.bib8)\]brought temporal relation extraction to clinical discharge summaries, followed by the Clinical TempEval shared tasks\[[6](https://arxiv.org/html/2608.12779#bib.bib9)\]on the THYME corpus\[[27](https://arxiv.org/html/2608.12779#bib.bib5)\], where best\-performing systems evolved from CRF/SVM classifiers to LSTMs\[[30](https://arxiv.org/html/2608.12779#bib.bib31)\]\. Surveys document persistent difficulties including implicit time anchors, inter\-sentence relations, and the gap between relation\-level extraction and usable patient timelines\[[20](https://arxiv.org/html/2608.12779#bib.bib19),[1](https://arxiv.org/html/2608.12779#bib.bib20)\]\. More recent work applies neural end\-to\-end methods to established clinical corpora\[[19](https://arxiv.org/html/2608.12779#bib.bib26)\], and LLM\-based approaches have begun to examine prompting and fine\-tuning for clinical temporal relation extraction\[[13](https://arxiv.org/html/2608.12779#bib.bib29),[4](https://arxiv.org/html/2608.12779#bib.bib27),[3](https://arxiv.org/html/2608.12779#bib.bib23),[36](https://arxiv.org/html/2608.12779#bib.bib28)\]as well as timeline extraction from medical case reports\[[32](https://arxiv.org/html/2608.12779#bib.bib30)\]\. However, these approaches typically require longitudinal records spanning multiple visits or rely on structured temporal metadata\[[10](https://arxiv.org/html/2608.12779#bib.bib24),[35](https://arxiv.org/html/2608.12779#bib.bib22)\]\. In contrast, CRAFT operates on single\-report clinical narratives where temporal cues are sparse or implicit\. To support evaluation in this underexplored setting, MedTempo provides expert\-annotated temporal trajectory benchmarks derived from real\-world adverse event narratives\.

Table 2\.1:Descriptive statistics for MedTempo by vaccine type\. Text Len\. denotes clinical narrative length in words; \#Stages denotes number of temporal stages\.MedTempoMedTempo\-TMedTempo\-NTVaccineMetricMedMinMaxMedMinMaxMedMinMaxPfizer\(1789/1019/770\)Text Len\.10211231910912163881112319\# Symp\.646464295464\# Stages20103110000Moderna\(1769/983/786\)Text Len\.83111056991390557111056\# Symp\.643064305429\# Stages209329000Janssen\(1789/1164/625\)Text Len\.871113171001282656111317\# Symp\.644374435433\# Stages20133213000

A parallel line of work applies iterative refinement to structured prediction, most notably the Self\-Refine paradigm\[[17](https://arxiv.org/html/2608.12779#bib.bib33)\], which iterates over model outputs using self\-generated feedback\. In the clinical domain, Hein et al\.\[[14](https://arxiv.org/html/2608.12779#bib.bib36)\]apply iterative refinement with human\-in\-the\-loop review cycles to improve extraction precision\. However, such approaches are not suitable for systematic benchmark evaluation across model tiers, as they conflate model capability with human reviewer effort\. CRAFT leverages iterative refinement for structured clinical temporal extraction in a fully automated setting, pairing a generator with a multi\-criterion verifier and enabling principled comparison across frontier and open\-weight models\.

## 3Dataset Creation\.

This section describes how MedTempo is constructed from VAERS\. We first introduce the source corpus and the information each report provides\. We then describe the filtering and stratified sampling that reduce the corpus to reports carrying temporal signal, followed by the annotation protocol that produces the gold\-standard timelines\. We close with summary statistics of the resulting benchmark\.

### 3\.1VAERS Dataset\.

The Vaccine Adverse Event Reporting System \(VAERS\), managed jointly by the CDC and FDA, is a passive surveillance database in which healthcare professionals, patients, and manufacturers submit reports of adverse events following immunization\[[7](https://arxiv.org/html/2608.12779#bib.bib32)\]\. Each report includes demographics, vaccination details, free\-text clinical narratives, and MedDRA\-coded symptom lists\. We focus on three widely administered COVID\-19 vaccines: Pfizer\-BioNTech, Moderna, and Janssen, covering reports from 2021 to 2024\.

### 3\.2Data Sampling\.

Table 3\.1:Annotation agreement and adjudication rates \(%\) overall and by subset\. Ann\. = Annotator; T = MedTempo\-T with temporal report only; NT = MedTempo\-NT with no temporal report\. Ambiguity exclusions are not applicable \(–\) in the T subset by definition\.CategoryMetricMedTempoTNTAgreementModel–Ann\. 1788291Model–Ann\. 2737886Inter\-Annotator939494AdjudicationAccepted809090Corrected91010Excluded11––Starting from the full VAERS corpus, we applied a multi\-stage filtering and stratified sampling pipeline\. We first removed non\-symptom MedDRA terms by building an exclusion list from three non\-clinical System Organ Classes \(Surgical and medical procedures,Social circumstances,Product issues\) combined with human\-annotated non\-symptom labels, yielding5,601excluded terms\. Reports with three or fewer distinct symptoms were discarded as they lack sufficient temporal variation\. Reports of ten or fewer words \(typically bare symptom lists without narrative context\) were also removed\. For reports between 11 and 30 words, we applied rule\-based temporal keyword filtering \(relative markers such asbefore/after, duration terms, date patterns\) to retain only those with explicit temporal cues; reports of 30 or more words were retained unconditionally\. Finally, stratified sampling balanced by year and report length was applied per vaccine to draw2,000 records each, preserving the distribution of the underlying VAERS corpus\. Appendix[A\.1](https://arxiv.org/html/2608.12779#S1.SS1)presents representative examples of reports removed under each criterion\.

### 3\.3Annotation of Temporal Sequence\.

Temporal timelines were produced via ahuman\-in\-the\-loopthree\-phase protocol:GPT\-4o mini\[[21](https://arxiv.org/html/2608.12779#bib.bib4)\]generated initial stage\-ordered timelines; two annotators with a medical NLP background independently reviewed and labeled each sequence; and all disagreements and uncertain cases were resolved through collaborative adjudication\. As shown in Table[3\.1](https://arxiv.org/html/2608.12779#S3.T1), human inter\-annotator agreement \(IAA\) was 93%\. After adjudication, 80% of LLM annotations were accepted without change, 9% were corrected, and 11% were excluded for temporal ambiguity, yielding a final corpus of 5,347 records\.

### 3\.4Dataset Statistics and Analysis\.

Table[2\.1](https://arxiv.org/html/2608.12779#S2.T1)summarizes the 3,166 temporally\-evident reports that form the primary benchmark subsetMedTempo\-T; the remaining 2,181 reports contain no temporal progression and are released separately asMedTempo\-NT\. Symptom count is consistent across vaccines \(median 6\), while narrative length varies: Pfizer\-BioNTech reports are longest \(median 102 words\) versus Moderna \(83\) and Janssen \(87\)\. Because symptom count is stable regardless of length, longer reports likely contribute richer contextual cues rather than additional adverse events, providing stronger signals for temporal extraction\. Figure[3\.1](https://arxiv.org/html/2608.12779#S3.F1)shows that thermoregulatory symptoms \(pyrexia, chills\) dominate early stages, while headache and fatigue are the most frequent overall, illustrating the diversity of symptom trajectories in the benchmark\.

![Refer to caption](https://arxiv.org/html/2608.12779v1/Figures/Dataset/top_15_symptoms.png)Figure 3\.1:Distribution of the most frequent symptoms \(left\) and the symptoms that occurred after \(right\)\. Each chart displays a total of 15 unique symptoms, representing the top 10 symptoms from each vaccine type and the overall dataset combined\.

## 4Method\.

This section presents CRAFT\. We first formalize the ordering task and define the stage\-based representation used throughout the paper, fixing the notation for reports, findings, and predicted timelines\. Then, we describe the framework itself: the generator that proposes a timeline, the verifier that scores it and returns feedback, and the loop that connects them\. Figure[4\.1](https://arxiv.org/html/2608.12779#S4.F1)gives an overview of both the dataset pipeline and the framework\.

![Refer to caption](https://arxiv.org/html/2608.12779v1/CRAFT_Figure_CR_V3.png)Figure 4\.1:Overview of MedTempo and CRAFT\.\(a\)Dataset construction pipeline from VAERS narratives to gold\-standard stage\-ordered timelines\.\(b\)CRAFT iterative generator–verifier loop, instantiated across four configurations \(CRAFT\-Full, PIVOT, GUIDE, CRAFT\-G\) and evaluated on four LLM backbones\.### 4\.1Problem Formulation and Temporal Representation\.

For each post\-vaccination reportrr, letxrx\_\{r\}denote its free\-text clinical narrative andℱ⁡\(r\)=\{f1,…,fn\}\\mathcal\{F\}\(r\)=\\\{f\_\{1\},\\dots,f\_\{n\}\\\}the provided list of adverse clinical findings \(MedDRA Preferred Terms from the VAERSSYMPTOMfields\)\. We treatℱ⁡\(r\)\\mathcal\{F\}\(r\)as given and do not perform entity extraction or coding\.

Our goal is to predict an explicit temporal ordering overℱ⁡\(r\)\\mathcal\{F\}\(r\)when the narrative expresses a*temporal progression*\(i\.e\., at least one new finding appears*after*previously mentioned findings\)\. We exclude concurrent onset, changes in severity or resolution without new onsets, and timelines inferable only from durations \(e\.g\., “finding A for 4 days, finding B for 3 days” without an explicit order\)\.

Formally, we produce an ordered sequence of non\-empty time buckets

ℬ⁡\(r\)=\(B1,B2,…,BK\),\\mathcal\{B\}\(r\)=\(B\_\{1\},B\_\{2\},\\dots,B\_\{K\}\),where each finding inℱ⁡\(r\)\\mathcal\{F\}\(r\)is assigned to exactly one bucketBkB\_\{k\}, grouping together findings that occurred at the same point in the patient’s clinical course, and indexkkorders the buckets from earliest to latest\. This representation is stored as a JSON list of buckets, which is the format used by both the generator and verifier below\. Models are evaluated solely on temporal structure \(ordering and grouping\); optional evidence snippets are not scored\.

### 4\.2CRAFT: Iterative Generator–Verifier Framework\.

CRAFT operates as a fully automated iterative loop: the generator proposes a candidate temporal sequence, the verifier scores it against structural and temporal constraints and returns targeted feedback, and the loop repeats until the candidate is accepted or a fixed iteration budget is exhausted\. Figure[4\.1](https://arxiv.org/html/2608.12779#S4.F1)\(b\) and algorithm[1](https://arxiv.org/html/2608.12779#alg1)illustrate this process\.

#### 4\.2\.1Generator Agent\.

At iterationii, the generator LLM implements

g:\(ℱ⁡\(r\),xr,feedback\(i−1\)\)⟶ℬ^\(i\)​\(r\),g:\\big\(\\mathcal\{F\}\(r\),\\,x\_\{r\},\\,\\text\{feedback\}^\{\(i\-1\)\}\\big\)\\;\\longrightarrow\\;\\hat\{\\mathcal\{B\}\}^\{\(i\)\}\(r\),wherefeedback\(i−1\)\\text\{feedback\}^\{\(i\-1\)\}is empty ati=1i\{=\}1and contains verifier guidance thereafter\.

CRAFT\-Full uses a full\-regeneration generator: a single prompt template with full task instructions at each iteration, with verifier feedback appended to the context when available\. The generator receives: \(i\) a task description and definition of temporal progression, including phenomena that should*not*be treated as progression; \(ii\) the finding listℱ⁡\(r\)\\mathcal\{F\}\(r\), which must appear exactly as given in the output; and \(iii\) the free\-text narrativexrx\_\{r\}\. It outputs a JSON bucket sequence using only findings fromℱ⁡\(r\)\\mathcal\{F\}\(r\)and avoiding unsupported temporal inferences\.

#### 4\.2\.2Verifier Agent\.

The verifier implements

v:\(ℱ⁡\(r\),xr,ℬ^\(i\)​\(r\)\)→\(decision\(i\),feedback\(i\),score\(i\)\),v:\\big\(\\mathcal\{F\}\(r\),\\,x\_\{r\},\\,\\hat\{\\mathcal\{B\}\}^\{\(i\)\}\(r\)\\big\)\\rightarrow\\big\(\\text\{decision\}^\{\(i\)\},\\,\\text\{feedback\}^\{\(i\)\},\\,\\text\{score\}^\{\(i\)\}\\big\),wheredecision\(i\)∈\{ACCEPT,REVISE\}\\text\{decision\}^\{\(i\)\}\\in\\\{\\text\{ACCEPT\},\\text\{REVISE\}\\\},feedback\(i\)\\text\{feedback\}^\{\(i\)\}describes issues to fix, andscore\(i\)∈\{0,…,5\}\\text\{score\}^\{\(i\)\}\\in\\\{0,\\dots,5\\\}supports early stopping\. The verifier first callsFormatTool, a deterministic helper module that normalizes the raw generator output into the JSON bucket schema without altering temporal content, then applies an additive rubric that assigns one point each for: valid JSON with earliest→\\rightarrowlatest ordering; placing non\-mentioned symptoms as‘‘none’’in the final group; using each symptom exactly once; grouping symptoms that occur around the same time; and ordering groups according to temporal cues in the narrative\. When the score meets thresholdθ\\theta, the verifier returnsACCEPT; otherwise it returnsREVISEwith targeted feedback for the next iteration\. If no candidate is accepted withinTmaxT\_\{\\max\}iterations, the loop terminates and the last candidate is returned\. The complete procedure is given in Algorithm[1](https://arxiv.org/html/2608.12779#alg1)\.

## 5Experimental Setup\.

We organize our evaluation around three research questions:

- •RQ1 Method effectiveness\.Does CRAFT\-Full outperform the baselines \(PIVOT, GUIDE\) across model tiers, and what drives the difference?
- •RQ2 Model capability\.How well do different LLM backbones perform on MedTempo, and is the capability ordering stable across configurations?
- •RQ3 Vaccine discrepancy\.To what extent do performance and error patterns vary across vaccine types, and are model rankings consistent under vaccine\-stratified evaluation?

The remainder of this section specifies the evaluation data, the model backbones and their settings, the implementation of each configuration, and the metrics used to score predicted timelines against the gold standard\.

### 5\.1Dataset\.

MedTempo contains 5,347 vaccine adverse\-event narratives across three COVID\-19 vaccine types, each paired with a provided symptom list\. We evaluate on the 3,166 reports from MedTempo\-T with temporal evidence of distinct symptom progression; the remaining reports contain no temporal progression and fall outside the primary benchmark\.

### 5\.2Models and Settings\.

We evaluate four LLMs:GPT\-4\.1\[[22](https://arxiv.org/html/2608.12779#bib.bib10)\],Claude Sonnet 4\.5\[[5](https://arxiv.org/html/2608.12779#bib.bib11)\],MedGemma\-27B\[[26](https://arxiv.org/html/2608.12779#bib.bib13)\], andLlama\-3\.3\-70B\[[18](https://arxiv.org/html/2608.12779#bib.bib12)\]\. Proprietary models are accessed via their respective APIs with deterministic decoding\. Open\-weight models run locally using Hugging Facetransformers\[[34](https://arxiv.org/html/2608.12779#bib.bib2)\]with 4\-bit quantization \(NF4 viabitsandbytes\[[11](https://arxiv.org/html/2608.12779#bib.bib3)\]\)\.

We evaluate five settings that systematically vary the generator and verifier components to isolate the contribution of each\. Our proposed method, CRAFT\-Full, pairs the full\-regeneration generator with the additive rubric verifier \(Section[4\.2\.1](https://arxiv.org/html/2608.12779#S4.SS2.SSS1)–[4\.2\.2](https://arxiv.org/html/2608.12779#S4.SS2.SSS2)\)\. Two baselines represent alternative design choices grounded in established paradigms: PIVOT pairs the full\-regeneration generator with an anchor\-based verifier inspired by the document\-creation\-time \(DCT\) anchoring tradition in temporal extraction\[[33](https://arxiv.org/html/2608.12779#bib.bib34),[9](https://arxiv.org/html/2608.12779#bib.bib35)\], where a fixed reference point serves as a hub for ordering events\.GUIDEpairs an edit\-conditioned generator, which applies targeted local edits to the previous candidate rather than regenerating from scratch, with the same anchor\-based verifier\. CRAFT\-G swaps in the edit\-conditioned generator while holding the additive verifier fixed, isolating the generator’s contribution; CRAFT w/o V removes the verification loop entirely, running one generator pass without feedback, isolating the verifier’s contribution\.

##### Edit\-conditioned generator \(CRAFT\-G, GUIDE\)\.

Uses a dedicated initialization prompt ati=1i\{=\}1, switching to a lightweight editing prompt that conditions on the previous candidate and verifier feedback ati\>1i\{\>\}1\.

##### Anchor\-based verifier \(PIVOT, GUIDE\)\.

Treats the vaccination date as the dominant time anchor and scores conservatively, starting from 5 and subtracting one point for clear violations: anchor\-order contradictions, grouping errors \(over\-merge or over\-split\), or overall inconsistency with temporal cues\.

Algorithm 1CRAFT: Iterative generator–verifier loop0:

ℱ⁡\(r\)\\mathcal\{F\}\(r\), narrative

xrx\_\{r\}, generator

GG, verifier

VV,

TmaxT\_\{\\max\}, threshold

θ\\theta
0:

ℬ^​\(r\)\\hat\{\\mathcal\{B\}\}\(r\)
feedback

←∅\\leftarrow\\emptyset
for

t=1,…,Tmaxt=1,\\dots,T\_\{\\max\}do

raw←G⁡\(ℱ⁡\(r\),xr,feedback\)\\text\{raw\}\\leftarrow G\\\!\\left\(\\mathcal\{F\}\(r\),\\,x\_\{r\},\\,\\textit\{feedback\}\\right\)

ℬ^​\(r\)←FormatTool​\(raw\)\\hat\{\\mathcal\{B\}\}\(r\)\\leftarrow\\textsc\{FormatTool\}\(\\text\{raw\}\)

score,feedback←V⁡\(ℱ⁡\(r\),xr,ℬ^​\(r\)\)\\text\{score\},\\;\\textit\{feedback\}\\leftarrow V\\\!\\left\(\\mathcal\{F\}\(r\),\\,x\_\{r\},\\,\\hat\{\\mathcal\{B\}\}\(r\)\\right\)

if

score≥θ\\text\{score\}\\geq\\thetathen

return

ℬ^​\(r\)\\hat\{\\mathcal\{B\}\}\(r\)
endif

endfor

return

ℬ^​\(r\)\\hat\{\\mathcal\{B\}\}\(r\)\{return last candidate\}

### 5\.3Implementation Details\.

All runs use fixed prompt templates that enforce a strict JSON schema and require every symptom in the provided list to appear exactly once, grouping symptoms into the same stage when the narrative does not support an internal order\. We use deterministic decoding \(no sampling\) withmax\_new\_tokens=512\.

Table 5\.1:General results stratified by vaccine type\. Metrics in %\.Bold= best,underline= second best within each model block\.†Proposed method\.ModelSettingJanssen \(n=1,164\)Moderna \(n=983\)Pfizer \(n=1,019\)TotalEM↑\\uparrowLCCS↑\\uparrow𝝉𝒃↑\\bm\{\\tau\_\{b\}\}\\uparrowEM↑\\uparrowLCCS↑\\uparrow𝝉𝒃↑\\bm\{\\tau\_\{b\}\}\\uparrowEM↑\\uparrowLCCS↑\\uparrow𝝉𝒃↑\\bm\{\\tau\_\{b\}\}\\uparrowEM↑\\uparrowLCCS↑\\uparrow𝝉𝒃↑\\bm\{\\tau\_\{b\}\}\\uparrowGPT\-4\.1PIVOT33\.6559\.5658\.8938\.5360\.7355\.4331\.8957\.4457\.0034\.6059\.2457\.20GUIDE33\.9961\.0260\.0736\.7060\.6856\.4831\.3057\.5757\.8533\.9759\.8058\.24CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}26\.4942\.9445\.7927\.9343\.5143\.6524\.5140\.0642\.3726\.3042\.1944\.02CRAFTG\{\}\_\{\\textup\{G\}\}29\.6855\.7155\.9233\.9456\.5252\.0829\.9254\.8653\.3131\.0855\.6953\.88CRAFT\-Full†34\.4361\.6859\.5840\.0663\.8456\.4532\.6858\.9356\.9735\.6161\.4657\.77Llama\-70BPIVOT24\.7650\.1851\.9830\.5852\.8149\.5326\.7750\.3352\.1327\.2251\.0451\.27GUIDE23\.7348\.5950\.8130\.2851\.4048\.5324\.7047\.9449\.1726\.0849\.2549\.58CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}18\.6441\.3454\.1023\.5544\.6851\.5920\.1841\.0851\.1220\.6642\.3052\.36CRAFTG\{\}\_\{\\textup\{G\}\}23\.8148\.5550\.6430\.4851\.4948\.7024\.9047\.9749\.2026\.2449\.2849\.57CRAFT\-Full†25\.8852\.0653\.4731\.9053\.8950\.6926\.7751\.4952\.9328\.0452\.4552\.43MedGemmaPIVOT17\.9539\.6743\.6325\.8943\.6542\.0819\.0039\.3741\.9220\.7540\.8142\.60GUIDE17\.5239\.5642\.3323\.9641\.3439\.2919\.3938\.5340\.3620\.1239\.7840\.75CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}13\.0333\.2144\.5215\.9036\.3840\.6114\.6735\.0042\.2014\.4444\.3642\.56CRAFTG\{\}\_\{\\textup\{G\}\}15\.5335\.9038\.5421\.9237\.8335\.6118\.1135\.7336\.6818\.3536\.4437\.03CRAFT\-Full†18\.2940\.2043\.8425\.9943\.9141\.5918\.8039\.4942\.0920\.8541\.1342\.57Claude\-4\.5PIVOT34\.5157\.3452\.7140\.3759\.4751\.0334\.8455\.5851\.6636\.4457\.4451\.85GUIDE34\.6954\.9850\.5338\.2356\.6148\.0334\.3553\.0948\.2135\.6854\.8849\.00CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}37\.1957\.7356\.0340\.8859\.8554\.2035\.6355\.2353\.0637\.8357\.5854\.51CRAFTG\{\}\_\{\\textup\{G\}\}33\.7461\.0457\.5840\.6763\.3953\.9934\.0659\.5754\.5035\.9961\.3055\.47CRAFT\-Full†35\.8961\.8256\.6841\.1864\.5154\.9834\.6558\.5452\.9237\.1461\.6054\.94

For generator–verifier configurations, we run up tomax\_iter=4 refinement iterations and accept outputs when the verifier score is at leastθ=3\\theta=3; the parameter\-selection procedure formax\_iterandθ\\thetais summarized in Appendix[A\.2](https://arxiv.org/html/2608.12779#S1.SS2)\. Full prompt templates for CRAFT\-Full are provided in Appendix[A\.3](https://arxiv.org/html/2608.12779#S1.SS3)\. More implementation details can be found at[https://github\.com/LEAF\-Lab\-Stevens/TemporalAnalysis](https://github.com/LEAF-Lab-Stevens/TemporalAnalysis)\.

Open\-weight experiments are run on a workstation equipped with two NVIDIA RTX A5000 GPUs \(24GB each\)\. Large models are loaded with device sharding across both GPUs and 4\-bit quantization\. API\-based experiments are executed using the official Python SDKs for the corresponding providers\.

### 5\.4Evaluation Metrics\.

We evaluate temporal sequences as ordered lists of buckets, where each bucket is a set of items and within\-bucket order is trivial\. Let the gold sequence beG=\(G1,…,Gm\)G=\(G\_\{1\},\\dots,G\_\{m\}\)and the predictionP=\(P1,…,Pn\)P=\(P\_\{1\},\\dots,P\_\{n\}\)\. After normalizationϕ⁡\(⋅\)\\phi\(\\cdot\)\(e\.g\., lower\-casing\), writeG~i=\{ϕ⁡\(x\)∣x∈Gi\}\\tilde\{G\}\_\{i\}=\\\{\\phi\(x\)\\mid x\\in G\_\{i\}\\\}andP~j=\{ϕ⁡\(x\)∣x∈Pj\}\\tilde\{P\}\_\{j\}=\\\{\\phi\(x\)\\mid x\\in P\_\{j\}\\\}; letrG​\(x\)r\_\{G\}\(x\)andrP​\(x\)r\_\{P\}\(x\)denote the bucket rank of itemxxin gold and prediction, andSSthe set of items appearing in both\. Within\-bucket duplicates are ignored; cross\-bucket duplicates are resolved by a check\-and\-fix moduleFormatTool\.

Strict Exact Match \(EM\)\[[24](https://arxiv.org/html/2608.12779#bib.bib15)\]\.

Pass/fail: the prediction must exactly reproduce the gold segmentation and inter\-bucket order \(within\-bucket order ignored\)\.

EM\(G,P\)=\{1,m=n​and​∀i∈\{1,…,m\}:G~i=P~i,0,otherwise\.\\mathrm\{EM\}\(G,P\)=\\begin\{cases\}1,&m=n\\ \\text\{and\}\\ \\forall\\,i\\in\\\{1,\.\.\.,m\\\}:\\tilde\{G\}\_\{i\}=\\tilde\{P\}\_\{i\},\\\\ 0,&\\text\{otherwise\.\}\\end\{cases\}
Kendall’sτb\\tau\_\{b\}\[[15](https://arxiv.org/html/2608.12779#bib.bib14)\]\.

For each unordered pair\{x,y\}⊂S\\\{x,y\\\}\\subset S, the pair is*concordant*ifsign⁡\(rG​\(y\)−rG​\(x\)\)=sign⁡\(rP​\(y\)−rP​\(x\)\)≠0\\operatorname\{sign\}\(r\_\{G\}\(y\)\{\-\}r\_\{G\}\(x\)\)=\\operatorname\{sign\}\(r\_\{P\}\(y\)\{\-\}r\_\{P\}\(x\)\)\\neq 0, and*discordant*if the signs are non\-zero and opposite\. LetNCN\_\{C\},NDN\_\{D\}be the concordant and discordant counts, andTGT\_\{G\},TPT\_\{P\}the pair counts tied in gold and prediction:

τb=NC−ND\(NC\+ND\+TG\)​\(NC\+ND\+TP\)∈\[−1,1\]\.\\tau\_\{b\}=\\frac\{N\_\{C\}\-N\_\{D\}\}\{\\sqrt\{\(N\_\{C\}\+N\_\{D\}\+T\_\{G\}\)\\,\(N\_\{C\}\+N\_\{D\}\+T\_\{P\}\)\}\}\\in\[\-1,1\]\.Ranges from−1\-1\(complete reversal\) to\+1\+1\(perfect agreement\); ties from within\-bucket equivalence are handled explicitly\.

Group\-Aware LCCS\[[12](https://arxiv.org/html/2608.12779#bib.bib1)\]\.

Treating each bucket as a token, LCCS is the longest common contiguous subsequence of\(G~1,…,G~m\)\(\\tilde\{G\}\_\{1\},\\dots,\\tilde\{G\}\_\{m\}\)and\(P~1,…,P~n\)\(\\tilde\{P\}\_\{1\},\\dots,\\tilde\{P\}\_\{n\}\):

L=max\{ℓ∣∃i,j:\(G~i,…,G~i\+ℓ−1\)=\(P~j,…,P~j\+ℓ−1\)\}\.L=\\max\\bigl\\\{\\,\\ell\\mid\\exists\\,i,j:\\,\(\\tilde\{G\}\_\{i\},\.\.\.,\\tilde\{G\}\_\{i\+\\ell\-1\}\)=\(\\tilde\{P\}\_\{j\},\.\.\.,\\tilde\{P\}\_\{j\+\\ell\-1\}\)\\bigr\\\}\.This metric complementsτb\\tau\_\{b\}by rewardingunbrokenspans of perfectly matched phases, penalising isolated segmentation errors\.

## 6Results\.

This section reports our empirical findings, organized around the three research questions\. We first compare CRAFT\-Full against the baselines, then examine how the four backbones compare to one another, and then test whether these patterns hold when performance is broken out by vaccine type\. Two further subsections follow: an ablation that isolates the contributions of the verifier and the generator, and case studies that illustrate the behavior behind the aggregate numbers\.

### 6\.1RQ1: CRAFT\-Full vs\. Baselines\.

Table[5\.1](https://arxiv.org/html/2608.12779#S5.T1)reports final performance across all four

Table 6\.1:Results at each iteration per model and setting\.M​@​tM@t: metricM∈\{EM,LCCS,τb\}M\\in\\\{\\text\{EM\},\\text\{LCCS\},\\tau\_\{b\}\\\}when the generate–verify loop is capped atttiterations\. AvgIters: mean iterations executed undert=4t\{=\}4\. Metrics in %\.Underline: best per setting across iterations\.Bold underline: best per model per metric\.†Proposed method\.LLMSettingAvgItersEM@1EM@2EM@3EM@4LCCS@1LCCS@2LCCS@3LCCS@4𝝉𝒃​@​1\\bm\{\\tau\_\{b\}\}@1𝝉𝒃​@​2\\bm\{\\tau\_\{b\}\}@2𝝉𝒃​@​3\\bm\{\\tau\_\{b\}\}@3𝝉𝒃​@​4\\bm\{\\tau\_\{b\}\}@4GPT\-4\.1PIVOT1\.273429\.7234\.7334\.7334\.6052\.1959\.4459\.3159\.2450\.7457\.2957\.2857\.20GUIDE1\.234834\.9234\.2234\.0633\.9760\.9660\.1859\.9759\.8058\.9158\.4458\.3658\.24CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}1\.000026\.30———42\.19———44\.02———CRAFTG\{\}\_\{\\textup\{G\}\}2\.940734\.8932\.7931\.7831\.0860\.9058\.1656\.6155\.6958\.3355\.8754\.4053\.88CRAFT\-Full†2\.986426\.9034\.8935\.6135\.6147\.1060\.9461\.4561\.4645\.7158\.1958\.1657\.77Llama\-70BPIVOT1\.202226\.9027\.2227\.2227\.2250\.1351\.0051\.0451\.0450\.0751\.2151\.2751\.27GUIDE1\.214826\.0826\.0826\.0826\.0849\.2349\.2549\.2549\.2849\.5449\.5949\.5849\.58CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}1\.000020\.66———42\.30———52\.36———CRAFTG\{\}\_\{\\textup\{G\}\}1\.264326\.0826\.2726\.2426\.2449\.2349\.3049\.2649\.2849\.5449\.5249\.5349\.57CRAFT\-Full†1\.213627\.4727\.9528\.0128\.0450\.9252\.3252\.4252\.4550\.2852\.3752\.4152\.43MedGemmaPIVOT1\.149620\.6620\.7520\.7520\.7540\.5640\.8240\.8340\.8142\.0842\.6042\.6342\.60GUIDE1\.117920\.1520\.1220\.1220\.1239\.8039\.7839\.7839\.7840\.8640\.7640\.7540\.75CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}1\.000014\.44———44\.36———42\.56———CRAFTG\{\}\_\{\\textup\{G\}\}1\.554220\.1518\.4718\.3518\.3539\.8036\.4936\.4436\.4440\.8637\.1337\.0337\.03CRAFT\-Full†1\.503220\.6620\.4720\.8520\.8540\.5640\.6441\.1341\.1342\.0842\.3242\.5742\.57Claude\-4\.5PIVOT1\.986137\.5835\.3336\.6636\.4462\.6556\.9257\.8157\.4456\.7352\.6152\.5551\.85GUIDE1\.994340\.3735\.1736\.4735\.6865\.3755\.5655\.7554\.8858\.5751\.4249\.9749\.00CRAFTw/o V\{\}\_\{\\textup\{w/o\\,V\}\}1\.000037\.83———57\.58———54\.51———CRAFTG\{\}\_\{\\textup\{G\}\}3\.635040\.0835\.9037\.4835\.9965\.3561\.6362\.5161\.3058\.4055\.5956\.0755\.47CRAFT\-Full†3\.462336\.5736\.5737\.4237\.1461\.3261\.4961\.6561\.6055\.7454\.8355\.0554\.94

models and five settings\. Against PIVOT, CRAFT\-Full gains\+\+1\.0 EM points for GPT\-4\.1 \(35\.61 vs\. 34\.60\),\+\+0\.8 for Llama\-3\.3\-70B \(28\.04 vs\. 27\.22\),\+\+0\.7 for Claude Sonnet 4\.5 \(37\.14 vs\. 36\.44\), and\+\+0\.1 for MedGemma\-27B \(20\.85 vs\. 20\.75\)\. Margins over GUIDE are uniformly larger \(GPT\-4\.1:\+\+1\.6; Llama:\+\+2\.0; Claude:\+\+1\.5; MedGemma:\+\+0\.7\)\. CRAFT\-Full achieves the highest EM in every model block compared to baselines, confirming it as the strongest configuration\. Although baselines such as PIVOT and GUIDE occasionally match or exceed CRAFT\-Full onτb\\tau\_\{b\}or LCCS, these metrics credit partial ordering agreement and can score highly even when the full trajectory structure is incorrect\. EM, which requires the entire stage\-wise grouping and ordering to match gold, is the most demanding metric for this task and the one on which CRAFT\-Full consistently leads\.

Table[6\.1](https://arxiv.org/html/2608.12779#S6.T1)explains how CRAFT\-Full wins\. CRAFT\-Full actively uses its refinement budget: AvgIters reaches 2\.99 for GPT\-4\.1 and 3\.46 for Claude, and performance builds across iterations\. For GPT\-4\.1, EM rises from 26\.90 ati=1i\{=\}1to 35\.61 ati=4i\{=\}4, a gain of 8\.7 points across rounds\. In contrast, PIVOT and GUIDE converge after a single pass in most instances \(AvgIters≈\\approx1\.2–2\.0\) because the anchor\-based verifier accepts outputs quickly without substantive ordering improvement\. For GPT\-4\.1, PIVOT peaks at EM@2==34\.73 and GUIDE degrades from its peak of 34\.92 ati=1i\{=\}1\. This confirms that CRAFT\-Full’s additive rubric provides richer, more actionable feedback that sustains improvement across the full refinement budget, whereas the anchor\-based verifier’s narrower signal cannot drive continued gains beyond the first pass\.

![Refer to caption](https://arxiv.org/html/2608.12779v1/gold_symptom_heatmap_Overall.png)\(a\)Ground Truth
![Refer to caption](https://arxiv.org/html/2608.12779v1/claude_craft_symptom_heatmap_Overall.png)\(b\)Claude, CRAFT\-Full

Figure 6\.1:Before–after symptom\-transition frequencies of each later symptom \(columns\) given the first symptom of the report \(rows; restricted to the most frequent first symptoms in the dataset\): \(a\) ground truth; \(b\) CRAFT\-Full on ClaudeFigure[6\.1](https://arxiv.org/html/2608.12779#S6.F1)shows that the gold and CRAFT\-Full transition matrices share a similarly diffuse distribution across symptom pairs and differ only in small cell values, confirming that CRAFT\-Full preserves progression patterns at the population scale\. Since this aggregate view does not reveal where refinement changes predictions, we turn to an instance level case study in Section[6\.5](https://arxiv.org/html/2608.12779#S6.SS5)\.

### 6\.2RQ2: Overall Model Performance\.

The capability ordering is stable across all configurations and metrics: Claude Sonnet 4\.5\>\>GPT\-4\.1\>\>Llama\-3\.3\-70B\>\>MedGemma\-27B\. Under CRAFT\-Full, Claude achieves the highest total EM \(37\.14\), followed by GPT\-4\.1 \(35\.61\), Llama \(28\.04\), and MedGemma \(20\.85\)\. This ordering holds without exception across all three metrics and all vaccine strata, indicating that MedTempo reliably differentiates model tiers\.

Beyond the stable ranking, models differ notably in how they respond to iterative refinement\. GPT\-4\.1 benefits most from additional iterations: under CRAFT\-Full, EM climbs from 26\.90 ati=1i\{=\}1to 35\.61 ati=4i\{=\}4\(\+\+8\.7 points\), with LCCS andτb\\tau\_\{b\}following similar upward trends \(47\.10→\\to61\.46 and 45\.71→\\to57\.77 respectively\)\. In contrast, Claude starts strong \(EM 36\.57 ati=1i\{=\}1\) but gains only\+\+0\.6 points across iterations, suggesting its first\-pass outputs already capture most of the temporal structure\. Llama and MedGemma show similarly flat iteration curves \(EM gains of\+\+0\.6 and\+\+0\.2 respectively\), but for a different reason: both converge quickly \(AvgIters 1\.2–1\.5\), indicating that the verifier accepts their outputs early rather than driving further improvement\. This divergence between models that saturate from high initial quality \(Claude\) and those that stall from limited capacity to act on feedback \(Llama, MedGemma\) highlights that iteration utility is tied to model capability\.

Figure[6\.2](https://arxiv.org/html/2608.12779#S6.F2)shows that distributional fidelity to gold tracks the capability ordering: Claude overlaps gold across all settings, while Llama and MedGemma fall short of gold at stage counts of four and above regardless of verifier choice\. GPT\-4\.1 is the one tier where verifier choice visibly reshapes the distribution, with CRAFT w/o V peaking at stage 2 and CRAFT\-Full recovering a spread close to gold\.

### 6\.3RQ3: Vaccine\-Stratified Performance\.

Performance is consistently stratified by vaccine type across all models and configurations: Moderna narratives yield the highest EM in every case, followed by Janssen, with Pfizer\-BioNTech systematically lowest\. Under CRAFT\-Full, the Moderna–Pfizer gap is 7\.4 points for GPT\-4\.1 \(40\.06 vs\. 32\.68\), 6\.5 for Claude Sonnet 4\.5 \(41\.18 vs\. 34\.65\), 5\.1 for Llama\-3\.3\-70B \(31\.90 vs\. 26\.77\), and 7\.2 for MedGemma\-27B \(25\.99 vs\. 18\.80\)\. This ordering is preserved across all settings and all four models without exception, indicating a systematic corpus\-level source of difficulty rather than a model\-specific effect\.

The vaccine gap is more pronounced on EM than on LCCS andτb\\tau\_\{b\}, suggesting that models make segmentation errors on Pfizer narratives specifically rather than systematically misranking symptoms within groups\. Since EM requires an exact match on both bucket composition and inter\-bucket order while LCCS rewards contiguous correct spans andτb\\tau\_\{b\}captures global pairwise ordering, the metric divergence points to Pfizer narratives being harder to segment into correct temporal groups rather than harder to order within those groups\. The capability ordering Claude\>\>GPT\-4\.1\>\>Llama\>\>MedGemma holds for every vaccine type under every configuration\.

Figure 6\.2:Predicted stage\-count distributions \(frequency %\) under all five settings vs\. gold, per model\. All models under\-segment relative to gold; the bias is more pronounced for weaker models \(MedGemma\-27B, Llama\-3\.3\-70B\) than for stronger ones \(GPT\-4\.1, Claude Sonnet 4\.5\)\.
### 6\.4Ablation Study\.

Using Table[6\.1](https://arxiv.org/html/2608.12779#S6.T1), we analyze two ablation groups: the verifier ablation \(CRAFT\-Full vs\. CRAFT w/o V\), which holds the generator fixed, and the generator ablation \(CRAFT\-G vs\. CRAFT\-Full\), which holds the verifier fixed\.

##### Verifier contribution\.

Removing the verification loop causes substantial performance drops for three of four models\. CRAFT\-Full gains\+\+9\.3 EM over CRAFT w/o V for GPT\-4\.1 \(35\.61 vs\. 26\.30\),\+\+7\.4 for Llama\-3\.3\-70B \(28\.04 vs\. 20\.66\), and\+\+6\.4 for MedGemma\-27B \(20\.85 vs\. 14\.44\)\. For Llama and MedGemma, virtually all gain is captured ati=1i\{=\}1: the verifier’s

Table 6\.2:Iteration trace for Example 1 \(GPT\-4\.1, CRAFT\-Full\)\.✓\\checkmark= matches gold;×\\times= grouping error\.IterScorePredicted StagesGold—\{Insomnia\}→\\to\{Muscle disorder\}→\\to\{Herpes zoster\}→\\to\{Memory impairment\}12/5\{Insomnia\}→\\to\{Muscle disorder\}→\\to\{Herpes zoster, Memory impairment\}×\\times23/5\{Insomnia\}→\\to\{Muscle disorder\}→\\to\{Herpes zoster\}→\\to\{Memory impairment\}✓\\checkmarkfirst\-pass feedback corrects schema violations in unverified outputs, after which the output is accepted without substantive ordering improvement\. For GPT\-4\.1, gains accumulate steadily throughi=3i\{=\}3\(AvgIters = 2\.99\), confirming that the verification loop provides genuine multi\-iteration value for capable models\. For Claude Sonnet 4\.5, CRAFT\-Full \(37\.14\) falls slightly below CRAFT w/o V \(37\.83\): the fixed thresholdθ=3\\theta\{=\}3does not recognize Claude’s near\-correct first\-pass output as satisfactory, forcing continued revision that introduces errors rather than correcting them\. This highlights the importance of threshold calibration for further improvement\.

##### Generator contribution\.

Replacing the full\-regeneration generator with the edit\-conditioned variant \(CRAFT\-G\) consistently degrades final performance\. For GPT\-4\.1, CRAFT\-G achieves a strong EM of 34\.89 ati=1i\{=\}1but degrades monotonically to 31\.08 byi=4i\{=\}4\(−\-3\.8 points\), while CRAFT\-Full builds to 35\.61\. The edit\-conditioned prompt is effective for initialisation but too constrained to sustain quality under continued verifier feedback across the full budget\. For MedGemma\-27B, CRAFT\-G is actively harmful: EM drops from 20\.15 ati=1i\{=\}1to 18\.35 byi=2i\{=\}2, indicating the model misinterprets edit\-only instructions and degrades its own output under revision\. CRAFT\-Full avoids this across all tiers by regenerating from full task context at every iteration\.

### 6\.5Case Study\.

The ablation analysis above quantifies verifier and generator contributions across models and iterations at the aggregate level, but does not reveal how the feedback loop operates on individual instances\. To provide insight into the refinement mechanism, we present a representative example showing how verifier feedback guides the generator toward the correct trajectory across iterations\. An additional case study is provided in the Appendix[A\.5](https://arxiv.org/html/2608.12779#S1.SS5)\.

Example \(GPT\-4\.1, CRAFT\-Full\):

Symptom list:Insomnia, Muscle disorder, Herpes zoster, Memory impairment

Narrative \(abridged\):“On 20Feb2021, the patient experienced could not sleep\. In Feb2021, the patient experienced effected upper part of body, arms, neck, shoulders, severe muscle condition, had tiny bit of shingles on my side, did not have memory of anything that happened / was out of my mind \[…\] They took her to the hospital and then the patient underwent MRI to see if her brain was okay\.”

Iteration 1Score: 2/5 — belowθ\\theta, continue

Verifier feedback:“Symptoms are grouped in temporal order and all are present, but ‘Herpes zoster’ and ‘Memory impairment’ should be in separate groups as they are not clearly described as occurring at the same time \[…\] For full marks, ensure symptoms grouped together are clearly simultaneous per the text\.”

Iteration 2Score: 3/5 —s≥θs\\geq\\theta, early stop ✓

Verifier feedback:—

Ati=1i\{=\}1, the model correctly orders all four symptom groups but over\-merges Herpes zoster and Memory impairment into a single stage, receiving a score of 2\. The score of 2 rather than 3 reflects that the over\-merge constitutes two rubric violations simultaneously: grouping symptoms without clear simultaneous support, and failing to maintain strict earliest\-to\-latest ordering within the merged group\. CRAFT\-Full’s verifier precisely identifies the grouping error and provides a targeted one\-sentence correction, demonstrating that its additive rubric produces informative, actionable feedback even when the overall ordering direction is already correct\. Ati=2i\{=\}2the model splits the two symptoms into separate stages, exactly matching the gold standard and receiving a score of 3, which meets the acceptance thresholdθ=3\\theta\{=\}3and triggers early stopping\. This example illustrates that a single round of targeted feedback is sufficient for a capable model to correct a local grouping error within the refinement budget\. Table[6\.2](https://arxiv.org/html/2608.12779#S6.T2)summarizes the predicted stages across iterations\.

## 7Conclusion\.

We presented CRAFT, a generator–verifier LLM framework for iterative temporal reasoning over clinical narratives, and MedTempo, an expert\-annotated benchmark of 5,347 adverse\-event narratives, to evaluate structured symptom trajectory construction under sparse temporal anchoring\. Experiments across four LLM backbones show that CRAFT consistently improves temporal ordering accuracy\. Ablation analysis confirms that both generator and verifier components contribute meaningfully, with CRAFT\-Full emerging as the strongest configuration across all model tiers\. Our results also reveal that refinement behavior varies by model capability, suggesting that adaptive verification strategies may further improve performance\. Future work will leverage the 2,181 non\-temporal reports from MedTempo\-NT as supervision for a learned temporal\-evidence identification module, extending CRAFT into a unified end\-to\-end framework for automatic progression detection and stage\-wise temporal ordering\.

## Acknowledgments\.

This work was supported in part by the US National Science Foundation grant IIS\-2245907, IIS\-2047843, IIS\-2437621, and an Amazon Research Award, Fall 2024\.

## References

- \[1\]G\. Alfattni, N\. Peek, and G\. Nenadic\(2020\)Extraction of temporal relations from clinical free text: a systematic review of current approaches\.Journal of Biomedical Informatics108,pp\. 103488\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2020.103488),[Link](https://www.sciencedirect.com/science/article/pii/S1532046420301167)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p1.1),[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[2\]S\. Alsayyahi and R\. Batista\-Navarro\(2023\)TIMELINE: exhaustive annotation of temporal relations supporting the automatic ordering of events in news articles\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 16336–16348\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.1016),[Link](https://aclanthology.org/2023.emnlp-main.1016/)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p1.1)\.
- \[3\]J\. J\. Andrew, J\. Potier, N\. Garcelon, A\. Burgun, and M\. Vincent\(2025\)Using large language models for temporal relation extraction from pediatric clinical reports\.JAMIA Open8\(6\),pp\. ooaf121\.External Links:[Document](https://dx.doi.org/10.1093/jamiaopen/ooaf121),[Link](https://academic.oup.com/jamiaopen/article/8/6/ooaf121/8340484)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[4\]J\. J\. Andrew, M\. Vincent, A\. Burgun, and N\. Garcelon\(2024\)Evaluating LLMs for temporal entity extraction from pediatric clinical text in rare diseases context\.InProceedings of the First Workshop on Patient\-Oriented Language Processing \(CL4Health\) @ LREC\-COLING 2024,D\. Demner\-Fushman, S\. Ananiadou, P\. Thompson, and B\. Ondov \(Eds\.\),Torino, Italia,pp\. 145–152\.External Links:[Link](https://aclanthology.org/2024.cl4health-1.18/)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[5\]Anthropic\(2025\)Claude sonnet 4\.5 system card\.Note:System CardExternal Links:[Link](https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf)Cited by:[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p1.1)\.
- \[6\]S\. Bethard, G\. Savova, M\. Palmer, and J\. Pustejovsky\(2017\)SemEval\-2017 Task 12: clinical TempEval\.InProceedings of the 11th International Workshop on Semantic Evaluation \(SemEval\-2017\),Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[7\]Centers for Disease Control and Prevention \(CDC\), Food and Drug Administration \(FDA\), agencies of the U\.S\. Department of Health and Human Services \(HHS\)\(1990\)Vaccine adverse event reporting system\.Note:[https://vaers\.hhs\.gov/data\.html](https://vaers.hhs.gov/data.html)Cited by:[§3\.1](https://arxiv.org/html/2608.12779#S3.SS1.p1.1)\.
- \[8\]J\. Chen, H\. Ouyang, J\. Ren, W\. Ding, W\. Hu, and Y\. Qu\(2024\)Timeline\-based sentence decomposition with in\-context learning for temporal fact extraction\.Note:Accepted to ACL 2024 \(main conference\)External Links:2405\.10288,[Document](https://dx.doi.org/10.48550/arXiv.2405.10288),[Link](https://arxiv.org/abs/2405.10288)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p1.1)\.
- \[9\]F\. Cheng and Y\. Miyao\(2018\)Inducing temporal relations from time anchor annotation\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),New Orleans, Louisiana,pp\. 1833–1843\.External Links:[Link](https://aclanthology.org/N18-1166/),[Document](https://dx.doi.org/10.18653/v1/N18-1166)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p1.1),[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p2.1)\.
- \[10\]H\. Cui, A\. Unell, B\. Chen, J\. A\. Fries, E\. Alsentzer, S\. Koyejo, and N\. H\. Shah\(2025\)TIMER: temporal instruction modeling and evaluation for longitudinal clinical records\.npj Digital Medicine8\(1\),pp\. 577\.External Links:[Document](https://dx.doi.org/10.1038/s41746-025-01965-9),[Link](https://pmc.ncbi.nlm.nih.gov/articles/PMC12475073/)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p1.1),[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[11\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),Cited by:[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p1.1)\.
- \[12\]D\. Gusfield\(1997\)Algorithms on strings, trees, and sequences: computer science and computational biology\.Cambridge University Press\.Cited by:[§5\.4](https://arxiv.org/html/2608.12779#S5.SS4.p7.1.1)\.
- \[13\]J\. He, L\. Rasmy, H\. Li, J\. Li, Z\. Sun, E\. Yu, D\. Zhi, and C\. Tao\(2024\)Prompting large language models for clinical temporal relation extraction\.External Links:2412\.04512,[Document](https://dx.doi.org/10.48550/arXiv.2412.04512),[Link](https://arxiv.org/abs/2412.04512)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[14\]D\. Hein, A\. Christie, M\. Holcomb, B\. Xie, A\. Jain, J\. Vento, N\. Rakheja, A\. H\. Shakur, S\. Christley, L\. G\. Cowell,et al\.\(2025\)Iterative refinement and goal articulation to optimize large language models for clinical information extraction\.NPJ Digital Medicine8\(1\),pp\. 301\.Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p3.1)\.
- \[15\]M\. G\. Kendall\(1945\)The treatment of ties in ranking problems\.Biometrika33\(3\),pp\. 239–251\.Cited by:[§5\.4](https://arxiv.org/html/2608.12779#S5.SS4.p5.1.1)\.
- \[16\]A\. Leeuwenberg and M\. Moens\(2018\)Temporal information extraction by predicting relative time\-lines\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 1237–1246\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1155),[Link](https://aclanthology.org/D18-1155/)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p1.1)\.
- \[17\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Welleck, B\. P\. Majumder, S\. Gupta, A\. Yazdanbakhsh, and P\. Clark\(2023\)Self\-refine: iterative refinement with self\-feedback\.External Links:2303\.17651Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p3.1),[§2](https://arxiv.org/html/2608.12779#S2.p3.1)\.
- \[18\]Meta AI\(2024\)Llama\-3\.3\-70B\-Instruct\.Note:Model cardReleased December 6, 2024External Links:[Link](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)Cited by:[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p1.1)\.
- \[19\]T\. Miller, S\. Bethard, D\. Dligach, and G\. Savova\(2023\)End\-to\-end clinical temporal information extraction with multi\-head attention\.InProceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks,Toronto, Canada,pp\. 50–61\.External Links:[Link](https://aclanthology.org/2023.bionlp-1.4/),[Document](https://dx.doi.org/10.18653/v1/2023.bionlp-1.4)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[20\]A\. L\. Olex and B\. T\. McInnes\(2021\)Review of temporal reasoning in the clinical domain for timeline extraction: where we are and where we need to be\.Journal of Biomedical Informatics118,pp\. 103784\.External Links:[Document](https://dx.doi.org/10.1016/j.jbi.2021.103784),[Link](https://www.sciencedirect.com/science/article/pii/S1532046421001131)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p1.1),[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[21\]OpenAI\(2024\)GPT\-4o system card\.External Links:[Link](https://openai.com/index/gpt-4o-system-card/)Cited by:[§3\.3](https://arxiv.org/html/2608.12779#S3.SS3.p1.1)\.
- \[22\]OpenAI\(2025\)GPT\-4\.1\.Note:Model releaseExternal Links:[Link](https://openai.com/index/gpt-4-1/)Cited by:[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p1.1)\.
- \[23\]J\. Pustejovsky, J\. Castaño, R\. Ingria, R\. Saurí, R\. Gaizauskas, A\. Setzer, G\. Katz, and D\. Radev\(2003\)TimeML: robust specification of event and temporal expressions in text\.InFifth International Workshop on Computational Semantics \(IWCS\-5\),Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p1.1)\.
- \[24\]P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang\(2016\)Squad: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2383–2392\.Cited by:[§5\.4](https://arxiv.org/html/2608.12779#S5.SS4.p2.1.1)\.
- \[25\]G\. Savova, S\. Bethard, W\. Styler, J\. Martin, M\. Palmer, J\. Masanz, and W\. Ward\(2009\)Towards temporal relation discovery from the clinical narrative\.InAMIA annual symposium proceedings,Vol\.2009,pp\. 568\.Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1)\.
- \[26\]A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly,et al\.\(2025\)MedGemma technical report\.External Links:2507\.05201,[Link](https://arxiv.org/abs/2507.05201)Cited by:[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p1.1)\.
- \[27\]W\. F\. Styler IV, S\. Bethard, S\. Finan, M\. Palmer, S\. Pradhan, P\. C\. de Groen, B\. Erickson, T\. Miller, C\. Lin, G\. Savova, and J\. Pustejovsky\(2014\)Temporal annotation in the clinical domain\.Transactions of the Association for Computational Linguistics2,pp\. 143–154\.Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[28\]X\. Su, P\. Howard, and S\. Bethard\(2025\)Transformer\-based temporal information extraction and application: a review\.External Links:2504\.07470,[Document](https://dx.doi.org/10.48550/arXiv.2504.07470),[Link](https://arxiv.org/abs/2504.07470)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p1.1)\.
- \[29\]W\. Sun, A\. Rumshisky, and Ö\. Uzuner\(2013\)Evaluating temporal relations in clinical text: 2012 i2b2 challenge\.Journal of the American Medical Informatics Association20\(5\),pp\. 806–813\.Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[30\]J\. Tourille\(2018\)Extracting clinical event timelines: temporal information extraction and temporal relation inference\.Ph\.D\. Thesis,Université Paris\-Saclay\.External Links:[Link](https://theses.hal.science/tel-01997223)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[31\]M\. Verhagen, R\. Gaizauskas, F\. Schilder, M\. Hepple, and J\. Pustejovsky\(2007\)SemEval\-2007 Task 15: TempEval temporal relation identification\.InProceedings of the Fourth International Workshop on Semantic Evaluations \(SemEval\-2007\),Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p1.1)\.
- \[32\]E\. Wang and A\. Weiss\(2025\)Extracting relative timelines from medical case reports using large language models\.AMIA Joint Summits on Translational Science proceedings\. AMIA Joint Summits on Translational Science,pp\. 598–606\.External Links:[Link](https://pmc.ncbi.nlm.nih.gov/articles/PMC12150726/)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[33\]L\. Wang, P\. Li, and S\. Xu\(2022\)DCT\-centered temporal relation extraction\.InProceedings of the 29th International Conference on Computational Linguistics,Gyeongju, Republic of Korea,pp\. 2087–2097\.External Links:[Link](https://aclanthology.org/2022.coling-1.182/)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p1.1),[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p2.1)\.
- \[34\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. Rush\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Link](https://aclanthology.org/2020.emnlp-demos.6/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[§5\.2](https://arxiv.org/html/2608.12779#S5.SS2.p1.1)\.
- \[35\]D\. Yu, R\. W\. Stidham, and V\. G\. V\. Vydiswaran\(2023\)A systematic temporal extraction pipeline for medical concepts in clinical notes\.InAMIA Annual Symposium Proceedings,pp\. 1314–1323\.Note:AMIA 2023; published in*AMIA Annu Symp Proc*\.External Links:[Link](https://pmc.ncbi.nlm.nih.gov/articles/PMC10785919/)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p2.1),[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[36\]C\. Yuan, Q\. Xie, and S\. Ananiadou\(2023\)Zero\-shot temporal relation extraction with chatgpt\.External Links:2304\.05454,[Document](https://dx.doi.org/10.48550/arXiv.2304.05454),[Link](https://arxiv.org/abs/2304.05454)Cited by:[§2](https://arxiv.org/html/2608.12779#S2.p2.1)\.
- \[37\]L\. Zhou, S\. Parsons, and G\. Hripcsak\(2008\)The evaluation of a temporal reasoning system in processing clinical discharge summaries\.Journal of the American Medical Informatics Association15\(1\),pp\. 99–106\.External Links:[Document](https://dx.doi.org/10.1197/jamia.M2467),[Link](https://pmc.ncbi.nlm.nih.gov/articles/PMC2274869/)Cited by:[§1](https://arxiv.org/html/2608.12779#S1.p1.1)\.

## AAppendix\.

### A\.1Data Sampling\.

Table[A\.1](https://arxiv.org/html/2608.12779#S1.T1)presents representative examples of reports removed under each criterion\.

### A\.2Hyperparameter Selection\.

Hyperparameter selection forTmaxT\_\{\\max\}andθ\\thetawas conducted using GPT\-4\.1 on a development sample of 100 instances drawn from MedTempo\-T\. Due to computational budget constraints, as each sweep configuration requires multiple LLM calls per instance across both generator and verifier, we limited the search to a single frontier model rather than conducting independent sweeps for all four backbones, and to a representative subset rather than the full evaluation set\. The selected values \(Tmax=4T\_\{\\max\}\{=\}4,θ=3\\theta\{=\}3\) are applied uniformly across all models and configurations in the main experiments\. While model\-specific tuning could yield marginal gains for individual backbones, uniform hyperparameters ensure a controlled comparison and reflect a realistic deployment scenario where per\-model calibration may not be feasible\. Tables[A\.2](https://arxiv.org/html/2608.12779#S1.T2)and[A\.3](https://arxiv.org/html/2608.12779#S1.T3)report the development set EM across the swept ranges\.

Table A\.1:Examples of records removed from the dataset\. This table presents representative samples of clinical reports excluded from the final dataset due to criteria such as brevity, lack of temporal information, or limited symptom mentions\.CategoryStandard SymptomsClinical TextReports with 3 or less symptoms\[Pharyngeal swelling\]patient called back the next day and stated her throat was swelling and had to take Benadryl\.\[Dysphagia, Epiglottitis\]Right side of epiglottis swelled up and hinder swallowing pictures taken Benadryl Tylenol taken\[Dizziness, Fatigue, Mobility decreased\]extreme fatigue, dizziness,\. could not lift my left arm for 72 hoursReports with less than 10 words\[Erythema, Pruritus, Rash, Swelling\]redness, bumps, itchiness, and local swelling\[Asthenia, Chills, Headache, Myalgia\]Headache, chills, muscle aches and weakness\[Chills, Dizziness, Injection site pain, Myalgia, Pyrexia\]Dizziness, chills, fever, muscle aches, pain at the injection siteReports with no temporal cues\[Headache, Nausea, Pain, Pyrexia, Urticaria, Vomiting\]Fever, headache, body aches, nausea and vomiting\. Hives\[Dizziness, Headache, Hypoaesthesia, Injection site pain\]Headache, sore arm to injection site, dizzy, numbness to right foot\. No medications taken\. Continues to have symptoms\.\[Arthralgia, Chills, Fatigue, Headache, Myalgia, Nausea, Pyrexia, Vomiting\]Repeated shaking with chills, headache, nausea and vomiting, muscle/joint aches, fatigue, fever\.

![Refer to caption](https://arxiv.org/html/2608.12779v1/gold_symptom_heatmap_Pfizer.png)Pfizer\-BioNTech
![Refer to caption](https://arxiv.org/html/2608.12779v1/gold_symptom_heatmap_Moderna.png)Moderna
![Refer to caption](https://arxiv.org/html/2608.12779v1/gold_symptom_heatmap_Janssen.png)Janssen
![Refer to caption](https://arxiv.org/html/2608.12779v1/gold_symptom_heatmap_Overall.png)Overall
\(a\) Ground Truth

![Refer to caption](https://arxiv.org/html/2608.12779v1/claude_craft_symptom_heatmap_Pfizer.png)Pfizer\-BioNTech
![Refer to caption](https://arxiv.org/html/2608.12779v1/claude_craft_symptom_heatmap_Moderna.png)Moderna
![Refer to caption](https://arxiv.org/html/2608.12779v1/claude_craft_symptom_heatmap_Janssen.png)Janssen
![Refer to caption](https://arxiv.org/html/2608.12779v1/claude_craft_symptom_heatmap_Overall.png)Overall
\(b\) Best Model \(Claude Sonnet 4\.5, CRAFT\-Full\)

Figure A\.1:Before–after symptom relationship heatmaps across vaccine types; rows denote the most frequent earlier symptoms and columns denote the most frequent subsequent symptoms\. Row \(a\) ground truth; row \(b\)CRAFT\-Full on ClaudeTable A\.2:Development EM \(%\) across maximum iteration budgetTmax∈\{2,…,10\}T\_\{\\max\}\\in\\\{2,\\dots,10\\\}for CRAFT\.TmaxT\_\{\\max\}is the upper cap on generator–verifier loop iterations per instance with early stopping enabled\. GPT\-4\.1, sample size 100\.2345678910CRAFT12\.520\.022\.520\.011\.019\.518\.520\.019\.0
Note\.Columns denoteTmaxT\_\{\\max\}\. Bold indicates the best EM\.Tmax=4T\_\{\\max\}\{=\}4is adopted in all main experiments\.

Table A\.3:Development EM \(%\) across verifier acceptance thresholdθ∈\{1,2,3,4\}\\theta\\in\\\{1,2,3,4\\\}for CRAFT\.θ\\thetais the minimum verifier score \(0–5 scale\) required to accept a candidate output\. GPT\-4\.1, sample size 100\.𝜽=𝟏\\bm\{\\theta\{=\}1\}𝜽=𝟐\\bm\{\\theta\{=\}2\}𝜽=𝟑\\bm\{\\theta\{=\}3\}𝜽=𝟒\\bm\{\\theta\{=\}4\}CRAFT41394337
Note\.Bold indicates the best EM\.θ=3\\theta\{=\}3is adopted in all main experiments\.

### A\.3CRAFT Prompt Templates\.

The generator prompt \(Box[A\.3](https://arxiv.org/html/2608.12779#S1.SS3)\) is used across all CRAFT iterations\. On the first pass, the placeholder fields$prev\_result\_blockand$feedback\_blockare empty; on subsequent passes, they carry the prior output and verifier feedback, respectively\. The verifier prompt \(Box[A\.3](https://arxiv.org/html/2608.12779#S1.SS3)\) scores each candidate on a 0–5 rubric; scores below 3 trigger re\-generation with actionable feedback\.

BoxA\.3: Generator PromptIgnore previous conversations\.TASK:Temporally order the provided list of adverse events based on their sequence of appearance in the clinical notes and extract the most specific mention for each adverse event from the notes\.Clinical Notes:\{symptom\_text\}Adverse Events to Temporally Order:\{symptom\_list\}PROCESSING INSTRUCTIONS:•Start with the provided adverse events list and reorder it based on the sequence implied in the clinical notes\.•If several adverse effects are mentioned together without a clear temporal order, group them into the same block \(inside a single dictionary\)\.•For each adverse event, extract the closest matching phrase from the clinical notes; if an adverse event is not mentioned, assign"none"as its value\.IMPORTANT RULES:•No Invention:Never add, remove, or modify symptom names from the provided list\.•Mention Each Symptom Once:Mention each symptom only once—at the first time it appears or becomes relevant\.•Specific vs\. Generic Terms:When a phrase matches both specific and generic symptoms, assign it to the specific term only\. If “Lip swelling” or “Pharyngeal swelling” is matched, do NOT include the generic “Swelling” unless there is a separate, explicit mention of general swelling elsewhere\.•Temporal Evidence Required:If multiple symptoms are mentioned across multiple sentences without clear timeline separation, group them together\. Do NOT treat different sentences as different times unless there is an explicit temporal indicator \(e\.g\., “then,” “after that,” “later,” specific dates or times\)\.•Unmentioned Symptoms:Group all symptoms with no original mention together in a separate block with"none"as their value\. This block should be placed after all the temporally ordered groups\.OUTPUT FORMAT:Return your answer only in valid JSON format—using a list of dictionaries to represent temporal progression and grouping:``` {[ {"Erythema": ["redness in neck"]}, {"Pain in extremity": ["sore arm"], "Pruritus": ["itchy feeling"]}, {"Swelling": ["mild arm swelling"]} ]} ``` When revising, consider both the feedback \(if any\) and the previous attempt result \(if provided\)\. Keep correct parts from prior attempts, but fix issues based on feedback\.\{prev\_result\_block\} \{feedback\_block\}

BoxA\.3: Verifier PromptGiven the original text and extracted symptoms below:Original Text:\{symptom\_text\}Symptoms to Extract:\{symptom\_list\}Current Result \(JSON\):\{initial\_result\}Scoring rubric\(0–5\), \+1 each if:1\.The JSON is valid and groups are earliest→\\tolatest\.2\.Non\-mentioned symptoms are presented as"none"in the last group\.3\.Every symptom in the list appears exactly once overall\.4\.Symptoms grouped together occur around the same time\.5\.Group ordering follows the text’s temporal cues\.Return ONLY one of the following JSON objects:•If score≥3\\geq 3:`\{"score": <int\>, "feedback": ""\}`•If score<3<3:`\{"score": <int\>, "feedback": "<specific fixes\>"\}`

### A\.4Heatmaps\.

Figure[A\.1](https://arxiv.org/html/2608.12779#S1.F1)compares ground\-truth before–after symptom relationships with the best\-performing Claude CRAFT\-Full configuration across all three vaccine types and overall\.

### A\.5Case Study 2: Verifier Miscalibration Induces Oscillation \(Claude Sonnet 4\.5, GUIDE\)\.

Symptom list:Chest pain, Fatigue, Carditis, Troponin increased

Narrative \(abridged\):“My son experienced chest pain and was very tired — Sat 6/19 \[…\] Monday 6/21 at 11am severe chest pain\. Taken to hospital — was hospitalized with heart inflammation and very high troponin numbers until Thursday 6/24\.”

Iteration 1Score: 2/5 — belowθ\\theta, continue

Verifier feedback:

> ASPECT=GROUPING; OP=MERGE\(g0, g1\)

Iteration 2Score: 2/5 — belowθ\\theta, continue

Verifier feedback:

> ASPECT=GROUPING; OP=SPLIT\(g0, move\_keys=\[Chest pain, Fatigue\], to=NEW\_AFTER\_\-1\); OP=SPLIT\(g1, move\_keys=\[Carditis, Troponin increased\], to=NEW\_AFTER\_1\)

Iterations 3–4Score: 2/5 — budget exhausted, return last output

Verifier feedback:\(repeatsi=1i\{=\}1andi=2i\{=\}2feedback alternately\)

Ati=1i\{=\}1, Claude produces the exact gold\-standard output, correctly separating the pre\-hospital symptoms \(Chest pain, Fatigue\) from the hospitalization findings \(Carditis, Troponin increased\)\. However, GUIDE’s verifier assigns a score of 2 and instructs aMergeoperation — because the narrative uses relative date markers \(“Sat 6/19”, “Monday 6/21”\) rather than explicit absolute anchors, the verifier cannot confidently confirm distinct temporal support for the two groups\. The model faithfully follows the instruction ati=2i\{=\}2, merging all symptoms into a single stage and producing a wrong output\. The verifier then issues aSplitinstruction, the model recovers the correct grouping ati=3i\{=\}3, and the cycle repeats\. The loop terminates ati=4i\{=\}4on a merge step, returning an incorrect final output despite the model having produced the correct answer twice\. This example illustrates how GUIDE’s verifier’s strict requirement for explicit temporal anchors is miscalibrated for narratives that express temporal order through relative date references, causing it to reject a correct first\-pass output and drive the model into an unresolvable oscillation\. Table[A\.4](https://arxiv.org/html/2608.12779#S1.T4)summarizes the predicted stages across all four iterations\.

Table A\.4:Model outputs across iterations for Example 2 \(Claude Sonnet 4\.5, GUIDE\)\.✓\\checkmark= matches gold;×\\times= grouping error\.IterScorePredicted StagesGold—\{Chest pain, Fatigue\}→\\to\{Carditis, Troponin increased\}12/5\{Chest pain, Fatigue\}→\\to\{Carditis, Troponin increased\}✓\\checkmark22/5\{Chest pain, Fatigue,Carditis, Troponin increased\}×\\times32/5\{Chest pain, Fatigue\}→\\to\{Carditis, Troponin increased\}✓\\checkmark42/5\{Chest pain, Fatigue,Carditis, Troponin increased\}×\\times