Evidence-Anchored Overdose Death Prediction from Death Investigation Reports

# Evidence-Anchored Overdose Death Prediction from Death Investigation Reports
Source: [https://arxiv.org/html/2610.19556](https://arxiv.org/html/2610.19556)
Ziqing YangAffiliation:Columbia University, USAKarli R\. HochstatterAffiliation:Friends Research Institute, USATalia NadelAffiliation:Friends Research Institute, USANicole D’AnnaAffiliation:The New York City Office of Chief Medical Examiner, USAJason K\. GrahamAffiliation:The New York City Office of Chief Medical Examiner, USAHannah JohnsonAffiliation:Independent Researcher, USAKelsey OpozdaAffiliation:The New York City Office of Chief Medical Examiner, USANabila El\-BasselAffiliation:Columbia University, USAAnubhav JangraAffiliation:Columbia University, USASmaranda MuresanAffiliation:Barnard College, Columbia University, USA

###### Abstract

Confirmation of overdose deaths in the United States is often delayed due to the complexities of certifying drug\-involved deaths\. Death investigation reports represent a rich but underutilized data source that could be used to predict whether a death is a suspected overdose \(i\.e\., a death preliminarily identified as an overdose prior to the final cause and manner of death determination\)\. We present an automated framework that classifies suspected overdoses from death investigation reports, while producing supporting explanations guided by expert knowledge\. Through an expert\-guided design process, we investigate 1\) what type of explanations are best for model prediction performance, and 2\) which architecture is better suited for this task: pipeline models that rationalize first and then predict, or self\-rationalizing models that jointly predict and rationalize? Our experiments show that a pipeline model with explanations grounded in the input death investigation reports outperforms the other settings\. Our best model also outperforms a rule\-based system currently in deployment by 15%, while allowing expert interventions for final decisions\.

††footnotetext:Corresponding authors:raavi02g@gmail\.com,anubhav@cs\.columbia\.eduandsmuresan@barnard\.edu\.## 1Introduction

The age\-adjusted rate of overdose death in the US jumped from 8\.9 per 100,000 population in 2003 to 31\.3 in 2023[Garnett and Miniño \(2024\)](https://arxiv.org/html/2610.19556#bib.bib35)\. Lag time for drug overdose deaths is generally longer than that for all other causes of death by more than six months[Ahmad et al\. \(2026\)](https://arxiv.org/html/2610.19556#bib.bib15), and the resulting delays prevent rapid surveillance and obstruct timely public health interventions[Gelberg et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib14)\. An underutilized source of early information lies in the narrative reports produced by on\-scene death investigators\. These reports are typically available shortly after a fatality and contain rich information about physical findings, substance involvement, and witness observations[Davis et al\. \(2020\)](https://arxiv.org/html/2610.19556#bib.bib31);[O’Donnell et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib32)\.

Figure 1:The two explanation types on an example death investigation report of mock data \(completely deidentified\)\. Feature\-verbalizationsRR\(top\) are predefined statements per expert\-identified feature; feature\-guided rationalesRGR\_\{G\}\(highlighted\) are verbatim sentences extracted from the report corresponding to the given expert\-identified feature\.Existing early\-detection systems depend on manual investigator input, ranging from a single suspected\-overdose flag[Hochstatter et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib33)to pre\-defined features extracted from the reports[Hochstatter et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib3);[Vannice et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib10), making these systems time\-intensive and dependent on a small, highly trained workforce\. Feature\-based ML pipelines[Goodman\-Meza et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib11)and traditional NLP[Ward et al\. \(2019\)](https://arxiv.org/html/2610.19556#bib.bib29)have been used to automate parts of this process, but these methods operate over death certificates, which may not be available until six months after the death[Spencer and Ahmad \(2017\)](https://arxiv.org/html/2610.19556#bib.bib8)\.

Figure 2:Overview of the proposed method\. We compare two types of explanations: feature\-guided rationalesRGR\_\{G\}\(sentences extracted from the report\) and feature\-verbalizationsRR\(predefined statements for each present feature\)\. A classifier model then predicts the overdose labelOOfrom the explanation, together with the report\. We show here the decoupled regime, in which explanation generation and prediction are two separate steps rather than a single joint generation\.While Large language models \(LLM\) have revolutionized NLP applications in various domains, their risks of hallucinations[Turpin et al\. \(2023\)](https://arxiv.org/html/2610.19556#bib.bib22);[Huang et al\. \(2025\)](https://arxiv.org/html/2610.19556#bib.bib34), non\-deterministic output[He and Lab \(2025\)](https://arxiv.org/html/2610.19556#bib.bib36), and black box nature requires care in applications in high\-stake domains such as overdose prediction\. To overcome these limitations, we adopt an expert\-guided design process[Cho et al\. \(2026\)](https://arxiv.org/html/2610.19556#bib.bib37), directly collaborating with domain experts to explore the use of open\-source LLMs for predicting overdose death directly from death investigation reports, while producing supporting explanations guided by expert knowledge\.

We investigate two questions: 1\) what type of explanations are best for model prediction performance and 2\) are pipeline models that rationalize first and then predict or self\-rationalizing models that jointly predict and rationalize[Wiegreffe et al\. \(2021\)](https://arxiv.org/html/2610.19556#bib.bib38)better for this task\. To answer the first question, we explore two types of explanations: 1\)feature verbalization, which are natural language descriptions of the expert\-provided features associated with a death investigation report, and 2\)feature\-guided rationales, which are explicit text spans extracted from the death investigation reports that correspond to an expert\-provided feature\. Figure[1](https://arxiv.org/html/2610.19556#S1.F1)shows an expert\-written mock death investigation report and the corresponding rationales obtained by the overdose prediction system:RRrepresents the feature verbalization for the feature "Injection Equipment on Scene", whileRGR\_\{G\}shows the exact text span from the death investigation report corresponding to this expert\-provided feature \(i\.e\., feature\-guided rationale\)\.

Our results show that a pipeline model with feature\-guided rationales using open\-source models111We focus on open\-source models, as proprietary models may raise data privacy and HIPAA compliance concerns in clinical settings with sensitive information\.is the best setup for this tasks\. Moreover, our best model achieves a 15% improvement in overdose prediction over an adopted expert rule\-based system that use death investigation reports for overdose predictions[Hochstatter et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib3)\. The pipeline model also allows experts to refine explanations during inference rather than starting from scratch, which could lead to substantial time savings per report\. We make three key contributions in this work:

- •To the best of our knowledge, we propose the first end\-to\-end system for automated decision support in overdose prediction from death investigation reports, eliminating the need for manual data extraction and enabling the system to scale\.
- •We present an interpretable approach, supporting an expert\-guided design in which feedback drives model refinement during both training and inference\.
- •We conduct a systematic comparison of explanation types \(feature verbalization and feature\-guided rationales\) and model types \(pipeline and self\-rationalizing models\) for overdose prediction\.

## 2Related Work

### 2\.1Explanation generation

Early work on model interpretability focused on post\-hoc explanation methods such as attention visualization\([Bahdanau et al\., 2016](https://arxiv.org/html/2610.19556#bib.bib18)\), saliency maps, and LIME\-style perturbation approaches\([Ribeiro et al\., 2016](https://arxiv.org/html/2610.19556#bib.bib19)\)\. However, these methods do not provide human\-readable rationales that domain experts can provide feedback upon\.[Lei et al\. \(2016\)](https://arxiv.org/html/2610.19556#bib.bib20)pioneered the extraction of human\-readable rationales from the input that are sufficient to support the model’s predictions\.

The advent of LLMs significantly improved rationale generation quality\.[Wei et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib21)demonstrated that chain of thought prompting that produces intermediate reasoning steps before answering improves performance on tasks\.[Sun et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib27)found that incorporating rationales to even a small amount of training data can considerably improve model performance\. Subsequent work, however, examined whether such rationales reflect the model’s true reasoning or merely serve as post\-hoc justifications\([Turpin et al\., 2023](https://arxiv.org/html/2610.19556#bib.bib22)\), motivating approaches that grounds rationales in input evidence rather than generating them freely\.[Wiegreffe et al\. \(2021\)](https://arxiv.org/html/2610.19556#bib.bib38)formalized this concern and proposed association tests to assess whether a model’s rationale and prediction are genuinely linked\. They further unveiled that free\-form rationales are not faithful by construction\. Extractive rationale, on the other hand, constrains explanations to verbatim spans of the source document[Wei Jie et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib26)and makes the reasoning auditable, unlike free\-form rationales\.

Expert feedback has emerged as a mechanism for improving model\-generated rationales\.[Saakyan and Muresan \(2024\)](https://arxiv.org/html/2610.19556#bib.bib2)introduced an in\-context learning framework that integrates scarce human annotations into model training to generate high\-quality explainable outputs, illustrating how expert feedback loops can enhance the accuracy of the model\.[Hase et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib4)demonstrated that LLMs can generalize from easy annotated data to harder instances via lightweight fine\-tuning, suggesting that carefully curated supervision signals can substitute for large\-scale human annotation\.

### 2\.2Medical NLP and Explanation\-Driven Decision Support

Medical NLP has a history of extracting structured signals from free text, spanning rule\-based systems\([Friedman et al\., 2004](https://arxiv.org/html/2610.19556#bib.bib23)\)and, more recently, transformer\-based models such as BioBERT\([Lee et al\., 2020](https://arxiv.org/html/2610.19556#bib.bib24)\)and ClinicalBERT\([Huang et al\., 2020](https://arxiv.org/html/2610.19556#bib.bib25)\), fine\-tuned specifically on biomedical corpora\.[McInerney et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib1)proposed an interpretable risk prediction model that uses LLMs to surface evidence passages in electronic health records supporting diagnostic hypotheses, demonstrating that evidence\-aligned predictions reduce diagnostic delays\.

Early approaches to overdose classification utilize traditional ML\([Goodman\-Meza et al\., 2022](https://arxiv.org/html/2610.19556#bib.bib11);[Tang et al\., 2023](https://arxiv.org/html/2610.19556#bib.bib12)\)techniques on death certificates\. More recent work has demonstrated the use of LLMs for structured information extraction from clinical free text with high precision\([Wiest et al\., 2024](https://arxiv.org/html/2610.19556#bib.bib13);[Funnell et al\., 2026](https://arxiv.org/html/2610.19556#bib.bib30)\)\.

Moreover, prior work has focused primarily on analysis from death certificates, which may not be available for more than six months\. Our work operate on death investigation reports, which are available within days of death rather than months enabling rapid identification of overdose deaths\.[Hochstatter et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib3)and[Harruff et al\. \(2023\)](https://arxiv.org/html/2610.19556#bib.bib9)also use death investigation reports to identify suspected overdose deaths\. However, their rule\-based systems require extensive expert annotation\. Other systems such as[Hochstatter et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib33)output only an expert provided global flag based on the totality of evidence, discarding the criterion\-level evidence that our automated and intervenable framework preserves\.

## 3Methods and Experimental Setting

LetIdenote the input death investigation report,O∈\{0,1\}\\textbf\{O\}\\in\\\{0,1\\\}the suspected overdose prediction, andV=\{v1,…,v\|V\|\}V=\\\{v\_\{1\},\\ldots,v\_\{\|V\|\}\\\}the set of expert\-defined features\. We also define the explanations𝐑\\mathbf\{R\}that justifies the predictionO\. The explanations are human\-readable and each explanation map back to a featureviv\_\{i\}\.

We define two forms of explanations \-feature\-verbalizations\(RR\): natural language template mapped to the expert\-provided featuresviv\_\{i\}andfeature\-guided rationales\(RGR\_\{G\}\): extracted evidence spans corresponding to the expert\-defined featuresviv\_\{i\}\. An example of the two types of explanation is provided in Figure[1](https://arxiv.org/html/2610.19556#S1.F1)\. Table[7](https://arxiv.org/html/2610.19556#A2.T7)in Appendix E shows all the expert\-defined features predictive of overdose death that can be associated with a death investigation report \(e\.g\., History of Prior Overdose\(s\), Benzodiazepine\(s\), Pills/Capsules on Scene\)\. We use𝐑∈\{R,RG\}\\mathbf\{R\}\\in\\\{R,R\_\{G\}\\\}to refer to either form when a statement applies to both\.

### 3\.1Dataset

Our dataset’s structure informs the design of our proposed methodology, including the explanation generation and prediction models\. The dataset comprises of 9,263 death investigation reports with two splits:

- •Feature\-annotated: Reports for which expert\-labeled boolean features\{vk\}\\\{v\_\{k\}\\\}and the final overdose labels𝐎\\mathbf\{O\}are available\.
- •Weakly\-annotated: Reports for which only the final overdose labels𝐎\\mathbf\{O\}are available, without feature annotations\{vk\}\\\{v\_\{k\}\\\}\.

Table 1:Overdose label distribution across the Feature\-annotated and Weakly\-annotated partitions\.YesandNodenote whether the death was certified as an overdose\.Table[1](https://arxiv.org/html/2610.19556#S3.T1)presents the distribution of overdose labels across the train, validation, and test sets, created by random sampling from the full pool of reports\. Overall, 64\.7% of reports are labeled as overdoses; given this class imbalance, we report macro\-F1 throughout\. Moreover, 39\.8% of reports lack expert feature annotations, motivating our use of models to infer them and further obtain downstream explanations𝐑\\mathbf\{R\}\.

### 3\.2Explanation Generation and Prediction Models

Our dataset contains the binary values for each feature only for the feature\-annotated partition\. To augment the weakly\-annotated partition with these missing annotations, we used five\-shot question answering withLlama\-3\.1\-8B[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib6)to extract feature labels\. The system prompt, per\-feature questions, and configuration details are provided in Appendix[A](https://arxiv.org/html/2610.19556#A1)\. Table[5](https://arxiv.org/html/2610.19556#A1.T5)provides the per\-feature extraction performance for the feature\-guided rationales\. We used this fully augmented feature\-labeled dataset to generate the two types of explanations\. We denote the completed feature value byv^k∈\{0,1\}\\hat\{v\}\_\{k\}\\in\\\{0,1\\\}, which corresponds to the annotated valuevkv\_\{k\}when available and otherwise to the feature value predicted by Llama\-3\.1\-8B\.

#### 3\.2\.1Feature Verbalization

Each variablevkv\_\{k\}\(e\.g\., “Drugs/Paraphernalia: Injection Equipment on Scene”\) is verbalized to a statementsks\_\{k\}\(e\.g\., “There is injection equipment reported on\-scene\.”\); the full set of mappings is provided in Table[7](https://arxiv.org/html/2610.19556#A2.T7)\. The feature\-verbalized explanation is then constructed by enumerating the statements for all variables indicated as present:

R=\{sk∣v^k=1,k=1,…,n\}R=\\\{s\_\{k\}\\mid\\hat\{v\}\_\{k\}=1,k=1,\\ldots,n\\\}
We obtained the feature indicators from two sources: for the feature\-annotated partition, we used the expert\-labeled indicators directly, whereas for the weakly\-annotated partition, we inferred them usingLlama\-3\.1\-8B, as described in Section[3\.2](https://arxiv.org/html/2610.19556#S3.SS2)\. The indicators were then mapped to their corresponding statementssks\_\{k\}and concatenated to form the feature\-verbalized explanations\.

##### Prediction Models\.

We considered two paradigms for producing the final prediction from the constructed explanations\.Pipeline\(I→RI\\rightarrow R;I​R→OI\\,R\\rightarrow O\), where the explanation generation and prediction are separated into two stages\. The first model generates the explanation from the input report, and the second model then takes the explanation, alongside the original report \(II\), as input to produce the final prediction\.Self\-rationalizing\(I→R,OI\\rightarrow R,O\): A single model receives the input report and produces both the explanation and the final prediction in a single pass\. We trained two decoder\-only LLMs for both pipeline and self\-rationalize models \(𝐈→𝐎\\mathbf\{I\}\\rightarrow\\mathbf\{O\},𝐈→R,𝐎\\mathbf\{I\}\\rightarrow R,\\mathbf\{O\}, andR→𝐎R\\rightarrow\\mathbf\{O\}\), namelyLlama\-3\.1\-8B Instruct[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib6)andQwen3\-8B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.19556#bib.bib7)\. These models are of comparable size but differ in their pretraining emphasis:Llama\-3\.1provides a strong general\-purpose baseline, whileQwen3offers strong performance on instruction\-following tasks\. We fine\-tuned each model with LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib16)on the constructed explanations and gold output labels\.

#### 3\.2\.2Feature\-guided Rationales

Feature\-guided rationales are extracted directly from the death investigation report where each rationale is a verbatim sentence\-level span222Our design intentionally extracts rationales at the sentence level rather than as arbitrary spans for simplicity and robustness\. Unbounded spans introduce additional noise, along with limiting the contextual knowledge from shorter spans, making the experts read the full sentence regardless of span width\.that can be traced back to the specific part of the report\. We cast this rationale extraction as a multi\-label sentence classification task\. Let𝐈=\{x1,…,xm\}\\mathbf\{I\}=\\\{x\_\{1\},\\ldots,x\_\{m\}\\\}denote themmsentences of a report\. For each sentence\-variable pair\(xi,vk\)\(x\_\{i\},v\_\{k\}\), we predict a binary labelr^i,kG∈\{0,1\}\\hat\{r\}^\{G\}\_\{i,k\}\\in\\\{0,1\\\}indicating whetherxix\_\{i\}constitutes evidence for featurevkv\_\{k\}\. The feature\-guided rationale is the set of sentences supporting at least one variable:

RG=\{xi∣∃vk∈V:r^i,kG=1\}R\_\{G\}=\\\{\\,x\_\{i\}\\mid\\exists\\,v\_\{k\}\\in V:\\hat\{r\}^\{G\}\_\{i,k\}=1\\,\\\}
Our dataset contains no gold\-labeled evidence spans for either partition\. We therefore construct silver\-standard labels forr^i,kG\\hat\{r\}^\{G\}\_\{i,k\}over the full dataset via a teacher–student pipeline\. We use an ensemble of teacher LLMs’ labels for each sentence–feature pair independently, and aggregate their outputs by majority vote to obtain the silver labeledRGs​i​l​v​e​rR\_\{G\}^\{silver\}for each report\. The teacher ensemble used to construct silver labels comprises MedGemma\-27B[Sellergren et al\. \(2026\)](https://arxiv.org/html/2610.19556#bib.bib5), Llama\-70B[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.19556#bib.bib6), and Qwen3\-30B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.19556#bib.bib7), inferred in full precision\.

These silver labels are then used to fine\-tune a student encoder\-only model, for sentence\-level multi\-label classification predictingr^i,kG\\hat\{r\}^\{G\}\_\{i,k\}for all featuresvk∈Vv\_\{k\}\\in Vsimultaneously given𝐈\\mathbf\{I\}\. At inference, the fine\-tuned student extractsRGR\_\{G\}directly from𝐈\\mathbf\{I\}, without the teacher ensemble\. For the student model, we usedClinical\-ModernBERT[Lee et al\. \(2025\)](https://arxiv.org/html/2610.19556#bib.bib17), a lightweight encoder\-only model pre\-trained on medical corpora\.

##### Prediction Models\.

Since these explanations are extracted by an encoder\-only model, self\-rationalization is not applicable for this explanation type, and we therefore consider only the pipeline models\.Pipeline \(I→RG;\[I\]​RG→OI\\rightarrow R\_\{G\};\\ \[I\]R\_\{G\}\\rightarrow O\):the paradigm in which a student model first extracts grounded rationalesRGR\_\{G\}fromII, and a separate classifier then predictsOOfromRGR\_\{G\}alone or from\(I,RG\)\(I,R\_\{G\}\)jointly, ablating the contribution of the original report at the classification stage\. We additionally investigate whether rationales alone are sufficient for classification,viz\.,whetherRG→O\{R\_\{G\}\}\\rightarrow Ocan match the performance of a model with full access to the original report\. This motivates a direct comparison betweenRG→OR\_\{G\}\\ \\rightarrow OandI→OI\\rightarrow O, isolating the information contributed by the extracted rationales versus the original narrative\.

#### 3\.2\.3Implementation Details

All decoder\-only models are fine\-tuned using LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib16)with rankr=32r=32and scaling factorα=32\\alpha=32, applied to all attention and feed\-forward projection layers on single NVIDIA A100\-40GB GPU\. Models are loaded in 4\-bit quantization and fine\-tuned with gradient checkpointing enabled\. The maximum sequence length is set to 4,096 tokens to accommodate longer reasoning traces\. Inference is performed using vLLM[Kwon et al\. \(2023\)](https://arxiv.org/html/2610.19556#bib.bib28)for efficient generation\.Clinical\-ModernBERTis fine\-tuned in full precision for the sentence\-level multi\-label classification task, with a classification head\.

## 4Results

Table[2](https://arxiv.org/html/2610.19556#S4.T2)reports accuracy and macro\-F1 across all model variants and evaluation splits\. The rule\-based baseline with expert\-human annotations, SPOT[Hochstatter et al\. \(2022\)](https://arxiv.org/html/2610.19556#bib.bib3), achieves 0\.721 Macro\-F1\. However, in contrast, the best overall configuration of our proposed approach \(Qwen3 withI,RG→OI,R\_\{G\}\\rightarrow O\), achieves 0\.831 Macro\-F1 without any expert knowledge\. This configuration illustrates that grounded rationales together with the original report result in the strongest prediction signal\. We examine the results through three views: the explanation type, pipeline versus self\-rationalizing models, and the data splits\.

Table 2:Comparison of all variants across test splits\.I→OI\\rightarrow O: direct prediction from report\.RG→O\\ R\_\{G\}\\rightarrow O: pipeline model with just feature\-guided rationale as input for prediction\.I,RG→OI,R\_\{G\}\\rightarrow O: pipeline model with feature\-guided rationales and report as input\.I→R,OI\\rightarrow R,O: self\-rationalized model with feature verbalization as explanations\.I,R→OI,R\\rightarrow O: pipeline model with report and feature verbalization as explanations as input for prediction\.Bold= best configuration per row\.We compute SPOT’s\([Hochstatter et al\., 2022](https://arxiv.org/html/2610.19556#bib.bib3)\)performance on our test split obtaining 0\.736 accuracy and 0\.721 macro\-F1\.

##### Analysis 1: Explanation types

Feature\-guided rationales outperform feature\-verbalized explanations on every split for both models, with the best feature\-guided model exceeding the best feature\-verbalized model by 2\.3% macro\-F1\. Apart from final prediction accuracy, we also compare per\-feature extraction quality on the feature\-annotated split for the feature\-guided rationale type and find that across all the features the model extracted features match the human extracted features around 75% of the cases and there is a discrepancy of at least one feature in the remaining 25%\. \(More details in Appendix[A](https://arxiv.org/html/2610.19556#A1)\)\. We also test whether just indicating the presence of feature alone carries the signal required to predict overdose and subsequently trained a logistic regression classifier on the binary variable indicators\. The best logistic regression model trained using feature\-guided rationales achieved 0\.621 macro\-F1 which is substantially lower than the corresponding best Qwen3 model by 33\.8%\. Therefore, knowing which features are present is not sufficient; the rationale text and report are needed to predict overdose outcomes, underscoring the value of feature\-guided rationales over feature\-verbalized explanations and bare feature presence\.

##### Analysis 2: Pipeline vs Self\-rationalizing Models\.

For feature\-verbalized explanations, the pipeline regime outperforms the self\-rationalizing regime for both Llama\-3\.1 \(0\.812 vs\. 0\.779 macro\-F1\) and Qwen3 \(0\.776 vs\. 0\.768\), i\.e\., conditioning the prediction on the report together with a pre\-generated explanation is more effective than generating the explanation and prediction in a single pass\. This comparison is only possible for feature\-verbalized explanations, since feature\-guided rationales use an encoder\-only student that cannot generate explanation and prediction jointly\.[Wiegreffe et al\. \(2021\)](https://arxiv.org/html/2610.19556#bib.bib38)report the opposite trend, where self\-rationalizing models outperform pipeline models\. Their pipeline, however, predicts from the explanation alone, and they show that explanations are often not sufficient without the original input\. Our pipeline gives the classifier both the death investigation report and the explanation, which avoids this limitation\. Our results for feature\-guided rationales also confirm their finding on sufficiency: the rationale alone is not sufficient, asRG→OR\_\{G\}\\rightarrow Operforms significantly worse than bothI→OI\\rightarrow OandI,RG→OI,R\_\{G\}\\rightarrow O\.

##### Analysis 3: Across different data splits\.

The macro\-F1 for weakly\-annotated split is higher than the corresponding feature\-annotated split for the best model which is surprising given the training data in the feature\-annotated split is human labeled\. One of the possible explanations for this can be that almost all samples in the feature\-annotated split contain features that are indicative of overdose and hence distinguishing between them is harder than weakly\-annotated split despite having human\-labeled data\. Another possible reason can be due to the class imbalance in the Feature\-annotated dataset\.

Table 3:F1 scores for sentence\-level classification without and with expert feedback integration\. % Improvement is computed asw/ Feedback−w/o Feedbackw/o Feedback×100\\frac\{\\text\{w/ Feedback\}\-\\text\{w/o Feedback\}\}\{\\text\{w/o Feedback\}\}\\times 100\. "S\.U\.H\." denotes "Substance Use History" and "D/P" denotes "Drugs/Paraphernalia"\.
### 4\.1Teacher Agreement for Silver Data Construction

We analyzed the pairwise Cohen’sκ\\kappaagreement between the three teacher models used to construct the silver\-labeled dataset for feature\-guided rationale extractionRGR\_\{G\}\(see Table[8](https://arxiv.org/html/2610.19556#A3.T8)\)\. Aggregate three\-way agreement between all the teacher models is moderate \(κ=0\.361\\kappa=0\.361\), with MedGemma\-LLaMa showing the strongest pairwise agreement \(κ=0\.500\\kappa=0\.500\) and Qwen3\-LLaMa the weakest \(κ=0\.284\\kappa=0\.284\)\.

Table 4:Qualitative analysis of three representative cases under the decoupled paradigm\. Evidence spans are extracted verbatim and abridged here\. Cases \(a\) and \(b\) show correct extraction of evidence; case \(c\) shows failure propagating from the extraction stage\.\(a\) Correct rejection\(b\) Missed suspected overdose\(c\) Extraction errorGold: No Pred: No ✓Gold: Yes Pred: No×\\timesGold: Yes Pred: No×\\timesEvidence

∙\\bulletperson with PMHx of CAD, HTN, DM, COPD,*remote h/o drug use*, depression and schizophrenia; found down in a shelter\.
∙\\bulletThe only drug use she is aware of is marijuana use*in their teens–20s*\.∙\\bulletDrank socially;*family denied*any known illicit drug use\.
∙\\bulletA*cut red straw*and a blue pill \(imprint B706\) found on a nearby table\.
∙\\bullet*Powder, appearing to be crushed pills*, in the clear plastic wrapper of a cigarette box\.∙\\bulletNo illicit drugs were found\.
∙\\bulletFamily stated the decedent was not known to use drugs or alcohol\.v^k\\hat\{v\}\_\{k\}

History of substance use:1
Drug paraphernalia: 0History of substance use: 0
Paraphernalia at scene:1
Loose/non\-prescribed pills:1History of substance use: 0
Paraphernalia at scene: 0
Loose/non\-prescribed pills: 0Analysis

The model correctly classifies this case as a non\-overdose\. A history\-of\-substance\-use flag is present, but the extracted spans reveal that the “remote h/o drug use” refers specifically to marijuana\. Since marijuana is excluded from the substance use history in the overdose prediction problem due to low risk, this is a false annotation that is correctly rejected using the additional context from the extracted evidence\.The model fails to classify this case as a suspected overdose despite extracting substantial scene evidence: drug packaging \(a clear plastic wrapper\), a loose pill not prescribed to the decedent, a cut straw, and powder/residue\. The rationale is accurate, but the classifier does not weigh the accumulated scene indicators strongly enough against the family’s denial of drug use\.Here the error originates in the extraction stage: the model extracts misleading spans, which propagate to an incorrect classification\. This failure mode is specific to the decoupled paradigm and marks the point where expert review of the rationale prior to prediction would intervene and help in downstream classification\.We found that the agreement varies considerably across features, which also correlates with annotation difficulty\. Scene evidence variables \(e\.g\.,Powder/Residue on Scene\(κ=0\.497\\kappa=0\.497\) andPackaging on Scene\) \(κ=0\.433\\kappa=0\.433\) that are easy to annotate tend to have higher agreement as compared to substance use history features, particularlyPrescription Medication\(s\)\(κ=0\.171\\kappa=0\.171\)\. This pattern is consistent with the F1 results in Table[3](https://arxiv.org/html/2610.19556#S4.T3), where the same feature showed the largest gains from expert feedback \(see Section[4\.2](https://arxiv.org/html/2610.19556#S4.SS2)\), suggesting that low inter\-teacher agreement is a reliable signal of features requiring targeted correction\.

### 4\.2Expert driven online testing

We fine\-tunedClinicalModernBERTon the silver\-labeled dataset for multi\-label sentence classification across the features inVVas described in[3\.2\.2](https://arxiv.org/html/2610.19556#S3.SS2.SSS2), producing the student model for feature\-guided rationale extractionRGR\_\{G\}\. Table[3](https://arxiv.org/html/2610.19556#S4.T3)reports per\-feature F1 scores for extraction performance on the feature\-annotated split with and without expert feedback integration\. The aggregate F1 improves from 0\.602 without feedback to 0\.684 with feedback, a 13\.6% gain\.

Expert feedback yields the most substantial improvements for rare or underspecified features \(eg\.Prescription Medication\(s\)andBenzodiazepines\)\. These improvements highlight the value of domain expertise in disambiguating forensic terminology\. Conversely, well\-represented features such asCocaineandKetamineremain stable, indicating that the teacher ensemble already provides strong silver supervision for these categories\.PCPachieves 0\.000 F1 in both conditions, likely due to extreme rarity in the dataset\. For physically observable scene evidence such asInjection EquipmentandStraw, feedback marginally hurts performance which may be attributed to noise\.

We provide the taxonomy of types of major errors found from expert feedback in Appendix[F](https://arxiv.org/html/2610.19556#A6)\.

## 5Case Study

To characterize where the pipeline paradigm succeeds and where it does not, we examine three representative cases from the feature\-annotated test set \(Table[4](https://arxiv.org/html/2610.19556#S4.T4)\)\. Together they separate two different failure modes: errors of aggregation, where evidence is correctly surfaced but incorrectly weighed, and errors of extraction, where the evidence never reaches the classifier at all and hence leads to error in the downstream prediction task as well\.

## 6Discussion

In our experiments, feature\-guided rationales consistently outperform feature verbalizations across all data splits, which highlights the value of grounding explanations in exact spans of the input\. Although feature\-guided rationales alone \(RG→OR\_\{G\}\\rightarrow O\) do not match the full report \(I→OI\\rightarrow O; Table[2](https://arxiv.org/html/2610.19556#S4.T2)\), they complement it: when we condition on bothIIandRGR\_\{G\}, we obtain our best configuration \(0\.831 macro\-F1\)\. This signal lies in the evidence text rather than in the features it represents, as a logistic regression over the same features reaches only 0\.621 macro\-F1\. In case \(a\) of Table[4](https://arxiv.org/html/2610.19556#S4.T4), for instance, the rationale reveals that a flagged substance use history refers only to marijuana, which allows the model to correctly reject the case\.

Our pipeline decouples explanation generation from prediction, which allows experts to correct the explanation behind each feature before it reaches the classifier\. When we incorporate expert feedback, rationale extraction F1 improves by 13\.6% \(Table[3](https://arxiv.org/html/2610.19556#S4.T3)\)\. In our study, experts most often remove on\-scene evidence that the model reads as a history of use and add synonyms that the model misses\. Without these corrections, such errors propagate to the final prediction \(Table[4](https://arxiv.org/html/2610.19556#S4.T4), case c\)\.

Prior work on overdose prediction either treats the model as a black\-box labeler\([Goodman\-Meza et al\., 2022](https://arxiv.org/html/2610.19556#bib.bib11);[Funnell et al\., 2026](https://arxiv.org/html/2610.19556#bib.bib30)\)or depends on manual annotation of death investigation reports\([Hochstatter et al\., 2022](https://arxiv.org/html/2610.19556#bib.bib3);[Harruff et al\., 2023](https://arxiv.org/html/2610.19556#bib.bib9)\)\. Our framework requires no manual annotation at inference, allows experts to refine the evidence behind each prediction, and uses only open\-weight models that run locally, so sensitive data never leaves the practitioner’s environment\.

## 7Conclusion

The practical bottleneck our system targets is timeliness\. Finalized overdose mortality data are delayed by months because confirmation depends on toxicology findings, autopsy reports, and sometimes other investigative information\. Whereas, the deployed alternatives that work on death investigation reports for identifying suspected overdose deaths depend on meticulous manual annotation that scales poorly\. Our paper presented a study that investigated what types of explanations and what type of explanation\-guided models \(pipeline or self\-rationalizing\) are best for this task, showing that a pipeline approach with feature\-guided rationales grounded in the death investigation report is the best setup for this task\. This setup enables an expert to refine machine\-extracted rationales rather than reading each report from scratch, offering an earlier signal at lower human cost and without asking public\-health practitioners to trust an opaque model\. The correctable explanation can increase the trustworthiness of the system: it keeps a human accountable for each determination while removing the manual\-extraction burden that currently limits surveillance throughput\. We see this as the central contribution for the public\-health community, not just a faster model but a model whose outputs domain experts can review and adjust as needed to ensure validity within their framework\.

## Limitations

Our evaluation is conducted on a single jurisdiction, and it remains an open question how well the learned models transfer to coroner or medical examiner offices with different documentation practices\. The expert\-defined features, while operationally validated, may not capture the full range of overdose\-relevant evidence across jurisdictions or evolving drug landscapes\. Additionally, the silver\-labeled training data for feature\-guided rationale extraction is inherently noisy, moderate aggregate teacher agreement \(κ=0\.361\\kappa=0\.361\) means that a fraction of sentence\-level labels are incorrect, potentially introducing systematic errors for the lowest\-agreement variables\. Finally, the expert feedback loop we evaluate is retrospective; prospective deployment would require careful interface design to ensure that corrections are both efficient and consistent across reviewers\.

## Ethical considerations

Robust protocols were put in place to prevent the release of confidential information\. This study was approved by our institutions IRB\. All data sharing occurred via a secure SharePoint environment that had been configured to be both Health Insurance Portability and Accountability Act \(HIPAA\) and Food and Drug Administration \(FDA\) compliant\. Model training and development occurred within a certified cloud\-based environment approved for use with Personally Identifiable Information \(PII\), Personal Health Information \(PHI\), and HIPAA data\. Before the research team had access to the data, narrative death investigation reports were de\-identified using the Presidio analyzer to prevent the unnecessary use and release of identifiable data during AI model training and development\.

## AI use disclosure

We have used generative AI tools to aid in writing the code for metric generation and generating tables\. We have not used AI tools in our core experiment implementation and have also refrained from using them for ideation, literature review, figure generation, or end\-to\-end paper or section writing\. We have reviewed all AI\-assisted work, validated the code used for metric generation, cross\-checked the illustration artifacts, and ensured that all authors had full agency over the paper\-writing process\. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI\.

## References

- Ahmadet al\.\(2026\)F\. B\. Ahmad, J\. A\. Cisewski, L\. M\. Rossen, and P\. SuttonProvisional drug overdose death counts\.Technical reportNational Center for Health Statistics\.Note:Accessed July 18, 2026External Links:[Document](https://dx.doi.org/10.15620/cdc/20250305008),[Link](https://www.cdc.gov/nchs/nvss/vsrr/drug-overdose-data.htm)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p1.1)\.
- Bahdanauet al\.\(2016\)D\. Bahdanau, K\. Cho, and Y\. BengioNeural machine translation by jointly learning to align and translate\.External Links:1409\.0473,[Link](https://arxiv.org/abs/1409.0473)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p1.1)\.
- Choet al\.\(2026\)D\. Cho, M\. Zhang, and R\. ChunaraExpert\-guided clinical text augmentation via query\-based model collaboration\.External Links:2509\.21530,[Link](https://arxiv.org/abs/2509.21530)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p3.1)\.
- Daviset al\.\(2020\)G\. G\. Davis, A\. B\. Cadwallader, C\. L\. Fligner, T\. P\. Gilson, E\. R\. Hall, K\. E\. Harshbarger, R\. Kronstrand, C\. T\. Mallak, J\. L\. McLemore, R\. A\. Middleberg, O\. L\. Middleton, L\. S\. Nelson, A\. Rogalska, E\. Tonsfeldt, J\. P\. Walterscheid, and R\. E\. WineckerPosition paper: recommendations for the investigation, diagnosis, and certification of deaths related to opioid and other drugs\.The American Journal of Forensic Medicine and Pathology41\(3\),pp\. 152–159\.External Links:[Document](https://dx.doi.org/10.1097/PAF.0000000000000550)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p1.1)\.
- Friedmanet al\.\(2004\)C\. Friedman, L\. Shagina, Y\. Lussier, and G\. HripcsakAutomated encoding of clinical documents based on natural language processing\.Journal of the American Medical Informatics Association11\(5\),pp\. 392–402\.Note:Epub 2004 Jun 7External Links:[Document](https://dx.doi.org/10.1197/jamia.M1552)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p1.1)\.
- Funnellet al\.\(2026\)A\. J\. Funnell, P\. Petousis, F\. Harel\-Canada, R\. Romero, A\. A\. T\. Bui, A\. Koncsol, H\. Chaturvedi, C\. Shover, and D\. Goodman\-MezaImproving drug identification in overdose death surveillance by using clinical natural language processing models\.Journal of Forensic Sciences71\(3\),pp\. 1329–1337\.External Links:[Document](https://dx.doi.org/10.1111/1556-4029.70281)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p2.1),[§6](https://arxiv.org/html/2610.19556#S6.p3.1)\.
- Garnett and Miniño \(2024\)M\. F\. Garnett and A\. M\. MiniñoDrug overdose deaths in the United States, 2003–2023\.Technical reportTechnical Report522,NCHS Data Brief,National Center for Health Statistics\.External Links:[Document](https://dx.doi.org/10.15620/cdc/170565)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p1.1)\.
- Gelberget al\.\(2024\)K\. H\. Gelberg, R\. Chase, M\. Lindstrom, K\. R\. Hochstatter, L\. Larochelle, and T\. HuntNeed for improved timeliness of reporting on drug overdose fatalities: the HEALing communities study\.Public Health Reports139\(1\),pp\. 72–78\.External Links:[Document](https://dx.doi.org/10.1177/00333549231161339)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p1.1)\.
- Goodman\-Mezaet al\.\(2022\)D\. Goodman\-Meza, C\. L\. Shover, J\. A\. Medina, A\. B\. Tang, S\. Shoptaw, and A\. A\. T\. BuiDevelopment and validation of machine models using Natural Language Processing to classify substances involved in overdose deaths\.JAMA Network Open5\(8\),pp\. e2225593\.External Links:[Document](https://dx.doi.org/10.1001/jamanetworkopen.2022.25593)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p2.1),[§6](https://arxiv.org/html/2610.19556#S6.p3.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3\.2\.1](https://arxiv.org/html/2610.19556#S3.SS2.SSS1.Px1.p1.1),[§3\.2\.2](https://arxiv.org/html/2610.19556#S3.SS2.SSS2.p2.1),[§3\.2](https://arxiv.org/html/2610.19556#S3.SS2.p1.1)\.
- Harruffet al\.\(2023\)R\. C\. Harruff, N\. A\. Yarid, W\. L\. Barbour, and Y\. H\. MartinMedical examiner response to the drug overdose epidemic in king county washington: “real\-time” surveillance, data science, and applied forensic epidemiology\.Journal of Forensic Sciences68\(5\),pp\. 1632–1642\.External Links:[Document](https://dx.doi.org/10.1111/1556-4029.15329)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p3.1),[§6](https://arxiv.org/html/2610.19556#S6.p3.1)\.
- Haseet al\.\(2024\)P\. Hase, M\. Bansal, P\. Clark, and S\. WiegreffeThe unreasonable effectiveness of easy training data for hard tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7002–7024\.External Links:[Link](https://aclanthology.org/2024.acl-long.378/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.378)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p3.1)\.
- He and Lab \(2025\)H\. He and T\. M\. LabDefeating nondeterminism in llm inference\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/defeating\-nondeterminism\-in\-llm\-inference/External Links:[Document](https://dx.doi.org/10.64434/tml.20250910)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p3.1)\.
- Hochstatteret al\.\(2022\)K\. R\. Hochstatter, S\. Rastogi, K\. Klein, C\. Tait\-Ozer, N\. El\-Bassel, and J\. GrahamPredicting accidental drug overdose as the cause of fatality in near real\-time using the suspected potential overdose tracker \(spot\): public health implications\.BMC Public Health22\(1\),pp\. 1311\.External Links:[Document](https://dx.doi.org/10.1186/s12889-022-13700-0)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p2.1),[§1](https://arxiv.org/html/2610.19556#S1.p5.1),[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p3.1),[Table 2](https://arxiv.org/html/2610.19556#S4.T2.5),[§4](https://arxiv.org/html/2610.19556#S4.p1.1),[§6](https://arxiv.org/html/2610.19556#S6.p3.1)\.
- Hochstatteret al\.\(2024\)K\. R\. Hochstatter, M\. Williams, S\. Latham, D\. Fenton, and A\. L\. FalzonRapid identification of suspected drug overdose deaths by death investigators, new jersey, 2020\.Public Health Reports139\(5\),pp\. 549–556\.External Links:[Document](https://dx.doi.org/10.1177/00333549241230921)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p3.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§3\.2\.1](https://arxiv.org/html/2610.19556#S3.SS2.SSS1.Px1.p1.1),[§3\.2\.3](https://arxiv.org/html/2610.19556#S3.SS2.SSS3.p1.1)\.
- Huanget al\.\(2020\)K\. Huang, J\. Altosaar, and R\. RanganathClinicalBERT: modeling clinical notes and predicting hospital readmission\.External Links:1904\.05342,[Link](https://arxiv.org/abs/1904.05342)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p1.1)\.
- Huanget al\.\(2025\)L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin, and T\. LiuA survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1––55\.External Links:ISSN 1558\-2868,[Link](http://dx.doi.org/10.1145/3703155),[Document](https://dx.doi.org/10.1145/3703155)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p3.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[§3\.2\.3](https://arxiv.org/html/2610.19556#S3.SS2.SSS3.p1.1)\.
- Leeet al\.\(2020\)J\. Lee, W\. Yoon, S\. Kim, D\. Kim, S\. Kim, C\. H\. So, and J\. KangBioBERT: a pre\-trained biomedical language representation model for biomedical text mining\.Bioinformatics36\(4\),pp\. 1234–1240\.External Links:ISSN 1367\-4803,[Document](https://dx.doi.org/10.1093/bioinformatics/btz682)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p1.1)\.
- Leeet al\.\(2025\)S\. A\. Lee, A\. Wu, and J\. N\. ChiangClinical modernbert: an efficient and long context encoder for biomedical text\.External Links:2504\.03964,[Link](https://arxiv.org/abs/2504.03964)Cited by:[§3\.2\.2](https://arxiv.org/html/2610.19556#S3.SS2.SSS2.p3.1)\.
- Leiet al\.\(2016\)T\. Lei, R\. Barzilay, and T\. JaakkolaRationalizing neural predictions\.External Links:1606\.04155,[Link](https://arxiv.org/abs/1606.04155)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p1.1)\.
- McInerneyet al\.\(2024\)D\. McInerney, W\. Dickinson, L\. Flynn, A\. Young, G\. Young, J\. van de Meent, and B\. WallaceTowards reducing diagnostic errors with interpretable risk prediction\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 7193–7210\.External Links:[Link](https://aclanthology.org/2024.naacl-long.399/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.399)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p1.1)\.
- O’Donnellet al\.\(2022\)J\. O’Donnell, R\. M\. Gladden, M\. Kariisa, and C\. L\. MattsonUsing death scene and toxicology evidence to define involvement of heroin, pharmaceutical morphine, illicitly manufactured fentanyl and pharmaceutical fentanyl in opioid overdose deaths, 38 states and the district of columbia, january 2018–december 2019\.Addiction117\(5\),pp\. 1483–1490\.External Links:[Document](https://dx.doi.org/10.1111/add.15768)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p1.1)\.
- Ribeiroet al\.\(2016\)M\. T\. Ribeiro, S\. Singh, and C\. Guestrin“Why should I trust you?”: explaining the predictions of any classifier\.InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,KDD ’16,New York, NY, USA,pp\. 1135–1144\.External Links:ISBN 9781450342322,[Document](https://dx.doi.org/10.1145/2939672.2939778)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p1.1)\.
- Saakyan and Muresan \(2024\)A\. Saakyan and S\. MuresanICLEF: in\-context learning with expert feedback for explainable style transfer\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 16141–16163\.External Links:[Link](https://aclanthology.org/2024.acl-long.854/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.854)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p3.1)\.
- Sellergrenet al\.\(2026\)A\. Sellergren, S\. Kazemzadeh, T\. Jaroensri, A\. Kiraly, M\. Traverse, T\. Kohlberger, S\. Xu, F\. Jamil, C\. Hughes, C\. Lau,et al\.MedGemma technical report\.External Links:2507\.05201,[Link](https://arxiv.org/abs/2507.05201)Cited by:[§3\.2\.2](https://arxiv.org/html/2610.19556#S3.SS2.SSS2.p2.1)\.
- Spencer and Ahmad \(2017\)M\. R\. Spencer and F\. AhmadTimeliness of death certificate data for mortality surveillance and provisional estimates\.Technical reportTechnical Report1,Vital Statistics Rapid Release,National Center for Health Statistics,Hyattsville, MD\.External Links:[Link](https://www.cdc.gov/nchs/data/vsrr/report001.pdf)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p2.1)\.
- Sunet al\.\(2022\)J\. Sun, S\. Swayamdipta, J\. May, and X\. MaInvestigating the benefits of free\-form rationales\.External Links:2206\.11083,[Link](https://arxiv.org/abs/2206.11083)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p2.1)\.
- Tanget al\.\(2023\)L\. A\. Tang, J\. Korona\-Bailey, D\. Zaras, A\. Roberts, S\. Mukhopadhyay, S\. Espy, and C\. G\. WalshUsing natural language processing to predict fatal drug overdose from autopsy narrative text: algorithm development and validation study\.JMIR Public Health and Surveillance9,pp\. e45246\.External Links:[Document](https://dx.doi.org/10.2196/45246)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p2.1)\.
- Turpinet al\.\(2023\)M\. Turpin, J\. Michael, E\. Perez, and S\. R\. BowmanLanguage models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.External Links:2305\.04388,[Link](https://arxiv.org/abs/2305.04388)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p2.1)\.
- Vanniceet al\.\(2022\)K\. Vannice, J\. Hood, N\. Yarid, M\. Kay, R\. Harruff, and J\. DuchinAccuracy of medical examiner’s assessment for near\-real\-time surveillance of fatal drug overdoses, King County, Washington, March 2017–February 2018\.Public Health Reports137\(3\),pp\. 463–470\.External Links:[Document](https://dx.doi.org/10.1177/00333549211008455)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p2.1)\.
- Wardet al\.\(2019\)P\. J\. Ward, P\. J\. Rock, S\. Slavova, A\. M\. Young, T\. L\. Bunn, and R\. KavuluruEnhancing timeliness of drug overdose mortality surveillance: a machine learning approach\.PLoS ONE14\(10\),pp\. e0223318\.External Links:[Document](https://dx.doi.org/10.1371/journal.pone.0223318),[Link](https://doi.org/10.1371/journal.pone.0223318)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p2.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=_VjQlMeSB_J)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p2.1)\.
- Wei Jieet al\.\(2024\)Y\. Wei Jie, R\. Satapathy, and E\. CambriaPlausible extractive rationalization through semi\-supervised entailment signal\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 5182–5192\.External Links:[Link](https://aclanthology.org/2024.findings-acl.307/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.307)Cited by:[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p2.1)\.
- Wiegreffeet al\.\(2021\)S\. Wiegreffe, A\. Marasović, and N\. A\. SmithMeasuring association between labels and free\-text rationales\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 10266–10284\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.804/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.804)Cited by:[§1](https://arxiv.org/html/2610.19556#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.19556#S2.SS1.p2.1),[§4](https://arxiv.org/html/2610.19556#S4.SS0.SSS0.Px2.p1.1)\.
- Wiestet al\.\(2024\)I\. C\. Wiest, D\. Ferber, J\. Zhu, M\. van Treeck, S\. K\. Meyer, R\. Juglan, Z\. I\. Carrero, D\. Paech, J\. Kleesiek, M\. P\. Ebert, D\. Truhn, and J\. N\. KatherPrivacy\-preserving large language models for structured medical information retrieval\.npj Digital Medicine7\(1\),pp\. 257\.External Links:[Document](https://dx.doi.org/10.1038/s41746-024-01233-2)Cited by:[§2\.2](https://arxiv.org/html/2610.19556#S2.SS2.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.2\.1](https://arxiv.org/html/2610.19556#S3.SS2.SSS1.Px1.p1.1),[§3\.2\.2](https://arxiv.org/html/2610.19556#S3.SS2.SSS2.p2.1)\.

## Appendix AFeature Extraction for the Weakly\-annotated Partition

For each feature, we paired the death investigation report with a question asking whether that variable was present, and the model answered Yes or No\. The prompt template is shown in Figure[3](https://arxiv.org/html/2610.19556#A1.F3)\. The average macro\-F1 across the features for the feature\-annotated split is 0\.967\. More detailed results can be found in Table[5](https://arxiv.org/html/2610.19556#A1.T5)\.

Table 5:Feature extraction accuracy for on Feature\-annotated test split\. H = human; AI = our systemVariablePosHPosAITPFPFNTNRecall𝜿\\boldsymbol\{\\kappa\}Substance use historyHistory of prior overdose\(s\)454141045000\.9110\.950Benzodiazepine\(s\)252222035200\.8800\.933Cocaine \(including Crack\)173173173003721\.0001\.000Injection drug use484747014970\.9790\.988Ketamine222005431\.0001\.000Opioid\(s\)164160160043810\.9760\.982PCP322015420\.6670\.799Other Rx used not as prescribed1077035350\.7000\.821Methamphetamine / other stimulant141414005311\.0001\.000Unspecified / other3673383380291780\.9210\.884Treatment history969292044490\.9580\.974Drugs / paraphernalia on sceneInjection equipment535252014920\.9810\.989Other paraphernalia\*20510310301023400\.5020\.558Packaging120120120004251\.0001\.000Pills / capsules444040045010\.9090\.948Powder / residue133133133004121\.0001\.000Smoking equipment898989004561\.0001\.000Straw or similar717070014740\.9860\.992

Prompt: Five\-shot feature extractionSystem Prompt\.
You are a Medical Investigator that answers questions based on provided context\. Answer the question ‘Yes’ or ‘No’ with direct evidence from the context\.
Do not infer specific drugs from generic mentions of terms like ‘substance use disorder’, ‘illicit substance’ etc\. For questions about history of substance use, answer ‘Yes’ only if the text explicitly mentions past use of that substance regardless of equipment found on scene\. For questions about equipment found on scene, answer ‘Yes’ only if the text explicitly mentions the presence of that equipment on the scene and not based on inferences\. Do not infer information\.
User Input\.
Examples:
\{Feature specific examples\}
Context:
\{report\}
Question:
\{feature\_question\}
Output\.
Yes / NoFigure 3:Prompt used for five\-shot feature indicator extraction with Llama\-3\.1\-8B\-Instruct\.### A\.1Generation Setting

We extract features with Llama\-3\.1\-8B\-Instruct using greedy decoding with five in\-context examples per feature\. Each question is posed separately against each section of the report, and the feature indicatorv^k\\hat\{v\}\_\{k\}is set to present if any section yields an affirmative response\.

### A\.2Question for Each Feature

Table[6](https://arxiv.org/html/2610.19556#A1.T6)lists the questionqkq\_\{k\}paired with each feature variablevkv\_\{k\}in the five\-shot extraction prompt described above\.

Table 6:Questions used to extract each feature variable from the MLI report\. Each question is substituted into the\{feature\_question\}slot of the prompt template\.

## Appendix BFeature\-verbalized Statements

Table[7](https://arxiv.org/html/2610.19556#A2.T7)lists the statements used to construct feature\-verbalized explanationsRR\. For each featurevkv\_\{k\}predicted as present \(v^k=1\\hat\{v\}\_\{k\}=1\), the corresponding statementsks\_\{k\}is included in the explanation\.

Table 7:Predefined natural\-language statements mapped to each of the feature variables, used to construct the feature\-verbalized explanationsRR\.
## Appendix CTeacher Ensemble Prompt

Figure[4](https://arxiv.org/html/2610.19556#A3.F4)shows the prompt used by each teacher model to extract feature\-guided rationales via question answering\. Prompts are reproduced verbatim from our experiments\.

Prompt: Feature\-guided rationale extractionGiven the following investigative report, answer the question by providing verbatim evidence from the text\. \{report\} Question: \{question\} \{instruction\} You are a medicolegal death investigator that answers questions strictly based on the provided report\. Follow these rules: Evidence Only:Output only sentences extracted verbatim from the report\. No Inference:Do not infer specific substances from generic mentions \(e\.g\., ‘‘substance use disorder,’’ ‘‘illicit substances’’\) or from equipment/paraphernalia unless explicitly stated\. Substance Use:Answer ‘‘history of substance use’’ questions only if the report explicitly mentions past misuse of the specific substance asked in the question\. Presence of the substance \(or something suspected to be the substance\) on the scene is not sufficient to indicate a history of misuse of the substance\. For prescription medications, there must some mention of use not as directed or prescribed; a history of taking or having been prescribed a medication with no context or evidence to suggest it was misused is not sufficient\. Equipment:Answer ‘‘equipment and paraphernalia found on scene’’ questions only if the report explicitly mentions the presence of the specified equipment or paraphernalia\. Unanswerable:If no explicit evidence is found, output only: Unanswerable\. Multiple Statements:If multiple disjointed pieces of text from the report provide evidence, enumerate only the minimal relevant fragments \(not full sentences\)\. Each fragment must be copied verbatim from the report and clearly enumerated\. Final responses must consist of only the supported pieces of text from the investigative report \(or Unanswerable\) answering the question andno extra informationthat does not answer the question\.

Figure 4:Prompt used for extracting feature\-guided rationales via question answering in teacher models\.### C\.1Teacher Agreement

Pairwise agreement among the three teacher models is reported in Table[8](https://arxiv.org/html/2610.19556#A3.T8)\.

Table 8:Cohen’sκ\\kappaagreement between teacher models for sentence\-level evidence extraction across 18 forensic variables\. Values represent pairwise and three\-way agreement in the silver dataset construction\.
### C\.2Teacher Accuracy

Teacher accuracy against expert feature labels is reported in Table[9](https://arxiv.org/html/2610.19556#A3.T9)\.

Table 9:Per\-variable extraction accuracy of three teacher models against human\-annotated labels across all Feature\-annotated cases \(n=5,580n=5\{,\}580; train / val / test = 4,483 / 552 / 545\)\. A variable is predicted positive if the model’s extracted span list is non\-empty\. PPV = 1\.000 for all variables and all models \(zero false positives\) and is omitted for brevity\. Cohen’sκ\\kappais computed against human labels as ground truth\.Llama\-70BMedGemma\-27BQwen3\-30BVariablePosHPosAIRecallκ\\kappaPosAIRecallκ\\kappaPosAIRecallκ\\kappaSubstance use historyBenzodiazepine\(s\)2111820\.8630\.9241760\.8340\.9061200\.5690\.717Cocaine \(including Crack\)160915950\.9910\.99415640\.9720\.98012950\.8050\.854Injection drug use5234960\.9480\.9713960\.7570\.8501400\.2680\.399Ketamine27271\.0001\.000260\.9630\.981200\.7410\.850Opioid\(s\)163815940\.9730\.98115860\.9680\.97712520\.7640\.821PCP53500\.9430\.971470\.8870\.939360\.6790\.808Prescription medication\(s\)2171450\.6680\.795940\.4330\.595250\.1150\.200Methamphetamine / other stimulant1411350\.9570\.9781300\.9220\.9581000\.7090\.826Unspecified / other371531070\.8360\.77433330\.8970\.85416640\.4480\.352Treatment history8347830\.9390\.9637920\.9500\.9705440\.6520\.761History of prior overdose\(s\)5024430\.8820\.9324260\.8490\.9113290\.6550\.776Drugs / paraphernalia on sceneInjection equipment5595560\.9950\.9975510\.9860\.9924910\.8780\.929Other paraphernalia∗22298580\.3850\.4297740\.3470\.3902600\.1170\.137Packaging112011130\.9940\.99611170\.9970\.99810170\.9080\.940Pills / capsules4223630\.8600\.9193950\.9360\.9641370\.3250\.471Powder / residue123912250\.9890\.99312270\.9900\.99410640\.8590\.904Smoking equipment9959930\.9980\.9999930\.9980\.9999220\.9270\.954Straw or similar8228150\.9910\.9958170\.9940\.9967740\.9420\.965Overall†0\.9760\.9100\.9760\.9090\.9340\.718

## Appendix DComparison with Logistic regression

We also see whether the use of language models is even necessary for the second stageRG→OR\_\{G\}\\rightarrow Oand whether simple models like logistic regression are sufficient to make the prediction\. We report three logistic regression baselines across three experiments\. Experiment 1 trains on human\-annotated evidence variables from the feature\-annotated split; because these annotations exist only for feature\-annotated cases, it can be evaluated on the feature\-annotated test subset alone\. Experiment 2 trains on model\-extracted evidence variables using the same train/test case IDs as Experiment 1 to directly compare with human annotations, but can be evaluated across all subsets\. Experiment 3 trains on model\-extracted variables pooled across all splits\. All models use balanced class weights\. Table[12](https://arxiv.org/html/2610.19556#A7.T12)reports macro\-F1\. We find that the performance remains subpar as compared to using language models and hence the context in the evidence and the report is required to actually get the final prediction\.

## Appendix EAnalysis where the evidence extracted is correct but the final prediction is not

For Llama\-3\.1\-8B, we also analyze the cases where the upstream evidence extraction is correct for all the features, but there is still a discrepancy in the prediction\. We find that the majority of the cases are correctly labeled\.

## Appendix FExpert feedback types

An analysis of types of expert feedback is present in Table[10](https://arxiv.org/html/2610.19556#A6.T10)\.

Table 10:Types of expert feedback on feature\-guided rationalesRGR\_\{G\}extracted by the student model\. Numbers in parentheses give the count of edited samples per type\.TypeError typeDescriptionOver\-extraction\(111\)On\-scene evidence as historyScene items cited as a history of usePrescription as misusePrescribed medications or medication lists cited as misuseGeneric mention as a specific drugVague mentions of drug use mapped to a specific substanceNeighboring scene categoryScene item assigned to an adjacent scene featureIndirect clinical signClinical signs or hospital visits cited as direct evidenceNegation or out\-of\-scope substanceDenials, abstinence, tobacco, or marijuana cited as evidenceMissed evidence\(70\)Synonyms and street namesSubstance named by a synonym or street nameDrug inside a listed historySubstance named within a list in the medical historySigns of treatmentTreatment programs or support meetingsBoundary \(23\)Fragment mergingFragmented spans merged into full sentences

## Appendix GAdditional Model Results

Table 11:Macro\-F1 of additional prediction models\.Table 12:Macro\-F1 for logistic regression baselines across evidence variable source and training distribution\. Exp\. 1: Feature\-annotated train samples trained with human evidence variables\. Exp\. 2: Feature\-annotated train samples trained with model\-extracted variables\. Exp\. 3: Entire train split trained with model\-extracted variables\. Dashes indicate evaluations unavailable by construction\.