面向可审计和校准AI的痴呆相关碰撞严重性预测：一个支持人工审查的选择性延迟框架

# Toward Auditable and Calibrated AI for Dementia-Related Crash Severity Prediction: A Selective Deferral Framework to Support Human Review
Source: [https://arxiv.org/html/2609.22694](https://arxiv.org/html/2609.22694)
Gaurab ChhetriAffiliation:Department of Computer ScienceAffiliation:Texas State University, USAEmail:[gaurab@txstate\.edu](mailto:)Anika BaitullahAffiliation:Ingram School of EngineeringAffiliation:Texas State University, USAEmail:[baitullah@txstate\.edu](mailto:)Ph\.DAffiliation:Ingram School of EngineeringAffiliation:Texas State University, USAEmail:[subasish@txstate\.edu](mailto:)

###### Abstract

Public crash databases increasingly support automated safety analysis, but crash severity prediction remains difficult to translate into public\-sector decision workflows when models are evaluated primarily as ordinary classifiers\. This study reframes dementia\-related crash severity modeling as a decision\-aware triage problem in which a system must classify crashes into no\-injury/property\-damage\-only \(O\), minor or moderate injury \(BC\), and fatal or severe injury \(KA\), while also controlling outcome leakage, reporting severe under\-triage, calibrating confidence, and preserving every raw prediction for audit\. Using 4,781 Texas crash records with structured fields and police narratives, we evaluate structured, narrative, fusion, calibrated fusion, BERT\-family, and local large\-language\-model baselines under a stratified 70/15/15 split\. In the reported split, leakage\-controlled Gemma obtains the highest observed macro\-F1 \(0\.545; 95% bootstrap CI \[0\.507, 0\.583\]\)\. The best calibrated fusion model obtains macro\-F1 of 0\.522 and expected calibration error of 0\.033\. Selective deferral improves performance among cases retained for automatic classification\. At 70% coverage, macro\-F1 rises to 0\.573 and severity cost falls to 0\.577, while deferred cases are treated as candidates for a proposed human\-review process and are not further evaluated in the present experiment\. The study contributes a reproducible, leakage\-controlled, and uncertainty\-aware evaluation framework for crash AI systems, emphasizing auditability and selective deferral rather than accuracy alone\.

#### Keywords:

Crash severity prediction, multimodal learning, leakage control, calibrated uncertainty, selective classification, public\-sector AI

## 1Introduction

In this study, crash severity prediction refers to model\-based classification of an observed crash record into a recorded injury\-severity category using structured crash descriptors and investigator narratives; it is distinct from predicting crash occurrence or crash risk\. For dementia\-related records, this task is complicated by record\-reported rather than clinically verified dementia status, infrequent severe outcomes, and narrative text that may contain post\-crash information related to the recorded severity\. These characteristics make leakage control, severe\-case performance, uncertainty, and auditability relevant alongside overall classification accuracy\.

This study treats dementia\-related crash severity prediction as decision support, not accuracy maximization\. The target classes are no\-injury/property\-damage\-only \(O\), possible or non\-incapacitating injury \(BC\), and killed or incapacitating injury \(KA\)\. Prior work shows that machine learning can improve prediction, but performance depends on class imbalance, metric choice, and deployment context\([Sattar et al\., 2023](https://arxiv.org/html/2609.22694#bib.bib9);[Kotsyubynska et al\., 2026](https://arxiv.org/html/2609.22694#bib.bib8);[Das, 2022](https://arxiv.org/html/2609.22694#bib.bib4)\)\. The key question is not only which model performs best, but when it should classify directly and when it should defer to human review\.

The framework evaluates severity prediction as a six\-layer decision architecture: data ingestion, leakage control, representation learning, probability prediction, severity\-aware decision rules, and audit logging \(Figure[1](https://arxiv.org/html/2609.22694#S1.F1)\)\. It combines structured crash variables and police narratives, tests whether performance remains after leakage control, and uses calibration and selective deferral so confidence becomes an operational signal\. This links multimodal crash modeling with cost\-sensitive learning, which separates probability estimates from decision rules under unequal error costs\([Elkan, 2001](https://arxiv.org/html/2609.22694#bib.bib5)\), and selective classification, which allows abstention when confidence is insufficient\([Geifman and El\-Yaniv, 2017](https://arxiv.org/html/2609.22694#bib.bib6)\)\. RQ1 asks whether police narratives improve KA, BC, and O classification beyond structured variables\. RQ2 tests how much performance remains after removing structured severity leakage and masking outcome\-revealing narrative terms\. RQ3 tests whether calibrated deferral reduces severe under\-triage, especially when KA crashes are classified as BC or O\. Together, these questions assess whether multimodal crash models are predictive, leakage\-aware, uncertainty\-aware, and suitable for selective deferral in support of potential human review\. A companion web page for this study is available at[https://ai\-in\-transportation\-lab\.github\.io/dementia\-hicss/](https://ai-in-transportation-lab.github.io/dementia-hicss/)\.

![Refer to caption](https://arxiv.org/html/2609.22694v1/figures/dementia/study-design.png)Figure 1:Leakage\-controlled multimodal crash triage architecture\.
## 2Literature Review

This review covers three research areas : dementia and older\-driver crash modeling, crash narrative mining, outcome leakage in crash reports, and the use of confidence, severity cost, and human review in triage\.

### 2\.1Dementia Crash Severity Prediction

Research on dementia and driving rarely predicts crash severity\. Prior work has largely focused on risk or behavior, with case studies describing the crash patterns associated with drivers with Alzheimer’s disease\([Carr et al\., 2000](https://arxiv.org/html/2609.22694#bib.bib26)\), and case\-control studies link dementia to higher crash and hospitalization risk\([Meuleners et al\., 2016](https://arxiv.org/html/2609.22694#bib.bib23)\)\. Severity is modeled instead for older drivers, and those methods inform ours: random parameters logit recovers injury severity factors\([Dzinyela et al\., 2023](https://arxiv.org/html/2609.22694#bib.bib25)\), and gradient boosting detects severe crashes under imbalance\([Hossain et al\., 2026](https://arxiv.org/html/2609.22694#bib.bib12);[Ma et al\., 2025](https://arxiv.org/html/2609.22694#bib.bib18)\)\(Table[1](https://arxiv.org/html/2609.22694#S2.T1)\)\.

### 2\.2Crash Narrative Mining

Structured fields support large scale modeling but omit the pre\-crash context officers record in free text\([Donoughe and Atkinson, 2015](https://arxiv.org/html/2609.22694#bib.bib11);[Johnson and Bugeja, 2018](https://arxiv.org/html/2609.22694#bib.bib14)\)\. Narratives carry recoverable severity signal: topic models with explainable XGBoost link topics to severity\([Li et al\., 2024](https://arxiv.org/html/2609.22694#bib.bib24)\), term frequency inverse document frequency \(TF\-IDF\) features drive interpretable prediction\([Arteaga et al\., 2020](https://arxiv.org/html/2609.22694#bib.bib10)\), semanticnn\-gram features classify hazardous actions\([Kwayu et al\., 2020](https://arxiv.org/html/2609.22694#bib.bib15)\), rule based tagging extracts structured fields\([Lai et al\., 2026](https://arxiv.org/html/2609.22694#bib.bib16)\), and multimodal fusion combines text with records and geometry\([Liu et al\., 2025](https://arxiv.org/html/2609.22694#bib.bib17)\)\(Table[1](https://arxiv.org/html/2609.22694#S2.T1)\)\.

Table 1:Representative dementia, older\-driver, and crash\-narrative studies for severity analysis\.
### 2\.3Leakage Control and Decision\-Aware Evaluation

Outcome leakage means training on signals that are not available at the decision time, and crash severity models rarely control it\. Text\-mining recoding studies recover misclassified work\-zone crashes\([Sayed et al\., 2021](https://arxiv.org/html/2609.22694#bib.bib20)\), wrong\-way crashes\([Hosseini et al\., 2023](https://arxiv.org/html/2609.22694#bib.bib13)\), and secondary crashes\([Zhang et al\., 2020](https://arxiv.org/html/2609.22694#bib.bib21);[Zheng et al\., 2015](https://arxiv.org/html/2609.22694#bib.bib22)\), showing that reports can differ from events\. Narrative leakage may involve explicit outcome terms as well as less direct documentation patterns that can act as severity proxies when recorded after crash outcomes or response activities are known\. The masking procedure targets identified outcome\-revealing terms and corpus\-derived phrases; however, the resulting inputs are treated as leakage\-controlled rather than assumed to be free of all potential outcome proxies\. Decision\-aware triage combines calibrated uncertainty, selective review queues, and asymmetric costs\([Guo et al\., 2017](https://arxiv.org/html/2609.22694#bib.bib7);[Geifman and El\-Yaniv, 2017](https://arxiv.org/html/2609.22694#bib.bib6);[Elkan, 2001](https://arxiv.org/html/2609.22694#bib.bib5);[Miller et al\., 2024](https://arxiv.org/html/2609.22694#bib.bib19)\); human\-AI use must also consider automation over\-reliance and reviewer workload\([Parasuraman and Riley, 1997](https://arxiv.org/html/2609.22694#bib.bib1);[Bansal et al\., 2021](https://arxiv.org/html/2609.22694#bib.bib2)\), while public\-sector deployment introduces accountability obligations\([Busuioc, 2021](https://arxiv.org/html/2609.22694#bib.bib3)\)\.

### 2\.4Research Gap

Within the literature reviewed here, dementia\-driving studies largely emphasize crash risk, while crash\-narrative severity studies primarily evaluate the predictive value of text\. Evidence appears more limited on how multimodal severity performance changes when identified post\-outcome narrative cues are masked\. The present study examines this question through paired full and leakage\-controlled evaluations of narrative and fusion models, followed by calibration and selective deferral using the leakage\-controlled predictions\.

## 3Data and Experimental Design

The study used Texas Department of Transportation Crash Records Information System \(CRIS\) data from 2017–2025, comprising approximately five million crash records with structured variables and police\-reported narratives\. Because dementia involvement is not directly coded in CRIS, candidate cases were identified through a high\-recall search using predefined dementia\-related terms, followed by manual contextual review of the retrieved narratives\. Cases were retained only when the narrative provided sufficient evidence that dementia or a closely related cognitive condition involved a crash participant; incidental or ambiguous mentions were excluded\. The final analytical dataset \(4,781 crashes\) was then constructed from the associated structured crash, roadway, vehicle, and person\-level variables, with missing\-data treatment and variable binning applied consistently according to predefined rules\. To control information leakage, terms and phrases that directly revealed dementia status or served as close proxies for the target label were excluded from model inputs\.

Dementia status is treated as crash\-record\-reported information and should not be interpreted as a clinically verified diagnosis\. The 25 structured predictors cover roadway/environment \(speed limit, weather, light, roadway type, alignment, surface, class, part, relation, intersection relation, traffic control\), crash event \(first harmful event, collision manner, other factor, primary/secondary contributing factors\), time/area \(weekday/weekend, rural/urban\), vehicle \(model\-year category, body style\), and person factors \(age category, gender, ejection, restraint, airbag\)\. Crash ID, year, raw severity, and the target were excluded from predictors\. Crash ID, year, target, and narrative were required; other missing structured values were codedNot Reported, while suppliedOther/Unknowncategories were retained\. Bins were used without data\-dependent estimation: speed≤\\leq30/35–50/55–80 mph, vehicle year 1950–1990/1991–2005/2006–2015/2016–2026, age 15–30/30–50/51–65/≥\\geq65, weekday/weekend, and rural/urban; other categorical groupings were retained as supplied\. The fixed stratified 70/15/15 split yielded 3,346/717/718 train/validation/test records, including 73 KA test cases\. Figure[2](https://arxiv.org/html/2609.22694#S3.F2)shows the annual severity distribution\.

Figure 2:Annual severity distribution for dementia\-related crash records\.The stratified split provides a common class distribution for model\-family comparison\. For audit, the retained artifact manifest includes the run configuration, data snapshots, split indices, trained models, Hugging Face logits, local LLM prompts and responses, validation and test predictions, all\-class metrics, calibration outputs, deferral tables, figures, and run logs\.

## 4Methods

The full setting excludes the target but otherwise retains the modeling inputs; the leakage\-controlled setting additionally masks outcome\-revealing narrative language\. Direct injury\-count or severity fields are absent from the structured predictors, so the intervention primarily affects narratives\. The mask combines 22 prespecified seed expressions with at most 75 corpus\-derived unigrams, bigrams, or trigrams having document frequency≥5\\geq 5, a severity/response root, and absolute smoothed one\-vs\-rest log odds≥2\.5\\geq 2\.5for O, BC, or KA\. Terms are ordered longest\-first and replaced case\-insensitively at token boundaries with\[MASK\_SEVERITY\]\. Figure[5](https://arxiv.org/html/2609.22694#S5.F5)shows representative corpus\-derived phrases\.

Structured predictors were treated as categorical: logistic models used one\-hot encoding and CatBoost received categorical variables directly\. Narratives were lowercased and represented by English\-stop\-word\-filtered TF\-IDF unigrams/bigrams \(min\_df=3; maximum 10,000 features\), with class weighting for structured and narrative logistic models\. Early fusion concatenated one\-hot and TF\-IDF features, while late fusion averaged validation\-tuned modality probabilities and calibrated late fusion applied temperature scaling\. The structured predictors are crash\-report fields, whereas the investigator narrative comes from the completed report and may contain hospital, transport, injury, or other post\-outcome documentation, motivating the leakage\-masking procedure\. Table[2](https://arxiv.org/html/2609.22694#S4.T2)summarizes the remaining model designs, and split indices, probabilities, prompts, responses, and logs are retained for audit\.

Table 2:Model families and audit design\.Note\.DistilBERT usesdistilbert\-base\-uncased; BERT\-Small usesgoogle/bert\_uncased\_L\-2\_H\-128\_A\-2; Gemma and Qwen use Ollama tagsgemma4:12b\-mlxandqwen3\.5:9b\-mlx\. Seeds, prompts, responses, probabilities, and runtime settings are retained\.

Two decision rules are evaluated\. The standard rule uses the highest predicted probability; the cost\-sensitive rule minimizes expected severity cost\. The author\-defined cost matrix encodes directional triage asymmetry rather than empirically estimated utilities: correct predictions have cost 0, O–BC confusion cost 1, O/BC→\\rightarrowKA cost 2, KA→\\rightarrowBC cost 3, and KA→\\rightarrowO cost 5\. Cost is therefore interpreted as a relative evaluation measure rather than a real\-world monetary or clinical cost\. Metrics include accuracy, macro\-F1, balanced accuracy, KA precision, KA recall, KA PR\-AUC, KA under\-triage, severity cost, Brier score, and expected calibration error\. Uncertainty on the fixed test set is estimated using 2,000 class\-stratified bootstrap resamples; the 2\.5th and 97\.5th percentiles form 95% confidence intervals, and paired model differences use common resample indices\. For selective deferral, records are ranked by calibrated confidence\. The system is evaluated at 100%, 90%, 80%, and 70% coverage, where lower coverage withholds a larger share of low\-confidence cases from automatic classification for potential human review\. Metrics below 100% coverage are calculated only on retained automatic predictions, and no outcomes are assigned to deferred cases\.

## 5Results

Table[3](https://arxiv.org/html/2609.22694#S5.T3)reports the stratified test results\. The majority baseline reaches 0\.532 accuracy but zero KA recall, confirming that accuracy is inadequate under class imbalance\. Structured CatBoost and logistic regression each reach KA recall of 0\.562 but incur higher severity costs through over\-escalation, indicating useful screening signal but limited specificity\.

Table 3:Main stratified test results\.In the reported split, leakage\-controlled Gemma has the highest observed macro\-F1, 0\.545 \[0\.507, 0\.583\], with balanced accuracy 0\.594 \[0\.548, 0\.638\], KA recall 0\.630 \[0\.521, 0\.740\], and KA PR\-AUC 0\.308 \[0\.242, 0\.390\], where brackets denote 95% bootstrap confidence intervals\. Early fusion obtains balanced accuracy 0\.547 \[0\.503, 0\.593\] and KA PR\-AUC 0\.285 \[0\.225, 0\.391\], while calibrated late fusion obtains 0\.539 \[0\.497, 0\.583\] and 0\.272 \[0\.213, 0\.358\], respectively\. Relative to early fusion, Gemma’s paired differences are 0\.022 \[\-0\.026, 0\.070\] for macro\-F1, 0\.047 \[\-0\.012, 0\.102\] for balanced accuracy, 0\.023 \[\-0\.088, 0\.110\] for KA PR\-AUC, and \-0\.019 \[\-0\.102, 0\.065\] for cost; these intervals include zero\. The KA\-recall difference is 0\.192 \[0\.041, 0\.329\]\. Thus, Gemma obtains the highest observed macro\-F1 in this split, but the results do not establish general model superiority\.

Figure 3:Leakage\-controlled model performance on the stratified test set\.Table[4](https://arxiv.org/html/2609.22694#S5.T4)and Figure[4](https://arxiv.org/html/2609.22694#S5.F4)show the effects of leakage masking\. Narrative logistic regression drops from 0\.516 to 0\.502 macro\-F1, with KA recall falling from 0\.329 to 0\.274\. Early and late fusion decline from 0\.533 to 0\.522 and 0\.539 to 0\.522, respectively\. These results show that outcome\-revealing terms aid prediction, especially for text\-only models, while the remaining performance suggests that narratives retain useful crash\-sequence information beyond leakage\. Figure[5](https://arxiv.org/html/2609.22694#S5.F5)further identifies severity words and multiword patterns such as ‘were transported,” ‘university hospital,” and “for possible injuries,” which seed\-only masking would miss\.

Table 4:Leakage audit for models evaluated in both full and leakage\-controlled settings\.Figure 4:Leakage sensitivity for models evaluated in both full and masked settings\.Figure 5:High\-support dataset\-derived leakage phrases discovered from crash narratives\.Figure[6](https://arxiv.org/html/2609.22694#S5.F6)highlights the main failure mode of the strongest leakage\-controlled classifier\. Gemma under\-triages 27 of 73 KA records while also over\-predicting KA for many non\-KA cases\. Its advantage therefore comes from a more severe\-case\-sensitive operating point rather than uniform improvement across classes, making it better suited for screening than fully automated severity classification\. Early fusion logistic regression and late fusion agree on 83\.3% of predictions, but their mean absolute probability distance is 0\.084, indicating that late fusion changes confidence more than labels\. Overall, Gemma is more aggressive in identifying KA cases, whereas calibrated late fusion provides more reliable probabilities and is better suited for selective deferral\.

Figure 6:Leakage\-controlled confusion matrix for Gemma, the strongest macro\-F1 model\.Calibration affects how the model can be used, even when it does not change the predicted class labels\. Late fusion and calibrated late fusion have identical classification metrics because temperature scaling preserves the class ranking\. However, in a leakage\-controlled setting, calibration reduces expected calibration error from 0\.089 to 0\.033 and Brier score from 0\.535 to 0\.520\. Figure[7](https://arxiv.org/html/2609.22694#S5.F7)shows the corresponding reliability curve and confidence distribution\. The main value of calibration is therefore operational and it makes confidence more reliable for deciding which records can be classified automatically, which records should be deferred for human review\.

Figure 7:Confidence\-band audit for calibrated late fusion\. Panel A splits test records into five equal\-sized confidence bands and shows correct versus incorrect predictions, with annotations giving band accuracy\. Panel B uses the same confidence bands to show true KA records split into detected and missed severe cases\.Table[5](https://arxiv.org/html/2609.22694#S5.T5)and Figure[8](https://arxiv.org/html/2609.22694#S5.F8)report selective deferral for calibrated late fusion\. At full coverage, the model classifies all 718 test records, with macro\-F1 of 0\.522, KA recall of 0\.384, KA under\-triage of 0\.616, and severity cost of 0\.675\. At 80% coverage, 144 records, including 16 KA records, are withheld from automatic classification, while macro\-F1 among retained automatic predictions rises to 0\.560 and severity cost falls to 0\.601\. At 70% coverage, retained\-case macro\-F1 rises to 0\.573 and severity cost falls to 0\.577\. KA recall among retained records changes only modestly because many deferred records are severe or ambiguous\. The risk\-coverage pattern shows that calibrated confidence identifies low\-confidence records, including severe or ambiguous cases, for potential review while improving performance on the retained automatic subset\.

Table 5:Selective deferral operating points for calibrated late fusion\.Note\. Metrics below 100% coverage are calculated on retained automatic predictions; deferred cases are not further evaluated and represent candidates for potential human review\.

Figure 8:Selective deferral operating points for calibrated late fusion\. Panel A shows macro\-F1 gain and severity\-cost reduction relative to full automation as more low\-confidence records are routed to review\. Panel B uses the same review levels to show the deferred queue size and the number of true KA records in that queue\.
## 6Behavioral Audit

Aggregate metrics summarize model performance but not boundary\-case behavior\. Table[6](https://arxiv.org/html/2609.22694#S6.T6)reports five saved\-prediction audits\. Early and late fusion agree on 83\.3% of leakage\-controlled test labels, which explains their similar headline metrics\. However, their probability estimates differ, with a mean absolute distance of 0\.084 and a maximum class\-probability change of 0\.398\. Thus, late fusion mainly changes boundary\-case uncertainty, slightly improving accuracy, while early fusion preserves stronger KA sensitivity\.

Treating the author\-defined matrix as an illustrative policy choice, sensitivity analysis compares lower\-asymmetry \(KA→\\rightarrowBC/O: 2/3\), baseline \(3/5\), and higher\-severe\-miss \(5/8\) scenarios\. Cost\-rule macro\-F1/KA recall/cost are 0\.542/0\.356/0\.529, 0\.516/0\.479/0\.671, and 0\.473/0\.630/0\.896, respectively, changing cost versus argmax by−11\.2%\-11\.2\\%,−0\.6%\-0\.6\\%, and\+9\.5%\+9\.5\\%\. In contrast, at 80% coverage, the same 144\-case deferral queue lowers retained\-case cost under all three scenarios to 0\.523 \(−12\.3%\-12\.3\\%\), 0\.601 \(−11\.0%\-11\.0\\%\), and 0\.742 \(−9\.2%\-9\.2\\%\)\. Thus, direct cost\-sensitive prediction is policy\-sensitive, whereas the observed deferral benefit is directionally stable across these illustrative scenarios\. Confidence better supports deferral, as correct calibrated\-fusion predictions have a mean confidence of 0\.622, compared with 0\.562 for errors, producing the risk\-coverage pattern in Figure[8](https://arxiv.org/html/2609.22694#S5.F8)\. Deferred records include 7 KA cases at 90% coverage, 16 at 80%, and 25 at 70%\. Figure[9](https://arxiv.org/html/2609.22694#S6.F9)shows that the system isolates an uncertainty region containing many BC records and a substantial fraction of KA records, rather than simply discarding low\-probability O cases\.

Table 6:Behavioral audits from saved test predictions\.Figure 9:Predicted probability and confidence distributions for calibrated late fusion\. Panel A shows macro\-F1 gain and severity\-cost reduction among retained automatic predictions as more low\-confidence records are withheld from automatic classification\. Panel B shows class\-specific confidence ECDFs used to construct the selective deferral rule\.The audit also clarifies the role of leakage control\. If masking had removed all useful narrative signal, narrative and fusion models would have approached structured\-only performance\. Instead, they decline modestly while retaining useful multimodal signal, indicating that police narratives contain both legitimate crash\-sequence information and outcome\-revealing language\. Figure[2](https://arxiv.org/html/2609.22694#S3.F2)adds context, since reporting practices and crash composition may vary over time\. Figure[5](https://arxiv.org/html/2609.22694#S5.F5)further shows that leakage extends beyond obvious severity words to ambiguous hospital\- and transport\-related phrases\. Conservative masking is therefore appropriate because the goal is to evaluate information available for triage, not language that merely reproduces final severity labels after the outcome is known\.

## 7Discussion

The results support three main findings\. First, accuracy alone is not enough for dementia\-related crash severity prediction\. The majority baseline reaches 0\.532 accuracy but fails on KA recall, while structured models detect more KA cases but over\-escalate many O and BC records\. The key question is whether a model can reduce severe under\-triage without creating an impractical review burden\. Second, multimodal modeling remains useful after leakage control, but its benefit is conditional\. Narrative masking reduces text\-based performance, indicating that part of the predictive signal comes from outcome\-revealing documentation\. However, performance does not collapse entirely, suggesting that the narratives also contain meaningful crash\-sequence information beyond explicit severity terms\. This matters because a triage model that mainly reads post\-crash outcome language is less defensible for public\-sector use\.

Third, the models serve different roles\. In the reported split, Gemma has the highest observed leakage\-controlled macro\-F1 and higher KA recall than early fusion, but paired bootstrap intervals do not establish general superiority across models\. Qwen has higher KA recall, but its lower macro\-F1 and higher severity cost show excessive escalation\. Calibrated late fusion is the better deferral model because its confidence estimates are more reliable, so severe\-case screening and deferral need not use the same model\. Deferral uses calibrated uncertainty to identify cases that could be considered for human review; reviewer accuracy, workload, disagreement, response time, and post\-review outcomes are not evaluated in the present study\. At 90% coverage, 72 of 718 records are withheld from automatic classification and retained\-case severity cost decreases from 0\.675 to 0\.635\. At 80%, 144 records are withheld and retained\-case severity cost is 0\.601\. At 70%, retained\-case severity cost is 0\.577 while 215 records are withheld from automatic classification\. These operating points characterize the tradeoff between retained\-set performance and the volume of cases that could be considered for review; they do not estimate end\-to\-end under\-triage or operational cost after human review\.

The remaining errors mainly reflect the BC–KA boundary\. KA records are most concerning when predicted as BC or O, especially when masked narratives have limited severe\-injury evidence\. Some BC records also receive high KA probability because response language can resemble severe\-outcome documentation even after masking\. Future error analysis should stratify saved predictions by crash configuration, roadway class, harmful event, lighting, narrative length, and year to test label ambiguity, missing covariates, reporting variation, and crash\-mechanism overlap\. Key limits are that dementia status is crash\-record reported, severity labels reflect local reporting, leakage terms are dataset\-specific, transformer baselines used a small imbalanced dataset, local LLM results depend on deterministic prompting and model availability, and the deferral rule uses confidence alone\. Future work should combine confidence, expected severity cost, class\-specific uncertainty, and analyst capacity in deferral policies\. Model rankings are based on the single stratified 70/15/15 split used in the reported experiment and are presented without confidence intervals or repeated\-split uncertainty estimates; they should therefore be interpreted as conditional on this evaluation split\.

## 8Conclusion

This study evaluates dementia\-related crash severity prediction as a leakage\-controlled and uncertainty\-aware triage problem\. The framework combines structured crash fields with police narratives and masks severity\-revealing terms within the narratives\. It evaluates structured, narrative, fusion, calibrated\-fusion, BERT\-family, and local prompt\-based models\. The framework also preserves raw predictions, logs, prompts, responses, calibration outputs, and deferral results for auditing purposes\. In the reported split, leakage\-controlled Gemma has the highest observed macro\-F1 \(0\.545; 95% CI \[0\.507, 0\.583\]\) and KA recall of 0\.630 \[0\.521, 0\.740\], without establishing general model superiority\. Calibrated late fusion is more useful for selective deferral because calibration reduces expected calibration error from 0\.089 to 0\.033 while preserving argmax predictions\. As coverage decreases, selective deferral lowers expected severity cost among retained automatic predictions while identifying uncertain cases that could be considered for human review\. The main contribution is an auditable decision framework in which crash severity models classify high\-confidence cases, identify uncertain cases for potential human review, and make each reported model result traceable to saved predictions, splits, logs, prompts, responses, and configuration\.

The study relies on available crash records and police narratives, so model performance may still reflect reporting quality, missing details, and jurisdiction\-specific wording patterns\. The dementia\-related crash subset is relatively specialized, which may limit generalizability to broader older\-driver or medical\-condition crash populations\. Future research should validate the framework across additional states, years, and crash\-reporting systems and evaluate human reviewer performance and end\-to\-end reviewer\-in\-the\-loop triage\.

## References

- Arteagaet al\.\(2020\)C\. Arteaga, A\. Paz, and J\. ParkInjury severity on traffic crashes: a text mining with an interpretable machine\-learning approach\.Safety Science132,pp\. 104988\.External Links:[Document](https://dx.doi.org/10.1016/j.ssci.2020.104988)Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.6.1.1.1)\.
- Bansalet al\.\(2021\)G\. Bansal, T\. Wu, J\. Zhou, R\. Fok, B\. Nushi, E\. Kamar, M\. T\. Ribeiro, and D\. S\. WeldDoes the whole exceed its parts? the effect of AI explanations on complementary team performance\.InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems,pp\. 1–16\.External Links:[Document](https://dx.doi.org/10.1145/3411764.3445717)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Busuioc \(2021\)M\. BusuiocAccountable artificial intelligence: holding algorithms to account\.Public Administration Review81\(5\),pp\. 825–836\.External Links:[Document](https://dx.doi.org/10.1111/puar.13293)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Carret al\.\(2000\)D\. B\. Carr, J\. Duchek, and J\. MorrisCharacteristics of motor vehicle crashes of drivers with dementia of the alzheimer type\.Journal of the American Geriatrics Society48\(1\),pp\. 18–22\.External Links:[Document](https://dx.doi.org/10.1111/j.1532-5415.2000.tb03023.x)Cited by:[§2\.1](https://arxiv.org/html/2609.22694#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.2.1.1.1)\.
- Das \(2022\)S\. DasArtificial intelligence in highway safety\.CRC Press\.Cited by:[§1](https://arxiv.org/html/2609.22694#S1.p2.1)\.
- Donoughe and Atkinson \(2015\)K\. Donoughe and J\. AtkinsonFacilitating work zone safety improvements through detailed crash report narratives\.Technical reportTransportation Research Board\.Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1)\.
- Dzinyelaet al\.\(2023\)R\. Dzinyela, E\. K\. Adanu, D\. Lord, and S\. IslamAnalysis of factors that influence injury severity of single and multivehicle crashes involving at\-fault older drivers: a random parameters logit with heterogeneity in means and variances approach\.Transportation Research Interdisciplinary Perspectives22,pp\. 100974\.External Links:[Document](https://dx.doi.org/10.1016/j.trip.2023.100974)Cited by:[§2\.1](https://arxiv.org/html/2609.22694#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.4.1.1.1)\.
- Elkan \(2001\)C\. ElkanThe foundations of cost\-sensitive learning\.InInternational joint conference on artificial intelligence,Vol\.17,pp\. 973–978\.Cited by:[§1](https://arxiv.org/html/2609.22694#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Geifman and El\-Yaniv \(2017\)Y\. Geifman and R\. El\-YanivSelective classification for deep neural networks\.Vol\.30\.Cited by:[§1](https://arxiv.org/html/2609.22694#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InInternational Conference on Machine Learning,pp\. 1321–1330\.Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Hossainet al\.\(2026\)A\. Hossain, X\. Sun, S\. Das, M\. Jafari, and J\. CodjoeInvestigating older driver crashes on high\-speed roadway segments: a hybrid approach with extreme gradient boosting and random parameter model\.Transportmetrica A: Transport Science22,pp\. 2362362\.External Links:[Document](https://dx.doi.org/10.1080/23249935.2024.2362362)Cited by:[§2\.1](https://arxiv.org/html/2609.22694#S2.SS1.p1.1)\.
- Hosseiniet al\.\(2023\)P\. Hosseini, S\. Khoshsirat, M\. Jalayer, S\. Das, and H\. ZhouApplication of text mining techniques to identify actual wrong\-way driving crashes in police reports\.International Journal of Transportation Science and Technology12,pp\. 1038–1051\.External Links:[Document](https://dx.doi.org/10.1016/j.ijtst.2022.12.002)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Johnson and Bugeja \(2018\)M\. Johnson and L\. BugejaBefore the crash: a systematic analysis of police narratives to determine the prevalence of pre\-crash factors in fatality road transport crashes\.Traffic Injury Prevention19,pp\. S156–S157\.External Links:[Document](https://dx.doi.org/10.1080/15389588.2018.1532212)Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1)\.
- Kotsyubynskaet al\.\(2026\)Y\. Kotsyubynska, N\. Kozan, V\. Chadiuk, A\. Kostyshyn, A\. Kotsyubynsky, and V\. FentsykMachine learning and deep learning for predicting traffic crash injury severity: a systematic review and meta\-analysis \(2014\-2025\)\.Journal of Road Safety37\(1\)\.External Links:[Document](https://dx.doi.org/10.33492/JRS-D-26-1-2721386)Cited by:[§1](https://arxiv.org/html/2609.22694#S1.p2.1)\.
- Kwayuet al\.\(2020\)K\. M\. Kwayu, V\. Kwigizile, J\. Zhang, and J\. OhSemantic n\-gram feature analysis and machine learning–based classification of drivers’ hazardous actions at signal\-controlled intersections\.Journal of Computing in Civil Engineering34\(4\),pp\. 04020015\.External Links:[Document](https://dx.doi.org/10.1061/%28ASCE%29CP.1943-5487.0000895)Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.8.1.1.1)\.
- Laiet al\.\(2026\)J\. Lai, F\. Yang, H\. Li, J\. Zhang, Y\. Feng, T\. Han, and Y\. ChenAdvancing traffic safety analysis: a novel lightweight rule\-based and part\-of\-speech tagging\-based approach for information extraction from crash reports\.Journal of Computing in Civil Engineering40\(1\),pp\. 04025105\.External Links:[Document](https://dx.doi.org/10.1061/JCCEE5.CPENG-6634)Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.9.1.1.1)\.
- Liet al\.\(2024\)P\. Li, S\. Chen, L\. Yue, Y\. Xu, and D\. A\. NoyceAnalyzing relationships between latent topics in autonomous vehicle crash narratives and crash severity using natural language processing techniques and explainable xgboost\.Accident Analysis & Prevention203,pp\. 107605\.External Links:[Document](https://dx.doi.org/10.1016/j.aap.2024.107605)Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.5.1.1.1)\.
- Liuet al\.\(2025\)Y\. Liu, Z\. Gao, H\. Ge, Z\. Chen, G\. Liang, Y\. Wang, and Y\. ZhangA multimodal deep learning approach for predicting traffic accident severity using crash records, road geometry, and textual descriptions\.Computer\-Aided Civil and Infrastructure Engineering40\(23\),pp\. 3773–3793\.External Links:[Document](https://dx.doi.org/10.1111/mice.70023)Cited by:[§2\.2](https://arxiv.org/html/2609.22694#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.7.1.1.1)\.
- Maet al\.\(2025\)Y\. Ma, H\. Liu, Z\. Duan, J\. Liu, and D\. ChenA driving risk prediction method for elderly drivers considering data imbalance and feature extraction\.Transportation Safety and Environment7\(3\),pp\. tdaf037\.External Links:[Document](https://dx.doi.org/10.1093/tse/tdaf037)Cited by:[§2\.1](https://arxiv.org/html/2609.22694#S2.SS1.p1.1)\.
- Meulenerset al\.\(2016\)L\. B\. Meuleners, J\. Ng, K\. Chow, and M\. StevensonMotor vehicle crashes and dementia: a population\-based study\.Journal of the American Geriatrics Society64\(5\),pp\. 1039–1045\.External Links:[Document](https://dx.doi.org/10.1111/jgs.14109)Cited by:[§2\.1](https://arxiv.org/html/2609.22694#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.22694#S2.T1.2.3.1.1.1)\.
- Milleret al\.\(2024\)A\. Miller, D\. Datta, V\. Sundharam, A\. Sarkar, G\. Rooney, and C\. LobbAI and decision support systems for crash preventability par processing\.Technical reportVirginia Tech Transportation Institute and Federal Motor Carrier Safety Administration\.Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Parasuraman and Riley \(1997\)R\. Parasuraman and V\. RileyHumans and automation: use, misuse, disuse, abuse\.Human Factors39\(2\),pp\. 230–253\.External Links:[Document](https://dx.doi.org/10.1518/001872097778543886)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Sattaret al\.\(2023\)K\. Sattar, F\. Chikh Oughali, K\. Assi, N\. Ratrout, A\. Jamal, and S\. Masiur RahmanTransparent deep machine learning framework for predicting traffic crash severity\.Neural Computing and Applications35\(2\),pp\. 1535–1547\.External Links:[Document](https://dx.doi.org/10.1007/s00521-022-07769-2)Cited by:[§1](https://arxiv.org/html/2609.22694#S1.p2.1)\.
- Sayedet al\.\(2021\)M\. A\. Sayed, X\. Qin, R\. J\. Kate, D\. Anisuzzaman, and Z\. YuIdentification and analysis of misclassified work\-zone crashes using text mining techniques\.Accident Analysis & Prevention159,pp\. 106211\.External Links:[Document](https://dx.doi.org/10.1016/j.aap.2021.106211)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Zhanget al\.\(2020\)X\. Zhang, E\. Green, M\. Chen, and R\. R\. SouleyretteIdentifying secondary crashes using text mining techniques\.Journal of Transportation Safety & Security12\(10\),pp\. 1338–1358\.External Links:[Document](https://dx.doi.org/10.1080/19439962.2019.1597795)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.
- Zhenget al\.\(2015\)D\. Zheng, M\. V\. Chitturi, A\. R\. Bill, and D\. A\. NoyceAnalyses of multiyear statewide secondary crash data and automatic crash report reviewing\.Transportation research record2514\(1\),pp\. 117–128\.External Links:[Document](https://dx.doi.org/10.3141/2514-13)Cited by:[§2\.3](https://arxiv.org/html/2609.22694#S2.SS3.p1.1)\.