Traceable LLM-Generated Hazard Scenarios for Operational Safety Analysis of Aviation Systems Using ASRS Reports

# Traceable LLM-Generated Hazard Scenarios for Operational Safety Analysis of Aviation Systems Using ASRS Reports
Source: [https://arxiv.org/html/2608.04697](https://arxiv.org/html/2608.04697)
###### Abstract

Operational hazard analysis of aviation system operations must consider interactions among weather, ATC actions, airspace constraints, aircraft operations, and human factors \- distinct from the functional hazard assessment applied at the aircraft\-system level\.

We present an AI\-assisted approach that generates candidate hazard scenarios from NASA’s Aviation Safety Reporting System \(ASRS\)\. Given a target adverse outcome, it produces a structured hypothesis as categorical factors and a narrative scenario describing an operational event sequence consistent with the structure\. Each scenario includes by a plausibility score from historical co\-occurrence evidence and traceability to the most similar held\-out ASRS reports\. We then propose a hybrid variant, conditioning narrative generation on a structured hypothesis produced via evolutionary abduction, improving correctness and reducing variability\. We evaluate multiple large language models, zero\-shot versus few\-shot prompting, and optional fine\-tuning, measuring how prompting and model choice affect the validity and realism of the generated structures and narratives\.

## IIntroduction

Operational hazard analysis in avionics requires anticipating hazardous events that emerge from interacting technical, environmental, and human factors, such as weather, air traffic control actions, airspace structure, flight conditions, aircraft, and crew behavior\. We target operational hazard analysis – reasoning about hazardous combinations arising during system operations – rather than the aircraft\- or system\-level FHA prescribed for avionics certification; the two operate at different levels and are complementary\. Many high\-consequence events are rare, and the space of plausible combinations is combinatorial, which makes manual scenario elicitation challenging and vulnerable to oversight\. In addition, analysts must hypothesize accident scenarios that are not only credible, but also novel\. At the same time, the aviation domain provides a large collections of historical reports that encode patterns of how contributing factors co\-occur in real operations\. This work investigates how generative AI can be engineered into an analyst\-in\-the\-loop workflow to explore plausible hazard scenarios while providing evidence to support review\.

We present an AI\-assisted approach, named Hazards Generator \(HaGen\), that generates candidate aviation hazard scenarios grounded in NASA’s Aviation Safety Reporting System \(ASRS\)\. The user specifies a target adverse outcome \(aResultcategory\), and optional operational context constraints\. For instance, the user can queryHaGento generate a hazard scenario resulting inAircraft Damagedfor a Part 121 air carrier, under VMC with clear skies during the final approach phase\. The approach produces two coupled representations: \(i\) a structured hypothesis as a set of categorical factors \(e\.g\., flight conditions, weather, airspace, operator/aircraft type, human factors\), and \(ii\) a narrative scenario specifying an event sequence consistent with those factors –– hence describing how such factors might combine into a particular unsafe situation, under the structured hypothesis conditions\. To support auditability, each scenario is accompanied by a plausibility score derived from historical ASRS co\-occurrence evidence and by traceability links to the most similar held\-out reports\. The method is intended for analyst\-in\-the\-loop use: it supports exploration and prioritization of candidate scenarios and is not a predictor of event likelihood\. The basic variant uses large language models to generate both the structured factors and the narrative under a predefined schema and admissible value sets\. We study zero\-shot vs\. few\-shot prompting and optional parameter\-efficient fine\-tuning that trains models to map the input prompt to a valid structured\-and\-narrative output\. The evaluation uses a sampled subset of ASRS reports with a held\-out test set\. We measure validity as the fraction of generations that violate the required schema, omit required fields, select values outside admissible domains, or fail to match the requested outcome category\. We assess realism by comparing each generated scenario to the closest report in the test set, using Jaccard similarity over the structured factors and BARTScore\[[40](https://arxiv.org/html/2608.04697#bib.bib45)\]for the narrative\. Results show that prompting strategy and model choice materially affect validity and realism\. In our factorial comparison, switching from zero\-shot to few\-shot prompting improves average validity by approximately 52% relative to zero\-shot, whereas naive fine\-tuning degrades it\. The best\-performing configuration generates structured factor sets whose realism is statistically indistinguishable from that of dedicated structured\-only baselines based on evolutionary search and causal structure discovery, while additionally producing narratives aligned with the operational language of ASRS reports and providing traceability to source events\. To improve controllability, we further implementHaGen\+, which conditions narrative generation on a structured hypothesis produced by a state\-of\-the\-art method based on evolutionary abduction: by construction, this eliminates invalid structured outputs, and empirically reduces the standard deviation of narrative realism scores from 0\.12 to 0\.02 \(an 83% relative reduction\) compared toHaGen\. Overall, our results suggest that combining structured constraints, narrative generation, and traceability can make generative AI a practical tool for avionics hazard scenario elicitation, provided that outputs are treated as decision support and subjected to expert review\. Code and artifacts are openly available for reproducibility111https://doi\.org/10\.5281/zenodo\.15428106\.

The rest of the work is organized as follows\. We first present preliminary background \(Section[II](https://arxiv.org/html/2608.04697#S2)\), followed by related work \(Section[III](https://arxiv.org/html/2608.04697#S3)\)\.HaGenandHaGen\+are presented in Section[IV](https://arxiv.org/html/2608.04697#S4)\. Research questions and experiments are described in Section[V](https://arxiv.org/html/2608.04697#S5)\. Results are presented in Section[VI](https://arxiv.org/html/2608.04697#S6)\. Discussion about the regulatory positioning is presented in Section[VII](https://arxiv.org/html/2608.04697#S7)\. After a discussion of threats to the study validity \(Section[VIII](https://arxiv.org/html/2608.04697#S8)\), Section[IX](https://arxiv.org/html/2608.04697#S9)concludes the paper\.

## IIBackground

HaGenis designed to leverage the experience gathered over the years about occurred accidents, just like safety engineers would do\. We harness the ASRS platform\[[21](https://arxiv.org/html/2608.04697#bib.bib12)\], managed by NASA\. This platform is dedicated to collecting and analyzing voluntary and confidential reports from pilots, air traffic controllers, crew members, and other aviation industry operators\.

Accidents are stored in a cause\-effect style: events related to aircraft components, weather conditions, human personnel involved, the airport and many other causes and conditions recorded for each accident as a set of mostly categorical variables, along with the resulting accident \(also categorical\) and a narrative with a natural language description of the accident\. The main entities are:

- •Environment: information regarding the flight conditions when the accident occurred, visibility conditions, working environment factors such as lighting or temperature\.
- •Aircraft\-related elements, e\.g\., the flight plan, the route, the flight phase, the maintenance status, the mission\.
- •Component: information about all components of the aircraft and their status \(e\.g\., design problem, failed, malfunctioning\)\.
- •Person: information about the persons involved, such as the flight crew, the air traffic control, or people working in maintenance, with their experience and qualification; information about the human factors that could cause mistakes such as distraction, confusion, stress, etc\.
- •Events, describing anomalies such as airspace violation, deviation of altitude, procedural errors, airbone or ground conflict, fire, as well as the event describing the final result, such as the type of accident and its consequences\.

As an example, consider the following event:

Weather: IcingFlight Conditions: IMCLight: NightMission: Cargo/FreightFlight Phase: ClimbAircraft Component: FCC FailedAircraft Component: Navigational Equipment and Processing FailedHuman Factors: Communication BreakdownResult: Landed in Emergency Condition\{internallinenumbers\*\}Narrative: ON DOWNWIND LEG FOR APCH AT 7000 FT\\ WE GOT A LEADING EDGE SLAT DISAGREEMENT MESSAGE ON EICAS AND THE LEADING EDGE LIGHT ILLUMINATED\. I DECLARED AN EMER AND ASKED TO HAVE THE FIRE TRUCKS MEET US\. WE RAN THE LEADING EDGE SLAT DISAGREEMENT CHKLIST IN THE QRH\\ NORMAL CHKLISTS\\ BRIEFED FLT ATTENDANTS AND MADE PA TO PAX AND LANDED\. THERE WERE 2 PREVIOUS WRITE\-UPS IN THE MAINT LOG FOR THIS SAME SCENARIO\.

This describes an incident in which the aircraft encountered a severe malfunction \(the Flight Control Computer failed, resulting in the loss of both the Flight Director Control Unit and the display units\)\. The incident occurred under Instrument Meteorological Conditions with prevalent icing conditions, significantly impairing visibility and navigational capabilities\.

The pilot observed an inability to reduce power, with the aircraft gaining speed despite the attempts\. Communication with departure ATC for priority handling was hampered by confusion, worsening the situation\. During the ascent the pilot regained visual contact with the runway, requested a 360\-degree manoeuvre, and safely landed on the departing runway

This is an example of how multiple co\-causes can contribute to an occurrence/incident\. Here the co\-occurring contributing factors are: icing, IMC, night lighting, the FCC failure with consequent loss of navigation/processing, and a communication breakdown \- jointly producing the emergency landing\.

HaGenis called to hypothesize accidents in this style, as a plausible combination of co\-occurring causes along with a textual description precisely characterizing the accident\. The accident has therefore astructuredpart \(the categorical variables describing its causes\) and anunstructuredpart \(a detailednarrativeand a shortcallback\)\.

## IIIRelated work

The aviation sector has recently experienced significant technological advancements that have transformed Aviation Safety Assessment \(ASA\)\. We first review traditional techniques for ASA, and then explore recent advancements attempting to automate part of the process\. Because our target is operational hazard analysis, we survey general risk\-analysis techniques used across aviation operations\. At the aircraft\-system level, certification practice \(ARP4761A\) instead centers on the Functional Hazard Assessment \(FHA\), which identifies system functions, associated failure conditions, and severity classifications; we describe it here as the system\-level counterpart to the operational analysis we address\.

### III\-ATraditional risk analysis techniques

Over the years, various structured methodologies have been developed to identify, evaluate, and mitigate risks in diverse domains, all conceived to support the analyst, acknowledging the critical role of human factors in system safety\. Traditional risk analysis techniques can be categorized into system\-focused approaches and human error analyses\. Each method is tailored to specific aspects of risk, providing tools to anticipate failures, analyse their potential impacts, and implement preventative or corrective measures\.

System\-focused approaches

- •HAZOP \(HAZard and OPerability Study\)is a structured, brainstorming\-based methodology for identifying potential risks and operational issues\. It examines the system as a collection of interconnected nodes, analysing operational parameters and deviations from expected behaviours\. HAZOP uses guidewords to stimulate discussions about potential risks\[[12](https://arxiv.org/html/2608.04697#bib.bib30)\]\.
- •FMECA \(Failure Modes, Effects, and Criticality Analysis\)is a bottom\-up approach that identifies and evaluates the failure modes of a product and their effects\. It helps establish actions to mitigate identified risks\[[15](https://arxiv.org/html/2608.04697#bib.bib31)\]\.
- •SFMEA \(Software Failure Modes and Effects Analysis\)is a software\-focused version of FMEA\. It analyses process flows and identifies areas for verification, validation, and testing\[[10](https://arxiv.org/html/2608.04697#bib.bib32)\]\.
- •CCA \(Common Cause Analysis\)identifies sequences of events caused by a common source \(e\.g\., human errors, manufacturing defects, external events\)\. It is crucial for preventing simultaneous failures from a single cause\[[25](https://arxiv.org/html/2608.04697#bib.bib33)\]\.
- •FTA \(Fault Tree Analysis\) and ETA \(Event Tree Analysis\)\. FTA evaluates combinations of events that could lead to risks or catastrophic consequences\[[37](https://arxiv.org/html/2608.04697#bib.bib34)\]\. ETA examines the consequences of an initial risk, describing pathways leading to potential outcomes, focusing on event progression\[[15](https://arxiv.org/html/2608.04697#bib.bib31)\]\.
- •Bow\-tie Analysis, also known as the butterfly model, links the causes of a hazardous event to its consequences\. It combines the visual frameworks of FTA and ETA into a single, comprehensive diagram\[[5](https://arxiv.org/html/2608.04697#bib.bib35)\]\.

Human error analyses

- •HEART \(Human Error Assessment and Reduction Technique\)quantifies the probability of human errors in specific tasks by considering ergonomic and environmental factors that affect performance\. It is useful for designing error\-reduction strategies\[[39](https://arxiv.org/html/2608.04697#bib.bib36)\]\.
- •HTA \(Hierarchical Task Analysis\)describes tasks by analysing operations necessary to achieve goals\. It focuses on user\-system interactions\[[13](https://arxiv.org/html/2608.04697#bib.bib37)\]\.
- •TRACER\-Liteanalyses cognitive and psychological errors, often used in fields like air traffic management\[[36](https://arxiv.org/html/2608.04697#bib.bib38)\]\.

These techniques provide valuable frameworks but have notable limitations: they depend on the team’s experience, and struggle with complex situations such as multiple simultaneous risks, as in ASRS, which can lead to severe accidents\.

Finally, they do not actively support the identification of new risks, but merely “guide” analysts in considering already known or hypothesized scenarios\[[33](https://arxiv.org/html/2608.04697#bib.bib39),[35](https://arxiv.org/html/2608.04697#bib.bib40),[21](https://arxiv.org/html/2608.04697#bib.bib12),[22](https://arxiv.org/html/2608.04697#bib.bib13)\]\. We use novel to mean a plausible combination of contributing factors not previously enumerated, rather than a previously\-unknown failure mode\.

### III\-BAutomatic generation of potential accidents

There exist some recent efforts to automate at least part of the task of a safety analyst\. Nouriet al\.designed a pipeline where LLMs play the role of the analyst in constructing the safety cases in the automotive domain\[[24](https://arxiv.org/html/2608.04697#bib.bib75),[23](https://arxiv.org/html/2608.04697#bib.bib76)\]\. While the overall process is supported, no means is provided for the actual hazard identification phase\. Pietrantuono first applied evolutionary algorithms to generate the structured set of discrete variables describing an accident in the ASRS report\[[29](https://arxiv.org/html/2608.04697#bib.bib67)\]\. He formulated the problem as acombinatorial causal optimization problem\(CCOP\), where theresultvariable is the final effect and all others are potential causes\.

The same author later devised a new evolutionary algorithm inspired by abductive reasoning, calledEvolutionary Abduction\(EVA\), to hypothesize potential multiple causes for an accident starting from the desired result specified as effect\[[30](https://arxiv.org/html/2608.04697#bib.bib77)\]\. Besides evolutionary strategies,causality\-based strategieshave been proposed in\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\. These exploit Causal Structure Discovery algorithms \(like\[[1](https://arxiv.org/html/2608.04697#bib.bib69),[32](https://arxiv.org/html/2608.04697#bib.bib68),[26](https://arxiv.org/html/2608.04697#bib.bib70)\]\) to learn the causal relations between variables from the ASRS dataset, and then generate the events by selecting variables proportionally to their causal strength on the desired effect\.

All these automatic techniques can generate the combinatorial description \(structuredpart\) of the accident, but none generate a textual description of the accident\. This is not merely a matter of completeness , but it affects its precision\. In fact, any combination of the above categorical variables can actually correspond to multiple accidents, depending, among other, on the causal chain connecting the cause variables with each other and with the effect\. For instance, the combinations of causes of the event described in Section[II](https://arxiv.org/html/2608.04697#S2)can correspond to different actual accidents, depending, among other factors, on the causal chain of the involved variables as well as on aspects not captured by the discrete set of variables\. Our aim is to generate finer\-grained potential accidents by also generating the narrative, uniquely describing the accident and with a greater level of detail\. To evaluate the quality of the hypothesized solutions, we leverage the same criteria as in existing work\[[29](https://arxiv.org/html/2608.04697#bib.bib67),[30](https://arxiv.org/html/2608.04697#bib.bib77),[28](https://arxiv.org/html/2608.04697#bib.bib57)\], checking how close the newly generated accidents are to accidents that actually occurred\.

TABLE I:Description of considered variables

## IVTheHaGenandHaGen\+techniques

### IV\-AHaGen

HaGenuses the LLM to generate relevant hazardous events\. It produces both thestructuraldescription \(categorical\-variable combination describing potential co\-occurring causes for a given accident type\) and thenarrative\(textual description detailing how those causes combine into the accident\)\.

The key aspect is therefore theprompt design\. The generated accident is required to becorrect– namely, with the structural description well\-formatted and with valid values – namely, the narrative description should match with the combinatorial description \- andrealistic\. We adopt two well\-known techniques: Few\-Shot Learning \(FSL\) and Zero\-Shot Learning \(ZSL\)\.

The prompt is shown below:

Generate structured output in JSON format to generate an aircraft accident report based on the dataset columns described\. Column Name: column description Possible Values: \[V1V\_\{1\},V2V\_\{2\}, \.\.,VnV\_\{n\}\] Narrative: A description of a plane crash from the point of view of a human operator\. Callback: A technical statement provided by a human operator involved in an aircraft accident\. In this case, consider "Result": "Air Traffic Control Issued Advisory / Alert"\. Expected JSON structure but with the fields populated by values: \{ Col0: ’’, Col1: ’’, \.\.\., Narrative: ’’, Callback: ’’ \}

In the prompt, the LLM is given the list of all categorical variables to consider as potential causes, their possible values, and the accident type it is asked to generate \(result\)\.

The output is required to be formatted as JSON\. The LLM is asked to generate both the categorical values for each variable and the textual narrative description of the accident, with guidelines asking thatdescription should be based on plausible scenarios with precise technical details\. The variables considered are listed in Table[I](https://arxiv.org/html/2608.04697#S3.T1)\. These variables are fields of the ASRS report record\. From the full set we retain 13 operational/contextual fields and the Result; we exclude fields specific to a second aircraft \(to limit prompt length, multi\-aircraft occurrences being rare\) and free\-form administrative fields not used as causal factors\. The prompts have approximately a length of 6,000 characters\.

### IV\-BHaGen\+

Although LLMs can generate both parts of the description of realistic accidents, there is no way to guide the generation, due to their black\-box nature\. WithHaGen\+we propose to steer the generation by an LLM, by providing it, as prompt context, with the structured part of the accident generated by the EVA algorithm \(see Section[III](https://arxiv.org/html/2608.04697#S3)\)\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\.

EVA exploitsabductionas means to mimic human reasoning in deriving potential causes for a given desired effect \(the accidentresult\)\. It automatically assesses howrealisticthe event is compared to a background knowledge made up of historical occurred accidents – that is part of the ASRS dataset\. EVA computes aplausibilityfitness function, defined as the perceived potential for a hypothesized solution to occur based on past occurrences ofpartsof the hypothesis: a plausible solution is one which has been observed at least once \(partsmay be single elements, pairs, or, generally,nn\-tuples\)\. Plausibility is distinct from probability: it depends on whether a cause\-effect pattern was observed at least once in the knowledge base \(recognition of a precedent\), not on how frequently it occurred \(a relative\-frequency estimate\)\. Formally: given a knowledge baseK​BKBof historical accidents, thek\-degreeof a solutionx=\{c,t\} \-cbeing the set of potential causes andtthe resulting accident type \(result\) \- is defined as follows\.

###### Definition 1\(k\-degree of a solution\(δk\\delta\_\{k\}\)\)

The k\-degree ofx\(δk\\delta\_\{k\}\(x\)\) is the number of distinctkk\-tuples of the potential cause variables sets\(withk≤\|k\\leq\|c\|\|\) that occurred at least once inK​BKBalong with the accident typett\.

Based on this, plausibility is:

###### Definition 2\(Plausibility\)

Plausibilityπ\(\\pi\(x\) of a solutionx=\{c,tt\}withp=\|c\|p=\|\\textbf\{c\}\|is:

π​\(x\)=∑k=1pδk​\(x\)\(pk\)=∑k=1pδk​\(x\)2p−1\\pi\(\\textbf\{x\}\)=\\sum\_\{k=1\}^\{p\}\\frac\{\\delta\_\{k\}\(\\textbf\{x\}\)\}\{\\binom\{p\}\{k\}\}=\\frac\{\\sum\_\{k=1\}^\{p\}\\delta\_\{k\}\(\\textbf\{x\}\)\}\{2^\{p\}\-1\}\(1\)

π​\(x\)\\pi\(x\)is the sum overkkof thekk\-degreesδk​\(x\)\\delta\_\{k\}\(x\), each normalized by the number\(pk\)\\binom\{p\}\{k\}of possiblekk\-tuples at that level – i\.e\., the average fraction of possible cause\-combinations, of each size, that have a precedent in KB\. For a hypothesis withp=2p=2causes\{a,b\}\\\{a,b\\\}and resulttt, the possible tuples are\{a\}\\\{a\\\},\{b\}\\\{b\\\}\(k=1k=1\) and\{a,b\}\\\{a,b\\\}\(k=2k=2\)\. If\{a,t\}\\\{a,t\\\}and\{a,b,t\}\\\{a,b,t\\\}were each observed in KB but\{b,t\}\\\{b,t\\\}was not, thenδ1=1\\delta\_\{1\}=1andδ2=1\\delta\_\{2\}=1, givingπ​\(x\)=δ1\(21\)\+δ2\(22\)=12\+11=1\.5\.\\pi\(x\)=\\frac\{\\delta\_\{1\}\}\{\\binom\{2\}\{1\}\}\+\\frac\{\\delta\_\{2\}\}\{\\binom\{2\}\{2\}\}=\\frac\{1\}\{2\}\+\\frac\{1\}\{1\}=1\.5\.

Candidate cause combinations for a given accident type are evolved via an evolutionary algorithm with newly\-defined abductive operators that maximize a plausibility fitness function, while constraining hypotheses to differ from past events \(Jaccard distance\>0\.1\>0\.1by default\)\.

By combining EVA with the LLM, we expect to improve the realism of proposed accidents as well as the explainability of what proposed, as the analyst will know the accident plausibility score and how it is derived, thus understandingwhythat generated solution is preferred over others\.

## VEvaluation

### V\-AResearch questions

We address the following research questions:

- •RQ1 \- IsHaGenable to generate valid accidents? This question validatesHaGenwith respect to its main objective, that is to generate correct,valid, accident events\. A generated event is consideredinvalidif it violates the prescribed format \(JSON\), does not contain the requested category, or includes categories that were not specified\. We distinguish*structural validity*—schema conformance, presence of required fields, admissible values, and match to the requestedResult—measured byGeneration Failure Ratio\(GFR\), from*semantic validity*\(whether the scenario is operationally coherent\), which GFR does not capture\. Semantic plausibility is assessed via the realism metrics \(RQ2\)\. We testHaGenin 12 configurations:i\)using three LLMs,ii\)with or without fine\-tuning, and withiii\)two prompting strategies\.
- •RQ2 \- IsHaGenable to generate realistic accidents? To automatically assess the realism of an accident, we measure the similarity between the accidents generated byHaGenand real accidents already occurred in the past \(test set\), as in\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\. This assessment is based on a set of similarity metrics applied to the accident report’s structured and unstructured parts\. Note that this is a conservative evaluation\. Indeed, an event not closely matching any test\-set event may still be plausible and realistic, but our similarity\-based evaluation cannot objectively assess plausibility beyond the known dataset\. We again consider the same 12 configurations in this RQ, in order to evaluate the impact of each factor on the performance in generating realistic events\.
- •RQ3 \- How doesHaGenperform compared to evolutionary\-based and causality\-based generation? The goal is to compareHaGenwith state\-of\-the\-art techniques in the ability of generating realistic accidents\. Let us note that, unlikeHaGen, evolutionary and causal algorithms cannot generate an unstructured description of the accident; thus, this comparison checks ifHaGencan compete in generating thestructuredpart compared to strategies specifically conceived for that purpose\.
- •RQ4 \- Does the combination of EVA andHaGeninHaGen\+improve performance? To evaluate this, we compareHaGen\+toHaGenconsidering only the unstructured parts of the generated hazards\.

### V\-BDataset

We randomly extracted a subset of10,00010,000accidents from the ASRS dataset\. After excluding events with empty result, we got9,9649,964accidents, 20% of which \(1,9931,993\) are allocated for evaluation purposes astest set\. The remaining events \(training set\) are used to fine\-tune the LLMs as well as to provide them with context in the few\-shot prompt strategy\.

### V\-CDesign of experiments

1: ATC Issued Advisory / Alert
2: ATC Issued Advisory / Alert / ATC Issued New Clearance
3: General Maintenance Action
4: General None Reported / Taken
5: General Maintenance Action / General None Reported / Taken
6: General Flight Canceled / Delayed
7: General Declared Emergency
8: General Physical Injury / Incapacitation
9: Flight Crew Took Evasive Action
10: Flight Crew DivertedFigure 1:Type of generated accidentIn all RQs, the LLM as well as the baselines are prompted to generate a specific type of accident\. We used 10 different types, reported in Figure[1](https://arxiv.org/html/2608.04697#S5.F1)\.

For RQ1 and RQ2, we compare 12 configurations ofHaGen, differing by the LLM employed, the prompting strategies, and application of fine\-tuning\. The LLMs considered are:Llama\-3\.1\-8B\(Llama\)\[[18](https://arxiv.org/html/2608.04697#bib.bib59)\],Ministral\-8B\(Ministral\)\[[20](https://arxiv.org/html/2608.04697#bib.bib60)\]andPhi\-3\.5\(Phi\)\[[19](https://arxiv.org/html/2608.04697#bib.bib61)\]\. All these models have been quantized with AWQ\[[17](https://arxiv.org/html/2608.04697#bib.bib62)\]\. We evaluate two prompt strategies: Zero\-Shot Learning \(ZSL\), which provides only the event features, and Few\-Shot Learning \(3SL\), which additionally includes three past accidents of the sameresulttype, randomly drawn from the training set\. We fine\-tune the LLMs with QLoRA\[[2](https://arxiv.org/html/2608.04697#bib.bib63)\]on a dataset of 1,000 examples \(100 perresult\), using ZSL prompts paired with JSON\-formatted past events from the training set as targets\. For RQ3 we compareHaGenwith all baseline algorithms as in the work by Pietrantuono and Russo\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\.

Finally, for RQ4, we combineEVAandHaGento exploit the advantages of both\. While EVA exhibited the best performance against all baselines in the original paper\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\], we run a further analysis per hazard type \(resulttype\) to confirm those results on our dataset, then compare the combination \(HaGen\+\) against the LLM alone \(HaGen\)\.

The LLMtemperatureis set to 1\. To account for randomness, the 10 generations \(one perresulttype\) are repeated 20 times each, for every baseline andHaGenconfiguration, followed by a statistical analysis of the differences among the 12 configurations\.

### V\-DBaselines

In RQ3 we consider three evolutionary\-based algorithms, three causal strategies, and a random baseline\. As they generate only structured descriptions, the comparison refers solely to the structured part\. The techniques used by the evolutionary algorithms are:Evolutionary AbductionEVA,Evolutionary SearchES,Genetic AlgorithmGA\. All are implemented injMetal, a known framework for experimenting with metaheuristics\[[4](https://arxiv.org/html/2608.04697#bib.bib66)\]\.ESandGAuse the default implementation ofjMetal, varied by Pietrantuono and Russo to make them capable of dealing with the combinatorial causal optimization problem \(CCOP\)\[[29](https://arxiv.org/html/2608.04697#bib.bib67)\]defined for the generation of accident events\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\. In particular, the crossover and mutation operators are varied to deal with the variable length of the accidents to generate\. The default setting for the probability of crossover and mutation as provided byjMetal, that is: 0\.9 and 1/nn, respectively, withnnbeing the number of variables\. For selection, both algorithms adopt binary tournament\. Likewise, we consider the original implementation ofEVA, also building onjMetal\.EVAhas custom operators, called factual, analogical, and hypothetical\-cause operators instead of crossover and mutation\.EVA’s hyperparameters are set at their default values resulting from a sensitivity analysis, as in the original paper\.

The three causal strategies leverage Causal Structure Discovery \(CSD\) algorithms to learn causal relationships \(and their strengths\) between potential causes and accident type, and then generate accidents by selecting causes proportionally to their learned causal strengths\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\. The three compared techniques differ for the CSD algorithm used: FGES\[[32](https://arxiv.org/html/2608.04697#bib.bib68)\], a score\-based algorithm; RFCI\[[1](https://arxiv.org/html/2608.04697#bib.bib69)\], a constraint\-based algorithm; and GFCI\[[26](https://arxiv.org/html/2608.04697#bib.bib70)\], a hybrid one\. These CSD algorithms learn a causal model from training data, represented as a DAGG=\(x,E\)\\textbf\{G\}=\(\\textbf\{x\},\\textbf\{E\}\), wherex=\{xc1,…,xcj,xt\}\\textbf\{x\}=\\\{x\_\{c\_\{1\}\},\\dots,x\_\{c\_\{j\}\},x\_\{t\}\\\};xcix\_\{c\_\{i\}\}are potential causes andxtx\_\{t\}is the effect of interest, namely the accident type\[[27](https://arxiv.org/html/2608.04697#bib.bib71)\]\. Arcsei,te\_\{i,t\}link cause variablesxcix\_\{c\_\{i\}\}toxtx\_\{t\}, with weightswi,tw\_\{i,t\}estimated via bootstrapping, representing the probability thatxci→xtx\_\{c\_\{i\}\}\\to x\_\{t\}exists using theTetradtoolbox\[[31](https://arxiv.org/html/2608.04697#bib.bib78)\]\)\.

The generation with the graph proceeds as follows:\(i\)The numberkkof causes to generate is randomly selected \(1≤k≤n1\\leq k\\leq n\);\(ii\)A value for the accident type to generate,xtx\_\{t\}, is sampled from its empirical distribution over the training set;\(iii\)thekkvariables to include as potential causes are selected proportionally to the weightswi,tw\_\{i,t\};\(iv\)For each selectedxcix\_\{c\_\{i\}\}, a valueviv\_\{i\}is sampled according to its empirical distribution in the training set\. As in\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\], the parameters for the CSD algorithms are the default ones used by theTetradimplementation, except for the number of bootstraps \(i\.e\., number of resampling\) raised to 50 to improve the accuracy\.

For all the above techniques, the same number of evaluations is used as in the original paper, namely 6,000 evaluations, covering all the accident types \(result\) defined in Fig\.[1](https://arxiv.org/html/2608.04697#S5.F1), with 600 evaluations for each type\.

Finally, the random baseline \(RAN\) selects both cause variables and their values uniformly at random\.

### V\-EEvaluation metrics

For RQ1 we use theGeneration Failure Ratio, as proportion of invalid accidents out of total number of generated accidents\.

For the other RQs, we measure the similarity between the generated and already\-occurred events, namedtest set\. Specifically, for thestructured part, we compute theJaccard distance\. Taking an accident event as a setE=e1,e2,…,enE=\{e\_\{1\},e\_\{2\},\.\.\.,e\_\{n\}\}, theJaccard distancebetween the generated eventEgE\_\{g\}and the event in the test setEtE\_\{t\}is:d=1−Eg∩EtEg∪Etd=1\-\\frac\{E\_\{g\}\\cap E\_\{t\}\}\{E\_\{g\}\\cup E\_\{t\}\}; thus,d∈\[0;1\]d\\in\[0;1\]\. As we are interested in similarity, the lowerdd, the better\.

To measure the similarity between the unstructured parts,we useBartScore\[[40](https://arxiv.org/html/2608.04697#bib.bib45)\]\.BartScoreis an evaluation metric that uses a pre\-trained BART model to assess generated text by treating evaluation as a text generation problem\[[16](https://arxiv.org/html/2608.04697#bib.bib64)\]\. The authors offer a fine\-tuned version of BART trained on the ParaBank2 dataset\[[11](https://arxiv.org/html/2608.04697#bib.bib65)\], a comprehensive collection of paraphrases\. We employed this fine\-tuned model to evaluate the similarity between two texts, framing the task as a paraphrase evaluation\. To be considered realistic, the accident generated should be similar toat least one accidentin the test set, hence we take the minimum distanceddandBartScorebetween the generated accident and all the accidents in the test set\.

## VIResults

### VI\-ARQ1 \- Valid accidents generation

Table[II](https://arxiv.org/html/2608.04697#S6.T2)shows the average GFR considering all1010result types and2020repetitions for eachHaGenconfiguration\. Using fine\-tuning strongly increases the GFR for almost all LLMs and prompt strategies \(except forLlamawith 3SL\)\.

In particular, these configurations generate events characterized by aresultnot aligned with the requested one\. Augmenting the LLMs context with already\-occurred events \(3SL prompt strategy\) reduces the GFR in almost all cases\.

The employed LLM strongly impacts the GFR\. TheLlamaconfigurations exhibit an average GFR of0\.320\.32, in contrast to the other two LLMs, which both show a GFR value of0\.620\.62\.

Finally, the best configuration employs no\-finetunedLlamawith ZSL\.Llamawith 3SL also shows good performance, both with and without fine\-tuning, with failure rates of12%12\\%and13%13\\%, respectively\. Moreover, these failures are evenly distributed across the differentresults\(with a maximum of55failed generations for the sameresult, both with and without fine\-tuning\), indicating that these configurations do not struggle with any specificresult\.

TABLE II:RQ1 – Generation Failure Ratiovsconfigurations
### VI\-BRQ2 \- Realistic accidents generation

Tables[III](https://arxiv.org/html/2608.04697#S6.T3)and[IV](https://arxiv.org/html/2608.04697#S6.T4)report the results for RQ2, showing the best performance achieved on the structured and unstructured parts of the generated accident, respectively\. For the structured part, Table[III](https://arxiv.org/html/2608.04697#S6.T3)reports the smallest Jaccard distance obtained by comparing each generated accident with every accident in thetest set, averaged over the1010accident types and2020repetitions\. For the unstructured part, Table[IV](https://arxiv.org/html/2608.04697#S6.T4)shows the average highestBartScore, computed again by comparing each generated accident with thetest set\.

TABLE III:RQ2 –HaGenconfigurations comparison on structured event part \(Jaccard distance\)TABLE IV:RQ2 \-HaGenconfigurations comparison on unstructured event part \(Narrative / Callback BartScore\)For the structured part, bothMinistralandPhiexhibit consistently lower performance across all configurations, regardless of the use of fine\-tuning or prompt type\. In contrast,Llamademonstrates superior performance, in line with the findings of RQ1\. Notably, the use of the 3SL prompt type significantly enhances its effectiveness, in contrast to fine\-tuning, which does not impact performance, except for ZSL\. The best Jaccard distance is achieved by fine\-tunedLlamawith ZSL; however, the high performance is due to the low number of correct generations \(only55out of200200\)\. In particular, this configuration generates configurations with aresultnot aligned with the requested one\. We run a statistical analysis to evaluate the difference between configurations and the impact of each factor \(LLM, prompt strategy, and fine\-tuning\)\.

For fine\-tuning and prompt strategy factors, as they have two levels, we run the Wilcoxon rank sum test\[[38](https://arxiv.org/html/2608.04697#bib.bib72)\]\(α=0\.05\\alpha=0\.05\) – both factors significantly impact the Jaccard distance \(<\.0001<\.0001\) To assess the significance of the LLM factor, we run the Kruskal\-Wallis \(KW\) test\[[14](https://arxiv.org/html/2608.04697#bib.bib73)\]to check if there is any difference between the compared LLMs \(p=2\.02×10−09p=$2\.02\\text\{\\times\}\{10\}^\{\-09\}$\), followed by a post\-hoc test for pairwise comparison with multiple comparison protection, the Dunn’s test\[[3](https://arxiv.org/html/2608.04697#bib.bib74)\]\. The result shows thatLlamais better thanMinistralandPhi, with ap=1\.59×10−06p=$1\.59\\text\{\\times\}\{10\}^\{\-06\}$andp=1\.49×10−08p=$1\.49\\text\{\\times\}\{10\}^\{\-08\}$, respectively\.

For the unstructured part,Llamashows the best performance using the 3SL prompt type, both with and without fine\-tuning\. ForCallback,Ministralachieves the best results using ZSL with fine\-tuning\. Notably, the differences among configurations are less pronounced than for the structured part\.

As for the impact of factors onBartScorefor theNarrativedescription, both fine\-tuning and prompt strategy significantly impactBartScore\(p=9\.54×10−06p=$9\.54\\text\{\\times\}\{10\}^\{\-06\}$andp=1\.91×10−06p=$1\.91\\text\{\\times\}\{10\}^\{\-06\}$, respectively\)\. The LLM factor is significant too \(p=1\.24×10−05p=$1\.24\\text\{\\times\}\{10\}^\{\-05\}$, KW test\), withPhibeing significantly different fromLlama\(p=1\.58×10−05p=$1\.58\\text\{\\times\}\{10\}^\{\-05\}$\) andMinistral\(p=0\.002p=0\.002\), Dunn’s test\. Finally, for theCallback, the prompt strategy impactsBartScore\(p=3\.82×10−06p=$3\.82\\text\{\\times\}\{10\}^\{\-06\}$\), while fine\-tuning does not\. The LLM impacts \(p=0\.00029p=0\.00029, KW test\) withPhisignificantly outperformingLlama\(p=0\.00033p=0\.00033\) but with worse performance thanMinistral\(p=0\.00949p=0\.00949\)\.

### VI\-CRQ3 \-HaGenperformance

Based on RQ1 and RQ2, we select the bestHaGenconfiguration and compare it with the baseline techniques\.Llamawith the3SLprompt and without fine\-tuning turned out to be the best, due to its strong performance in GFR and in generating both the structured and unstructured part\.

Henceforth, with “HaGen” we refer to this configuration in the following\. The comparison is only on thestructuredpart, as the baselines do not generate theunstructureddescription\.

Figure[2](https://arxiv.org/html/2608.04697#S6.F2)shows the comparison ofHaGenwith the baselines in terms of average Jaccard distance\. Like in RQ2, we take into account the smallest Jaccard distance obtained by comparing each generated event with every events in thetest set\. Thus, forHaGenwe average that value over the1010accident types and2020repetitions\. UnlikeHaGen, which generates one event perresult, other algorithms generate several events\. For the evolutionary algorithms , we consider the average Jaccard distance of the last generation, as evolutionary algorithms aim to enhance the quality of generations step by step, with the last generation expected to be the best one\. For the causal algorithms andRAN, all the generated events are independent\. As in\[[28](https://arxiv.org/html/2608.04697#bib.bib57)\], they are organized into generations for comparison with evolutionary algorithms\. In this case, we consider the best generation, the one with the best average Jaccard distance\.

![Refer to caption](https://arxiv.org/html/2608.04697v1/x1.png)Figure 2:RQ3 \- Average Jaccard distanceThe Kruskal\-Wallis test \(α=0\.05\\alpha=0\.05\) reveals significant differences between techniques; according to the post\-hoc Dunn’s test all algorithms differ significantly fromRAN, but there are no significant differences between the other algorithms\.

### VI\-DRQ4 \-HaGen\+performance

![Refer to caption](https://arxiv.org/html/2608.04697v1/x2.png)Figure 3:RQ4 \- Distribution of generated eventsWith RQ4 we assess the combination of the beststructureddescription generation strategy EVA and the generation of thenarrativedescription byHaGen\. Indeed, whileHaGencan generate the structured description with good results in terms of Jaccard distance, it occasionally generates invalid solutions \(RQ1\), a problem that the other algorithms do not have\.

As the overall performances of the baseline techniques considering allresulttypes together are indistinguishable \(RQ3, Fig\.[2](https://arxiv.org/html/2608.04697#S6.F2)\), we run an analysis byresulttype in order to assess the best baseline on the structured part to be combined withHaGen\. Figure[3](https://arxiv.org/html/2608.04697#S6.F3)shows the distribution of all accidents distinct byresulttype\.EVAhas the lowest overall median value for 7 out or 10resulttype; importantly,EVAis also the most stable technique across allresulttypes, as in almost all cases the distribution is concentrated around the median \(smaller variance\), whereas the other algorithms produce only a few high\-quality events\.

To exploit the performance and robustness ofEVAand the narrative generation capability ofHaGen, we defineHaGen\+\.HaGen\+takes the structured part generated byEVAas input \(in the prompt\), and generates the corresponding unstructured part compliant with the structured input\.

In order to get the highest benefit fromEVA, we first investigated why it did not perform well in 3 out of 10 accident types\. Since EVA is an evolutionary algorithm, the population evolves over generations: although we forced it to initially generate solutions for eachresulttype, the solutions of some type are discarded as the population evolves\. Therefore, even though we select the best solution of that type encountered during the evolution, this is not necessarily the last generation’s solution\. In such cases, the Jaccard distance of those solutions might be not optimal, as they evolved over fewer generations\. Fig[4](https://arxiv.org/html/2608.04697#S6.F4)reports the generation from which the solution of each type were picked up\. In three cases, solutions come from initial generations, hence they are suboptimal\. We thus forcedEVAto evolve the solutions of allresulttype\.

![Refer to caption](https://arxiv.org/html/2608.04697v1/x3.png)Figure 4:Generation from which solutions of each type are takenHaGen\+integrates this improvement ofEVA\. We hereafter compareHaGen\+againstHaGen\. The immediate advantage is in terms of Generation Failure Ratio \(GFR\)\. Table[V](https://arxiv.org/html/2608.04697#S6.T5)reports the GFR ofHaGenbyresulttype: although we selected the best configuration, the GFR ranges from 5% to up to 25% of invalid accidents \(average 13%\)\.HaGen\+solves this problem, as all accidents generated by combining EVA withHaGenarevalid\.

TABLE V:Generation failure ratio ofHaGenin its best configuration for eachresult \(R\)In terms of realism, the Jaccard distances for the structural part are equivalent \(HaGen\+uses EVA, which is shown to be equivalent toHaGenin RQ3, Fig\.[2](https://arxiv.org/html/2608.04697#S6.F2)\)\. However, by taking advantage of the greater stability of EVA \(Fig\.[3](https://arxiv.org/html/2608.04697#S6.F3)\), the Jaccard distance ofHaGen\+exhibits lower variance\. Fig\.[5](https://arxiv.org/html/2608.04697#S6.F5)reports the overall distributions ofHaGenandHaGen\+for the Jaccard distance\. TheLevene’s hypothesis test for checking homoscedasticity reports app\-value of7\.17×10−147\.17\\text\{\\times\}\{10\}^\{\-14\}, confirming that variances are significantly unequal\. Similarly, Fig\.[6](https://arxiv.org/html/2608.04697#S6.F6)compares the distributions ofHaGenandHaGen\+for theNarrativeandCallbackBartScore\. Forced by the higher stability of EVA in the structured part generation,HaGen\+offers a smaller variance for theNarrativeandCallbacktoo – the Levene’s test reports app\-value of0\.00340\.0034and1\.14×10−091\.14\\text\{\\times\}\{10\}^\{\-09\}, respectively\.

![Refer to caption](https://arxiv.org/html/2608.04697v1/x4.png)Figure 5:RQ4 \-HaGen\+vs\.HaGenin generating realistic structured part![Refer to caption](https://arxiv.org/html/2608.04697v1/x5.png)Figure 6:RQ4 \-HaGen\+vs\.HaGenin generating realisticNarrativeandCallback

## VIIRegulatory Positioning

A practical concern for any AI\-assisted safety technique is whether it can be integrated with frameworks recognized by European Union Aviation Safety Agency \(EASA\) and Federal Aviation Administration \(FAA\)\. We positionHaGen/HaGen\+as a candidate alternative means of compliance \(AMC\) \- or as a tool supporting an accepted means of compliance \- rather than as an existing means of compliance, and identify what adoption on a certification programme would still require\.

### VII\-APlacement in the ARP4761A process

ARP4761A\[[34](https://arxiv.org/html/2608.04697#bib.bib1)\],‘Guidelines for Conducting the Safety Assessment Process on Civil Aircraft, Systems, and Equipment’, dated December20232023and published by SAE International, present guidelines for performing safety assessments of civil aircraft, systems, and equipment\. We select this standard as it may be used when addressing compliance with certification requirements by both EASA \(eg\. CS Parts 23, 25, 27, and 29\) and FAA \(e\.g\. 14 CFR Parts 23, 25, 27, and 29, 33 and 35\)\. This standard prescribes an iterative, qualitative\-and\-quantitative safety assessment process articulated in the Aircraft and System Functional Hazard Assessments \(AFHA/SFHA\), the Preliminary Aircraft/System Safety Assessments \(PASA/PSSA\), and the final Aircraft/System Safety Assessments \(ASA/SSA\)\. The process begins with the AFHA, which reviews aircraft\-level functions, identifies failure conditions, and classifies severity; the resulting safety objectives are then passed to the PASA, which assesses the architecture, derives safety requirements, and allocates Development Assurance Levels that flow down to the PSSA\.

HaGenandHaGen\+support the hazard\-identification input that feeds operational hazard analysis; they do not themselves perform FHA steps such as severity classification, which remain with qualified analysts\. They do not replace any assessment, nor do they produce certification credit: their outputs are candidate scenarios to be reviewed, classified, and, if retained, entered into the hazard log by qualified analysts\. More precisely,HaGenandHaGen\+produce two coupled artifacts\. Thestructuredpart – a combination of co\-occurring categorical factors – provides operational context that can inform the identification of failure conditions in the FHA, rather than constituting a failure condition itself\. Thenarrativepart – the textual description of an event sequence consistent with those factors – helps analysts classify severity and validate scenario plausibility, both core AFHA activities\.

### VII\-BAnalyst\-in\-the\-loop and alignment with EASA initiatives\.

The positioning ofHaGen/HaGen\+as an assistant to safety analysts is consistent with two converging strands of EASA policy\. First, theData4Safetyprogramme222https://www\.easa\.europa\.eu/en/domains/safety\-management/data4safetyexplicitly promotes a proactive, data\-driven paradigm in which large corpora of operational safety data are analyzed collaboratively to surface systemic risks; our use of the ASRS corpus to ground candidate hazard scenarios is methodologically aligned with this paradigm\. Second, the EASA Artificial Intelligence Roadmap\[[6](https://arxiv.org/html/2608.04697#bib.bib2),[8](https://arxiv.org/html/2608.04697#bib.bib3)\]and the associated Concept Paper on Level 1 and Level 2 machine\-learning applications\[[7](https://arxiv.org/html/2608.04697#bib.bib4),[9](https://arxiv.org/html/2608.04697#bib.bib5)\]classify AI systems according to the degree of autonomy granted to the machine\. In this taxonomyHaGen/HaGen\+falls withinLevel 1A – Human Augmentation: the tool augments the analyst’s exploration of plausible hazard combinations, but every output is subjected to expert review, classification and acceptance before being entered into any certification\-relevant artifact\. No decision is delegated to the model, and generated scenarios are never treated as certified safety items\.

### VII\-COpen issues

Formal acceptance would still require validation on real certification programmes; the definition of an operational design domain for the LLM covering training\-data coverage, admissible outcome categories and supported variables; monitoring for model drift and hallucination consistent with the learning assurance, AI explainability and continuous safety management objectives of the EASA Concept Paper\[[9](https://arxiv.org/html/2608.04697#bib.bib5)\]\.HaGen/HaGen\+thereforeaugments– and does not replace – the classical hazard identification techniques by ARP4761A\.

## VIIIThreats to validity

ASRS is a voluntary reporting system, biased toward hazards that occurred, were noticed and survivable, while catastrophic are under\-represented\. As it underlies scenario generation and evaluation, this bias should be considered when interpreting results\. To assess the plausibility and realism of the generated accidents, we measure their similarity to occurred events, as in the related literature\[[29](https://arxiv.org/html/2608.04697#bib.bib67),[30](https://arxiv.org/html/2608.04697#bib.bib77),[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\. However, this measurement is conservative, as an event may differ from those previously observed while still being plausible – a qualitative rating from experts could capture these cases and better highlight the generation performance\. For the structured part, we adopted the Jaccard distance \(as in\[[29](https://arxiv.org/html/2608.04697#bib.bib67),[30](https://arxiv.org/html/2608.04697#bib.bib77),[28](https://arxiv.org/html/2608.04697#bib.bib57)\]\)\. For the unstructured part, we usedBartScore, a metric to evaluate text similarity\.BartScoreis measured with a fine\-tuned version of BART on ParaBank2, a dataset designed for paraphrasing, by reformulating the similarity assessment as a paraphrase evaluation\. While it is a widespread metric, there are many metrics for text similarity, which could yield different results\. To account for results variability, we deployed 12HaGenconfigurations and statistically assessed their differences\. The selection of potential\-cause variables and the accident\-type definition may influence the results\. We use 13 variables, excluding fields related to multi\-aircraft occurrences; extending the approach to such scenarios is future work\.

Finally, the LLMs employed are a snapshot of a rapidly evolving landscape: newer models may perform better\.

## IXConclusion and Future Work

We proposed an AI\-assisted strategy to generate hypotheses of potential accidents in avionics\. We analyzed 12 configurations across three large language models, two prompting strategies, and the use of fine\-tuning, assessing their ability to generate plausible and realistic accidents\. We compared our approach with state\-of\-the\-art methods for structured accident generation, evaluated its capability to produce fine\-grained narratives, and improved the pipeline by hybridizing narrative generation with the best\-performing structured generator\.

A natural extension is the development of a novel plausibility metric that goes beyond historical co\-occurrence evidence and incorporates causal knowledge of the avionics domain\. By reasoning over causal relations among contributing factors rather than their mere joint occurrence, the score would better reflectwhya hypothesized scenario is plausible, providing the analyst with a more reliable and interpretable ranking signal\.

## References

- \[1\]D\. Colombo, M\. H\. Maathuis, M\. Kalisch, and T\. S\. Richardson\(2012\)Learning high\-dimensional directed acyclic graphs with latent and selection variables\.The Annals of Statistics40\(1\)\.External Links:ISSN 0090\-5364,[Link](http://dx.doi.org/10.1214/11-AOS940),[Document](https://dx.doi.org/10.1214/11-aos940)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p2.1),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p2.9)\.
- \[2\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: Efficient Finetuning of Quantized LLMs\.InNIPS ’23: Proceedings of the 37th International Conference on Neural Information Processing Systems,pp\. 10088–10115\.External Links:2305\.14314Cited by:[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p2.1)\.
- \[3\]O\. J\. Dunn\(1964\)Multiple comparisons using rank sums\.Technometrics6\(3\),pp\. 241–252\.External Links:ISSN 00401706,[Link](http://www.jstor.org/stable/1266041)Cited by:[§VI\-B](https://arxiv.org/html/2608.04697#S6.SS2.p3.5)\.
- \[4\]J\. J\. Durillo and A\. J\. Nebro\(2011\)JMetal: a java framework for multi\-objective optimization\.Advances in Engineering Software42\(10\),pp\. 760–771\.External Links:ISSN 0965\-9978,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.advengsoft.2011.05.014),[Link](https://www.sciencedirect.com/science/article/pii/S0965997811001219)Cited by:[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p1.2)\.
- \[5\]C\. J\. Edwards\(1994\)Developing a Safety Case with an Aircraft Operator\.InProceedings of the Second Annual Safety\-Critical Systems Symposium,London, UK,pp\. 93–104\.Cited by:[6th item](https://arxiv.org/html/2608.04697#S3.I1.i6.p1.1)\.
- \[6\]European Union Aviation Safety Agency \(EASA\)\(2020\-02\)Artificial intelligence roadmap 1\.0: a human\-centric approach to ai in aviation\.Technical reportEASA\.Note:Accessed: 2025\-02\-11External Links:[Link](https://www.easa.europa.eu/en/downloads/109668/en)Cited by:[§VII\-B](https://arxiv.org/html/2608.04697#S7.SS2.p1.2)\.
- \[7\]European Union Aviation Safety Agency \(EASA\)\(2021\)Concept paper: first usable guidance for level 1 machine learning applications\.Technical reportTechnical ReportIssue 01,EASA\.Note:Accessed: 2026\-03\-11External Links:[Link](https://www.easa.europa.eu/en/downloads/134357/en)Cited by:[§VII\-B](https://arxiv.org/html/2608.04697#S7.SS2.p1.2)\.
- \[8\]European Union Aviation Safety Agency \(EASA\)\(2023\-05\)Artificial intelligence roadmap 2\.0: a human\-centric approach to ai in aviation\.Technical reportEASA\.Note:Accessed: 2026\-03\-11External Links:[Link](https://www.easa.europa.eu/en/downloads/137919/en)Cited by:[§VII\-B](https://arxiv.org/html/2608.04697#S7.SS2.p1.2)\.
- \[9\]European Union Aviation Safety Agency \(EASA\)\(2024\)Concept paper: guidance for level 1 & 2 machine learning applications\.Technical reportTechnical ReportIssue 02,EASA\.Note:Accessed: 2026\-03\-11External Links:[Link](https://www.easa.europa.eu/en/downloads/139504/en)Cited by:[§VII\-B](https://arxiv.org/html/2608.04697#S7.SS2.p1.2),[§VII\-C](https://arxiv.org/html/2608.04697#S7.SS3.p1.1)\.
- \[10\]P\. Haapanen and A\. Helminen\(2002\)Failure mode and effects analysis of software\-based automation\.Reliability Engineering and System Safety75\(1\),pp\. 29–40\.External Links:[Document](https://dx.doi.org/10.1016/S0951-8320%2801%2900116-1)Cited by:[3rd item](https://arxiv.org/html/2608.04697#S3.I1.i3.p1.1)\.
- \[11\]J\. E\. Hu, A\. Singh, N\. Holzenberger, M\. Post, and B\. Van Durme\(2019\)Large\-scale, diverse, paraphrastic bitexts via sampling and clustering\.InProceedings of the 23rd Conference on Computational Natural Language Learning \(CoNLL\),M\. Bansal and A\. Villavicencio \(Eds\.\),Hong Kong, China,pp\. 44–54\.External Links:[Link](https://aclanthology.org/K19-1005/),[Document](https://dx.doi.org/10.18653/v1/K19-1005)Cited by:[§V\-E](https://arxiv.org/html/2608.04697#S5.SS5.p3.1)\.
- \[12\]B\. Kirwan and L\. K\. Ainsworth \(Eds\.\)\(1992\)A guide to task analysis\.Taylor & Francis,London, UK\.External Links:ISBN 978\-0748400570Cited by:[1st item](https://arxiv.org/html/2608.04697#S3.I1.i1.p1.1)\.
- \[13\]B\. Kirwan\(1994\)A guide to practical human reliability assessment\.Taylor & Francis,London, UK\.External Links:ISBN 978\-0748400037Cited by:[2nd item](https://arxiv.org/html/2608.04697#S3.I2.i2.p1.1)\.
- \[14\]W\. H\. Kruskal and W\. A\. Wallis\(1952\)Use of ranks in one\-criterion variance analysis\.Journal of the American Statistical Association47\(260\),pp\. 583–621\.External Links:[Document](https://dx.doi.org/10.1080/01621459.1952.10483441)Cited by:[§VI\-B](https://arxiv.org/html/2608.04697#S6.SS2.p3.5)\.
- \[15\]N\. G\. Leveson\(1995\)Safeware: system safety and computers: a guide to preventing accidents\.Addison\-Wesley,Reading, MA, USA\.External Links:ISBN 978\-0201119725Cited by:[2nd item](https://arxiv.org/html/2608.04697#S3.I1.i2.p1.1),[5th item](https://arxiv.org/html/2608.04697#S3.I1.i5.p1.1)\.
- \[16\]M\. Lewis, Y\. Liu, N\. Goyal, M\. Ghazvininejad, A\. Mohamed, O\. Levy, V\. Stoyanov, and L\. Zettlemoyer\(2020\)BART: denoising sequence\-to\-sequence pre\-training for natural language generation, translation, and comprehension\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 7871–7880\.Cited by:[§V\-E](https://arxiv.org/html/2608.04697#S5.SS5.p3.1)\.
- \[17\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)AWQ: activation\-aware weight quantization for llm compression and acceleration\.External Links:2306\.00978,[Link](https://arxiv.org/abs/2306.00978)Cited by:[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p2.1)\.
- \[18\]A\. @\. M\. Llama Team\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p2.1)\.
- \[19\]Microsoft\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.External Links:2404\.14219,[Link](https://arxiv.org/abs/2404.14219)Cited by:[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p2.1)\.
- \[20\]Mistral AI Team\(2024\-10\)Un Ministral, des Ministraux\.External Links:[Link](https://mistral.ai/news/ministraux)Cited by:[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p2.1)\.
- \[21\]NASAAviation safety reporting system\.Note:https://asrs\.arc\.nasa\.govOnline; accessed 31 December 2024Cited by:[§II](https://arxiv.org/html/2608.04697#S2.p1.1),[§III\-A](https://arxiv.org/html/2608.04697#S3.SS1.p5.1)\.
- \[22\]National Transportation Safety Board \(NTSB\)\(n\.d\.\)Aviation accident database & synopses\.Note:Accessed: 9 January 2025External Links:[Link](http://www.ntsb.gov/_layouts/ntsb.aviation/index.aspx)Cited by:[§III\-A](https://arxiv.org/html/2608.04697#S3.SS1.p5.1)\.
- \[23\]A\. Nouri, B\. Cabrero\-Daniel, F\. Törner, H\. Sivencrona, and C\. Berger\(2024\)Engineering safety requirements for autonomous driving with large language models\.In2024 IEEE 32nd International Requirements Engineering Conference \(RE\),pp\. 218–228\.External Links:[Document](https://dx.doi.org/10.1109/RE59067.2024.00029)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p1.1)\.
- \[24\]A\. Nouri, B\. Cabrero\-Daniel, F\. Torner, H\. Sivencrona, and C\. Berger\(2024\)Welcome Your New AI Teammate: On Safety Analysis by Leashing Large Language Models\.InProceedings of the IEEE/ACM 3rd International Conference on AI Engineering \- Software Engineering for AI,CAIN ’24,pp\. 172–177\.External Links:ISBN 9798400705915,[Link](https://doi.org/10.1145/3644815.3644953),[Document](https://dx.doi.org/10.1145/3644815.3644953)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p1.1)\.
- \[25\]S\. of Automotive Engineers \(SAE\)\(2010\)Certification considerations for highly\-integrated or complex aircraft systems\.Technical reportTechnical ReportARP4754A,SAE International,SAE,Warrendale, PA, USA\.Cited by:[4th item](https://arxiv.org/html/2608.04697#S3.I1.i4.p1.1)\.
- \[26\]J\. M\. Ogarrio, P\. Spirtes, and J\. Ramsey\(2016\)A hybrid causal search algorithm for latent variable models\.InProceedings of the Eighth International Conference on Probabilistic Graphical Models,A\. Antonucci, G\. Corani, and C\. P\. Campos \(Eds\.\),Proceedings of Machine Learning Research, Vol\.52,pp\. 368–379\.External Links:[Link](https://proceedings.mlr.press/v52/ogarrio16.html)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p2.1),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p2.9)\.
- \[27\]J\. Pearl\(2009\)Causality: models, reasoning and inference\.2nd edition,Cambridge University Press,USA\.Cited by:[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p2.9)\.
- \[28\]R\. Pietrantuono and S\. Russo\(2025\)Automatic generation of plausible co\-occurring causes for effects explanation or prediction\.ACM Transactions on Intelligent Systems and Technology,pp\. 1–32\.Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p2.1),[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p3.1),[§IV\-B](https://arxiv.org/html/2608.04697#S4.SS2.p1.1),[2nd item](https://arxiv.org/html/2608.04697#S5.I1.i2.p1.1),[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p2.1),[§V\-C](https://arxiv.org/html/2608.04697#S5.SS3.p3.1),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p1.2),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p2.9),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p3.7),[§VI\-C](https://arxiv.org/html/2608.04697#S6.SS3.p3.2),[§VIII](https://arxiv.org/html/2608.04697#S8.p1.1)\.
- \[29\]R\. Pietrantuono\(2021\)Automated hypotheses generation via combinatorial causal optimization\.In2021 IEEE Congress on Evolutionary Computation \(CEC\),pp\. 399–407\.External Links:[Document](https://dx.doi.org/10.1109/CEC45853.2021.9504816)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p1.1),[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p3.1),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p1.2),[§VIII](https://arxiv.org/html/2608.04697#S8.p1.1)\.
- \[30\]R\. Pietrantuono\(2023\)An evolutionary strategy for automatic hypotheses generation inspired by abductive reasoning\.InProceedings of the Companion Conference on Genetic and Evolutionary Computation,GECCO ’23 Companion,pp\. 235–238\.External Links:ISBN 9798400701207,[Link](https://doi.org/10.1145/3583133.3590568),[Document](https://dx.doi.org/10.1145/3583133.3590568)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p2.1),[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p3.1),[§VIII](https://arxiv.org/html/2608.04697#S8.p1.1)\.
- \[31\]J\. Ramsey, K\. Zhan, M\. Glymour, R\. Sanchez Romero, B\. Huang, I\. Ebert\-Uphoff, S\. M\. Samarasinghe, E\. A\. Barnes, and C\. Glymour\(2018\)TETRAD \- A Toolbox for Causal Discovery\.In8th International Workshop on Climate Informatics,Cited by:[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p2.9)\.
- \[32\]J\. D\. Ramsey\(2015\)Scaling up greedy causal search for continuous variables\.Technical reportSAE International,Center for Causal Discovery,Pittsburgh, PA, USA\.External Links:1507\.07749,[Link](https://arxiv.org/abs/1507.07749)Cited by:[§III\-B](https://arxiv.org/html/2608.04697#S3.SS2.p2.1),[§V\-D](https://arxiv.org/html/2608.04697#S5.SS4.p2.9)\.
- \[33\]RVS GroupIncident and accident database\.Note:www\.rvs\.uni\-bielefeld\.de/publications/compendium/incidents\_and\_accidents/index\.htmlAccessed: 9 January 2025Cited by:[§III\-A](https://arxiv.org/html/2608.04697#S3.SS1.p5.1)\.
- \[34\]SAE International\(2023\)Guidelines for conducting the safety assessment process on civil aircraft, systems, and equipment\.Technical reportTechnical ReportARP4761A,SAE International\.Note:Accessed: 2026\-03\-11Cited by:[§VII\-A](https://arxiv.org/html/2608.04697#S7.SS1.p1.1)\.
- \[35\]B\. Schroeder and G\. A\. Gibson\(2010\)A large\-scale study of failures in high\-performance computing systems\.IEEE Transactions on Dependable and Secure Computing7\(4\),pp\. 337–350\.External Links:[Document](https://dx.doi.org/10.1109/TDSC.2009.4)Cited by:[§III\-A](https://arxiv.org/html/2608.04697#S3.SS1.p5.1)\.
- \[36\]S\. T\. Shorrock\(2002\)Error classification for safety management: finding the right approach\.InProceedings of the International Conference on Safety Management,London, UK,pp\. 56–63\.Cited by:[3rd item](https://arxiv.org/html/2608.04697#S3.I2.i3.p1.1)\.
- \[37\]W\. Vesely, J\. Dugan, J\. Fragola, J\. I\. Minarick, J\. Railsback, and M\. Stamatelatos\(2002\-08\)Fault tree handbook with aerospace applications\.Technical reportNASA Office of Safety and Mission Assurance\.Cited by:[5th item](https://arxiv.org/html/2608.04697#S3.I1.i5.p1.1)\.
- \[38\]F\. Wilcoxon\(1945\)Individual comparisons by ranking methods\.Biometrics Bulletin1\(6\),pp\. 80–83\.External Links:ISSN 00994987,[Link](http://www.jstor.org/stable/3001968)Cited by:[§VI\-B](https://arxiv.org/html/2608.04697#S6.SS2.p3.5)\.
- \[39\]J\. C\. Williams\(1988\)A Data\-Based Method for Assessing and Reducing Human Error to Improve Operational Performance\.Proceedings of the IEEE Conference on Human Factors in Electronics,pp\. 436–442\.Cited by:[1st item](https://arxiv.org/html/2608.04697#S3.I2.i1.p1.1)\.
- \[40\]W\. Yuan, G\. Neubig, and P\. Liu\(2021\)BARTScore: evaluating generated text as text generation\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 27263–27277\.Cited by:[§I](https://arxiv.org/html/2608.04697#S1.p2.1),[§V\-E](https://arxiv.org/html/2608.04697#S5.SS5.p3.1)\.