The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting

# The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting
Source: [https://arxiv.org/html/2609.16267](https://arxiv.org/html/2609.16267)
Hisham IhshaishAffiliation:School of Computing and Creative Technologies, University of the West of England, Bristol, UKAffiliation:Department of Computer Science, Birzeit University, Birzeit, PalestinePeter MayhewAffiliation:School of Computing and Creative Technologies, University of the West of England, Bristol, UKAffiliation:GE Aerospace, Cheltenham, UKTasnim M\. A\. ZayetAna Del AmoAffiliation:GE Aerospace, Clearwater, FL, USA\*Corresponding authors: hisham\.ihshaish@uwe\.ac\.uk; peter\.mayhew@geaerospace\.com

###### Abstract

Many operational cases are documented more than once, at different workflow stages and for different purposes, yet model evaluations normally select one of these records before model comparison begins\. We treat that selection as part of the evaluation and compare matched records of the same cases under fixed labels and splits in three systems: GE Aerospace repair events, NASA ASRS safety reports and NHTSA vehicle recalls\. Across the three GE fields, for events whose label comes from parts transactions independently of the narratives, held\-out macro\-F1 ranged from 0\.33 to 0\.91\. A difference of 0\.46 separated the customer report, written before shop work, from the technician report, written after diagnosis but before the transaction that generates the label\. That difference is substantially larger than the representation and architecture differences tested on the same events\. The public systems showed different patterns: the NHTSA defect summary remained strongest under every model family tested, whereas the ASRS analyst synopsis outperformed the reporter narrative under learned sequence models but not under lexical baselines\. Secondary analyses showed that some model comparisons were also record\-dependent\. Evaluations should be run on the information available at the intended decision point and should report how both the record and the label were produced\.

Keywords:matched\-record evaluation; text classification; documentation context; industrial AI; maintenance records; safety reporting; evaluation validity

## 1Introduction

In 1796 the Astronomer Royal, Nevil Maskelyne, dismissed his assistant David Kinnebrook after months of disagreement over the times each recorded for the same stellar transits\. The episode became the founding case of the personal equation once Bessel treated such differences as systematic properties of observation\[[1](https://arxiv.org/html/2609.16267#bib.bib41)\], although Kinnebrook’s surviving letters show that the assistant understood the problem better than the standard account allows\[[2](https://arxiv.org/html/2609.16267#bib.bib44)\]\. The lesson, that a measurement is read together with the process that produced it, applies to the text that information systems now classify, where the process differs in one further respect\. A repair, a safety report or a recall campaign generates several written accounts as it moves through a workflow, written by different people, at different stages, for different purposes\. Such accounts are the working material of industrial text analytics\. Maintenance work orders in computerised maintenance management systems carry record quality as a standing constraint on downstream analysis\[[3](https://arxiv.org/html/2609.16267#bib.bib17),[4](https://arxiv.org/html/2609.16267#bib.bib18)\]\. The NASA Aviation Safety Reporting System holds tens of thousands of voluntary reports in its Aircraft category alone, and every vehicle manufacturer must lodge recall filings with its regulator\. When such text is classified, evaluation practice selects one of these records, and the selection is rarely treated as part of what is being evaluated\.

We use*record*for any distinctly produced textual account of a case: a named free\-text field within one filing, or a separately submitted narrative\. The records of one case are not interchangeable accounts of the same content\. They differ in who wrote them, what was known when they were written and what they were written to do\. A classifier never reads the case; it reads a record of the case\. The label it is scored against is produced by a further process, sometimes independent of every record and sometimes inside the workflow that wrote one of them\. Measured performance can therefore reflect documentation as well as modelling\. The questions such classifications serve are practical ones, which assembly to order before a unit reaches the shop, which reports a safety analyst might examine first and how a regulator might code a campaign that affects an entire fleet\. Clinical NLP has met one form of this problem: classifiers confounded by the institution that produced a note\[[5](https://arxiv.org/html/2609.16267#bib.bib29)\], and retrieval rankings that fail to carry across documentation settings\[[6](https://arxiv.org/html/2609.16267#bib.bib38)\]\.

The workflow setting differs from the astronomical case in one respect: the two astronomers timed one event from the same information, whereas later records in a workflow can contain information that did not yet exist when the earlier ones were written\. A repair of a cockpit display unit illustrates the difference\. The customer reports “Display blanks intermittently in flight”; the technician, after diagnosis, writes “Fault isolated to processor assembly; key panel checked serviceable”; the repair record states “Replaced processor assembly”\. All three describe one return, and each later record holds information the earlier ones could not\. Diagnosis generated new information between the first record and the second, and the completed repair generated more before the third\. Because the three records are different information states, each becomes a legitimate model input only once the information it contains exists, and a high score on the third record says nothing about a classifier for the first, which addresses a different problem\. In a fleet of more than ten thousand units in service, an evaluation on the wrong record misjudges a triage aid before it is deployed\.

Although data source and documentation context are known to condition NLP systems \(Section[2](https://arxiv.org/html/2609.16267#S2)\), much less is known about what happens when alternative records of the same cases are exchanged under a common target\.

This study asks two questions on matched records from three record systems, one proprietary and two public \(Figures[1](https://arxiv.org/html/2609.16267#S3.F1)and[2](https://arxiv.org/html/2609.16267#S4.F2)\)\. The systems are GE Aerospace repair events, extending our earlier classification study of those records\[[7](https://arxiv.org/html/2609.16267#bib.bib1)\]with a different question; safety reports from the NASA Aviation Safety Reporting System \(ASRS\); and vehicle recall campaigns reported to the National Highway Traffic Safety Administration \(NHTSA\)\. First, how much does classification performance change when a model reads a different record of the same case? Second, do conclusions about modelling approaches remain the same across those records? Within every comparison, cases, targets and evaluation splits are fixed, and model families and training schedules are held constant \(Section[3](https://arxiv.org/html/2609.16267#S3)\)\. We also explore whether disagreement between records, or between models reading different records of a case, can identify records that warrant review\. The practical question behind both is whether an organisation should improve the model, improve the information captured before the decision or change which records the system is allowed to use\. Throughout, we distinguish who wrote each record, when in the workflow it was written and for what purpose, and we make the same distinction for the label\. The contribution is an evaluation design that places alternative routine records of the same cases on the same held\-out scale as the modelling choices tested and ties every comparison to the workflow stage at which the record exists\.

The design is set out in Section[3](https://arxiv.org/html/2609.16267#S3)and applied to the three systems in Sections[4](https://arxiv.org/html/2609.16267#S4)to[6](https://arxiv.org/html/2609.16267#S6)\. Section[7](https://arxiv.org/html/2609.16267#S7)draws the engineering implication, that a benchmark is only as valid as the records a system can read when the decision is taken\.

## 2Related Work

### 2\.1Evaluation validity and data context

Whether an evaluation supports its claims depends on whether its task and measures represent the phenomenon being claimed\. A systematic review of large language model benchmarks finds construct validity to be a recurring concern\[[8](https://arxiv.org/html/2609.16267#bib.bib40)\]\. Industrial AI research reaches a similar position from the engineering side: a recent meta\-review organises eighty\-two data issues across seven stages of the data lifecycle, from source and quality onward, and proposes managing them at every stage\[[9](https://arxiv.org/html/2609.16267#bib.bib28)\]\. Models can also exploit decision rules that succeed on standard benchmarks yet fail under more demanding conditions\[[10](https://arxiv.org/html/2609.16267#bib.bib22)\]\. What counts as leakage depends on the intended use of the model\[[11](https://arxiv.org/html/2609.16267#bib.bib23)\], and in clinical prediction whether every predictor was available at the moment the model is intended to be used is a standing applicability item\[[12](https://arxiv.org/html/2609.16267#bib.bib47)\]\. The same concern arises here through the record selected as model input\.

### 2\.2Multiple records and documentation context

The most direct evidence that documentation context conditions model behaviour comes from clinical NLP, where classifiers are confounded by the institution that produced a note and provenance\-aware adjustment changes their conclusions\[[5](https://arxiv.org/html/2609.16267#bib.bib29)\]\. In clinical embedding\-based retrieval, the context variables of the documentation explain as much retrieval\-performance variance as the choice of embedding model, and model–corpus interactions make rankings non\-portable across documentation settings\[[6](https://arxiv.org/html/2609.16267#bib.bib38)\]\. Those studies compare institutions, corpora or query conditions\. A close same\-case design compares note types of the same admissions for a given outcome, with a single model trained on all notes and the note types subsampled to equal length\[[13](https://arxiv.org/html/2609.16267#bib.bib48)\], and finds that the most useful note type depends on the outcome\. Here the variation is within the case and the procedure is refitted to each record at its full length: the label and the evaluated cases stay fixed while a different routinely produced record of each case is supplied\. Multi\-view learning starts from the same multiplicity but asks how records should be combined\[[14](https://arxiv.org/html/2609.16267#bib.bib33)\], and a systematic review of multi\-view document classification finds fusion gains that are consistent but modest\[[15](https://arxiv.org/html/2609.16267#bib.bib34)\]\. Matched\-record evaluation first asks what each record does on its own, and in deployed systems a retrieval component may make that choice, which makes it a property of the deployed system as well as of the evaluation\. Prediction\-time availability, data provenance and documentation context are established concerns\[[12](https://arxiv.org/html/2609.16267#bib.bib47),[5](https://arxiv.org/html/2609.16267#bib.bib29),[6](https://arxiv.org/html/2609.16267#bib.bib38)\], and multi\-view classification routinely fits a separate classifier to each view of the same labelled instances\[[16](https://arxiv.org/html/2609.16267#bib.bib35),[14](https://arxiv.org/html/2609.16267#bib.bib33)\]\. The distinction here is that the views are routine records generated by an organisational workflow, with different producers, purposes and availability times\. The performance difference between those records, with the same procedure refitted to each record, is treated as the estimand itself and not as a baseline for fusion\. The design then tests whether model comparisons survive exchanging the record while cases, labels and evaluation partitions remain fixed\.

### 2\.3Maintenance and safety text

Aviation and maintenance text has been classified with sparse statistical learners\[[17](https://arxiv.org/html/2609.16267#bib.bib6),[18](https://arxiv.org/html/2609.16267#bib.bib7)\], recurrent models over embeddings\[[7](https://arxiv.org/html/2609.16267#bib.bib1),[19](https://arxiv.org/html/2609.16267#bib.bib9)\]and, more recently, domain\-pretrained transformers and large language models\[[20](https://arxiv.org/html/2609.16267#bib.bib13),[21](https://arxiv.org/html/2609.16267#bib.bib14),[22](https://arxiv.org/html/2609.16267#bib.bib21),[23](https://arxiv.org/html/2609.16267#bib.bib36)\]\. Most evaluations use public safety corpora, mainly ASRS, whose analyst\-assigned labels and de\-identified text differ in kind from the maintenance records operators hold\[[24](https://arxiv.org/html/2609.16267#bib.bib15),[25](https://arxiv.org/html/2609.16267#bib.bib16)\], and work on aviation maintenance records remains comparatively thin\[[26](https://arxiv.org/html/2609.16267#bib.bib19),[27](https://arxiv.org/html/2609.16267#bib.bib20)\]\.

The technical\-language\-processing literature established maintenance work orders as a distinct NLP object, telegraphic and written under time pressure, with record quality as a persistent constraint\[[3](https://arxiv.org/html/2609.16267#bib.bib17),[28](https://arxiv.org/html/2609.16267#bib.bib8)\]that reduces the accuracy of downstream analyses, often invisibly\[[4](https://arxiv.org/html/2609.16267#bib.bib18)\]\. Maintenance text now serves to correct metadata in computerised maintenance management systems \(CMMS\), retrieve related records, extract causal relations and derive repair subtasks\[[29](https://arxiv.org/html/2609.16267#bib.bib24),[30](https://arxiv.org/html/2609.16267#bib.bib25),[31](https://arxiv.org/html/2609.16267#bib.bib26),[32](https://arxiv.org/html/2609.16267#bib.bib27)\]\. Classifier disagreement has identified ambiguous or incorrect CMMS metadata\[[29](https://arxiv.org/html/2609.16267#bib.bib24)\], and the audit here extends that idea to a label generated separately from any narrative\. Maintenance research is also moving toward systems that use tools and multiple agents for diagnostic and prescriptive work\[[33](https://arxiv.org/html/2609.16267#bib.bib42),[34](https://arxiv.org/html/2609.16267#bib.bib43)\], which raises the question of what information is available to the system at each stage\. In the maintenance and safety studies reviewed here, the input record is generally treated as a fixed property of the dataset\.

## 3Matched\-record study design

Procedurem0m\_\{0\}Procedurem1m\_\{1\}S⁡\(ra,m0\)S\(r\_\{a\},m\_\{0\}\)S⁡\(ra,m1\)S\(r\_\{a\},m\_\{1\}\)S⁡\(rb,m0\)S\(r\_\{b\},m\_\{0\}\)S⁡\(rb,m1\)S\(r\_\{b\},m\_\{1\}\)Recordrar\_\{a\}Recordrbr\_\{b\}ΔR\\Delta\_\{R\}ΔR\\Delta\_\{R\}ΔM​\(ra\)\\Delta\_\{M\}\(r\_\{a\}\)ΔM​\(rb\)\\Delta\_\{M\}\(r\_\{b\}\)D=ΔM​\(ra\)−ΔM​\(rb\)D=\\Delta\_\{M\}\(r\_\{a\}\)\-\\Delta\_\{M\}\(r\_\{b\}\): does the model comparison change with the record?Figure 1:The matched\-record design as a record\-by\-procedure table of held\-out scoresS⁡\(r,m\)S\(r,m\)on the same cases and labels:ΔR\\Delta\_\{R\}compares records under one refitted procedure,ΔM\\Delta\_\{M\}compares procedures on one record, andDDasks whether the procedure comparison changes with the record\.Each system supplies matched records: caseiicarries recordsXi\(1\),…,Xi\(V\)X\_\{i\}^\{\(1\)\},\\dots,X\_\{i\}^\{\(V\)\}and one labelYiY\_\{i\}\. We fix a modelling proceduremm, an architecture with its preprocessing and training schedule, and writeS⁡\(r,m\)S\(r,m\)for the held\-out macro\-F1 \(the F1 score averaged with equal weight over classes\) of the model that procedure fits on recordrr, scored over the shared held\-out cases, cases set aside from all training \(Figure[1](https://arxiv.org/html/2609.16267#S3.F1)\)\. Two estimands are defined on this table of scores\. The record contrast

ΔR​\(ra,rb,m\)=S⁡\(ra,m\)−S⁡\(rb,m\)\\Delta\_\{R\}\(r\_\{a\},r\_\{b\};m\)\\;=\\;S\(r\_\{a\},m\)\-S\(r\_\{b\},m\)\(1\)compares two records under one procedure: cases, targets, splits, model family and training schedule are fixed, and each record’s model is fitted and evaluated separately\.ΔR\\Delta\_\{R\}is therefore the end\-to\-end consequence of changing the record, with the procedure refitted to each record\. The model contrastΔM​\(r,m1,m0\)=S⁡\(r,m1\)−S⁡\(r,m0\)\\Delta\_\{M\}\(r;m\_\{1\},m\_\{0\}\)=S\(r,m\_\{1\}\)\-S\(r,m\_\{0\}\)compares two models on one record, and its stability across records is measured by the interaction

D=ΔM​\(ra,m1,m0\)−ΔM​\(rb,m1,m0\),D\\;=\\;\\Delta\_\{M\}\(r\_\{a\};m\_\{1\},m\_\{0\}\)\-\\Delta\_\{M\}\(r\_\{b\};m\_\{1\},m\_\{0\}\),\(2\)which is nonzero when the observed benefit of a modelling choice depends on the record read\. When the two model contrasts differ in sign, the ranking of the two models reverses between records\. We define one further quantity, used once: the cross\-record transferT\(ra→rb;m\)T\(r\_\{a\}\\\!\\to\\\!r\_\{b\};m\), the score of the model fitted on recordrar\_\{a\}when it reads recordrbr\_\{b\}of the same cases, which combines record differences with the shift between training and evaluation text \(Section[6\.2](https://arxiv.org/html/2609.16267#S6.SS2)\)\. Every model\-performance comparison in the study is one of these quantities, with paired inference over the shared test cases \(Section[5\.2](https://arxiv.org/html/2609.16267#S5.SS2)\)\.

#### Decision\-time availability:

LetHi​\(τ\)H\_\{i\}\(\\tau\)denote the information about caseiithat exists by workflow timeτ\\tau, and let recordvvbe produced at timeτv\\tau\_\{v\}by a recording processgvg\_\{v\}, its author, purpose, selection and wording, so thatXi\(v\)=gv​\(Hi​\(τv\)\)X\_\{i\}^\{\(v\)\}=g\_\{v\}\\bigl\(H\_\{i\}\(\\tau\_\{v\}\)\\bigr\)\. Two records of one case can differ becauseτa≠τb\\tau\_\{a\}\\neq\\tau\_\{b\}, so that different information existed, or becausega≠gbg\_\{a\}\\neq g\_\{b\}, so that the same information was recorded differently, andΔR\\Delta\_\{R\}measures the joint consequence of both without separating them\. A decision taken at timeτ\\taucan read only records that exist by then and that the deployed system is permitted to access\. Writing𝒫\\mathcal\{P\}for the set of records the system may access, the admissible set is

𝒜i\(τ\)=\{v:τv≤τ,v∈𝒫\}\.\\mathcal\{A\}\_\{i\}\(\\tau\)=\\\{\\,v:\\tau\_\{v\}\\leq\\tau,\\ v\\in\\mathcal\{P\}\\,\\\}\.\(3\)An evaluation is decision\-time valid with respect to information availability only if every input it uses lies in𝒜i​\(τ\)\\mathcal\{A\}\_\{i\}\(\\tau\), whether the input is fixed in advance or selected at run time by a retrieval policy,π⁡\(i,τ\)∈𝒜i​\(τ\)\\pi\(i,\\tau\)\\in\\mathcal\{A\}\_\{i\}\(\\tau\)for a single record andπ⁡\(i,τ\)⊆𝒜i​\(τ\)\\pi\(i,\\tau\)\\subseteq\\mathcal\{A\}\_\{i\}\(\\tau\)for several\.

The three systems place the design under three of the possible relationships between record production and label production \(Figure[2](https://arxiv.org/html/2609.16267#S4.F2)\)\. The GE label is produced independently of every narrative, from parts transactions; the NHTSA label is the regulator’s coding of the report that contains the fields; the ASRS label is assigned inside the analyst workflow that also writes one of the records\. The systems differ accordingly in how directly a measured contrast can be interpreted: the record comparison is most directly interpretable where the label is produced independently of every record, and most entangled where record and label are co\-produced\. Independence bears on how the comparison is interpreted, and the validity of the label is a separate question\. The transaction\-derived label has failure modes of its own \(Section[6\.4](https://arxiv.org/html/2609.16267#S6.SS4)\)\.

## 4Data

Three record systems are used, each supplying three records per case \(Figure[2](https://arxiv.org/html/2609.16267#S4.F2); Table[1](https://arxiv.org/html/2609.16267#S4.T1)\)\. Held\-out partitions are constructed as described in Section[5\.2](https://arxiv.org/html/2609.16267#S5.SS2), with the exact\-duplicate rule given with the NHTSA dataset, and Figure[3](https://arxiv.org/html/2609.16267#S4.F3)shows the token\-length distributions of the public systems’ records\. Tables, figures and sections numbered S1 onward are in the Supplementary Material\.

\(a\)GE Aerospacerepair eventCustomer\-reported faultoperator, pre\-shopTechnician reporttechnician, diagnosisRepair actiontechnician, post\-repairPartstransactionsOutcome label4 classesfrom the repair itself, independent of the narratives\(b\)NASA ASRSsafety reportReporter narrativereporter, at submissionSupplemental narrativesecond reporterASRS analyst processingAnalyst synopsisanalyst, same stepPrimary problembinary taskother analyst information\(c\)NHTSArecall campaignDefect summarymanufacturer reportConsequencemanufacturer reportRemedymanufacturer reportNHTSA analysis of the reportComponent class16 classesfull Part 573 report

Figure 2:The three record systems on one plan: each case yields several records \(shaded, producer and stage inside\) and one target \(thick border\)\. The GE target comes from parts transactions, bypassing the narratives; the ASRS analyst and the NHTSA regulator code their targets from the reports, and the ASRS analyst also writes the synopsis\. Dashed inputs mark information beyond the modelled records\.Table 1:The records and their provenance, with median lengths in tokens\. GE medians cannot be exported \(dashes; mean character lengths in Section S1\)\. Each record’s relation to the label is shown in Figure[2](https://arxiv.org/html/2609.16267#S4.F2)\.### 4\.1GE Aerospace LRU dataset

Each time an avionics line\-replaceable unit \(LRU\) is removed from an aircraft and returned to a maintenance, repair and overhaul \(MRO\) shop, the repair generates free text at successive workflow stages, and these narratives feed failure reporting and corrective\-action processes\[[35](https://arxiv.org/html/2609.16267#bib.bib10),[36](https://arxiv.org/html/2609.16267#bib.bib12),[37](https://arxiv.org/html/2609.16267#bib.bib11)\]\. The dataset covers one civil cockpit\-display LRU family with more than 10k units in service: 5792 repair events after the exclusions needed for reliable transaction linkage and unambiguous classes \(Section S1\), of which 4633 train and 1159 are held out\. Three free\-text fields of each event are modelled, the*customer\-reported fault*, the*technician report*and the*repair action*\(Figure[2](https://arxiv.org/html/2609.16267#S4.F2)\), of the form shown by the synthetic record in Section[1](https://arxiv.org/html/2609.16267#S1)\. The label is the assembly actually replaced, taken from the parts transactions and joined to the narratives by the return identifier\. Replaced parts are the operational reference because they record the maintenance action independently of the narratives and drive spares forecasting\. Their agreement with the narratives is examined in Section[6\.4](https://arxiv.org/html/2609.16267#S6.SS4)\. The task covers the two most frequently replaced sub\-assemblies as four mutually exclusive classes, chosen for label determinacy as much as frequency\.

The fields differ in when they exist relative to the label \(Figure[2](https://arxiv.org/html/2609.16267#S4.F2)\)\. The customer text exists before any shop work and the technician report is written after diagnosis has identified the fault, so both precede the parts transaction that produces the label, at different information stages\. The repair action is written during or after the repair that generates the label, so classifying it is a post\-outcome task, automated coding and narrative–transaction reconciliation, in which the field’s account of the outcome is legitimately available\. How much of that field’s value is lexical restatement of the outcome is tested by masking in Section[6\.2](https://arxiv.org/html/2609.16267#S6.SS2)\.

### 4\.2NASA ASRS dataset

ASRS holds voluntary, de\-identified safety reports, each with one*primary problem*assigned by an ASRS analyst among 18 categories of markedly unequal prevalence\. We reformulate classification as independent binary tasks with majority\-class undersampling to 50:50 and report the largest equipment\-related category in full: all 44597 Aircraft records through 2021 with a non\-blank primary problem, against an equal undersample of the remaining categories\.

Two further records of each case accompany the reporter’s narrative \(Table[1](https://arxiv.org/html/2609.16267#S4.T1)\)\. The*synopsis*is a short summary written by the analyst who processes the report and assigns the primary problem, drawing on overflow text and fields absent from the public record\[[38](https://arxiv.org/html/2609.16267#bib.bib30)\]\. On the Aircraft task it accompanies 89186 of 89194 records, at a median 19 tokens against the narrative’s 178\. Where two parties reported the same event, the second reporter’s*supplemental narrative*is attached, and 7637 task records carry one, at a median 91 tokens\. Analysts merge multiple reports of one event into one database record, choose which reports receive detailed analysis and edit narratives during processing and de\-identification, which replaces tail numbers, operators and equipment identifiers with placeholders\[[39](https://arxiv.org/html/2609.16267#bib.bib31)\]\.

Figure 3:Token\-length distributions of the records of the same cases in the two public systems, as log\-binned histograms scaled to their highest bin with medians in parentheses\. The ASRS records of one case differ in length by an order of magnitude and the supplemental narrative is bimodal, with a short\-record mode and a full\-account mode, whereas the NHTSA fields are similar in length\. GE distributions cannot be exported \(field statistics in Section S1\)\.
### 4\.3NHTSA recall campaigns

Every United States vehicle recall campaign is reported to NHTSA under 49 CFR Part 573, and the recall record carries three narrative fields from that report: a*defect summary*describing the problem, a*consequence*field describing the safety risk and a*remedy*field describing the corrective action\. The target is NHTSA’s component classification of the campaign, collapsed to the top level of its taxonomy\. NHTSA derives it from its own analysis of the manufacturer’s report, with acknowledged subjectivity in the mapping\[[40](https://arxiv.org/html/2609.16267#bib.bib37)\], so the target is a coding of the case by the regulator, separate from the text fields\. All campaigns with 2000–2026 campaign numbers were retrieved through the public recall API\[[41](https://arxiv.org/html/2609.16267#bib.bib32)\], 16626 after de\-duplication, and categories with at least 300 campaigns define a 16\-class task with 3325 campaigns held out\. Boilerplate recurs across filings, 39% of consequence texts being exact duplicates of another campaign’s, so campaigns sharing an identical text in any field are confined to one side of the split\. The fields are short \(medians 41, 19 and 35 tokens\) and the component name can appear verbatim in any of them, which motivates the class\-vocabulary masking condition\. Campaign numbers encode the filing year, which supports the out\-of\-time check in Section[6\.2](https://arxiv.org/html/2609.16267#S6.SS2)\(class list and snapshot details in Section S1\)\.

## 5Models and analysis

We use four kinds of model, each for a different diagnostic purpose, with shared\-representation controls that separate the record from representation choice where a comparison requires it \(Section[5\.1](https://arxiv.org/html/2609.16267#S5.SS1)\)\. Preprocessing is limited to lower\-casing and punctuation stripping before tokenisation into whitespace\-delimited alphanumeric units, keeping part identifiers and shorthand intact, with no stemming, lemmatisation or stop\-word removal\.

The task is single\-label classification of a tokenised narrative, withC=4C=4assembly classes for GE,C=2C=2per NASA task andC=16C=16component classes for NHTSA\. Each narrative is read through a frozen word embedding, a learned mapping from words to vectors that stays fixed while the classifier trains\. Tokens outside an embedding’s vocabulary map to a single shared fallback vector\. Representation\-specific differences, including vocabulary coverage and out\-of\-vocabulary handling, are analysed in Section S5\.

### 5\.1Models

The lexical baseline is word TF\-IDF \(term frequency weighted by inverse document frequency\) with linear classifiers, plus a character n\-gram variant, and asks whether surface vocabulary alone carries a difference\. The learned sequence model is a bidirectional long short\-term memory network \(BiLSTM\)\[[42](https://arxiv.org/html/2609.16267#bib.bib45),[43](https://arxiv.org/html/2609.16267#bib.bib46)\], which reads the record as an ordered sequence over frozen word embeddings\. Its embeddings are word2vec vectors trained on the corresponding training record or field\. The general\-purpose GloVe\-200\[[44](https://arxiv.org/html/2609.16267#bib.bib2)\]serves as a shared\-representation control, and on GE the comparison also uses Avi2Vec, a word2vec embedding trained on aviation maintenance text\[[7](https://arxiv.org/html/2609.16267#bib.bib1),[45](https://arxiv.org/html/2609.16267#bib.bib3),[46](https://arxiv.org/html/2609.16267#bib.bib4)\]\. The pooling comparator is a perceptron over the same frozen embeddings with mean pooling, which averages the word vectors and discards word order\. It separates sequence reading from representation and suppliesm0m\_\{0\}in Equation[2](https://arxiv.org/html/2609.16267#S3.E2)\. The pretrained transformer, RoBERTa\-base\[[47](https://arxiv.org/html/2609.16267#bib.bib39)\], is fine\-tuned end\-to\-end on each public task and asks whether the results persist under a pretrained contextual encoder\. No pretrained checkpoint can be brought into the GE computing environment, so its transformer comparator is trained from scratch\. Further controls and every architectural and training constant are in Sections S2 and S3\. Together these models test whether the record differences persist across lexical, sequential and contextual representations\.

### 5\.2Inference

We split each dataset once into 80% training and 20% held\-out test data before any preprocessing or tuning, fitting tokeniser and vocabulary statistics on the training partition only\. Performance is summarised by macro\-averaged F1, weighting the operationally distinct classes equally\. Under the class imbalance of the GE task, accuracy would reward majority\-class prediction\. Each neural configuration is estimated by three independent trainings of the same procedure\. A difference is reported as its mean over trainings, with a paired bootstrap 95% interval obtained by resampling the shared held\-out cases jointly, conditional on the trained models\. The range over trainings is given separately\. The training\-averaged estimand and the resample and permutation counts per family are in Section S2\. Significance statements come from paired approximate\-randomisation tests, Holm\-corrected within each preplanned family of comparisons\. Stratified 10\-fold cross\-validation appears in the supplement as description only\[[48](https://arxiv.org/html/2609.16267#bib.bib5)\]\. The paper’s conclusions rest on the held\-out paired estimates\. Unless a model is named, matched\-record differences are reported for the record\- or field\-trained word2vec BiLSTM, and for the Avi2Vec BiLSTM on GE\.

### 5\.3Comparator implementation and sensitivity analyses

We built an independent implementation of the GE and ASRS pipelines to test dependence on the initial codebase, and the two agree \(Section S3\)\. The principal analyses carry an evidential status, preplanned, prospective extension, post hoc or prior\-study design, defined and listed in Section S7\. Within each record the sequence and pooling models share that record’s embedding, so eachΔM\\Delta\_\{M\}in Equation[2](https://arxiv.org/html/2609.16267#S3.E2)is representation\-matched\.

### 5\.4Audit procedure for model–label disagreements

Misclassified test records are manually reviewed against their narratives and coded as model errors, narrative–transaction inconsistencies or faults outside the label space \(category definitions and worked examples in Section S6\)\. Coders saw narrative, transaction label and prediction together, so anchoring toward the model’s reading cannot be excluded\. No formal inter\-rater statistic was recorded and correctly classified records were not audited\. The audit covers all 168 technician\-input misclassifications in the initial extraction’s 1279\-record test set and predates the removal of the earlier\-build\-variant records\. Of the audited records, 162 remain in the final dataset \(Section S6\)\. The audit was performed by GE maintenance engineers and is treated as exploratory\.

## 6Results

Table[2](https://arxiv.org/html/2609.16267#S6.T2)collects the matched\-record contrasts and secondary quantities with paired uncertainty\. All inferential results use the held\-out comparisons \(Section[5\.2](https://arxiv.org/html/2609.16267#S5.SS2)\)\.

Table 2:Held\-out record and model comparisons\. Panel A gives matched\-record differencesΔ¯\\bar\{\\Delta\}with the same procedure fitted separately to each record\. Panel B gives the cross\-record transferTTand the secondary quantities\. Values are three\-training means with paired case\-bootstrap 95% intervals and the range over trainings\. \*GE interval endpoints could not be exported\. Status vocabulary is defined in Section S7\.SystemContrastΔ¯\\bar\{\\Delta\}\[95% CI\]Range over trainingsStatusA: Matched records \(fitted separately\)GEcustomer→\\rightarrowtechnician\+0\.456\+0\.456\*\+0\.419\+0\.419to\+0\.492\+0\.492prior\-study designGEtechnician→\\rightarrowrepair action\+0\.127\+0\.127\*\+0\.104\+0\.104to\+0\.152\+0\.152prior\-study designASRSsynopsis−\-narrative\+0\.015\+0\.015\[0\.011, 0\.019\]\+0\.011\+0\.011to\+0\.021\+0\.021preplannedASRSsynopsis−\-narrative, RoBERTa\+0\.015\+0\.015\[0\.011, 0\.019\]\+0\.011\+0\.011to\+0\.018\+0\.018prospective extensionNHTSAsummary−\-consequence\+0\.115\+0\.115\[0\.099, 0\.131\]\+0\.113\+0\.113to\+0\.118\+0\.118preplannedNHTSAsummary−\-remedy\+0\.148\+0\.148\[0\.131, 0\.165\]\+0\.129\+0\.129to\+0\.160\+0\.160preplannedB: Cross\-record transfer and secondary quantitiesASRSprimary→\\tosupplemental \(TT\)\+0\.179\+0\.179\[0\.157, 0\.202\]\+0\.170\+0\.170to\+0\.187\+0\.187preplannedASRScomparable\-length subset\+0\.010\+0\.010\[−\-0\.013, 0\.033\]\+0\.001\+0\.001to\+0\.017\+0\.017post hocASRSinteractionDD\(Eq\.[2](https://arxiv.org/html/2609.16267#S3.E2)\)\+0\.017\+0\.017\[0\.013, 0\.021\]\+0\.014\+0\.014to\+0\.020\+0\.020post hoc### 6\.1Magnitude of the record differences

Under the Avi2Vec–BiLSTM procedure the three GE fields reach macro\-F1 of 0\.327 on the held\-out cases \(customer\-reported fault\), 0\.783 \(technician report\) and 0\.910 \(repair action\), means over three independent trainings\. Reading a different field of the same events therefore moves performance by 0\.583 across the three fields, and by 0\.456 between the customer and technician fields, both written before the transaction that generates the label\. Customer to technician is\+0\.419\+0\.419to\+0\.492\+0\.492in every training, technician to repair action\+0\.104\+0\.104to\+0\.152\+0\.152\. The same ordering appears under the TF\-IDF baselines and both recurrent architectures\. The gain appears in every class and is largest for the rarest \(the rarest class’s values in Section S3\)\. Figure[4](https://arxiv.org/html/2609.16267#S6.F4)places these record differences against the representation and model differences measured on the same events under the same evaluation: the largest modelling difference tested moves performance by 0\.092, the record by up to 0\.583\. The quantities in Figure[4](https://arxiv.org/html/2609.16267#S6.F4)are descriptive magnitudes and are not components of a variance decomposition\.

The ASRS analyst’s synopsis outperforms the reporter’s narrative in every training of a learned sequence model, by 0\.011–0\.021 macro\-F1 across both embeddings, averaging\+0\.015\+0\.015\[0\.011, 0\.019\]\. The advantage is absent under the lexical baselines: word TF\-IDF is level across the records, and character n\-grams favour the narrative nominally \(per\-configuration results in Table S9\)\. The fine\-tuned RoBERTa shows the same advantage, averaging\+0\.015\+0\.015\[0\.011, 0\.019\], an estimate that coincides with the recurrent models’ to three decimals\. The primary–supplemental comparison produces the largest raw ASRS difference, and it is a cross\-record transfer: fixed narrative models reading the second reporter’s account of the same 1530 held\-out cases lose 0\.165–0\.187 macro\-F1 in all six models\. The number combines record differences with the shift between training and evaluation text, and Section[6\.2](https://arxiv.org/html/2609.16267#S6.SS2)separates the two\.

Figure 4:Record differences against representation and model differences on one scale, GE task, held\-out evaluation\. All entries are means over three trainings \(Table S5\); the bracketed values on the two smallest rows are the largest single\-training differences\.NHTSA component classification is strongest from the defect summary, lower from the consequence field and lowest from remedy text under both TF\-IDF and the field\-trained BiLSTM \(values per run in Section S4\)\. The summary exceeds the consequence field by\+0\.115\+0\.115\[0\.099, 0\.131\] and the remedy field by\+0\.148\+0\.148\[0\.131, 0\.165\], with 11 of 12 paired contrasts Holm\-significant\. Under the BiLSTM the per\-training differences span\+0\.113\+0\.113to\+0\.118\+0\.118and\+0\.129\+0\.129to\+0\.160\+0\.160\(Table[2](https://arxiv.org/html/2609.16267#S6.T2)\)\. TF\-IDF, a single deterministic evaluation, gives\+0\.094\+0\.094and\+0\.143\+0\.143\. Chosen as different record and label production settings, the two public systems test whether record sensitivity appears there and are not replications of the GE task\.

### 6\.2Interpreting the record differences

The GE ordering follows the workflow stage at which each field is written \(Section[4](https://arxiv.org/html/2609.16267#S4)\) and is already visible to lexical models, where TF\-IDF rises from 0\.362 on customer text to 0\.891 on repair actions\. Explicit outcome terminology contributes to the later fields without accounting for the difference, since a keyword rule over the curated outcome terms reaches 0\.292 on the repair\-action field and masking those terms lowers the BiLSTM by 0\.013 with GloVe\-200 and 0\.038 with Avi2Vec\. Representation choice is smaller again: Avi2Vec exceeds GloVe\-200 by 0\.049 on repair\-action text, 0\.028 of it associated with tokens that only the domain vocabulary covers, and the advantage was not observed on the de\-identified ASRS text \(Sections S3 and S5\)\.

The ASRS synopsis result is smaller and harder to interpret, because synopsis and target are produced within the same analyst workflow \(Section[3](https://arxiv.org/html/2609.16267#S3)\)\. Literal category\-label tokens alone do not explain the advantage: masking every token that appears in the category labels, in both records, costs the synopsis 0\.005 and leaves it above the narrative, and the record with fewer label tokens is the one that scores higher\. The direction held across the five redrawn balanced tasks\. The much larger transfer loss on the supplemental account behaves differently, falling to about zero as the second account approaches the primary in length \(Figure[6](https://arxiv.org/html/2609.16267#S6.F6), right\)\. With the selection and editing documented in Section[4\.2](https://arxiv.org/html/2609.16267#S4.SS2), that pattern points to completeness and curation, and the analyses do not support reporter identity alone as an explanation \(all conditions in Section S4\)\.

The NHTSA ordering follows the purpose of each field and holds under three different conditions\. The fields are similar in length \(Figure[3](https://arxiv.org/html/2609.16267#S4.F3)\), so length does not explain it\. Masking every token that occurs in the sixteen class labels, at training and evaluation, lowers every score yet preserves the full ordering, all twelve masked contrasts Holm\-significant\. Trained on campaigns filed 2000–2021 and evaluated on the 2022–2026 filings, the summary remains strongest under both models in every training, all six prospectively specified BiLSTM contrasts Holm\-significant\. Grouping near\-duplicate filings to one side of the split, token\-set Jaccard≥\\geq0\.80 in any field, affected 39\.5% of the held\-out campaigns in the main split\. It changes absolute scores, which are not comparable across the two test samples, and leaves the ordering intact in every training \(prospective extension\)\. The two lower fields are less separable: with one embedding shared across the fields the consequence–remedy difference largely disappears while the summary stays best \(Figure[5](https://arxiv.org/html/2609.16267#S6.F5), right\)\. The difference therefore extends beyond literal component\-name overlap and is associated with where the content relevant to the target is expressed within the filing \(Section S4 and Figure S3\)\. Across the three systems the differences are organised by who wrote the record, when and for what purpose, without any of the three being separately identified\. Table[3](https://arxiv.org/html/2609.16267#S7.T3)asks an evaluation to report those three properties\.

Figure 5:Held\-out macro\-F1 across the three NHTSA fields under the plain model families \(left\) and under the sensitivity conditions \(right\), with the class\-label vocabulary masked for TF\-IDF and the BiLSTM and one embedding shared across the fields for the BiLSTM\. Masking lowers absolute performance while the defect summary remains strongest, and the shared embedding makes the consequence–remedy ordering less stable\.
### 6\.3Secondary: record dependence of model comparisons

Except for the RoBERTa comparisons, a prospective extension, these secondary analyses are post hoc\. On the same ASRS cases the ranking of the lexical baseline against the sequence model reverses between the records in every training pair, TF\-IDF exceeding the BiLSTM on narratives and trailing it on synopses \(Figure[6](https://arxiv.org/html/2609.16267#S6.F6), left\)\. The benefit of sequence reading over pooling is also larger on synopses\. The interactionDDof Equation[2](https://arxiv.org/html/2609.16267#S3.E2)averages\+0\.017\+0\.017\[0\.013, 0\.021\] under the record\-trained embeddings and persists at reduced size, though less uniformly, under the shared\-representation control\. RoBERTa reproduces the synopsis advantage at the same size as the recurrent models, and its benefit over the BiLSTM does not differ by record \(DDspans−0\.006\-0\.006to\+0\.006\+0\.006\), while its benefit over TF\-IDF does \(\+0\.011\+0\.011to\+0\.018\+0\.018, every interval excluding zero\)\. The record dependence therefore lies between lexical and learned sequence models, and the pretrained encoder and the recurrent reader did not differ by record \(per\-configuration values and the NHTSA RoBERTa results in Section S4\)\.

Figure 6:ASRS record dependence appears differently across model families and across record completeness\. On the left, held\-out macro\-F1 of each model family on the narrative and on the synopsis, as three\-training means on a y\-axis spanning 0\.870–0\.895, where the BiLSTM \(dashed\) and RoBERTa lie within 0\.001 of each other, near 0\.875 on the narrative and 0\.890 on the synopsis\. On the right, the cross\-record transfer loss on the supplemental narrative by the ratio of supplemental to primary length, as mean and range over six trainings with bin sizes shown\.
### 6\.4Exploratory: record disagreement as review evidence

The remaining analyses are exploratory\. Combining the two ASRS records by mean probability exceeds the better single record by 0\.005–0\.013 in every configuration \(allpHolm≤0\.0104p\_\{\\mathrm\{Holm\}\}\\leq 0\.0104\), a gain of the size the multi\-view literature reports \(Section[2\.2](https://arxiv.org/html/2609.16267#S2.SS2)\)\. Cross\-record disagreement, which affects 11\.3% of held\-out cases, captured more of the reference model’s errors than selecting the same number of least confident predictions at every review budget from 1% to 30%\. Given the balanced task and the synopsis’s origin \(Section[4\.2](https://arxiv.org/html/2609.16267#S4.SS2)\), this supports prioritising post\-coding review and does not estimate screening yield in service \(Section S6\)\.

The GE audit turns to disagreement between the model’s reading of the narrative and the transaction\-derived label\. Of the 168 technician\-text misclassifications audited in the initial extraction \(Section[5\.4](https://arxiv.org/html/2609.16267#S5.SS4)\), three quarters \(126\) were coded as narrative–transaction inconsistency, a fifth \(35\) as model errors and seven as faults outside the label space\. The headline share lies between 74\.1% and 77\.8% under any coding of the six audited records later removed from the dataset \(Section S6\)\. Within the limits of Section[5\.4](https://arxiv.org/html/2609.16267#S5.SS4), narrative–transaction inconsistency accounts for most of the apparent model error among the contested records\.

## 7Discussion

The 0\.456 gap between the GE customer and technician fields opens before the parts transaction exists, and every model change we tested moved the score by less, 0\.049 for the domain embedding and 0\.092 for sequence reading against pooling \(Figure[4](https://arxiv.org/html/2609.16267#S6.F4)\)\. Before a team compares models on maintenance text it should therefore know which record the model will read in service\. Capture practice was not manipulated, so 0\.456 is the difference between two records as they are written today and says nothing about the return from changing intake\.

The two GE gaps have different interpretations, because the customer\-to\-technician transition introduces new diagnostic evidence whereas the later repair\-action field can also restate the completed outcome\. A model reading the customer field cannot recover findings that did not exist when the customer wrote, and can at best predict them\. Part of the 0\.127 gap between the technician and repair\-action fields is also restatement: a keyword rule over the curated outcome terms alone reaches 0\.292 on that field, and masking those terms costs the BiLSTM 0\.038\. In the notation of Section[3](https://arxiv.org/html/2609.16267#S3), the customer\-to\-technician transition necessarily changesHi​\(τ\)H\_\{i\}\(\\tau\)because diagnosis has occurred, but it also changesgvg\_\{v\}, andΔR\\Delta\_\{R\}does not separate the two\. The technician\-to\-repair transition again changes both, and the later field can also state the completed maintenance action directly\. The intervention implied by a limitation inHi​\(τ\)H\_\{i\}\(\\tau\), capturing more before the decision, differs from the one implied bygvg\_\{v\}, better models, standardised wording or curation, and the matched\-record comparison gives their joint size and not the return from either alone\.

The small ASRS synopsis gain of 0\.015 arises inside the analyst workflow that also produces the target, so it is best read as evidence about curation within one workflow \(Section[3](https://arxiv.org/html/2609.16267#S3)\)\. Field purpose, by contrast, organises the NHTSA ordering, which holds under masking, out\-of\-time evaluation and near\-duplicate grouping\. Across the systems the observed patterns are consistent with differences in producer, workflow stage and record purpose, although those factors are not separately identified\.

In the secondary ASRS analyses the lexical baseline and the sequence model also change order between the reporter narrative and the analyst synopsis in every training pair, while RoBERTa and the BiLSTM show no consistent record\-dependent difference\. A model advantage measured on one record therefore need not transfer unchanged to another record of the same cases\.

For the GE triage setting, the scale of the record difference shifts attention from model choice to information availability\. The technician record is not thereby a better input for triage, because it is created only after diagnosis\. It identifies the information available at triage, where a model reads the customer field and reaches 0\.327, as the constraint that no tested model change relieves\. Structured intake or access to further contemporaneous records could address that constraint, although the benefit of either intervention has not been measured here\. Table[3](https://arxiv.org/html/2609.16267#S7.T3)illustrates, for one record per system, the context such an evaluation should report: when the record exists, who produced it for what purpose, the decision it would inform and the relation of the label to the records\.

Table 3:Illustrative decision contexts for interpreting record\-based classifier evaluations, one record per system\. The decision column names the use each evaluation would inform; only the GE decision is a documented operational use\. The ASRS label uses information beyond the synopsis; the NHTSA field is one part of the filing the regulator codes from\.For retrieval\- and tool\-using systems, decision\-time validity becomes an access\-control problem\[[34](https://arxiv.org/html/2609.16267#bib.bib43),[33](https://arxiv.org/html/2609.16267#bib.bib42)\]: the evaluation has to fix the admissible set of Equation[3](https://arxiv.org/html/2609.16267#S3.E3), records that exist by the decision and that the system may read, for the decision assessed \(Figure[7](https://arxiv.org/html/2609.16267#S7.F7)\)\. A post\-repair record retrieved at run time lies outside the triage set exactly as it would if supplied as a fixed input\.

Figure 7:Decision\-time availability of records\. Above, the general case: each recordX\(v\)X^\{\(v\)\}becomes available at its production timeτv\\tau\_\{v\}, and a decision taken at timeτ\\taumay read only the records available by then that the deployed system is permitted to access\. Below, the GE repair event: the customer\-reported fault exists before shop work, the technician report after diagnosis and the repair action after the repair, and the dashed lines mark the decisions of triage, part prediction at diagnosis and post\-repair reconciliation\. The records whose lines have begun at a decision, and that the deployed system may read, form its admissible set, Equation[3](https://arxiv.org/html/2609.16267#S3.E3)\. The parts transaction, from which the label is taken, follows the repair action\.The numerical effects should not be generalised beyond the systems studied\. The GE difference bundles a change in what was known \(HiH\_\{i\}\) with changes in author, purpose, length and vocabulary \(gvg\_\{v\}\), which move together with the field\. The ordering holds in every training and under the lexical and recurrent models tested, but the attribution of the 0\.456 among those causes stays open\. The GE evidence covers one LRU family, one organisation, about a dozen record authors and two assemblies, so rare faults and other sites are untested\. The ASRS task is balanced by undersampling and capped at 256 tokens, with a synopsis co\-produced with the target, so it does not estimate performance at operational prevalence, and NHTSA covers one regulator and sixteen top\-level component classes\.

The review evidence is weaker still\. The audit covered only misclassified records, unblinded and without an agreement statistic, so its 126 of 168 attributions to narrative–transaction inconsistency do not estimate how common that inconsistency is, and disagreement as a review aid needs blinded, controlled validation before use in screening \(Section S6\)\.

A prospective information\-capture intervention would give the clearest causal follow\-up: an intake form altered to elicit specific diagnostic detail, with downstream performance measured again at the same decision point and under the same models, would separate the bundled causes of the GE gap\. A blinded audit with correctly classified controls would test the review finding, and replication across further LRU families, sites and maintenance systems would show how far the observed sizes carry\. The same problem leads to active information acquisition, a system that identifies which observation a decision still lacks and asks for it before the decision is taken\.

## 8Conclusion

Which record a classifier reads moved its score on the GE repair events more than any model change we tested\. The gap between the customer and technician fields, both written before the parts transaction, was 0\.456 macro\-F1, against 0\.092 for the largest modelling change tested\. Smaller and differently shaped dependences appeared in the two public systems, each consistent with who wrote the record, when and for what purpose, and some model comparisons changed with the record\. The GE fields are not interchangeable, because each exists at a different workflow stage and supports a different decision\. Accordingly, an evaluation should identify the decision being supported, restrict inputs to the records available at that point, document how the target was constructed and compare modelling approaches within that information boundary\. The design applies wherever a case is documented more than once and the production of the label can be established\.

## Data Availability

ASRS reports are public at[https://asrs\.arc\.nasa\.gov/search/database\.html](https://asrs.arc.nasa.gov/search/database.html)and NHTSA campaigns come from the public recall API\[[41](https://arxiv.org/html/2609.16267#bib.bib32)\]\. The analysis code, public\-data snapshots, analysis plans and archived public\-system results are at[https://github\.com/ihshaish/source\-variation](https://github.com/ihshaish/source-variation)\. The GE Aerospace dataset and the Avi2Vec vectors are proprietary and cannot be shared\.

## CRediT author statement

H\.I\. and P\.M\.:Conceptualization, Methodology, Formal analysis, Writing \(original draft, review and editing\), with Software and Data curation by P\.M\. and Supervision by H\.I\.T\.M\.A\.Z\. and A\.D\.A\.:Methodology, Validation, Writing \(review and editing\)\.

## Funding and conflicts of interest

This research was supported by GE Aerospace\. P\.M\. and A\.D\.A\. are employees of GE Aerospace and contributed in the author roles listed above\. Beyond those authorial contributions, the sponsor had no role in study design, analysis, interpretation or the decision to publish\.

## References

- \[1\]S\. Schaffer\(1988\)Astronomers mark time: discipline and the personal equation\.Science in Context2\(1\),pp\. 115–145\.External Links:[Document](https://dx.doi.org/10.1017/s026988970000051x)Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p1.1)\.
- \[2\]M\. D\. Lund\(2025\)“A Vitious Way of Observing”: Kinnebrook and the Prehistory of the Personal Equation\.Isis116\(3\),pp\. 461–484\.External Links:[Document](https://dx.doi.org/10.1086/736892)Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p1.1)\.
- \[3\]M\. P\. Brundage, T\. Sexton, M\. Hodkiewicz, A\. Dima, and S\. Lukens\(2021\)Technical language processing: unlocking maintenance knowledge\.Manufacturing Letters27,pp\. 42–46\.External Links:[Document](https://dx.doi.org/10.1016/j.mfglet.2020.11.001)Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p1.1),[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[4\]A\. Conte, C\. Bolland, L\. Phan, M\. Brundage, and T\. Sexton\(2021\)Impact of data quality on maintenance work order analysis: a case study in historical HVAC maintenance work orders\.InProceedings of the PHM Society European Conference,Vol\.6\.External Links:[Document](https://dx.doi.org/10.36001/phme.2021.v6i1.2814)Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p1.1),[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[5\]X\. Ding, Z\. Sheng, M\. Yetişgen, S\. Pakhomov, and T\. Cohen\(2023\)Backdoor adjustment of confounding by provenance for robust text classification of multi\-institutional clinical notes\.InAMIA Annual Symposium Proceedings,pp\. 923–932\.Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[6\]Y\. Mikkelsen\(2026\)Clinical context variables collectively rival model choice in embedding\-based retrieval: multi\-corpus benchmark study\.JMIR Medical Informatics14,pp\. e94241\.External Links:[Document](https://dx.doi.org/10.2196/94241)Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[7\]P\. Mayhew, H\. Ihshaish, J\. I\. Deza, and A\. Del Amo\(2023\)Maintenance automation using deep learning methods: a case study from the aerospace industry\.InArtificial Neural Networks and Machine Learning – ICANN 2023,Lecture Notes in Computer Science, Vol\.14263,pp\. 295–307\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-44204-9%5F25)Cited by:[§1](https://arxiv.org/html/2609.16267#S1.p5.1),[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1),[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[8\]A\. M\. Bean, R\. O\. Kearns, A\. Romanou, F\. S\. Hafner, H\. Mayne, J\. Batzner,et al\.\(2025\)Measuring what matters: construct validity in large language model benchmarks\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2511\.04703Cited by:[§2\.1](https://arxiv.org/html/2609.16267#S2.SS1.p1.1)\.
- \[9\]X\. Li, Y\. Cheng, C\. Møller, and J\. Lee\(2025\)Data issues in industrial AI systems: a meta\-review and research strategy\.Computers in Industry173,pp\. 104361\.External Links:[Document](https://dx.doi.org/10.1016/j.compind.2025.104361)Cited by:[§2\.1](https://arxiv.org/html/2609.16267#S2.SS1.p1.1)\.
- \[10\]R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. Wichmann\(2020\)Shortcut learning in deep neural networks\.Nature Machine Intelligence2,pp\. 665–673\.External Links:[Document](https://dx.doi.org/10.1038/s42256-020-00257-z)Cited by:[§2\.1](https://arxiv.org/html/2609.16267#S2.SS1.p1.1)\.
- \[11\]S\. Kapoor and A\. Narayanan\(2023\)Leakage and the reproducibility crisis in machine\-learning\-based science\.Patterns4\(9\),pp\. 100804\.External Links:[Document](https://dx.doi.org/10.1016/j.patter.2023.100804)Cited by:[§2\.1](https://arxiv.org/html/2609.16267#S2.SS1.p1.1)\.
- \[12\]K\. G\. M\. Moons, J\. A\. A\. Damen, T\. Kaul, L\. Hooft, C\. L\. Andaur Navarro, P\. Dhiman,et al\.\(2025\)PROBAST\+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods\.BMJ388,pp\. e082505\.External Links:[Document](https://dx.doi.org/10.1136/bmj-2024-082505)Cited by:[§2\.1](https://arxiv.org/html/2609.16267#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[13\]C\. Hsu, S\. Karnwal, S\. Mullainathan, Z\. Obermeyer, and C\. Tan\(2020\)Characterizing the value of information in medical notes\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 2062–2072\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.187)Cited by:[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[14\]Z\. Yu, Z\. Dong, C\. Yu, K\. Yang, Z\. Fan, and C\. L\. P\. Chen\(2025\)A review on multi\-view learning\.Frontiers of Computer Science19\(7\),pp\. 197334\.External Links:[Document](https://dx.doi.org/10.1007/s11704-024-40004-w)Cited by:[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[15\]M\. M\. Mirończuk\(2026\)Document classification pattern recognition via information fusion: a systematic review of multimodal and multiview representation approaches\.Information Fusion132,pp\. 104247\.External Links:[Document](https://dx.doi.org/10.1016/j.inffus.2026.104247)Cited by:[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[16\]M\. M\. Mirończuk, J\. Protasiewicz, and W\. Pedrycz\(2019\)Empirical evaluation of feature projection algorithms for multi\-view text classification\.Expert Systems with Applications130,pp\. 97–112\.External Links:[Document](https://dx.doi.org/10.1016/j.eswa.2019.04.020)Cited by:[§2\.2](https://arxiv.org/html/2609.16267#S2.SS2.p1.1)\.
- \[17\]L\. Tanguy, N\. Tulechki, A\. Urieli, E\. Hermann, and C\. Raynal\(2016\)Natural language processing for aviation safety reports: from classification to interactive analysis\.Computers in Industry78,pp\. 80–95\.External Links:[Document](https://dx.doi.org/10.1016/j.compind.2015.09.005)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[18\]S\. D\. Robinson, W\. J\. Irwin, T\. K\. Kelly, and X\. O\. Wu\(2015\)Application of machine learning to mapping primary causal factors in self\-reported safety narratives\.Safety Science75,pp\. 118–129\.External Links:[Document](https://dx.doi.org/10.1016/j.ssci.2015.02.003)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[19\]A\. Nanyonga, H\. Wasswa, and G\. Wild\(2023\)Phase of flight classification in aviation safety using LSTM, GRU, and BiLSTM: a case study with ASN dataset\.In2023 International Conference on High Performance Big Data and Intelligent Systems \(HDIS\),Macau, China,pp\. 24–28\.External Links:[Document](https://dx.doi.org/10.1109/HDIS60872.2023.10499521)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[20\]C\. Chandra, X\. Jing, M\. V\. Bendarkar, K\. Sawant, L\. Elias, M\. Kirby, and D\. N\. Mavris\(2023\)Aviation\-BERT: a preliminary aviation\-specific natural language model\.InAIAA AVIATION 2023 Forum,Note:AIAA 2023\-3436External Links:[Document](https://dx.doi.org/10.2514/6.2023-3436)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[21\]A\. Tikayat Ray, B\. F\. Cole, O\. J\. Pinon Fischer, R\. T\. White, and D\. N\. Mavris\(2023\)AeroBERT\-classifier: classification of aerospace requirements using BERT\.Aerospace10\(3\),pp\. 279\.External Links:[Document](https://dx.doi.org/10.3390/aerospace10030279)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[22\]A\. Kumar, A\. Farahat, and C\. Gupta\(2025\)Predicting maintenance actions from historical logs using domain\-specific LLMs\.InProceedings of the Asia Pacific Conference of the PHM Society,Vol\.5\.Note:Published online 13 January 2026External Links:[Document](https://dx.doi.org/10.36001/phmap.2025.v5i1.4652)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[23\]X\. Li, F\. I\. Romli, S\. A\. Md Ali, M\. A\. Md Zhahir, and J\. Tang\(2026\)Domain\-adapted deep learning for aviation incident classification with multiple labels and risk assessment\.Engineering Applications of Artificial Intelligence173,pp\. 114454\.External Links:[Document](https://dx.doi.org/10.1016/j.engappai.2026.114454)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[24\]C\. Yang and C\. Huang\(2023\)Natural language processing \(NLP\) in aviation safety: systematic review of research and outlook into the future\.Aerospace10\(7\),pp\. 600\.External Links:[Document](https://dx.doi.org/10.3390/aerospace10070600)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[25\]A\. Nanyonga, K\. Joiner, U\. Turhan, and G\. Wild\(2025\)Applications of natural language processing in aviation safety: a review and qualitative analysis\.InAIAA SciTech 2025 Forum,Note:AIAA 2025\-2153; arXiv:2501\.06210External Links:[Document](https://dx.doi.org/10.2514/6.2025-2153)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[26\]M\. Kála, A\. Lališ, and T\. Vojtěch\(2022\)Analyzing aircraft maintenance findings with natural language processing\.Transportation Research Procedia65,pp\. 238–245\.External Links:[Document](https://dx.doi.org/10.1016/j.trpro.2022.11.028)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[27\]M\. J\. Scott, O\. Kirkpatrick, W\. Verhagen, V\. Kekoc, B\. Teunisse, J\. Zhang, H\. Fayek, and P\. Marzocca\(2024\)Application of natural language processing for aircraft defect tracking in maintenance operations\.InProceedings of the 34th Congress of the International Council of the Aeronautical Sciences \(ICAS\),Florence, Italy\.Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p1.1)\.
- \[28\]F\. Akhbardeh, T\. Desell, and M\. Zampieri\(2020\)MaintNet: a collaborative open\-source library for predictive maintenance language resources\.InProceedings of the 28th International Conference on Computational Linguistics: System Demonstrations,pp\. 7–11\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.coling-demos.2)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[29\]A\. Deloose, G\. Gysels, B\. De Baets, and J\. Verwaeren\(2023\)Combining natural language processing and multidimensional classifiers to predict and correct CMMS metadata\.Computers in Industry145,pp\. 103830\.External Links:[Document](https://dx.doi.org/10.1016/j.compind.2022.103830)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[30\]S\. M\. R\. Naqvi, M\. Ghufran, C\. Varnier, J\. Nicod, K\. Javed, and N\. Zerhouni\(2024\)Unlocking maintenance insights in industrial text through semantic search\.Computers in Industry157–158,pp\. 104083\.External Links:[Document](https://dx.doi.org/10.1016/j.compind.2024.104083)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[31\]B\. Hershowitz, M\. Hodkiewicz, T\. Bikaun, M\. Stewart, and W\. Liu\(2024\)Causal knowledge extraction from long text maintenance documents\.Computers in Industry161,pp\. 104110\.External Links:[Document](https://dx.doi.org/10.1016/j.compind.2024.104110)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[32\]V\. Giordano and G\. Fantoni\(2025\)Decomposing maintenance actions into sub\-tasks using natural language processing: a case study in an Italian automotive company\.Computers in Industry164,pp\. 104186\.External Links:[Document](https://dx.doi.org/10.1016/j.compind.2024.104186)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1)\.
- \[33\]A\. Crespo\-Márquez and J\. F\. Gómez Fernández\(2026\)Agentic AI for maintenance management: a process\-centric review and a staged framework for industrial adoption\.Computers & Industrial Engineering220,pp\. 112285\.External Links:[Document](https://dx.doi.org/10.1016/j.cie.2026.112285)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1),[§7](https://arxiv.org/html/2609.16267#S7.p6.1)\.
- \[34\]M\. A\. Farahani, M\. I\. Khan, and T\. Wuest\(2026\)Hybrid agentic AI and multi\-agent systems in smart manufacturing\.Journal of Manufacturing Systems86,pp\. 612–623\.External Links:[Document](https://dx.doi.org/10.1016/j.jmsy.2026.04.002)Cited by:[§2\.3](https://arxiv.org/html/2609.16267#S2.SS3.p2.1),[§7](https://arxiv.org/html/2609.16267#S7.p6.1)\.
- \[35\]M\. Pelt, K\. Stamoulis, and A\. Apostolidis\(2019\)Data analytics case studies in the maintenance, repair and overhaul \(MRO\) industry\.MATEC Web of Conferences304,pp\. 04005\.External Links:[Document](https://dx.doi.org/10.1051/matecconf/201930404005)Cited by:[§4\.1](https://arxiv.org/html/2609.16267#S4.SS1.p1.1)\.
- \[36\]T\. Sexton, M\. Hodkiewicz, M\. P\. Brundage, and T\. Smoker\(2018\)Benchmarking for keyword extraction methodologies in maintenance work orders\.InProceedings of the Annual Conference of the Prognostics and Health Management Society \(PHM\),Vol\.10\.External Links:[Document](https://dx.doi.org/10.36001/phmconf.2018.v10i1.541)Cited by:[§4\.1](https://arxiv.org/html/2609.16267#S4.SS1.p1.1)\.
- \[37\]O\. Candell, R\. Karim, and P\. Söderholm\(2009\)eMaintenance—information logistics for maintenance support\.Robotics and Computer\-Integrated Manufacturing25\(6\),pp\. 937–944\.External Links:[Document](https://dx.doi.org/10.1016/j.rcim.2009.04.005)Cited by:[§4\.1](https://arxiv.org/html/2609.16267#S4.SS1.p1.1)\.
- \[38\]NASA Aviation Safety Reporting SystemAbout ASRS data, ASRS database online\.Note:[https://asrs\.arc\.nasa\.gov/search/dbol/aboutdata\.html](https://asrs.arc.nasa.gov/search/dbol/aboutdata.html)Accessed August 2026Cited by:[§4\.2](https://arxiv.org/html/2609.16267#S4.SS2.p2.1)\.
- \[39\]NASA Aviation Safety Reporting SystemASRS program briefing\.Note:[https://asrs\.arc\.nasa\.gov/docs/ASRS\_ProgramBriefing\.pdf](https://asrs.arc.nasa.gov/docs/ASRS_ProgramBriefing.pdf)Data through December 2025; accessed August 2026Cited by:[§4\.2](https://arxiv.org/html/2609.16267#S4.SS2.p2.1)\.
- \[40\]National Highway Traffic Safety AdministrationVehicle safety recall completion rates report: report to Congress\.Note:[https://www\.nhtsa\.gov/sites/nhtsa\.dot\.gov/files/documents/13376\-recall\_completion\_rates\_rtc\-tag\_final\.pdf](https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/13376-recall_completion_rates_rtc-tag_final.pdf)Accessed August 2026Cited by:[§4\.3](https://arxiv.org/html/2609.16267#S4.SS3.p1.1)\.
- \[41\]National Highway Traffic Safety AdministrationNHTSA datasets and APIs: recalls\.Note:[https://www\.nhtsa\.gov/nhtsa\-datasets\-and\-apis](https://www.nhtsa.gov/nhtsa-datasets-and-apis)Accessed August 2026Cited by:[§4\.3](https://arxiv.org/html/2609.16267#S4.SS3.p1.1),[Data Availability](https://arxiv.org/html/2609.16267#Sx1.p1.1)\.
- \[42\]S\. Hochreiter and J\. Schmidhuber\(1997\)Long short\-term memory\.Neural Computation9\(8\),pp\. 1735–1780\.External Links:[Document](https://dx.doi.org/10.1162/neco.1997.9.8.1735)Cited by:[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[43\]M\. Schuster and K\. K\. Paliwal\(1997\)Bidirectional recurrent neural networks\.IEEE Transactions on Signal Processing45\(11\),pp\. 2673–2681\.External Links:[Document](https://dx.doi.org/10.1109/78.650093)Cited by:[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[44\]J\. Pennington, R\. Socher, and C\. D\. Manning\(2014\)GloVe: global vectors for word representation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1532–1543\.External Links:[Document](https://dx.doi.org/10.3115/v1/D14-1162)Cited by:[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[45\]T\. Mikolov, K\. Chen, G\. Corrado, and J\. Dean\(2013\)Efficient estimation of word representations in vector space\.arXiv preprint arXiv:1301\.3781\.Cited by:[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[46\]B\. Chiu, G\. Crichton, A\. Korhonen, and S\. Pyysalo\(2016\)How to train good word embeddings for biomedical NLP\.InProceedings of the 15th Workshop on Biomedical Natural Language Processing \(BioNLP\),pp\. 166–174\.External Links:[Document](https://dx.doi.org/10.18653/v1/W16-2922)Cited by:[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[47\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§5\.1](https://arxiv.org/html/2609.16267#S5.SS1.p1.1)\.
- \[48\]C\. Nadeau and Y\. Bengio\(2003\)Inference for the generalization error\.Machine Learning52\(3\),pp\. 239–281\.External Links:[Document](https://dx.doi.org/10.1023/A%3A1024068626366)Cited by:[§5\.2](https://arxiv.org/html/2609.16267#S5.SS2.p1.1)\.