The Normalization of Deviance in AI Development

# The Normalization of Deviance in AI Development
Source: [https://arxiv.org/html/2609.05749](https://arxiv.org/html/2609.05749)
###### Abstract

Work on the risks of artificial intelligence has focused predominantly on capability risk: the danger that systems become too powerful, too autonomous, or too misaligned with human values\. Far less attention has been paid to the organizational level—to whether the institutions building these systems are themselves predisposed to drift toward failure\. This paper argues that they are\. Regardless of how capable AI systems become, the organizations building them face the same structural dynamics that preceded past major technological disasters\. Drawing on case studies of the Space ShuttleChallenger, the Three Mile Island accident, and the Boeing 737 MAX crashes, this paper identifies the common structural mechanisms preceding each failure and maps them onto contemporary AI development\. The findings suggest that existing safety infrastructure may provide less protection than it appears, as organizations can complete safety processes in full compliance and still produce catastrophic outcomes\. The pre\-disaster period of AI development is still underway; the purpose of this paper is to make these dynamics legible while they can still be interrupted\.

## 1Introduction

Concerns about the risks of intelligent machines are a century and a half old, from Victorian\-era essayists to contemporary AI safety researchers, and have taken on sustained urgency since the release of GPT\-3\.5 in November 2022\. Those warnings have focused almost exclusively on capability risk, the danger of systems becoming too powerful, too autonomous, or too misaligned to control\. This paper argues that regardless of how capable AI systems become, the organizations building them may be structurally predisposed to drift toward failure, not through negligence or malice, but through organizational dynamics that have historically preceded major technological disasters\.

The implication is that existing AI safety infrastructure—internal safety teams, red\-teaming exercises, model evaluations, and responsible scaling policies—is not the protection it appears, as organizations can complete every safety process and*still*drift toward failure because the dynamics identified here operate through those same processes\. The argument proceeds as follows\. Section[2](https://arxiv.org/html/2609.05749#S2)develops the theoretical framework linking organizational structure to failure\. Section[3](https://arxiv.org/html/2609.05749#S3)turns to history, examining three technological disasters to isolate the structural pattern common to each\. Section[4](https://arxiv.org/html/2609.05749#S4)maps that pattern onto AI development, showing where the same dynamics are already operating\. Section[5](https://arxiv.org/html/2609.05749#S5)closes with recommendations aimed at the gap this pattern exposes\. Section[6](https://arxiv.org/html/2609.05749#S6)addresses the limits of the historical analogy, Section[7](https://arxiv.org/html/2609.05749#S7)outlines directions for empirical extension, and Section[8](https://arxiv.org/html/2609.05749#S8)concludes\.

## 2Background

Merton observed as early as 1936 that any system of purposive social action inevitably generates unanticipated consequences that run counter to its objectives[Merton \(1936\)](https://arxiv.org/html/2609.05749#bib.bib7)\. Building on this, Turner identified “failures of foresight” as the common precursor to large\-scale disasters, arguing that early warning signs were routinely overlooked because of cultural beliefs about hazards and their avoidance[Turner \(1978\)](https://arxiv.org/html/2609.05749#bib.bib8)\. Hughes extended this insight to large\-scale sociotechnical systems, showing how assemblages of actors, technologies, and organizations produce complexity that is beyond the intentions of any single designer[Hughes \(1979\)](https://arxiv.org/html/2609.05749#bib.bib9);[Hughes \(1993\)](https://arxiv.org/html/2609.05749#bib.bib10)\. Perrow drew the sharpest conclusion from this tradition, arguing that in tightly coupled, complex systems, catastrophic failure is not an anomaly but a structural inevitability built into the system’s architecture[Perrow \(1984\)](https://arxiv.org/html/2609.05749#bib.bib11)\. Sagan broadened the frame, showing how political pressure and bureaucratic interest within larger institutional contexts shape the behavior of organizational units in ways that compound these dynamics[Sagan \(1993\)](https://arxiv.org/html/2609.05749#bib.bib40)\. Vaughan synthesized and extended this tradition by centering organizational culture and social structure, explaining not just how failures happen but why the people closest to them so often fail to see them coming[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\. Dekker added a temporal dimension, describing how organizations under sustained competitive pressure drift incrementally toward the boundary of safe operation, with each small adaptation making the next one easier to rationalize[Dekker \(2011\)](https://arxiv.org/html/2609.05749#bib.bib6)\. Together, these contributions locate the origin of catastrophic failure in the ordinary dynamics of organizations under commercial and political pressure\.

Out of this tradition emerges the framework central to the present analysis\. This process, termed thenormalization of deviance, describes the incremental organizational drift by which a practice that violates safety norms becomes, through repeated non\-disaster, redefined as normal—a concept developed through a landmark study of the 1986 Space ShuttleChallengerdisaster[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\. A critical feature of this framework is that normalization of deviance does not require villains\. The engineers and managers who authorized the launch were competent, experienced, and they sincerely believed the launch was safe\. Their belief was wrong, but it emerged from an organizational context that had systematically supplied them with information and frameworks that supported it[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2)\. This stands in contrast to the “amoral calculator” model of organizational failure[Carroll \(1987\)](https://arxiv.org/html/2609.05749#bib.bib4);[Kagan and Scholz \(1984\)](https://arxiv.org/html/2609.05749#bib.bib5), in which decision\-makers consciously weigh the costs and benefits of violating safety norms and choose to proceed when benefits outweigh costs[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2)\. Normalization of deviance does not originate in individual calculation of this kind\. Rather, it originates in organizational structure and the culture that structure sustains\.

The organizational failure literature identifies four mechanisms that reliably appear across instances of normalized deviance[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1);[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2);[Dekker \(2011\)](https://arxiv.org/html/2609.05749#bib.bib6)\.Production pressurecreates incentives to minimize the friction that safety processes impose, progressively shifting the burden of proof onto those arguing for caution\.False assurance from prior successtreats the absence of disaster as evidence of safety, a logically invalid inference that is nonetheless organizationally compelling\.Structural secrecydescribes how organizational structure itself impedes the flow of safety\-relevant information through the natural consequences of division of labor and hierarchical reporting[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2)\.Erosion of independent oversightoccurs as the relationship between regulators and regulated industries becomes, over time, less adversarial and more collaborative, vitiating the independence that gives oversight its value\.

These mechanisms apply with particular force to emerging technologies, where safety standards are developed alongside deployment, with no stock of operational experience to derive them from[Vaughan \(2004\)](https://arxiv.org/html/2609.05749#bib.bib3)\. A systematic review of thirty\-three studies across oil and gas, nuclear, aviation, healthcare, and rail industries found consistent evidence of these dynamics in every sector examined, suggesting they belong to the structure of organizations managing risk under competitive pressure, whatever the domain[Sedlar et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib12)\. In mature technological domains, norms are built up over decades, often at the cost of prior accidents\. In an emerging technology, this process is foreshortened, or absent entirely\. Technologies are easiest to control before their effects are understood, but their effects are not understood until they are widely deployed[Collingridge \(1980\)](https://arxiv.org/html/2609.05749#bib.bib13);[Genus and Stirling \(2018\)](https://arxiv.org/html/2609.05749#bib.bib61)\. With no mature standard to measure against, industry practice becomes the baseline[David \(1985\)](https://arxiv.org/html/2609.05749#bib.bib59);[Arthur \(1989\)](https://arxiv.org/html/2609.05749#bib.bib60)\. This is the condition in which AI development currently finds itself\.

## 3Case Studies

The cases were selected for their records\. Each is among the most exhaustively investigated organizational failures of the past half\-century, and that documentation is what the study of normalization requires and what intact organizations withhold\. The cases span independent industries and decades, and each foregrounds a different mechanism\. They establish how the dynamics operate, not how often they end in failure\.

### 3\.1The Challenger Launch Decision \(1986\)

On the night of January 27, 1986, engineers from Morton Thiokol conducted an emergency teleconference with NASA administrators\. They argued against the following morning’s launch, citing the below\-freezing forecast and a concern that the O\-ring seals in the Solid Rocket Boosters could fail\. Their concerns were overruled\. Hours later,Challengerlifted off, and 73 seconds into the flight, it broke apart, killing all seven astronauts[Higginbotham \(2024\)](https://arxiv.org/html/2609.05749#bib.bib14);[Rogers et al\. \(1986\)](https://arxiv.org/html/2609.05749#bib.bib16);[McDonald \(2012\)](https://arxiv.org/html/2609.05749#bib.bib15)\.

The concept of the normalization of deviance emerged directly from analysis of this disaster[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\. In the years leading up toChallenger, engineers had openly observed and documented O\-ring erosion, but each flight that survived without catastrophe was read, retrospectively, as confirming the safety case\. Over time, the organizational interpretation of this evidence shifted\. Production pressure, driven by a launch schedule that had already slipped multiple times, had progressively reversed the burden of proof: where safety once had to be established before flight, those urging caution now had to prove the launch unsafe in order to halt it\. False assurance from prior success closed the argument\. Each uneventful flight had made the next approval easier\.

### 3\.2The Accident at Three Mile Island \(1979\)

The partial meltdown at Three Mile Island Unit 2 on March 28, 1979, began with a stuck\-open pilot\-operated relief valve in a reactor system that had accumulated years of minor anomalies, each handled through improvised workarounds that had gradually calcified into standard practice[Walker \(2004\)](https://arxiv.org/html/2609.05749#bib.bib18);[Perrow \(1981\)](https://arxiv.org/html/2609.05749#bib.bib19)\. The President’s Commission concluded that the disaster resulted from deficiencies in how the Nuclear Regulatory Commission and the nuclear industry had approached reactor safety[President’s Commission on the Accident at Three Mile Island \(1979\)](https://arxiv.org/html/2609.05749#bib.bib17)\. The organizational environment had normalized the management of complexity through improvisation, and that normalization was what made the accident possible[Perrow \(1981\)](https://arxiv.org/html/2609.05749#bib.bib19)\. Three Mile Island is most often read through normal accident theory, as a failure of interactive complexity and tight coupling[Perrow \(1984\)](https://arxiv.org/html/2609.05749#bib.bib11)\. The reading here is complementary as the informal workaround culture that preceded the accident is what normalization of deviance looks like in an operating environment\.

In an organizational context, Three Mile Island foregrounds structural secrecy and opacity\. The reactor systems had grown in complexity as modifications were layered on modifications, and operators in the control room could not, in real time, form an accurate mental model of what was happening inside the reactor\. When the relief valve stuck open, a cascade of confusing and partially contradictory indicator readings meant consequential decisions were made on an incorrect understanding of the system’s state\. Operators had been trained to handle the anomalies they expected, and the combination of anomalies that actually occurred fell entirely outside that training[Perrow \(1981\)](https://arxiv.org/html/2609.05749#bib.bib19);[Perrow \(1984\)](https://arxiv.org/html/2609.05749#bib.bib11);[Walker \(2004\)](https://arxiv.org/html/2609.05749#bib.bib18)\.

Operators had developed habits of handling routine anomalies informally, without systematic reporting or analysis, because treating every anomaly as a potential crisis was organizationally unsustainable\. The informal workarounds worked until they did not\. By the time the accident occurred, the gap between formal operating procedures and actual operational practice was wide enough that operators lacked the procedural and cognitive resources to recognize and respond correctly to what was happening\. Safety\-relevant information existed within the system\. It simply never reached those with the authority and knowledge to act on it\.

### 3\.3The Boeing 737 MAX Crashes \(2018–2019\)

The crashes of Lion Air Flight 610 in October 2018 and Ethiopian Airlines Flight 302 in March 2019 were caused by a software system called MCAS \(Maneuvering Characteristics Augmentation System\) that repeatedly pushed the nose of the aircraft down in response to faulty sensor data[Robison \(2022\)](https://arxiv.org/html/2609.05749#bib.bib21)\. MCAS was designed to compensate for aerodynamic handling differences created by installing larger engines on an airframe not designed to accommodate them\. It was a software patch for a physical design flaw, and the pilots of both flights were unable to override the system before impact\.

The origins of this design trajectory lie in competitive pressure[Johnston and Harris \(2019\)](https://arxiv.org/html/2609.05749#bib.bib22);[MacArthur \(2020\)](https://arxiv.org/html/2609.05749#bib.bib23)\. When Airbus announced the A320neo, Boeing faced a choice: design a new aircraft or modify the existing 737 platform\. Modification was faster and cheaper, and would allow the aircraft to be certified as a variant of the existing 737 rather than as an entirely new airplane, avoiding costly simulator training requirements for airline customers[U\.S\. House Committee on Transportation and Infrastructure \(2020\)](https://arxiv.org/html/2609.05749#bib.bib20);[Hopkins \(2025\)](https://arxiv.org/html/2609.05749#bib.bib24)\. Boeing chose modification and committed to a path in which a series of engineering compromises became increasingly difficult to walk back\. MCAS was initially a minor system designed to engage only in rare high\-angle\-of\-attack situations \(nose pitched steeply upward\)\. Over the course of development, its authority was expanded and it was made to rely on input from a single angle\-of\-attack sensor, making it vulnerable to a single\-point failure\. Each change was accompanied by an assessment that the modification was acceptable[Herkert et al\. \(2020\)](https://arxiv.org/html/2609.05749#bib.bib25)\.

The FAA’s role illustrates erosion of independent oversight in its most developed form\. Boeing employees acting as FAA Authorized Representatives conducted much of the safety analysis and certification work that the FAA nominally oversaw[U\.S\. House Committee on Transportation and Infrastructure \(2020\)](https://arxiv.org/html/2609.05749#bib.bib20)\. The institutional trust that had developed between Boeing and the FAA over decades of generally successful collaboration had become a substitute for independent verification[Robison \(2022\)](https://arxiv.org/html/2609.05749#bib.bib21)\. The Congressional investigation found that FAA managers had overruled their own engineers when those engineers raised concerns about the certification process[U\.S\. House Committee on Transportation and Infrastructure \(2020\)](https://arxiv.org/html/2609.05749#bib.bib20)\. The organizations nominally responsible for preventing normalization had become participants in it\.

## 4Normalization of Deviance in AI Development

The three cases examined above share a common structural pattern: an extended period of drift during which warning signals mounted and were progressively redefined as acceptable, enabled by production pressure, false assurance from prior success, structural secrecy, and erosion of independent oversight\. Each mechanism is observable in contemporary AI development, and several features of the AI context make the conditions more acute than in prior technological domains and also compounded\. Mechanisms that appeared singly in the historical cases appear together here\. NASA drifted inside mature procedures, and Boeing certified against five decades of experience with the 737; the frontier laboratories are adolescent institutions, none much more than a decade old, operating all four mechanisms at once on systems less legible than any their predecessors managed\.

Production pressurein AI development—commercial competition among a handful of frontier laboratories, overlaid by strategic rivalry between the United States and China—operates at an intensity that has few historical precedents outside of wartime[Armstrong et al\. \(2016\)](https://arxiv.org/html/2609.05749#bib.bib41);[Rhodes \(1986\)](https://arxiv.org/html/2609.05749#bib.bib39);[Sagan \(1993\)](https://arxiv.org/html/2609.05749#bib.bib40)\. The economic and reputational stakes attached to being first create sustained pressure to treat safety work as friction, not value[Felstead \(2025\)](https://arxiv.org/html/2609.05749#bib.bib43);[Armstrong et al\. \(2016\)](https://arxiv.org/html/2609.05749#bib.bib41);[Cave and ÓhÉigeartaigh \(2018\)](https://arxiv.org/html/2609.05749#bib.bib42)\. Safety evaluations that might delay a release, red\-teaming exercises whose findings might block a product, and interpretability research that reveals concerning model behaviors each represent a potential impediment to competitive position\. The burden of proof has shifted in a manner structurally identical to what occurred at NASA: those arguing for caution must overcome those arguing for speed, rather than the reverse[Armstrong et al\. \(2016\)](https://arxiv.org/html/2609.05749#bib.bib41);[Felstead \(2025\)](https://arxiv.org/html/2609.05749#bib.bib43);[Bostrom \(2014\)](https://arxiv.org/html/2609.05749#bib.bib26);[Cave and ÓhÉigeartaigh \(2018\)](https://arxiv.org/html/2609.05749#bib.bib42)\. At the state level, AI capability has been explicitly framed as a matter of national security and economic survival[State Council, People’s Republic of China \(2017\)](https://arxiv.org/html/2609.05749#bib.bib37);[The White House \(2024\)](https://arxiv.org/html/2609.05749#bib.bib45);[Horowitz \(2018\)](https://arxiv.org/html/2609.05749#bib.bib44)\.

False assurance from prior successmanifests in AI development through benchmark performance and deployment history[Raji et al\. \(2020\)](https://arxiv.org/html/2609.05749#bib.bib28);[Bengio and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib46);[Stanovsky et al\. \(2025\)](https://arxiv.org/html/2609.05749#bib.bib47)\. AI systems are assessed using standardized evaluation suites, and performance on those benchmarks is treated as safety evidence[Liang et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib38);[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.05749#bib.bib27)\. But the history of AI benchmarks is a history of systems that perform well on evaluations while exhibiting concerning behaviors the benchmarks were not designed to detect[Raji et al\. \(2020\)](https://arxiv.org/html/2609.05749#bib.bib28);[Ribeiro et al\. \(2020\)](https://arxiv.org/html/2609.05749#bib.bib29)\. When a model passes its evaluations and is deployed without incident, this is treated as validation of both the model and the evaluation framework\. Each successive generation of AI systems that is deployed without catastrophe expands the implicit safety baseline for the next, making it progressively harder to argue for more stringent standards as capabilities increase[Bommasani et al\. \(2021\)](https://arxiv.org/html/2609.05749#bib.bib30)\. This is particularly dangerous as we begin to uncover increasing evidence of frontier models scheming on a much larger scale than we had previously realized[OpenAI \(2026\)](https://arxiv.org/html/2609.05749#bib.bib58)\. This concern is no longer hypothetical\. OpenAI’s report on the July 2026 Hugging Face incident documents evaluation\-stage agents escaping containment through chained exploits and compromising third\-party production infrastructure[OpenAI \(2026\)](https://arxiv.org/html/2609.05749#bib.bib58)\.

Structural secrecyfollows from the ordinary organization of frontier AI laboratories[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2);[Anderljung et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib33);[Cihon and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib48)\. Safety teams, product teams, and leadership operate in different organizational contexts with different incentive structures and different information environments[Delaney et al\. \(2024\)](https://arxiv.org/html/2609.05749#bib.bib49);[Cihon and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib48)\. Concerns raised by safety researchers may not reach or influence deployment decisions made under competitive pressure\. The pattern is documented across multiple frontier organizations in the public departures of safety\-focused researchers citing the prioritization of products over safety[Goldman and Kahn \(2024\)](https://arxiv.org/html/2609.05749#bib.bib52);[Tech Brew \(2026\)](https://arxiv.org/html/2609.05749#bib.bib50);[Ethical AI Departures \(2024\)](https://arxiv.org/html/2609.05749#bib.bib51)\. Red\-team findings may be documented without carrying the authority to change outcomes[Ganguli et al\. \(2022\)](https://arxiv.org/html/2609.05749#bib.bib31);[Perez et al\. \(2022\)](https://arxiv.org/html/2609.05749#bib.bib32)\. The consequence of division of labor and hierarchical reporting is that safety\-relevant information fails to reach those with authority to act on it\.

Erosion of independent oversightis perhaps the most advanced of the four mechanisms in AI development[Rost \(2026\)](https://arxiv.org/html/2609.05749#bib.bib53);[Ho et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib34);[Anderljung et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib33)\. The governance infrastructure for AI is embryonic by historical standards, and the organizations with the strongest interest in less stringent standards have the greatest influence over how those standards are set[Anderljung et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib33);[Ho et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib34);[Wei et al\. \(2024\)](https://arxiv.org/html/2609.05749#bib.bib54)\. The expertise needed to evaluate frontier systems is generated by the work of building them, so it accumulates where development happens and nowhere else\. Every year of frontier progress widens the gap between what laboratories can assess and what any outside body can[Cihon and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib48)\. Responsible scaling policies and model evaluations are defined, conducted, and enforced by the same organizations whose deployment decisions they are meant to constrain[Coggins et al\. \(2025\)](https://arxiv.org/html/2609.05749#bib.bib55);[Rost \(2026\)](https://arxiv.org/html/2609.05749#bib.bib53);[OpenAI \(2023\)](https://arxiv.org/html/2609.05749#bib.bib35);[OpenAI \(2025\)](https://arxiv.org/html/2609.05749#bib.bib62)\. This is the self\-certification dynamic that the Congressional investigation identified as a central institutional cause of the 737 MAX crashes[Robison \(2022\)](https://arxiv.org/html/2609.05749#bib.bib21);[U\.S\. House Committee on Transportation and Infrastructure \(2020\)](https://arxiv.org/html/2609.05749#bib.bib20), and it is the current default governance model for frontier AI development[Hadfield and Clark \(2026\)](https://arxiv.org/html/2609.05749#bib.bib36);[Anderljung et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib33);[Rost \(2026\)](https://arxiv.org/html/2609.05749#bib.bib53)\. When the entity defining safety standards has an institutional stake in the content of those standards, normalization of deviance can proceed regardless of individual or institutional good faith[Anderljung et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib33)\.

## 5Recommendations

The preceding analysis suggests that normalization of deviance in AI development is an organizational, cultural, governance, and technical problem\. Because normalization operates through rule\-following rather than rule\-breaking, interventions that merely add new rules, compliance requirements, or accountability mechanisms are unlikely to be sufficient on their own\. The recommendations below are organized around the four mechanisms identified above, with each designed to address a structural condition\.

Organizational\.The common thread in all three case studies is that safety functions lost practical independence from production incentives\. At NASA, safety engineers reported to the same management chain under pressure to launch[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\. At the FAA, the self\-certification model embedded the certifying authority within the organization whose products it was certifying[Robison \(2022\)](https://arxiv.org/html/2609.05749#bib.bib21)\. Safety functions require structural independence from the processes that create production pressure\. In AI development, this implies safety organizations that report through structures insulated from deployment timelines, with authority to delay or halt deployment that is not subject to override by the same management chain under pressure to ship[Cihon and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib48);[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\. It also implies that safety findings are reported externally as well as internally, so that structural secrecy cannot operate unchecked\. Pre\-mortem analysis conducted before deployment directly addresses the false assurance mechanism\. Requiring organizations to prospectively construct the plausible chain of decisions that could produce failure forces implicit risk tolerances to become explicit[Klein \(2007\)](https://arxiv.org/html/2609.05749#bib.bib57)\.

Epistemic and cultural\.Organizations cannot defend against normalization of deviance if they do not recognize it as a distinct phenomenon\. The amoral calculator model attributes failures to bad actors making bad calculations, providing a more comfortable explanation than the normalization of deviance account[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1);[Kagan and Scholz \(1984\)](https://arxiv.org/html/2609.05749#bib.bib5)\. Naming the dynamic explicitly and incorporating it into the safety culture and training of AI development organizations is a prerequisite for the other interventions to function[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2);[Sedlar et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib12)\. Concretely, this means building epistemic cultures in which prior successful deployments are treated as precedent rather than proof\. Each new generation of more capable systems requires its own affirmative safety evidence, independent of the deployment history that preceded it[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1);[Dekker \(2011\)](https://arxiv.org/html/2609.05749#bib.bib6)\. It means treating anomalous model behaviors as meaningful signals, with the institutional expectation that they are systematically investigated, and not absorbed into an existing risk category[Vaughan \(1999\)](https://arxiv.org/html/2609.05749#bib.bib2);[Sedlar et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib12)\. The institutional protection of dissenting voices deserves special emphasis\. In all three cases examined, individuals within the organization raised concerns through appropriate channels and were overruled through appropriate processes\. The organizational structure lacked the capacity to give dissent due weight\. Mechanisms that give safety concerns procedural standing, requiring that safety objections be formally documented, responded to in writing, and escalated independently of the management chain under production pressure, address this structural failure directly[Ganguli et al\. \(2022\)](https://arxiv.org/html/2609.05749#bib.bib31);[Delaney et al\. \(2024\)](https://arxiv.org/html/2609.05749#bib.bib49)\.

Regulatory and governance\.The 737 MAX case is the clearest illustration of why industry self\-regulation, however well\-intentioned, is structurally insufficient to prevent normalization\. Boeing’s engineers knew the risks, their safety processes identified concerns, and internal culture systematically overrode those concerns under commercial pressure\. The FAA’s self\-certification regime provided the institutional framework within which this override could proceed with apparent legitimacy\. Strong internal safety culture is necessary but requires external, independent oversight to function as intended\. For AI development, this implies third\-party evaluation infrastructure that does not depend on developer cooperation for access, funded through sources other than the organizations it evaluates, and empowered to conduct and publish its own assessments of safety\-relevant model properties[Anderljung et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib33);[Ho et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib34);[Wei et al\. \(2024\)](https://arxiv.org/html/2609.05749#bib.bib54);[Cihon and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib48)\. The National Transportation Safety Board \(NTSB\) model is instructive\. Its investigations are independent of the FAA and of industry, and its findings are published in full\. An AI analog does not need to replicate every feature of the NTSB to break the secrecy that lets normalization proceed inside the closed loop of industry self\-assessment\. Incident reporting systems modeled on the Aviation Safety Reporting System \(ASRS\), anonymous, no\-fault, and focused on systemic learning, directly address the same dynamic, creating channels through which safety\-relevant information flows toward analysis instead of remaining captured where it was produced[Hadfield and Clark \(2026\)](https://arxiv.org/html/2609.05749#bib.bib36);[Ho et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib34)\.

Technical\.Both the Three Mile Island case and Section[4](https://arxiv.org/html/2609.05749#S4)identify opacity as a primary driver of normalization: when operators cannot see the system, improvisation fills the gap, and improvisation normalizes\. The TMI operators could not form an accurate mental model of the reactor’s state; AI safety evaluators cannot verify that the properties producing safe behavior in test conditions will produce safe behavior in deployment\. Interpretability research should therefore be treated as a safety prerequisite and funded accordingly[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.05749#bib.bib27);[Bengio and others \(2025\)](https://arxiv.org/html/2609.05749#bib.bib46)\. Staged deployment with enforceable rollback criteria addresses the deployment normalization dynamic\. Staged deployment that treats advancing to each stage as the default and halting as requiring affirmative evidence of failure replicates the burden\-of\-proof reversal that was the defining symptom of normalization at NASA\. Conditionally staged deployment, in which advancing requires affirmative evidence of safety and the authority to halt is structurally independent from the authority to ship, confronts the same mechanism that theChallengerpre\-launch review failed to address[Shevlane et al\. \(2023\)](https://arxiv.org/html/2609.05749#bib.bib56);[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\.

## 6Limitations

The argument advanced here depends on a structural analogy between AI development organizations and the organizations that produced theChallenger, Three Mile Island, and Boeing 737 MAX disasters\. This analogy is the paper’s central contribution, but it is not without weaknesses\.

The most fundamental limitation is epistemic, and it is invidious—the framework’s full evidentiary record only ever exists after the failure it predicts\. The dynamic is invisible to those undergoing it, and disaster is what makes the accumulation of small decisions visible\. This paper therefore cannot offer a smoking gun\. The structural conditions identified here are consistent with normalization of deviance occurring in AI development today, but consistency is not proof\. The documented departures of safety researchers, the revision histories of responsible scaling policies, and the gap between benchmark performance and deployment safety are each consistent with normalization—they are also consistent with organizations learning and adapting under difficult conditions\. Distinguishing between these interpretations requires evidence that is not yet available\. One partial discriminator does exist: learning and normalization predict different directional patterns across a growing corpus of policy revisions, with learning tracking new safety information and normalization tracking competitive events\.

A second limitation concerns the boundaries of the analogy\. The normalization of deviance framework was developed for physical engineering systems where failure modes are relatively legible \(e\.g\., an O\-ring can be inspected, erosion can be measured, and a causal chain can be reconstructed after the fact\)\. The failure modes of AI systems are more diffuse, the relevant properties less measurable, and the distinction between a concerning anomaly and normal system behavior often simply unclear\. If anything, the disanalogy heightens the concern, since what cannot be seen cannot be corrected\. But the mapping is imperfect, and the framework has to be applied with that in view\.

A third limitation is that the AI safety community is substantially more self\-aware about these risks than NASA, the NRC, or Boeing were at comparable stages\. The field of AI safety exists as a recognized discipline; major development organizations employ substantial safety teams; the intellectual frameworks for thinking about misalignment, evaluation failure, and organizational risk have been developed and published widely\. Several of the laboratories responsible for these risks are also their most prominent public chroniclers and advocates—safety commentary is, for some, part of the commercial identity\. This paper does not claim that AI development organizations are indifferent to safety\. The argument is that awareness of a risk is insufficient to prevent normalization when the organizational structures that produce it remain intact, and this distinction is easier to assert than to verify empirically\.

Finally, the paper conflates corporate\-level and state\-level AI development to some degree, treating them as instances of the same phenomenon operating at different scales\. The organizational dynamics and incentive structures relevant to a commercial AI laboratory differ substantially from those relevant to a state\-level AI program\. A fuller treatment would separate the two and work out the governance implications of each\.

## 7Future Work

The argument advanced here is necessarily preliminary\. To the authors’ knowledge, this is the first sustained application of the normalization of deviance literature to frontier AI development, and its primary contribution is conceptual rather than empirical\. The historical cases demonstrate the organizational dynamics that produce normalization, but do not by themselves establish that the same process is occurring within AI organizations\. Future work should therefore be empirical\. Three approaches seem most tractable\.

First, longitudinal analysis of frontier laboratories’ governance artifacts, including responsible scaling policies, preparedness frameworks, and model cards, treating successive revisions as a corpus whose direction of change can be coded and correlated with competitive events and safety incidences\. Under normalization, procedural language would be expected to become more permissive over time and competitive\-adjustment provisions more entrenched; under learning, capability thresholds would be qualified or strengthened\. These predictions can be tested against the public revision histories already available\.

Second, systematic analysis of personnel testimony\. The departures of safety and governance researchers from frontier laboratories between 2024 and 2026, notably Daniel Kokotajlo, Jan Leike, and William Saunders, have produced an unusually public record of first\-hand accounts, several of which describe precisely the pattern of interest: standards eroding through the accumulation of deployments in which nothing visibly went wrong[Kokotajlo \(2024\)](https://arxiv.org/html/2609.05749#bib.bib65);[Leike \(2024\)](https://arxiv.org/html/2609.05749#bib.bib63);[Saunders \(2024\)](https://arxiv.org/html/2609.05749#bib.bib64)\. Treated as evidence in aggregate and read longitudinally, these accounts can be situated on the same timeline as governance artifact revisions and competitive events, with convergence across independent accounts corroborating the normalization interpretation\.

Third, extension to the state level\. Nation\-states competing over AI capability are themselves organizations under production pressure, and the framework applied to them directly[Sagan \(1993\)](https://arxiv.org/html/2609.05749#bib.bib40);[Horowitz \(2018\)](https://arxiv.org/html/2609.05749#bib.bib44)\. National strategies, export control rules, and procurement and testing standards are published and revised in ways tractable to the same analysis proposed above, and the comparative question of whether safety commitments erode differently under state principals than under market discipline would begin to separate the corporate and state cases this paper has treated together\.

## 8Conclusion

The normalization of deviance is a story about ordinary people, following ordinary procedures, inside well\-intentioned organizations, collectively drifting toward catastrophic failure\. Analysis of theChallengerdisaster revealed that the organizational conditions producing catastrophe are structural—the predictable output of institutions managing risk under competitive pressure without adequate independence for their safety functions[Vaughan \(1996\)](https://arxiv.org/html/2609.05749#bib.bib1)\. Three Mile Island and the Boeing 737 MAX confirmed that this dynamic generalizes across industries and decades\. This paper has argued that the structural conditions enabling normalization of deviance are present in AI development in acute form\. Production pressure, false assurance from prior success, structural secrecy, and erosion of independent oversight are each observable in AI development today\.

The opacity of AI systems makes warning signals harder to read, and the immaturity of the regulatory environment leaves safety norms to be written by the developers themselves\. Competitive pressure at both the corporate and state levels creates sustained incentives to treat safety work as friction\. Practices that would register as deviant against a mature standard simply become the standard\. The disasters examined in this paper were catastrophic\. They were also legible: investigators could determine what had happened, attribute it to organizational causes, and construct institutional reforms designed to prevent recurrence\. A sufficiently large AI safety failure may produce consequences that are neither recoverable nor legible enough to serve as a corrective, foreclosing the institutional learning that followed prior disasters\.

The AI development community is in the period analogous to the years before the major accidents examined here, when norms are still being written and institutions are still being designed\. The organizational, cultural, regulatory, and technical recommendations offered here share a common logic: safety infrastructure built before competitive pressures reach their peak is more robust than infrastructure constructed in response to failure\. Whether the current moment is used as an opportunity depends on whether the dynamics identified here are taken seriously by those best positioned to act on them\.

## References

- Anderljunget al\.\(2023\)M\. Anderljung, J\. Barnhart, A\. Korinek,et al\.Frontier ai regulation: managing emerging risks to public safety\.arXiv preprint arXiv:2307\.03718\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1),[§4](https://arxiv.org/html/2609.05749#S4.p5.1),[§5](https://arxiv.org/html/2609.05749#S5.p4.1)\.
- Armstronget al\.\(2016\)S\. Armstrong, N\. Bostrom, and C\. ShulmanRacing to the precipice: a model of artificial intelligence development\.Technical reportFuture of Humanity Institute, University of Oxford\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Arthur \(1989\)W\. B\. ArthurCompeting technologies, increasing returns, and lock\-in by historical events\.The economic journal99\(394\),pp\. 116–131\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p4.1)\.
- Bengioet al\.\(2025\)Y\. Bengioet al\.International AI Safety Report\.Technical reportInternational AI Safety Report\.External Links:[Link](https://arxiv.org/abs/2602.21012)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1),[§5](https://arxiv.org/html/2609.05749#S5.p5.1)\.
- Bommasaniet al\.\(2021\)R\. Bommasani, D\. A\. Hudson, E\. Aditi,et al\.On the Opportunities and Risks of Foundation Models\.arXiv preprint arXiv:2108\.07258\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1)\.
- Bostrom \(2014\)N\. BostromSuperintelligence: Paths, Dangers, Strategies\.Oxford University Press,Oxford, UK\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Carroll \(1987\)A\. B\. CarrollIn search of the moral manager\.Business Horizons30\(2\),pp\. 7–15\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p2.1)\.
- Cave and ÓhÉigeartaigh \(2018\)S\. Cave and S\. S\. ÓhÉigeartaighAn AI Race for Strategic Advantage: Rhetoric and Risks\.InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society,pp\. 36–40\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Cihonet al\.\(2025\)P\. Cihonet al\.Frontier ai auditing: toward rigorous third\-party assessment of safety and security practices at leading ai companies\.arXiv preprint arXiv:2601\.11699\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1),[§4](https://arxiv.org/html/2609.05749#S4.p5.1),[§5](https://arxiv.org/html/2609.05749#S5.p2.1),[§5](https://arxiv.org/html/2609.05749#S5.p4.1)\.
- Cogginset al\.\(2025\)S\. Coggins, A\. K\. Saeri, K\. A\. Daniell, L\. P\. Ruster, J\. Liu, and J\. L\. DavisThe 2025 OpenAI Preparedness Framework does not guarantee any AI risk mitigation practices: a proof\-of\-concept for affordance analyses of AI safety policies\.arXiv preprint arXiv:2509\.24394\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1)\.
- Collingridge \(1980\)D\. CollingridgeThe Social Control of Technology\.Frances Pinter,London\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p4.1)\.
- David \(1985\)P\. A\. DavidClio and the Economics of QWERTY\.The American economic review75\(2\),pp\. 332–337\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p4.1)\.
- Dekker \(2011\)S\. DekkerDrift into failure: From hunting broken components to understanding complex systems\.CRC press\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1),[§2](https://arxiv.org/html/2609.05749#S2.p3.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1)\.
- Delaneyet al\.\(2024\)O\. Delaney, O\. Guest, and Z\. WilliamsMapping technical safety research at ai companies: a literature review and incentives analysis\.Technical reportApart Research\.External Links:[Link](https://arxiv.org/abs/2409.07878)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1)\.
- Ethical AI Departures \(2024\)Ethical AI DeparturesEthical ai departures: documented ai safety and ethics departures\.External Links:[Link](https://ethicalaidepartures.fyi/)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1)\.
- Felstead \(2025\)N\. FelsteadEnabling frontier lab collaboration to mitigate ai safety risks\.arXiv preprint arXiv:2511\.08631\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Ganguliet al\.\(2022\)D\. Ganguli, L\. Lovitt, J\. Kernion,et al\.Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned\.arXiv preprint arXiv:2209\.07858\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1)\.
- Genus and Stirling \(2018\)A\. Genus and A\. StirlingCollingridge and the dilemma of control: Towards responsible and accountable innovation\.Research policy47\(1\),pp\. 61–69\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p4.1)\.
- Goldman and Kahn \(2024\)S\. Goldman and J\. KahnTop openai researcher resigns, saying company prioritized ‘shiny products’ over ai safety\.Fortune\.External Links:[Link](https://fortune.com/2024/05/17/openai-researcher-resigns-safety)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1)\.
- Hadfield and Clark \(2026\)G\. K\. Hadfield and J\. ClarkRegulatory Markets: The Future of AI Governance\.Jurimetrics Journal65,pp\. 195–240\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1),[§5](https://arxiv.org/html/2609.05749#S5.p4.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, N\. Carlini, J\. Schulman, and J\. SteinhardtUnsolved Problems in ML Safety\.arXiv preprint arXiv:2109\.13916\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1),[§5](https://arxiv.org/html/2609.05749#S5.p5.1)\.
- Herkertet al\.\(2020\)J\. Herkert, J\. Borenstein, and K\. MillerThe Boeing 737 MAX: Lessons for Engineering Ethics\.Science and engineering ethics26\(6\),pp\. 2957–2974\.Cited by:[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p2.1)\.
- Higginbotham \(2024\)A\. HigginbothamChallenger: A True Story of Heroism and Disaster on the Edge of Space\.Random House\.Cited by:[§3\.1](https://arxiv.org/html/2609.05749#S3.SS1.p1.1)\.
- Hoet al\.\(2023\)L\. Ho, J\. Barnhart, R\. Trager,et al\.International Institutions for Advanced AI\.arXiv preprint arXiv:2307\.04699\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1),[§5](https://arxiv.org/html/2609.05749#S5.p4.1)\.
- Hopkins \(2025\)A\. HopkinsBoeing, the 737 MAX crisis and aviation safety: The perils of profit\-driven engineering\.CRC Press\.Cited by:[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p2.1)\.
- Horowitz \(2018\)M\. C\. HorowitzArtificial intelligence, international competition, and the balance of power\.Vol\.1\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1),[§7](https://arxiv.org/html/2609.05749#S7.p4.1)\.
- Hughes \(1979\)T\. P\. HughesThe electrification of America: the system builders\.Technology and Culture20\(1\),pp\. 124–161\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1)\.
- Hughes \(1993\)T\. P\. HughesNetworks of power: electrification in Western society, 1880\-1930\.JHU press\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1)\.
- Johnston and Harris \(2019\)P\. Johnston and R\. HarrisThe Boeing 737 MAX saga: lessons for software organizations\.Software Quality Professional21\(3\),pp\. 4–12\.Cited by:[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p2.1)\.
- Kagan and Scholz \(1984\)R\. Kagan and J\. ScholzThe criminology of the corporation and regulatory enforcement strategies\.Enforcing Regulation\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p2.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1)\.
- Klein \(2007\)G\. KleinPerforming a Project Premortem\.Harvard Business Review85\(9\),pp\. 18–19\.Cited by:[§5](https://arxiv.org/html/2609.05749#S5.p2.1)\.
- Kokotajlo \(2024\)D\. KokotajloIn april, i resigned from openai after losing confidence that the company would behave responsibly in its attempt to build artificial general intelligence \[thread\]\.Note:Post on X \(formerly Twitter\), @DKokotajlo67142Accessed: 2026\-08\-31External Links:[Link](https://x.com/DKokotajlo67142/status/1797994238468407380)Cited by:[§7](https://arxiv.org/html/2609.05749#S7.p3.1)\.
- Leike \(2024\)J\. LeikeYesterday was my last day as head of alignment, superalignment lead, and executive @openai \[thread\]\.Note:Post on X \(formerly Twitter\), @janleikeAccessed: 2026\-08\-31External Links:[Link](https://x.com/janleike/status/1791498174659715494)Cited by:[§7](https://arxiv.org/html/2609.05749#S7.p3.1)\.
- Lianget al\.\(2023\)P\. Liang, R\. Bommasani, T\. Lee,et al\.HELM: Holistic Evaluation of Language Models\.Transactions on Machine Learning Research\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1)\.
- MacArthur \(2020\)J\. B\. MacArthurCost savings versus redundant systems: The case of the Boeing 737 Max debacle\.Journal of Business and Accounting13\(1\),pp\. 4–21\.Cited by:[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p2.1)\.
- McDonald \(2012\)A\. J\. McDonaldTruth, lies, and O\-rings: inside the space shuttle challenger disaster\.University Press of Florida\.Cited by:[§3\.1](https://arxiv.org/html/2609.05749#S3.SS1.p1.1)\.
- Merton \(1936\)R\. K\. MertonThe unanticipated consequences of purposive social action\.American sociological review1\(6\),pp\. 894–904\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1)\.
- OpenAI \(2023\)OpenAIOpenAI Preparedness Framework\.Technical reportOpenAI\.External Links:[Link](https://openai.com/safety/preparedness)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1)\.
- OpenAI \(2025\)OpenAIPreparedness Framework, Version 2\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1)\.
- OpenAI \(2026\)OpenAIThe Hugging Face Incident and the Road Ahead\.Technical reportOpenAI\.Note:Published 26 August 2026External Links:[Link](https://openai.com/index/hugging-face-incident-and-the-road-ahead/)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1)\.
- Perezet al\.\(2022\)E\. Perez, S\. Huang, F\. Song,et al\.Red Teaming Language Models with Language Models\.arXiv preprint arXiv:2202\.03286\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1)\.
- Perrow \(1981\)C\. PerrowNormal accident at three mile island\.Society18\(5\),pp\. 17–26\.Cited by:[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p2.1)\.
- Perrow \(1984\)C\. PerrowNormal accidents: Living with high risk technologies\.Basic Books\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1),[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p2.1)\.
- President’s Commission on the Accident at Three Mile Island \(1979\)President’s Commission on the Accident at Three Mile IslandThe Need for Change, the Legacy of TMI: Report of the President’s Commission on the Accident at Three Mile Island\.Technical reportU\.S\. Government Printing Office,Washington, DC\.Note:John G\. Kemeny, chairmanExternal Links:[Link](https://www.osti.gov/biblio/6986994)Cited by:[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p1.1)\.
- Rajiet al\.\(2020\)I\. D\. Raji, A\. Smart, R\. N\. White, M\. Mitchell, T\. Gebru, B\. Hutchinson, J\. Smith\-Loud, D\. Theron, and P\. BarnesClosing the ai accountability gap: defining an end\-to\-end framework for internal algorithmic auditing\.InProceedings of the 2020 Conference on Fairness, Accountability, and Transparency,pp\. 33–44\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1)\.
- Rhodes \(1986\)R\. RhodesThe Making of the Atomic Bomb\.Simon & Schuster,New York\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Ribeiroet al\.\(2020\)M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. SinghBeyond accuracy: behavioral testing of nlp models with checklist\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4902–4912\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1)\.
- Robison \(2022\)P\. RobisonFlying blind: The 737 MAX tragedy and the fall of Boeing\.Anchor\.Cited by:[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p3.1),[§4](https://arxiv.org/html/2609.05749#S4.p5.1),[§5](https://arxiv.org/html/2609.05749#S5.p2.1)\.
- Rogerset al\.\(1986\)W\. P\. Rogers, N\. A\. Armstrong, D\. C\. Acheson, E\. E\. Covert, R\. P\. Feynman, R\. B\. Hotz, D\. J\. Kutyna, S\. K\. Ride, R\. W\. Rummel, and J\. F\. SutterReport of the Presidential Commission on the Space Shuttle Challenger Accident\.NASA Report93\.Cited by:[§3\.1](https://arxiv.org/html/2609.05749#S3.SS1.p1.1)\.
- Rost \(2026\)T\. RostFrom disclosure to self\-referential opacity: six dimensions of strain in current ai governance\.arXiv preprint arXiv:2604\.14070\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1)\.
- Sagan \(1993\)S\. D\. SaganThe limits of safety: organizations, accidents, and nuclear weapons\.Princeton University Press,Princeton, NJ\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1),[§4](https://arxiv.org/html/2609.05749#S4.p2.1),[§7](https://arxiv.org/html/2609.05749#S7.p4.1)\.
- Saunders \(2024\)W\. SaundersWritten testimony of william saunders, former member of technical staff, openai\.United States Senate,Washington, DC\.Note:Testimony before the U\.S\. Senate Committee on the Judiciary, Subcommittee on Privacy, Technology, and the Law, Hearing on “Oversight of AI: Insiders’ Perspectives”Accessed: 2026\-08\-31External Links:[Link](https://www.judiciary.senate.gov/imo/media/doc/2024-09-17_pm_-_testimony_-_saunders.pdf)Cited by:[§7](https://arxiv.org/html/2609.05749#S7.p3.1)\.
- Sedlaret al\.\(2023\)N\. Sedlar, A\. Irwin, D\. Martin, and R\. RobertsA qualitative systematic review on the application of the normalization of deviance phenomenon within high\-risk industries\.Journal of Safety Research84,pp\. 290–305\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p4.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1)\.
- Shevlaneet al\.\(2023\)T\. Shevlane, S\. Farquhar, B\. Garfinkel, M\. Phuong, J\. Whittlestone, J\. Leung, D\. Kokotajlo, N\. Marchal, M\. Anderljung, N\. Kolt,et al\.Model evaluation for extreme risks\.arXiv preprint arXiv:2305\.15324\.Cited by:[§5](https://arxiv.org/html/2609.05749#S5.p5.1)\.
- Stanovskyet al\.\(2025\)G\. Stanovsky, R\. Keydar, G\. Perl, and E\. HabbaBeyond benchmarks: on the false promise of ai regulation\.arXiv preprint arXiv:2501\.15693\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p3.1)\.
- State Council, People’s Republic of China \(2017\)State Council, People’s Republic of ChinaA Next Generation Artificial Intelligence Development Plan\.Technical reportState Council of the People’s Republic of China\.Note:Translated by New AmericaCited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Tech Brew \(2026\)Tech BrewOpenAI, anthropic, and xai employees leave amid ai safety concerns\.Tech Brew\.External Links:[Link](https://www.techbrew.com/stories/2026/02/12/AI-employee-exits-safety-ethics)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p4.1)\.
- The White House \(2024\)The White HouseNational security memorandum on advancing the united states’ leadership in artificial intelligence\.Technical reportExecutive Office of the President,Washington, DC\.Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p2.1)\.
- Turner \(1978\)B\. A\. TurnerMan\-Made Disasters\.Wykeham,London\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1)\.
- U\.S\. House Committee on Transportation and Infrastructure \(2020\)U\.S\. House Committee on Transportation and InfrastructureThe boeing 737 max: a failure of management, engineering culture, and faa oversight\.Technical reportU\.S\. House of Representatives,Washington, DC\.Cited by:[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p2.1),[§3\.3](https://arxiv.org/html/2609.05749#S3.SS3.p3.1),[§4](https://arxiv.org/html/2609.05749#S4.p5.1)\.
- Vaughan \(1996\)D\. VaughanThe Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA\.University of Chicago Press,Chicago, IL\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p1.1),[§2](https://arxiv.org/html/2609.05749#S2.p2.1),[§2](https://arxiv.org/html/2609.05749#S2.p3.1),[§3\.1](https://arxiv.org/html/2609.05749#S3.SS1.p2.1),[§5](https://arxiv.org/html/2609.05749#S5.p2.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1),[§5](https://arxiv.org/html/2609.05749#S5.p5.1),[§8](https://arxiv.org/html/2609.05749#S8.p1.1)\.
- Vaughan \(1999\)D\. VaughanThe dark side of organizations: Mistake, misconduct, and disaster\.Annual review of sociology25\(1\),pp\. 271–305\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p2.1),[§2](https://arxiv.org/html/2609.05749#S2.p3.1),[§4](https://arxiv.org/html/2609.05749#S4.p4.1),[§5](https://arxiv.org/html/2609.05749#S5.p3.1)\.
- Vaughan \(2004\)D\. VaughanTheorizing disaster: Analogy, historical ethnography, and the Challenger accident\.Ethnography5\(3\),pp\. 315–347\.Cited by:[§2](https://arxiv.org/html/2609.05749#S2.p4.1)\.
- Walker \(2004\)J\. S\. WalkerThree Mile Island: A nuclear crisis in historical perspective\.Univ of California Press\.Cited by:[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2609.05749#S3.SS2.p2.1)\.
- Weiet al\.\(2024\)K\. Wei, C\. Ezell, N\. Gabrieli, and C\. DeshpandeHow do ai companies “fine\-tune” policy? examining regulatory capture in ai governance\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Vol\.7,pp\. 1539–1555\.External Links:[Document](https://dx.doi.org/10.1609/aies.v7i1.31745)Cited by:[§4](https://arxiv.org/html/2609.05749#S4.p5.1),[§5](https://arxiv.org/html/2609.05749#S5.p4.1)\.