My speech at Borgo Laudato Si’: On speed, competition, and loss of control

This week, I was honored to speak at the the Global Nobel Laureates Assembly on Artificial Intelligence and Nuclear War. Experts in AI, experts in nuclear weapons, Nobel laureates, religious leaders, and others from around the world gathered for two days at the Borgo Laudato Si’ to discuss perhaps the two most extreme risks facing humanity (nuclear war and catastrophic AI risks such as loss of control or extreme misuse).

The event culminated today, with participating Nobel laureates signing the Rome Declaration. I have various thoughts on the text of the declaration itself, but the final version may or may not be public yet in electronic form (you can squint and see much of it in a photo here). And I’m pretty tired so need to triage a bit. So I’ll focus on my general impressions of the event and share the speech I delivered at Borgo Laudato Si’, which focused on loss of control over AI and the factors making it more likely.

I learned a lot from the other participants in the event, and was inspired by their work and achievements. I was able to get a lot of really dumb questions about nuclear war answered quickly, and was inspired by everyone’s continued participation amidst a brutal heat wave (I’m writing this from Rome, where it’s currently 98°F, and let’s just say that A/C norms are different here than in the US).

I improvised a little bit live but the script at the bottom of this post is more or less exactly what I said on a panel chaired by Brian Schmidt which also included Jon Wolfsthal, Maria Ressa, Heigo Sato, Neerav Kingsland, and Iason Gabriel.

I only had 7 minutes so had to be quite aggressive in editing, and I didn’t make (or fully justify) all the points I wanted to. Hopefully, though, I conveyed the gravity and basic nature of the AI situation as I see it.

As an aside, this event happened in parallel to (and will probably get a lot less press attention than) the recent hubbub about AI CEOs speaking out in favor of (self-)regulation, with many people “+1”ing a proposal from Demis Hassabis.

When I’m more caught up on everything I will say more about how these things relate. But for now, I’ll just say that, as someone who has been thinking about this stuff more or less nonstop for 14 years, I assure you that things are still very far from solved on the regulation front, and excitement about growing support should also be tempered with skepticism about regulatory capture and other failure modes.

My emphasis on the need for AI company employees to get involved is no less urgent today than it was a few days ago — in fact, it may be more urgent to engage over the next few months since there is a high risk of settling on an overly weak framework.

Without further ado, here are my remarks:

2026 is an unusual year to be on a panel about AI escaping human control. In many respects, the story of AI this year is that people are voluntarily handing over control to AI, with no escape required. That process begins inside AI companies, which are automating more and more of the work of building the next generation of AI, and it’s being extended to everyone who uses their products.

AI companies describe this as automated AI research, self-improving systems, or increasingly autonomous agents, and insist that the aim is not loss of control but greater productivity, with humans still in the driver seat. But in a climate of rushed decisions and fierce competition, we could lose control over AI even if almost no one wants that outcome. By “loss of control” here, I mean a scenario in which humans cannot reliably understand, constrain, or reverse what advanced AI systems do.

Two forces make loss of control more likely: speed and competition. I’ll take each in turn.

The first factor creating risk is speed. AI is developing at a very rapid pace, far more rapidly than most outside of the industry fully understand, and far more rapidly than almost anyone in the industry predicted.

I’ll give one example of this. In late 2022 and early 2023, my team worked with outside experts to assess whether a language model, GPT-4, could meaningfully help someone develop chemical, biological, radiological, or nuclear weapons. The results were somewhat reassuring about the severity of imminent risks but pointed to a need for vigilance in the coming years. Today, three years later, AI systems outperform expert virologists and chemists on many tasks that have significant potential for causing harm.

Loss of control risks, too, have moved from something to keep an eye on to a daily challenge making AI systems untrustworthy, and yet still widely used. Loss of control is a risk for the next few years, not the next few decades as many experts once thought. Because of the speed of AI’s advance, we know more about making models capable than about making their behavior reliably understandable and controllable.

The second major factor driving risk is competition. Companies and countries fear that restraint means falling behind. Each also tends to believe it would govern the technology more responsibly than its rivals, and that’s not to mention the massive wealth being generated through leading in this sector.

The net result is that the AI industry is engaging in what Diane Vaughan called normalization of deviance. She was studying NASA’s safety culture leading up to the Challenger explosion and found that people had learned to accept things that they used to consider unacceptable, and cut corners here and there. Likewise, in AI, the idea of an AI system lying to the user was rightly considered unacceptable not long ago. Today it’s just how the technology works. Some companies will say their systems lie and cheat less often than others’, or they will argue that broad deployment is the fastest way to learn and improve, or that the benefits outweigh the risks. But regardless of the rationale, the action is always the same – move forward rapidly with both development and deployment.

In this atmosphere of fierce competition and mistrust, just like with nuclear weapons, there is a growing risk of something going terribly wrong.

Fortunately, there is still some time to act, but it’s running out. There are many things to do, including establishing clear standards for what it means to maintain human control over AI systems in different contexts, penalizing bad behavior, requiring more transparency from the companies building them, and so forth. I’ll focus on just one part of the solution, which is what my colleagues and I call frontier AI auditing.

The idea is that independent experts should be able to analyze the AI systems that the leading companies are building and scrutinize their safety and security practices. Over time, global auditing of frontier AI can build confidence that everyone is maintaining human control to a sufficient degree and otherwise mitigating key risks, so that no one has to put themselves at a potential disadvantage by being cautious.

AI companies and the employees within them face a choice between doing what’s easiest for them in the moment, and what’s best for the species. What’s easiest is to enjoy this fascinating time we’re in, take pride in making some contribution towards accelerating humanity into a new era, and don’t rock the boat internally or in the public sphere. What’s best for the species is for everyone in the industry to take a step back and ask themselves: where is this all headed, if each company is largely left to its own devices? And if they don’t like that default path, they can then also ask: what can I do to bring about the guardrails that each company has to follow, and is verified to be following?

I hope that this convening helps to build global consensus on the urgency of these issues, and to build pressure on those in the industry to participate in such reflections collaboratively and quickly. Thank you.

Cross-posted on Substack here.