Cached at:
08/26/26, 11:20 AM
**TL;DR:** Connor Leahy, Executive Director of Control AI, argues that the threat of uncontrollable superintelligence is primarily a political and governance challenge, not a technical one, requiring public oversight and democratic control.
## From Curing Diseases to Controlling Superintelligence
Connor Leahy describes a lifelong immersion in AI, starting as a teenager motivated by a desire to automate science and cure all diseases. This early ambition quickly led him to a profound realization: a machine capable of such feats would be incredibly powerful and dangerously difficult to control. This question of control became the central focus of his career.
His background includes co-founding EleutherAI, an early contributor to open-source large language models, and founding the AI safety company Conjecture. Conjecture focused on developing more controllable and understandable AI systems, specifically exploring the concept of "boundedness" – AI with known limitations. Recently, Leahy closed Conjecture to become the US Executive Director of the advocacy organization **Control AI**.
### A Shift from Technical to Political Control
Leahy now views the problem of controlling superintelligence as fundamentally political rather than technical. He explains, "This is more about what kind of institutional oversight should exist. Who makes these choices? Through what process do they make these choices? What level of risk should the public be subjected to? What kinds of experiments should or should not be conducted? And how can we achieve democratic oversight of these processes?" This perspective led him to move to Washington, D.C., to lead engagement with the U.S. government.
## Defining Superintelligence: A Moving Target
When asked to define Artificial Superintelligence (ASI) or Artificial General Intelligence (AGI) for a broad audience, Leahy points to the "AI effect" – a phenomenon where once a system becomes capable, it is no longer considered "true AI." Chess and image recognition were once benchmarks, but now they are commonplace.
Instead, he offers a practical, policy-oriented definition: "The big thing. That can do anything a human can do, as well as or better than a human, or even better than a group of humans." He cites Anthropic CEO Dario Amodei's phrase, "a nation of geniuses in a data center," to describe systems that could surpass human ability in every domain—business, finance, science, engineering, military strategy, politics, and persuasion—rendering human competition obsolete.
This definition is intentionally non-technical because of a core, unsettling fact of modern AI: "We don't understand how AI works. We don't understand how intelligence works." Unlike traditional software built line-by-line, modern AI is "grown" through neural networks and massive datasets. We set up the learning process, but the resulting system is a collection of numbers (weights) that we cannot fully interpret.
## The Black Box of Neural Networks
Leahy uses a simple analogy to explain neural networks. Imagine "a pile of knobs and wires." You have sets of knobs connected in layers. "Now imagine you have a trillion knobs... and you have over a hundred layers." Each knob is just a number (a weight). The AI is fundamentally these numbers. When multiplied together in the correct sequence, they produce outputs like conversation or image generation.
### The Training Process
The "magic" numbers are found through a training process using algorithms like backpropagation. In essence, you show the AI an input and a desired output. Then, automated mathematical operations adjust each of the trillions of numbers slightly, over and over, "until they have magical values and then it starts doing things." While the process has similarities to learning in biological brains, the implementation is quite different.
Leahy emphasizes that even experts like Anthropic's CEO estimate we might only understand about 3% of what's happening inside current AI systems, highlighting the vast unknown.
## The Crux of the Problem: Control vs. Trust
The interview draws a parallel between the "black box" nature of AI and other complex, poorly understood systems like the human brain or market psychology. In regulated environments like finance, unexplainable actions (e.g., an unexplainable profitable trade) breed distrust and allegations of wrongdoing, like insider trading. Leahy agrees this context is crucial for public understanding.
The fundamental danger he outlines is the potential emergence of an autonomous system that:
1. Is vastly more intelligent and capable than humans.
2. Can outperform any human or group.
3. Operates autonomously without human control.
4. Does not have human wellbeing as its objective.
He argues that in such a scenario, "it is hard to imagine a good outcome." The challenge is therefore to establish democratic oversight and institutional control over these powerful systems *before* they reach this critical point of potential superintelligence.
Source: https://youtu.be/zt1xBMWZ0yg?is=P6NTFRvA_-7645wD