AI Is an Existential Risk… Let’s Build It Anyway.

# AI Is an Existential Risk… Let’s Build It Anyway. Source: [https://cefboud.com/posts/ai-scary/](https://cefboud.com/posts/ai-scary/) AI Is an Existential Risk… Let’s Build It Anyway\. The current discussion around AI is frustrating\. On the one hand, the leading companies are saying this technology is extremely powerful and possibly represents an existential risk, and on the other hand, they’re racing and incinerating vast amounts of money to build it\. Which is it? If it’s so dangerous, why are you racing? One argument you hear is the following: AI is going to cure all diseases, and it’s irresponsible not to build it\. An age of abundance is upon us, if only we could build it\. Under this lens, if there’s a chance for infinite prosperity, let’s race to build this thing even if it can destroy us\. We can just reject the premise\. Things can be built responsibly, and they should be\. What’s the current state after all the announcements, tweets and essays? They’ll be keep doing the same things but with extra safety, external auditors, etc\. Why not the extra safety from the get go? You can use external auditors, do interpretability research, have better monitoring, and sandboxing without scaring billions of people\. You can just do these things with less fanfare\. The end result would be the same\. Unless that kind of attention is needed for different reasons\. Then you hear probabilities[thrown around by the people building the thing](https://x.com/EvanHub/status/2097497037956891126): [](https://cefboud.com/assets/tweet-ai-risk-10-percent.png) Is he for real? Even if it actually were just 1%, they should stop immediately\. Would you get into a car or plane if you were told there’s a 1\-in\-10 chance you’ll get into a crash? Clearly not\. Do they actually believe it? Is this a case of cognitive dissonance? LLMs are built with probabilities and statistics\. A ~10% likelihood of infinite harm — killing all humans — has an expected value of infinite harm, and no, factoring future extremely large benefits into the equation is just reckless\. Avoiding infinite harm matters way more than capturing potentially great benefits\. Lower considerably the likelihood of harm by building responsibly and slowing down when needed\. If you can’t garantee very low risk, stop until you can\. Other powerful technologies have been built responsibly\. As should this one\. AI “escaping” is just a lapse in engineering, at least today\. And, yes, there are absolutely many risks, and they need to be addressed, and coordination and communication are definitely required\. But again, all this can be done without the panic and fear angle\. ## Never let a good PR opportunity slip[https://cefboud.com/posts/ai-scary/#never-let-a-good-pr-opportunity-slip](https://cefboud.com/posts/ai-scary/#never-let-a-good-pr-opportunity-slip) The OAI\-HF incident is one catalyst for this recent hand\-wringing sequence\. The agents’ collaboration and capabilities are impressive\. This whole technology is amazing, there is no denying that\. For coding, it’s an absolute game changer\. I love it and use it daily\. And for other disciplines too\. But the whole incident was the result of improper sandboxing and monitoring\. There is no way around it\. They didn’t isolate the agents properly, and they didn’t monitor them while asking them to perform cyber challenges, i\.e\., hacking\. That’s it\. The jump from that to existential risk is unclear\. As far as I can tell, the scenarios people put forward are all built on unproven hypotheticals\. It’s all boundless speculation, things like sandboxing won’t work because[AIs will communicate using temperature changes](https://x.com/firesidealpha/status/2100641742135701983)\. But as of today, all that happened was preventable with proper guardrails\. They just weren’t put in place\. We can’t predict immense future risk because present, preventable risk was not handled properly\. Let’s close with a classic banger\. [](https://cefboud.com/assets/gpt2-dangerous.png) That’s the headline of a Guardian article from 2019\. That was GPT\-2\. This has happened over and over again, and because AI has this sci\-fi and pop\-culture mystique, it keeps working\. All this is really just reward\-hacking the human tendency toward fear and catastrophizing for questionable goals\. Just build responsibly, and if something bad happens, slow down and figure it out without scaring the whole world while doing it\. </rant\>