@VraserX: I think this gets the most important question backwards. The fact that intelligent people inside AI labs are afraid of …

I think this gets the most important question backwards.

The fact that intelligent people inside AI labs are afraid of a catastrophic outcome is a reason to take the risk seriously. It is not evidence that catastrophe is the most likely outcome, and it certainly does not prove that stopping capability progress would make humanity safer.

We still do not know that scaling current systems leads to an uncontrollable recursively self-improving superintelligence. We do not know that such a system would inevitably seek power, escape control or become hostile. Those are plausible failure modes worth enormous amounts of research, but possibility is being presented here as trajectory.

And a temporary capability ban has a gigantic problem. You are not pausing physics. You are pausing whichever actors agree to the pause.

If American labs stop while less transparent states, military programs or other actors continue, we may end up with exactly the outcome you fear, except the most powerful systems are developed somewhere with less scrutiny, less interpretability research and fewer safety constraints.

There is also another side of the risk equation that almost disappears from this argument. Advanced AI could accelerate medicine, biology, clean energy, materials science, education and scientific discovery on a scale we have never experienced. Delaying that technology is not a morally neutral action either. People will die during those lost years from diseases and problems that more capable intelligence might have helped solve.

The sane position is not “race blindly” and it is not “freeze progress.”

Build fast enough that responsible actors remain at the frontier, while making evaluations, interpretability, security, containment and international coordination scale even faster.

I take AI risk seriously precisely because the stakes are enormous. But existential fear cannot become a substitute for evidence.

If we are genuinely approaching superintelligence, humanity should be extraordinarily careful.

We should also be extraordinarily careful about deciding that humanity must not reach it.

For sure. We need to solve the worlds biggest problems and ASI is the only way.

The hardest governance problem with superintelligence may not be stopping it from controlling humans.

It may be stopping a small number of humans from controlling everyone else through it.