Why you can’t do RSI as fast as you like - https://t.co/YG05zw2vnG

Intelligence Has a Speed Limit

In AI circles, Recursive Self-Improvement (RSI) is talked about in almost magical terms: a model tweaks itself, becomes smarter, uses that intellect to rewrite itself again, and this leads to super-intelligence. It’s an intuitive and appealing idea. Talk to anyone from control theory though (the discipline that brought you the auto-pilot etc.), and they’ll raise an eyebrow because they know the expectation is naive in a very specific way.

We all know self-improvement is possible. But any self-improving loop has a maximum safe speed, set by how fast you can generate trustworthy evidence that a change was actually an improvement.

Control theory’s small-gain theorem says a feedback loop stays stable when the product of its gains stays below one. Gain is a measure of amplification. Here it means how large a modification the system makes per unit of evidence (that the modification helps.) The entire appeal of RSI is high gain: rapid progress.

In the 80-90s, adaptive control was a big subject: we got global stability proofs for controllers that revise their own models on the fly. But within two years, researchers at MIT published Rohrs’ counterexample. They ran proven adaptive control policies against plants that violated the assumptions in mild but realistic ways, and the systems went haywire.

A self-improving AI system rewriting itself has the same structure. Its self-model is worst exactly where it has least real-world experience, and the harder it optimizes against that model the deeper it pushes into the gaps. Worse still, an unstable adaptive system often doesn’t blow up right away. It bursts: runs beautifully for a while, then behaves unpredictably and without warning - it may even settle back down.

The cause is loss of persistent excitation, a vulnerability directly analogous to how AI models fail on out-of-distribution (OOD) scenarios. Ground truth comes from contact with the world, not from benchmarks. And the system must respond with speeds at which consequences unfold. Accelerating confidence past evidence creates a higher gain system with small stability margins.

Evolution and civilization are the counterexamples people like to cite, but both prove the point. Each improved its own substrate without bursting: each ran slowly relative to feedback, in parallel, with no single agent holding both modification power and evaluation authority.

Rohrs’ counterexample redirected the field into a decade of building robustness in, always by trading peak performance for more stability margin. These translate directly to RSI progress:

• Slow down and stop adapting when the error is inside the noise floor.

• Anchor parameters toward a trusted prior - bound total distance from a verified checkpoint.

• Treat a saturated benchmark as a sensor that went dead, not as a win.

Systems that reliably improve themselves will be governed, damped, slow, and bounded by margins that look wasteful. Because stability is the speed limit.