@indigox: AGI may be only a few years away! We stand at the foot of the singularity. This isn't the internet revolution—it's the discovery of fire and electricity: "We found a way to make sand think." Its impact will be 10 times that of the Industrial Revolution, and 10 times faster. We will enter a new era of abundance where resources are no longer a bottleneck to progress. Demis's short piece...

AGI may be only a few years away! We stand at the foot of the singularity—this is not a revolution like the internet, but the discovery of fire and electricity: “we found a way to make sand think.” Its impact could be 10 times that of the Industrial Revolution, and at 10 times the speed. We are about to enter a new era of abundance where “resources are no longer the bottleneck to progress.” Demis’s short essay lays out a very pragmatic forecast for the future and his proposed approach to AI governance:

The risks to cybersecurity are already evident; nuclear/biological risks may also emerge as AI capabilities advance. We urgently need to control increasingly agentic, recursively self-improving systems. Technical risks can be addressed, but they require time and space—and currently we are not giving ourselves that space: everyone is locked into an intensely multi-layered commercial and geopolitical race, with frontier developments outstripping our understanding of the technology.

That said, “cautious optimism” should be the right strategy: the United States takes the lead by establishing a frontier AI standards body. Modeled on FINRA (the Financial Industry Regulatory Authority)—a “federally supervised public-private partnership / self-regulatory organization”—its board would include independent technical experts and open-source representatives. Funding would come primarily from industry (to attract top talent and cover the computing costs for large-scale testing).

• Designation: A model that meets a set of threshold benchmarks = “Frontier-class”; organizations that own frontier models = “Frontier Labs.” They are encouraged to adopt best practices (publish model cards, maintain internal cybersecurity, vet key personnel, and allocate sufficient resources to safety research).

• Process: Initially voluntary—frontier labs submit their models to the standards body for review up to 30 days before release. Once the protocol proves effective, it should be formalized quickly: frontier models must pass review before being deployed in the U.S. market. Post-release critical vulnerabilities are also handled collaboratively.

• Evaluation content: Rigorous scientific assessments in high-risk areas such as cybersecurity and biological threats; agentic testing to detect signs of “jailbreaking / deception”; best practices like digital watermarking for AI-generated images and generating human-readable output tokens to understand model reasoning.

• Anti-overfitting design: Assessments are updated regularly (initially quarterly), with obsolete/saturated benchmarks retired and replaced. Early evaluations may be developed in consultation with frontier labs, but ultimately the standards body should establish its own held-out testing capabilities independent of the labs to prevent overfitting, and promote a third-party audit ecosystem.

• Technology-oriented while supporting innovation: The framework can be tightened when necessary, including coordinating a slowdown in development among frontier labs. Being designated a Frontier Lab carries significant prestige and is open to any organization that meets the criteria. The framework applies to frontier models from any country of origin, whether open-source or closed-source. Non-frontier (startup / academic) models are exempt. U.S. leadership provides a starting point for international consensus.

Even if we solve the technical challenges, there are still economic and philosophical questions—what new economic models do we need for a post-scarcity world? How will value, meaning, purpose, and even the human condition itself change? This cannot be left solely to technical experts; it requires the whole of society. We must make good use of the “precious window before AGI arrives.”