Governance of superintelligence

# Governance of superintelligence Source: [https://openai.com/index/governance-of-superintelligence/](https://openai.com/index/governance-of-superintelligence/) There are many ideas that matter for us to have a good chance at successfully navigating this development; here we lay out our initial thinking on three of them\. First, we need some degree of coordination among the leading development efforts to ensure that the development of superintelligence occurs in a manner that allows us to both maintain safety and help smooth integration of these systems with society\. There are many ways this could be implemented; major governments around the world could set up a project that many current efforts become part of, or we could collectively agree \(with the backing power of a new organization like the one suggested below\) that the rate of growth in AI capability at the frontier is limited to a certain rate per year\. And of course, individual companies should be held to an extremely high standard of acting responsibly\. Second, we are likely to eventually need something like an[IAEA⁠\(opens in a new window\)](https://www.iaea.org/)for superintelligence efforts; any effort above a certain capability \(or resources like compute\) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security, etc\. Tracking compute and energy usage could go a long way, and give us some hope this idea could actually be implementable\. As a first step, companies could voluntarily agree to begin implementing elements of what such an agency might one day require, and as a second, individual countries could implement it\. It would be important that such an agency focus on reducing existential risk and not issues that should be left to individual countries, such as defining what an AI should be allowed to say\. Third, we need the technical capability to make a superintelligence safe\. This is an[open research question⁠](https://openai.com/index/our-approach-to-alignment-research/)that we and others are putting a lot of effort into\. But the governance of the most powerful systems, as well as decisions regarding their deployment, must have strong public oversight\. We believe people around the world should democratically decide on the[bounds and defaults⁠](https://openai.com/index/how-should-ai-systems-behave/)for AI systems\. We don't yet know how to design such a mechanism, but[we plan to experiment⁠](https://openai.com/index/democratic-inputs-to-ai/)with its development\. We continue to think that, within these wide bounds, individual users should have a lot of control over how the AI they use behaves\. Given the risks and difficulties, it’s worth considering why we are building this technology at all\. At OpenAI, we have two fundamental reasons\. First, we believe it’s going to lead to a much better world than what we can imagine today \(we are already seeing early examples of this in areas like education, creative work, and personal productivity\)\. The world faces a lot of problems that we will need much more help to solve; this technology can improve our societies, and the creative ability of everyone to use these new tools is certain to astonish us\. The economic growth and increase in quality of life will be astonishing\. Second, we believe it would be unintuitively risky and difficult to stop the creation of superintelligence\. Because the upsides are so tremendous, the cost to build it decreases each year, the number of actors building it is rapidly increasing, and it’s inherently part of the technological path we are on, stopping it would require something like a global surveillance regime, and even that isn’t guaranteed to work\. So we have to get it right\.