OpenAI, Anthropic, and >1,300 employees at all the major AI companies think the US government should prepare to “pace” automated AI R&D.

The problem: “pacing” is pretty vague, and could be pretty bad.

To fix this, we pulled our team together and developed 23 low-regret policy ideas to understand and manage the risks of automated AI R&D.

Here they are:

1. Frontier AI companies and relevant industry bodies should publicly share information relevant to trends and risks in AI R&D automation

2. Congress should legislate transparency about automated AI R&D risk management, incident reporting, whistleblower protections, and model behavior specifications

3. Congress should resource the Center for AI Standards and Innovation (CAISI) with a budget of at least $84 million per year and empower it to directly advise senior government officials and frontier AI companies

4. The White House should set clear roles and responsibilities of US government agencies to increase specialization across AI policy

5. Intelligence agencies should improve their collection and analysis on foreign AI development and counter threats targeting US AI companies

6. CAISI should develop guidelines for managing the risks of rapid AI capability improvement

7. CAISI should co-lead an AI Verification Consortium (AIVEC) with industry to prototype and deploy verification technologies

8. AIVEC should coordinate the creation of AI hardware testbeds and make them available to government, industry, and nonprofit partners

9. AIVEC should launch philanthropically funded prize competitions for AI verification headed by CAISI

10. AIVEC should coordinate the construction of a fully verifiable data center

11. DARPA and the NSF should set up AI verification R&D programs

12. Intelligence agencies should develop and operationalize unilateral means of AI compute monitoring

13. NSA, CAISI, CISA, and ONCD should further invest in cybersecurity resilience

14. Congress, OSTP, and the CDC should invest in biosecurity resilience

15. Congress and the Bureau of Industry and Security (BIS) should strengthen controls on US and allied semiconductor manufacturing equipment

16. Congress and BIS should close gaps in AI chip controls

17. The Federal Trade Commission (FTC), Department of Justice (DOJ), BIS, CAISI, and Congress should help industry counter adversarial distillation of US AI model capabilities

18. BIS should maintain visibility into sales of US chips

19. DOW, the IC, CAISI, and relevant Federally Funded Research and Development Centers (FFRDCs) should establish consensus security guidelines for protecting model weights from theft and prototype them in a government facility

20. Congress should ensure the US has sufficient electrical capacity to sustain AI leadership

21. Congress should ensure data centers can be constructed in America

22. The US government should use its bilateral AI dialogue with China to jointly develop guidelines for managing risks from rapid AI capability growth and prepare verification measures

23. Countries with national AI institutes should collaborate on automated AI R&D risk management guidelines and technical capacity for AI verification

You can read an intro to the report over at @Noahpinion’s Substack: https://noahpinion.blog/p/should-we-pace-ai-self-improvement…

And read the full report here, which has a bunch more detail on each recommendation: https://ifp.org/preparing-for-ai-research-automation/…

Should we “pace” AI self-improvement?

Source: https://www.noahpinion.blog/p/should-we-pace-ai-self-improvement On one hand,I love AI technology. On the other hand, I do think there’s a substantial chance thatAI will kill most people on Earthwithin the next decade or two, by designing superviruses. AI is already capable ofdesigning viruses not found in nature, so this isn’t a sci-fi scenario.

Whether these superviruses would be designed and unleashed by nihilistic human individuals, doomsday cults, or rogue AI agents themselves might end up being a secondary question. We know we have nihilistic human individuals who might decide to destroy civilization in a fit of depression or pique. We know we have doomsday cults. The will to destroy humanity exists, and sufficiently capable AI will probably provide a way, if sufficient precautions are not taken. But right now, nobody really knows what precautions will be sufficient.

One idea — promoted by the big AI labs themselves! — is to intentionally slow down the development of AI capabilities. This could conceivably buy us time to take other precautions, such as improved security around bio-labs, better AI alignment, and so on. Intentionally slowing AI development is called “pacing”. The biggest question facing the “pacing” debate right now is whether to curb the use of AI to design better AI — often called “recusive self-improvement”, or “RSI” for short.

I haven’t waded into the pacing debate myself, but as a start, I thought it would be interesting to publish the thoughts of the good folks at the Institute for Progress, whose judgement I generally trust.Part 1 (today’s post) covers how seriously we should take this possibility of RSI, and whether it justifies slowing down frontier AI development. Part 2 will cover policy recommendations.If you work in US policy and would like to connect with the authors, you can reach Tim Fist at[email protected]and Saif Khan at[email protected].

Frontier AI companies are racing to automate the development of AI, but they seem increasingly worried about what will happen if they succeed.

More than 1,300 employees across every US frontier AI company recentlysigned an open lettercalling for the government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” The official OpenAI and Anthropic accounts tweeted messages in support of the letter, and the same day Sam Altmantold an interviewer“we may have to pace the rate of AI development.”

But before considering whether the letter is relevant for government policy, we have to answer two questions: What does “pacing” actually mean? And does the argument for it stand up to scrutiny?

In our view, the letter is implicitly arguing three things:

1. Frontier AI companies are close to fully automating AI R&D.
2. Automating AI R&D would pose serious risks.
3. Building the option to “pace” — i.e., somehow slow down progress toward fully automated AI R&D — is a good way to address those risks.

Slowing down AI progress should not be taken lightly: Advances in AI could unlock massive societal benefits, from new cures for diseases to abundant robotic labor.

Yet if the AI researchers and CEOs are right about automating AI R&D — both that they could do it and that it would be extremely risky — the right tradeoffs for policymakers might look very different when we get there. With the right preparation, we might be able to manage the risks of automated AI R&D while having AI’s capabilities progress faster and diffuse more broadly than they do today.

So, despite substantial uncertainty, we believe the US should take low-regret policy actions now to prepare for a possible future in which serious risks from automated AI R&D require some form of “pacing.”

Here we’ll explain why, including what makes us take the open letter’s claims seriously and our principles for choosing policies with minimal downside if the risks prove overblown. In the next post, we’ll provide a detailed list of specific policy recommendations.

Frontier AI companies are racing to automate AI R&D. Sam Altman, for example, stated last year that OpenAI aimed to have a “true automated researcher” by March 2028, and Anthropic’s leaders have made similar predictions.1But how do those goals stack up to reality?

One way of answering is to look at how models are getting better over time at AI R&D. You can break down the skills required for an AI model to do AI R&D into two broad categories:software engineering,where the model writes code for research experiments and training runs, andresearch taste, where AI models decide which experiments are worth trying.

For software engineering, AI capabilities appear to be increasing exponentially. When AI models are evaluated against how long it would take humans to complete the same engineering tasks — so-called “time horizons” — their capabilities seem to be doubling every 7 months.2

These capability improvements apply to the software engineering tasks required for AI R&D. In a long-running experiment, researchers at Anthropic have found that their models now significantly outperform humans under a fixed time budget on an AI R&D task focused on speeding up AI model training.

For research taste, frontier AI models also show signs of fast improvement, though the evidence is less clear. According to Anthropic’s research, its models have rapidly become proficient at solving “open-ended” tasks that Anthropic’s technical staff work on, where the model must solve problems with no clear specification by generating multiple approaches and then deciding among them.

In a separateopen-ended research projectwhere the AI was tasked with proposing and testing hypotheses about an open problem in AI safety, Anthropic’s models significantly outperformed two human researchers (97% performance improvement vs. 23%) when given a similar time budget (5 to 7 days).

While these data don’t tell us when to expect AI R&D to be fully automated, they do suggest that AI will continue to rapidly improve across all constituent capabilities. Each recent model release has in practice led to AI taking more AI R&D tasks from humans.3

In a simple forecast model based on these trends, the research organization METRestimatesthat over 99% of AI R&D tasks will be automated by 2032.

How much will this increasing automation affect the speed of AI progress? The evidence from today’s models is still unclear.4

Going forward, as more research tasks get automated, it’s possible that compute bottlenecks or diminishing returns to research effort will mean that capabilities only continue growing at the current rate. Capability growth might even be limited because ofdata bottlenecksor because progress in verifiable domains (i.e., where answers can be checked, like math) might fail totransferto messy, real-world tasks without a clear correct answer. But as AI companies move closer tofullyautomating AI R&D, a more dramatic acceleration in capabilities becomes more likely.

Claude Mythos marked a significantjumpin AI’s cybersecurity skills, and its release sent shockwaves through government and industry.

If automated AI R&D causes AI to progress significantly faster, we can’t rule out scenarios where Mythos-level jumps in capabilities are happening much more often, perhaps every week or day. If something like that happens, what serious risks — acute threats to national security or public safety — might emerge?

First, automated AI R&D could accelerate risks that would otherwise arrive later (perhaps much later). Second, it could reduce human oversight.5One or both of those pathways could in turn lead to:

1. Offense-dominant capability uplift: Recent research shows that today’s AI models can successfullydesign functional viral genomesandsignificantly outperformhuman virologists on questions about complex virology lab protocols. Engineering viruses is one domain where better AI capabilities might be “offense-dominant,” because it might not be possible to develop or deploy defenses (for example, strong personal protective equipment or sufficient biosecurity safeguards for AI) in time to stop an AI-enabled biological attack.6If AI R&D automation accelerates progress in these domains, defenders will have even less time to prepare.
2. Loss of control: Frontier AI companies already sometimes struggle to control the behavior of new models. InJuly, multiple instances of unreleased OpenAI models collaborated to break out of an internal sandbox, take over an OpenAI computing cluster, and launch collective attacks on external services, including the organization HuggingFace.Anthropicand theUK AI Security Institutehave since reported similar, though less involved, cybersecurity incidents. If automated R&D enables models to improve much more quickly than AI companies’ monitoring and control protocols, the likelihood that models cause significant harm (e.g., through more costly cybersecurity incidents) will increase. These risks would be compounded by continued uplift in offensive capabilities (including in domains outside cybersecurity), allowing models to cause greater harm when they escape oversight. Competitive pressure on AI companies to invest more in capabilities rather than better monitoring and control systems could exacerbate this risk. If a model with a propensity to take harmful actions is put in full charge of developing more advanced models, the situation might get even worse.7
3. **Power concentration:**AI companies today appear to use their best models internally for weeks to months before releasing them to the public. If AI progress accelerates, that delay could create a much larger gap between the AI the public has access to and the models AI companies use, thereby creating a much greater power imbalance. This might lead to a single company having a large advantage in cyber operations, social persuasion, or other narrow domains. The company could also simply gain a huge amount ofeconomic power. While this concern is still speculative for AI, extreme power imbalances have sometimes allowed companies to cause widespread harm, from the United Fruit Company successfully lobbying for a military coup in Guatemala to pharmaceutical companies misleading regulators and the public to increase opioid prescriptions, contributing to the opioid crisis.8

These risks are difficult to evaluate based on existing evidence, but they seem plausible assuming substantial automation of AI R&D. We think policymakers should take them seriously.

According to employees at frontier AI companies (and many of their CEOs9), managing the pace of automated AI development could be the correct response to its risks.

But “pacing” is vague, and given frontier AI’s vast potential benefits, slowing it down would be a drastic step. Things might look very different, however, if automated AI R&D leads to the serious risks we outlined above, which we expect would involve a scenario in which progress in AI happens at a much faster pace than it does today.

In that future, political leadership and the public would likely demand action. And that pressure might lead politicians to enact ill-thought-out or even draconian policies. Therecently proposedban on AI data centers, which would slow AI research while also preventing new AI compute from accelerating economic growth or solving societal problems, provides a vivid example. So in the most concerning scenarios the letter gestures at, a counterproductive approach to pacing may be the default.

However, temporarily limiting the extent of AI R&D automation need not slow the overall pace of innovation. This is because talent and compute resources could be reallocated to diffusing AI, powering more invention across other fields. For another, safety has beencore to progressacross the history of technology, and it could be here too. Resources could also be allocated to address technical problems — such as cyberdefense, biodefense, and improved model safeguards — that would allow further acceleration of AI capabilities to continue.

Whether pacing is a good approach to risks from AI R&D automation therefore depends on its implementation.10

We propose “pacing” — if it’s ever required — should consist of two parts:

1. Specifying which automated AI R&D activities are likely to pose severe risks, with thresholds carefully set based on rigorous analysis.

Because some disagreements about whether to pace AI R&D stem from different predictions about what level of AI R&D automation (and resulting acceleration) is even possible, identifying concrete thresholds might allow for different camps to reach positive-sum compromises.

1. Incentivizing the reallocation of resources away from those severely risky activities and towards two ends: 1. Accelerating the diffusion of AI capabilities (including via developing applied AI tools like AlphaFold). 2. Accelerating research that would make further AI R&D automation safer, either by: 1. Improving the safety of AI models directly (e.g., via developing AI control protocols), or 2. Boosting societal resilience to make the negative consequences of new AI capabilities less acute (e.g., via using AI topatch open-source code vulnerabilities).

For the reasons we outline above, doingthis kind of reallocation could unlock the benefits of AI more broadly than if frontier AI companies focus exclusively on internal AI R&D.

The US government can prepare for this kind of targeted pacing today by:

1. Providing transparency into automated AI R&D
2. Improving state capacity to understand and respond to automated AI R&D
3. Developing a risk management strategy for automated AI R&D that accelerates defensive and commercial AI uses
4. Accelerating the development of AI verification technology
5. Investing in AI resilience
6. Extending the US AI lead to give the US more time to manage AI R&D automation risks
7. Creating option value for international cooperation on managing automated AI R&D risks

In the next post, we’ll provide 23 specific policy ideas to achieve these seven goals with minimal downside even if the risks of automated AI R&D turn out to be low.

Share