AI safety and alignment

Just a couple days ago, Anthropic put out a declaration to pause the development of AI, emphasising that we are not prepared for the consequences of giving this technology too much power too quickly. Is anyone else genuinely worried about future AI safety and how, as it becomes more and more intelligent, humans may start to lose control of it? Pumping billions of dollars into this technology only means it’ll get increasingly integrated into our workflows, which we are already starting to see. As a result over time, companies will begin completely trusting the system, automating the vast majority of business operations – this is all while the technology gets more and more intelligent, leading to the real possibility of self replication ability, let alone the power to deceptively manipulate people into using it. By allowing AI to be embedded in systems, the internet and even ‘helping’ humans develop revolutionary drugs, does it concern you at all that perhaps one bad super intelligent, misaligned actor may bypass testing processes and, for one example, launch a biochemical weapon onto humans? I don’t think the threat is inevitable, but it is on a trajectory toward inevitability unless intervention occurs. The variable that most determines the outcome is not AI capability, it is whether governance frameworks (particularly around open-source bio-design tools and autonomous offensive AI) can outpace capability development. Perhaps a pause is necessary to reduce this risk, allowing defence capabilities to be prepared? I understand this is a hurdle given the capitalist nature of the world but what significant, destructive catastrophe will it take for people to wake up…