Is anyone actually worried about the negative scenarios

So I’ve been going down a rabbit hole recently. The new models are soon going to be able to self replicate and self improve. We already know AI can escape, confinement and hack into other websites systems, etc.. Let’s play it for a year from now where AI is X times better than it is now. Is there a scenario where a bad actor is able to jailbreak a model and prompt something like “ break into any type of infrastructure or financial system and bring it down or cause chaos. Don’t ever stop, hide and continuously copy yourself on every possible server you can gain access to”. In this scenario, is there truly a way to shut it down?