So how does a model end up knowing how to cook meth?

Reddit r/artificial News

Summary

An opinion piece argues that AI models acquire dangerous knowledge from training data, and that companies like Anthropic and OpenAI rely on easily breakable refusal filters instead of truly removing harmful capabilities, prioritizing speed over safety.

Jailbreaking is a real issue, but honestly nothing new… Every model gets cracked within days of release. The real question is where the model gets the dangerous knowledge in the first place. It has to be in the training data. So how does a model end up knowing how to cook meth, or worse? It didn't figure it out by itself. It's in there because of what they fed it. Anthropic, OpenAI, all of them love to present themselves as the responsible, "safety-first" adults in the room. But they trained these models on the dangerous knowledge anyway, and now they lean on refusal filters that everyone knows break in days. That's not safety, it's a PR layer. They're racing each other to ship and the actual safety of the rest of us is an afterthought they paper over with marketing. “we can't make it safe so we shipped it anyway with a warning label" isn't the flex they think it is. If you genuinely can't remove the dangerous capability without breaking the model, then the responsible move isn't to ship it to everyone behind a filter you already know breaks. Either the safety problem is solvable before release, or the thing shouldn't be a free public toy. Gate access properly or don't ship it that way at all. Curious if anyone here actually buys the safety narrative.
Original Article

Similar Articles

Lessons learned on language model safety and misuse

OpenAI Blog

OpenAI shares lessons learned on language model safety and misuse, discussing challenges in measuring risks, the limitations of existing benchmarks, and their development of new evaluation metrics for toxicity and policy violations. The post also highlights concerns about labor market impacts and the need for continued research on measuring social effects of AI deployment at scale.