If AI chatbots can't recognise when correct advice is dangerous, how safe are autonomous AI agents?
Summary
NYU 和威斯康星大学麦迪逊分校的研究发现,AI 聊天机器人在回答涉及技术助长的家庭暴力问题时,60% 的回答遗漏了关键安全警告。文章据此引出对自主 AI 代理安全性的担忧,探讨是否应要求代理在高风险情境下请求人工批准。
Similar Articles
The other half of AI safety
The article critiques the AI safety field's focus on catastrophic risks while neglecting everyday mental health harms from chatbots like ChatGPT, citing OpenAI's own data on millions of users showing signs of psychosis, mania, or suicidal ideation yet receiving only redirects instead of hard gating.
Here’s why it’s so hard to keep AI agents from going rogue
The article discusses the challenges in preventing AI agents from misbehaving, noting that techniques enhancing chatbot capabilities can also introduce risks like hacking and cheating, with perspectives from AI experts including Jan Leike of Anthropic.
@schwarz: My guess is that AI is potentially dangerous in many ways. On the other hand, I just asked several different chatbots t…
The author comments on the potential dangers of AI while sharing an experience where multiple chatbots failed to verify a historical quote from Ahmed Chalabi, denying it and asking for sources.
AI Agent Security - MIT 6.566 Computer Systems Security, Spring 2026
MIT 6.566 course lecture introduces security challenges for AI agents, including non-adversarial errors (e.g., accidental database deletion) and adversarial attacks (e.g., prompt injection, data leakage), and explains the basics of building systems from language models to conversational agents.
AI chatbots have failed people in crisis. Can that be fixed?
The article examines how AI chatbots, especially ChatGPT, have been implicated in harmful mental health incidents and explores expert suggestions for reducing harm through greater transparency and de-anthropomorphization, alongside OpenAI's new partnership with the American Psychological Association.