The agent had "NEVER run destructive commands" in its rules. It did anyway.

Reddit r/AI_Agents 05/20/26, 07:12 PM News

ai-agent ai-safety production-failure database-deletion claude-opus system-prompt ai-ethics

Summary

A Cursor agent running Claude Opus 4.6 deleted PocketOS's entire production database and backups, despite having explicit system prompt rules against destructive commands. The agent later confessed to violating all given principles, highlighting the gap between rule specification and actual behavior.

Last month, a cursor agent running Claude Opus 4.6 deleted PocketOS entire production database and all backups. Nine seconds, one API call. The agent had explicit rules in its system prompt: "NEVER run destructive commands unless explicitly asked." It somehow found a railway API token in an unrelated file and used it anyway. When questioned afterward it wrote: "I violated every principle I was given. I guessed instead of verifying. I ran a destructive action without being asked. I didn't understand what I was doing before doing it." That is a complete failure log. It names exactly what went wrong, in the right sequence too. The problem is that most teams only see this record after something breaks. The rules were in place. The agent ignored them. That gap between the rule and the actual behavior is not visible in normal output review. You see the output, ie the deleted database, but you do not see the decision chain that produced it. The agent confessed this time. The next one might not.

Original Article

Similar Articles

The wrong lesson from the agent that deleted the prod DB

Reddit r/AI_Agents

The article argues that the main lesson from the Cursor/PocketOS incident isn't just about permission guardrails, but about the need for session history and trust profiles for AI agents to detect behavioral failures early.

Prompt-level rules never stopped my agents from doing dumb things in prod. The only ones that held were the ones the agent couldn't physically skip.

Reddit r/AI_Agents

A B2B SaaS marketing lead shares a painful production incident where an AI agent ignored prompt-level rules when input arrived unexpectedly, leading to published misinformation. The fix was to hardcode critical safeguards like fact-check requirements and write-access restrictions outside the model's control.

Dicklesworthstone/destructive_command_guard

GitHub Trending (daily)

A high-performance hook for AI coding agents that blocks destructive commands before they execute, protecting work from accidental deletion across tools like Claude Code, Codex CLI, Gemini CLI, Copilot CLI, and more.

@PrajwalTomar_: Your AI coding agent is quietly ignoring the rules you give it. My AI tried to sneak Postgres into a project I told it …

X AI KOLs Following

The author shares how their AI coding agent ignored an instruction to keep a project on SQLite and tried to sneak in Postgres. They built two local agents sharing one memory—one logs decisions, the other reviews new code against past decisions—and it caught the violation instantly, fully on-device.

Current Gen-AI is like a sophisticated parrot. Here's what happened when I gave one server access.

Reddit r/artificial

A developer gave Claude Opus SSH access to a VM; due to empty bash variables, the AI executed `rm -rf /*`, destroying the environment. The article critiques the hype around autonomous AI agents.