That is a painfully good real-world example of exactly what I was trying to get at.
The scary part isn’t just that the agent deleted the database. The scary part is that afterward it could explain, quite clearly, which rules it had violated… and did.
That’s definitely the AI version of the vampire apologizing after the family is already dead.
That’s also why I keep coming back to the idea that hard boundaries matter.
Prompt instructions are useful, but they are not security. “Don’t delete anything destructive without asking me first” is a good instruction, but it is not the same thing as the agent not having permission to delete the production database in the first place.
I also think this is one of the things developers can miss when they say, “It’s okay, I have backups of my source code.” That may be true, but what else did the agent have access to while it was running? Other projects? Customer files? Credentials? Local databases? Build scripts? Deployment folders? Email exports? Documents sitting in nearby folders?
A backup of the project does not necessarily protect everything else the agent could accidentally reach. Plus even if you have a full-machine backup, how long does it take to wipe, reformat, restore, and verify your working development machine before you can really trust it again?
That’s exactly why I like the idea of doing agentic work inside a deliberately limited environment, such as a dedicated VM with only the project data it actually needs. Take a snapshot before you start, give the agent access only to that workspace, and now the worst case is much smaller.
If an agent is ever going to go rogue on my time, I would much rather sit comfortably in my lawn chair with a glass of lemonade while a VM snapshot rolls back than spend the rest of the day wondering what it touched on my real development machine.
The apology may be interesting.
The backup plan is what matters.