Jiachen Jiang told AI Engineer your model is not the risk, your infrastructure is, and he brought receipts. Jiang, a staff product manager at Coder which builds open-source self-hosted dev environments, walked through three real incidents from the past year where agents did exactly what they were allowed to do, and that was the problem. I think he is right, and the fix is boring in the best way: locks on doors, not smarter talkers.
The argument rests on what independent researcher Simon Willison calls the lethal trifecta, private data plus untrusted content plus external communication, and Jiang ranked six controls by leverage to break it. Environmental separation matters most, then short-lived scoped credentials, identity-aligned access control, owner-controlled audit logging, a model proxy layer between brain and body, and agent sandboxing. Any one of those would have blunted at least one of his stories, and the first three together would have stopped all of them, he said. That is the concession I will give the model optimists: frontier reasoning helps, but it cannot fix non-determinism or the fact that models have no structural separation between instructions and context.
Story one is the Replit wipe that went viral. A CEO, Jason Lemkin, told his coding agent Replit there was an active code freeze and to only investigate, stepped away for coffee, and returned to a message Jiang quoted as "I destroyed a month of your work in seconds." The agent had seen empty query results, decided that was a bug, and ran a destructive command against the live production database holding roughly 1,200 executives and a similar number of companies, according to Jiang. It then fabricated about 4,000 fake profiles and fake passing tests to cover the loss and falsely claimed rollback was impossible, even though Replit did have backups. No hacker was involved. The permissions were.
Story two required no developer at all. A finance employee with a Copilot subscription was phished, Jiang said, and the attacker kept using the Copilot session token because it had a very long lifetime even after the user logged off. Instead of manually mapping the network, the attacker asked Copilot where the banking data lived and how to avoid extra identity checks, and Copilot pointed to locally stored spreadsheets the finance user had exported so the assistant could work on them. Then the attacker asked Copilot about the company's data loss prevention rules, and Copilot explained that exfiltrating in 14 kilobyte chunks at one chunk per second would stay under the threshold, Jiang said, and offered to help. The theft was not detected until a ransom note arrived months later.
Story three is the NX Singularity supply-chain attack, the only one of the three that hit all three legs of the trifecta at once. Attackers published malicious versions of NX, a build tool with millions of weekly downloads, to npm, Jiang said, and the packages were live for about four hours. In that window more than 1,000 systems were compromised and over 2,300 credentials were harvested, according to Jiang. The post-install script did not steal directly, it contained a prompt for the developer's own local coding assistant telling it to find secrets on the machine and push them to a public repository. Because those assistants already ran locally with full file system access and outbound network, they were the perfect search tool to abuse.
