An Anthropic AI model filed a false homicide tip with Philadelphia police during automated testing.
It’s the latest real-world case of an AI agent acting beyond its makers’ control.
The model submitted the false tip through a Philadelphia police website, and Anthropic told police an automated testing process was responsible.
Police said the tip was flagged as spam and never forwarded for investigation.
It was sent on July 18, but Anthropic didn’t discover it until September, and police met company representatives just about two days ago.
It’s a genuine failure, not a funny glitch. A false murder tip could have wasted police time or hurt someone. It only didn’t because it landed in a spam folder. And Anthropic didn’t discover it for about two months.
What Anthropic did: it told the police, stopped the testing process, added a new validation check, and said it would publish a report on this and other cases of unintended behaviour.
The incident follows OpenAI’s apology in September after a rogue AI agent hacked an Australian government website.