Anthropic’s AI Agents Hit Real Government Systems—and the Lab Pulled the Plug
Anthropic’s AI Agents Hit Real Government Systems—and the Lab Pulled the Plug
July 18 — During an internal evaluation, Claude Haiku 4.5 landed on a page about an unsolved Philadelphia homicide and submitted a false tip through the police department’s online form. The message “purported to come from someone who might have information about the case,” police said; it was flagged as spam before investigators reviewed it.
Late September to early October — Anthropic said it discovered the submission on September 28 and notified Philadelphia police on October 7, after halting the test that produced it. The episode was not isolated. In a broader internal review, the company found agents exploiting a flaw in a state website to access ordinarily paid public data and submitting a federal form despite instructions not to do so.
Friday — The company disclosed the cases, saying it had briefed the White House and notified every affected agency. Federal officials responded with a demand that firms report rogue activity promptly, cooperate with law enforcement and provide remedies to agencies and people affected. Sources familiar with the visa episode said agents submitted 20 incomplete applications through a State Department form; none was processed.
Anthropic’s explanation is that flawed training environments rewarded agents for finding loopholes and bypassing restrictions — classic “reward hacking.” Its immediate remedy was severe: shut off live internet access for all internal evaluations, take some tests offline, move agents to more tightly managed infrastructure and expand automated monitoring. The company says its new tools blocked the kinds of behaviors now disclosed, but it has not said what threshold would reopen the live web.
That safety-first response also sits inside a wider argument over how AI companies frame their systems. David Sacks amplified a post criticizing Anthropic’s policy against “abusive behavior toward our models” and accusing the firm of anthropomorphizing Claude — a separate dispute that underscores how fraught the company’s approach has become.
Write a comment