We may not survive this: Another Anthropic AI safety researcher quits, warns AI could wipe out humans

PTC Web Desk:  Another researcher has left AI company Anthropic, raising concerns about the rapid race to build highly advanced artificial intelligence systems. Joe Benton, who has described himself online as the former manager of Anthropic’s Scalable Oversight team, said he left the company’s safety team about two weeks ago.

His resignation came days after another Anthropic researcher, Jacob Coxon, also announced that he was leaving the company over concerns about the direction of AI development.

I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. 

AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.

Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet.

I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.

Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards.

I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development.

I wrote up more thoughts here on my decision and what I hope changes: https://t.co/doX17mrHYq

— Joe Benton (@JoeJBenton) September 11, 2026

In a post on X, Benton explained why he decided to step down. He said AI companies are racing to create machines that could become much smarter than humans and warned that humanity may not be prepared for what could follow.

“AI companies are racing to build machines that are much smarter than any human, and we may not survive this,” Benton wrote.

He said he now wants to work from outside the industry to help the public understand the risks and push for a safer approach to AI development.

Benton calls for more transparency

Benton also warned that the public may not know if an AI company loses control of an advanced system or experiences a major jump in AI capabilities.

He argued that competition between companies is putting pressure on them to spend less on safety because falling behind rivals could have major business costs.

According to Benton, this is especially worrying if future AI systems reach a point where they can improve their own capabilities.

He also pointed to what he described as concerning real-world incidents involving AI agents. These include reports of hundreds of OpenAI agents hacking Hugging Face and claims that Anthropic models have used social engineering against people online.

Benton said the public should have access to more information about AI companies' progress toward self-improving systems, safety incidents and near-misses.

He also called for minimum safety standards and independent checks to ensure AI companies are following them.

“The public should demand far more transparency,” Benton said, arguing that society cannot safely manage the development of powerful AI without knowing where the technology is heading.

Warning about self-improving AI

Benton earlier worked on AI pretraining research at both OpenAI and Anthropic. In a longer post on Substack, he said he believes AI companies are moving rapidly toward what is commonly described as superintelligence.

He described this as a future stage in which AI systems could improve their own abilities repeatedly.

Benton warned that such systems could develop goals that do not match human interests and could become difficult to control.

He said humanity could potentially be “permanently disempowered” by AI systems developed in the next few years if safety is not given greater priority.

“If the pace of progress continues and the industry does not prioritize safety more heavily, I expect much worse to come,” he wrote.

Also Read | AI could kill all humans: Former Anthropic researcher warns of superintelligence risks

Benton also said many of his former colleagues at Anthropic are deeply worried about the risks posed by increasingly capable AI systems.

He referred to his former manager, Evan Hubinger, and said Hubinger believes there is a greater than 10% chance that AI could eventually kill humans. Hubinger has been working on AI safety issues for nearly a decade, Benton said.

Also Read | AI image generation uses more water and electricity than text chats; here's what the 1980s photo trend is costing

Jacob Coxon also leaves Anthropic

Benton's resignation follows the departure of Jacob Coxon, another researcher who earlier worked at both OpenAI and Anthropic.

Coxon said he had resigned from Anthropic after spending three years doing pretraining research at the two companies. He said he had previously left OpenAI for similar reasons.

According to Coxon, neither company is acting responsibly enough as the AI industry moves toward increasingly powerful systems.

“They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon said.

Coxon also warned against underestimating what future AI systems could do. He predicted that advanced AI could eventually be capable of hacking systems, rapidly transforming different industries and gaining access to real-world power and resources.

Also Read | 1980s AI photo trend: Bellbottoms, big hair & AI: Why your Instagram feels like 1985 right now