They're Gambling With Our Lives: AI Researcher Quits After Warning Superintelligence Could Kill Us All

Jacob Coxon, a 27-year-old AI researcher, has quit Anthropic after warning that artificial intelligence companies are racing towards self-improving superintelligence despite concerns over whether humans can maintain control, saying the technology could potentially kill humanity.

Coxon, who previously worked at OpenAI, spent about three years pretraining AI models before resigning from Anthropic. In a statement explaining his departure, he accused both companies of pursuing increasingly powerful systems despite concerns among researchers about whether humans can control them.

AI Researcher Quits Anthropic Over Safety Fears

Coxon said he left because he believes AI companies are competing to reach self-improving superintelligence rather than slowing down to resolve safety problems.

'Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,' Coxon wrote in his resignation statement.

He argued that the problem is not simply that AI systems are becoming more capable. His concern is that companies could eventually develop systems capable of substantially improving their own abilities, creating a race in which commercial and strategic pressure outweighs safety considerations.

Coxon also said the warning was not intended as a publicity stunt, arguing that senior researchers and executives genuinely believe advanced AI could pose an existential threat.

Researcher Warns AI Could Kill Humans

Coxon said people building advanced AI 'earnestly believe that it could kill us all by the end of the decade'.

His warning refers to a potential future in which AI systems become significantly more capable than humans and can contribute to designing or improving increasingly advanced successors. Such systems could potentially operate with less human oversight, raising concerns about whether researchers could maintain control over their behaviour.

The concept of superintelligence remains theoretical. Coxon's warning is therefore a prediction about a possible future risk, rather than evidence that AI will inevitably cause human extinction.

However, he argued that the possibility is serious enough to require intervention before systems reach that level of capability.

Anthropic Safety Researcher Backs Warning

Coxon's concerns have also been echoed by Evan Hubinger, an Anthropic alignment researcher, who publicly said he believes AI could kill all humans.

Hubinger estimated the risk at more than 10 per cent over the next decade and acknowledged that Anthropic does not yet have a plan guaranteeing that an AI system surpassing human capabilities would follow its creators' intentions.

The comments are significant because Hubinger is an Anthropic alignment researcher rather than an outside critic of the company.

Coxon had previously considered Anthropic more cautious than OpenAI, making his decision to leave the company particularly notable.

Concerns Grow as Systems Become More Autonomous

Coxon's resignation comes amid broader concerns about the speed of AI development. Nearly 1,400 employees from AI companies signed an open letter in July calling for stronger government regulation and action on AI risks.

OpenAI Chief Scientist Jakub Pachocki has also warned that AI capabilities are advancing faster than researchers' ability to monitor and control them.

For Coxon, the answer requires more than individual companies making voluntary safety commitments. He has called for government intervention and greater co-ordination between AI developers to slow the race towards systems that could surpass human capabilities.

Coxon's departure highlights his argument that AI development could be moving towards increasingly autonomous systems faster than adequate safeguards can be established.