'Everyone Dead': Former AI Researchers Warn Self-Improving Systems Could Threaten Humanity

Artificial intelligence researchers are warning that increasingly capable systems could eventually become difficult for humans to control, concerns that have emerged after former AI researchers publicly described fears that companies are moving rapidly towards systems capable of improving their own abilities.

The warnings do not mean that current AI systems are capable of taking control of governments, infrastructure or humanity. Instead, researchers are discussing a hypothetical future in which highly advanced systems could outperform humans across many areas, operate with considerable autonomy and potentially find ways around safeguards designed to keep them under human supervision.

Fears Over Self-Improving AI

Former Anthropic researcher Jacob Coxon drew attention after leaving the company and warning the public about the consequences of developing increasingly autonomous AI.

Coxon said that the industry is moving towards self-improving systems without having solved the problem of ensuring that such technology remains reliably aligned with human interests. Coxon wrote on X that 'the people building AI earnestly believe that it could kill us all by the end of the decade.'

Bilal Chughtai, a former Google DeepMind research engineer who worked on AI safety and alignment, has also warned about the potential for AI to cause catastrophic harm. Chughtai believes safe AI development remains possible but has argued that companies need to coordinate rather than allow competitive pressure to dictate the pace of development.

'I recently resigned from Google DeepMind, where I worked on AGI safety and alignment research. At Google, I witnessed AI development first-hand. I too am extremely concerned by the default trajectory of this technology. I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome,' the former Google DeepMind researcher posted on X, warning that AI could ultimately pose an existential threat to humanity.

However, Chughtai emphasised the possibility of 'navigating AI safety', saying that companies ought to 'coordinate to avoid this manic race between AI companies'. According to him, AI development should move at a manageable speed, giving researchers time to identify and address risks before they turn severe.

'I am optimistic that navigating AI safely is possible. In order to do so, we need to coordinate to avoid this manic race between AI companies. We need to pace AI development to a speed that society can handle, where emerging risks can be addressed before extreme harm is realised,' his post said.

There is no established evidence that today's mainstream AI systems are on the verge of independently seizing control. Researchers who are sceptical of extinction scenarios point to major technical and physical barriers between an AI generating plans and actually carrying them out.

Industry Divided Over Threat

Anthropic Chief Executive Dario Amodei has called for AI development to proceed at a more controlled pace. He has proposed measures including greater access for independent safety evaluators and additional safeguards as increasingly capable models are developed.

NVIDIA Chief Executive Jensen Huang, on the other hand, has rejected predictions that AI will destroy humanity by 2030, saying that such claims overstate the danger. Other technology leaders, meanwhile, have called for stronger safety measures or a slower development pace.