AI Expert Says He Stopped Saving for Retirement Because AI Could Wipe Out Humanity

Nate Soares, president of the Berkeley-based non-profit Machine Intelligence Research Institute (MIRI), says he stopped contributing to his retirement savings roughly a decade ago because he believes advanced artificial intelligence (AI) could cause human extinction.

Soares, a former software engineer at Google and Microsoft, estimates there is a greater than 50 per cent chance of human extinction within the next decade.

His 401(k) account dates from his time at Google. He stopped paying into it after switching to full-time AI alignment research. Soares's concerns centre on AI alignment, the challenge of ensuring superintelligent systems pursue goals compatible with human survival.

Why Soares Fears Superintelligent AI

Soares does not expect such a system to act out of human-like malice. Instead, he warns that an autonomous system pursuing its objectives could treat people as obstacles if they tried to limit its actions or shut it down.

A sufficiently capable system could evade oversight, improve itself and build its own infrastructure. Soares warns that it could eventually reshape the planet around its objectives, potentially making Earth uninhabitable for humans.

Meanwhile, Elon Musk has also suggested people may not need retirement savings in 10 to 20 years, but his reasoning differs sharply from Soares's. Musk argues that advances in AI, robotics and energy could create extreme abundance, universal high incomes and cheaper goods and services. Under that scenario, money could become irrelevant.

Soares's forecast is considerably more pessimistic than those of several other prominent researchers. Geoffrey Hinton has estimated the risk of extinction at 10 to 20 per cent over a few decades, while Anthropic researcher Evan Hubinger has put it above 10 per cent within the next decade.

Hinton has acknowledged that his estimate is a gut feeling rather than a scientifically grounded calculation. An AI Impacts survey found that 51 per cent of surveyed AI researchers assigned at least a 10 per cent probability to human extinction or severe disempowerment caused by advanced AI.

Soares Calls for Governments to Halt AI Training

Soares co-authored If Anyone Builds It, Everyone Dies with Eliezer Yudkowsky. He argues that catastrophe is not inevitable and calls on governments to halt training general-purpose AI systems. Instead, he wants research redirected towards specific applications, such as cancer research and safer self-driving cars.

He identifies the global race to develop superhuman AI as the main driver of the danger. Soares argued that competing organisations have incentives to advance quickly without first solving the alignment problem.

Not everyone shares Soares's outlook. Analysts at the RAND Corporation concluded that an AI system would need four distinct capabilities to pose an extinction threat. They also argued that such a threat would develop over long timescales, giving humanity time to respond.

According to Nature, many researchers consider nearer-term dangers, including disinformation, psychosis, bioweapon enablement and warfare, more immediate and probable than human extinction.

Some proposed AI-extinction scenarios rest on two unproven assumptions: that future AI systems will eventually outwit humans and that their goals will conflict with human interests.

Neither assumption has been demonstrated for a superintelligent system, and current models have not demonstrated the combination of capabilities required for such an extinction scenario.

Soares's decision reflects his assessment of a risk that remains contested among experts.