Humanity Needs an AI Nonproliferation Treaty

America’s biggest AI companies have spent the past few years churning out advanced new models as fast as they possibly can.

Now the industry is screaming out with a nearly unified voice: Please, for the love of God, make us slow down.

In July, more than 1,300 employees from leading AI firms, including several top executives at OpenAI, Anthropic, Google, and Meta, signed an open letter warning that AI capabilities could soon accelerate “beyond our ability to understand or control the resulting systems.” The letter urges the U.S. government to “develop the technical and governance tools” necessary to “deliberately pace” AI progress.

In a comment accompanying the open letter, Shengjia Zhao, the chief scientist at Meta’s Superintelligence Labs, wrote that AI is on the cusp of creating “unprecedented social and safety risks.” Matthew Rahtz, an engineer at Google, warned that coordination was necessary to “avoid catastrophic harms.” And Leo Gao, an OpenAI researcher, compared the current pace of AI progress to “a runaway nuclear chain reaction.” Plenty of others in the industry have warned that we’ve already crossed a threshold from which there is no turning back.

They worry that the most dystopian fears of an AI future—chatbot-generated bioweapons, major hacking attacks on banks or governments, rogue AI agents wreaking havoc on society—could soon become regular occurrences.

Meanwhile, the public backlash against AI data centers is growing louder by the day.

But a data-center moratorium, the leading political proposal on offer, is totally mismatched to the actual problem of runaway AI development.

So is the notion that AI companies can be trusted to pause their own models before things get out of hand.

These companies will never unilaterally disarm.

Neither, for that matter, will the countries where they’re based.

The only way to truly slow things down might be to take the nuclear-reaction analogy seriously.

What humanity needs is an international AI nonproliferation treaty.

For decades, technology experts have warned that AI systems will eventually become so powerful as to be dangerous.

In just the past few months, “eventually” has begun to look a lot more like “now.” AI systems can already hack seemingly secure systems, invent new viruses, create hyperrealistic deepfakes, and even outsmart the humans who built them.