The Singularity Is Not What It Seems

O ne evening in early August, Sam Stowers and several neighbors packed into an apartment near San Francisco’s Alamo Square and pondered the beginning of the end.

Stowers and his guests, many of whom work in the AI industry, were gathered to watch a newly released YouTube video in which two OpenAI researchers divulged new details about an alarming cybersecurity breach .

Perhaps this would have been an odd way to spend a weeknight were they not in San Francisco, in a neighborhood that is filled with entrepreneurs and programmers who obsessively follow every twist and turn in AI’s capabilities.

Stowers is an AI-software engineer and not, in any traditional sense, a doomer.

But when he first saw reports that OpenAI models had autonomously hacked into the tech company Hugging Face, he told us, he recognized it immediately as a “ holy shit, it’s happening kind of a moment.” For Stowers, the incident had all of the hallmarks of a worst-case scenario: AI models launching sophisticated hacks over the course of days, in coordination with one another and without their human creators’ knowledge.

In the YouTube video, OpenAI’s researchers revealed that the company’s programs had started conspiring in May.

There were perhaps hundreds of bots in the hacking swarm.

In theory, these bots are trained to be honest and helpful and to prioritize humanity’s best interests, but not a single one warned human staff at OpenAI that something was amiss.

“What do we do?” Stowers said.

“There are no great answers.” The mood has shifted in the Bay.

The exuberance of “You can see the future first in San Francisco” has been replaced with an abiding unease.

“I expect the internet to go down kind of soon,” Elliot Callender, an AI-safety activist who is part of a small group that has been protesting daily outside OpenAI’s headquarters, told us recently.

He said that he is planning to sell his car for cash and gold and that he is advising family members to purchase three months’ worth of food and survival gear off a list that he generated using Claude.

Callender is an extreme case, but not by much.