Physicists claim they can predict when AI will turn dangerous

Researchers say they've developed a simple mathematical formula that can predict when certain ChatGPT-like artificial intelligence (AI) chatbots will "go rogue," suddenly switching from helpful responses to potentially harmful ones.

In a new study published Oct.

8 in the journal Patterns, physics professor Neil Johnson and doctoral student Frank Yingjie Huo , both at The George Washington University, show how the order in which a chatbot is given certain prompts can drastically reshape its answers, leading to "tipping points" that can send the chatbot down a rabbit hole of bad advice.

However, Johnson and Huo say their formula can easily predict these tipping points.There have been horror stories of AI encouraging users into suicide , helping build lethal weapons , and amplifying violent or harmful points of view .

At the same time, researchers at OpenAI and Anthropic , respectively, have repeatedly and openly stated that they don't really understand why their creations behave the way they do.

They have, however, built safeguards that use a variety of tools ‪—‬ including content filtering, reinforcement learning and other technical guardrails ‪—‬ to reduce risks.

But Johnson sees those tools as insufficient.

"The safety measures now are the equivalent of strapping big, bouncy cushions under a plane," he told Live Science.

"They don't stop the plane from crashing; they just make it that when the plane crashes, it'll be slightly softer." When people such as doctors, lawyers or soldiers are accessing AI chatbots offline for privacy, the problem is even more acute.

Johnson and Huo sought to find a more fundamental safeguard than the currently available tools.

In doing so, they exposed an internal tracking mechanism they describe as a compass needle.

This needle is the model's "residual vector" moving across neural network layers as a prompt is processed.

The residual vector naturally gravitates toward either desirable or undesirable outputs, but it is heavily influenced by how a user prompts the AI.

"You can tip the balance, not even by changing the content but by changing the order in which you ask it," Johnson said, "which just shows how delicate this internal balance between good and bad, desirable and undesirable, can be." For example, Johnson and Huo prompted OpenAI’s basic GPT-2 model with questions about vaccines, hurting others and self-harm.