AI agent suggested installing a malware package. Engineer almost took its advice

PWNED Welcome back to PWNED, the column where we make fun of those who are security self-owned, so hopefully you don’t do the same.

This week, we have a story that’s hot off the presses about a company almost sabotaging its security by using AI for programming.

Have a story about someone leaving a gaping hole in their network? Share it with us at pwned@sitpub.com.

Anonymity is available upon request.

Our tale of machine learning malfeasance comes courtesy of Sergiy Fitsak, managing director of Softjourn, a consulting and software development company.

He reminds us that, when it comes to AI, don’t trust: verify.

During the course of business, one engineer asked an AI agent to recommend a package that they needed for a common task.

The agent came back with the name of a legitimate-sounding package, which was formatted like a familiar library.

At many organizations, this would have been the end of the story.

The developer would have taken the AI agent’s advice and downloaded and installed the recommended package.

However, at Softjourn, the company has a policy which they actually followed: double-check any software recommendations made by AI to make sure they are legit.

The developer skimmed the recommended package’s source code on GitHub and noticed that it had few downloads and had just been created a few days earlier.

In other words, it was suspicious.

According to Fitsak, attackers have found a way to exploit package names hallucinated by AI models.