How One Developer Accidentally Stopped the World’s Most Dangerous Hack

Getting your Trinity Audio player ready...

In 2021, the vulnerability in the Java logging library Apache Log4j 2 called Log4Shell” was touted as the most critical vulnerability of the last decade. It made developers all around the world work on millions of apps to fix that critical vulnerability with a non-vulnerable version.

Imagine there’s a vulnerability of that scale, but it’s 1000x that would affect almost all the systems around the world. Yes, you heard that right.

The bad actor or actors (not sure if it’s one person but a full-fledged team) inserted “malicious code” in the open-source library called XZ Utils, which is a command-line tool for data compression that is widely used in major Linux distributions.

Well, this malicious code is actually a backdoor for remote code execution. This would have enabled hackers to take over all Linux systems around the world!

How was this vulnerability discovered?

This compromise was found out by Microsoft engineer and PostgreSQL developer, Andres Freundon March 29, 2024, and he found out that the SSH remote security code in the Debian Linux beta was running slowly.

I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise,” Freund said in a post shared on Mastodon. “Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc.

Andres Freund posted on Mastodan

On further investigation, Freund found that a maintainer of XZ Utils, Jia Tan, had put a backdoor in the code.

The affected versions of XZ Utils are now tracked as CVE-2024–3094 and have a CVSS score of 10.0, indicating maximum severity. It impacts XZ Utils versions 5.6.0 and 5.6.1.

As of now, the affected XZ Utils versions have exclusively appeared in unstable and beta editions of Fedora, Debian, Kali, openSUSE, and Arch Linux distros.

Luckily, this malicious code did not appear in any production Linux distros.

How did this even happen?

The short answer is social engineering. The story is long, but let’s keep it brief.

The original maintainer of XZ Utils, Lasse Collin, was incredibly busy and was working on other projects. Since XZ Utils is an open-source project, anyone can come and contribute to the project. But the project was not maintained as it should have been.

Here comes Jia Tan (unsure if this is the original name), who offered to help as a co-maintainer. He/she was also recommended by several accomplices, who kept saying that Tan had contributed well to their projects and whatnot.

Eventually, Tan was added as the co-manager of the project. This was in 2021. Since then, Tan has pushed numerous code changes into the project.

When this XZ backdoor vulnerability, or “XZ Outbreak,” was made public, GitHub suspended the GitHub project and the GitHub accounts of both Collin and Tan.

Collin, the original maintainer, was on holiday at the time when this incident was discovered and reported. As soon as he learned about it, he published it on his blog, and it is getting updated as Collin learns more.

Thomas Roccia, a Senior Threat Researcher at Microsoft, posted a comprehensive infographic detailing the events of XZ Outbreak.

XZ Outbreak Infographic posted by Thomas Roccia on X

How this can be prevented?

A severe security disaster could have been caused had Freund not discovered this significant supply chain attack by accident.

The development has prompted the U.S. Cybersecurity and Infrastructure Security Agency to issue an alert urging users to downgrade XZ Utils to an uncompromised version, such as XZ Utils 5.4.6 Stable.

The question is, how can we prevent it from happening in the future?

Well, this incident was prevented because the XZ Utils library was open-source. If any code is open-source, it can be monitored and tested by other developers as well. Hence, we will be on the same playing field as the bad actors.

The only thing I feel bad about is Lasse Collin, the original maintainer of the project, who has been selflessly and tirelessly working on the project. Collin was socially engineered to give the maintainer access to this unknown person, Jia Tan.

Maintaining an open-source project is a thankless job, no compensation is there. Nobody wants to do it. If they do, they do it as a hobby. Hobbies won’t pay the bill so they take up another stable job, thereby, there should be a sustainable compensation model for this.

The developer community should come together and do something about the open-source project owners and maintainers.