How one volunteer stopped a backdoor from exposing Linux systems worldwide

How one volunteer stopped a backdoor from exposing Linux systems worldwide

Photo by Amelia Holowaty Krales / The Verge

Linux, the most widely used open source operating system in the world, narrowly escaped a massive cyber attack over Easter weekend, all thanks to one volunteer.

The backdoor had been inserted into a recent release of a Linux compression format called XZ Utils, a tool that is little-known outside the Linux world but is used in nearly every Linux distribution to compresses large files, making them easier to transfer. If it had spread more widely, an untold number of systems could have been left compromised for years.

And as Ars Technica noted in its exhaustive recap, the culprit had been working on the project out in the open.

The vulnerability, inserted into Linux’s remote log-in, only exposed itself to a single key, so that it could hide from scans of public computers. As Ben Thompson writes in Stratechery. “the majority of the world’s computers would be vulnerable and no one would know.”

The story of the XZ backdoor’s discovery starts in the early morning of March 29th, as San Francisco-based Microsoft developer Andres Freund posted on Mastodon and sent an email to OpenWall’s security mailing list with the heading: “backdoor in upstream xz/liblzma leading to ssh server compromise.”

Freund, who volunteers as a “maintainer” for PostgreSQL, a Linux-based database, noticed a few strange things over the past few weeks while running tests. Encrypted log-ins to liblzma, part of the XZ compression library, were using up a ton of CPU. None of the performance tools he used revealed anything, Freund wrote on Mastodon. This immediately made him suspicious, and he remembered an “odd complaint” from a Postgres user a couple of weeks earlier about Valgrind, Linux’s program that checks for memory errors.

After some sleuthing, Freund eventually discovered what was wrong. “The upstream xz repository and the xz tarballs have been backdoored,” noted Freund in his email. The malicious code was in versions ​​5.6.0 and 5.6.1 of the xz tools and libraries.

Shortly after, enterprise opensource software company Red Hat sent out an emergency security alert for users of Fedora Rawhide and Fedora Linux 40. Ultimately, the company concluded that the beta version of Fedora Linux 40 contained two affected versions of the xz libraries. Fedora Rawhide versions likely received versions 5.6.0 or 5.6.1 as well.

PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity. Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be redeployed.

Although a beta version of Debian, the free Linux distribution, contained compromised packages, its security team acted swiftly to revert them. “Right now no Debian stable versions are known to be affected,” wrote Debian’s Salvatore Bonaccorso in a security alert to users on Friday evening.

Freund later identified the person who submitted the malicious code as one of two main xz Utils developers, known as JiaT75, or Jia Tan. “Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the “fixes” mentioned above,” wrote Freund in his analysis, after linking several workarounds that were made by JiaT75.

JiaT75 was a familiar name: they’d worked side-by-side with the original developer of .xz file format, Lasse Collin, for a while. As programmer Russ Cox noted in his timeline, JiaT75 started by sending apparently legitimate patches to the XZ mailing list in October of 2021.

Other arms of the scheme unfolded a few months later, as two other identities, Jigar Kumar and Dennis Ens, began emailing complaints to Collin about bugs and the project’s slow development. However, as noted in reports by Evan Boehs and others, “Kumar” and “Ens” were never seen outside the XZ community, leading investigators to believe both are fakes that existed only to help Jia Tan get into position to deliver the backdoored code.

Image: Screenshot from The Mail Archive

An email from “Jigar Kumar” pressuring the developer of XZ Utils to relinquish control of the project.

“I am sorry about your mental health issues, but its important to be aware of your own limits. I get that this is a hobby project for all contributors, but the community desires more,” wrote Ens in one message, while Kumar said in another that “Progress will not happen until there is new maintainer.”

In the midst of this back and forth, Collins wrote that “I haven’t lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things,” and suggested Jia Tan would take on a bigger role. “It’s also good to keep in mind that this is an unpaid hobby project,” he concluded. The emails from “Kumar” and “Ens” continued until Tan was added as a maintainer later that year, able to make alterations, and attempt to get the backdoored package into Linux distributions with more authority.

The xz backdoor incident and its aftermath are an example of both the beauty of open source and a striking vulnerability in the internet’s infrastructure.

The lesson from the xz fiasco is that investments in maintenance and sustainability are unsexy and probably won't get a middle manager their promotion but pay off a thousandfold over many years.

But try selling that to a bean counter

— FFmpeg (@FFmpeg) April 2, 2024

A developer behind FFmpeg, a popular open-source media package, highlighted the problem in a tweet, saying “The xz fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion dollar corporations expect free and urgent support from volunteers.” And they brought receipts, pointing out how they dealt with a “high priority” bug affecting Microsoft Teams.

Despite Microsoft’s dependence on its software, the developer writes, “After politely requesting a support contract from Microsoft for long term maintenance, they offered a one-time payment of a few thousand dollars instead...investments in maintenance and sustainability are unsexy and probably won’t get a middle manager their promotion but pay off a thousandfold over many years.”

Details of who is behind “JiaT75,” how they executed their plan, and the extent of the damage are being unearthed by an army of developers and cybersecurity professionals, both on social media and online forums. But that happens without direct financial support from many of the companies and organizations who benefit from being able to use secure software.