For the second in our Backdoors and Ethics series, Daniel digs into the Linux XZ Utils Backdoor and the impact it had on this thriving open-source community.
Often, we underestimate the ability and contribution of individuals in the bigger picture of a story or event. However, the hero in this tale pulled the thread where others didn’t and uncovered a shocking revelation.
One Friday, a Debian system was experiencing performance problems with SSH. Andres Freund found himself troubleshooting the issue as CPU cycles were consumed at a higher rate on all SSH logins, and Valgrind (a computer memory monitoring tool) generated many errors.
Freund’s attention to detail, and some good luck, led him to discover that the problems were due to an update made to XZ Utils, where someone had intentionally added a backdoor to the compression software. He quickly disclosed all he knew to the Open-Source Security list.
Many theorized about the complexity of the backdoor. Thomas Roccia (@fr0gger_ on X), A researcher at Microsoft, published a graphic on Mastodon to visualize the extent of the nearly successful backdoor, showing how its reach could have made the SolarWinds event of 2020 seem minor.
XZ Utils is a household name for anyone running a Unix-based operating system. It is responsible for lossless data compression and many critical functions in the compressing and decompressing of data across numerous operations. Additionally, it supports the legacy .lzma format. Its importance is hard to overstate.
The scariest element of the backdoor is that nobody saw the code the attacker intended to run, making the possible intent infinite. The code added to XZ Utils versions 5.6.0 and 5.6.1 modified how the software functions by manipulating the SSHD (an executable file used in creating remote SSH connections). Anyone who obtained a predetermined encryption key could stash any code in an SSH login certificate, upload, and execute it on the backdoor device.
Upon further research and some reverse engineering, it seems the backdoor was years in the making. As far back as 2021, a user named JiaT75 had been making suspicious commits to open-source projects.
In 2022, JiaT75 seemingly teamed up with another unknown user named Jigar Kumar to support a new patch to XZ Utils strategically. Alongside these two operators, several other unknown sources pressured the long-time maintainer of XZ Utils to bring on an additional developer to help – JiaT75.
Early 2023 saw JiaT75’s first commits to XZ Utils. Their activity increased over time until February 2024, when JiaT75 issued commits to versions 5.6.0 and 5.6.1 of XZ Utils. These updates implemented the backdoor, and the new community of open-source developers began to appeal to developers of Ubuntu and Debian to merge these updates into the operating systems. Eventually, one of the two updates made its way onto many releases for the aforementioned products.
The open-source community is a fragile space, often propped up by people’s passions alone. With that in mind, it’s both unimaginable and clear to see that our reliance on something precariously kept alive and maintained by those passions can often not be enough to safeguard it from orchestrated attacks.
The effort and time spent in setting this up is staggering and hopefully serves as a stark reminder for larger companies utilizing these open-source products that payment and work-security for the open-source community’s developers is a must.
If you’d like to talk more about security and get your project off the ground or moved forward further, reach out on freshconstruct.com.
Then why not sign up for our newsletter! Keep up to date about the content your business needs to know.