A software developer has uncovered a years-long effort to bypass the security of an extremely popular tool in Linux – catching what would have been a disastrous hacking exploit only weeks before it potentially reached the mainstream.

One of the biggest fears among cyber security experts is the possibility of a ubiquitous technology housing an unknown critical vulnerability, and this nightmare scenario is exactly what was uncovered over the Easter long weekend.

On Friday, security experts were made aware of a backdoor mechanism in XZ Utils – an open-source, massively popular compression software used to manage the size of big files such as system backups and software packages.

Lauded for its speed and size-reduction capabilities, XZ Utils is used with most major distributions of customisable operating system Linux and can also be found in countless Linux and macOS applications.

The exploit itself is quite technical, but in short, it works by intercepting key-decryption operations related to SSH – a remote-login protocol adopted by countless global servers.

Once these operations are intercepted, they are redirected towards malicious “backdoor” code which allows the attacker to pass technical arguments and ultimately execute remote code on a targeted system.

The exploit appears to be intended to allow unauthorised access to affected systems at the level of any authorised administrator.

“This might be the best executed supply chain attack we've seen described in the open, and it's a nightmare scenario: malicious, competent, authorised upstream in a widely used library,” said cryptography engineer Filippo Valsorda.

Soon after its discovery, the backdoor was picked up by the Common Vulnerabilities and Exposures catalogue as CVE-2024-3094 and assigned a security score of 10/10 – the highest measurable level of severity.

Furthermore, the exploit – which is present in versions 5.6.0 and 5.6.1 of XZ Utils – requires no system privileges or user interaction with the attacker whatsoever.

Chance discovery prevents calamity

The exploit was discovered by Andres Freund, a Microsoft software engineer and developer who sent an open callout to alert software developers of the backdoor.

Freund made the discovery while troubleshooting some performance issues he noticed in a Debian operating system’s SSH processes which were consuming too many computing resources.

“Hi, after observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks… I figured out the answer,” wrote Freund.

“The upstream xz repository and the xz tarballs have been backdoored.”

After some careful observation, Freund realised a sizeable security exploit was taking place and issued his callout, earning widespread thanks from the global security community.

Valsorda noted the issue was likely “caught by chance”, while malware and security expert Vesselin Bontchev suggested the backdoor was only picked up because it made an observable impact on computing performance.

“If you're going to backdoor something, make sure that your changes don't impact its performance,” wrote Bontchev.

“If your backdoor makes the thing half a second slower, some nerd is going to dig it up.”

Is it state-sponsored?

The culprit of the backdoor is Jia Tan, a pseudonymous developer who has been observed making suspicious changes on online development platform GitHub since 2021.

Tan first contributed code to XZ Utils in April 2022, building up reputation on the project until its community trusted them enough to commit changes to the code.

Eventually, Tan managed to sneakily insert the backdoor code into XZ Utils – requiring a mix of technical know-how, social engineering and time investment which has led experts to suspect a possible state-sponsored entity was behind the affair.

“If this timeline is correct, it’s not the modus operandi of a hobbyist,” said tech enthusiast Michal Zalewski.

“All signs point to this being a professional, for-pay operation — and it wouldn’t be surprising if it was paid for by a state actor.”

Over 96 per cent of the world’s top million web servers are said to run on Linux – including those for Wikipedia, Facebook and Google – and while not all Linux installations utilise XZ Utils, cyber security experts were rightfully shocked to discover the deep-seated vulnerability was likely close to reaching two dominant Linux distributions, Red Hat and Debian.

In February, Tan issued commits containing the backdoor for versions 5.6.0 and 5.6.1 of XZ Utils – before appealing to developers for Ubuntu, Debian and Red Hat to merge the updates into their systems.

Luckily, the discovery has narrowly allowed vendors to address the vulnerability.

An urgent security alert from Red Hat revealed the operating system Fedora Linux 40 contains affected version of XZ libraries, as well as developmental version Fedora Rawhide.

“PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity,” said Red Hat.

Red Hat encouraged users to revert to 5.4.x versions of Fedora 40, and to wait for Rawhide to be reverted to an unaffected version.

The US’ Cyber Security and Infrastructure Security Agency (CISA) also sounded alarms over the exploit, recommending users downgrade to an uncompromised version of XZ Utils, hunt for any malicious activity in their systems, and report “positive findings” to CISA.