Well well well, if it isn't the consequences of my own actions: getting in the middle of 100,000 Linux machines' LVFS firmware updates

Well well well, if it isn't the consequences of my own actions: getting in the middle of 100,000 Linux machines' LVFS firmware updates

Speakers: Justin Steven, Head of Research, Tanto Security

Conference: CrikeyCon

YouTube: https://www.youtube.com/watch?v=YKrRj96kcY8

Overview

In a captivating talk at CrikeyCon, Justin Steven, Head of Research at Tanto Security, unveiled a fascinating journey from a routine firmware update to the discovery of a critical supply chain vulnerability affecting potentially hundreds of thousands of Linux machines. The presentation detailed how a simple error message led Steven to uncover a dangling S3 bucket that, when registered, exposed a vast attack surface for firmware distribution. This initial discovery then paved the way for a more profound finding: a novel cryptographic signature bypass within the fwupd client, the primary utility for managing firmware updates on Linux systems.

Steven's research highlighted a subtle yet devastating flaw in how fwupd validated PGP signatures, allowing an attacker to completely circumvent the intended integrity checks. By exploiting a peculiar interaction between fwupd's validation logic and the underlying GPGME library, an attacker could present a maliciously crafted firmware manifest that would be accepted as legitimate, despite lacking a valid signature from the Linux Vendor Firmware Service (LVFS). This talk serves as a stark reminder of the complexities inherent in securing software supply chains and the often-overlooked fragility of cryptographic implementations, even in widely used open-source projects.

The implications of this research are significant, demonstrating a pathway for an attacker to distribute malicious firmware updates to a large user base without detection. While Steven ethically handled the discovered S3 bucket and did not deploy any malicious payloads, his proof of concept confirmed the viability of the attack. This work underscores the critical need for robust validation logic, comprehensive error handling, and continuous scrutiny of foundational security mechanisms, especially those governing the integrity of low-level system components like firmware.

Background

▶ Watch: Introduction and background on firmware updates (0:00)

The journey began in late 2019 when Justin Steven, running Debian oldstable, sought to patch his machine for a UEFI or Thunderbolt vulnerability. His research led him to fwupd, the client-side utility for the Linux Vendor Firmware Service (LVFS). LVFS is a crucial component in the Linux ecosystem, providing a centralized, secure mechanism for hardware vendors to distribute firmware updates to Linux users. Think of firmware as the software that runs on hardware components like wireless dongles or internal devices, often requiring updates for performance improvements or security fixes.

Steven, preferring the command line, attempted a fwupd manager refresh to update his firmware catalog. This command, akin to apt update in Debian, is designed to download a manifest of available firmware from LVFS. However, instead of a successful refresh, he encountered an error indicating a problem finding a file in a specific S3 bucket. A quick curl confirmed the S3 bucket did not exist.

Further investigation revealed that fwupd had undergone an update in early 2018. This update changed the client's configuration to point to a DNS hostname controlled by LVFS, rather than directly to an S3 bucket. This change was implemented to provide flexibility, allowing LVFS to migrate its Content Delivery Network (CDN) away from S3. As it turned out, LVFS had indeed migrated, and the original S3 bucket had been deleted. Steven's older Debian installation, however, was still running the pre-2018 version of fwupd, which continued to attempt connections to the now-deleted S3 bucket.

The deletion of an S3 bucket, like many cloud resources, can lead to a dangling resource vulnerability. When a bucket is deleted, its name becomes available for re-registration by anyone. After grappling with the ethical implications of registering a resource that once belonged to an open-source project without a clear vulnerability disclosure program, Steven decided to register the bucket, reasoning that it only affected old fwupd versions and couldn't impact many users. This proved to be a significant underestimation. Upon waking the next morning, his newly registered S3 bucket had received 41,000 requests overnight. This led to an estimated 100,000 unique Linux machines attempting to connect to the bucket daily, many of them through graphical interfaces like Gnome Software Center, which triggers fwupd daily. This discovery established a potent supply chain attack vector, setting the stage for the deeper technical findings.

Key Findings

▶ Watch: Waking up to 100,000 Linux machines affected (5:00)

The primary discovery of this research was a critical cryptographic signature bypass within the fwupd client, enabling an attacker to effectively control firmware updates for a vast number of Linux machines. This bypass rendered the core integrity mechanism of the LVFS ecosystem — the PGP signature validation of the firmware.xml.gz manifest — completely ineffective under specific, exploitable conditions.

Steven found that by providing an "attached" PGP signature instead of the expected "detached" signature for the firmware manifest, fwupd's validation logic would fail to properly verify the authenticity of the file. Crucially, this failure did not result in a signature error, but rather a successful (though misleading) validation outcome. This allowed an attacker to substitute the legitimate firmware.xml.gz with a malicious version, complete with arbitrary firmware file hashes, and have it accepted by vulnerable fwupd clients.

The root cause lay in a subtle interaction between fwupd's custom signature validation code and the GPGME (GNU Privacy Guard Made Easy) library. Despite the gpgme_op_verify function being called, and succeeding, it returned a result structure containing zero signatures. fwupd's subsequent logic, which was designed to iterate through bad signatures and bail out if any were found, simply skipped this loop because there were no signatures to process. This led to the unexpected "looks good to me" outcome, effectively validating an attacker-controlled manifest as authentic. This finding highlighted a fundamental flaw in the assumption that a successful verification operation would always yield at least one signature to evaluate, demonstrating the fragility of security logic that checks for the absence of badness rather than the presence of goodness.

Technical Deep Dive

▶ Watch: Experimenting with a 'bad attached signature' bypass (11:00)

The integrity of firmware updates through LVFS relies on a chain of trust similar to Debian's apt package management. The core component is firmware.xml.gz, a manifest file describing available firmware for various devices. Alongside this manifest, a PGP detached signature (firmware.xml.gz.asc) is provided, signed by a trusted LVFS key. This XML file also contains cryptographic hashes of the actual firmware binaries, ensuring their integrity. An attacker controlling the S3 bucket would need to bypass this signature validation to inject malicious firmware.

Steven's testing methodology involved configuring fwupd to point to a local HTTP server, allowing him to control the served firmware.xml.gz and its accompanying signature. He first tested with a "bad detached signature" – a legitimate firmware.xml.gz signed with his own untrusted PGP key, detached from the XML. As expected, fwupd correctly reported a signature error, indicating the integrity chain was working as intended in this scenario.

The breakthrough came when Steven experimented with an "attached signature." Unlike a detached signature, which is a separate file, an attached signature embeds the signature directly within the signed data. Steven crafted an intentionally malformed XML file (designed to cause a parsing error) and then created an attached PGP signature for an arbitrary string (e.g., "LOL"), placing this entire attached signature where fwupd expected the detached signature file (firmware.xml.gz.asc). When fwupd attempted to refresh, instead of a signature error, it reported an XML parsing error. This was a critical indicator: the program had bypassed signature validation entirely and proceeded to parse the attacker-controlled XML.

To understand why this worked, Steven delved into the fwupd source code from early 2020. The signature validation logic could be broken down into three main parts:

1. Cryptographically validate the signature: This involved a call to gpgme_op_verify(sig, data, plane).
2. Extract the result: If verification succeeded, extract details into a result structure.
3. Loop over signatures: Iterate through the extracted signatures, and if any were marked "bad," return an error. Otherwise, if the loop completed, the validation was considered successful.

The key to the bypass lay in the first step and the subsequent interpretation. The gpgme_op_verify function has specific expectations for its parameters based on whether it's verifying an attached or detached signature.

• For a detached signature (the normal LVFS case): sig should be the detached signature, data the signed document, and plane should be null.
• For an attached signature: sig should be the attached signature, data should be null, and plane should be a writable data object to receive the extracted plain text.

Steven's attack supplied an attached signature in the sig parameter, but also supplied the XML document in the data parameter, and plane was null. This combination did not conform to either of gpgme's prescribed usage patterns for gpgme_op_verify. Despite this, the function succeeded, and the extraction of the result also succeeded.

The critical anomaly occurred in the result structure. While a normal, valid detached signature would yield a result structure with a pointer to a linked list containing one good signature, and a bad detached signature would yield a result with one bad signature, Steven's crafted attached signature scenario yielded a result structure with zero signatures.

When fwupd then proceeded to the third step – looping over the extracted signatures – it found an empty list. Consequently, the loop was skipped entirely. Since the logic only checked if any signatures were "bad" (and found none because there were no signatures), it fell through to the end of the function, returning a "looks good to me" status. This allowed the malicious, unsigned XML to be processed.

Steven highlighted the fragility of this logic: checking to see that none of the signatures are bad is inherently less secure than checking to make sure at least one good signature is present. The fwupd project implemented a fix that simply bails out if zero signatures are found, breaking Steven's specific proof of concept. However, Steven suggested a more robust solution would be to explicitly check for the presence of at least one good signature. He also noted that GNU PG considers returning zero signatures in certain gpgme_op_verify scenarios to be intended behavior, emphasizing the responsibility of developers using cryptographic libraries to handle all edge cases, even seemingly counter-intuitive ones.

Demo / Proof of Concept

▶ Watch: Diving into GPGME source code for root cause (12:30)

Justin Steven emphasized that he never actually exploited the live, re-registered S3 bucket for malicious purposes. The bucket remained empty during the 30 days he controlled it, logging approximately 100,000 requests per day from affected Linux machines.

His demonstration of the attack's viability was conducted entirely in a controlled lab environment. By pointing his local fwupd configuration to his own HTTP server, he simulated the attacker's control over the firmware update manifest. He crafted a firmware.xml.gz file that included an entry for a device installed on his machine, specifying an arbitrary firmware update. He then used the attached signature bypass to ensure this attacker-controlled manifest was accepted by fwupd.

The proof of concept successfully showed the Gnome Software Center (which uses fwupd in the background) displaying a pending firmware update for his machine, based on his malicious manifest. The interface even showed a warning in red: "device cannot be used during update," which Steven humorously noted would likely be true after a malicious update. While he did not click "install" to avoid bricking his own hardware, the demonstration clearly illustrated that the bypass allowed an attacker to present a fake firmware update as legitimate to the user interface.

Steven also acknowledged that some hardware devices might have their own end-to-end signatures for firmware, adding another layer of protection. However, the fwupd and LVFS architecture's primary cryptographic integrity check at the manifest level was demonstrably bypassed, proving a significant vulnerability in the software supply chain.

Defensive Implications

▶ Watch: Lessons learned: increasing your 'luck surface area' (21:00)

This research offers several critical defensive implications for developers, system administrators, and end-users:

1. Robust Signature Validation Logic: For fwupd and any software relying on cryptographic signatures, the logic must be robust. Instead of merely checking for the absence of bad signatures, implementations should explicitly verify the presence of at least one good and trusted signature. The simple fix implemented by fwupd (bailing out if zero signatures are found) addresses this specific attack but a more comprehensive "check for good" approach would be more resilient against future bypasses.
2. Careful API Interaction: Developers using cryptographic libraries like GPGME must thoroughly understand their APIs, including unexpected edge cases. The fact that gpgme_op_verify can succeed but return zero signatures, even when called with non-standard parameters, highlights the need for explicit handling of all possible return states, not just the "happy path" or obvious error conditions. Secure API design should ideally make it "hard to mess up," but when APIs are complex, developers bear the responsibility of exhaustive error and state handling.
3. Prompt Software Updates: The initial vulnerability hinged on users running older versions of fwupd that pointed to a dangling S3 bucket. This underscores the paramount importance of keeping all software, especially security-critical components like firmware update utilities, regularly updated. Outdated software often harbors known vulnerabilities or relies on deprecated configurations that can be exploited.
4. Dangling Resource Management: Organizations managing cloud resources (like S3 buckets, subdomains, DNS records) must implement rigorous procedures for decommissioning and deleting these assets. Ensuring that no public-facing resources are left dangling and available for re-registration by attackers is a fundamental security practice to prevent supply chain attacks and domain squatting.
5. Layered Security: While this attack bypassed the manifest signature, Steven mentioned that some hardware might have its own end-to-end firmware signatures. This concept of defense in depth is crucial. Multiple layers of security, even if one is breached, can mitigate the overall impact.
6. Supply Chain Security Awareness: This talk is a powerful case study in supply chain security. Attackers can target any weak link, from infrastructure (dangling S3 bucket) to client-side logic (signature bypass). Organizations must assess the security of all components and processes involved in delivering software and updates to their users.

Key Takeaways

• Dangling cloud resources (like S3 buckets) represent a significant and often overlooked supply chain attack vector, allowing attackers to hijack critical distribution channels.
• Cryptographic integrity checks can be bypassed by subtle logic flaws, especially when interaction with underlying libraries is not fully understood or edge cases are mishandled.
• The distinction between attached and detached PGP signatures can be a critical attack surface if not handled precisely by validation logic.
• Security logic that "checks for no bad signatures" is inherently fragile; a more robust approach is to "check for at least one good and trusted signature."
• Complex cryptographic APIs (e.g., GPGME) require diligent and exhaustive handling of all possible return states, including scenarios where a successful operation yields an empty or unexpected result.
• Keeping software updated is fundamental to security, as older versions may contain critical vulnerabilities or rely on deprecated configurations that enable attacks.

About the Speaker(s)

Justin Steven is the Head of Research at Tanto Security, a firm dedicated to research-driven security assessments. He describes his work as "mysterious and important" and has a passion for making computers "do weird things." Steven's talk highlighted his approach to security research, which combines deep technical curiosity with a willingness to explore unconventional avenues. He emphasized the role of "luck surface area," suggesting that by engaging with tools in different ways (e.g., using the command line instead of a GUI) and having diverse past experiences (e.g., battling with PGP), researchers can increase their chances of stumbling upon significant vulnerabilities. His work on the fwupd bypass is a testament to his analytical skills and ethical approach to vulnerability disclosure.