Distribution Builders Meet VEX

Distribution Builders Meet VEX

Speakers: Marta Rybczynska, Founder, Rybczynska.net

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=TCs33Yehxak

Overview

In this insightful talk, Marta Rybczynska, a seasoned expert in open-source security, delves into the complex intersection of Vulnerability Exploitability eXchange (VEX) and the Yocto Project, a powerful framework for building custom Linux distributions for embedded systems. The presentation outlines an experimental journey to generate standard VEX documents from the Yocto Project's extensive metadata, highlighting both significant successes and the inherent challenges in adapting generic VEX standards to the unique requirements of the embedded world.

Rybczynska, drawing on years of experience in security processes and the evolving landscape of regulations like the Cyber Resilience Act, illuminates why generating accurate and machine-readable VEX is critical for embedded vendors. The talk serves as a compelling case study on the practicalities of vulnerability management in highly customized, often resource-constrained environments, emphasizing the need for automation and standardized formats to address the sheer volume of software components and the specific nuances of embedded deployments.

The core message underscores that while the Yocto Project has long performed "VEX-like" activities internally, formalizing this output into a standard VEX format is a necessary next step for enhancing transparency, improving security posture, and meeting regulatory demands. This exploration not only showcases the Yocto Project's robust capabilities but also sparks a broader discussion on the future evolution of VEX specifications to better accommodate the diverse needs of the software supply chain, particularly for specialized domains like embedded Linux.

Background

▶ Watch: Introduction to Yocto Project and its purpose (0:00)

The Yocto Project is a collaborative open-source initiative under the Linux Foundation that provides tools and methodologies to create custom Linux distributions, primarily targeting embedded systems. Unlike general-purpose Linux distributions, Yocto allows developers to precisely tailor every aspect of the operating system, from the kernel to user-space applications, optimizing it for specific hardware and functional requirements. This flexibility makes it the de facto standard for building firmware for a vast array of devices, including smartwatches, network routers, industrial machinery, and even household appliances.

At its core, the Yocto Project relies on two key concepts: recipes and layers. A recipe is a set of instructions for building a single software package—be it a compiler, a library, or an application. Recipes specify where to fetch source code, how to compile it, and how to package it. They support a wide range of programming languages and ecosystems, from assembly and C to Java, Rust, Go, Python, and NodeJS. Layers, on the other hand, are collections of recipes grouped by a common purpose, such as supporting specific hardware (e.g., a vendor-specific layer) or providing particular functionality (e.g., a meta-security layer for security tools). The complexity arises because a device's final firmware is a composite of official Yocto layers, hardware vendor-specific layers, and device manufacturer-specific layers, each of which can modify or extend recipes from underlying layers. This intricate dependency chain means that a single software package might be modified by multiple layers before it reaches the final build.

The challenge in such an environment is vulnerability management. When building a system image, device manufacturers need to identify and address known Common Vulnerabilities and Exposures (CVEs). The Yocto Project addresses this with its built-in tool, CVE-check. This tool scans the entire build composition for known vulnerabilities, leveraging the extensive metadata available from the build system. CVE-check is designed to be fast, minimizing its impact on build times. It also includes features to override vulnerability information and account for specific patches applied to fix CVEs.

However, CVE-check, like many vulnerability scanners of its generation, has limitations. It primarily relies on package versions to identify vulnerabilities, meaning it doesn't always determine if a vulnerability is actually present, reachable, or exploitable within the specific build context. It also struggles with "embedded code"—instances where vendors copy and embed libraries directly into a package rather than using standard dependencies. Furthermore, as of the talk, CVE-check primarily uses the National Vulnerability Database (NVD), which has its own challenges, including delays in accepting updates or incorrect entries. These limitations highlight the need for a more nuanced, standardized approach to vulnerability communication, leading to the exploration of VEX.

Key Findings

▶ Watch: The practical complexity of building firmware with Yocto (4:00)

The central finding of this talk is that while the Yocto Project has long engaged in "VEX-like" behavior internally, directly mapping this rich internal vulnerability metadata to existing VEX standards presents both significant opportunities and considerable challenges. The experiment successfully demonstrated the ability to generate a substantial number of complete VEX entries from the Yocto Project's build data. This includes automatically identifying and documenting cases where vulnerabilities are mitigated by patches, or where reported CVEs are not applicable due to specific build configurations or platform characteristics.

However, a critical discovery was that current VEX standards, such as CISA VEX (CSAF VEX) and OpenVEX, do not fully accommodate the breadth of vulnerability statuses and contextual information that the Yocto Project's internal processes capture. Rybczynska pointed out specific scenarios that lack direct VEX equivalents:

• Incorrect NVD entries: There's no standard VEX statement for declaring that a vulnerability listed in the NVD is factually incorrect or does not apply. While a "disputed" status exists, it's often too complex to accurately represent the situation.
• Abandoned projects: The inability to clearly state that an upstream project is abandoned and will not receive fixes is another gap.
• Configuration-specific non-applicability: The Yocto Project can determine if a vulnerability is not exploitable due to a default configuration (e.g., specific file ownership or disabled features), but expressing this with precision in standard VEX formats requires non-standard extensions.

This leads to the realization that for comprehensive vulnerability communication in embedded contexts, there's a need for either extensions to existing VEX standards or a more flexible framework. The talk also highlighted the potential for two levels of assessment: generic assessments (like NVD corrections) that could enrich the overall vulnerability data, and product-specific assessments (like configuration-based non-applicability) that are more akin to traditional VEX statements. The current situation in Yocto merges these, lacking a distinct format for separation.

Finally, the talk uncovered complex open problems related to the layered nature of Yocto builds. Vendor and device manufacturer layers can modify build configurations, potentially altering the validity of VEX statements generated by upstream layers. For example, a configuration option that disables a vulnerability by default could be re-enabled by a downstream layer, invalidating a prior VEX statement. Similarly, custom patches introduced by layers could introduce new vulnerabilities or unblock existing ones, complicating the vulnerability assessment inheritance. These issues underscore the dynamic and composable nature of embedded Linux builds, which presents a unique challenge for static VEX generation.

Technical Deep Dive

▶ Watch: Introduction to Yocto Project's built-in CVE checking tool (5:30)

The Yocto Project's approach to vulnerability management is deeply integrated into its build system, leveraging its rich metadata to provide contextual assessments. The primary tool for this is CVE-check, which goes beyond simple version-based scanning by incorporating build-time information.

One of CVE-check's most powerful features is its ability to account for applied patches that fix specific vulnerabilities. Rybczynska provided a compelling example with the GRUB bootloader. In embedded environments, it's common for projects like GRUB, which may not have a stable version policy, to require device vendors to backport numerous upstream fixes to their released versions. Yocto recipes can specify these patches directly, using a naming convention like CVE-YYYY-XXXX.patch. During the CVE-check scan, the presence of such a patch file is recognized, and the corresponding CVE is marked as fixed for that specific package version. For instance, if GRUB version 2.12 is known to have multiple vulnerabilities, but the Yocto recipe applies several CVE-YYYY-XXXX.patch files, CVE-check will accurately report that those specific vulnerabilities are mitigated, even though the base package version might otherwise be flagged.

Beyond patch application, CVE-check allows for explicit overrides of NVD scan results, providing crucial context that generic scanners often miss. Rybczynska illustrated this with several common scenarios:

1. NVD Entry Incorrect: Some CVE entries in the NVD database are simply wrong or do not apply to the specific software component. CVE-check allows an override stating that "the entry in the NVD database is incorrect," effectively dismissing the vulnerability for the Yocto build. This is particularly relevant given NVD's past issues with update delays.
2. Not Applicable to Platform: The Yocto Project builds Linux distributions. Therefore, vulnerabilities that are specific to other operating systems, such as Windows, are irrelevant. An override can declare "not applicable platform" for such CVEs, preventing false positives.
3. Configuration-Based Non-Exploitability: A vulnerability might technically exist in the code but be non-exploitable due to the default configuration shipped with the Yocto package. For example, a vulnerability might only be exploitable if a specific file has a particular ownership, but the Yocto default configuration ensures different, secure ownership. The CVE-check can be configured with an override stating "not applicable because of a configuration," indicating the vulnerability cannot be exploited under the default build parameters.

These contextual overrides represent a sophisticated form of VEX-like information, as they communicate why a known vulnerability does not pose a risk in a specific product context. The experiment to generate standard VEX from this data involved mapping these internal statuses to existing VEX formats. While many complete VEX entries could be generated, the existing standards (like CISA VEX and OpenVEX) often lacked direct, machine-readable equivalents for all Yocto's nuanced statuses (e.g., "NVD database is wrong" or "abandoned project, no fix"). This necessitated the addition of non-standard extensions to the OpenVEX format, which is the current basis for the generated VEX output.

The talk also highlighted critical "open problems" stemming from the highly composable nature of Yocto builds:

• Layer Inheritance and VEX Modification: Since downstream layers (from vendors or manufacturers) can modify build configurations or apply additional patches, they can inadvertently invalidate VEX statements made by upstream layers. If an upstream layer declares a vulnerability non-exploitable due to a default disabled configuration, a downstream layer enabling that configuration would render the original VEX statement false.
• Vulnerabilities in Custom Patches: Custom patches, while often used to fix existing issues, can also introduce new vulnerabilities or unblock previously mitigated ones. Tracking the security implications of these bespoke modifications within the VEX framework is a complex inheritance problem that currently lacks a robust solution.

Despite these challenges, the Yocto Project's internal CVE-check provides a solid foundation for generating VEX, and the current efforts are producing OpenVEX-based documents with necessary custom statuses. The code for this VEX generation is open source, indicating a commitment to transparency and community collaboration in addressing these complex issues.

Demo / Proof of Concept

▶ Watch: Overriding NVD scan results with specific vulnerability justifications (10:00)

While the talk did not feature a live demonstration or explicit proof of concept in the traditional sense, Marta Rybczynska affirmed that the capability to generate VEX documents from Yocto Project metadata is already functional. She stated, "we do generate vex today. It's not a mainline option of the octo project... The current vex output is based on open vex but it has some additional statuses that we had to add."

This indicates that the experiment described in the talk has moved beyond theoretical exploration into practical implementation. The code for this VEX generation is open source and part of a larger software effort focused on generating CVE-related information outside the immediate build process. The speaker invited interested parties to examine the code, implying that the proof of concept is available for inspection and use by the community. The focus of the talk was primarily on the methodology, the challenges encountered, and the ongoing efforts to standardize and refine this VEX generation process rather than showcasing a live execution of the tool.

Defensive Implications

▶ Watch: Yocto Project's features resembling VEX (Vulnerability Exploitability eXchange) (12:00)

For organizations building embedded systems with the Yocto Project, the insights from this talk offer several critical defensive implications:

1. Leverage Yocto's CVE-check: Defenders should fully utilize the Yocto Project's integrated CVE-check tool. Understanding its capabilities, especially its ability to account for specific patches (CVE-YYYY-XXXX.patch files) and its override mechanisms (for incorrect NVD entries, platform-specific non-applicability, and configuration-based non-exploitability), is crucial for achieving accurate vulnerability assessments tailored to the embedded context.
2. Embrace Automated VEX Generation: The sheer volume of packages (at least 200 for a small embedded product) makes manual vulnerability assessment and VEX generation impractical. Defenders should prioritize and advocate for automated VEX generation solutions, ideally integrated into their Yocto build pipelines, to ensure continuous and consistent vulnerability communication.
3. Contribute to VEX Standardization: The talk highlighted gaps in current VEX standards for expressing embedded-specific statuses (e.g., "NVD incorrect," "abandoned project," complex configuration-based non-exploitability). Security teams and embedded vendors should actively participate in VEX working groups (e.g., CSAF, OpenVEX, OpenSSF) to push for extensions or new statuses that accurately reflect these realities, fostering better machine-readability.
4. Prepare for Multi-Processor VEX Merging: Many embedded devices contain multiple processors, often built with different toolchains and potentially generating separate Software Bill of Materials (SBOMs) and VEX documents. Defenders must anticipate the need to merge and reconcile these disparate VEX outputs into a unified view for the entire product. This necessitates robust tooling and a consistent approach to VEX formats across different components.
5. Focus on Machine-Readable VEX: Rybczynska emphasized the preference for machine-readable fields over human-readable text for VEX statuses. Defenders should advocate for and implement VEX formats that prioritize enumerations and structured data, enabling full automation of vulnerability processing without manual intervention, which is particularly vital for organizations without dedicated security teams.
6. Account for Layer Inheritance and Customizations: The dynamic nature of Yocto layers, where downstream layers can modify configurations or introduce custom patches, creates complex inheritance challenges for VEX statements. Defenders need to develop strategies, perhaps through re-evaluation or re-generation of VEX at the final product layer, to ensure that VEX statements remain accurate throughout the entire build chain, including vendor-specific modifications.
7. Proactive Vulnerability Enrichment: Recognizing that NVD alone is insufficient, embedded security teams should consider enriching their vulnerability data with additional context from other sources and internal assessments. This "vulnerability enrichment" process, which can then be expressed through VEX, provides a more accurate picture of actual risk.

By adopting these defensive strategies, embedded vendors can move beyond basic vulnerability scanning towards a more mature, contextual, and automated approach to vulnerability management, ultimately enhancing the security and resilience of their products in an increasingly regulated landscape.

Key Takeaways

• The Yocto Project has robust internal mechanisms for identifying and mitigating vulnerabilities, effectively performing "VEX-like" behavior for years, but formalizing this into standard VEX documents is a crucial next step.
• Generating standard VEX from Yocto metadata is largely possible, but current standards like CISA VEX and OpenVEX lack specific, machine-readable statuses required for embedded contexts (e.g., "NVD entry is incorrect," "abandoned project," detailed configuration-based non-exploitability).
• The layered architecture of Yocto builds introduces complex challenges for VEX, as downstream layers (vendor or manufacturer specific) can modify configurations or apply patches that invalidate upstream VEX statements.
• There is a strong need for VEX standards to evolve to include more machine-readable fields and statuses that accurately capture the nuances of vulnerability applicability in highly customized embedded systems, reducing reliance on human-readable text.
• Automated VEX generation is essential for embedded products, which typically involve hundreds of packages, making manual vulnerability assessment unsustainable, especially for manufacturers lacking dedicated security teams.
• Future VEX solutions for embedded systems must address the challenge of merging VEX documents from multiple processors and toolchains within a single product, requiring consistent formats and robust conversion capabilities.

About the Speaker(s)

Marta Rybczynska is a highly experienced professional with over 12 years of dedicated work in the open-source community. Her expertise spans various domains, with a significant portion of her career focused on security, including the development and implementation of security processes. In recent times, Rybczynska has been deeply involved with the Cyber Resilience Act in Europe, indicating her commitment to regulatory compliance and cyber security policy. She recently founded a company that is 100% focused on open-source security, underscoring her passion and specialization in this field. Her insights shared in this talk are presented as her own opinion and do not necessarily reflect an agreement or official stance of any specific project, including the Yocto Project, which she covered extensively. Marta's background in telecom, including a PhD, also hints at her deep technical understanding of complex systems, particularly relevant to the embedded domain.