Fragile by Design: Large-Scale Evidence of Supply Chain Risk

Fragile by Design: Large-Scale Evidence of Supply Chain Risk

Speakers: Gary Schwartz (NetRise)

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=cvnQhjpDnyo

Overview

In his compelling VulnCon talk, "Fragile by Design: Large-Scale Evidence of Supply Chain Risk," Gary Schwartz of NetRise delivered a sobering assessment of the systemic issues plaguing software supply chain security. Moving beyond the often-repeated adage of supply chain fragility, Schwartz presented extensive research data demonstrating a pervasive visibility problem that fundamentally undermines organizations' ability to understand, prioritize, and manage risk effectively. The core message is clear: current security tools and methodologies, largely reliant on manifest-declared components and application-layer insights, provide an incomplete and dangerously misleading picture of an organization's true security posture.

Schwartz, representing NetRise, a software supply chain security vendor specializing in binary analysis, shared insights derived from analyzing nearly 19,000 firmware and container assets over three years. His research revealed that a vast majority of vulnerabilities and critical misconfigurations reside in components that are entirely invisible to traditional scanning methods. This hidden risk landscape extends beyond conventional CVEs to encompass critical non-CVE issues and the profound implications of upstream open-source dependencies and maintainer trust.

The talk serves as a critical call to action for security professionals, urging a paradigm shift from superficial scanning to deep binary analysis and comprehensive inventory management. By exposing the significant gap between reported vulnerabilities and actual deployed software components, Schwartz underscores the urgent need for tools and processes that reveal the "software reality" – a complete and accurate understanding of everything running within an environment. Without this foundational visibility, effective prioritization becomes impossible, leaving organizations vulnerable to a wide array of unaddressed threats, as evidenced by recent supply chain incidents like those involving Trivy, Light LLM, and Axios.

Background

▶ Watch: Systematic issues in supply chain vulnerability (0:40)

The notion of a "fragile" software supply chain has become a pervasive, almost trite, topic in cybersecurity discussions. Yet, as Gary Schwartz emphasizes, its persistent relevance stems from deeply ingrained, systemic issues that continue to challenge even the most sophisticated security teams. The problem, as NetRise's research vividly illustrates, is not merely about the existence of vulnerabilities, but about a fundamental lack of visibility into the actual components that comprise modern software.

Historically, vulnerability management and software composition analysis (SCA) have largely relied on analyzing manifests, source code, or the application layer of software. This approach assumes that declared dependencies and readily available metadata accurately reflect the deployed software stack. Tools like the National Vulnerability Database (NVD), while crucial for tracking Common Vulnerabilities and Exposures (CVEs) associated with Common Platform Enumerations (CPEs), are designed to report on known issues within declared product versions and application-level components. This system works well for vendors like Cisco, which diligently report and patch vulnerabilities in their application-layer products, leading to thousands of CPEs and CVEs.

However, this traditional paradigm falters dramatically when confronted with the realities of complex software, particularly firmware and container images. These artifacts often embed a multitude of components—operating system kernels, open-source libraries, statically linked binaries, and vendored code—that are not explicitly declared in application manifests. The problem is exacerbated by legacy systems, where source code or SBOMs are often unavailable, leaving defenders blind during incident response. This fundamental disconnect between what is declared and what is actually present creates a vast "invisible" attack surface. Schwartz highlights that this lack of comprehensive inventory is the root cause of the prioritization problem, as organizations struggle to manage risks they cannot even see, let alone assess or mitigate. NetRise, with its focus on analyzing binary images, directly addresses this gap by seeking to reveal the full "software reality" at the deepest possible level.

Key Findings

▶ Watch: 99% of vulnerabilities in invisible components (6:30)

NetRise's extensive research, based on the analysis of materials collected over three years, provides compelling, large-scale evidence of the systemic visibility gaps in the software supply chain. The findings are stark and underscore the limitations of current security paradigms.

The research encompassed two primary datasets: almost 19,000 firmware assets across more than 380 organizations, and a smaller but similarly revealing set of container images. Across these firmware assets, NetRise identified a staggering 467 million vulnerabilities, which consolidated into just under 500,000 unique vulnerabilities. This sheer volume alone highlights the scale of the challenge.

A critical discovery emerged from the analysis of container images: 25% of the components within these images were identifiable only through binary analysis. This means a quarter of the software components in containers would remain entirely invisible to traditional manifest-based or source code scanning tools. The implications for firmware were even more pronounced:

• Almost 60% of firmware assets had no manifest-declared components at all. This means that for nearly two-thirds of the analyzed firmware, relying on an SBOM or manifest would yield absolutely no information about its internal composition.
• Overall, two-thirds of the components found within the analyzed firmware packages were invisible – they were not listed in any manifests.
• Perhaps the most alarming statistic: 98.7% (almost 99%) of all vulnerabilities identified in firmware resided within these invisible components. This demonstrates that the vast majority of an organization's vulnerability exposure is completely unaddressed by conventional scanning methods.

Schwartz illustrated the NVD's limitations with a powerful example. While the NVD effectively tracks application-layer issues for vendors like Cisco (35,000 CPEs, 6,500 CVEs), other companies producing fundamental networking devices might have zero CPEs and CVEs listed. Yet, NetRise's binary analysis of devices from such vendors revealed an average of 2,700 CVEs per device. This massive discrepancy is largely attributable to vulnerabilities in underlying components, particularly the Linux kernel, which is ubiquitous in modern devices but often invisible to application-centric scanning.

Further analysis highlighted the concentrated nature of risk: the top 10 components (including Glib and the Linux kernel) accounted for 93% of the total vulnerability exposure. Critically, these components are predominantly not part of the application layer. The Linux kernel, in particular, contributes orders of magnitude more vulnerabilities than other components. The research also uncovered instances of incredibly old, unmaintained code, with CVEs dating back to 1999 still present in devices, indicating a severe lack of scrutiny and proactive removal of unnecessary components.

Beyond CVEs, the research exposed significant non-CVE risks. Binary analysis frequently uncovered critical misconfigurations such as:

• User IDs with no passwords
• Thousands of expired certificates
• Public and private key pairs stored on the same device

These findings are not just security flaws; they are indicators of a profound lack of oversight, suggesting that "no one's looking" at the underlying device configurations.

Schwartz also delved into the broader implications for open-source software. He argued that the current open-source model, which allows broad contributions, creates a "trust problem." While SCA tools typically examine downstream dependencies, they often fail to account for upstream dependencies. A malicious contribution several layers up in the dependency chain can silently compromise downstream projects. The speaker referenced recent incidents involving Trivy, Light LLM, and Axios, noting that these compromises were not CVE-related but rather supply chain events, highlighting the inadequacy of CVE-centric approaches.

The concept of "blast radius" was illustrated with an example involving Anthropic packages. A hypothetical divestment from Anthropic would involve identifying 476 package names, 3,434 package versions across seven ecosystems, leading to 127,000 direct or transitive package versions and impacting 202 million Docker Hub pulls. This demonstrates the immense complexity and difficulty of identifying and remediating exposure to a single component across a vast ecosystem.

Finally, NetRise's research extended to maintainer trust and provenance. By analyzing over 100 million repositories and tens of thousands of packages on GitHub, NetRise developed methods to infer the geographic origin of contributors based on behavioral patterns like working hours and holiday schedules. This was demonstrated by identifying a malicious contribution to the Beef Project, an open-source pen-testing tool, originating from Russia, based on these work patterns rather than spoofed IP addresses. This capability provides a crucial layer of trust intelligence, enabling organizations to make informed decisions about their open-source dependencies.

Technical Deep Dive

▶ Watch: Filtering vulnerabilities for true impact (16:00)

NetRise's approach to uncovering supply chain risk is rooted in a sophisticated methodology that goes far beyond traditional manifest scanning. The core of their technique involves binary analysis – directly inspecting the compiled software artifacts rather than relying on declared metadata.

The process begins by unpacking and reverse engineering firmware images or container images. This involves recreating the complete file system of the artifact. By doing so, NetRise can identify all components present, regardless of whether they are explicitly declared in a manifest, statically linked, or vendored. This deep inspection is critical because, as Schwartz demonstrated, a significant portion of software components (25% in containers, two-thirds in firmware) are otherwise invisible.

This binary-level analysis allows NetRise to identify components like the Linux kernel, which is a massive source of vulnerabilities but often overlooked by application-layer tools. While the NVD and CPE system are effective for tracking vulnerabilities in specific product versions (e.g., cisco:ios:15.0), they are ill-equipped to track issues within an embedded Linux kernel or a statically linked version of OpenSSL or zlib. Schwartz referenced a talk by Craig Heffner and Peter Eastman, which further highlighted how static linking and vendoring can introduce vulnerable versions of libraries that are not reflected in declared manifests, creating a dangerous disconnect between what a developer intends and what is actually built into the final artifact.

Beyond identifying CVEs in hidden components, the technical deep dive extends to uncovering non-CVE risks. By rebuilding the file system, NetRise can scan for critical misconfigurations that signify poor security hygiene. Examples include:

• Passwordless user accounts (e.g., user IDs with no passwords)
• Thousands of expired certificates
• Presence of public and private key pairs on the same device

These are not vulnerabilities in the traditional sense, but fundamental security flaws that can be exploited, likened by Schwartz to leaving the "keys in the front door" even if all windows are sealed.

A significant aspect of NetRise's technical contribution is its ability to refine vulnerability prioritization. While acknowledging the importance of scores like EPSS (Exploit Prediction Scoring System), Schwartz emphasized that these are just one component. NetRise's binary analysis allows for a highly granular filtering process:

1. Network Accessibility: By understanding the full file system and network configurations, NetRise can determine which CVEs are accessible over the network.
2. Auto-run at Startup: The analysis identifies components and processes that auto-run at system startup (e.g., via cron jobs or systemd units).
3. Weaponization & CISA KEV: Filtering for vulnerabilities that are known to be weaponized or appear on the CISA Known Exploited Vulnerabilities (KEV) list.

By combining these filters, an initial set of 2,700 CVEs found on a device could be quickly triaged down to a handful of truly critical, actionable vulnerabilities, transforming "noise" into focused remediation targets. This shifts the focus from sheer volume to contextualized risk.

NetRise's analysis also extends into the realm of trust intelligence for open-source components. To determine the blast radius of a compromised dependency, their tools analyze direct and transitive package versions across various ecosystems. The Anthropic example demonstrated how a single component can ripple through hundreds of thousands of versions and millions of Docker Hub pulls, making traditional remediation efforts impractical without this comprehensive mapping.

For identifying malicious contributions, NetRise goes beyond simple IP address logging. Their system analyzes developer work patterns, including typical working hours and holiday schedules. This behavioral analysis was instrumental in identifying a Russian-based contributor to the Beef Project, despite any attempts to mask their origin. This sophisticated provenance analysis provides a deeper understanding of the human element in the supply chain.

Finally, to operationalize these insights, NetRise has developed a CLI tool that functions as a policy engine. This tool can be integrated directly into build pipelines. It allows organizations to define custom policies, such as disallowing contributions from specific nation-states or failing a build if certain risky components or patterns are detected. This proactive enforcement mechanism is designed to prevent compromised dependencies from ever entering automated builds, thereby closing the narrow windows of vulnerability that attackers often exploit. The definition of "firmware" in this context was clarified as the "software package that gets installed on our hardware device," which can include Linux or other real-time operating systems.

Demo / Proof of Concept

▶ Watch: The trust problem in open source supply chains (18:30)

While Gary Schwartz's talk did not feature a live, interactive demonstration of NetRise's tools in action, the presentation itself served as a powerful proof of concept through the extensive data and visualizations derived from NetRise's research. Schwartz shared numerous charts, statistics, and real-world examples that illustrate the capabilities and findings of their binary analysis platform. The detailed breakdown of invisible components, the discrepancy between NVD data and actual device vulnerabilities, the identification of non-CVE risks, and the complex dependency mapping for Anthropic and the Beef Project all functioned as compelling evidence of NetRise's analytical power. These case studies and aggregated data points effectively demonstrated what their technology can achieve and why it is essential for understanding the true "software reality."

Defensive Implications

▶ Watch: Policy engines prevent compromised dependencies in builds (26:00)

The insights presented by Gary Schwartz offer crucial defensive implications for organizations grappling with the complexities of software supply chain security. The overarching message is a call for a fundamental shift in defensive strategy, moving beyond superficial scanning to embrace a deeper, more comprehensive understanding of deployed software.

1. Embrace Binary-Level Analysis: Defenders must recognize the severe limitations of manifest-based and application-layer scanning. The vast majority of vulnerabilities and hidden components reside in the binary image, requiring tools capable of unpacking, reverse engineering, and rebuilding the file system of firmware and container images. This shift is non-negotiable for achieving true visibility.
2. Prioritize Comprehensive Inventory: The talk repeatedly emphasized that an "inventory problem" precedes a "prioritization problem." Organizations must establish a complete and accurate inventory of all software components, including operating system kernels, statically linked libraries, and vendored code, not just declared dependencies. This foundational software reality is essential before any meaningful risk assessment can occur.
3. Expand Risk Assessment Beyond CVEs: Defensive strategies must evolve beyond a singular focus on CVEs. The prevalence of critical non-CVE risks such as passwordless accounts, expired certificates, and exposed key pairs demonstrates that basic security hygiene issues are rampant and easily exploitable. Scanning for these misconfigurations at the binary level should become a standard practice.
4. Adopt Contextualized Prioritization: Simply relying on CVE severity scores (e.g., CVSS, EPSS) is insufficient. Defenders should implement advanced prioritization filters that consider network accessibility, whether a vulnerability is auto-run at startup, its presence on the CISA KEV list, and its weaponization status. This allows security teams to triage thousands of potential issues down to a manageable, actionable few, maximizing the impact of remediation efforts.
5. Scrutinize Open Source Provenance and Dependencies: Given the "trust problem" in open source, organizations must gain deeper insights into their upstream dependencies. This includes understanding the blast radius of any potential compromise and employing tools that can analyze maintainer trust and contributor provenance (e.g., identifying geographic origin based on work patterns). This intelligence is vital for managing risks associated with malicious contributions.
6. Integrate Policy Enforcement into Build Pipelines: To prevent compromised or untrusted components from entering the software supply chain, organizations should implement policy engines directly within their automated build processes. These tools can enforce custom policies (e.g., blocking contributions from certain regions, flagging specific risky components) and automatically fail builds that violate these rules, thereby shifting security left and preventing vulnerabilities at the source.
7. Demand Deeper Transparency from Vendors: The Log4j experience highlighted how long it can take for vendors to ascertain their full exposure. Defenders should be prepared to perform their own binary analysis on third-party products to understand their true risk posture, rather than solely relying on vendor attestations. This empowers organizations to proactively manage risks in their purchased software.
8. Proactive Component Cleanup: Identifying and removing unnecessary or outdated components (e.g., ancient CVEs from 1999) is a straightforward yet often overlooked defensive measure. Reducing the attack surface by eliminating irrelevant code reduces the burden on security teams and simplifies prioritization.

In essence, Schwartz advocates for a proactive, comprehensive, and context-aware approach to supply chain security, moving from reactive, incomplete scanning to a holistic understanding of the "software reality" and its associated risks.

Key Takeaways

• Visibility is the Primary Challenge: Traditional security tools provide an incomplete picture, missing a significant portion of software components and their associated vulnerabilities, particularly in firmware and container images.
• Binary Analysis is Essential: Deep binary analysis, which unpacks and recreates the file system of software artifacts, is critical to uncover invisible components and nearly 99% of vulnerabilities hidden from manifest-based scanning.
• Non-CVE Risks Are Prevalent and Critical: Beyond traditional CVEs, widespread misconfigurations (e.g., passwordless accounts, exposed keys, expired certificates) represent significant, often overlooked, security flaws that indicate a lack of basic oversight.
• Contextual Prioritization is Key: Effective vulnerability management requires filtering beyond severity scores. Prioritization must consider factors like network accessibility, auto-run status, weaponization, and presence on the CISA KEV list to focus on truly actionable risks.
• Trust Intelligence for Open Source is Imperative: Understanding the blast radius of open-source dependencies and the provenance of contributors (e.g., geographic origin inferred from work patterns) is vital to mitigate risks from malicious or untrusted components.
• Proactive Policy Enforcement: Integrating policy engines into build pipelines allows organizations to define and enforce security policies, preventing compromised or untrusted dependencies from ever entering automated builds.

About the Speaker(s)

Gary Schwartz is a speaker representing NetRise, a software supply chain security vendor. His work at NetRise focuses on understanding and analyzing binary images, including firmware and container images, to uncover hidden vulnerabilities and risks. He shared research conducted by NetRise over the last three years, demonstrating systematic issues in the software supply chain and advocating for comprehensive visibility as a critical foundation for effective security prioritization and management.