The Dependency Mirage: Hidden Vulnerabilities in Your Compiled Binaries

The Dependency Mirage: Hidden Vulnerabilities in Your Compiled Binaries

Speakers: Craig Heffner, Principal Security Researcher, NetRise; Peter Eacmen, Director of Research, NetRise

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=4TuSNHXdWTo

Overview

In "The Dependency Mirage: Hidden Vulnerabilities in Your Compiled Binaries," Craig Heffner and Peter Eacmen from NetRise shed light on a critical, yet often overlooked, aspect of software supply chain security: the presence of undeclared and vulnerable dependencies embedded directly within compiled binary files. The talk, delivered at VulnCon, challenges the conventional wisdom that software bill of materials (SBOMs) and traditional software composition analysis (SCA) tools provide a complete picture of an application's dependencies. Instead, Heffner and Eacmen argue that a significant blind spot exists in the form of statically linked libraries, vendored code, and build-time injected components, which can harbor severe security vulnerabilities unbeknownst to developers and security teams.

The speakers, leveraging their extensive experience in firmware and binary analysis—with Heffner notably being the creator of the widely used Binwalk tool—present compelling case studies that demonstrate how organizations are routinely exposed to critical CVEs through these hidden dependencies. They highlight that while source code analysis is a well-established practice, the true "ground truth" of what actually executes on a system lies within the compiled binary. This distinction is particularly crucial in an era where software supply chain attacks are on the rise and modern programming languages, such as Go and Rust, increasingly favor static linking by default.

This presentation is vital for anyone involved in software development, security, or compliance, as it exposes fundamental flaws in current dependency management practices. By illustrating how traditional methods fail to identify these embedded vulnerabilities, the talk underscores the urgent need for a paradigm shift towards comprehensive binary analysis. It provides actionable insights for security, development, and leadership teams to proactively address these hidden risks, emphasizing that a "trust but verify" approach to all dependencies, even internally generated SBOMs, is paramount for robust software security.

Background

▶ Watch: The Dependency Mirage: Understanding the core problem (1:30)

The increasing complexity of modern software development has led to an intricate web of dependencies, forming a software supply chain that spans numerous components, contributors, and build environments. Just as a grocery store consumer might scrutinize food labels to understand ingredients and origins, organizations strive to understand the components within their software. The rise of Software Bill of Materials (SBOMs) and Software Composition Analysis (SCA) tools has been a direct response to this need, aiming to provide transparency into declared dependencies, typically by analyzing source code, package manifests (like package.json, requirements.txt, pom.xml), and system package managers (e.g., RPM, Debian packages).

However, Heffner and Eacmen argue that this approach, while necessary, is fundamentally incomplete. The "Dependency Mirage" arises because the final compiled binary often contains dependencies that are not explicitly listed in source code manifests or tracked by system package managers. This problem stems from several factors:

1. Statically Linked Libraries: Instead of relying on a shared library installed system-wide (dynamic linking), developers might choose to compile a library's code directly into their application's binary. This bundles the library's functionality, making the application self-contained but also opaque to traditional dependency scanners. If the statically linked library contains a vulnerability, updating the system-wide shared library will not patch the embedded version.
2. Vendored Code: A common practice involves copying a third-party library's source code directly into a project's own repository. This "vendoring" allows developers to have full control over the code, including making custom modifications or ensuring a specific version is used, independent of external repositories. However, it also means that the vendored code is no longer tracked as an external dependency by package managers and can easily become outdated and vulnerable without detection.
3. Build Environment Injection: The environment in which software is compiled can introduce dependencies or variations that are not explicitly part of the source code. This includes specific compiler versions, optimization flags, system-level libraries available during compilation, and even build caches. Different build environments can lead to functionally identical binaries containing different versions of underlying components, some of which may be vulnerable.

The speakers illustrated these issues with real-world customer scenarios. In one instance, a customer’s system, when scanned by traditional methods (e.g., RPM), reported OpenSSL version 3.0.7, which was considered secure. However, NetRise’s binary analysis revealed that a Python shared object on the same system had OpenSSL version 3.0.0 statically linked within it. This older, vulnerable version was completely invisible to the customer's conventional security tools, exposing them to unknown risks.

Another compelling example involved the common Unix utility rsync. While package managers reported a specific zlib version (e.g., 1.3.1.1), NetRise discovered that rsync contained a vulnerable version of zlib due to its authors having "blindly copied the zlib source code into their own source tree" years ago. By default, rsync compiles and uses this vendored version rather than the system's zlib library, perpetuating an unpatched vulnerability within the rsync binary itself. These examples underscore that the problem is not hypothetical but a tangible and widespread issue impacting operational security.

Key Findings

▶ Watch: Another hidden vulnerability: zlib source copied into rsync (6:10)

The central revelation of this talk is the existence of a "Dependency Mirage," where the perceived list of software dependencies, often derived from SBOMs and package managers, significantly deviates from the actual dependencies embedded within compiled binaries. This mirage leads to a dangerous blind spot, leaving organizations unknowingly exposed to critical vulnerabilities. The speakers presented several key findings that highlight the scope and impact of this issue:

Firstly, traditional SCA tools and package managers are fundamentally insufficient for identifying hidden dependencies. As demonstrated by the customer case studies, relying solely on declared dependencies (e.g., RPM output) can provide a false sense of security. The OpenSSL example, where RPM reported version 3.0.7 but a statically linked Python shared object contained vulnerable 3.0.0, clearly illustrates this gap. Similarly, the rsync example showed how vendored code (zlib) can bypass conventional detection mechanisms.

Secondly, the problem is not isolated but widespread and impactful at scale. In one specific customer asset analyzed by NetRise, 47 separate components with hidden dependencies were identified. This breakdown included:

• 14 instances of statically linked libraries, where components were compiled directly into the binary.
• 3 vendored libraries, indicating code copied directly into the project's source tree.
• 30 instances of build-time injection, where dependencies were introduced through the compilation process itself, often without explicit declaration.

Crucially, normal SCA tools found none of these 47 issues, leaving the customer entirely unaware of these potential attack vectors. This finding strongly suggests that many organizations are operating with a significant, unaddressed attack surface.

Thirdly, modern programming languages like Go and Rust exacerbate the problem due to their default compilation behaviors. Both languages are designed to produce self-contained binaries by statically linking almost all dependencies, with the notable exception of libC. While this simplifies deployment by eliminating runtime dependency issues, it means that every module and its transitive dependencies are baked into the final binary. As Craig Heffner explained, a simple cargo.toml file listing five direct dependencies in a Rust project might result in 10, 15, or even 20 modules being pulled in and statically compiled. If a vulnerability is discovered in any of these embedded versions six months down the line, the entire application must be recompiled and redeployed, a process that is often complex and time-consuming, especially for a vast and poorly understood dependency tree.

Finally, the talk emphasizes that the compiled binary represents the "ground truth" of what actually runs on a computer. Source code analysis is valuable, but it's the binary that executes instructions and interacts with the system. Therefore, neglecting binary analysis means missing a critical layer of security scrutiny, leaving organizations vulnerable to unintentional inclusions, supply chain attacks, or malicious injections that might bypass source code checks.

Technical Deep Dive

▶ Watch: Practical steps for inspecting software for hidden components (14:00)

Understanding the "Dependency Mirage" necessitates a technical appreciation of how these hidden components manifest within binaries and the methods—both manual and automated—to uncover them. The speakers delved into the technical underpinnings of binary analysis, highlighting the decision points that introduce these hidden dependencies and the tools available for investigation.

The presence of hidden dependencies stems from fundamental choices made during the software build process. These choices dictate what code ultimately makes it into the final executable:

1. Library Version Selection: Developers use package managers like npm, pip, or Conan to declare dependencies. They can specify exact versions, major versions, or simply "latest." However, the transitive dependencies pulled in by these choices can be vast and often poorly understood, leading to a much larger set of compiled code than initially intended.
2. Static vs. Dynamic Linking: This is a critical decision. Dynamic linking relies on shared libraries (e.g., DLLs on Windows, shared objects on Linux) that are installed system-wide. The application loads these at runtime. Static linking, conversely, embeds a copy of the library's compiled code directly into the application binary. While static linking creates self-contained executables, it means any updates to the system-wide library will not affect the embedded version, requiring a full recompile and redeployment to patch vulnerabilities.
3. Optimization and Compiler Flags: Build tools (e.g., makefiles, configure scripts) allow developers to pass numerous flags to compilers. These flags can enable or disable features, change optimization levels, or even dictate which versions of certain libraries are used. For instance, the speakers suggested that for the Python OpenSSL issue, there was likely an option to prevent static linking of SSL, forcing Python to use the system's OpenSSL. Similarly, rsync has an option to use the system zlib instead of its vendored version. Overlooking or misconfiguring these flags can inadvertently bake in vulnerable components.
4. Build Caches: Package managers and build systems often use caches to speed up compilation. If these caches are not regularly updated or properly managed, an older, vulnerable version of a dependency might be pulled from the cache instead of the latest secure version, even if the manifest specifies the newer version.

To manually uncover these hidden dependencies, security researchers can employ a series of techniques:

1. Extraction: For complex software packages like containers, large firmware images, or ISOs, the first step is to deconstruct them into their individual components. Tools like Binwalk (created by Craig Heffner) and unblob are invaluable here, capable of identifying and extracting embedded files, archives, and filesystems from opaque binary blobs.
2. String Analysis: A basic but effective technique involves running the strings utility on a binary and then greping for known library names, function names, or debug messages. For example, strings mybinary | grep OpenSSL might reveal hardcoded version strings or function calls related to OpenSSL. While this can be tedious for large binaries and requires prior knowledge of what to search for, it can quickly identify obvious inclusions.
3. ELF Header Inspection: For known binary formats like ELF (Executable and Linkable Format) on Linux, standard tools can inspect headers to identify dynamically linked libraries, imported functions, and symbols. This primarily reveals runtime dependencies that the binary expects to find on the system. While not directly identifying statically linked code, it provides crucial context about the binary's external interactions. For instance, if an ELF binary imports libssl.so, it tells you it's using OpenSSL, and you can then investigate the system's libssl.so for vulnerabilities.
4. Vendor Name Search: Often, vendors embed their names or abbreviations into function names, debug messages, or file paths within their components. Searching for these specific strings (grep -r "Foobar" in extracted files) can help identify proprietary or third-party code that might otherwise go unnoticed.

The manual nature of these techniques, however, quickly becomes impractical for large, complex systems with hundreds or thousands of binaries. This limitation highlights the necessity for automated binary analysis solutions that can perform these tasks at scale, correlating findings across an entire software ecosystem to provide a comprehensive and accurate dependency graph.

Demo / Proof of Concept

▶ Watch: Building an organizational binary analysis program (19:45)

While no live, interactive demonstration of the NetRise platform was performed during the talk, the speakers presented compelling customer use cases that served as practical proofs of concept for their methodology. These real-world scenarios effectively illustrated how NetRise's approach to binary analysis uncovers hidden vulnerabilities that evade traditional software composition analysis (SCA) tools and SBOMs.

The primary "demonstrations" were the detailed accounts of the customer's experiences:

1. OpenSSL Vulnerability in Python Shared Object: The speakers described a situation where a customer's system reported OpenSSL version 3.0.7 via standard package management utilities (like RPM). However, NetRise's analysis of the compiled binaries identified a Python shared object containing a statically linked, vulnerable version of OpenSSL 3.0.0. This finding was critical because the customer was completely unaware of this older, insecure dependency embedded within their operational software. The screenshot shown during the talk depicted NetRise's product interface reporting the vulnerable OpenSSL version and its association with the python binary.
2. zlib Vulnerability in rsync: Another customer scenario involved the widely used rsync utility. While the system's package manager might have indicated a specific zlib version (e.g., 1.3.1.1), NetRise's deep dive into the rsync binary revealed that it contained an older, vulnerable version of zlib. This was attributed to rsync's developers having copied the zlib source code directly into their project (vendoring) rather than dynamically linking against the system's zlib library. The screenshot highlighted how NetRise's tool identified zlib within the rsync binary, pointing to the hidden dependency.

These case studies, backed by specific version numbers and the stark contrast between what traditional tools reported versus what binary analysis uncovered, served as powerful evidence for the "Dependency Mirage." They demonstrated that NetRise's analytical capabilities could identify 47 separate hidden dependencies within a single customer asset, including 14 statically linked libraries, 3 vendored libraries, and 30 instances of build-time code injection, none of which were detected by conventional SCA. This effectively proved the concept that binary analysis is indispensable for a complete and accurate understanding of software dependencies and associated vulnerabilities.

Defensive Implications

▶ Watch: Q&A: How to identify original assets from compiled binaries (33:20)

The insights from "The Dependency Mirage" carry significant defensive implications, necessitating a multi-faceted approach involving security teams, development teams, and leadership. Addressing hidden binary dependencies requires a shift in mindset and investment in new processes and tools.

For Security Teams:

• Build Binary Analysis Capabilities: Security teams must develop or acquire the expertise and tools for binary analysis. This goes beyond traditional source code scanning and requires understanding compiled code. Tools like Binwalk and unblob are starting points for extraction, but comprehensive analysis requires more advanced platforms.
• Establish Baselines and Ground Truth: When evaluating new software or updates, security teams need to establish a baseline of what's actually inside the binaries. This involves initial manual verification to confirm findings (true positives vs. false positives) and then documenting these findings, including how they were discovered.
• Create a Database of Correlations: Over time, patterns will emerge. For example, consistently finding a specific version of zlib embedded in rsync binaries. Documenting these correlations helps in future analyses and triaging.
• Continuous Monitoring: Software is not static. As new features are added, bugs are patched, and dependencies evolve, continuous monitoring of compiled binaries is essential to prevent regressions and the introduction of new, undocumented vulnerabilities.
• Triage and Verification: Expect initial pushback from development teams when reporting hidden dependencies. Security teams need experienced personnel capable of triaging findings, presenting evidence, and verifying the presence and impact of these vulnerabilities.

For Development Teams:

• Analyze Compiled Binaries Post-Build: Don't just rely on source code analysis. After the build process, perform checks on the final compiled binaries to identify unexpected libraries or components. This can catch issues introduced by the build environment or transitive dependencies.
• Understand and Document Build Processes: For every project, thoroughly document the build process, including specific compile flags, configure options, and features enabled/disabled. Understand the implications of each choice, especially regarding static vs. dynamic linking. This knowledge is crucial for making informed decisions and debugging issues.
• Review Source Code for Vendored Components: Actively search for and identify vendored code within your repositories. Treat vendored code as an explicit dependency that requires regular updates and security scrutiny, just like external libraries.
• Comprehensive Patching Strategies: When a vulnerability is found in a library, do not assume a single update will resolve all instances. Perform a full rebuild of the entire application suite or, at minimum, a thorough analysis to ensure all instances of the vulnerable library (including statically linked and vendored versions) are updated. Maintain a well-documented list of all dependency locations.
• Mindful Dependency Management: Be aware of how package managers (npm, pip, Conan) pull in transitive dependencies. Regularly review the full dependency tree, not just direct dependencies, and understand the implications of static linking, particularly with languages like Go and Rust.

For Leadership:

• Strategic Investment: Recognize that building robust binary analysis capabilities is a significant investment in both time and money. It requires dedicated teams, specialized tools, and new processes. This is no longer a luxury but a necessity for modern software security.
• Prioritize Security Over Features: Leadership must prioritize security and supply chain integrity alongside product features and profitability. The long-term cost of a breach due to an unknown vulnerability far outweighs the upfront investment in comprehensive security.
• Empower Security Teams: Provide security teams with the necessary guidance, resources, and authority to conduct thorough analyses and implement new processes. Support their findings, even if they challenge existing assumptions within development.
• Set SBOM Requirements: Evolve SBOM requirements to go beyond declared dependencies. Mandate the documentation of external vendors (especially closed-source), specific build flags and compile options, and the rationale behind static vs. dynamic linking decisions. This ensures a more accurate and comprehensive understanding of the software's true composition.
• Foster a Culture of "Trust but Verify": Instill a company-wide culture where all dependencies, whether from third-party vendors or internally generated, are subject to rigorous verification. This proactive stance is critical in mitigating the risks posed by the "Dependency Mirage."

Key Takeaways

• SBOMs and Manifests are Often Incomplete: While valuable, traditional SBOMs, manifest files, and package manager reports frequently present an incomplete picture of an application's true dependencies. They often miss components that are statically linked, vendored, or injected during the build process, leading to a "Dependency Mirage." Organizations must adopt a "trust but verify" approach.
• Hidden Dependencies are Prevalent in Compiled Binaries: Every compiled binary contains potential hidden dependencies, which can be vulnerable. These include statically linked libraries (e.g., OpenSSL in a Python shared object), vendored code (e.g., zlib in rsync), and build-time injected components. Traditional SCA tools typically fail to detect these.
• Modern Languages Exacerbate the Problem: Languages like Go and Rust, by design, favor static linking of almost all dependencies into the final binary. While this simplifies deployment, it means that every transitive dependency is baked in, making vulnerability remediation (requiring full recompilation) and dependency tracking significantly more complex.
• Binary Analysis Provides the "Ground Truth": Source code analysis is crucial, but it's the compiled binary that actually executes on a system. Therefore, comprehensive binary analysis is indispensable for understanding the true composition of software, identifying all embedded components, and uncovering vulnerabilities that would otherwise remain hidden. It complements, rather than replaces, source code analysis.
• Proactive Measures are Essential: Organizations must invest in binary analysis capabilities, document build processes meticulously, and enforce continuous monitoring. Development teams need to be aware of the implications of static linking and vendoring, while leadership must prioritize resources for robust security practices to mitigate the risks of unknown dependencies.
• Most Organizations Likely Have Hidden Vulnerabilities: Given the pervasive nature of static linking, vendoring, and complex build environments, it is highly probable that most organizations currently have hidden, vulnerable dependencies within their deployed software that are not tracked by their existing security tools.

About the Speaker(s)

Craig Heffner is a Principal Security Researcher at NetRise and is widely recognized in the security community as the creator of Binwalk, an open-source tool for analyzing, reverse engineering, and extracting firmware images. His extensive background spans over a dozen years, focusing on understanding the intricate details of what lies within binary blobs and firmware. Heffner's expertise lies in developing novel methods for deep binary analysis, making him a leading voice in uncovering hidden software dependencies and vulnerabilities.

Peter Eacmen serves as the Director of Research at NetRise. With a career also extending over a dozen years, Eacmen has specialized in dissecting firmware and other binary assets to identify underlying components and security risks. Together with Craig Heffner, he brings a wealth of practical experience in offensive cyber operations and engineering to the challenge of securing the software supply chain. The NetRise team also includes other notable contributors such as Michael Scott, a former Marine with experience in offensive use cases for cyber operations, and Anthony Federson, an engineer with an illustrious career in the field.