From Discovery to Fix: What 10,000 Open Source Projects Reveal About CVE Remediation

From Discovery to Fix: What 10,000 Open Source Projects Reveal About CVE Remediation

Speakers: Mor Weinberger, Software Architect, Aqua

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=IkMwLyik2KA

Overview

In this insightful talk at FIRSTCON, Mor Weinberger, a Software Architect at Aqua, challenges the security industry's long-standing obsession with vulnerability discovery. While we've achieved unprecedented capabilities in finding security issues through advanced scanners, feeds, and contextual analysis, the corresponding backlog of unaddressed vulnerabilities never seems to shrink. Weinberger, along with co-author Leo Kaplan, an open-source maintainer and expert, presents a comprehensive analysis derived from scanning hundreds of open-source container projects daily for an entire year, processing over 10 million findings and charting 40,000 CVE journeys.

The core revelation of the research is both startling and actionable: approximately 90% of all Common Vulnerabilities and Exposures (CVEs) already have a fix available. For critical vulnerabilities, this number rises to an impressive 96.5%. This fundamentally shifts the problem from one of vulnerability availability to one of propagation – getting the existing fixes deployed into live systems. Despite the readily available patches, a significant 40% of these remediable CVEs persist within container images for over six months, exposing organizations to known and often exploitable risks.

Weinberger dissects this "gap" by segmenting the CVE lifecycle into two distinct journeys: from discovery to fix, and from fix to deployment. The talk explores the evolving role of Artificial Intelligence (AI) in both accelerating discovery and inadvertently overwhelming maintainers, alongside practical strategies for defenders to significantly reduce their attack surface. By highlighting the critical disconnect between finding vulnerabilities and effectively remediating them, Weinberger provides a roadmap for a more efficient and secure open-source ecosystem, emphasizing the critical role of maintainers and strategic deployment practices.

Background

▶ Watch: Introduction: The growing gap between discovery and fix (0:00)

For the past decade, the security industry has predominantly focused its efforts on enhancing vulnerability discovery. The proliferation of sophisticated scanners, enriched threat intelligence feeds, and advanced contextual analysis tools has led to an unprecedented ability to identify security flaws. However, this relentless pursuit of faster discovery has paradoxically resulted in an ever-expanding backlog of vulnerabilities that organizations struggle to address. This problem underscores a fundamental imbalance: while we excel at finding issues, our capacity to remediate them at scale has lagged significantly.

The CVE life cycle provides a framework for understanding this challenge. Ideally, a vulnerability is reported (by researchers, maintainers, or even AI), a fix is swiftly developed and committed by maintainers, a new release incorporating the fix is published, project maintainers rebuild their images to pull the updated version, and finally, users become aware of the new tags and update their deployments. In an ideal scenario, as seen with projects like curl, this entire process might take only a few days.

However, real-world complexities introduce significant friction at each stage. Many CVEs initially lack an available fix, leaving users with no immediate recourse. Even when maintainers develop a patch, publishing a new release can be delayed for various reasons. Subsequently, downstream project maintainers must rebuild their container images with the latest fixed versions, a step often overlooked or deprioritized. Finally, end-users frequently remain unaware that newer tags contain critical security updates, leading to prolonged exposure. This cumulative lag, particularly in containerized environments, creates a substantial security debt. To accurately diagnose and address these systemic issues, Weinberger and his team concluded that the CVE journey must be conceptually split: the first part, "Discovery to Fix," primarily involves maintainers and the vulnerability reporting ecosystem, while the second, "Fix to Deployment," focuses on the challenges faced by consumers in integrating and operationalizing those fixes.

Key Findings

▶ Watch: Maintainer overload: 'Death by a thousand slopes' from AI reports (8:00)

The research conducted by Mor Weinberger and Leo Kaplan, spanning 10,000 open-source container projects over a year and analyzing 10 million findings across 40,000 CVE journeys, revealed several critical insights that fundamentally redefine the challenge of vulnerability management:

1. Fix Availability is Not the Problem: A staggering 90% of all detected CVEs across the scanned projects already have a patch or fix available. This figure rises even higher for critical vulnerabilities, with 96.5% of critical CVEs having an existing fix. This directly contradicts the common perception that many vulnerabilities are unfixable, indicating that the industry's focus on "finding more problems faster" overlooks the fact that solutions for most existing problems are already on the shelf.
2. Propagation Lag is the Core Issue: Despite the high availability of fixes, the problem lies in their propagation. The study found that approximately 40% of CVEs with an available fix persist within container images for more than six months. This significant delay means that even when maintainers have done their job by developing and releasing a patch, users remain exposed to known vulnerabilities for extended periods, turning "fixed" vulnerabilities into ongoing risks.
3. Container Bloat is a Major Contributor: A significant portion of CVEs found in container images stems from bloated base images. Many containers are treated like "tiny VMs," including numerous extra packages, build tools, development headers, and utilities that are not essential for the application's core functionality in a production environment. The research indicated that a substantial number of these CVEs (around 56%) originate from such unnecessary components.
4. The "Latest" Tag is Often a Quick Win: Simply pulling the latest tag of a container image can yield substantial security improvements. The data showed that around 80% of latest tag images have fewer CVEs, leading to an average 40% reduction in CVE count compared to older versions. This highlights a simple yet often neglected action with high impact.
5. AI's Dual Impact: While AI demonstrates undeniable prowess in vulnerability discovery (e.g., Anthropic Opus finding 21 CVEs in Firefox in one month, OpenAI Codex surfacing thousands of high-severity issues), it also creates a significant burden on open-source maintainers. The flood of low-quality, AI-generated reports, termed "AI slope," leads to maintainer fatigue, exemplified by Curl's Daniel Stenberg shutting down his bug bounty program and the libxml2 maintainer stepping down. This indicates that AI is a force multiplier, but its application must prioritize quality and value over sheer volume.
6. CVE Assignment Matters: A fix that isn't properly flagged with a CVE can remain "invisible" to users, preventing them from prioritizing urgent updates. The process of CVE assignment, often managed by CVE Numbering Authorities (CNAs) like OpenJS or Aqua, is crucial for transforming a quiet code fix into a clear security signal, enabling timely remediation.

These findings collectively underscore that the security industry needs to pivot its focus from solely discovering new vulnerabilities to effectively managing the remediation and deployment of existing fixes.

Technical Deep Dive

▶ Watch: The bloat problem: Most containers are bloated with unnecessary packages (23:00)

The deep dive into CVE remediation begins by dissecting the two critical "journeys": from discovery to fix, and from fix to deployment. Each journey presents unique technical challenges and human factors.

Journey 1: From Discovery to Fix – The Maintainer's Burden

The initial phase revolves around identifying and patching vulnerabilities. AI has emerged as a powerful, yet double-edged, sword here.

• AI's Power in Discovery: The talk highlights the undeniable capabilities of AI models in automated vulnerability discovery. Anthropic's Opus model, for instance, reportedly found 500 previously unknown vulnerabilities in open-source projects, including 21 CVEs in Firefox within a single month – a number exceeding all other combined sources for the preceding year. Similarly, OpenAI's Codex model scanned millions of commits, surfacing tens of thousands of high-severity issues. This demonstrates AI's capacity to find bugs at a scale human researchers cannot match.
• The "AI Slope" and Maintainer Fatigue: The flip side of AI-driven discovery is the overwhelming volume of low-quality reports, dubbed "AI slope." Daniel Stenberg, the core maintainer of curl (a foundational piece of software), was forced to shut down his project's bug bounty program due to a "death by a thousand slopes" – a flood of AI-generated vulnerability reports, most of which were garbage. Each report required significant time from Stenberg and his team to read, reproduce, triage, and ultimately dismiss. This problem isn't confined to security reports; Hugging Face's CEO reported their biggest open-source repositories were receiving new pull requests every three minutes, making GitHub "unusable." This has even led GitHub to consider a "kill switch" for pull requests, underscoring the severity of the quality-over-quantity problem.
• Responsible vs. Irresponsible Disclosure: To mitigate maintainer fatigue and ensure efficient remediation, the talk outlines principles of responsible disclosure:

1. Report Privately: Avoid public disclosure before a fix is available.
2. Discuss CVE Assignment: Collaborate with maintainers directly regarding CVE IDs.
3. Help with Fix and Verification: Offer assistance in drafting and validating patches.
4. Reduce Fatigue: Work with maintainers, not against them, to streamline the process.

Conversely, irresponsible practices include asking for CVEs to "pad a resume," public disclosure before a fix, making maintainers figure out CVE assignments themselves, and complaining about slow fixes. An example of irresponsible practice was shown with libxml2, where a third-party database (Vol DB) assigned a CVE without the maintainer's knowledge, based on an unreproducible proof-of-concept (POC), causing significant wasted effort for the maintainer.

• The Role of CNAs: CVE Numbering Authorities (CNAs) play a crucial role in bridging the gap between a fix and its recognition as a security patch. Organizations like OpenJS have become CNAs to responsibly assign CVEs for their projects, ensuring proper disclosure and signaling. The speaker's company, Aqua, also acts as a CNA, as demonstrated when assisting the Ollama project. After Ollama shipped a fix for a critical vulnerability but couldn't assign a CVE themselves, Aqua stepped in as a CNA, validated the submission, informed MITRE, and gave Ollama a heads-up, leading to a CVE assignment within days. This process is vital because a fix without a CVE is "invisible" to users, who won't prioritize an update if they don't know it's a security patch.

Journey 2: From Fix to Deployment – The User's Challenge

Once a patch is committed, the journey shifts to ensuring users actually receive and implement it.

• Release and Propagation Delays: The first step is for the project to release a new version containing the fix. The speed of this release varies, often depending on the vulnerability's severity and the project's release cycle (hours for critical, weeks/months for lower severity). Following this, vulnerability scanners need to update their databases with the new CVE information, including affected components and versions, to correctly flag vulnerable software.
• Dependency Management: Consuming fixes becomes more complex with dependencies. Direct dependencies are relatively straightforward, but transitive dependencies (dependencies of dependencies) introduce significant lag and complexity.
• Container Rebuilds: Downstream projects and container images must then be rebuilt to pull in the fixed versions. This process itself can take hours to months.
• "Why Not Just Latest?": A common question is why users don't simply use the latest version. The reasons are practical:
• API Changes: Newer versions may introduce breaking API changes.
• Dependency Conflicts: Upgrading a single dependency might cause conflicts with others.
• Backports: Users often rely on backports, where a patch from a newer version is applied to an older, supported version (common with Linux distributions). Waiting for backports can delay remediation.

Research Methodology and Data:

The insights are grounded in extensive research:

• 250 container image projects scanned daily for a year.
• Detection of approximately 8,000 unique CVEs.
• Analysis of 40,000 CVE journeys.
• Processing of over 10 million CVE findings.

This scale of data allowed for robust statistical analysis of propagation lag and the efficacy of various remediation strategies.

Demo / Proof of Concept

▶ Watch: Simple solution: Pull the latest tag for 40% CVE reduction (25:35)

While the talk did not feature a live coding demo in the traditional sense, it extensively detailed practical, actionable strategies and specific tools that serve as proof-of-concept for effective CVE remediation. These demonstrations of "how it works" provide concrete methods for addressing the propagation gap.

1. Bloat Reduction through Optimal Container Tags:

• The Problem: The research highlighted that most containers are "bloated" with unnecessary packages (build tools, dev headers, utilities) that contribute significantly to the CVE count. For example, a default Python image can contain hundreds to thousands of CVEs due to its "kitchen sink" approach, suitable for building but not production.
• The Solution: The speaker advocated for prioritizing minimal or slim images in production. These images strip out non-essential components, drastically reducing the attack surface. By comparing a default Python image to its slim counterpart, the talk illustrated how the CVE count could be reduced to just a few dozen.
• Multi-Stage Builds: For scenarios requiring a full build environment, multi-stage builds were presented as the ideal solution. This technique allows developers to compile their application using a comprehensive base image (e.g., with build tools) but then ship the final, compiled artifact in a separate, minimal runtime image. This ensures core functionality is preserved while only necessary components are deployed to production.

1. Automated Dependency Upgrades:

• The Challenge: Managing application dependencies, especially direct ones, for security updates can be a manual and time-consuming process.
• The Solution: Tools like Dependabot and Renovate were showcased as effective, free automation solutions. These tools integrate into a Continuous Integration (CI) pipeline, scan project dependencies, compare them against vulnerability databases and registry latest versions, and automatically open pull requests to upgrade dependencies.
• Trade-offs: While highly beneficial, these tools require human review, especially for major version bumps, and robust test suites to ensure upgrades don't introduce breaking changes. Their primary focus is on direct application dependencies, offering a strong starting point for automation.

1. Direct Patching with Copacetic:

• The Innovation: The traditional method of applying a fix often involves rebuilding the entire container image, which can be time-consuming and resource-intensive.
• The Solution: Copacetic, a CNCF sandbox project, was introduced as an innovative alternative. Based on Buildkit (Docker's default builder), Copacetic allows for patching vulnerable packages directly within an existing container image. It achieves this by adding a lightweight layer containing only the specific patch, eliminating the need to rebuild the entire image from scratch. This significantly reduces the overhead and time required to deploy security fixes, offering a more agile approach to remediation.

These practical examples and tool recommendations demonstrate that effective strategies exist to close the CVE remediation gap, moving beyond just discovery to active, efficient fix deployment.

Defensive Implications

▶ Watch: Q&A: Is AI a force multiplier for fixing vulnerabilities? (36:00)

The research presented by Mor Weinberger offers clear and actionable guidance for defenders seeking to improve their security posture, particularly in containerized environments. The defensive implications span strategic shifts in development practices, operational processes, and engagement with the broader open-source ecosystem.

1. Prioritize Bloat Reduction as a Primary Security Control:

• Highest Leverage Action: The talk unequivocally identifies bloat reduction as the "highest leverage security action" that is often overlooked. Defenders should actively audit and eliminate unnecessary packages, build tools, development headers, and utilities from their production container images.
• Adopt Minimal Images: Make minimal or slim base images the default for production deployments.
• Implement Multi-Stage Builds: Leverage multi-stage builds to separate build-time dependencies from runtime environments, ensuring that only essential components are shipped to production. This significantly reduces the attack surface and the number of CVEs that need to be managed.

1. Embrace Consistent Update Strategies:

• Regularly Pull Latest Tags: Given that 80% of latest tags have fewer CVEs and can reduce overall CVE count by 40%, organizations should establish automated processes to regularly pull and deploy the latest stable tags of their base images and dependencies.
• Automate Dependency Management: Integrate tools like Dependabot and Renovate into CI/CD pipelines to automatically scan for and propose upgrades for application dependencies. While human review and robust testing are still necessary, automation streamlines the process.
• Treat Rebuilds as Security Patches: Shift the perception of container image rebuilds from mere maintenance tasks to critical security patching operations. Schedule regular rebuilds to ensure that all underlying layers and dependencies incorporate the latest security fixes.

1. Understand and Leverage CVE Signaling:

• Importance of CVEs: Recognize that a fix without a corresponding CVE is often "invisible." Defenders need to understand the role of CVE Numbering Authorities (CNAs) in providing the critical signal that turns a quiet code fix into an urgent security patch.
• Monitor CVE Feeds Actively: Ensure security scanning tools are consuming up-to-date CVE databases to accurately identify vulnerabilities and their remediation status.

1. Support and Engage with Open-Source Maintainers:

• Recognize Maintainer Burden: Understand that open-source maintainers are often volunteers facing immense pressure, particularly from the "AI slope" of low-quality reports.
• Practice Responsible Disclosure: If discovering vulnerabilities, engage privately, offer assistance with fixes and verification, and actively work to reduce maintainer fatigue.
• Contribute and Sponsor: Consider contributing to or sponsoring critical open-source projects. Sustaining these projects is vital for the entire ecosystem's security.

1. Explore Advanced Patching Mechanisms:

• Copacetic for Agile Patching: Investigate and potentially adopt technologies like Copacetic that allow for direct patching of vulnerable packages within containers by adding lightweight layers. This can significantly reduce the time and resources required for remediation compared to full image rebuilds, especially in critical situations.

By implementing these defensive strategies, organizations can move beyond simply discovering vulnerabilities to proactively and efficiently remediating them, closing the critical gap between fix availability and actual deployment.

Key Takeaways

• Bloat Reduction is the Highest Leverage Security Action: Most CVEs in containers are a "delete problem," not a "fix problem." By adopting minimal or slim images and implementing multi-stage builds, organizations can drastically reduce their attack surface and the number of vulnerabilities they need to manage.
• Fixes Exist, Propagation Lags: Over 90% of CVEs already have an available fix, but getting these fixes deployed to production is the primary challenge. Regularly pulling the latest tags (which can reduce CVEs by 40%) and automating dependency upgrades are critical steps to close this gap.
• Treat Container Rebuilds as Security Patches: Don't view container rebuilds as mere maintenance. They are a fundamental mechanism for applying security fixes and must be scheduled regularly as part of a proactive security strategy.
• AI is a Force Multiplier, Not a Replacement: While AI excels at vulnerability discovery, its value comes from generating high-quality, actionable reports, assisting with reproduction, and drafting candidate fixes. The focus should be on using AI to help humans and reduce maintainer fatigue, not to generate overwhelming volumes of low-quality "AI slope."
• Respect and Support Open-Source Maintainers: Open-source maintainers are volunteers who sustain a critical part of our digital infrastructure. Engaging respectfully, offering help, contributing, and considering sponsorship are vital to the health and security of the open-source ecosystem.
• CVEs Provide Critical Signal: A fix without a CVE is often invisible to users. Responsible CVE assignment by CVE Numbering Authorities (CNAs) is essential for turning a quiet code fix into an urgent security signal, enabling users to prioritize and apply necessary updates.

About the Speaker(s)

Mor Weinberger is a Software Architect at Aqua, where his work focuses on building vulnerability-free container images and other artifacts. With a decade of experience in vulnerability management and container security, Mor brings deep expertise to the challenges of securing modern software supply chains. His research, co-authored with Leo Kaplan, an open-source maintainer and expert, provides a unique perspective by integrating both the consumer and maintainer viewpoints on CVE remediation.