Alpha-Omega: What We've Learned From Funding Open Source Security Over the Past 3Years, What's Ahead

Alpha-Omega: What We've Learned From Funding Open Source Security Over the Past 3 Years, What's Ahead

Speakers: Michael Windsor, Co-founder Alpha-Omega, Linux Foundation

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=8gZPwB3JKAU

Overview

In this VulnCon talk, Michael Windsor, co-founder of Alpha-Omega, a project under the Linux Foundation, shared profound insights from three years of dedicated efforts to bolster open-source software security. Co-founded by Google and Microsoft, and later joined by Amazon and City, Alpha-Omega's mission extends beyond mere funding; it aims to catalyze sustainable change within open-source ecosystems. Windsor's presentation delves into the triumphs, challenges, and evolving strategies employed by Alpha-Omega to address the pervasive security vulnerabilities in the digital supply chain.

The talk highlights a critical shift in approach: from attempting to secure the "top 100" most critical projects or broadly applying static analysis across thousands, to a more targeted, community-centric strategy dubbed "cleaning the beach." This evolution acknowledges that simply injecting resources into projects is often insufficient; true security improvements stem from fostering a culture of ownership, establishing dedicated security roles, and building robust processes. Windsor underscores the necessity of moving away from a "free as in beer" mentality towards open source, urging consumers and maintainers alike to recognize the inherent risks and responsibilities.

Ultimately, Windsor’s discussion is a call to action for the entire industry. It emphasizes that open-source security is a shared problem requiring collective investment, strategic engagement, and a fundamental change in how dependencies are perceived and managed. Alpha-Omega's journey offers a candid look at the complex economics of open-source maintenance and provides actionable strategies for both upstream projects and downstream consumers to build a more resilient software ecosystem, with a keen eye on future challenges like AI security and incident response.

Background

▶ Watch: Alpha Omega's mission: causing change in open source security (0:50)

The genesis of Alpha-Omega stemmed from a recognition of the vast and intricate security challenges within the open-source landscape. Initially, the project conceptualized its efforts into two streams: "Alpha" focused on the supposed "top 100 most critical" open-source projects, while "Omega" aimed to tackle the hundreds of thousands of smaller, less-resourced projects. However, early experiences quickly revealed the limitations of these initial assumptions. Windsor explains that identifying the "top 100" was contentious, and more importantly, many obvious critical projects already received significant attention. Crucially, simply providing money to these projects did not inherently make them more secure because their problems were not solely about resources but rather about underlying processes and culture.

This led to a significant evolution in Alpha-Omega's strategy for "Alpha" projects. Instead of broad funding, the focus shifted to identifying "points of leverage" where judicious spending could catalyze change, foster leadership, and create lasting impact beyond Alpha-Omega's direct involvement. For the "Omega" problem—the immense scale of unmaintained or under-secured smaller projects—initial attempts involved scaled static analysis, finding vulnerabilities, and pushing pull requests across thousands of projects (e.g., the top 2,000 projects in PyPI). While technically feasible, this approach proved largely "not actionable." The cost per project was reasonable, but the fixes often weren't integrated, and downstream consumers couldn't effectively de-risk or respond. This "Pacific garbage patch" analogy highlighted the futility of trying to clean the entire ocean at once.

The breakthrough for the Omega problem came with the "cleaning the beach" approach. This strategy involves focusing on specific communities or ecosystems, helping them identify their supply chains, establish methodologies, and pay attention to upstream dependencies. The recent XZ situation, where a malicious backdoor was discovered in a widely used compression utility, served as a stark reminder of the systemic issue: many open-source projects, much like corporations, often consume upstream code with an implicit, often unfounded, trust, failing to scrutinize its security posture. This upstream neglect creates a fertile ground for supply chain attacks, underscoring the urgent need for a more proactive and integrated security approach across the open-source ecosystem.

Key Findings

▶ Watch: Challenges of scaled open source security: Pacific Garbage Patch (2:50)

Alpha-Omega's three years of funding open source security have yielded several critical findings, leading to a refined approach centered on catalytic change rather than mere resource injection.

Firstly, the most impactful strategy has been the creation of dedicated security staffing roles. Windsor highlighted the success of funding a Security Engineer in Residence at the Python Software Foundation, a role filled by Seth Larson. Before this, security was "everyone's job, which means it's no one's job." Seth's presence transformed a backlog of issues and committee debates into actionable initiatives, design documents, and programs, normalizing security within the Python ecosystem. This pattern has been successfully replicated across other major ecosystems, including the Eclipse Foundation, Apache, Ruby, Rust, Node.js, and OpenJS, all of which now have dedicated security staff driving real change. Alpha-Omega has allocated approximately 60% of its spending budget to these staffing engagements, underscoring their perceived effectiveness.

Secondly, audits serve as a crucial starting point, not just for identifying vulnerabilities but as a strong signal of a project's security posture and willingness to engage. The mere interest in undergoing an audit reveals much about a project's security culture. Alpha-Omega is increasingly exploring ways to reduce the cost of these audits, potentially through "rapid security reviews" (e.g., a four-hour review spread over weeks with a six-month check-in), to get an early signal and test a project's commitment to remediation. The Eclipse Foundation has been a leader in this area.

Thirdly, Alpha-Omega prioritizes leverage and public good over short-term fixes or commercial interests. The goal is to fund initiatives that cause change beyond a single project or individual, creating lasting impact. A prime example is the funding for Homebrew to implement Sigstore for end-to-end verification. This integration benefits the entire Homebrew ecosystem and bootstraps Sigstore's viability, demonstrating how a targeted investment can ripple through a wide user base. Similarly, the project is biased towards "shovel-ready work" with short-term, measurable impact, preferring clear plans and dedicated personnel over speculative research. Proposals from commercial entities with limited community contribution, where the benefit is primarily for a single company, are generally declined, as seen in the example of a company seeking funding for its open-sourced product with only internal contributors.

Finally, Alpha-Omega has discovered the immense value of inter-ecosystem collaboration. By funding across various communities, they've been able to connect security leads from Python, PyPI, Ruby, RubyGems, Rust, and Crates.io, fostering shared learning and the adoption of best practices. The trusted publishing specification that became critical for PyPI's security is now being adopted by Rust and RubyGems, a direct result of these facilitated connections through quarterly roundtables. This community-driven trust and knowledge sharing amplify the impact of Alpha-Omega's investments far beyond direct financial contributions.

Technical Deep Dive

▶ Watch: Effective scaled security: the 'cleaning the beach' approach (4:00)

Alpha-Omega's strategic funding has supported the implementation and incubation of several foundational security technologies and methodologies designed to enhance the integrity and trustworthiness of the open-source supply chain.

One significant technical contribution is the integration of Sigstore within Homebrew. Sigstore is an open-source standard for signing, verifying, and protecting software supply chains. In the context of Homebrew, this means implementing end-to-end verification from the build side down to the brew client. This process allows for the verification of attestations—cryptographically verifiable proofs about the software's origin and characteristics—across the entire build process. This mechanism helps predict and prevent tampering, ensuring the authenticity of packages installed via Homebrew. While still in beta due to some limitations with GitHub's OIDC (OpenID Connect) and service accounts, this initiative is a powerful example of applying cutting-edge supply chain security practices to a widely used package manager, benefiting millions of users.

Another critical area of focus is the development and adoption of trusted publishing specifications for package managers. Originating within PyPI's efforts to enhance publishing security, this specification provides a framework for secure package uploads and distribution. By fostering cross-ecosystem collaboration, Alpha-Omega has facilitated its adoption by other major package managers like Rust (Crates.io) and RubyGems. This standardization helps to mitigate risks associated with compromised maintainer accounts or malicious package injections, establishing a more reliable chain of custody for published software.

The talk also underscores the importance of foundational technologies like SLSA (Supply-chain Levels for Software Artifacts) and reproducible builds. SLSA is a framework that provides increasing levels of assurance about the integrity of a software artifact and its supply chain, particularly focusing on preventing tampering and improving build integrity. Reproducible builds ensure that given the same source code, build environment, and build instructions, anyone can produce bit-for-bit identical binaries. This eliminates a significant attack surface by making it impossible for malicious actors to inject code during the build process without detection. Windsor highlights Go's inherent capability to build everything from source, referencing GitHub repos or URLs with a non-repudiation index transparency log, effectively neutralizing many SLSA build system attacks. Similarly, the Eclipse Foundation's Adoptium project for Java runtimes and Debian's strides towards reproducible package builds are cited as leading examples, with Red Hat verifying Adoptium's reproducibility. These efforts aim to normalize practices that minimize trust decisions about opaque build systems.

Looking ahead, Alpha-Omega is actively exploring the use of AI for scaled security analysis. Initial experiments have shown remarkable potential in using AI to find vulnerabilities in code. The vision includes leveraging AI for "depth-first supply chain audits"—performing mini-audits to understand the governance and security posture of dependencies through a recursive traversal. While remediation remains a challenge, AI's ability to identify issues at scale could be transformative for tackling the "Omega" problem.

Finally, a significant technical hurdle identified is the lack of common open source metadata. Windsor laments the current state where every organization and researcher rebuilds basic data pipelines—crawling NVD, OSV, or GitHub Archive to gather public security data and perform dependency resolution. This redundant effort consumes vast resources before any meaningful analysis can begin. Alpha-Omega aims to create a "trustworthy public good" for this metadata, designed for extensibility and integration with private data. The goal is to make it as trivial as running a SQL query to perform security analysis at scale, fostering innovation for both commercial and open-source security solutions.

Demo / Proof of Concept

▶ Watch: Funding security engineers at Python and Eclipse Foundations (5:40)

The talk did not feature a live demonstration or proof-of-concept specific to a vulnerability exploit or tool. Instead, Michael Windsor's presentation focused on the strategic and operational initiatives undertaken by Alpha-Omega to improve open-source security at scale, sharing lessons learned and future directions. The content was primarily conceptual and analytical, illustrating the impact of funding and strategic interventions through real-world examples of project improvements and ecosystem-wide shifts, rather than through a technical showcase of a specific exploit or defensive mechanism.

Defensive Implications

▶ Watch: Audits reveal project security posture and culture (6:30)

The insights shared by Michael Windsor carry profound defensive implications for any organization consuming open-source software, which, in today's landscape, is virtually every enterprise.

Firstly, the most critical takeaway for defenders is to internalize the mantra: "It's your code. It's your problem." Organizations must abandon the fallacy that open-source dependencies are external entities for which they bear no responsibility. As Windsor starkly puts it, "You have no vendor relationship. You have no contract. You have no nothing." Every line of code ingested, regardless of its origin, becomes part of the organization's attack surface with total access to build and production environments. This demands a shift from treating open source as "free as in beer" to understanding it as "free as in puppies"—initially appealing but requiring significant, ongoing care and attention to prevent costly damage.

Defenders should move beyond a reactive "fix-the-vulnerability" mindset to focus on establishing robust security processes and fostering a security-aware culture. The talk emphasizes that the process and culture of an organization in fixing vulnerabilities are more important than the current number of known vulnerabilities. If a project or organization lacks the culture and people to prioritize and implement security, even deep audits or mandated fixes will be ineffective.

Leveraging compliance is presented as a pragmatic strategy for driving internal security prioritization. Windsor notes that "you can't sell security, but compliance wins every day of the week." With impending regulations like CRA (Cyber Resilience Act) and 14028 (likely referencing NIST SP 800-140B or similar standards related to secure software development), governments are increasingly forcing industry to raise its security posture. Organizations can use these mandates to secure budget and resources for open-source security initiatives that might otherwise be deprioritized in favor of growth-oriented features.

A re-evaluation of dependency criticality is also crucial. Defenders should not rely on superficial metrics like GitHub stars or number of dependents. As Windsor illustrates, even the smallest, seemingly insignificant dependency (like a "left-pad-like text string formatter") can have total access to an environment. The risk should be assessed based on the potential impact if that dependency is compromised, assuming it has full access. This necessitates prioritizing scrutiny and controls around all ingested code.

Organizations must implement a strategic "fix, fork, or forgo" approach to managing dependencies.

• Fixing involves active engagement with upstream maintainers, contributing PRs, or even funding vendors to address issues directly in the upstream project. This proactive stance builds relationships and ensures fixes percolate through the supply chain.
• Forking entails taking ownership of a dependency, especially if it's critical but unmaintained or slow to patch. While resource-intensive, it provides direct control over security. Windsor cautions that unmaintained, outdated dependencies are, in effect, "hard forks" that are unsanctioned and unknown.
• Forgoing means eliminating unnecessary dependencies. This is often the most effective defense, reducing the attack surface by removing code that isn't strictly essential. A hybrid approach involves hard-forking and maintaining only the minimal necessary components of a larger library. Defenders need to establish stringent ingestion policies to prevent engineers from easily adding insecure or unvetted packages (e.g., "npm install FU").

Finally, and perhaps most importantly, defenders must build relationships with upstream maintainers. Windsor likens this to an "adopt a highway mindset," where company engineers become points of contact for specific open-source projects. Proactive engagement—reaching out before a crisis, offering help, and expressing appreciation—can significantly improve response times and collaboration during a security incident. The Log4Shell crisis highlighted the chaos when thousands of companies tried to contact overwhelmed maintainers with no prior relationship. Establishing these connections when there isn't a fire is paramount for effective incident response and long-term security.

Key Takeaways

• Catalytic Funding, Not Just Direct Funding: Alpha-Omega's experience shows that simply providing money to open-source projects is often insufficient; the most impactful approach is to catalyze sustainable change by fostering culture, processes, and leadership within ecosystems.
• Dedicated Security Roles are Transformative: Appointing Security Engineers in Residence (e.g., Seth Larson for Python) effectively shifts security from being "everyone's job" (and thus no one's) to a prioritized, actionable function, driving significant improvements across major open-source projects.
• Organizations Must Own Their Open Source Security: Downstream consumers must internalize that every open-source dependency is "their code and their problem," requiring proactive engagement, robust processes, and a shift from a "free as in beer" mentality to one acknowledging inherent risks and responsibilities.
• Unsustainable Economics Threaten Core Infrastructure: The current funding models for critical open-source infrastructure like package managers are often fragile, relying on flat donations and cloud credits. These "charities" face escalating operational and security costs, creating a significant future risk for the entire software ecosystem.
• Relationship Building is Paramount for Defense: Proactive engagement and building trust with upstream open-source maintainers are critical defensive strategies, enabling better incident response and collaborative security improvements before a crisis hits.
• Future Focus on Scaled Solutions: Alpha-Omega's future efforts will concentrate on normalizing co-funding models, leveraging AI for scaled security analysis, incubating foundational technologies like Sigstore and SLSA, promoting reproducible builds, and developing a public good for common open-source metadata to facilitate widespread security analysis.

About the Speaker(s)

Michael Windsor is a co-founder of Alpha-Omega, a pivotal project within the Linux Foundation dedicated to enhancing the security of open-source software. Through Alpha-Omega, he works to identify and fund initiatives that catalyze sustainable security improvements across various open-source ecosystems. His work involves collaborating with major technology companies like Google, Microsoft, Amazon, and City, as well as numerous open-source foundations and communities. Windsor's insights are deeply rooted in practical experience, having navigated the complex challenges of funding, governance, and community engagement in the pursuit of a more secure software supply chain. He frequently speaks on the economics of open-source security and the strategic approaches required to address systemic vulnerabilities. Michael Scaveta is also mentioned as a co-founder of Alpha-Omega.