Airflow Beach Cleaning - Supply Chain Security with Community in Mind

Airflow Beach Cleaning - Supply Chain Security with Community in Mind

Speakers: Michael Windsor, Co-founder, Alpha Omega; Jarrick, Apache Airflow Maintainer & PMC Member

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=BXkrkneY2-g

Overview

In this compelling talk from VulnCon, Michael Windsor, co-founder of Alpha Omega, and Jarrick, an Apache Airflow maintainer, shed light on a pragmatic and community-focused approach to securing the open-source supply chain. Moving beyond the daunting "Pacific garbage patch" of vulnerabilities, they advocate for "beach cleaning"—a targeted strategy to improve security for specific, high-impact open-source projects and their upstream dependencies. The core of their methodology centers on human connection, direct engagement, and a clear understanding of economic incentives.

The presentation delves into the critical lessons learned from incidents like XZ Utils, highlighting the pervasive issue of open-source projects treating their dependencies as "code from the sky," devoid of responsibility. Alpha Omega, a fund dedicated to enhancing open-source security with significant backing from industry giants like Microsoft, Amazon, and Google, partnered with the Apache Airflow project to pilot this new strategy. Their collaboration demonstrates how focused effort on a project with strong community engagement and clear business value can create a cascading positive effect throughout the dependency graph.

This article will explore how Airflow, a massive and widely used orchestration engine, proactively audited its 700+ dependencies not just for technical flaws, but for their overall risk posture and maintainer engagement. The speakers detail the "Fix, Fork, or Forget" framework for managing dependencies and present real-world case studies that underscore the power of maintainer-to-maintainer communication in driving tangible security improvements upstream. The insights shared offer a vital roadmap for any organization or project grappling with the complexities of open-source supply chain security.

Background

▶ Watch: Alpha Omega's mission: turning money into open source security (0:00)

Alpha Omega was founded in 2021-2022 by Michael Windsor and Michael Skaveta with a mission to funnel resources into open-source security, ultimately raising approximately $23 million, largely from Microsoft, Amazon, and Google. Initially, the fund aimed to address the "alpha" (top 100 critical projects) and "omega" (the long tail of hundreds of thousands of smaller projects). However, experience revealed that "alpha" projects often had sufficient attention, leading Alpha Omega to pivot towards highly leveraged activities that could impact the broader "omega" scale. This meant working with foundations and package ecosystems rather than individual projects.

The XZ Utils incident served as a stark wake-up call, exposing the fragility of small, seemingly innocuous projects from both a social and ecosystem perspective. It highlighted a critical flaw in how open-source projects manage their dependencies: they often treat upstream components with the same disinterest they complain about from corporations using their own code—as if the code simply "comes down from the sky." This led to the fundamental realization: "Your software, right, is that shiny sports car on top. And that's how much we're treating the stuff underneath it... Every dependency you bring in, it's not their code. It's not someone else's problem. It's your problem." From a compliance, security, and business perspective, the entire dependency tree becomes the responsibility of the consuming project.

Early attempts by Alpha Omega to automate vulnerability discovery and fix propagation at scale proved frustrating. While some vulnerabilities were found and patched, the impact felt like "going out to the Pacific garbage patch with a rowboat and coming back with some plastic"—insignificant in the grand scheme. This led to a strategic shift: instead of trying to clean the entire ocean, they decided to focus on cleaning one beach at a time. The ideal "beach" for this approach possesses a community that cares about security, is aware of existing issues, and is open to collaboration. Apache Airflow perfectly fit this description, boasting a large, active community and strong stakeholder interest in improving its security posture.

Key Findings

▶ Watch: New strategy: clean one open-source 'beach' at a time (4:10)

The collaboration between Alpha Omega and Apache Airflow yielded several profound insights into effective open-source supply chain security. Firstly, the "beach cleaning" methodology, which prioritizes a targeted, human-centric approach, proved significantly more effective than broad, automated vulnerability scanning. Instead of a full-scale code audit of Airflow itself, Alpha Omega funded an audit of Airflow's 719 dependencies, specifically assessing their risk from a supply chain perspective. This shift in focus from the consuming project to its upstream components was a critical strategic decision.

A second key finding was the paramount importance of human connection and direct communication. While automated security reports often go unheeded, a direct outreach from a maintainer of a dependent project (like Airflow) to an upstream maintainer dramatically increases the likelihood of a rapid and effective fix. This maintainer-to-maintainer dialogue fosters a sense of shared responsibility and mutual aid, transforming what could be an adversarial interaction into a collaborative one. This highlights that supply chain security is fundamentally a human problem first, and technical second.

Finally, the talk underscored the economic realities of open-source security. Projects like Airflow, which are "close to revenue" for major businesses (e.g., Amazon, Microsoft, Google, Astronomer offering Airflow as a service), can more easily justify allocating engineering resources to security improvements, including those upstream. This clear Return on Investment (ROI) makes it easier to fund and sustain security initiatives. Conversely, smaller projects, often maintained by volunteers, may lack the resources or organizational structure to accept direct funding, necessitating alternative approaches like direct maintainer engagement or community-driven forks. This understanding shapes how organizations like Alpha Omega prioritize their funding and engagement strategies.

Technical Deep Dive

▶ Watch: Airflow partnership: auditing 719 dependencies for risk (5:00)

Apache Airflow stands as a testament to the scale and complexity of modern open-source projects. As an orchestration engine with extensive integrations, its dependency tree is vast. Jarrick detailed Airflow's impressive statistics: over 256 active pull requests in a single week, 62 unique authors, 129 active issues, more than 16,000 GitHub projects starring or forking it, and over 3,200 contributors. This level of activity underscores both its importance and the challenge of securing its extensive ecosystem.

Airflow had already made significant strides in its internal security posture before partnering with Alpha Omega. These improvements included establishing a dedicated security team (comprising 15 people, with five actively engaged), creating detailed security processes, implementing a security model, enhancing communication with security reporters, disabling inherently insecure features, hardening CI workflows, and introducing reproducible builds. These efforts significantly reduced the number of reported vulnerabilities by 2023, demonstrating a strong commitment to security.

The core of the technical deep dive revolved around Airflow's approach to its dependencies. With 579 basic dependencies and over 700 total when optional dependencies are included, the task was immense. The process began with generating Software Bill of Materials (SBOMs) using CycloneDX of Airflow's releases. These SBOMs provided the foundational data for identifying direct and transitive dependencies. From this comprehensive list, an initial set of 16 "prime candidates" was selected for deeper investigation.

Automated assessment tools played a role, with OpenSSF Scorecards being used to provide an initial security posture rating for these dependencies. Jarrick supplemented this with additional analysis to determine which dependencies were most critical to Airflow's functionality and the broader industry. This data-driven approach generated specific, actionable recommendations for each dependency, such as proposing the addition of security policies, following up on insecure workflows, suggesting the implementation of trusted publishing, addressing unpatched vulnerabilities, and proposing mandatory code reviews. This blend of automated insights and human-driven action formed the backbone of their dependency management strategy.

The aim was not merely to identify problems but to actively engage and assist. Airflow maintainers reached out to their upstream counterparts, offering to submit pull requests and provide direct support to improve security. This proactive, collaborative stance, rooted in the understanding that "we are only as secure as our dependencies," proved far more effective than simply reporting issues.

Demo / Proof of Concept

▶ Watch: Introducing Apache Airflow: a huge, active project (6:05)

While a traditional live coding demo was not part of the talk, the speakers presented a powerful "experiment in progress" that served as a proof of concept for their human-centric, "beach cleaning" methodology. This experiment involved engaging with the initial 16 identified high-priority dependencies, proposing specific security improvements, and tracking responses. The key finding was the dramatic difference in reaction when a report came from a fellow maintainer (Jarrick) versus an anonymous or automated security report. Upstream projects that had previously ignored automated vulnerability reports responded within days or even hours to Jarrick's direct, personal outreach.

The talk highlighted three specific case studies that exemplify the "Fix, Fork, or Forget" framework in action:

1. Chroner Project (Fork/Fix): Chroner was a small but critical dependency for many Airflow versions. Its maintainer threatened to remove the project due to concerns about the Cyber Resilience Act (CRA) and lack of time. Airflow initiated public discussions on a GitHub issue, and several community members volunteered to fork the project. Ultimately, a more sustainable solution emerged: the entire Chroner project moved to Pallet Echo, an open-source steward, with Jarrick and another Airflow maintainer becoming co-maintainers. This effectively "saved" the project through a collaborative fork and subsequent fix by a new, dedicated team.

1. Vortake (Transitive Dependency - Fix): Vortake represented a challenging transitive dependency for Airflow, making direct intervention difficult. Through discussions with stakeholders, Airflow identified this as a critical issue. As a direct result of these discussions, the Google security team (one of Airflow's major stakeholders) stepped in and began actively working on patching older versions of Vortake to address security concerns for Airflow. This demonstrated the power of stakeholder involvement in driving fixes for complex transitive issues.

1. Flask (Dependency - Forget): Another dependency, Flask (connected to Vortake), proved extremely difficult to upgrade, with previous attempts failing. Despite efforts from the Google security team to patch it, a sustainable solution wasn't found. After engaging with the maintainers and assessing the situation, Airflow made the strategic decision to forget this dependency. They switched to FastAPI, effectively removing the problematic component from their dependency tree. This illustrates that sometimes, the most secure option is to deprecate and replace a problematic dependency.

These case studies collectively demonstrate the practical application of the "Fix, Fork, or Forget" framework, showcasing how direct engagement, community support, and strategic decision-making can lead to tangible improvements in open-source supply chain security.

Defensive Implications

▶ Watch: Visualizing Apache Airflow's vulnerability trends over time (7:15)

The insights from the "Airflow Beach Cleaning" talk offer crucial defensive implications for any organization or project relying on open-source software. The foundational principle is that every dependency is "your code, your problem." This means organizations must extend their security and compliance responsibilities to their entire dependency graph, rather than treating upstream components as external entities.

Defenders should proactively establish vendor relationship management practices for their open-source dependencies, mirroring how they manage commercial software vendors. This involves knowing who the upstream maintainers are, establishing channels of communication, and understanding their security posture. The talk explicitly questions, "How many of you are in contact with any of your upstream suppliers?" highlighting a pervasive gap that needs urgent addressing. If risk management people would deem it insane to have no contact with a commercial vendor, the same logic must apply to open-source suppliers.

Implementing the "Fix, Fork, or Forget" framework provides a clear strategic approach to managing problematic dependencies:

• Fix: Actively contribute patches, security policies, or process improvements back to the upstream project. This is the ideal scenario, fostering collaboration.
• Fork: If upstream engagement fails or is insufficient, be prepared to fork the project and maintain security patches internally. Critically, this should be a deliberate, sanctioned fork, not an unmanaged, unmaintained copy.
• Forget: If a dependency poses intractable security risks or maintenance burdens, consider removing it entirely by switching to an alternative or refactoring. This is a powerful, albeit sometimes costly, way to eliminate risk.

Furthermore, defenders must recognize that "there are no small dependencies." As Michael Windsor emphasized, "Every dependency in your tree has total access to your build environment... and total access to your production environment." Regardless of its perceived size or criticality, any dependency can become a vector for exfiltrating secrets or introducing malicious code. Relying solely on sandboxing or call graphs is insufficient; a proactive, human-centric approach to understanding and securing these components is paramount.

Finally, organizations should leverage tools like CycloneDX for SBOM generation and OpenSSF Scorecards for initial dependency assessment. However, these tools should serve as a starting point, guiding human-led investigations and fostering direct engagement with maintainers. Prioritizing security investments in open-source projects that are "close to revenue" for the business can yield significant ROI, as the justification for allocating engineering resources is clearer.

Key Takeaways

• Supply chain security is a human problem first, technical second. Direct, maintainer-to-maintainer communication is far more effective than automated reports in addressing upstream vulnerabilities.
• Every dependency is "your code, your problem." Organizations must take full responsibility for the security, compliance, and correctness of their entire open-source dependency graph.
• The "Fix, Fork, or Forget" framework provides a practical, actionable strategy for managing and mitigating risks associated with upstream dependencies.
• Prioritize human connections with upstream open-source maintainers, treating them like critical vendors in your supply chain to build trust and facilitate collaboration.
• No dependency is "small." Every component, regardless of size, has potential access to build and production environments, necessitating a thorough security posture.
• Invest strategically in projects close to revenue. For projects that directly enable business monetization, the ROI for security improvements is clearer, making it easier to justify and fund upstream security efforts.

About the Speaker(s)

Michael Windsor is the co-founder of Alpha Omega, a fund established in 2021-2022 alongside Michael Skaveta. Alpha Omega's mission is to invest money into improving the security of open-source software, having raised approximately $23 million with significant contributions from Microsoft, Amazon, and Google. Michael's work at Alpha Omega focuses on highly leveraged activities to secure the long tail of open-source projects, driven by lessons learned from incidents like XZ Utils and a commitment to fostering collaborative security initiatives.

Jarrick is an Apache Airflow maintainer, committer, and PMC (Project Management Committee) member. He plays a crucial role in the development and security of Apache Airflow, a widely used open-source orchestration engine. Jarrick has been instrumental in leading Airflow's internal security improvements, including establishing a dedicated security team and processes. His practical experience as a maintainer and his direct engagement with upstream dependencies were central to the "beach cleaning" initiative, highlighting the effectiveness of maintainer-to-maintainer collaboration in enhancing open-source supply chain security.