Security response for open source ecosystems

Security response for open source ecosystems

Speakers: Fraser Tweedale

Conference: CrikeyCon

YouTube: https://www.youtube.com/watch?v=SprxtL-nL7w

Overview

In this essential CrikeyCon talk, Fraser Tweedale, a seasoned security professional from Red Hat, tackles the critical, yet often overlooked, subject of building and sustaining effective security response mechanisms within open-source ecosystems. The presentation, "Security response for open source ecosystems," unpacks the growing imperative for robust security practices in a world increasingly reliant on open-source software, driven by both market realities and emerging regulatory frameworks. Tweedale draws heavily on his practical experience in establishing the Haskell programming language's security response team, offering a candid, step-by-step guide for other projects and foundations.

The talk highlights that open-source software forms the bedrock of the modern software supply chain, making its security a universal concern. Tweedale argues that projects lacking a dedicated security apparatus risk being left behind as new standards and regulations, such as the EU Cyber Resilience Act (CRA) and NIST SSDF, mandate greater accountability for supply chain security. His insights are particularly valuable for large open-source projects, programming language ecosystems, and foundations grappling with how to professionalize their security posture without overwhelming volunteer contributors.

Beyond the technicalities of vulnerability disclosure and tooling, Tweedale delves into the often-challenging topic of sustainability and funding for open-source security work. He presents various models for ensuring that critical security tasks are not left to the goodwill of unpaid volunteers, emphasizing the moral and economic obligations of commercial entities and governments that extensively leverage open-source software. The talk serves as a call to action, urging a shift from passive consumption to active contribution to the security of the shared digital infrastructure.

Background

▶ Watch: Importance of open source security & regulations (0:40)

The pervasive nature of open-source software (OSS) means it underpins virtually every modern software supply chain. As Fraser Tweedale succinctly puts it, "There's very little software out there these days that is not using open-source somewhere in the supply chain." This ubiquity, while fostering innovation, introduces significant security challenges. Organizations are increasingly mandated by external pressures to address these, moving beyond mere awareness to concrete action.

A significant driver for this shift is the evolving regulatory landscape. Tweedale specifically cites ISO 27001, the NIST Secure Software Development Framework (SSDF) in conjunction with Executive Order 14028 (mandating secure development practices for US government procurement), and crucially, the EU Cyber Resilience Act (CRA). The CRA, passed in 2023, is particularly impactful as it extends its reach not only to market participants (vendors and buyers) but also explicitly to "open source software stewards"—foundations and organizations overseeing free and open-source projects. These stewards are now required to "put in place cyber security policies that promote the effective handling of vulnerabilities, foster the voluntary reporting of vulnerabilities and promote sharing of information concerning discovered vulnerabilities." This marks a pivotal moment, shifting the responsibility from an optional best practice to a legal obligation for many.

Against this backdrop, open-source projects and ecosystems that lack effective security response mechanisms face an existential threat; they "will be left behind." This necessity applies broadly: to operating systems, programming languages and their library ecosystems (e.g., Python/PyPI, JavaScript/npm), major frameworks (Spring, Rails), applications (LibreOffice), network servers (Apache, Nginx), and large umbrella projects like Android or Kubernetes.

Tweedale’s personal journey into this space began with the Haskell Foundation. As a Red Hat employee working with diverse languages, he spends his free time with Haskell. When the Haskell Foundation recognized the need for a security apparatus, including an advisory database, and issued a tech proposal, Tweedale—with over a decade of experience in security products at Red Hat—offered extensive feedback. His expertise led to an invitation from then-executive director David Christensen to lead the effort, resulting in the formation of the Haskell security response team. This real-world case study forms the practical backbone of the talk, demonstrating how a project can move from identifying a need to implementing a functional security response.

Key Findings

▶ Watch: Essential skills and diversity for a security team (6:00)

The core "findings" of Tweedale's talk are not research discoveries but rather a comprehensive set of practical guidelines and best practices for establishing and sustaining an open-source security response team. These findings are distilled from his direct experience with the Haskell Foundation and his extensive background in enterprise security.

First, effective security response requires a dedicated, skilled team. Tweedale emphasizes that a single individual is insufficient. A team must possess a diverse set of skills, including deep technical knowledge of the project/language, specialized cybersecurity know-how (application security, incident response, GRC, penetration testing, vulnerability research), software development capabilities for tooling, and strong communication skills. Diversity in team composition (cultural, linguistic, time zones) is crucial for round-the-clock response and resilience against burnout or absences. The Haskell team started with five members and grew to six, deliberately maintaining a reserve pool of qualified volunteers for future expansion or attrition.

Second, defining and prioritizing the scope of work is paramount. Given limited resources, a security team cannot do everything. The most critical activities are incident response (triaging, analyzing, proposing action for security reports) and coordinating with maintainers and stakeholders. Publishing advisories for discovered issues is another high-priority task. Other important, but potentially optional or outsourced, activities include overseeing project infrastructure security, developing developer tooling (though for Haskell, downstream tooling development is out of scope), and community engagement (reports, guides, conferences, office hours). Tweedale stresses that incident response and advisory publication must always take precedence.

Third, standardized processes and clear communication are essential. This includes defining and publicizing clear reporting and disclosure processes for external parties. Internally, teams need private communication channels (chat, mailing lists) and regular meetings, often scheduled to accommodate different time zones. Regular activity reports (e.g., quarterly) keep the community informed. External engagement through conferences and office hours also fosters trust and collaboration.

Finally, Tweedale highlights the absolute necessity of sustainability and adequate funding. He presents the grim reality of "a not good scenario" where critical issues remain unfixed because maintainers are tied up with day jobs. The bottom line is that "Open source ecosystems need personnel with capacity and authority to respond immediately to security incidents." Empty words of support or minimal funding are insufficient. Viable models include company employees working on security full-time/part-time, foundation funding, grants (e.g., German Sovereign Tech Fund), or even paid leads for volunteer teams. He also underscores the significant moral obligation of commercial users and governments, who derive trillions of dollars in value from open source, to contribute financially or in-kind to its security. Without such support, developers' obligations to fix issues are "zero."

Technical Deep Dive

▶ Watch: Advisory formats, CVEs, and numbering authority (14:00)

The technical heart of building an open-source security response apparatus lies in the management and dissemination of vulnerability information. Tweedale elaborates on several key aspects, from advisory content to advanced tooling.

Advisory Structure and Content:

A fundamental decision for any security team is whether to use its own namespace for advisories or rely solely on CVE identifiers. The Haskell project opted for its own namespace, HSC advisories, but also includes corresponding CVEs or GitHub Security Advisories as aliases. The main advantage of becoming a CVE Numbering Authority (CNA) is the ability to suppress bogus CVEs—a significant problem for some projects due to automated or low-quality submissions—and to produce high-quality CVE records directly. However, smaller projects can start by leveraging existing CNAs and apply for CNA status later.

The content of an advisory must be comprehensive. Key elements include:

• A detailed description of the vulnerability.
• Clearly indicated affected packages and their versions.
• CVSS scores for severity assessment.
• Common Weakness Enumerations (CWEs), particularly useful for programming language ecosystems to categorize vulnerability types.
• Aliases for related identifiers across different namespaces.
• Numerous references to reports, proofs of concept (PoCs), commits (introducing/fixing the issue), blog posts, and other relevant documentation.

Tweedale shows that for Haskell, an advisory's source is a simple Markdown file with TOML metadata at the front, demonstrating a lightweight, version-controllable approach. The advisory database itself is a Git repository, allowing for easy tracking of changes and community contributions.

Advisory Consumption and Integration:

To ensure broad reach and utility, advisories need to be consumable by automated tooling and other databases. The Haskell team exports its advisories to OSV (Open Source Vulnerabilities), an advisory information interchange format. They also produce an HTML index and a snapshot format for downstream tooling.

OSV.dev, a Google-run project, is highlighted as a critical aggregator. It ingests and consolidates advisories from numerous databases across various ecosystems (e.g., npm, Maven, Debian). OSV itself uses a JSON format based on a defined schema. Projects wishing to integrate with OSV.dev must publish their OSV export somewhere and then file an issue with OSV.dev, potentially providing assistance with specific versioning schemes or package enumeration. Tweedale demonstrates OSV.dev, showing how users can filter advisories by ecosystem and view individual advisory details, including rendered Markdown descriptions and metadata.

Package Repository Integration and Audit Tooling:

Integrating advisory information directly into package repositories (e.g., PyPI, npm, Hackage for Haskell) is crucial. This allows users to see which package versions are vulnerable directly within the package index. Additionally, package indexes should provide clear information on how and where to report security issues, ideally with an online form or mechanism.

For developers, audit tooling is essential. When a developer has a build plan or a lock file specifying dependencies and their versions, they need tools to analyze these artifacts for vulnerable package versions. This involves consuming advisory information and using it to identify problematic dependencies. Call analysis, while more advanced and not always feasible, can further refine detection. A recommended starting point is OSV-Scanner, a tool that leverages the aggregated OSV database to scan dependency trees.

Suppressing False Positives with VEX:

A common challenge in dependency scanning is the prevalence of false positives. A package might have a known vulnerability, but in the specific context of a program's usage (e.g., a vulnerable function is never called, or a specific configuration mitigates the risk), it may not actually be exploitable. To address this, Tweedale introduces VEX (Vulnerability Exploitability eXchange), a data model for exploitability information. By integrating VEX data into audit tooling and distributing it (e.g., via advisory or package databases), developers and users can avoid being "spammed with a million false positives."

Insights from Aggregate Data:

Beyond individual advisories, the aggregate data from a vulnerability database offers powerful insights. It can reveal the most common issue types within an ecosystem, allowing for the creation of an "OWASP Top 10 for Haskell packages." This information can guide community education, help identify problematic or abandoned packages, and inform proactive security efforts. Tweedale also stresses the importance of including historical advisories in the database to build a comprehensive risk profile.

Embargoed Advisories and Cross-Ecosystem Collaboration:

Security teams must manage embargoed advisories for critical, undisclosed vulnerabilities. This requires setting a clear impact threshold for what qualifies for an embargo and maintaining a private list of downstream distributors and stakeholders to inform. For vulnerabilities spanning multiple ecosystems, collaboration is vital. Tweedale highlights VINT (Vulnerability Information and Notification Toolkit), an initiative by CERT.org, as a powerful system for cross-vendor collaboration on embargoed security issues. Joining VINT is strongly recommended for any established security response team.

Demo / Proof of Concept

▶ Watch: Addressing sustainability and funding for open source security (24:00)

Fraser Tweedale demonstrated the practical application of several concepts discussed, specifically focusing on the OSV.dev platform and the structure of a Haskell security advisory.

He presented the OSV.dev website, illustrating its function as a central aggregator for open-source vulnerability advisories. He showed how the platform allows users to filter advisories by various ecosystems (e.g., npm, Maven, Debian), highlighting the breadth of its coverage. By filtering for "package," the Haskell package database, he demonstrated how specific advisories for the Haskell ecosystem are presented. The demo showcased an individual advisory page, rendering the Markdown description along with its associated TOML metadata, making it clear how the structured advisory data is consumed and displayed in a user-friendly format. This visual tour reinforced the importance of standard formats like OSV for interoperability and the value of aggregated vulnerability information for both developers and security professionals.

Defensive Implications

▶ Watch: Moral obligations and economic value of open source (28:00)

The talk offers a wealth of actionable intelligence for defenders, encompassing both strategic and tactical recommendations for securing open-source software supply chains.

For Open-Source Projects and Ecosystems:

1. Establish a Dedicated Security Response Team: This is paramount. Projects must move beyond ad-hoc responses. The team needs diverse skills (technical, cybersecurity, development, communication) and sufficient capacity. Start small, define a clear scope focused on incident response and advisory publication, and build outwards.
2. Define and Publicize Processes: Clear vulnerability reporting and disclosure processes are non-negotiable. This facilitates responsible disclosure and reduces friction for external researchers. Internal communication and regular meetings are also vital for team cohesion and effective incident management.
3. Implement Robust Advisory Systems:

• Choose an advisory namespace (e.g., HSC advisories) and consider becoming a CVE Numbering Authority (CNA) if the project's scale warrants it, primarily to control the quality of CVEs and suppress bogus reports.
• Structure advisories comprehensively, including descriptions, affected versions, CVSS scores, CWEs, aliases, and extensive references.
• Export advisories to standard formats like OSV (Open Source Vulnerabilities) and integrate with aggregators like OSV.dev to maximize reach and utility for downstream tooling.

1. Integrate Security into Package Repositories: Package indexes should display vulnerability information alongside package versions and provide clear, accessible mechanisms for reporting security issues.
2. Leverage Audit Tooling: Encourage and support the use of tools like OSV-Scanner to analyze dependency trees for vulnerable components.
3. Adopt VEX for Exploitability Information: Implement Vulnerability Exploitability eXchange (VEX) data to reduce false positives in vulnerability scans, thereby improving developer signal-to-noise ratio and focusing efforts on truly exploitable issues.
4. Engage in Cross-Ecosystem Collaboration: For embargoed issues, join initiatives like VINT (Vulnerability Information and Notification Toolkit) to coordinate disclosures securely across multiple vendors and ecosystems.

For Commercial Organizations and Governments:

1. Recognize and Act on Moral Obligations: Organizations deriving significant value (estimated at $8.8 trillion annually in demand-side value, per Harvard Business School research) from open-source software have a direct responsibility to contribute to its security. This contribution should be direct or indirect through foundations.
2. Fund Open-Source Security Work:

• Direct Funding: Employ developers full-time or part-time to work on the security of critical open-source dependencies.
• Foundation Support: Contribute to open-source foundations that, in turn, fund security teams or specific security projects (e.g., audits, tooling).
• Grants & Bounties: Support grant programs (like the German Sovereign Tech Fund) or establish bug bounty programs.
• Subscription Models: Explore direct funding models where commercial users subscribe to support security teams, potentially receiving early heads-up on issues or consulting access.

1. Prioritize Capacity and Authority: Ensure that funded personnel have both the capacity (time, resources) and the authority to drop other work and respond immediately to security incidents when they arise.
2. Influence Policy: Governments should actively fund FOSS security work, establish grants programs, and use their procurement power (as per EO 14028) to incentivize secure development practices within their supply chains.

The overarching defensive implication is a call for a paradigm shift: from passive consumption and expectation of free security labor to active, financially supported contribution to the shared open-source security commons. Failure to do so will inevitably lead to an unsustainable and insecure software ecosystem, with severe consequences for all dependent parties.

Key Takeaways

• Open-source security is supply chain security: With open source forming the foundation of nearly all modern software, its security is a non-negotiable requirement driven by both market reliance and increasing regulatory mandates like the EU Cyber Resilience Act.
• Dedicated security teams are essential: Open-source projects need skilled, diverse teams with clear scopes focused on incident response and advisory publication, rather than relying on ad-hoc volunteer efforts.
• Standardized advisories and tooling are critical: Leveraging formats like OSV and platforms like OSV.dev, along with audit tooling and VEX data, is vital for effective vulnerability communication, detection, and false positive reduction.
• Sustainability requires funding: Critical open-source security work cannot rely solely on volunteers. Commercial entities and governments that depend on open source have a moral and economic obligation to contribute financially to its security.
• Proactive engagement is key: Projects should define clear reporting processes, engage with their communities, and collaborate across ecosystems (e.g., via VINT) to enhance collective security posture.
• Capacity and authority are paramount: Funded security personnel must have the time and the organizational backing to prioritize and immediately address security incidents, preventing long-term exposure to critical vulnerabilities.

About the Speaker(s)

Fraser Tweedale is a security professional with over a decade of experience at Red Hat, where he works on security-related products. His day-to-day work involves a mix of programming languages including C, Python, Golang, and TypeScript. In his personal time, he has a passion for functional programming, particularly with the Haskell language. This unique blend of enterprise security expertise and deep engagement with a specific open-source ecosystem positioned him to lead the establishment of the Haskell Foundation's security response team, drawing on his practical understanding of how open-source security response works in practice. He has spoken at CrikeyCon multiple times, demonstrating a long-standing commitment to the security community.