How to Hack and Defend (Your) Open Source?

How to Hack and Defend (Your) Open Source?

Speakers: Roman Zhukov, Principal Engineer, Intel

Conference: Hacktivity

YouTube: https://www.youtube.com/watch?v=DQDAuNG0jG0

Overview

In his Hacktivity 2023 talk, "How to Hack and Defend (Your) Open Source?", Roman Zhukov, a Principal Engineer at Intel, delivered a critical examination of the escalating security challenges within the open source software supply chain. Zhukov underscored the ubiquitous, yet often unacknowledged, reliance on open source components across all industries, from small startups to multinational corporations. He highlighted that the security of this foundational software is not an external problem but a direct responsibility for every organization and developer, emphasizing the phrase "this is your open source, not someone else's."

The presentation meticulously explored both the offensive tactics employed by malicious actors and the defensive strategies necessary to safeguard open source ecosystems. Zhukov provided a stark overview of recent, high-profile attacks, including dependency confusion, typosquatting, trojan injection, and repository hijacking, demonstrating how easily widely used tools and packages can be compromised. Beyond mere vulnerability patching, he advocated for a holistic approach to open source security, integrating robust practices across secure sourcing, development, build, and release phases, with a particular focus on Software Bill of Materials (SBOMs) and the Supply-chain Levels for Software Artifacts (SLSA) framework.

Zhukov's talk served as a wake-up call, urging attendees to move beyond passive consumption of open source to proactive security stewardship. He stressed that while the sheer volume of open source components makes comprehensive oversight daunting, strategic application of industry best practices, automation, and a "trust but verify" mindset can significantly mitigate risks. The session was packed with actionable advice, recommended tools, and frameworks, making it an invaluable guide for anyone involved in developing, deploying, or managing software that relies on open source.

Background

▶ Watch: Recent real-world open source supply chain attacks (0:00)

The modern software landscape is overwhelmingly built upon open source components. Roman Zhukov highlighted staggering statistics that underscore this reliance: over one billion vulnerable dependencies are downloaded each month, and nearly 90% of codebases contain open source components that are more than four years out of date. This pervasive adoption, coupled with a lagging approach to security maintenance, creates a fertile ground for supply chain attacks. The problem is exacerbated by the often-automatic ingestion of dependencies, where developers may unknowingly incorporate malicious or vulnerable code.

Zhukov framed the problem by presenting a series of real-world incidents from 2023:

• July 2023: Malicious actors created Dependabot lookalikes that, when enabled by default in GitHub repositories, issued suspicious pull requests designed to steal developer credentials.
• August 2023: A malicious tool named VMConnect, downloaded over 200 times, mimicked the legitimate VMware vSphere client (which sees 45,000 downloads per month). The fake tool incorporated malware, delivered via PyPI, compromising users' systems.
• September 2023: Malicious JavaScript npm packages were crafted using popular brand names like Airbnb and PayPal, containing code that exfiltrated personal information to command-and-control servers.

These incidents illustrate the diverse methods attackers use to infiltrate the open source supply chain, leveraging trust, automation, and human error. To combat these threats, Zhukov pointed to established industry frameworks and reports, including the OpenSSF Top 10 Open Source Software Risks, and documents from NIST, CIS, and CISA. He emphasized that understanding these frameworks is crucial for organizations to prioritize their security efforts, rather than attempting to "do everything at once." The talk built upon the concept of a Secure Development Life Cycle (SDLC), adapting it to the specific nuances of open source software by focusing on four critical areas: secure sourcing, secure development, build and release, and the importance of Software Bill of Materials (SBOMs).

Key Findings

▶ Watch: Key threats and defenses for open source sourcing (8:20)

The core findings of Roman Zhukov's presentation revolve around the dual nature of open source in modern software development: its indispensable utility and its inherent, often underestimated, security risks. He demonstrated that attackers are increasingly sophisticated, exploiting common development practices and the sheer volume of open source components to achieve their objectives.

First, Zhukov emphasized that open source supply chain attacks are diverse, prevalent, and directly impact every organization. He detailed how attackers leverage various techniques, including:

• Dependency Confusion: Exploiting package manager resolution logic to trick build systems into pulling malicious public packages instead of intended private ones, as seen with the CarePay incident.
• Typosquatting/Masquerading: Creating packages with names similar to popular ones (e.g., Airbnb, PayPal, fake LLM models, VMConnect) to trick developers into downloading malware.
• Trojan Injection/Protestware: Embedding malicious code within seemingly legitimate packages or using automation (like the Dependabot lookalikes) to introduce vulnerabilities.
• Repository or Package Hijacking: Gaining unauthorized control over legitimate open source projects or developer accounts to inject malicious code.

Second, the talk highlighted self-hosted CI/CD runners as a critical and often overlooked attack vector. Zhukov explained that while convenient, non-ephemeral self-hosted runners, when misconfigured or lacking sufficient isolation, can provide attackers with a direct pathway from a malicious commit in a public repository to an organization's internal network. This allows for the exfiltration of secrets, privilege escalation, and lateral movement, effectively bypassing traditional perimeter defenses.

Third, Zhukov championed the crucial role of Software Bill of Materials (SBOMs) in mitigating these risks. He presented a compelling case study illustrating how the presence of SBOMs can drastically reduce the time required to assess and mitigate vulnerabilities (from weeks to hours or days), especially during widespread incidents like Log4j. However, he also acknowledged that the adoption and verification of SBOMs are still in their early stages, with challenges in format standardization and accuracy.

Finally, a pervasive theme was the necessity of a "trust but verify" approach. Zhukov argued that relying solely on the perceived trustworthiness of open source projects or package managers is insufficient. Organizations must implement continuous verification processes, rigorous pipeline automation, and a strong foundation of classic IT security principles to defend against these evolving threats effectively. He stressed that enabling basic security features in Source Code Management (SCM) platforms like GitHub or GitLab can prevent up to 80% of malicious activities.

Technical Deep Dive

▶ Watch: Hacking and defending self-hosted CI/CD runners (25:00)

Roman Zhukov's talk provided a comprehensive technical exploration of how open source software can be compromised and subsequently defended, delving into specific attack vectors and architectural considerations.

Attack Vectors and Mechanisms:

Zhukov outlined several prevalent attack techniques:

1. Dependency Confusion: This attack exploits package managers' search order. If a private package has the same name as a public package, a misconfigured build system might fetch the public (and potentially malicious) version from a public repository (e.g., npm, PyPI) instead of the intended private one. The CarePay incident, involving a fake package related to Apache Kafka, served as a prime example. Attackers often scan for lock files containing dependency lists to identify potential targets for this method.

1. Typosquatting / Masquerading: Attackers create malicious packages with names very similar to legitimate, popular ones (e.g., vmconnect instead of vmware-vsphere, or packages mimicking Airbnb and PayPal brands). These packages often contain malware designed to exfiltrate data to Command and Control (C2) servers. The recent trend of fake Large Language Model (LLM) models also falls into this category, leveraging current technology trends.

1. Trojan Injection / Protestware: This involves embedding malicious code within a seemingly benign package. The speaker cited the example of Dependabot lookalikes that, once integrated into a repository, would issue pull requests containing code designed to steal developer credentials. This method often preys on the automation inherent in modern development workflows.

1. Repository or Package Hijacking: This occurs when attackers gain unauthorized access to a legitimate open source project's repository or a developer's package manager account. Once hijacked, the attacker can inject malicious code into new releases, affecting all downstream consumers. The compromise of the co package on npm was mentioned as a notable instance.

Compromising the CI/CD Pipeline:

Zhukov illustrated a common attack blueprint:

• Compromised Source: Malicious code is submitted to a repository, either directly or through a hijacked account.
• Compromised Build Platform: The malicious code traverses the CI/CD pipeline, potentially compromising the build environment itself.
• Bad Dependency / CI/CD Bypassing: The malicious code might introduce a backdoor or branch out to execute unauthorized actions, leveraging the build system's permissions.

Attackers identify targets by using tools like Wappalyzer or BuiltWith to scan websites for technologies in use, then examining lock files for dependency lists. They also look for popular but deprecated packages or leverage poorly protected/expired credentials.

Self-Hosted Runner Attacks:

A significant portion of the deep dive focused on self-hosted CI/CD runners (e.g., GitHub Actions runners, GitLab CI runners). These runners execute jobs locally, offering greater control over hardware and system environments, which is often necessary for specific use cases. However, they pose a critical security risk, especially when configured as non-ephemeral runners.

• Ephemeral vs. Non-Ephemeral: Ephemeral runners are designed to perform a single job and then be terminated, unregistered, and auto-updated. Non-ephemeral runners, conversely, persist across multiple jobs.
• The Attack Flow: If a public repository uses non-ephemeral self-hosted runners without sufficient checks, an attacker can become a contributor (even via a minor documentation commit). They then push a malicious commit containing a malicious workflow. The non-ephemeral runner, residing in the organization's internal network, checks out the attacker's code. This allows the attacker to:
• Steal secrets (API keys, credentials).
• Take over workflow permissions.
• Escape from the end-user machine where the runner is executed.
• Establish connections to external C2 servers, enabling lateral movement within the internal infrastructure.
• Attacker Reconnaissance: Attackers can identify self-hosted runners by searching public GitHub repos for the runs-on command. They can further identify non-ephemeral runners by checking for actions (e.g., checkout step) that contain repository cleaning logic, indicating the runner is designed to persist.

Software Bill of Materials (SBOMs) and SLSA:

Zhukov emphasized SBOMs as a foundational element for securing the supply chain.

• Purpose: SBOMs provide an inventory of software components, their licenses, and dependencies. They are crucial for:
• Component Inventory: Knowing what's inside your software.
• License Inventory: Managing legal compliance.
• Dependency Understanding: Mapping the entire dependency graph.
• Benefits: SBOMs enable automated, at-scale risk assessment. In a scenario like the Log4j vulnerability, an organization with SBOMs could assess its exposure in hours or days, compared to weeks of manual scanning.
• Challenges: The SBOM ecosystem is still maturing. While formats like CycloneDX and SPDX exist, challenges remain in standardizing their use, ensuring accuracy, and verifying their provenance.
• Vulnerability Exploitability eXchange (VEX): An extension to SBOMs, VEX provides information about whether a specific vulnerability affects a product or component, reducing false positives and aiding prioritization.
• SLSA (Supply-chain Levels for Software Artifacts): This framework, primarily focused on build hardening (especially SLSA v1.0), defines levels of supply chain integrity. Achieving SLSA Level 1, which ensures a correct build looks like what it's supposed to, is considered achievable for most organizations. SLSA emphasizes provenance – an attestation showing how a software artifact was built, its dependencies, and who signed it – acting as a "passport for builds."

Demo / Proof of Concept

▶ Watch: Understanding the SALSA framework for build hardening (35:00)

While Roman Zhukov's presentation was rich with practical examples and detailed explanations of attack methodologies, it did not feature a live, interactive demonstration or proof-of-concept. Instead, he effectively illustrated potential attack scenarios and how they could be executed by walking through diagrams and referencing real-world incidents. The "how to hack" sections served as conceptual proofs of concept, detailing the steps an attacker would take to exploit vulnerabilities like dependency confusion, typosquatting, or self-hosted CI/CD runners, thereby providing attendees with a clear understanding of the offensive techniques without a live execution.

Defensive Implications

▶ Watch: Q&A: Escaping virtualized self-hosted runner environments (39:50)

Roman Zhukov provided a robust framework for defending against open source supply chain attacks, categorized into practices and tools for each of his focus areas: secure sourcing, secure development, build and release, and SBOMs.

Secure Sourcing and Development:

• Practices:
• Use Trusted Sources: Prefer trusted package managers and ingestion channels.
• Assess Repository Health: Evaluate the security posture of open source projects, including their Mean Time To Resolution (MTTR) for CVEs, commit frequency, and overall security practices.
• Zero-Trust Verification: Implement rigorous verification for all ingested packages, including checksums, signatures, and vulnerability scans, even for trusted sources.
• Continuous Updates: Regularly update dependencies to the latest stable versions. While Zhukov expressed reservations about strict version pinning, he acknowledged its use in specific contexts.
• Curate Critical Components Locally: For larger organizations, maintaining a local, curated repository of critical open source components can provide an additional layer of control and security.
• Package Configuration Audits: Regularly audit configuration files for package managers (e.g., for PyPI, NuGet) to ensure they align with security policies.
• Enable SCM Security Features: Activating built-in security features in platforms like GitHub or GitLab can prevent up to 80% of malicious activities.

• Tools:
• Source Health Check: OpenSSF Scorecards (to which Zhukov contributes) and Deps.dev are recommended for assessing repository health.
• Local Package Management: Utilize native tools embedded in SCMs and local package management solutions.
• Dependency/CVE Detection & Updating: Zhukov recommended a suite of tools including SonaType, Dependabot, Renovate, Trivy, and SNYK for vulnerability scanning and automated updates.

Software Bill of Materials (SBOMs):

• Practices:
• Generate SBOMs: Don't get stuck on format; the priority is to generate something to gain visibility into components.
• Validate Provenance & Signatures: For responsible vendors, ensure SBOMs are signed and include provenance information, ideally conforming to SLSA standards.
• Create Your Own SBOMs: Even without government mandates (like those for US government contractors), generating internal SBOMs provides invaluable inventory and auditing capabilities.
• Leverage VEX: Consume and share Vulnerability Exploitability eXchange (VEX) documents to understand the actual impact of disclosed vulnerabilities.

• Tools:
• SBOM Generation: Syft, Tern, CycloneDX CLI, and various SPDX tools were listed.
• SBOM Verification: Emerging tools are available to verify SBOM conformance and accuracy, ensuring they meet defined requirements.

Build and Release (Securing CI/CD and Runners):

• Practices:
• Classic IT Security for Runners: Treat self-hosted runners as critical infrastructure. Isolate them in a DMZ (Demilitarized Zone) or dedicated network segments.
• Harden Access and Token Management: Implement robust controls for credentials used by CI/CD pipelines and runners.
• Leverage SCM Permissions & Branch Protection: Enforce strict branch protection rules and use code owners to ensure only authorized individuals can merge code.
• Use Ephemeral Runners: Configure self-hosted runners to be ephemeral (one job and killed) whenever possible, with auto-updating and unregistration enabled.
• Prevent Runner PR Approvals: Crucially, prevent automated runners from approving pull requests, especially for production or main branches, requiring human oversight.
• Manage Secrets: Implement a dedicated secret management solution.
• Scan Binaries: Scan compiled binaries for CVEs and malware after the build process, in addition to source code scanning.
• Check Infrastructure as Code: Do not overlook scanning infrastructure-as-code scripts for vulnerabilities.
• Adopt SLSA: Implement the Supply-chain Levels for Software Artifacts (SLSA) framework to harden the build platform and ensure provenance. Work towards reproducible builds to ensure consistency and verifiability.

• Tools:
• Repo/Runner Settings Check: OpenSSF Scorecards now include checks for runner configurations.
• Secret Management/Scanning: Implement both secret management systems for credentials and secret scanning tools for detecting embedded secrets in code.
• Classic Network Security: Apply traditional network security measures (firewalls, IDS/IPS) to protect runner environments.
• CVE/Malware/Script Scanning: Zhukov specifically highlighted CVBin, an open-source binary analysis tool developed by Intel, for detecting CVEs in binaries.

Key Takeaways

• Trust But Verify: Never blindly trust open source components or automated processes. Continuously verify the integrity, security posture, and provenance of all ingested software.
• Automate and Oversee Pipelines: Implement robust automation for security checks throughout the CI/CD pipeline, but ensure proper oversight and human intervention points to prevent malicious automation.
• Leverage Industry Frameworks and Tools: Don't reinvent the wheel. Utilize established frameworks like OpenSSF Top 10, NIST C2C2F, SLSA, and recommended tools (e.g., OpenSSF Scorecards, Trivy, Dependabot) to guide and streamline security efforts.
• Self-Hosted Runners are a Critical Attack Surface: Treat self-hosted CI/CD runners with extreme caution, isolating them with classic IT security measures (DMZ) and configuring them as ephemeral to prevent network escape and privilege escalation.
• SBOMs are Essential for Rapid Response: Implement Software Bill of Materials (SBOMs) generation and consumption to quickly identify and assess exposure to vulnerabilities, drastically reducing response times during security incidents.
• Basic Security Hygiene Prevents Most Attacks: Enabling foundational security features in Source Code Management (SCM) systems and adhering to classic IT security principles can prevent a significant majority (up to 80%) of open source supply chain attacks.

About the Speaker(s)

Roman Zhukov is a Principal Engineer at Intel, bringing over 15 years of experience in IT security, encompassing infrastructure security, endpoint security, and security operations. Hailing from Ireland, Zhukov is a passionate advocate for open source security, evident in his contributions to initiatives like OpenSSF Scorecards. Intel itself is recognized as a top-tier contributor to open source projects, underscoring Zhukov's practical experience in securing large-scale open source development. His expertise bridges the gap between traditional enterprise IT security and the evolving challenges of the open source supply chain, providing a well-rounded perspective on both offensive and defensive strategies.