Snow Nor Rain Nor Dependency Confusion: How to...

Snow Nor Rain Nor Dependency Confusion: How to...

Speakers: Jessica Smith, Justin Engler

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=NDKWQ3YZuCQ

Overview

This talk, "Snow Nor Rain Nor Dependency Confusion: How to...", delivered by Jessica Smith and Justin Engler, engineers on the offensive security (red team) at Block, delves into the intricacies of the dependency confusion vulnerability. The presentation provides a comprehensive overview of what dependency confusion is, details a real-world red team operation conducted by Block to exploit it, and outlines the various strategies and challenges encountered in remediating this critical issue. The speakers emphasize that this was one of their team's inaugural red team engagements, offering valuable insights into the practicalities of identifying, exploiting, and defending against a vulnerability that poses a significant remote code execution (RCE) risk in modern software supply chains.

The core problem addressed is the potential for package managers to prioritize a malicious public package over an intended private internal package when both share the same name. This can lead to an attacker's code being executed on a developer's machine. Smith and Engler not only explain the technical mechanics of this attack but also share their experience in crafting stealthy payloads, achieving persistence, exfiltrating sensitive data, and demonstrating impact within a corporate environment. The latter half of the talk focuses on the complex, often messy, process of implementing effective, scalable, and developer-friendly defenses, highlighting the trade-offs and challenges of various mitigation strategies.

The talk is highly relevant for security professionals, developers, and infrastructure teams concerned with software supply chain security. It provides a practical case study of a well-known vulnerability, demonstrating its real-world impact and offering actionable advice for both offensive and defensive security practitioners. By sharing their journey from initial exploitation to long-term remediation, Smith and Engler offer a candid look at the challenges of securing complex enterprise development environments.

Background

▶ Watch: Artifactory as a package manager proxy for public and private packages. (01:00)

The concept of dependency confusion was brought to prominence by Alex Berson's seminal Medium post in 2017. At its heart, the vulnerability exploits the way modern package managers resolve dependencies when organizations use internal package repositories. Companies often configure a package repository, such as Artifactory, Azure Artifacts, Nexus, or Zype, to act as a proxy or middleman between developers and public package registries like npm or pip. When a developer executes a command like npm install or pip install, their machine is configured to first query the internal Artifactory instance. Artifactory then queries the public registry for the requested package and serves it back to the developer, often caching it for future use.

A key benefit of this setup is the ability to host private packages internally, allowing developers to share proprietary code within the company without open-sourcing it. The dependency confusion vulnerability arises when a package with the same name exists in both the public and private package stores. In such a conflict, package managers typically default to serving whichever package has the higher version number. This behavior creates an attack vector: if an attacker discovers the name of a private internal package, they can publish a malicious package with the identical name to a public repository (e.g., npm, PyPI) but with a higher version number. When a developer subsequently attempts to install the legitimate private package, their package manager, following its resolution logic, will instead pull and install the attacker's malicious public package, leading to remote code execution (RCE) on the developer's machine.

While theoretically applicable to any language with public package stores (Python's pip, Java's Maven, RubyGems), JavaScript and npm are frequently targeted for several reasons. Firstly, discovering internal JavaScript dependency names is often easier because developers can inadvertently include package.json files in source maps, making them accessible from the browser. This contrasts with server-side languages, where dependency lists are typically harder to obtain without source code leaks or open-sourcing. Secondly, npm packages facilitate RCE more directly through install-time scripts. The package.json configuration file allows defining a scripts attribute with hooks like pre-install or post-install. These hooks execute arbitrary code during the package installation process, making it trivial for an attacker to achieve RCE immediately upon installation, rather than waiting for the package to be referenced and executed within the application code. Defenders and security scanners often flag custom pre-install or post-install hooks as potentially malicious, a detail attackers might consider for stealthier operations.

Key Findings

▶ Watch: Why JavaScript/npm is a common target: package.json discovery and pre-install... (04:00)

The red team operation at Block, described by Jessica Smith and Justin Engler, yielded several critical findings regarding the prevalence, exploitability, and remediation of dependency confusion.

Vulnerability Scope and Targeting:

• Using a script called confused, the team scanned Block's internal repositories and identified a staggering 984 npm packages that were potentially vulnerable to dependency confusion.
• For their initial exploitation attempt, they strategically chose a non-critical internal package responsible for linting ownership (the owner out lint package). This choice was deliberate to minimize potential disruption to production systems, as a failure in linting would not break core functionality.

Exploitation and Evasion Techniques:

• The first attempt involved a "naive" malicious package uploaded to npm, which was detected and removed by npm within approximately one week. This highlighted the need for more sophisticated evasion.
• The improved malicious package incorporated several stealth techniques:
• Targeted Execution: Inspired by Stuxnet, the payload was encrypted. The decryption key was derived from the MD5 hash of a specific, company-wide configuration file present on all Block laptops. This ensured the payload would only execute on Block machines, preventing unintended execution elsewhere and complicating analysis.
• Obfuscation: The package was given a legitimate-sounding description and included benign, commonly used JavaScript code snippets from Stack Overflow in its body. This made the package appear less suspicious and helped it evade detection. This refined package was never caught by npm and remained active until the operation concluded.

Successful Compromise and Impact:

• The malicious package, specifically version 0.31, was successfully picked up by Block's internal Artifactory instance, overriding the legitimate private package.
• After a few weeks, the team observed downloads of their malicious package (version 0.32 and higher) as new projects were initiated and developers ran install scripts. This led to the successful compromise of approximately 20 developer machines, providing the red team with shells.
• A key learning from this phase was the initial mistake of placing the payload in /tmp, which resulted in lost persistence upon machine reboots. For persistence, the team later modified the crontab file. While this required user interaction (accepting a prompt to allow the terminal to edit crontab), developers, accustomed to installing software, typically accepted it. The speakers noted that asking for root privileges would have been a more robust approach.

Post-Exploitation and Data Exfiltration:

• The red team established a command and control (C2) infrastructure using Python, with both client and server components, secured by mTLS for strong authentication and encrypted communication.
• They used tools like truffle hog to scan for secrets on compromised laptops. Pillage included:
• Private SSH keys
• Keychain files
• Data from Block's internal access control system (detailing user-to-app and app-to-app access)
• Significant amounts of source code
• A GitHub Personal Access Token (PAT)
• To demonstrate impact beyond data exfiltration, the team performed a code injection. Jessica Smith observed a developer editing files for a new project. After the developer finished, she injected a message ("hey if you see this talk to offsec") into one of the files. The developer subsequently committed and pushed this modified file to GitHub, and the message was indeed noticed by reviewers, proving the ability to inject malicious code into the development pipeline.

Remediation Challenges:

• A significant finding was that implementing fixes retroactively in a "messy package environment" (lacking consistent naming conventions or restrictions) was considerably more difficult than a proactive, "greenfield" approach. This highlighted the importance of early security integration in development practices.

Technical Deep Dive

▶ Watch: Red team operation success: Gaining 20 shells, achieving persistence via cron... (09:30)

The technical depth of the dependency confusion vulnerability lies in the interaction between package managers, internal repositories, and public registries. Block's red team operation meticulously exploited these interactions and explored various defensive mechanisms.

Exploitation Mechanism:

The core of the attack leverages the package resolution logic. When a developer's machine, configured to use an internal repository like Artifactory, requests a package (e.g., npm install my-private-package), the following sequence typically occurs:

1. The package manager queries Artifactory.
2. Artifactory first checks its private package store for my-private-package.
3. If not found privately, or if configured to proxy, Artifactory then queries the public package registry (e.g., npm).
4. If my-private-package exists in both private and public stores, the package manager (or Artifactory, depending on configuration) will typically prioritize the version with the highest version number.

An attacker, knowing the name of a private package, publishes a malicious version to the public registry with a higher version number. This ensures that when a developer requests the package, the malicious public version is served, leading to RCE.

Targeting npm for RCE:

npm is particularly susceptible due to:

• Package Name Discovery: Front-end JavaScript applications often expose package.json files in source maps, which can be scraped from a browser. This provides attackers with a list of potential internal package names.
• Install-Time Hooks: The package.json file allows for scripts such as pre-install or post-install. These hooks execute arbitrary commands during the package installation process. For example, a malicious package.json might include:

This preinstall script would execute the attacker's shell script immediately upon installation.

Payload Obfuscation and Targeting:

To bypass detection and ensure targeted execution, the red team implemented a sophisticated payload:

• Stuxnet-inspired Targeting: The malicious payload itself was encrypted. The decryption key was generated by taking the MD5 hash of a specific, known file that existed on all Block employee laptops. This ensured that the payload would only decrypt and execute on a Block machine, preventing accidental execution on external systems and making analysis by security researchers more difficult. While MD5 is cryptographically weak for collision resistance, its use here for a unique identifier hash was sufficient for the targeting mechanism.
• Social Engineering/Stealth: The malicious npm package was given a plausible, albeit generic, description. Furthermore, it was padded with legitimate, commonly used JavaScript code snippets sourced from Stack Overflow. This made the package appear less suspicious to automated scanners and human reviewers, contributing to its success in remaining undetected.

Command and Control (C2):

The C2 infrastructure was built using Python for both the client (on compromised machines) and the server. To ensure secure and authenticated communication, mTLS (mutual Transport Layer Security) was implemented. This provided strong assurance that only authorized clients (the compromised Block machines) could communicate with the C2 server, preventing unauthorized access or interception of exfiltrated data.

Defensive Strategies Explored and Implemented:

1. Placeholder Packages:

• Mechanism: Registering packages with the same names as internal private packages on public registries (e.g., npm). This "squats" on the name, preventing attackers from publishing their own malicious versions.
• Pros: Quick to implement (scripted), security team-driven, protects against developers bypassing Artifactory (e.g., using extra index URL in Python).
• Cons: Does not prevent recurrence (requires continuous scanning and registration), leaks internal package names, GitHub API limitations (only indexes default branch), "internet citizenship" concerns. This was a quick stop-gap measure.

1. Requiring Scopes (npm-specific):

• Mechanism: Utilizing npm's scoped packages (e.g., @block/my-package). Only members of the @block organization can publish packages under that scope.
• Pros: Good developer best practice, simplifies internal package identification, doesn't leak package names to the public.
• Cons: npm-specific, difficult to enforce culturally and technically (gating developers), hard to retrofit to existing packages, still leaks package names to npm itself. Block ultimately decided not to implement this as a primary security control.

1. Artifactory Exclude Rules:

• Mechanism: Configuring Artifactory to explicitly not look externally for packages matching a specific pattern (e.g., a regex like ^Square.* for packages starting with "Square").
• Pros: Language agnostic, prevents package name leakage (Artifactory doesn't query externally), transparent to developers.
• Cons: Requires consistent naming conventions (which Block lacked initially), suffers from the recurrence problem (needs regular updates), potential performance impact with a large number of rules (Block had ~1000 rules, but didn't observe significant impact). This was used to some extent.

1. Artifactory Priority Resolution:

• Mechanism: Configuring Artifactory to prioritize a specific internal source. If a package exists in that priority source, Artifactory serves it immediately without checking external registries.
• Pros: Language agnostic, Artifactory-level, no package name leakage, transparent to developers, elegant solution.
• Cons: Edge case with "vestigial forks": If a developer previously created a private fork of a public package (e.g., an old private fork of React) and then stopped maintaining it, enabling priority resolution would cause developers to receive the outdated private fork instead of the current public version, breaking functionality. This requires significant cleanup of such forks. Also, managing multiple internal sources adds complexity. This is being gradually implemented as vestigial forks are cleaned up.

1. Pinning Dependency Versions:

• Mechanism: Explicitly specifying exact version numbers for all dependencies in project configuration files.
• Pros: Reduces the risk of unexpected package updates, including malicious higher-versioned packages.
• Cons: Not a direct dependency confusion fix, and the complexities of dependency pinning itself are a separate topic.

The overall strategy for Block involved a combination of exclude rules and gradually enabling priority resolution, coupled with ongoing cleanup of legacy packages.

Demo / Proof of Concept

▶ Watch: Pillage obtained: Private SSH keys, keychain files, internal app access data,... (11:00)

The talk itself serves as a detailed account of a real-world red team operation, which functioned as a comprehensive proof of concept for the dependency confusion vulnerability. The speakers walked through the entire lifecycle of the attack, from initial reconnaissance to post-exploitation and impact demonstration.

Initial Attempt and Refinement:

• The first demonstration involved a "naive" malicious package uploaded to npm. This package was quickly detected and removed by npm within approximately a week, highlighting the need for more sophisticated evasion techniques.
• The subsequent, successful proof of concept incorporated advanced stealth. The malicious package was designed to execute only on Block machines by using a hash of an internal configuration file as a decryption key for its payload. It also included benign JavaScript code and a plausible description to avoid detection.

Exploitation and Shell Acquisition:

• A key visual proof was a screenshot of Block's internal Artifactory instance, clearly showing that for version 0.31 of the targeted owner out lint package, Artifactory was resolving it from the public source (npm internal cache), rather than the private one.
• The screenshot further indicated that starting from version 0.32, the malicious package began to receive downloads. This occurred a few weeks after deployment, as new projects were started and developers ran npm install commands.
• The result was the acquisition of approximately 20 shells on developer laptops, confirming successful remote code execution.

Persistence and Data Exfiltration:

• The team demonstrated persistence by modifying the crontab file on compromised machines. While this required a user prompt (which was typically accepted by developers), it showed the ability to maintain access across reboots (though an initial mistake of placing the payload in /tmp caused some shells to be lost).
• Post-exploitation, the red team showcased the exfiltration of sensitive data, including private SSH keys, keychain files, internal access control system data, source code, and a GitHub Personal Access Token (PAT). This demonstrated the high impact of gaining access to developer machines.

Code Injection Demonstration:

• A particularly impactful demonstration involved directly influencing the software development lifecycle. Jessica Smith observed a developer working on a new project. After the developer saved a file, she injected a message into it: "hey if you see this talk to offsec."
• The developer subsequently committed this modified file and pushed it to GitHub. The message was indeed seen by reviewers, proving the red team's ability to inject arbitrary code into the company's source code repository, a critical supply chain compromise. This served as a powerful illustration of the potential for malicious code to enter production systems.

The entire red team operation, as described, served as a robust, real-world proof of concept for the dependency confusion vulnerability, demonstrating its exploitability, the effectiveness of various evasion techniques, and the significant impact it can have on an organization.

Defensive Implications

▶ Watch: Remediation strategy 2: Artifactory priority resolution and challenges with '... (21:00)

The red team operation at Block provides crucial defensive implications for organizations grappling with software supply chain security. The primary takeaway is that proactive implementation of security controls is vastly more effective and less disruptive than retroactive remediation in a complex, existing environment.

Prioritize Artifactory-Level Controls:

• Artifactory Priority Resolution: This is presented as one of the most elegant and effective solutions. By configuring Artifactory to prioritize internal package sources, organizations can ensure that private packages are always served first, preventing external lookups for those names. However, this requires a thorough audit and cleanup of "vestigial forks" – outdated private forks of public packages – to avoid breaking developer workflows. Organizations should plan for this cleanup phase.
• Artifactory Exclude Rules: For organizations with less consistent naming conventions, exclude rules can serve as a valuable layer. By explicitly telling Artifactory not to query public registries for specific internal package names or patterns, the risk of confusion is mitigated. This requires an ongoing process to identify new private packages and update the rules.

Enforce Consistent Naming Conventions:

• The lack of consistent naming for internal packages significantly complicated Block's remediation efforts. Implementing and enforcing scoped packages (e.g., @yourorg/package-name in npm) from the outset is a strong preventative measure. While Block did not adopt this as a primary security control due to implementation challenges, it remains a best practice for clarity and defense.

Automated Scanning and Placeholder Packages:

• Regularly scan internal codebases and package repositories (like Artifactory) to identify all private-only package names. Tools like confused are essential for this.
• As a stop-gap or supplementary measure, automate the registration of placeholder packages on public registries for all identified private package names. This "squats" on the names, preventing attackers from publishing malicious versions. Be mindful of the recurrence problem (new private packages need new placeholders) and the potential for leaking internal package names.

Developer Education and Policy Enforcement:

• Educate developers about the risks of dependency confusion and the importance of adhering to established package management practices.
• Discourage or prevent developers from bypassing central package repositories by using mechanisms like extra index URL in Python, as these can negate Artifactory-level defenses.
• Implement policies that restrict who can publish packages to internal repositories and enforce naming conventions.

Robust Monitoring and Alerting:

• Monitor Artifactory logs and package manager activity for unusual patterns, such as unexpected downloads of internal package names from public sources, or sudden version bumps that resolve to external packages.
• Implement alerts for any package installation that involves custom pre-install or post-install scripts, as these are common vectors for RCE.

Supply Chain Security Mindset:

• Dependency confusion is a facet of broader software supply chain security. Organizations must adopt a holistic approach to securing their development pipelines, including vetting third-party dependencies, managing internal package repositories, and controlling code contributions.

Least Privilege and User Prompts:

• Review the privileges required by package installation processes. If an install script requires elevated permissions (e.g., to modify crontab), ensure it's absolutely necessary. Educate users to be cautious about accepting prompts for elevated privileges during routine software installations.

By combining these layered defenses, organizations can significantly reduce their exposure to dependency confusion attacks and strengthen their overall software supply chain security posture.

Key Takeaways

• Dependency confusion is a critical and exploitable RCE vulnerability: It poses a significant threat to organizations using internal package repositories, allowing attackers to execute arbitrary code on developer machines.
• npm is a prime target: Due to the ease of discovering internal package names via package.json in source maps and the direct RCE capabilities of pre-install hooks, JavaScript projects using npm are particularly susceptible.
• Effective exploitation requires stealth and targeting: Naive attacks are quickly caught. Successful exploitation, as demonstrated by Block's red team, involves sophisticated techniques like Stuxnet-inspired payload encryption (using a hash of a company-specific file) and obfuscation with benign code to evade detection.
• Remediation is complex, especially retroactively: Fixing dependency confusion in an established, "messy" environment (without consistent naming or controls) is significantly harder than implementing proactive measures. A multi-faceted approach is essential.
• Artifactory-level controls are powerful but have caveats: Features like Priority Resolution and Exclude Rules offer robust defenses by controlling how packages are resolved. However, they require careful implementation, potentially involving extensive cleanup of "vestigial forks" or ongoing maintenance for dynamic environments.
• Red teaming is invaluable for driving remediation: The red team operation at Block effectively highlighted the vulnerability's impact, leading to prioritized and comprehensive defensive actions. Demonstrating real-world impact, such as code injection into GitHub, can be a powerful motivator for change.

About the Speaker(s)

Jessica Smith is an Engineer on the offensive security team, also known as the red team, at Block. Her work focuses on identifying and exploiting vulnerabilities within Block's systems to improve their overall security posture. This talk covered one of the very first red team operations conducted by her team, which has been in existence for approximately two years.

Justin Engler is also an Engineer on the offensive security team (red team) at Block. Alongside Jessica, he contributed to the red team operation detailed in this presentation, which involved exploiting dependency confusion and developing strategies for remediation.

Block, where both Jessica and Justin are employed, is described as a global technology company with a focus on Financial Services.