(Supply) Chain Reaction

(Supply) Chain Reaction

Speakers: Iddo Eldor, Vulnerability Researcher, Mirror Security

Conference: Hacktivity

YouTube: https://www.youtube.com/watch?v=aFIdt6H5aSE

Overview

In this Hacktivity presentation, Iddo Eldor, a seasoned vulnerability researcher now leading efforts in software supply chain security at Mirror Security, sheds light on the escalating threat landscape of software supply chain attacks. The talk, titled "Supply Chain Reaction," dissects the multifaceted nature of these attacks, moving beyond theoretical concepts to highlight real-world vulnerabilities and sophisticated exploitation techniques. Eldor emphasizes that the software supply chain encompasses every element touching an application throughout its Software Development Life Cycle (SDLC), including third-party components, proprietary code, deployment methods, infrastructure, interfaces, protocols, and developer tools.

Eldor underscores the critical importance of addressing these threats, citing a Gartner prediction that 45% of organizations will experience a software supply chain attack by 2025. This alarming statistic highlights the profound impact these attacks can have on business operations, data integrity, and customer trust. The talk provides a comprehensive overview of how attackers compromise popular dependencies to achieve widespread impact, exploiting the inherent trust in commonly used libraries and systems.

The presentation distinguishes itself by delving into practical attack scenarios, including novel vulnerability research conducted by Eldor's team. It covers a spectrum of attack vectors, from well-known techniques like typosquatting and dependency confusion to complex exploits involving package manager vulnerabilities and the compromise of artifact storage systems. Eldor's work culminates in the demonstration of a sophisticated privilege escalation chain, transforming a medium-severity vulnerability into a critical one through a clever combination of user enumeration, SQL injection, and optimized token leakage.

Background

▶ Watch: Speaker introduction and software supply chain definition (0:00)

The software supply chain has become an increasingly lucrative target for advanced persistent threat (APT) actors and other malicious entities, leading to a significant increase in both the volume and sophistication of attacks. The core challenge stems from the interconnected nature of modern software development, where applications rely heavily on a vast ecosystem of third-party libraries, open-source components, and shared infrastructure. This interconnectedness means that compromising a single, popular dependency can yield a widespread impact across countless software applications and systems, potentially granting access to valuable data or critical systems across various industries.

Companies must navigate a complex landscape where they are not only responsible for their own organizational security but also for the security of all customers who receive their software. Beyond the direct business implications of downtime, data loss, and revenue impact, organizations are compelled to adhere to stringent regulatory and compliance standards such as SOC 2, ISO, and NIST. A robust supply chain security posture is thus not merely a risk mitigation strategy but a fundamental commitment to data protection, enhancing reputation and trustworthiness among customers, partners, and regulators.

Eldor outlines several common attack vectors that exploit trust and vulnerabilities within the supply chain:

• Typosquatting / Package Name Confusion: Attackers register subtly different variations of popular package names, often leveraging Unicode characters to create visually similar but malicious packages. Developers, in a hurry, might copy-paste commands like pip install some_library from blog posts or Stack Overflow, inadvertently installing malicious code that could lead to persistent threats.
• Dependency Confusion: This attack occurs when a malicious package is uploaded to a public or private repository with a name identical to an internal package used by the target organization. Build systems, when resolving dependencies, might prioritize the malicious public package over the intended private one.
• Repo Jacking: Malicious actors register usernames and create repositories that were previously used by an organization but have since been abandoned or renamed. Any project still relying on dependencies from the old, now-controlled repository will fetch code from the attacker, potentially containing malware. Eldor notes that mitigations for repo jacking have been bypassed four times in the last two years alone, highlighting its persistence.
• Package Manager Vulnerabilities: Specific package managers have suffered from critical flaws. RubyGems, the Ruby package manager, was vulnerable to a guessing attack allowing attackers to override the next version of any package. npm, the Node.js package manager, had a similar vulnerability two years prior and more recently suffered from "manifest confusion," allowing attackers to modify the manifest data to hide malicious dependencies and scripts from appearing in the npm registry, even though they still existed in the package.json file and would execute upon installation.
• Credential Theft and Lateral Movement: Compromising a domain, navigating to a "forgot password" service, and clicking a validation link can lead to the theft of repository or package manager credentials. This initial compromise can facilitate lateral movement from artifact storage servers or CI/CD pipelines to the entire SDLC ecosystem, effectively compromising the build environment.
• Build Environment Compromise: As seen in high-profile incidents like SolarWinds and 3CX, malicious code can be injected during the build process before the final artifact is created. Eldor also mentions a specific vulnerability found by SonarSource in TeamCity by JetBrains, a popular CI/CD platform, demonstrating that even secure build tools can have exploitable flaws.

These attack methods collectively underscore the pervasive and sophisticated nature of supply chain threats, necessitating a proactive and multi-layered defense strategy.

Key Findings

▶ Watch: Package manager vulnerabilities (RubyGems, npm) (5:45)

Iddo Eldor's research at Mirror Security culminated in several significant findings, demonstrating a practical approach to uncovering and exploiting vulnerabilities within the software supply chain. The team focused their efforts on a widely used artifact storage product, which Eldor deliberately left unnamed but described as a company with a strong security posture, evidenced by a bug bounty SLA of two days (though practically resolved in minutes). Despite this, Eldor's team uncovered a series of critical issues.

Initially, their investigation revealed multiple existing exploits in older versions of the product, along with several duplicates of previously reported vulnerabilities. Notably, they found a way to bypass a Brute Force mitigation, which could have significant implications for credential-based attacks. The most impactful finding, however, emerged from a deeper dive into a recent medium-severity CVE. Through meticulous root cause analysis, Eldor's team successfully transformed this medium-severity vulnerability into a critical privilege escalation exploit.

The core of this critical exploit chain involved several interconnected discoveries:

1. User Enumeration Primitive: A method to enumerate valid users for a specific instance of the artifact storage product. This primitive was particularly potent because the email content sent for enumeration originated from the trusted artifact storage instance domain, not a suspicious external domain, making it highly effective for phishing campaigns.
2. SQL Injection in Proprietary Query Language: The discovery of a SQL injection vulnerability embedded within a proprietary query language parser. This flaw allowed attackers to manipulate the order by clause in queries, providing a powerful primitive for data exfiltration.
3. Blind Boolean-Based SQL Injection for Token Leakage: Leveraging the SQL injection, Eldor's team developed a blind Boolean-based SQL injection technique. This allowed them to systematically leak sensitive data, specifically the admin reset password token, which is generated as a Base64 encoded string containing a UUID v4 and a timestamp, valid for 30 minutes.
4. Optimized Token Exfiltration: To overcome the time constraints of the 30-minute token validity, the team devised an optimized bit-by-bit exfiltration method for the Base64 token. This significantly reduced the number of requests required, making the token leakage feasible within the validity window and enabling subsequent admin password reset.

These findings collectively illustrate a complete attack chain, starting from reconnaissance and phishing enablement, progressing through complex technical vulnerabilities, and culminating in full administrative control over a critical component of the software supply chain.

Technical Deep Dive

▶ Watch: SQL injection to leak admin password reset token (14:00)

The technical exposition of Iddo Eldor's research provides a granular look at the methods employed to achieve critical privilege escalation in a widely deployed artifact storage product. The attack chain meticulously crafted by Eldor's team begins with reconnaissance and user enumeration, vital steps for any targeted offensive operation.

The first primitive discovered was a robust user enumeration technique. Eldor noted that finding target instances of this artifact storage product in the wild was remarkably easy, thanks to publicly available search engines like Shodan. Furthermore, identifying managed instances often only required enumerating subdomains of a company's main domain. Their research revealed approximately 30,000 servers running the product in the wild, with an average of 10 users per server, though some instances boasted up to 500 users. This enumeration method was particularly effective for phishing, as the controlled mail content would be sent from the trusted artifact storage instance's domain, lending it significant legitimacy compared to external phishing attempts like those seen targeting PyPI maintainers. While the exact bug enabling this enumeration could not be fully disclosed due to ongoing sensitivities, its impact as an initial foothold was clear.

The next critical step in the attack chain involved leveraging this enumeration to target the admin user. The strategy was to trigger the "forgot password" service for the admin account. If the default admin username had been changed, the previously discovered user enumeration primitive would easily reveal the new name. When the "forgot password" request is made, a temporary token is generated in the database. This token is a Base64 encoded string comprising a UUID version 4 and a timestamp, making it valid for half an hour. The objective then became to leak this token.

To achieve token leakage, Eldor's team embarked on an intensive vulnerability research effort. They downloaded all available versions of the product, noting that the binaries contained symbols and were not obfuscated, which greatly aided in understanding code changes between versions. Their approach involved enumerating the attack surface by identifying requests accessible to a non-privileged authenticated user. Initially, decompiling the product and manually searching for allow annotations for user-accessible services proved challenging due to the product's complexity, with "many files, many services" leading to numerous false positives from additional checks for roles like "repo manager" or "admin."

To overcome this manual audit bottleneck, Eldor innovatively employed a Large Language Model (LLM). Specifically, they utilized Code Llama version 2 to sort potential attack surface paths by complexity. The LLM was "taught" to prioritize code paths with many constraints and those involving Attacker Controlled Input (ACI), assigning them higher scores. This methodology was complemented by a strong advocacy for patch diffing, digging deep into known vulnerabilities to pinpoint their root causes and detect new variants, ensuring that old bugs within a patch were also factored into the scoring.

After this LLM-assisted sorting, manual auditing of the highest-scoring paths commenced. This led to the discovery of a "good old SQL injection" within a proprietary query language parser. The vulnerability manifested when adding a filter to queries, where the order by columns could be controlled by the attacker without constraint. The database used by the product, however, did not allow chaining multiple queries, meaning the attacker could only control how existing SELECT queries were sorted.

This specific limitation led to the implementation of a blind Boolean-based SQL injection. By querying for all bundles for a user and manipulating the order by clause, the attacker could infer information based on the sorted order of results. For instance, to leak the admin reset password token (an alphanumeric Base64 string), the attacker would iterate through each possible character (52 letters, 10 numbers, 3 special characters) for each position in the token. If a character matched, the results would be sorted in a predictable "wanted" way; otherwise, they would be sorted differently. This technique, while effective, was computationally intensive, requiring a significant number of requests that, on average, would exceed the token's half-hour validity period.

To optimize the token exfiltration and ensure it could be completed within the 30-minute window, Eldor's team devised a more efficient bit-by-bit comparison technique. Recognizing that each character of the token is one byte and the most significant bit is always zero, they could iterate through each bit for each character. Instead of guessing characters, they would simply ask, "Is this bit 'on'?" If the bit was "on," the results would sort in the desired way; if not, they would sort differently. This method drastically reduced the number of requests, enabling the successful leakage of the time-sensitive admin reset password token and subsequent admin password reset. This sophisticated multi-stage attack chain highlights the ingenuity required to transform seemingly minor vulnerabilities into critical compromises.

Demo / Proof of Concept

▶ Watch: Optimizing token leakage with bit-by-bit SQLi (16:00)

While Iddo Eldor did not present a live, interactive demonstration during the talk, the "Technical Deep Dive" section effectively served as a detailed walkthrough of the proof-of-concept. Eldor meticulously described each step of the attack chain, from user enumeration to the final privilege escalation, providing pseudo-code snippets and explaining the logical flow of the exploits.

The absence of a live demo was acknowledged by Eldor, who stated at one point, "due to time constraints I will not show all the case studies from the last two years." However, the descriptive nature of his presentation, particularly regarding the SQL injection and the optimized token leakage, provided a clear, actionable methodology that functions as a theoretical proof-of-concept. The level of detail, including the specific characteristics of the token (Base64, UUID v4, timestamp, 30-minute validity) and the optimization technique (bit-by-bit comparison), strongly implies that this attack was successfully executed and validated by his team. The focus was on conveying the "how it worked" rather than a live "watch it work."

Defensive Implications

▶ Watch: Challenges in verifying compiled artifact to source code (19:45)

The insights gleaned from Eldor's presentation offer crucial guidance for defenders seeking to bolster their software supply chain security. The primary takeaway is the shift from reactive detection to proactive prevention, emphasizing that attacks are often discovered by EDRs or other means only after a compromise has occurred, as exemplified by the 3CX attack. The goal should be to prevent malicious code from ever reaching production environments and customers.

Key defensive strategies include:

• Vigilant Monitoring of Maintainers and Dependencies: Organizations must extend their security perimeter beyond their immediate codebase to scrutinize the behavior and security posture of upstream maintainers. This includes monitoring for signs of compromise (e.g., maintainer credentials for sale on dark web marketplaces) or malicious intent, such as "protestware" or maintainers going rogue (like the mock library author who exfiltrated and sold developer data). Defenders should look for suspicious commits, deleted issues, or unusual activity, acknowledging that not every repository benefits from "1,000 eyes" like the Linux kernel, where even academic researchers have intentionally introduced vulnerabilities.
• Secure Patching and Update Hygiene: While updating to the latest minor or general versions is crucial for bug fixes, improvements, and new features, Eldor warns that new versions can also introduce new attack surfaces and bugs. Many discovered vulnerabilities are variants of previously reported issues where the patch was not "hermetic," allowing for bypasses. A robust patch management process should include thorough testing and validation to ensure that updates don't inadvertently introduce new risks.
• Comprehensive Transitive Dependency Management: The security of an application is not just about its direct dependencies but also the entire tree of transitive dependencies. Organizations must adopt tools and practices that scan and monitor the security of the entire dependency graph, as a vulnerability deep within a transitive dependency can be just as exploitable as one in a direct one.
• Advanced Malicious Code Detection: Beyond traditional vulnerability scanning, which typically focuses on non-intentional flaws, detecting malicious code requires more sophisticated approaches:
• Static Code Analysis: Employing rules and heuristics to identify suspicious patterns, such as pushes without review, commits at unusual hours (e.g., midnight), or a developer modifying an area of the codebase for the first time.
• Machine Learning (ML): Training ML models to learn and cluster malicious patterns in code.
• Artifact-to-Source Code Comparison: This is highlighted as a "super important" but "very complex problem," especially for compiled languages like Java or C. The objective is to verify that the compiled package code precisely corresponds to its source code, ensuring no tampering has occurred during the build process. Eldor discusses several approaches, each with challenges:
• Reproducible Builds: While ideal, these have proven "very hard and not deterministic" in practice.
• Open Source Project Verification: Convincing all open-source project maintainers to verify every step of their development and build process is "kind of not realistic."
• Binary to Source Compression (Reverse Engineering): This involves using reverse engineering techniques and many heuristics to retrieve information lost due to compiler optimizations, or teaching an AI to do so (a "whole different presentation").
• Local Compilation: The seemingly simple solution of "bring all the source code to your computer, compile it, go to lunch" is presented with a caveat, acknowledging that even the CI/CD environment where this compilation occurs can be compromised, as demonstrated in the talk.

Ultimately, defending against supply chain attacks requires a holistic approach that combines technical controls, process improvements, and a deep understanding of the attack surface, from the earliest stages of development through to deployment and ongoing maintenance.

Key Takeaways

• Software supply chain attacks are rapidly increasing in volume and sophistication, with Gartner predicting 45% of organizations will experience one by 2025.
• Common attack vectors include typosquatting, dependency confusion, repo jacking, and exploiting vulnerabilities in package managers like RubyGems and npm.
• Deep vulnerability research, aided by tools like LLMs for attack surface analysis and patch diffing, can transform seemingly medium-severity bugs into critical privilege escalation exploits.
• A complete attack chain can involve user enumeration, SQL injection in proprietary systems, and optimized blind Boolean-based techniques to exfiltrate time-sensitive credentials like admin reset tokens.
• Defenders must focus on proactive prevention, including vigilant monitoring of maintainer activity, securing transitive dependencies, and implementing advanced malicious code detection methods.
• Verifying the integrity of compiled artifacts against their source code is a critical, yet complex, challenge that requires robust solutions beyond current industry practices.

About the Speaker(s)

Iddo Eldor is an experienced cybersecurity professional with a decade of expertise as a vulnerability researcher in various cyber offensive companies. He has a strong background in uncovering and exploiting security flaws in complex systems. Currently, Iddo serves at Mirror Security, where he leads efforts focused on combating software supply chain attacks, a critical and evolving area of cybersecurity. His practical knowledge extends to teaching, as he has developed and delivers a 4-day course on real-world vulnerability research in instant messaging applications. Eldor's work emphasizes a hands-on, analytical approach to understanding and mitigating sophisticated threats within the software development ecosystem.