Supply-Chain Attacks in Machine Learning Frameworks

Supply-Chain Attacks in Machine Learning Frameworks

Speakers: Yue Gao, Ilia Shumailov, Kassem Fawaz

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

This talk, presented by Yue Gao, Ilia Shumailov, and Kassem Fawaz at MLSys 2025, delves into the critical and rapidly escalating issue of supply chain attacks within the machine learning ecosystem. The speakers highlight how the increasing complexity and reliance on open-source software in ML frameworks create a unique and significantly larger attack surface compared to traditional software. They argue that existing safeguards, while robust for conventional software, are often insufficient to protect against sophisticated attacks targeting both the software and machine learning layers. The core of their work identifies a novel class of supply chain attacks that exploit the dynamic nature of Python's runtime to bypass current defenses, injecting vulnerabilities or disabling protective measures in ML models and pipelines.

The presentation underscores the urgent need for a paradigm shift in how the ML community approaches security. With incidents like the XZ Utils backdoor serving as a stark reminder of supply chain vulnerabilities in traditional software, the speakers draw parallels to emerging threats in ML, citing real-world compromises in popular frameworks like PyTorch and Hugging Face. Their research reveals a concerning lack of awareness and discussion around critical security practices within the open-source ML community, advocating for increased vigilance, adoption of more secure programming paradigms, and a fundamental re-evaluation of dependency management in ML projects.

The significance of this work lies in its comprehensive analysis of the unique threat landscape facing machine learning. By dissecting the mechanisms of these attacks, the speakers provide crucial insights for practitioners, infrastructure teams, and model builders to understand the subtle yet potent ways in which ML projects can be compromised. Their findings serve as a clarion call to action, urging the community to move beyond traditional security mindsets and develop ML-specific defenses that account for the intricate interplay between software dependencies, dynamic runtimes, and the inherent vulnerabilities of machine learning models themselves.

Background

▶ Watch: Introduction to ML supply chain attacks & XZ example (0:00)

The landscape of software development, particularly in machine learning, is heavily reliant on open-source components, forming intricate supply chains. In this model, an application's source code depends on a hierarchy of first-layer, second-layer, and often many more layers of external libraries and packages. A supply chain attack typically involves an adversary compromising one of these upstream dependencies by injecting malicious code or a backdoor, which then propagates downstream to affect the end application or users. A prominent example from traditional software security is the XZ Utils outbreak of 2024, where an attacker meticulously gained trust within the open-source community over several years, eventually becoming a maintainer of the critical XZ compression library. This allowed them to inject a sophisticated SSH backdoor, threatening widespread system compromise. The success of such attacks is often attributed to the strain on open-source maintainers, who frequently lack the time and resources to rigorously vet contributions and manage project security.

For machine learning projects, the threat of supply chain attacks is not only present but accelerating, particularly with the rapid proliferation of large language models (LLMs). The speakers cite several recent incidents that underscore this burgeoning problem. In 2023, PyTorch was compromised when a hacker published malicious dependencies on PyPI, the Python Package Index. Similarly, Hugging Face, a central hub for ML models and datasets, was found to allow arbitrary code execution when users employed third-party models, creating a direct entry point for model providers to inject backdoors. Even internal incidents, such as a Bytedance intern modifying internal PyTorch source code to disrupt training by altering random seeds, optimizers, data loaders, and model weights, highlight the fragility of ML pipelines. These events suggest an exponential growth trajectory for supply chain attacks in ML frameworks.

Despite the existence of a rich set of safeguards for traditional open-source software supply chains—including code signing, security matrices, reproducible builds, and role-specific security practices for maintainers, providers, and consumers—ML applications remain highly vulnerable. The fundamental reason for this vulnerability, as articulated by the speakers, is the significantly larger and more complex attack surface presented by machine learning frameworks. First, ML frameworks inherently possess more complicated software supply chains, often depending on a greater number of Python packages. A measurement of the top 50 ML and Linux packages on GitHub revealed that ML packages, on average, have more dependencies. Second, ML itself introduces a distinct set of vulnerabilities that have persisted for a decade, such as adversarial examples, data poisoning, and model stealing. With LLMs, new attack vectors like prompt injection and jailbreaking further expand this surface. Crucially, the target of attackers has shifted from compromising operating systems or stealing user information to manipulating or exfiltrating ML models, allowing for less conspicuous but equally damaging objectives. This unique confluence of software and ML-specific vulnerabilities defines the challenging security landscape for the ML community.

Key Findings

▶ Watch: Why ML has a larger attack surface (4:00)

The central and most critical finding presented in this talk is the identification of a new class of supply chain attacks specifically tailored for machine learning models. These attacks uniquely exploit vulnerabilities at both the software level (primarily Python's dynamic runtime) and the machine learning level, allowing adversaries to achieve their objectives without triggering traditional security alerts or ML-specific defenses. The speakers demonstrate that a few simple Python tricks can enable attackers to bypass existing safeguards, presenting a more severe and insidious threat.

Specifically, the key findings include:

1. Python Runtime as a Root Cause: The dynamic nature of the Python runtime is pinpointed as a primary enabler for these novel attacks. When a Python package is imported, it is granted broad access to the importing application's source code and all objects in its runtime environment. This allows a compromised dependency to override global and local objects within downstream applications, including core ML framework functionalities.

1. Two Main Attack Vectors:

• Injection of Vulnerabilities: Attackers can directly inject malicious logic into the ML pipeline or inference process. This can range from subtly altering model behavior (e.g., injecting backdoors into inference functions by adding specific patterns or prompt injections into tokenizers for LLMs) to compromising the entire ML pipeline by changing seemingly innocuous parameters (e.g., switching an image downscaling interpolation mode to nearest neighbor, which might appear harmless but can significantly increase the attack surface for adversarial inputs).
• Bypassing or Disabling Defenses: Compromised packages can actively subvert existing ML defenses. An illustrative example is overriding the SoftMax function to inflate margins between top classes, thereby defeating margin-based model stealing detection mechanisms without traditional malicious code or suspicious behavior.

1. Evasion of Existing Safeguards: These new attacks are particularly potent because they do not rely on suspicious system calls or network traffic, which are typically flagged by traditional supply chain security tools. Furthermore, they can be designed not to trigger machine learning defenses, as they modify the underlying mechanisms that defenses rely upon or operate at a level below typical detection.

1. Low Security Awareness in ML Open Source: An analysis of the top 50 machine learning repositories on GitHub using an LLM revealed a concerning lack of discussion and implementation of critical supply chain security safeguards.

• Dependency Count: ML packages generally have a higher number of dependencies compared to traditional Linux packages, expanding their attack surface.
• Security Discussion Disparity: Introductory and example ML projects (e.g., "AI for beginners") showed more discussion around security, while fundamental ML frameworks (e.g., PyTorch, TensorFlow) had very limited engagement with supply chain security issues. Advanced projects built on these frameworks fell in between.
• Limited Critical Safeguards: Discussions predominantly focused on incidental safeguards like version pinning and typo detection (often byproducts of efforts to ensure reproducible builds), rather than critical measures such as secure authentication, code isolation, and sandboxing.
• Maintainer Resistance: A concerning signal was the rejection of security-focused pull requests by maintainers who perceived them as lacking value, highlighting a fundamental gap in security awareness and prioritization within the ML community.

These findings collectively paint a picture of an ML ecosystem that is uniquely susceptible to sophisticated supply chain attacks, necessitating a fundamental shift in security posture and development practices.

Technical Deep Dive

▶ Watch: Defining the ML ecosystem threat model (6:00)

The core of the technical vulnerability discussed in the talk stems from the dynamic and flexible nature of the Python runtime environment. Unlike compiled languages with stricter scope and object management, Python allows for significant manipulation of its runtime state, which can be exploited by malicious actors within a dependency chain. The speakers detail how this flexibility, while beneficial for rapid prototyping and development, becomes a critical security weakness in the context of supply chain attacks.

When a Python package is imported, it gains access to the entire runtime environment of the downstream application. This means a compromised dependency can perform two primary actions that enable these novel attacks:

1. Overriding Global Objects: A malicious dependency can import a core ML framework (e.g., PyTorch, TensorFlow) and then programmatically alter its globally accessible attributes or functions. For instance, if a dependency imports PyTorch before the main application, it can modify a PyTorch function's default parameters or even replace the function entirely. Subsequent calls to that function by the downstream user will then unknowingly execute the compromised version. This allows for subtle yet impactful alterations to ML models or pipelines without direct modification of the main application's source code.

1. Navigating the Stack Environment and Overriding Local Variables: More sophisticated attacks can exploit Python's introspection capabilities to navigate the call stack. This allows a compromised dependency to identify and modify local variables or parameters in the calling functions, even if those variables were not explicitly passed into the dependency's functions. This technique is particularly insidious because it bypasses conventional parameter passing and can affect code execution in ways that are extremely difficult to trace or anticipate.

Based on these fundamental Python tricks, the speakers illustrate two primary categories of attacks:

1. Injecting Vulnerabilities

This category focuses on embedding malicious logic or weakening the security posture of ML models and pipelines.

• Direct Backdoor Injection: Instead of traditional backdoor methods that require control over training data or the training process, a supply chain attacker can inject backdoors directly into the inference process. For example, a compromised dependency could modify an image processing function to detect a specific pixel pattern. If this pattern is present in an input image, the function could be altered to misclassify it or trigger an arbitrary action. For Large Language Models (LLMs), this could manifest as injecting specific prompt injection patterns into a tokenizer library. When the tokenizer processes certain input text, the malicious pattern is appended or modified, forcing the LLM to behave in an unintended way (e.g., generating harmful content, leaking sensitive information).

• Subtle Pipeline Compromise: Attackers can introduce vulnerabilities by changing seemingly innocuous parameters in ML pipeline components. A notable example provided is altering the interpolation mode used for downscaling images. If a compromised package changes the default or specified interpolation mode from a secure method (e.g., bicubic or bilinear) to nearest neighbor, this might go unnoticed by developers. However, the nearest neighbor interpolation is known to be less robust and can increase the susceptibility of a model to adversarial examples, effectively expanding the attacker's surface for future attacks without directly inserting a "malicious" function. Realizing this subtle change has security implications requires deep domain knowledge, making it a highly effective stealth attack.

2. Bypassing or Disabling Defenses

This category of attacks focuses on subverting existing security measures designed to protect ML models.

• Defeating Model Stealing Defenses: Many defenses against model stealing attacks (where an adversary reconstructs a proprietary model by querying it) rely on detecting suspicious query patterns. One such defense identifies queries that are close to the decision boundary, meaning the margin between the top two predicted classes is very small. A compromised dependency can exploit this by replacing or overriding the standard SoftMax function (or equivalent final activation layer) within the ML framework. The malicious SoftMax implementation would artificially increase the margin between the top two predicted classes for all outputs. This makes it appear as if the model is making very confident predictions, thereby circumventing margin-based model stealing detection mechanisms. From a traditional code perspective, this isn't overtly malicious; it's just a different mathematical implementation, yet its impact on security is profound.

These attacks are particularly dangerous because they operate stealthily. They don't typically involve suspicious system calls, unusual network traffic, or overt malicious code that traditional security tools are designed to detect. Furthermore, because they manipulate the fundamental components of ML frameworks, they can preemptively disable or render ineffective ML-specific defenses, creating a potent and difficult-to-detect threat vector.

Experimental Setup & Results

▶ Watch: Injecting ML backdoors and prompt injections via dependencies (8:00)

The "experimental setup" in this research primarily involved a series of analytical studies and measurements rather than traditional empirical experiments with new models or systems. The speakers leveraged Large Language Models (LLMs) for code analysis and performed quantitative assessments of dependency chains to understand the current state of supply chain security awareness and practices within the open-source ML community.

Dependency Analysis

• Methodology: The researchers conducted a simple measurement of the number of direct and transitive dependencies for the top 50 machine learning packages and the top 50 non-machine learning (Linux-focused) packages hosted on GitHub. This was likely done by analyzing requirements.txt, setup.py, or similar dependency manifests.
• Headline Result: They found that, overall, machine learning packages have significantly more dependencies compared to their non-ML counterparts. This quantitative finding directly supports their argument that ML frameworks inherently possess a larger software supply chain attack surface.

GitHub Repository Security Analysis via LLM

• Methodology: An LLM was employed to analyze the issues and pull requests (PRs) of the top 50 machine learning repositories on GitHub. The LLM was tasked with identifying and classifying discussions and proposed changes relevant to supply chain security. This automated analysis allowed for a broad survey of community engagement with security topics.
• Key Findings on Security Awareness:
• Security Discussion Disparity: The analysis revealed a clear pattern:
• Most Secured (perceived): Introductory and example projects (e.g., "AI for beginners," "100 days of machine learning code") showed the most discussion and activity related to supply chain security. This is likely because these projects are often designed with reproducibility and clear dependency management in mind, which incidentally benefits security.
• Least Secured: Fundamental machine learning frameworks (e.g., PyTorch, TensorFlow, Scikit-learn) exhibited very limited discussion or proactive measures concerning supply chain security. This is a critical concern, as these foundational libraries underpin the entire ML ecosystem.
• Intermediate: Advanced projects built upon these frameworks displayed an intermediate level of security discussion.
• Limited Critical Safeguards: The types of safeguards most frequently discussed in ML repositories were predominantly version pinning and typo detection. While useful, these are often byproducts of efforts to ensure reproducible builds rather than explicit, dedicated supply chain security initiatives. Critical safeguards such as secure authentication mechanisms, code isolation, and sandboxing were found to be very limited in discussion and implementation.
• Comparison to Non-ML Community: A similar analysis was performed for non-machine learning GitHub projects. The pattern of security awareness was found to be broadly similar across both ML and non-ML communities. However, the speakers emphasized that given the far greater complexity and unique attack vectors in ML supply chains, the ML community should possess a significantly higher level of security awareness and proactive measures.
• Rejected Security Pull Requests: A highly concerning observation was the identification of instances where contributors attempted to submit security-focused pull requests, only for these PRs to be rejected by maintainers. The stated reason for rejection was often a perception that the security contribution lacked value or was not a priority, indicating a fundamental gap in security prioritization within some parts of the ML open-source community.

These results collectively highlight a significant gap between the escalating threat of ML supply chain attacks and the current state of security awareness and practice within the open-source machine learning ecosystem. The low emphasis on critical safeguards and even resistance to security contributions underscore the urgent need for a cultural shift.

Practical Implications

▶ Watch: Subtle pipeline compromise: insecure image interpolation example (9:30)

The findings from this research carry profound practical implications for a wide range of stakeholders in the machine learning ecosystem, from individual practitioners to large-scale infrastructure teams. The core message is that the unique vulnerabilities of ML supply chains necessitate a departure from traditional security mindsets and the adoption of more rigorous, ML-specific security practices.

For Practitioners (ML Developers, Data Scientists)

• Extreme Dependency Vigilance: The most immediate implication is the need for extreme caution when selecting and integrating open-source packages. Developers must go beyond simply checking for functionality and actively scrutinize the security posture of every dependency. This includes:
• Maintainer Activity: Prioritize packages with active, well-staffed maintenance teams. Avoid packages solely maintained by one or two individuals, as these are prime targets for takeover attempts (as seen with XZ Utils).
• Community Engagement: Look for signs of robust community engagement, security discussions, and a history of promptly addressing vulnerabilities.
• Dependency Auditing: Regularly audit the entire dependency tree of projects, not just direct dependencies, to identify potential weak links. Tools that visualize and analyze dependency graphs can be invaluable.
• Understanding Python's Dynamics: Developers should gain a deeper understanding of Python's dynamic runtime and how objects can be overridden. This knowledge empowers them to identify unusual behavior or potential injection points in their code or imported libraries.
• Reproducible Builds with Security in Mind: While version pinning is crucial for reproducibility, it also serves as a baseline security measure by preventing unexpected updates. However, it's not a complete solution; even pinned versions can harbor vulnerabilities.

For Infrastructure Teams and Model Builders

• Holistic Attack Surface Analysis: Infrastructure teams deploying ML models must recognize that the attack surface extends beyond traditional network and system boundaries. It includes the entire software supply chain and the intrinsic vulnerabilities of ML models themselves.
• Enhanced Runtime Monitoring: Develop and deploy advanced monitoring solutions capable of detecting subtle anomalies in the Python runtime environment. This includes monitoring for unexpected object overrides, modifications to core ML framework functions (e.g., SoftMax), or changes in pipeline parameters (e.g., interpolation modes) that could indicate a supply chain compromise.
• Code Isolation and Sandboxing: Implement stringent code isolation and sandboxing mechanisms for ML workloads, especially when running code from third-party or untrusted sources (e.g., Hugging Face models). This can limit the blast radius of a compromised dependency, preventing it from accessing or modifying critical system components or other models.
• Static and Dynamic Analysis for ML: Invest in tools for static and dynamic analysis that are specifically designed to detect ML-specific vulnerabilities within code and models, rather than relying solely on generic software security scanners.
• Encouraging Safer Languages: While Python's ecosystem is dominant, for highly critical components or where performance and security are paramount, infrastructure teams might consider encouraging the use of compiled languages (like Rust, Go, C++) with stronger type systems, memory safety, and stricter module boundaries. This reduces the risk of dynamic runtime manipulation.

Tradeoffs and Limitations

• Ease of Use vs. Security: Python's dynamic nature and extensive open-source ecosystem are precisely what make it so popular and productive for ML development. Imposing stricter security controls (e.g., sandboxing, less dynamic behavior) often comes at the cost of development speed, flexibility, and ease of integration, creating a significant tradeoff.
• Performance Overhead: Runtime monitoring, extensive sandboxing, and using compiled languages can introduce performance overhead, which is a critical consideration for high-throughput ML inference and training systems.
• Cultural Shift: The research highlighted a lack of security awareness and even resistance among some open-source maintainers. Overcoming this requires a significant cultural shift, which is often the hardest aspect of implementing widespread security improvements.
• Complexity of ML-Specific Defenses: Developing robust, ML-specific defenses against these new classes of attacks is inherently complex, requiring deep expertise in both software security and machine learning.

In essence, the practical implication is that security in ML is no longer a peripheral concern but a fundamental design principle that must be integrated into every stage of the development and deployment lifecycle, from dependency selection to runtime monitoring, with a keen eye on Python's unique vulnerabilities.

Key Takeaways

• ML Supply Chains are Uniquely Vulnerable: Machine learning frameworks have a significantly larger and more complex attack surface due to their extensive dependency chains and inherent ML-specific vulnerabilities (e.g., adversarial examples, prompt injection).
• Python Runtime is a Critical Weakness: The dynamic nature of Python allows compromised dependencies to easily override global and local objects in downstream applications, bypassing traditional security checks.
• New Class of Attacks Emerges: Attackers can now inject vulnerabilities (e.g., backdoors in inference, subtle pipeline parameter changes like interpolation modes) or disable existing defenses (e.g., subverting model stealing detection by overriding SoftMax) without triggering traditional software or ML-specific security alerts.
• Low Security Awareness in Open Source ML: There is a concerning lack of discussion and implementation of critical supply chain safeguards (e.g., code isolation, sandboxing) in fundamental ML open-source projects, with some maintainers even rejecting security-focused contributions.
• Urgent Need for Proactive Measures: Practitioners must exercise extreme vigilance in dependency selection, prioritize well-maintained packages, and consider stricter isolation/sandboxing. Infrastructure teams should implement enhanced runtime monitoring and explore more secure language paradigms for critical components.
• Cultural Shift is Essential: Raising security awareness for both contributors and maintainers across the ML community is paramount to foster a more secure development ecosystem.

About the Speaker(s)

The talk "Supply-Chain Attacks in Machine Learning Frameworks" was presented by Yue Gao, Ilia Shumailov, and Kassem Fawaz. Based on the information provided in the talk bundle and transcript, they are researchers actively involved in investigating and presenting on the critical area of machine learning security, particularly focusing on the vulnerabilities within the open-source software supply chain that underpins modern ML development. Their work presented at MLSys 2025 highlights their expertise in both systems security and machine learning.