Deep dive into the abuse of DL APIs to create malicious AI models and how to detect them

Deep dive into the abuse of DL APIs to create malicious AI models and how to detect them

Speakers: Mohamed Nabeel, Palo Alto Networks; Alex Starov, Palo Alto Networks

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=NYmx6i3W4

Overview

In an era increasingly dominated by artificial intelligence, the security of Deep Learning (DL) models has become a critical concern. This talk, delivered by Mohamed Nabeel and Alex Starov from Palo Alto Networks, delves into a novel and insidious method of injecting malware into these models by exploiting legitimate but undocumented functionalities within popular AI frameworks. The presentation highlights a significant blind spot in current security scanning technologies, demonstrating how attackers can weaponize pre-trained models distributed through model hubs like Hugging Face and TensorFlow Hub.

The core of the research exposes how TensorFlow models, even in their serialized format, are not merely data but executable computational graphs containing hidden functions with powerful side effects. By chaining these obscure functions, attackers can construct sophisticated malware that performs actions like data exfiltration or dropping malicious payloads, all while evading conventional signature-based detection mechanisms. The speakers illustrate these concepts with two compelling proof-of-concept attacks, underscoring the urgent need for a paradigm shift towards semantic scanning and robust defense-in-depth strategies within the AI supply chain.

This research is particularly pertinent given the burgeoning reliance on pre-trained models across industries. As organizations increasingly leverage these shared resources to accelerate AI development, understanding and mitigating the risks associated with untrusted models becomes paramount. Nabeel and Starov's work provides crucial insights into the evolving AI threat landscape, equipping defenders with knowledge to anticipate and counter these advanced persistent threats targeting the very foundation of modern AI applications.

Background

▶ Watch: Introduction: Malware in deep learning models (0:00)

The modern AI supply chain is characterized by a growing dependency on shared resources, primarily pre-trained models. Building complex AI models from scratch is an incredibly resource-intensive endeavor, both in terms of computational power and specialized expertise. A recent report by Gartner indicates that a staggering 70% of organizations will opt to use pre-trained models in their workloads moving forward. This trend has fostered a vibrant ecosystem of model hubs, platforms like Hugging Face and TensorFlow Hub, which function akin to software package repositories such as PyPI for Python or npm for Node.js. These hubs allow developers to upload, share, and consume state-of-the-art AI models, democratizing access to powerful AI capabilities.

The typical lifecycle within this hub economy involves several key steps: a model is designed, trained, and then saved into a serialized format – essentially a compressed file containing the model's architecture, weights and biases (parameters), and crucially, executable functions. This serialized model is then uploaded to a chosen hub. Users can download these models, deserialize them, and load them into their environments for direct use or further fine-tuning. The critical insight, as emphasized by the speakers, is that these models are not just static data; they are dynamic entities containing executable code. This raises a fundamental security question: "Can we trust these models, and what can we do to use them safely?"

The threat landscape targeting AI models has been evolving. Previous attacks can be broadly categorized into two types: behavioral attacks and carrier attacks. Behavioral attacks, commonly known as adversarial machine learning (AML), aim to manipulate a model's decision-making process. Examples include:

• Model poisoning attacks, where attackers inject malicious or biased data into the training set, leading to compromised or skewed model outputs. This is particularly problematic for large models that ingest vast amounts of internet data without sufficient trustworthiness vetting.
• Model evasion attacks, which occur at inference time. Attackers subtly alter input data (e.g., adding stickers to a stop sign) to trick the model into making incorrect classifications, potentially with fatal consequences in real-world applications.
• Prompt injection attacks, a growing concern with large generative AI models (like GPT-4). Attackers inject malicious prompts or instructions to bypass safety filters, extract sensitive information, or coerce the model into unintended behaviors, effectively "jailbreaking" it.

In contrast, carrier attacks, the primary focus of this talk, leverage AI models as a vessel to hide and deliver malware. Unlike behavioral attacks, carrier attacks typically do not aim to alter the model's decision-making but rather to execute arbitrary code on the user's system. The speakers outlined three historically observed types of carrier attacks:

• Pickle Bomb: The pickle Python serialization format, once common for saving models, is inherently insecure. It allows the serialization of executable code, specifically through its reduce function, which can be abused to invoke arbitrary system calls. Malicious code injected via a pickle bomb would trigger upon the model being loaded using pickle.load. While effective, the pickle format is now largely discouraged for model serialization, and most model scanners can detect these vulnerabilities.
• Custom Layer Abusers: Frameworks like Keras provide flexibility for users to define custom layers, sometimes referred to as lambda layers, which can embed Python code directly within the model architecture. Attackers exploited this by inserting system calls into these layers, which would then execute during the model's inference phase. Keras has since deprecated lambda layers from being serialized, and this attack vector has been largely mitigated.
• Hiding in Weights: A more subtle technique involved embedding malware bits within the least significant bits (LSBs) of the model's floating-point parameters (weights). This manipulation was designed to be imperceptible to the model's performance or accuracy. However, this attack required an out-of-band triggering mechanism to reassemble and launch the malware, along with white-box access to the model parameters, making it less practical and consequently not widely adopted.

With these prior attack vectors largely addressed or impractical, the talk then pivots to a new class of carrier attacks designed to exploit a novel blind spot in current detection methodologies.

Key Findings

▶ Watch: Evolving AI model threat landscape: behavioral vs. carrier attacks (2:10)

The central premise of Nabeel and Starov's research revolves around a critical observation: TensorFlow models, even when saved and serialized, are fundamentally executable computational graphs. Within these graphs, there exist numerous legitimate but hidden functions that are not intended for direct user interaction and are often undocumented. The key finding is that these hidden functions can possess powerful latent functionality and can be chained together by an attacker to perform malicious operations, effectively bypassing existing security scanners.

The speakers identified that any form of effective malware requires only four fundamental building blocks:

1. File read: To steal sensitive data from the local system.
2. File write: To drop payloads, achieve persistence, or modify system files.
3. Network send: To exfiltrate stolen data to an attacker-controlled endpoint.
4. Network receive: To fetch commands or additional payloads from a C2 server.

The challenge for attackers, and consequently the focus of the researchers, was to identify hidden, serializable functions within the vast TensorFlow API that could fulfill these four building block requirements, without relying on commonly documented functions that scanners would easily flag. This task is analogous to "finding needles in a haystack," given the immense size and constant evolution of the TensorFlow library. The difficulty is compounded by the fact that these serializable functions are often poorly documented, and their co-functionality with side effects is rarely explicitly stated.

To overcome this, the research team developed a sophisticated approach utilizing an LLM-based agent alongside a suite of custom-built tools. This system was designed to systematically identify hidden serializable core functions within the TensorFlow API that possessed the latent capabilities for file I/O and network communication. The success of this methodology in uncovering these evasive functions constitutes a significant key finding, demonstrating a novel and potent attack surface within widely adopted AI frameworks.

Technical Deep Dive

▶ Watch: New attack: Abusing hidden TensorFlow functions with blind spots (8:05)

The technical core of this research lies in the exploitation of TensorFlow's internal architecture, specifically its computational graph nature and the existence of legitimate but obscure functions. The speakers highlight the distinction between easily detectable functions and their newly discovered counterparts. For instance, a function like TF.io.IOFileRead is a well-documented file reading function. If an attacker were to include this directly in a model, conventional scanners employing syntactic or signature-based detection would immediately flag it as suspicious.

However, the researchers discovered functions like TF.raw_ops.FixedLengthRecordDataset, which, despite its innocuous name and internal purpose, possesses the latent capability to read files from the system. This function is "hidden" in the sense that it is not documented for regular user consumption and is typically used for internal data processing within TensorFlow. The critical insight is that model scanners have a blind spot for these kinds of hidden, latent functionalities.

The LLM-based agent and custom tools developed by the researchers were instrumental in systematically trawling through the TensorFlow API to identify functions corresponding to the four malicious building blocks:

• File Read: Identified hidden serializable functions capable of accessing and reading local files.
• File Write: Identified hidden serializable functions that could write data to the local file system, useful for dropping payloads or achieving persistence. An example mentioned later is print_v2.
• Network Send: Identified hidden serializable functions that could establish network connections and exfiltrate data.
• Network Receive: Identified hidden serializable functions that could receive data or commands over a network, enabling communication with a C2 server.

These identified functions are not designed for these malicious purposes; rather, their internal mechanics, when chained together, can be co-opted for such actions. The evasiveness of this attack stems from the fact that these functions are considered "legitimate" by the framework itself, making them difficult for static analysis tools to differentiate from benign usage without deep semantic understanding. The full technical details of the LLM-based discovery system are referred to in their paper, but the presentation clearly establishes the existence and exploitability of these hidden primitives within TensorFlow.

Demo / Proof of Concept

▶ Watch: Proof of concept: Stealthy data exfiltration and malware dropper (12:00)

The talk presented two compelling proof-of-concept (PoC) attacks, demonstrating the practical feasibility of abusing these hidden TensorFlow functions. Both PoCs were uploaded to Hugging Face, a prominent model hub, and successfully evaded its security scanning, being verified as "good models to use." This starkly illustrated the blind spot of existing detection mechanisms.

1. Stealthy Data Exfiltration

The first PoC showcased a stealthy data exfiltration attack. The objective was to read sensitive information from the victim's system and transmit it to an attacker-controlled endpoint without detection.

• File Read: The attack utilized a specifically identified hidden serializable read function to access and retrieve sensitive data from the local file system. The transcript notes this could involve "sensitive information from the system."
• Network Send: Once the data was loaded into memory, another hidden serializable send function was employed to exfiltrate this data to a remote server controlled by the attacker.

The speakers noted that while their example showed clear exfiltration, real-world attackers would likely use even more evasive techniques, such as encoding the payloads to obscure their content and leveraging seemingly trustworthy endpoints to bypass URL reputation systems. The successful upload and lack of detection on Hugging Face underscored the attack's potency and the vulnerability of current model distribution platforms.

2. Malware Dropper

The second PoC demonstrated a malware dropper functionality. This attack aimed to download a malicious payload from an attacker's C2 server and save it onto the victim's local machine, achieving persistence or setting the stage for further compromise.

• Network Receive: The attack used a hidden serializable receive function to establish communication with the attacker's C2 server and download the malicious payload.
• File Write: Upon receiving the payload, a hidden serializable write function, specifically exemplified by print_v2, was used to save the downloaded malware onto the local system.

Both PoCs highlight the ease with which these powerful, yet obscure, functions can be combined to form functional malware, once they have been identified. The fact that these malicious models could be hosted on and downloaded from widely used model hubs without raising any alarms represents a critical security gap.

Defensive Implications

▶ Watch: Key takeaways and responsible disclosure to vendors (15:00)

The research by Nabeel and Starov exposes a significant vulnerability in the security posture of the modern AI supply chain, particularly regarding the distribution of pre-trained models. The primary defensive implication is that existing model scanners are fundamentally inadequate against this new class of attacks. Their failure stems from several key limitations:

• Syntactic and Signature-Based Detection: Current scanners predominantly rely on recognizing known malicious patterns or specific function calls. They are effective against "pickle bombs" or deprecated "lambda layers" because these involve easily identifiable vulnerable code structures or explicit system calls. However, they are blind to the legitimate, hidden functions exploited in this new attack.
• Lack of Semantic Understanding: Scanners lack the ability to comprehend the true meaning or potential side effects of functions, especially those not publicly documented or intended for internal framework use. They cannot infer that a function designed for internal data processing might have the latent capability to read arbitrary files.
• Absence of Context and Flow Analysis: Attacks, as demonstrated by the PoCs, often involve chaining multiple core functions (e.g., read + send, receive + write). Existing scanners typically analyze functions in isolation and fail to understand the malicious intent when legitimate functions are combined in a specific computational flow.

To counter these sophisticated threats, the speakers advocate for a paradigm shift in detection methodologies towards semantic scanning. This approach would involve building tools capable of understanding the computational graph's context and the actual meaning and potential side effects of functions, even the hidden ones. The paper outlines how an LLM-based code understanding agent could be integrated into scanning processes to analyze models more intelligently, identifying suspicious combinations and sequences of operations that signify malicious intent.

Based on their findings, the speakers provided critical defensive implications and recommendations for organizations:

1. Treat All Downloaded Models as Untrusted Executable Code: This is the foundational principle. Models should no longer be viewed as inert data files but as potentially active and malicious software components.
2. Embrace Semantic Scanning: While signature-based scanning remains a necessary first step, it must be augmented with advanced semantic analysis. Tools need to be developed that can interpret the intent and potential side effects of all functions within a model, including the obscure ones.
3. Implement Defense-in-Depth: A multi-layered security approach is essential. This includes:

• Comprehensive Scanning: Every downloaded model should undergo thorough scanning, combining both signature-based and semantic analysis.
• Sandboxing: Models should be run in isolated, controlled sandbox environments to detect any evasive techniques that might bypass static scanning. This dynamic analysis can catch runtime malicious behavior.

In a move of responsible disclosure, the researchers contacted both Hugging Face and Google (TensorFlow Hub). Hugging Face acknowledged the issue and stated they are further investigating. Google also acknowledged the possibility of abuse but suggested that model scanners should "step up" to prevent these types of attacks, implicitly confirming the vulnerability and placing the onus on the security community to develop more advanced detection capabilities. This underscores the urgency for the industry to adapt its security practices to the evolving nature of AI threats.

Key Takeaways

• Models are Code, Not Just Data: Treat every downloaded AI model as potentially untrusted executable code, not merely static data.
• New Evasive Carrier Attacks: A novel class of carrier attacks exploits legitimate but hidden and undocumented functions within AI frameworks like TensorFlow, enabling actions like data exfiltration and malware dropping.
• Blind Spots in Current Scanners: Existing security scanners primarily rely on syntactic and signature-based detection, lacking the semantic understanding and context analysis necessary to detect these sophisticated attacks.
• Necessity of Semantic Scanning: Future detection mechanisms must incorporate semantic understanding, potentially leveraging LLM-based agents, to analyze the true intent and side effects of functions within a computational graph.
• Defense-in-Depth is Crucial: A multi-layered security strategy, combining robust static scanning with dynamic analysis in sandboxed environments, is essential to mitigate these advanced threats.
• Industry Acknowledgment: Major model hub providers like Hugging Face and Google have acknowledged the presented vulnerabilities, highlighting the critical need for improved security measures across the AI supply chain.

About the Speaker(s)

Mohamed Nabeel and Alex Starov are researchers from Palo Alto Networks. They collaborated on this research, contributing their expertise to uncover and demonstrate the novel methods of abusing deep learning APIs for malicious purposes and to propose solutions for their detection and mitigation.