Reversing Large Deep Learning AI Models - Yashodhan Vivek Mandke

Reversing Large Deep Learning AI Models - Yashodhan Vivek Mandke

Speakers: Yashodhan Vivek Mandke, PhD Student in Satellite Security, Researcher

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=OkCd2KSvBA4

Overview

In this insightful Nullcon talk, Yashodhan Vivek Mandke delves into the critical, yet often overlooked, domain of reversing deep learning (DL) models. While the current discourse in AI security frequently centers on Large Language Models (LLMs) and peripheral attacks like prompt injection, Mandke argues that true security and robust defense necessitate a deeper understanding of the model's internal architecture and mathematical foundations. His presentation illuminates methods for dissecting complex AI models, exposing their core components, and identifying vulnerabilities that lie beyond superficial interactions.

Mandke, a PhD student specializing in satellite security with a diverse research background spanning IoT, space, and AI product development, highlights that AI models are not merely applications but intricate systems governed by mathematical and statistical rules. By treating AI as a system, security professionals can move beyond surface-level threat modeling to understand the model's purpose and its inherent assets: data, algorithms, and the model itself. This talk serves as a crucial guide for researchers and defenders seeking to fortify AI systems from the inside out, providing a framework for analyzing model integrity, confidentiality, and availability (ICA) at a foundational level.

The presentation introduces novel approaches, including the surprising analogy of AI models to control systems, to assess their stability and pinpoint weaknesses. Mandke demonstrates how to leverage tools and techniques to extract sensitive information, such as model weights, architecture details, and even mathematical functions, which can be exploited for sophisticated attacks or leveraged for more effective defensive strategies. This deep dive into the internal mechanics of AI models offers a paradigm shift in AI security, urging the community to explore and secure the complex mathematical heart of these increasingly pervasive technologies.

Background

▶ Watch: Introduction to reversing deep learning models and AI security. (0:00)

The journey of Artificial Intelligence has evolved dramatically, from simple regression problems to the sophisticated multimodal architectures prevalent today. However, the foundational principles remain consistent. At the core of any AI development project is the model, which dictates how data is processed and insights are generated. The lifecycle involves gathering requirements, sourcing diverse data (from IoT sensors, websites, etc.), rigorous data cleaning and feature selection, model training, and finally, deployment, often managed through MLOps pipelines that include continuous retraining. This centrality of the model underscores its critical role in determining both the potential impact of an attack and its resilience.

Traditional IT and IoT systems prioritize Confidentiality, Integrity, and Availability (CIA), with confidentiality often being dominant in IT and availability in Operational Technology (OT) systems. Mandke asserts that for AI systems, the focus shifts to Integrity, Confidentiality, and Availability (ICA), with integrity becoming paramount. This is because AI models, when integrated across various domains, are not simple applications but complex systems with unique mathematical and statistical underpinnings. Securing AI therefore demands adherence to these underlying mathematical rules. The fundamental assets of any AI system are its data, algorithms, and the model itself. Threat modeling for AI must transcend surface-level concerns like identity and authentication, instead focusing on the model's specific purpose and addressing ICA dominance accordingly.

To reverse-engineer AI models effectively, one must understand their common formats. Mandke introduces several key formats:

• ONNX (Open Neural Network Exchange): An open-source, interoperable format designed for deploying models across different platforms, from cloud servers to edge devices and microcontrollers. ONNX models can even be converted into C code for ARM-based microcontrollers using the CMSIS-NN library, allowing inference on resource-constrained hardware.
• SafeTensors: Developed by Hugging Face, this format specifically addresses serialization vulnerabilities, offering enhanced security. It is widely adopted by large language models, including many deepfake models hosted on Hugging Face, which currently number around 52. Tensors are fundamental mathematical structures with dimensions, generalizing concepts like scalars, vectors, and matrices.
• PyTorch (pth or pt): A tensor-based framework developed by Meta, commonly using .pth or .pt file extensions for saved models.
• TensorFlow Lite (tflite): Google's lightweight format for deploying TensorFlow models on mobile and edge devices.

These formats encapsulate the intricate mathematical structures and learned parameters of deep learning models, making them the primary targets for reverse engineering efforts.

Key Findings

▶ Watch: Understanding AI security: ICA dominance and system challenges. (2:55)

Mandke's research yielded several crucial findings regarding the reversibility and inherent vulnerabilities of large deep learning AI models:

1. Accessibility of Internal Architecture and Weights: Contrary to the perception of AI models as opaque black boxes, Mandke demonstrated that tools like Netron (an open-source model viewer designed for debugging) can effectively break the confidentiality of these models. Netron allows an attacker or researcher to visualize the entire neural network architecture, including individual layers, operations (e.g., convolution, ReLU, pooling), and critically, the weights associated with each neuron. These weights, which determine the model's intelligence and predictive strength, can often be downloaded directly as Numpy files, essentially allowing an attacker to copy the core intellectual property and functionality of a model.

1. Sparsity as a Critical Vulnerability Indicator: A significant discovery was the role of sparsity (the percentage of zero values within a model's tensors or matrices) in indicating a model's susceptibility to attack. Mandke highlighted that models with 0% sparsity, meaning they are "dense matrices" with no zero values, are deeply connected. In such models, even minor changes or "perturbations" to the weights can propagate widely throughout the architecture, making them highly vulnerable to weight poisoning and manipulation. He explicitly noted that many large deep learning models, including some GPT architectures, exhibit 0% sparsity, posing a substantial threat.

1. AI Models as Control Systems for Stability Analysis: A novel and groundbreaking finding was Mandke's conceptualization of AI models as feedback control systems. By applying classical control theory concepts like Root Locus and Bode Plot analysis, he demonstrated a method to assess a model's stability. This approach allows researchers to identify specific points or frequency ranges where a model becomes unstable or highly sensitive to changes. For instance, Bode plots can reveal whether a model's weights are more vulnerable to high-frequency (e.g., sharp edges in images) or low-frequency (e.g., gradual color shifts) perturbations, enabling targeted and efficient attacks rather than prolonged, resource-intensive brute-forcing.

1. Layer-by-Layer Attack Planning: The ability to analyze the stability of individual layers within a deep neural network was another key finding. By examining the weights and their stability characteristics layer by layer, attackers can precisely identify the most vulnerable layers, allowing for a more strategic and time-efficient attack plan. This granular analysis provides a pathway to optimize attack vectors and minimize computational resources.

1. Specific Mathematical Functions as Attack Surfaces: Mandke pointed out that fundamental mathematical operations within DL models, such as conv2D (convolution) and matmul (matrix multiplication), can themselves be attack surfaces. He noted that conv2D has reported CVEs, and matmul operations are susceptible to Denial of Service (DoS) attacks if manipulated with incorrect dimensions, highlighting vulnerabilities in the underlying mathematical libraries.

These findings collectively underscore the necessity of moving beyond perimeter security in AI and focusing on the internal integrity and architecture of deep learning models to build truly resilient systems.

Technical Deep Dive

▶ Watch: Overview of common AI/DL model formats for reversing. (4:45)

The technical core of Mandke's presentation revolves around dissecting the internal workings of deep learning models and applying unconventional analytical frameworks to uncover vulnerabilities. He exemplified this using GoogleNet, also known as Inception, a convolutional neural network (CNN) architecture from approximately 8-9 years ago, foundational for computer vision tasks like object detection and recognition. GoogleNet, with its 22 layers, comprises combinations of convolutional (C) and pooling (P) layers. These layers perform mathematical operations, both linear (multiplication, addition) and non-linear (activation functions like ReLU), to learn patterns and build intelligence from input data.

Mandke demonstrated the use of Netron, an open-source tool, to visualize and inspect a GoogleNet model. Upon loading the ONNX-formatted GoogleNet model into Netron, the entire architecture becomes navigable. Key insights derived from this visualization include:

• Input Dimensions: For GoogleNet, the input is typically 1 3 224 * 224. This translates to one sample, three color channels (RGB), and an image resolution of 224x224 pixels. Understanding these input parameters is crucial for crafting targeted attacks.
• Layer-Specific Operations: Netron reveals the sequence of operations within each layer, such as conv2D, ReLU (Rectified Linear Unit), and Max Pooling. Mandke noted that conv2D operations have reported CVEs, making them potential targets for fuzzing and exploitation.
• Weight Extraction: The most critical revelation is the accessibility of weights (W). Weights are fundamental hyperparameters in any neural network, determining its predictive strength and intelligence. Netron allows an attacker to inspect the values of these weights, which are typically represented as multi-dimensional tensors. For instance, a four-dimensional tensor of weights signifies critical features captured across four dimensions. Crucially, these weights can be downloaded as Numpy files, enabling an attacker to copy the entire learned intelligence of a model. This poses a significant threat, as a substantial portion of AI development effort goes into training these weights.

Beyond architectural inspection, Mandke introduced a groundbreaking analogy: treating AI models as feedback control systems. This allows for the application of classical control theory to analyze model stability, a concept typically used in engineering to ensure system reliability.

• Stability Analysis: Just as control systems need to be stable, AI models need to make reliable predictions. An unstable model is inherently easier to attack.
• Root Locus: This control system concept helps identify the roots of a system, showing where it becomes unstable or stable. In the context of AI, it reveals points of high sensitivity to changes in weights, indicating potential attack vectors.
• Bode Plot: This analysis tool provides insights into a system's magnitude and phase response across different frequencies. Mandke demonstrated how a Bode plot can differentiate between an original model (blue line) and an adversary-attacked model (red line). The crossover points indicate frequencies where the model is vulnerable. For images, this translates to high-frequency components (e.g., sharp borders or sudden color changes) versus low-frequency components (e.g., gradual color gradients). If a model's Bode plot shows vulnerability at high frequencies, an attacker knows to focus on perturbing those specific image features. Conversely, lower magnitude responses indicate higher attack probability.
• Layer-by-Layer Stability: This approach extends the control system analogy to individual layers. By downloading the Numpy files for specific layer weights, their stability can be analyzed. Mandke showed a graph where the first layer's weights exhibited vibrations, indicating susceptibility to high-frequency attacks, thus allowing for precise attack planning.

Mandke also highlighted the vulnerability of sparsity. When a model's tensors have 0% sparsity (meaning all values are non-zero), it indicates a "dense matrix." Such a dense network implies deep connectivity, where even a minor perturbation to a single weight can propagate extensively, leading to significant changes in model behavior. This inherent characteristic of many state-of-the-art models, including GPTs, makes them highly susceptible to weight poisoning attacks. Furthermore, functions like matmul (matrix multiplication) are dimension-sensitive; feeding incorrect dimensions can lead to errors, potentially creating a Denial of Service (DoS) attack.

This detailed technical analysis, combining direct architectural inspection with an innovative control systems framework, provides a comprehensive methodology for understanding, identifying, and exploiting deep learning model vulnerabilities at their mathematical core.

Demo / Proof of Concept

▶ Watch: ONNX format: open-source interoperability and deployment. (5:00)

During the talk, Mandke provided a compelling demonstration of how to reverse-engineer and analyze a deep learning model, along with a conceptual proof of concept for perturbing its weights.

The primary demonstration involved using the Netron tool to inspect the GoogleNet (Inception) model. Mandke loaded the ONNX-formatted model into Netron, showcasing its ability to:

1. Visualize the Entire Architecture: The audience could see the complex network of layers, including convolutional, pooling, and activation functions, laid out graphically.
2. Identify Input Parameters: He highlighted the input tensor dimensions, such as 1 3 224 * 224, explaining that this represents one sample, three RGB channels, and a 224x224 pixel image size.
3. Inspect Tensor Properties and Weights: Mandke navigated to specific convolutional layers and selected the W (weights) tensor. He then revealed the actual numerical values of these weights, emphasizing their multi-dimensional (e.g., four-dimensional) tensor structure. Crucially, he showed that Netron allows these weights to be downloaded as Numpy files, illustrating the ease with which the core intelligence of a model can be extracted.

For the proof of concept regarding attacks, Mandke described and showed outputs related to weight perturbation:

• Manipulating Weights: He explained that by identifying susceptible weights (e.g., from stability analysis), an attacker could make minor, targeted changes to their values. He displayed a comparison of initial weights versus perturbed weights, demonstrating that even "really minor" differences in values could lead to significant changes in model behavior, especially in dense matrices (0% sparsity) and when targeting high-frequency components.
• Attack Visualization: Mandke presented an output showing the effects of an attack on an image. The "dark areas" in different image channels (channel one, channel two, channel three) visually represented where the attack was happening, indicating how the model's processing of visual features was being manipulated. This illustrated that the attack was not a superficial prompt injection but an internal corruption affecting how the model "sees" and interprets data.
• New Tools for Fuzzing and Poisoning: To further practicalize his research, Mandke announced two new open-source repositories:
• Tensor: Designed for fuzzing SafeTensors models.
• Torture: Focused on fuzzing and poisoning PyTorch models.

These tools provide concrete mechanisms for researchers and attackers to apply the principles discussed in the talk, enabling systematic testing and exploitation of model vulnerabilities at a deeper level.

While the full execution of these attacks was not performed live due to time and computational constraints, the combination of architectural visualization, weight extraction, and the conceptual demonstration of perturbation effects, coupled with the introduction of practical tools, effectively conveyed the feasibility and impact of reversing and attacking deep learning models from within.

Defensive Implications

▶ Watch: PyTorch and TensorFlow Lite for edge devices. (8:00)

The insights presented by Yashodhan Vivek Mandke carry profound implications for strengthening the defensive posture of AI systems. Moving beyond the prevalent focus on perimeter-level attacks like prompt injection, Mandke's work emphasizes the critical need to secure the core mathematical foundations of deep learning models.

1. Prioritize Model File Integrity and Confidentiality: The ease with which tools like Netron can extract model architectures and weights (often as Numpy files) underscores a fundamental vulnerability. Organizations must treat trained model files (ONNX, SafeTensors, PyTorch .pth, TensorFlow Lite) as highly sensitive assets. This requires:

• Strict Access Control: Limiting who can access, download, or inspect deployed model files.
• Encryption at Rest and in Transit: Ensuring model files are encrypted wherever they are stored or moved.
• Integrity Verification: Implementing robust cryptographic hashing and digital signatures for model files to detect any unauthorized tampering or alteration of weights. Any deviation should trigger alerts and potentially model rollback.

1. Address Sparsity Vulnerabilities: The finding that models with 0% sparsity (dense matrices) are highly susceptible to minor weight perturbations is a significant concern, especially as many large, advanced models exhibit this characteristic. Defenders should:

• Assess Model Sparsity: Incorporate sparsity analysis into model evaluation pipelines to understand inherent risks.
• Explore Sparsification Techniques: Research and implement techniques to introduce controlled sparsity into models without significantly degrading performance, potentially increasing resilience against targeted weight poisoning.
• Robustness Training: Employ adversarial training methods that specifically account for small, high-frequency perturbations to model weights, making the model more robust to such attacks.

1. Implement Control System-Based Monitoring: Mandke's novel approach of viewing AI models as control systems offers a new paradigm for defense. Defenders can leverage this by:

• Stability Monitoring: Develop tools and methodologies to continuously monitor the "stability" of deployed AI models, potentially adapting Root Locus and Bode Plot analyses. Unexpected shifts in stability metrics could indicate an ongoing attack or anomalous behavior.
• Frequency-Based Anomaly Detection: Use frequency domain analysis (as suggested by Bode plots) to detect perturbations targeting specific frequency components of input data (e.g., high-frequency image features). This could help identify subtle adversarial examples that exploit model weaknesses.

1. Harden Core Mathematical Operations: Vulnerabilities identified in fundamental operations like conv2D (with reported CVEs) and matmul (susceptible to DoS via dimension manipulation) highlight the need for:

• Secure Library Usage: Prioritize the use of security-hardened and regularly audited mathematical libraries for deep learning frameworks.
• Input Validation: Implement stringent validation on input tensor dimensions and data types, especially for functions like matmul, to prevent DoS attacks.
• Fuzzing Internal Components: Proactively fuzz core mathematical functions within models using tools like Mandke's "Tensor" and "Torture" to uncover and patch vulnerabilities before they are exploited.

1. Secure the MLOps Pipeline End-to-End: Model integrity can be compromised at any stage, from data ingestion to deployment. Defenders must:

• Secure Training Data: Protect against data poisoning, which directly impacts learned weights.

Secure Training Environments: Ensure the integrity of the training infrastructure to prevent malicious injection of code or data.

• Model Versioning and Rollback: Maintain immutable versions of trained models and have mechanisms for rapid rollback to a known good state if an attack is detected.
• Supply Chain Security for Models: For models obtained from third-party repositories (like Hugging Face), verify their integrity and provenance before deployment.

By adopting these multi-faceted defensive strategies that extend to the deepest layers of AI model architecture and mathematical logic, organizations can build more resilient and trustworthy AI systems capable of withstanding sophisticated internal attacks.

Key Takeaways

• Deep AI Security is Crucial: AI security must move beyond peripheral attacks (like prompt injection) to focus on the model's core architecture and mathematical foundations.
• Model Confidentiality is Vulnerable: Tools like Netron can easily reverse-engineer deep learning models, exposing sensitive internal details like architecture, input parameters, and critically, weights.
• Weights are the Crown Jewels: Model weights are the essence of a model's intelligence and can be extracted as Numpy files, allowing for replication or targeted poisoning attacks.
• Sparsity is a Security Indicator: Models with 0% sparsity (dense matrices) are highly interconnected and thus extremely vulnerable to even minor weight perturbations, as changes propagate widely.
• Control Systems Theory Offers New Defenses: Analyzing AI models as feedback control systems using techniques like Root Locus and Bode Plots can identify stability weaknesses and pinpoint specific "frequencies" (e.g., high-frequency image features) where models are most susceptible to attack.
• New Tools for Deeper Attacks/Defenses: Emerging open-source tools like Tensor (for SafeTensors fuzzing) and Torture (for PyTorch fuzzing/poisoning) empower researchers to explore and exploit these deep vulnerabilities.

About the Speaker(s)

Yashodhan Vivek Mandke is a dedicated researcher currently pursuing his PhD in satellite security. His academic and professional journey is marked by extensive research across diverse and cutting-edge domains, including space research, IoT security, IoT product development, and AI product development. Mandke's background as an ML developer who transitioned into a cybersecurity researcher provides him with a unique perspective, enabling him to bridge the gap between AI development and security vulnerabilities. His work, as presented at Nullcon, reflects a deep commitment to exploring and understanding the fundamental security challenges inherent in artificial intelligence systems.