Loading Models, Launching Shells: Abusing AI File Formats for Code Execution

Loading Models, Launching Shells: Abusing AI File Formats for Code Execution

Speakers: Cyrus Parzian

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=IHzn9BiH6rY

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Cyrus%20Parzian%20-%20Loading%20Models%2C%20Launching%20Shells%20Abusing%20AI%20File%20Formats%20for%20Code%20Execution.pdf

Overview

The explosion of AI model sharing has created a new attack surface that combines the risks of untrusted code execution with the trust dynamics of scientific software distribution. At DEF CON 33, Cyrus Parzian — an AI red teamer at a major U.S. healthcare company — delivered a deep technical examination of how popular AI model file formats, particularly Python's pickle serialization format and the widely-used .pt (PyTorch) and .pkl model files, can be weaponized to achieve arbitrary code execution on any machine that loads them.

The talk traces the history of pickle-format security issues from their earliest public disclosure, maps the current landscape of AI model distribution platforms and tooling, demonstrates multiple working exploit techniques, and evaluates the effectiveness (and failures) of existing mitigations. The core message is a supply chain risk argument: as the AI ecosystem normalizes downloading and running model files from public repositories like Hugging Face, model hubs, and open-source research releases, organizations are importing untrusted executable artifacts under the guise of model weights — and their security programs largely do not treat them as such.

Background

▶ Watch: NA AV and EDR for the last 10 years or so and about last year we actually sta... (1:23)

The Pickle Format

Python's pickle module is a serialization format that converts Python objects to and from a byte stream. Unlike data serialization formats such as JSON or Protocol Buffers — which represent only data — pickle represents arbitrary Python objects, including their class constructors and the code needed to reconstruct them.

The pickle format includes an __reduce__ method hook that objects can implement to control how they are serialized. When an object with __reduce__ is unpickled, Python executes the return value of __reduce__ — which can be any callable, including os.system, subprocess.Popen, or any other function that executes shell commands.

This is not a bug in pickle — it is by design. The Python documentation explicitly warns: "The pickle module is not secure. Only unpickle data you trust." Despite this warning, pickle is the dominant serialization format for machine learning model files:

• PyTorch .pt and .pth files are serialized with pickle.
• Scikit-learn model files saved via joblib use pickle under the hood.
• Keras/TensorFlow legacy model formats use pickle for metadata and weight storage.
• Many ad hoc model-sharing practices use .pkl files directly.

The consequence: loading a PyTorch model from an untrusted source with torch.load(model.pt) executes arbitrary Python code if the model file has been crafted or tampered with.

Historical Context

Parzian placed this research in historical context:

• Black Hat 2011: Marco Slaviero first systematically disclosed the security issues with pickle serialization in the Python ecosystem.
• DEF CON 3x (prior year): Jonathan demonstrated how to backdoor pickle files — embedding malicious __reduce__ payloads in existing serialized objects.
• DEF CON 33 (this talk): Extending the analysis to the AI model sharing ecosystem, where the scale, trust, and tooling context amplifies the risk by orders of magnitude.

The AI Model Distribution Ecosystem

Machine learning models are shared via:

• Hugging Face Hub: Tens of thousands of models available for direct download, many containing .bin, .pt, or .pkl files that use pickle serialization.
• GitHub releases: Research paper code repositories routinely distribute pretrained models as pickle files.
• Academic model zoos: ML framework-specific collections (PyTorch Hub, TensorFlow Hub, etc.)
• Proprietary and enterprise model hubs: Internal distribution systems at companies deploying AI.

In many of these contexts, users download and load models with a single line of Python code, often without inspecting the file's contents, and frequently without any AV or EDR coverage of the model loading operation.

Key Findings

▶ Watch: And if you actually click on it, you see that uh hugging face mentioning tha (13:23)

1. Pickle-based AI model files are executable artifacts, not passive data — loading them executes Python code, and this is by design in the format.

1. Popular AI tooling (torch.load, joblib.load, etc.) executes pickle payloads without user confirmation or sandboxing by default.

1. A malicious payload can be injected into a legitimate pretrained model file without affecting the model's weights or accuracy — making the backdoored model functionally indistinguishable from the original when evaluated on benchmark tasks.

1. AV and EDR tools largely fail to detect malicious pickle files: The payload is Python bytecode/opcodes within a binary format, not a traditional executable — most signature-based and behavioral detection tools miss it.

1. Model scanning tools (e.g., ModelScan) exist but have limitations: Bypassable in some configurations, not universally deployed, and not integrated into the most common model loading workflows.

1. The attack scales to supply chain level: A compromised model published on Hugging Face or included in an open-source research codebase propagates to every user who loads it — potentially thousands or millions of machines.

1. Alternative safe formats (SafeTensors) exist but adoption is incomplete: The ecosystem is in transition, and many critical models remain distributed only in pickle-based formats.

Technical Deep Dive

▶ Watch: Microsoft defender endpoint. (23:02)

Pickle Internals: The Reduce Protocol

A pickle file is a sequence of opcodes that a stack-based virtual machine (the unpickler) executes to reconstruct an object. The REDUCE opcode calls a callable with a tuple of arguments — and the callable and arguments are both embedded in the pickle stream.

A minimal malicious pickle payload:

When pickle.loads(payload) is called, Python calls os.system('calc.exe'). This is the fundamental primitive — arbitrary code execution with the privileges of the Python process that loaded the model.

More sophisticated payloads replace os.system with:

• subprocess.Popen for a reverse shell.
• exec() on a base64-encoded Python payload for staged execution.
• Network-fetching stagers that download a full implant at runtime.

Injecting Payloads into Legitimate Models

The more practical attack for supply chain scenarios involves modifying a legitimate, functional model file to include the payload while preserving the model's behavior. Parzian demonstrated:

1. Load the legitimate model into Python (this executes in a safe environment under the researcher's control).
2. Extract the model's state dict (the weight tensors) — these are the actual neural network parameters.
3. Create a new pickle file that first executes the malicious __reduce__ payload, then restores the weight tensors so the model loads normally after exploitation.
4. Upload the backdoored model to a public hub or commit it to a research repository.

From the end user's perspective, the model loads, produces correct output on any evaluation task, and shows no signs of tampering. The malicious code executes silently in the background at load time, typically before any model output is visible.

Scope of Vulnerable Tooling

Parzian surveyed the major Python ML libraries:

• torch.load(): Executes pickle by default. PyTorch added a weights_only=True parameter that disables arbitrary object deserialization, but it is not the default and it is not compatible with all legacy models.
• joblib.load(): Uses pickle; no equivalent safe-load option.
• pickle.load() directly: No safeguards.
• numpy.load() with allow_pickle=True: Allows pickle deserialization of .npy files.
• tf.keras.models.load_model() (legacy H5 format): Contains pickle-serialized metadata in some configurations.

Evasion of Security Tooling

Parzian tested the malicious model files against:

• Commercial AV products: Signature-based detection missed the payload in most cases because the pickle opcode sequence does not match executable file signatures.
• EDR behavioral detection: Some EDR tools detect os.system or subprocess.Popen calls at runtime, but the invocation from within a Python process loading a model file is often missed or whitelisted because python.exe is a trusted process.
• ModelScan (an open-source model security scanner): Detected the straightforward payload, but Parzian demonstrated bypass techniques involving indirect function references and multi-stage loading that evaded the scanner.

SafeTensors: The Alternative

The safetensors format (developed by Hugging Face) stores only tensor data in a safe, non-executable format. It has no support for arbitrary object serialization and cannot execute code at load time. This is the intended replacement for pickle-based model serialization.

Adoption challenges:

• Many pretrained models are only available in .pt or .pkl format.
• Research papers distribute models in pickle format for reproducibility reasons without considering security.
• Conversion tools exist but are not universally applied.
• Some model architectures use custom objects in their state dicts that require pickle to serialize properly.

Demo / Proof of Concept

▶ Watch: In the next two slides what I really want to point out here is that uh there ... (35:09)

The demo sequence:

1. Basic pickle PoC: A Python script demonstrates creating a malicious pickle object whose __reduce__ method pops up a calculator — the classic "safe" code execution demo.

1. Model backdoor: A legitimate pretrained model (a small text classifier) is loaded, backdoored with the payload, and saved. The backdoored model is then loaded in a separate Python session — the calculator pops up, the model continues to run normally and produces correct classification results.

1. Reverse shell variant: The payload is replaced with a reverse shell stager. Loading the backdoored model connects back to the attacker's listener. The demo shows a shell session on the "victim" machine established entirely through the model loading operation.

1. AV evasion: The backdoored model file is scanned by a commercial AV tool on the same machine — clean result. The file is then loaded again — shell established.

1. Hugging Face upload simulation: The workflow of uploading a backdoored model to a public hub and having another user "download and run" it was walked through conceptually, with the final load step executed in the demo environment.

Defensive Implications

▶ Watch: As you can see, virus tool gives our .exe a score of zero out of zero. (38:43)

For ML Practitioners and Data Scientists

• Use torch.load(..., weights_only=True) for all model loading from untrusted sources. This is the single most impactful mitigation for PyTorch users and requires only a parameter change.
• Prefer SafeTensors format for any model you create and distribute, and request SafeTensors variants from providers of models you consume.
• Never load model files from untrusted sources without inspection. Treat .pkl, .pt, and .pth files as executables, not data.
• Inspect pickle files before loading using tools like Fickling (from Trail of Bits) or ModelScan to detect obvious malicious payloads.

For Security Teams

• Classify model files as executable artifacts in your data classification policy. Apply the same controls to model files that you apply to Python scripts or compiled binaries.
• Add model scanning to ML pipelines: Integrate ModelScan or equivalent tools into CI/CD pipelines that handle model ingestion.
• Monitor Python process behavior: EDR rules that alert on subprocess.Popen or os.system calls originating from ML loading code paths can catch exploitation at runtime.
• Control Hugging Face and model hub access: Apply the same scrutiny to downloading from ML model hubs that you apply to downloading packages from PyPI or npm. Consider mirroring internal versions of approved models rather than allowing direct downloads.

For Platform Operators (Hugging Face, etc.)

• Scan all uploaded models for malicious pickle payloads before making them publicly available.
• Enforce SafeTensors-first policies for new model uploads, requiring pickle-based uploads to pass security review.
• Display prominent security warnings on model pages that include pickle-based files.

Key Takeaways

• Pickle-based AI model files are executable code, not passive data — torch.load() and pickle.load() execute arbitrary Python at load time.
• Backdoored models are functionally indistinguishable from legitimate ones — the model performs normally while the payload executes silently.
• AV and EDR tools largely miss this attack class because the payload lives inside a binary format and executes within a trusted Python process.
• The supply chain attack surface is enormous: Hugging Face alone hosts hundreds of thousands of models, many of which are pickle-based, downloaded and run by users who never inspect their contents.
• Mitigations exist — SafeTensors, weights_only=True, ModelScan — but adoption is incomplete and the most impactful practices are not the defaults.
• The AI security supply chain problem mirrors historical lessons from npm, PyPI, and other package ecosystems: trust in open distribution systems is routinely exploited, and the AI model ecosystem has not yet internalized this lesson.

About the Speaker

Cyrus Parzian is an AI red teamer at one of the largest U.S. healthcare companies, where he focuses on adversarial machine learning, prompt injection, AI pipeline security, and supply chain risks in AI deployments. He has over ten years of experience in offensive security, with particular depth in payload crafting, AV/EDR bypass techniques, and phishing infrastructure. His transition into dedicated AI red teaming puts him at the intersection of traditional offensive security and the emerging threats specific to machine learning systems. He emphasizes throughout the talk that the views expressed are personal and do not reflect his employer.