Advanced Bypass Techniques and a Novel Detection Approach

Smashing Model Scanners: Advanced Bypass Techniques and a Novel Detection Approach

Speaker: Itay Ravia, Head of Aim Labs, Aim Security

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=jjiE9XzJo0M

Reading Time: ~8 minutes

Type: Briefing

TL;DR

Static model scanners used to vet AI models from repositories like Hugging Face are fundamentally unable to catch malicious code embedded in model files, because the problem of exhaustively analyzing arbitrary code is NP-hard. Itay Ravia of Aim Security demonstrates dozens of bypass techniques against every major static scanner and proposes a dynamic tracing approach — running models in a sandbox and observing their system calls — as the only reliable detection method.

Introduction

The open-source AI ecosystem runs on trust. Data scientists routinely pull models from Hugging Face — a repository hosting nearly two million models — and load them directly into their environments without questioning whether those files might execute malicious code at load time or inference time. The current defense is a class of tools called model scanners, which attempt to statically analyze model file bytes and flag dangerous imports or function calls before loading occurs.

Itay Ravia, Head of Aim Labs at Aim Security and author of the Echo Leak research, came to Black Hat USA 2025 to deliver an uncomfortable message: those static scanners are not just inadequate — they are structurally incapable of doing their job. The same limitations that defeated first-generation antivirus signatures apply here, and the attack surface is enormous given how deeply serialization formats like Pickle are baked into the ML ecosystem.

The Threat: Code Execution Hidden in Model Files

▶ Watch: The Risk of Third-Party Models (02:00)

When a model is loaded, two windows of code execution exist: load time (when the model file is deserialized) and inference time (when the model is run). Ravia focuses on the former, where the most dangerous vectors live.

Model files consist of two components: the numerical weights and the architecture — the code that describes how the model processes data. While weights are purely numerical, architecture often requires actual executable code, serialized using formats like Pickle, ONNX, Keras, Joblib, and SafeTensors. Of these, Pickle is the most dangerous and the most prevalent. Its assembly-like opcode set — covering operations like GLOBAL (import), stack manipulation, memory read/write, and object instantiation — was designed for Python object serialization, not for security, and it provides virtually unlimited code execution capability to anyone who embeds malicious logic.

The community response has been static scanners: tools that read the raw bytes of a model file, identify which functions and modules are imported, and cross-reference those against a deny list. If os.system appears, the model is flagged. The problem, Ravia argues, is that this deny-list approach cannot scale.

Bypass Category 1: The Infinite Unsafe Function Space

▶ Watch: The Deny List Problem (10:00)

The first category of bypasses exploits the fact that Python has a virtually unlimited number of functions that eventually wrap unsafe operations. Ravia used an AI agent to automatically generate fifty wrapper functions around unsafe system calls, none of which appeared on any static scanner's deny list.

He demonstrated one concrete example: MLflow's project.backend.local._run_entry_point, which wraps CLI command execution. When a model using this function was uploaded to Hugging Face and users queried vendor tools to assess its safety, the tools could not determine whether it was safe — even after a month of community exposure.

This is not a failure of engineering effort but of mathematics: determining whether an arbitrary Python function is a wrapper around an unsafe call is a generalization of the halting problem and is NP-hard. No static scanner can enumerate all unsafe functions in advance.

Bypass Category 2: Stack/Memory Desynchronization in Pickle

▶ Watch: Pickle Opcode Desynchronization (16:00)

The second major bypass class exploits how static scanners simplify Pickle analysis. Pickle is a stack-based virtual machine with over 60 opcodes, but most scanners only emulate a subset — they handle GLOBAL (import) and REDUCE (instantiate) opcodes but ignore others, leaving their internal emulation of the stack and memory perpetually out of sync with what an actual Python unpickler would see.

Ravia's demonstration: an attacker pushes the string "os" onto the Pickle stack, then immediately uses the builtins.str instantiation opcode — which converts a string to a string (a no-op) — and stores the result in memory cell 0. The scanner records builtins.str as what is stored in memory. The actual unpickler stores "os". When the attacker subsequently retrieves memory cell 0 and imports [memory_cell_0].system, the scanner believes it imported builtins.str.system (not on the deny list), while the runtime imports os.system. Clean bypass, every time.

An eight-line proof-of-concept using string splitting to reconstruct "os" at runtime was uploaded to Hugging Face and remained unflagged.

Bypass Category 3: Joblib's Dynamic Byte Blob

▶ Watch: Joblib Format Bypass (20:00)

Joblib is essentially Pickle with NumPy array bytes interspersed throughout the file. The length of each NumPy array blob is determined dynamically — it cannot be computed without actually running the file. When a static scanner encounters one of these blobs, it cannot determine where it ends and must halt analysis at that point.

An attacker can place the dangerous GLOBAL os system opcodes before the blob (so they are stashed in Pickle memory), insert the blob, and then emit the GET and REDUCE opcodes after the blob. The scanner never sees the second half of the execution. Every import listed in the model's visible header is legitimate — indistinguishable from the imports of real Joblib models.

The DeepSeek Edge Case and SafeTensors

Even the supposedly safe SafeTensors format is not immune. Ravia noted that a Black Hat Asia talk earlier in the year demonstrated flows from SafeTensors loading back into Pickle loading. More immediately, DeepSeek's model repository on Hugging Face contains a modeling_deepseek Python file that is entirely unscanned — it cannot be statically analyzed because it is Python source code defining a custom architecture. A scanner can only report on the weights file, not on the code that loads them.

The implication: every custom architecture, including DeepSeek at the time of its release, starts as an unknown quantity that static scanners must either reject outright or silently pass.

The Solution: Dynamic Scanning via Sandboxed Execution

▶ Watch: The Dynamic Scanner Approach (26:00)

Ravia draws an explicit analogy to the evolution of endpoint detection: first-generation AV used static signatures; modern EDR uses behavioral analysis. Model scanning needs the same transition.

His proposed dynamic scanner works by running the model inside a sandbox and tracing its system-level operations using tools similar to strace. The key insight is that the expected behavior of a legitimate model is extremely narrow: read the model file, interact with weights and memory, perform mathematical operations. What a model should never do: spawn new processes, execute CLI commands, make network connections, or write to arbitrary filesystem paths.

This creates an exhaustive allowlist of legitimate operations — something impossible for static function analysis but trivially achievable for system call observation. Demonstrated against the Joblib exploit:

• The static scanner: stops at the byte blob, reports only legitimate imports.
• The dynamic scanner: detects a call to libc system() with the full CLI command string visible.

Against the MLflow _run_entry_point bypass: the dynamic scanner detects new process creation, regardless of which high-level function triggered it, without needing to know MLflow exists.

Against a Keras Lambda function executing CLI commands: the dynamic scanner observes the system call; the static scanner can only say "Lambda function detected — ambiguous."

Notable Quotes

"Static scanners have inherent shortcomings, and they can never be in front of the attackers — always behind."

— Itay Ravia ▶ 30:00

"We can actually spawn numerous bypasses [from each bypass method]. Unfortunately we don't have the time to go into all of them."

— Itay Ravia ▶ 08:00

"A model either loading or inference only makes a very limited set of operations. It doesn't make any sense for a model to execute CLI commands, make network traffic, or create new processes."

— Itay Ravia ▶ 26:00

"This is not just a problem with Pickle. Even SafeTensors — it has safe in its name — unfortunately that's not the case."

— Itay Ravia ▶ 06:00

Key Takeaways

• Static model scanners are structurally broken. The problem of determining whether a Python function is ultimately unsafe is NP-hard — a deny-list approach will always lag behind attacker creativity.
• Pickle's stack/memory model enables trivial desynchronization attacks. Scanners that only emulate a subset of opcodes can be trivially confused about what is actually imported and executed.
• Joblib's dynamic byte blobs create an analysis halt condition that any attacker can exploit to hide imports after the blob in plaintext.
• Dynamic sandbox tracing is the correct paradigm. Because legitimate models have an extremely narrow set of expected system behaviors, an allowlist of acceptable syscalls is both exhaustive and easy to define.
• Data scientists need tooling, not warnings. Hugging Face's current UX of "I don't know if this is safe — you decide" is not a workable security posture for production ML pipelines.

Slides: No slide PDF was available for this talk.