Achieving Zen: Combining Mathematical and Programmatic Deep Learning Model Representations for Attribution and Reuse

Achieving Zen: Combining Mathematical and Programmatic Deep Learning Model Representations for Attribution and Reuse

Speakers: David Oygenblik

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=38ENoGDpjMY

Overview

When a self-driving car crashes because its sign recognition model was backdoored, how do investigators determine what happened? They need to recover the model's architecture, set up a testing environment, and apply white-box analysis techniques like NeuralCleanse or OBD scan -- but modern proprietary models mix open-source base models with custom layer implementations, making this process nearly impossible without source code. This talk presents Zen, a memory forensics tool that bridges this investigation gap by recovering both the mathematical representation (weights, layer shapes, graph structure) and the programmatic representation (model inference code, function signatures, bytecode) of deployed deep learning models from CPU and GPU memory dumps.

Presented by David Oygenblik, Zen introduces a novel model fingerprinting and attribution system that can identify which open-source base model a proprietary model was derived from (with 97%+ similarity matching), determine the differences, and automatically generate patches to reconstruct a testable version of the proprietary model -- even when up to 83.3% of the code has been changed from the base. The tool was evaluated across seven model families including YOLO, ResNet, MobileNet, BEiT, and LLaMA.

Background

▶ Watch: Motivation: investigating a backdoored self-driving car model (0:00)

Modern proprietary deep learning models are rarely built from scratch. Engineers typically download an open-source model from platforms like GitHub or Hugging Face, customize the architecture with novel layers from research papers or their own implementations, retrain on proprietary data, and deploy. This creates a challenge for security investigators: the deployed model combines known (open-source) and unknown (proprietary) components.

To apply white-box testing techniques for backdoor detection, investigators need three things: model source code (to instrument with analysis tools), a testing model environment (to run the model), and knowledge of the model's architecture (layer types, shapes, and sizes). If the developer used custom layer implementations that are unknown to the investigator, none of these prerequisites can be met.

Prior work in model recovery from memory focused on extracting mathematical representations (weights, layer shapes, sizes) but neglected the programmatic representation -- the actual code needed to construct layers, run forward passes, and execute inference. Without the code, recovered weights are useless because you cannot build the model to run them through.

Key Findings

▶ Watch: Zen's dual recovery: mathematical and programmatic representations (3:00)

Zen recovers both mathematical and programmatic model representations from memory, creating a model fingerprint that combines both:

Mathematical recovery (building on prior work): Extracts model weights, layer shapes, layer sizes, graph structure, and data pointers from CPU and GPU memory dumps.

Programmatic recovery (Zen's novel contribution): Enumerates all functions from memory, filters them using previously recovered layer information and data pointers to isolate functions necessary for model inference (discarding logging, telemetry, and other non-inference functions), then recursively recovers all dependency functions needed by each inference function. The result is the complete code required to perform a model forward pass.

Model attribution: Zen compares the combined fingerprint against a base model library of open-source models. A novel attribution algorithm compares both mathematical components (weights, layer shapes, graph structure) and programmatic components (function signatures, bytecode), achieving 97%+ similarity in correctly matching proprietary models to their base models.

Patch generation: Once the base model is identified, Zen determines the differences between the proprietary and base models, identifying changed code (modified functions) and unique code (added functions). It sets up a local environment matching the open-source base model and generates and applies patches to create a testable mock of the proprietary model. Zen successfully patched up to 46 functions and handled cases with up to 83.3% code change from the base model.

End-to-end validation: Pre-deployment performance matches post-Zen performance in all test cases (except a minor difference in nanoGPT due to different training resume methodology), confirming that Zen recovers everything needed for accurate inference.

Technical Deep Dive

▶ Watch: Model fingerprinting and base model attribution algorithm (4:00)

Zen's programmatic recovery works through a filtering and dependency analysis pipeline:

1. Function enumeration: All functions are recovered from memory with their signatures, arguments, local variables, and bytecode.

1. Inference function filtering: Using the mathematical recovery results (layer types, shapes, sizes, data pointers), Zen filters the function list to retain only those involved in model inference. This eliminates logging, telemetry, debugging, and other non-essential functions that would complicate analysis.

1. Recursive dependency resolution: For each remaining function, Zen analyzes the bytecode to identify all functions that the inference function depends on, recursively recovering the complete dependency tree.

1. Fingerprint construction: The mathematical representation (weights, graph, layer info) and programmatic representation (filtered functions, bytecode, signatures) are combined into a model fingerprint.

The attribution algorithm applies a two-phase comparison:

• Mathematical comparison: Compares weights, layer shapes, and graph structure between the unknown model and base model candidates
• Programmatic comparison: Compares function signatures, bytecode sequences, and code structure

The heat map of similarity scores shows clear diagonal matching: customized models match their true base models with 97%+ similarity while showing low similarity to other model families.

Patch generation handles two categories of differences: changed code (functions present in both but modified) and unique code (functions only in the proprietary model). Patches are generated to transform the base model's code into the proprietary model's code, enabling test environment setup without access to the original deployment infrastructure.

Demo / Proof of Concept

▶ Watch: Evaluation: 97%+ attribution accuracy across seven model families (6:00)

The evaluation covered seven model families across multiple tasks:

• Object detection: YOLO
• Image classification: ResNet, MobileNet, BEiT
• Text generation: LLaMA, nanoGPT

Models were trained on datasets including COCO, VisDrone, CIFAR-10, and OpenWebText. Zen took CPU and GPU memory dumps for each custom model deployment as input.

The end-to-end performance validation confirmed that post-Zen inference performance matches pre-deployment performance across all test cases. The patch generation evaluation showed Zen successfully handling up to 46 patched functions and up to 83.3% code change from the base model (typical for small models like MobileNetV2 where minor changes represent a large percentage of the codebase).

A copyright investigation case study demonstrated Zen's use for model attribution: determining whether a deployed model is derived from a copyrighted open-source model. The paper also discusses potential evasion strategies against Zen and corresponding defenses.

Defensive Implications

▶ Watch: Patch generation handles up to 83.3% code change from base model (6:30)

Zen enables several defensive capabilities that were previously difficult or impossible:

1. Backdoor investigation: When a deployed model is suspected of being backdoored (as in the self-driving car scenario), Zen enables investigators to recover the model, identify its base, reconstruct a testable environment, and apply white-box analysis tools without access to the original source code or deployment infrastructure.

1. Copyright and IP enforcement: Zen can attribute deployed models to their open-source origins, enabling enforcement of model licenses and identification of unauthorized model reuse.

1. Incident response for AI systems: When an AI-powered system behaves unexpectedly, Zen provides a forensic pathway from memory dump to testable model, enabling post-incident analysis that was previously limited to black-box observation.

1. Supply chain verification: Organizations deploying third-party AI models can use Zen to verify that the deployed model matches what was contracted, detecting unauthorized modifications or substitutions.

The main limitation is that Zen requires a base model library for attribution -- if the proprietary model is built on a base model not in the library, attribution fails. However, a one-time reverse engineering effort can add any model to the library for future use.

Key Takeaways

• Zen is the first memory forensics tool to recover both mathematical and programmatic representations of deployed deep learning models
• Model fingerprinting combines weights, layer shapes, graph structure, function signatures, and bytecode for 97%+ attribution accuracy
• Automatic patch generation enables reconstruction of testable proprietary models from identified base models, handling up to 83.3% code change
• Pre-deployment and post-Zen inference performance match, confirming complete recovery
• Evaluated across seven model families (YOLO, ResNet, MobileNet, BEiT, LLaMA, nanoGPT) spanning object detection, classification, and text generation
• Enables backdoor investigation, copyright enforcement, and incident response for AI systems

About the Speaker(s)

David Oygenblik presented the work with enthusiasm and clarity, demonstrating clear knowledge of both the forensic analysis pipeline and the deep learning model landscape. He engaged directly with audience questions about threat models (acknowledging that GPU/CPU memory dump access is required) and scope limitations (base model must be in the library). He noted that the tool is built on top of prior memory forensics work and is available for experimentation.