Hidden Risks: Pitfalls in Learnable Obfuscation for Private A.I.

Hidden Risks: Pitfalls in Learnable Obfuscation for Private A.I.

Speakers: Mihail-Iulian Pleșa, Security Researcher, Orange Services

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=uNXGIrTh9oU

Overview

In an era defined by stringent data privacy regulations like GDPR and the increasing need for collaborative data analysis across multiple, sensitive data silos, privacy-preserving machine learning (PPML) has emerged as a critical field. This talk by Mihail-Iulian Pleșa, a Security Researcher at Orange Services, delves into a specific class of PPML techniques known as learnable obfuscation. While these methods promise efficient training of machine learning models on sensitive, transformed data, Pleșa critically evaluates their security guarantees, revealing significant pitfalls and demonstrating how two prominent learnable obfuscation schemes, Neurocrypt and Instahide, have failed to uphold their claimed security in realistic scenarios.

The presentation highlights a fundamental dichotomy in PPML: "provable" methods, grounded in strong cryptographic guarantees, versus "non-provable" or "learnable" methods, which prioritize efficiency but often lack rigorous security proofs. Pleșa argues that while learnable obfuscation offers compelling performance advantages, its current state of research is akin to a "game" where new methods are proposed and subsequently broken. Through detailed technical explanations and live demonstrations, the talk exposes the underlying flaws in the security models of these efficient techniques, emphasizing the necessity for extreme caution when considering their deployment in sensitive applications.

The core message of the talk is a stark warning: the efficiency and utility offered by learnable obfuscation methods come at a significant, often unacknowledged, security cost. Despite mathematical proofs of security, these proofs frequently rely on unrealistic assumptions about an attacker's capabilities, particularly their inability to obtain even a small subset of original and transformed data pairs. This oversight renders many proposed learnable obfuscation schemes vulnerable to practical attacks, underscoring the gap between theoretical security and real-world applicability in the evolving landscape of private AI.

Background

▶ Watch: Introduction: The Need for Private A.I. (0:00)

The imperative for privacy-preserving machine learning (PPML) stems from two primary drivers: regulatory mandates and distributed data ownership. Regulations like GDPR impose strict requirements on how personal data, especially sensitive categories, can be processed and shared. Simultaneously, scenarios often arise where multiple parties, each holding a subset of sensitive data, need to collectively train a machine learning model without compromising the confidentiality of their individual contributions. A pertinent example provided by Pleșa involves three hospitals, each possessing patient data related to a specific disease. To train a robust model for disease detection, data from all three hospitals is crucial. However, legal and ethical constraints prohibit direct sharing of raw patient information.

PPML methods aim to resolve this dilemma by enabling collaborative model training while safeguarding the underlying data. These methods broadly fall into two categories:

1. Provable Methods: These are typically based on strong cryptographic principles and offer formal mathematical proofs of security, ensuring that an attacker, regardless of their computational power, cannot reverse the transformation to recover original data.

• Homomorphic Encryption (HE): Allows computations directly on encrypted data, yielding encrypted results that, when decrypted, match the results of computations on plaintext. While theoretically robust, HE suffers from increased running times and significant computational overhead, making it impractical for large-scale machine learning tasks.
• Secure Multi-Party Computation (SMPC): Protocols designed for scenarios where multiple parties wish to jointly compute a function over their combined private inputs, without revealing their individual inputs to each other. SMPC offers strong privacy guarantees but is often associated with high communication overhead and complexity, limiting its practical deployment.
• Differential Privacy (DP): Achieves privacy by adding carefully calibrated noise to data or model outputs, ensuring that the presence or absence of any single individual's data point does not significantly alter the outcome. DP provides formal security guarantees, ensuring data cannot be "denoised," but can sometimes impact model utility.

1. Non-Provable Methods (Learnable Obfuscation): In contrast, these methods prioritize efficiency and utility, often achieving performance closer to traditional machine learning. However, they generally lack the formal security proofs characteristic of cryptographic approaches. Pleșa characterizes the research landscape in this area as a constant "game": a method is proposed, then weeks or months later, it is broken by another researcher.

• Masking: Involves applying a consistent geometric transformation to each data sample. For instance, in image datasets, the same random, noisy image might be added to every original image. The transformed images appear random, but the consistency of the transformation allows a model to learn features.
• Mixing: Combines multiple data samples to create new, obfuscated samples. An example is taking an image of a cat and an image of a dog, then blending them into a single image labeled with a weighted combination (e.g., "70% dog, 30% cat"). The mixed image itself might not clearly represent either original class, preserving confidentiality, yet a model can learn from these weighted labels.
• Permutation: A supplementary technique often used in conjunction with masking or mixing. It involves rearranging parts of the data, such as rotating rows of pixels in an image by a fixed number of positions.

The general workflow for learnable obfuscation involves multiple parties agreeing on a specific transformation. Each party applies this identical transformation to their local, private data. The transformed data is then pooled (e.g., on a central server or cloud), and a machine learning algorithm is trained on this combined, obfuscated dataset. Crucially, the aim is privacy-preserving training, not inference. After training, the model is distributed back to the parties, who then use it locally on their own transformed data for inference. This distinguishes it from scenarios where inference itself occurs on encrypted data in the cloud.

Key Findings

▶ Watch: Security: Cryptography vs. Learnable Obfuscation (9:00)

Mihail-Iulian Pleșa's talk unveils several critical findings regarding the state of learnable obfuscation in privacy-preserving AI, challenging its perceived security and practical viability:

1. Efficiency vs. Provable Security Trade-off: Learnable obfuscation methods are undeniably efficient. They involve computationally light operations like adding random data, mixing samples, or simple permutations, allowing machine learning models to achieve good accuracy on transformed data (e.g., 96% on MNIST, 80% on CIFAR-10). However, this efficiency comes at a severe cost to provable security, a stark contrast to cryptography-based methods which are provably secure but computationally expensive.

1. Flawed Security Models and Assumptions: The central flaw in many learnable obfuscation schemes, including the prominent examples Neurocrypt and Instahide, lies in their unrealistic security models. Their formal security proofs, while often mathematically sound within their defined assumptions, fail to account for practical attacker capabilities. Specifically, these models assume that an attacker has no access whatsoever to pairs of original and transformed data.

1. Vulnerability to "Known Plaintext" Attacks: In real-world scenarios, it is highly probable that an attacker can obtain a small subset of original-transformed data pairs. This can happen, for instance, when private datasets are augmented with publicly available data (e.g., ImageNet, CIFAR-10) to improve model performance. Even a minuscule number of such pairs (e.g., 50 images out of 60,000) is sufficient to completely compromise the privacy of the entire transformed dataset.

1. Neurocrypt's Failure: Misapplication of One-Time Pad Equivalence: Neurocrypt, a masking technique, claimed information-theoretic security by proving equivalence to a one-time pad. While the mathematical proof itself was correct, the underlying assumption that an attacker would never see any plaintext-ciphertext pairs (or original-transformed pairs) rendered it insecure. An attacker, given even a few pairs, can train a simple neural network to reverse the obfuscation, effectively "denoising" the entire dataset.

1. Instahide's Failure: Public Weights and Linear Algebra: Instahide, a mixing technique, based its security on the subset sum problem, a known hard problem in cryptography. Its proof asserted that breaking Instahide would imply solving subset sum, thus making Instahide secure. However, the talk reveals that the mixing "weights" (e.g., 70% dog, 30% cat) are not secret; they are inherently part of the labels provided to the machine learning model during training. This leakage allows an attacker to bypass the complex subset sum problem entirely and instead solve a simple linear system of equations to reconstruct the original images.

1. Partial Security Guarantees: Even when learnable obfuscation methods offer some formal guarantees, these are often limited to specific privacy goals (e.g., preventing data reconstruction) and do not extend to others (e.g., preventing membership inference attacks). This piecemeal approach to security means that a method might protect against one threat while leaving data vulnerable to another.

1. Need for Caution and Realistic Modeling: The overarching finding is a call for extreme caution. While promising, learnable obfuscation methods are not yet mature enough for sensitive real-world applications. Future research must focus on developing more realistic attacker models and providing comprehensive, provable security guarantees that account for the practical realities of data sharing and potential information leakage.

Technical Deep Dive

▶ Watch: Case Study: Instahide and its Vulnerability (22:45)

The core of Pleșa's technical exposition revolves around contrasting the rigorous security definitions in cryptography with the often-flawed approaches in learnable obfuscation, using Neurocrypt and Instahide as prime examples of where theoretical proofs diverge from practical security.

Security Definitions: Cryptography vs. Learnable Obfuscation

In cryptography, security is typically defined in an "all-or-nothing" manner. For instance, indistinguishability under chosen plaintext attack (IND-CPA) for an encryption scheme means that an attacker, even after querying an encryption oracle multiple times with chosen plaintexts, cannot distinguish between the encryptions of two challenge plaintexts. This is a very strong guarantee: no information about the plaintext can be gleaned from the ciphertext. As Pleșa notes, if an attacker queries an oracle for "this is an apple" and "this is a car," receives ciphertexts, and then is given a challenge ciphertext, they cannot determine if it encrypts "apple" or "car" even with prior knowledge of other encryptions. This guarantees 100% mathematical certainty of data confidentiality.

Learnable obfuscation, however, cannot achieve this level of security. Pleșa illustrates this with a simple example: if an ML model is trained on obfuscated MNIST (handwritten digits, grayscale, simple to learn, ~99% accuracy) versus obfuscated CIFAR-10 (color images, 10 classes, more complex, ~70-80% accuracy), an attacker observing the final model's accuracy can infer which dataset was used. This means information is leaked, precluding cryptographic-level indistinguishability.

Given this inherent limitation, learnable obfuscation methods aim for more modest, partial security goals:

• Data Reconstruction: The attacker should not be able to recover the original, exact data samples (e.g., images) from the transformed data. This is often the primary focus.
• Membership Inference: The attacker should not be able to determine if a particular data sample (e.g., a patient's X-ray) was included in the training dataset. This is particularly relevant for sensitive scenarios like healthcare.
• Reverse Engineering: The attacker should not be able to deduce or reverse the transformation mechanism itself.

Pleșa points out that current learnable methods typically offer guarantees for only some of these goals, often relying on information-theoretic measures or partial proofs. There is currently no single learnable method that formally guarantees all three goals simultaneously. A recent paper from the previous year is cited as a promising step towards formalizing security, specifically proving resistance to data reconstruction under the condition that an attacker does not possess "enough examples of plaintext and obfuscated data." This represents a shift towards conditional security rather than absolute.

Case Study 1: Neurocrypt (Masking)

Neurocrypt is a masking technique that proposes using a random neural network to transform data. The core idea is that all participating parties agree on the same random neural network (initialized with random weights). Each party processes their private images through this network, then pools the transformed images for training. The model is then trained on these transformed images.

The authors of Neurocrypt formally claimed their method was equivalent to a one-time pad, implying information-theoretic security—the strongest possible form of security, where an attacker, regardless of computational power, gains no information. Their proof was mathematically sound.

The Flaw: The critical vulnerability lay in the security model's unrealistic assumption: the proof was valid only if the attacker had no access to any original-transformed data pairs. This is rarely the case in practice. As Pleșa explains, to improve classification accuracy, private datasets (e.g., 1,000 private images) are often augmented with large public datasets (e.g., 10,000 public images from ImageNet or CIFAR-10). An attacker can easily obtain samples from these public datasets, apply the known transformation (since the random neural network is shared), and thus acquire original-transformed pairs.

The Attack: With even a small number of such pairs (e.g., 50 pairs from a 60,000-image MNIST dataset), an attacker can train a neural network to learn the inverse transformation. This "recovery network" can then be used to reconstruct the original images from the entire obfuscated dataset, effectively breaking Neurocrypt's privacy guarantees. The demo showed that with 50 pairs, recognizable features could be reconstructed, and increasing the number of pairs significantly improved reconstruction quality.

Case Study 2: Instahide (Mixing)

Instahide is a mixing technique. It takes groups of images and creates mixed images by computing weighted sums of the original images (e.g., 50% dog + 50% cat). A crucial additional step is a random sign flip applied to each pixel of the resulting mixed image.

Instahide's security proof claimed a reduction to the subset sum problem, a computationally hard problem in cryptography. The argument was that if an attacker could break Instahide, they would effectively solve the subset sum problem, implying Instahide's security because subset sum is considered intractable.

The Flaw: Again, the proof rested on an unrealistic assumption: that the mixing weights were secret. In reality, for the machine learning model to learn effectively from mixed images, it needs to know the composition of the mix. For example, if an image is 70% dog and 30% cat, this information is provided as part of the label during training. Therefore, the mixing weights are public.

The Attack: Since the weights are public, an attacker can treat the original images as unknowns in a linear system of equations. Given the mixed images and the public weights, the attacker simply solves this system. This is "high school math," not a cryptographically hard problem. The random sign flip component is addressed by the attacker considering the absolute values of pixels, and then using other methods to refine the sign, leading to highly recognizable reconstructed images. The demo showed clear recovery of original images (e.g., a car) even if the exact pixel signs were perturbed.

In both cases, the fundamental disconnect was between the theoretical models of attacker capabilities (assuming zero access to crucial information) and the practical realities of how these systems would be deployed and how attackers operate.

Demo / Proof of Concept

▶ Watch: Demo: Neurocrypt Image Reconstruction Attack (30:50)

Mihail-Iulian Pleșa presented two compelling live demonstrations, one for Neurocrypt and one for Instahide, to illustrate their vulnerabilities. Both demos used standard datasets and showcased how the systems, despite their theoretical claims, could be easily compromised under realistic attack scenarios.

Neurocrypt Demo

The Neurocrypt demonstration utilized the MNIST dataset, which consists of grayscale images of handwritten digits.

1. Transformation: The speaker first instantiated a simple linear neural network initialized with random weights. This network served as the random transformation function. An original MNIST image (e.g., a digit '8') was passed through this random network, resulting in a highly noisy, seemingly uninformative transformed image. Pleșa emphasized that the same random network is applied to all images in the dataset, a crucial detail for the subsequent attack.

1. Model Training: A standard neural network classifier (a "classic neural network with several layers") was then trained on the transformed MNIST data. This classifier achieved a respectable accuracy of 96%, demonstrating that the obfuscated data still retained sufficient features for effective learning.

1. Inference: The demo clarified the inference process: to use the trained model, new input images must also be transformed by the same random neural network. Feeding an original image (e.g., a digit '3') directly into the model trained on transformed data resulted in an incorrect classification (e.g., '6'). However, first transforming the '3' and then feeding it to the model yielded the correct classification. This confirmed the utility aspect of Neurocrypt.

1. The Attack: This was the critical part. Pleșa demonstrated the attack scenario where an attacker has access to a very small subset of original-transformed image pairs. For a training dataset of 60,000 MNIST images, the attacker was assumed to have access to only 50 pairs of original and their corresponding transformed images. The attacker then trained a separate neural network using these 50 pairs to learn the inverse transformation (i.e., how to reconstruct an original image from its transformed version).

• The results showed that even with just 50 pairs, the attacker's recovery network could reconstruct the original images with "quite good" quality, allowing most features of the digits to be distinguished.
• The speaker then interactively increased the number of attacker samples to 100 and then 500, demonstrating a visible improvement in the quality of the reconstructed images, further highlighting the system's fragility.

Instahide Demo

The Instahide demonstration used the CIFAR-10 dataset, which contains color images across 10 classes, but was restricted to three classes for faster demo execution.

1. Transformation (Mixing): The demo showed how Instahide generates mixed images. For example, an image of a car and an image of a bird were combined using weighted sums (e.g., 34% car, 66% bird). After computing this sum, a random sign flip was applied to each pixel. The resulting image was noisy and did not clearly represent either a car or a bird. This illustrated the obfuscation process.

1. Model Training: A simple convolutional neural network classifier was trained on the transformed CIFAR-10 dataset. This model achieved an accuracy of 80%, confirming that Instahide also preserved enough features for effective learning, similar to Neurocrypt.

1. The Attack: The Instahide attack capitalized on the leakage of mixing weights. Pleșa explained that the weights (e.g., 58% airplane, 42% bird) are necessarily public because they are used as labels during the training process for the classification model.

• Instead of training a complex neural network, the attacker simply leveraged these public weights to solve a linear system of equations. Each original image was treated as an unknown in the system.
• The random sign flip was handled by the attacker by considering the absolute value of each pixel, with further methods available to refine the sign later.
• The demo displayed a reconstructed image (e.g., a car) that was clearly identifiable, demonstrating a successful recovery of the original content, albeit with some perturbation to pixel signs. This showcased how a fundamental cryptographic assumption (secrecy of weights) was violated, leading to a straightforward, non-ML-based attack.

Both demos powerfully underscored the central thesis: the security claims of these learnable obfuscation methods are often built on unrealistic assumptions about attacker capabilities, leading to practical vulnerabilities that can be exploited with minimal effort or information.

Defensive Implications

▶ Watch: Q&A: Practicality of Learnable Obfuscation Today (40:50)

The findings presented by Mihail-Iulian Pleșa carry significant defensive implications for organizations and researchers considering privacy-preserving machine learning. The primary takeaway is a strong call for caution and a re-evaluation of current practices in deploying learnable obfuscation techniques.

1. Extreme Caution with Learnable Obfuscation: The most immediate implication is that current learnable obfuscation methods, despite their efficiency, are not suitable for scenarios requiring strong, provable privacy guarantees. Organizations handling highly sensitive data (e.g., patient records, financial information, national security data) should be extremely wary of relying on these techniques. The speaker explicitly stated that, for now, he would not recommend using learnable obfuscation in practice for critical privacy needs.

1. Scrutinize Security Assumptions: Defenders and researchers must move beyond simply accepting published security proofs at face value. It is imperative to critically examine the underlying assumptions of these proofs, particularly concerning the attacker model. Questions to ask include:

• Does the model realistically account for an attacker's ability to obtain any original-transformed data pairs, however small the quantity?
• Are all "secret" components truly secret, or can they be inferred or leaked through other channels (e.g., training labels, public data)?
• Does the proof guarantee against all relevant privacy attacks (reconstruction, membership inference, reverse engineering), or only a subset?

1. Prioritize Provable Methods for High-Stakes Scenarios: For applications where data confidentiality is paramount, provable methods such as Secure Multi-Party Computation (SMPC) and Homomorphic Encryption (HE) remain the recommended choice, despite their performance drawbacks. While SMPC might incur a "large communication overhead," Pleșa noted that its running time might be acceptable for "small data sets." These methods offer mathematical guarantees that, unlike learnable obfuscation, do not crumble under realistic attacker models.

1. Acknowledge and Mitigate Information Leakage: The demonstrations highlighted how even subtle forms of information leakage (e.g., mixing public with private data, public weights in labels) can completely undermine security. Defenders must conduct thorough threat modeling to identify all potential channels through which an attacker might gain access to pairs of original and transformed data or infer transformation parameters. If such leakage is unavoidable, the learnable obfuscation method is likely compromised.

1. Invest in Robust Research and Development: The field of learnable obfuscation is still evolving. The mention of a "paper from last year" that offers some hope by formalizing conditional security suggests that future advancements might lead to more secure methods. However, until these theoretical breakthroughs translate into robust, practically secure implementations, significant research and validation are needed. This includes developing new methods with stronger, more realistic security proofs and rigorous empirical testing against sophisticated attacks.

1. Avoid False Sense of Security: The irony that both Neurocrypt and Instahide had formal proofs of security, yet failed, underscores the danger of a false sense of security. Implementing a system based on such proofs without understanding their limitations can lead to catastrophic data breaches. It is crucial to understand that a "proof" is only as strong as its foundational assumptions.

In summary, while learnable obfuscation is "efficient" and "works" in terms of model utility, its current security guarantees are often built on shaky ground. Defenders must exercise extreme caution, prioritize provable methods for sensitive applications, and demand more robust, realistically modeled security proofs from any new learnable obfuscation technique before considering its adoption.

Key Takeaways

• Learnable obfuscation offers an efficient approach to privacy-preserving machine learning training, allowing models to achieve good accuracy on transformed data (e.g., 96% on MNIST, 80% on CIFAR-10).
• However, many current learnable obfuscation methods, including Neurocrypt and Instahide, are fundamentally insecure due to their reliance on unrealistic security assumptions in their formal proofs.
• The primary flaw is the assumption that an attacker will never obtain any pairs of original and transformed data, an assumption easily violated in real-world scenarios (e.g., mixing private with public datasets).
• Neurocrypt (masking) failed because even 50 original-transformed image pairs allowed an attacker to train a neural network to reconstruct original images, despite its theoretical equivalence to a one-time pad.
• Instahide (mixing) failed because its mixing weights were leaked via training labels, allowing an attacker to bypass a cryptographically hard problem (subset sum) and simply solve a linear system of equations to recover original images.
• For applications requiring strong, provable privacy guarantees (e.g., healthcare), cryptography-based methods like Secure Multi-Party Computation (SMPC) or Homomorphic Encryption (HE) are currently the more secure, albeit less efficient, options.
• Future research is attempting to formalize conditional security for learnable obfuscation, but for now, extreme caution is warranted, and security assumptions must be rigorously scrutinized before deployment.

About the Speaker(s)

Mihail-Iulian Pleșa is a Security Researcher at Orange Services. His work focuses on evaluating the security and privacy implications of emerging technologies, particularly in the domain of artificial intelligence and machine learning. His expertise lies in identifying and demonstrating practical vulnerabilities in systems that claim privacy-preserving capabilities.