Robustness of client-side scanning for illegal content detection on E2EE platforms
Shubham Jain (PhD Student · Imperial College London)
AI Village @ DEF CON 29 · Day 3 · AI Village
Overview
The proliferation of end-to-end encrypted (E2EE) messaging platforms has revolutionized digital communication, offering unparalleled privacy and security to billions of users worldwide. However, this very security has raised significant concerns regarding the potential for these platforms to be abused for sharing illegal content, such as Child Sexual Abuse Material (CSAM), undetected by service providers. In response, a controversial solution known as Client-Side Scanning (CSS) has been proposed by researchers and policymakers, aiming to detect illicit content before it is encrypted and sent.

Key moments
- 4:00 Client-Side Scanning: How it works and concerns
- 10:00 Demonstrating adversarial attacks on perceptual hashing
- 12:00 High false negative rates for modified illegal images
- 13:40 Increased threshold leads to unacceptable false positives
- 15:00 Attack generates diverse perturbations, defeating database expansion
- 18:00 Apple confirms client-side scanning for CSAM detection
Robustness of client-side scanning for illegal content detection on E2EE platforms
Speakers: Shubham Jain, PhD Student, Imperial College London
Conference: AI Village @ DEF CON 29
YouTube: https://www.youtube.com/watch?v=6HWa8CuiuOA
Overview
The proliferation of end-to-end encrypted (E2EE) messaging platforms has revolutionized digital communication, offering unparalleled privacy and security to billions of users worldwide. However, this very security has raised significant concerns regarding the potential for these platforms to be abused for sharing illegal content, such as Child Sexual Abuse Material (CSAM), undetected by service providers. In response, a controversial solution known as Client-Side Scanning (CSS) has been proposed by researchers and policymakers, aiming to detect illicit content before it is encrypted and sent.
In this critical talk at AI Village @ DEF CON 29, Shubham Jain, a PhD student at Imperial College London, presented groundbreaking research challenging the fundamental robustness of these proposed client-side scanning systems. Jain's work demonstrates that CSS, as currently conceived and utilizing perceptual hashing algorithms, is highly vulnerable to evasion through subtle adversarial perturbations.
The research highlights a significant flaw in the proposed technical approach to a pressing societal problem. By showcasing the ease with which an adversary can bypass detection with minimal visual changes to an image, the talk underscores the false sense of security that CSS might provide, while simultaneously introducing profound privacy and security risks. This work is particularly timely, given recent announcements by major tech companies regarding their intent to deploy similar client-side content scanning technologies, making its findings crucial for ongoing debates about privacy, security, and content moderation in encrypted environments.
Background
▶ Watch: Client-Side Scanning: How it works and concerns (4:00)
The landscape of digital communication has been irrevocably shaped by the widespread adoption of end-to-end encryption (E2EE). Beginning with platforms like Tech Secure in 2013, followed by Signal, WhatsApp (adapting Signal's protocol in 2014), Telegram (2016), and Messenger's secret chats, E2EE has become the default for over two billion users. Major players like Instagram and Messenger are projected to adopt E2EE by default across their platforms by 2022, solidifying its role as the cornerstone of secure private communication. E2EE ensures that only the sender and intended recipient can read a message or view shared content; the servers routing the data have no access to the plaintext, effectively rendering them blind to the content itself.
While E2EE offers vital protection against surveillance and data breaches, it has simultaneously ignited a contentious debate: how to prevent the dissemination of illegal content, particularly CSAM, when service providers cannot inspect the data flowing through their systems. The inability of servers to "see" encrypted content means that traditional server-side content moderation techniques are rendered ineffective.
Historically, one of the earliest proposed solutions to this dilemma was key escrow systems. Under this model, encryption keys would be stored by a third party or the service provider, accessible under exceptional circumstances, such as a law enforcement request. However, key escrow has been widely rejected by the privacy and security community due to profound concerns. Critics argue that it is practically impossible to build a secure, large-scale key escrow system that cannot be compromised or abused by malicious actors, governments, or private organizations. The potential for mass surveillance and the inherent risk of a single point of failure far outweighed any perceived benefits, leading to its widespread dismissal.
In the wake of key escrow's failure, a new paradigm emerged: Client-Side Scanning (CSS). This approach shifts the detection mechanism from the server to the user's device. The premise is that before content (e.g., an image) is encrypted and sent, a digital fingerprinting algorithm within the messaging app would scan it. This fingerprint would then be compared against a database of known illegal content hashes. If a match is found, the content is prevented from being sent, and further action is taken. If no match, the content proceeds to be encrypted and transmitted.
While proponents argue that CSS offers a privacy-preserving alternative to key escrow by keeping the content scanning local to the device, it has faced its own torrent of criticism. Two primary concerns dominate the debate:
- Increased Attack Surface: Introducing a client-side scanning component adds complex code to user devices, creating new potential vulnerabilities that adversaries could exploit. A compromised scanning algorithm or database could lead to false positives, censorship, or even remote code execution.
- Mission Creep: Perhaps the most significant concern is the potential for CSS systems to expand beyond their initial scope. Critics fear that once such a mechanism is in place, governments or powerful entities could pressure platforms to scan for other types of content deemed undesirable, shifting from illegal content to politically sensitive material, hate speech, or even copyrighted works. The question of "who controls the database" and "what is inside the database" becomes paramount, raising fears of pervasive censorship and surveillance.
It is against this backdrop of E2EE's benefits, the challenges of illegal content, and the controversial nature of CSS proposals that Shubham Jain's research poses a fundamental question: even if we accept the privacy trade-offs, is client-side scanning a robust and effective solution against a motivated adversary?
Key Findings
▶ Watch: High false negative rates for modified illegal images (12:00)
The central finding of this research is a resounding "no": client-side scanning, as currently proposed and implemented using perceptual hashing algorithms, is fundamentally not a robust solution for detecting illegal content on E2EE platforms. The talk meticulously demonstrated that an adversary can almost always evade detection, even under a black-box attack model, by introducing subtle, imperceptible perturbations to images.
Specifically, the research revealed:
- Near-Universal Evasion: The presented adversarial attack successfully modified more than 99.9% of images to evade detection across five popular perceptual hashing algorithms. This high success rate was observed even under a wide range of detection thresholds, indicating a systemic vulnerability rather than an edge case.
- Minimal Perturbations: The adversarial modifications are visually imperceptible or result in images that remain "very similar" to the original. This means that a human observer would not easily distinguish the modified content from the original, yet the system's fingerprint would be sufficiently altered to bypass detection.
- Ineffectiveness of Proposed Defenses: The study rigorously tested two commonly suggested defensive strategies against such evasion:
- Increasing the detection threshold (T): While a higher threshold might intuitively seem to make evasion harder, the research showed it comes at an unacceptable cost. For Facebook's PDQ algorithm, increasing the threshold from 30 to 90 (on a scale of 0-256, where 56 is the max difference PDQ can output) reduced evasion from 97% to 67%. However, this also drastically increased the false positive rate from 1 in 1,000 to approximately 1 in 3 safe images being falsely flagged. Such a high false positive rate renders the system practically unusable.
- Expanding the database with adversarial hashes: The idea here is to pre-compute and add hashes of known adversarial variants to the database. However, the research demonstrated that their attack can produce multiple, diverse perturbations for a single image. The resulting adversarial hashes are "widely dispersed" in the hash space, meaning adding a few pre-computed adversarial hashes would not cover the vast number of possible evasive variants. This defense is therefore ineffective against a versatile adversary.
- Significant False Negative Rates: Even at a relatively low threshold of 30, 97% of 100 modified illegal images successfully evaded detection when checked against a database of 100,000 randomly selected Imagenet images. At a higher threshold of 90, a still significant 67% of modified images evaded detection. These high false negative rates indicate that CSS systems, as designed, would be highly unreliable in catching illegal content if targeted by an adversary.
These findings collectively demonstrate that current client-side scanning proposals, reliant on perceptual hashing, are fundamentally flawed. They offer a false promise of detection while simultaneously opening the door to privacy and mission creep concerns, without even delivering on their core security objective of robust illegal content identification.
Technical Deep Dive
▶ Watch: Increased threshold leads to unacceptable false positives (13:40)
The core of client-side scanning (CSS) proposals relies on perceptual hashing algorithms. These are a class of image fingerprinting techniques designed to generate a unique "fingerprint" or hash for an image, such that visually similar images produce similar (or identical) hashes, while perceptually different images produce distinct hashes. They have been widely used for tasks like efficient image search and duplicate detection. A key feature is their ability to measure image similarity without requiring access to the original images, theoretically offering some privacy by only comparing hashes.
The talk highlighted Facebook's PDQ (Perceptual Digest Quantization) as a commonly used open-source perceptual hashing algorithm. The process typically works as follows:
- Image Fingerprinting: An image is fed into the perceptual hashing algorithm, which processes its visual characteristics (e.g., luminance, frequency components, texture) to produce a fixed-length binary string – the "fingerprint" or hash.
- Robustness to Transformations: A well-designed perceptual hash is robust to common, non-perceptible image transformations. For example, resizing an image, converting it to grayscale, or making minor color adjustments should ideally result in a fingerprint that is either identical or very close to the original image's fingerprint. A completely different image, however, would yield a distinctly different fingerprint.
For detection, CSS systems maintain a database of fingerprints of known illegal content. When a user attempts to send an image:
- The user's device generates a perceptual hash of the outgoing image.
- This generated hash is then compared against all hashes in the database.
- Distance Metric: The similarity between hashes is measured using a distance metric. For binary fingerprints, the Hamming distance is typically used, which counts the number of differing bits between two hashes. PDQ, for instance, produces a hash that can have a distance from 0 to 256 (though the talk mentions "0 to 56" specifically for PDQ's effective range, with 56 representing a significant difference).
- Detection Threshold (T): A predefined threshold
Tis established. If the distance between the generated hash and any hash in the database is less than or equal toT, the system flags a "match," indicating potential illegal content. If all distances are greater thanT, no match is found. PDQ commonly recommends a threshold between 20 and 90 for effective operation.
The choice of T is crucial, balancing between:
- True Positives (TP): Illegal images correctly identified.
- False Negatives (FN): Illegal images missed (evaded detection).
- True Negatives (TN): Safe images correctly identified as safe.
- False Positives (FP): Safe images falsely flagged as illegal.
Increasing T generally leads to more TPs and fewer FNs but at the cost of more FPs.
The adversarial attack presented in the talk exploits the inherent properties of these perceptual hashing algorithms. The adversary's goal is to take an illegal image and apply a small, visually imperceptible perturbation such that its newly generated perceptual hash has a distance greater than the threshold T from all hashes in the detection database. This effectively makes the modified image appear "different enough" to the algorithm to evade detection, even though to a human eye, it remains essentially the same illegal content.
The attack leverages black-box adversarial machine learning (ML) algorithms, adapting approaches from prior work (e.g., Iliac Tower, 2018). In a black-box setting, the adversary does not have direct access to the internal workings of the hashing algorithm or the database. Instead, they interact with the CSS system as an oracle, submitting images and observing whether they are flagged or not. Through iterative perturbations and observing the system's responses, the adversary can strategically modify the image's pixels to push its hash beyond the detection threshold.
The researchers demonstrated the attack's efficacy across five popular hashing algorithms, achieving over 99.9% success. For instance, when targeting PDQ with a threshold of 30, their attack successfully made 97% of 100 modified illegal images evade detection against a database of 100,000 images. Even at a higher threshold of 90, 67% of images still evaded.
The study further investigated defensive strategies and found them wanting.
- Increasing
Tas a defense: While it does reduce the false negative rate from 97% (at T=30) to 67% (at T=90), this comes at an unacceptable cost to privacy and usability. The false positive rate skyrockets from 1 in 1000 at T=30 to approximately 1 in 3 images at T=90. This means legitimate, safe content would be flagged with alarming frequency, rendering the system practically useless and deeply intrusive. - Expanding the database with adversarial hashes: This defense relies on the assumption that an attack only produces a limited, predictable set of adversarial variants. However, the research showed that their attack can generate multiple, diverse perturbations for a single image. The resulting hashes of these adversarial images are "widely dispersed" in the hash space. This diversity means that simply adding a few pre-computed adversarial hashes to the database would be insufficient to cover all possible evasive variants, making this defense impractical and ineffective.
In essence, the technical deep dive reveals that the fundamental design choices of current CSS proposals, particularly their reliance on perceptual hashing, make them inherently susceptible to adversarial manipulation. The properties that make perceptual hashes useful for similarity detection also make them vulnerable to targeted evasion, where subtle changes can catastrophically alter the hash without affecting human perception.
Demo / Proof of Concept
▶ Watch: Attack generates diverse perturbations, defeating database expansion (15:00)
While the talk did not feature a live, interactive demonstration of the adversarial attack in action, the speaker effectively presented compelling visual evidence and quantitative results that served as a robust proof of concept for their claims.
Shubham Jain displayed several images to illustrate the effectiveness and subtlety of their adversarial perturbations. The audience was shown pairs of images: the original illegal content and its adversarially modified counterpart. The key observation highlighted by the speaker was that "our attack does not modify image significantly; the two images are still very similar." This visual evidence underscored the core premise of their work: an adversary can evade detection by client-side scanning systems with changes that are virtually imperceptible to the human eye.
Specifically, the presentation included:
- Images that had been successfully attacked for different detection thresholds, such as 30 and 70 (referring to the distance metric threshold for PDQ).
- A single original image, shown alongside its versions that had been attacked for four different, increasing thresholds. This demonstrated how the attack could be adapted to bypass various detection sensitivities, consistently producing visually similar, yet algorithmically distinct, outputs.
These visual examples, combined with the quantitative data on evasion rates (e.g., 97% at threshold 30, 67% at threshold 90) and false positive rates (1 in 3 safe images flagged at threshold 90), formed a powerful proof of concept. They empirically validated that the proposed client-side scanning systems, even when employing widely accepted perceptual hashing algorithms like Facebook's PDQ, are not robust. The demonstration clearly illustrated that the "small perturbation" required for evasion is indeed small enough to maintain the malicious intent and visual content of the original image, making it a viable strategy for a determined adversary.
Defensive Implications
▶ Watch: Apple confirms client-side scanning for CSAM detection (18:00)
The findings presented in this talk carry profound implications for anyone considering, designing, or implementing client-side scanning (CSS) systems, particularly those relying on perceptual hashing. The central message for defenders is clear: current CSS proposals, as designed, are fundamentally unreliable and easily evadable by a motivated adversary.
Here are the key defensive implications:
- Do Not Rely on Current Perceptual Hashing for Robust Detection: The research definitively shows that perceptual hashing algorithms, while useful for other applications like duplicate detection, are not robust enough for high-stakes illegal content detection in an adversarial setting. Defenders should abandon the notion that these algorithms can provide a secure and reliable means of preventing the spread of illicit material on E2EE platforms.
- Re-evaluate the Foundation of CSS: The vulnerability is not an implementation bug but an inherent characteristic of how perceptual hashes are designed to work. They are built to be somewhat forgiving to minor changes, which is precisely what adversaries exploit. Any future CSS proposals must fundamentally reconsider their underlying content identification mechanisms, moving beyond simple perceptual hashing.
- Understand the Trade-offs of Threshold Tuning: Increasing the detection threshold, a seemingly intuitive defensive measure, is not a viable strategy. As demonstrated, it leads to an unacceptable explosion in false positives, effectively breaking the system for legitimate users. This highlights the inherent conflict between robust detection and user privacy/usability within the current CSS paradigm.
- Database Expansion is Insufficient: Expanding the database with adversarial hashes is also not an effective defense. The ability of black-box adversarial ML to generate diverse and numerous perturbations means that maintaining a comprehensive database of all possible evasive variants is computationally infeasible and practically impossible.
- Focus on Alternative Research Directions: Instead of refining flawed CSS approaches, defenders and researchers should explore fundamentally different cryptographic or privacy-enhancing technologies that could address the challenge of illegal content without compromising E2EE or introducing easily bypassable scanning. This might include exploring more robust forms of content attestation, zero-knowledge proofs, or secure multi-party computation, though these also come with their own complexities and challenges.
- Acknowledge the False Sense of Security: Implementing CSS based on current proposals risks creating a false sense of security for the public and policymakers, suggesting that the problem of illegal content on E2EE is "solved," when in reality, it remains highly vulnerable to evasion. This could divert resources and attention from truly effective, privacy-preserving solutions.
- Consider the Broader Implications: Beyond the technical flaws, the talk's timely mention of Apple's client-side scanning announcement underscores the urgency. The deployment of such systems, even with the best intentions, could set a dangerous precedent without robust technical foundations, paving the way for mission creep and mass surveillance capabilities that are easily circumvented by those they aim to stop.
In summary, defenders must recognize that client-side scanning, as currently conceptualized using perceptual hashing, offers a technically weak solution to a complex problem. Relying on it would be akin to building a security wall with easily identifiable and exploitable weak points, while simultaneously compromising the privacy of all users. A fundamental re-evaluation of the approach is urgently needed.
Key Takeaways
- Client-Side Scanning (CSS) is not Robust: Proposed CSS systems, relying on perceptual hashing algorithms for illegal content detection on E2EE platforms, are highly vulnerable to adversarial evasion.
- Near-Perfect Evasion Achieved: Adversaries can modify over 99.9% of images with small, visually imperceptible perturbations to bypass detection, even in a black-box setting.
- Defenses are Ineffective: Traditional defensive strategies like increasing detection thresholds lead to unacceptable false positive rates (e.g., 1 in 3 legitimate images flagged at threshold 90), rendering the system unusable. Expanding databases with adversarial hashes is also futile due to the diversity of possible evasive perturbations.
- Perceptual Hashing is Fundamentally Flawed for this Task: The inherent properties of perceptual hashes, designed for similarity detection, make them susceptible to targeted manipulation that alters the hash without changing the content's illicit nature.
- Urgent Re-evaluation Needed: Given the deployment of CSS technologies by major companies, a critical re-evaluation of their technical robustness is imperative to prevent a false sense of security and avoid the introduction of easily bypassable, privacy-invasive systems.
About the Speaker(s)
Shubham Jain is a PhD student at Imperial College London, where his research focuses on computational privacy. He is affiliated with the computational privacy group at Imperial College London. His work, including the research presented in this talk, explores critical security and privacy challenges in modern digital systems, particularly concerning end-to-end encryption and content moderation. This particular study was a collaborative effort with a fellow PhD student and his advisor.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
Apple and government regulators spent 2021 trying to sell client-side scanning as a privacy-preserving silver bullet for E2EE platforms. Jain mathematically dismantles that illusion by proving Facebook's PDQ and similar perceptual hashes completely collapse against basic adversarial perturbations. This is mandatory viewing for anyone forced to evaluate CSS proposals.
Heather Calloway (CISO) — STRONG ACCEPT
If you are fielding questions from regulators, your board, or your general counsel about using client-side scanning to satisfy content moderation mandates on E2EE platforms, this is required reading. It mathematically proves the proposed technical 'compromise' is a failure, giving you the ammunition to kill flawed compliance architectures before they reach engineering.