Private Devices No Longer Private: The Broader Implications of Apple's Content Scanning Push

Nadim Kobeissi

Swiss Cyber Storm 2021 · Day 1 · Arena

Overview

In this thought-provoking talk at Swiss Cyber Storm, Nadim Kobeissi, a cryptographer and privacy advocate, dissects Apple's controversial proposal to implement client-side scanning for Child Sexual Abuse Material (CSAM) on iPhones. Delivered in the wake of Apple's initial announcement in August 2021 and its subsequent "pause," Kobeissi’s presentation serves as a critical examination of the technical, ethical, and societal ramifications of such a system. He argues passionately that while the stated goal of protecting children is laudable, the proposed technology represents a dangerous paradigm shift, fundamentally eroding the concept of private devices and setting a perilous precedent for mass surveillance.

Watch on YouTube · Slides

Visual summary for Private Devices No Longer Private: The Broader Implications of Apple's Content Scanning Push by Nadim Kobeissi
Visual summary for Private Devices No Longer Private: The Broader Implications of Apple's Content Scanning Push by Nadim Kobeissi

Key moments

  1. 0:00 Welcome and talk overview by host
  2. 4:00 Apple's problematic on-device photo scanning feature explained
  3. 6:40 Technical criticisms: Apple's misleading security promises
  4. 10:40 Apple's history of yielding to government demands
  5. 15:00 Civic criticism: Device privacy and mission creep concerns
  6. 18:40 The 'bad guys' scenario and dangerous precedent
  7. 23:00 Apple temporarily halts feature release due to pushback
  8. 25:00 Q&A: Communicating the long-term precedent to the public

Private Devices No Longer Private: The Broader Implications of Apple's Content Scanning Push

Speakers: Nadim Kobeissi

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=L8Scmhuqoco

Overview

In this thought-provoking talk at Swiss Cyber Storm, Nadim Kobeissi, a cryptographer and privacy advocate, dissects Apple's controversial proposal to implement client-side scanning for Child Sexual Abuse Material (CSAM) on iPhones. Delivered in the wake of Apple's initial announcement in August 2021 and its subsequent "pause," Kobeissi’s presentation serves as a critical examination of the technical, ethical, and societal ramifications of such a system. He argues passionately that while the stated goal of protecting children is laudable, the proposed technology represents a dangerous paradigm shift, fundamentally eroding the concept of private devices and setting a perilous precedent for mass surveillance.

Kobeissi's central thesis is that Apple's on-device scanning, even with its stated privacy-enhancing cryptographic features, opens the door to mission creep and government overreach. He challenges Apple's claims regarding auditability and resistance to governmental pressure, drawing on historical examples of the company succumbing to demands from various regimes. The talk aims to convince the audience that concerns raised by privacy advocates are not paranoid, but rather a necessary warning about the long-term consequences for civil liberties in an increasingly digital and interconnected world.

The core of the controversy revolves around the shift from server-side content moderation, common with cloud services like Google Photos, to on-device scanning of personal property. Kobeissi meticulously details how this seemingly subtle distinction transforms a personal device into a potential informant, capable of silently reporting its owner's content to authorities. This article will delve into Kobeissi's arguments, exploring the technical mechanics of Apple's proposal, its vulnerabilities, and the broader implications for privacy and freedom in a global context.

Background

▶ Watch: Welcome and talk overview by host (0:00)

The impetus for Apple's proposal stemmed from increasing pressure on technology companies to combat the proliferation of CSAM online. Organizations like the National Center for Missing and Exploited Children (NCMEC) in the United States have been vocal about the need for tech platforms to do more. This pressure culminated in Apple's August 2021 announcement, outlining three new measures aimed at protecting children, published in a document titled "Apple's Expanded Protection for Children."

While two of these measures – new tools in the Messages app to warn children and parents about sexually explicit photos, and updates to Siri and Search for more appropriate answers to sensitive queries – were largely uncontroversial and framed as enhanced parental controls, the third measure sparked widespread alarm. This was the proposed technology in iOS and iPadOS designed to detect known CSAM images stored in iCloud Photos by performing on-device matching using a database of known CSAM image hashes. This client-side scanning approach immediately distinguished Apple's plan from existing server-side content scanning employed by competitors like Google Photos or Microsoft OneDrive, where content is scanned on the service provider's servers after upload.

Kobeissi highlighted that the critical difference lies in the ownership and control of the scanning process. With Google Photos, scanning occurs on Google's servers, which are their property and governed by their terms of service. Apple's proposal, however, intended to turn every iPhone into a local scanner, silently and automatically checking users' private photos on their own device. This distinction formed the bedrock of the privacy community's opposition, arguing that it fundamentally alters the relationship between a user and their personal property, transforming a private device into a potential surveillance tool. The initial "rumblings" on platforms like Twitter, notably from Professor Matthew Green of Johns Hopkins University, signaled immediate concerns about the implications of such a client-side tool.

Key Findings

▶ Watch: Technical criticisms: Apple's misleading security promises (6:40)

Nadim Kobeissi systematically dismantled Apple's arguments and assurances regarding its CSAM scanning proposal, highlighting several critical findings:

  1. Deceptive Marketing and Technical Reality: Apple claimed the feature "only applies to photos that the user chooses to upload to iCloud Photos" and that they were "not looking for CSAM on iPhones." Kobeissi directly contradicted this, stating that the scanning technology works entirely offline and locally on the device. Enabling iCloud Photos was merely a "politeness" or "courtesy" from Apple; the core technology did not depend on it. This implied that Apple could, at any point, activate the local scanning universally, regardless of iCloud Photos settings.
  2. Lack of Auditability in a Closed Ecosystem: Apple's senior vice president of software engineering, Craig Federighi, asserted that "security researchers are constantly able to introspect what's happening in Apple's iPhone software." Kobeissi dismissed this as "ridiculous." He pointed out that iOS is one of the most closed operating systems, requiring researchers to apply for an exclusive, loaned "Apple Security Research Device Program" with stringent conditions. This closed environment makes genuine, independent auditability of such a sensitive system practically impossible, leaving users to trust Apple's claims without verification.
  3. Vulnerability to Government Pressure: Apple unequivocally stated that "governments [could not] force Apple to add non-CSAM images to the hash list" and that "Apple would refuse such demands." Kobeissi provided compelling counter-examples:
  • Apple dropped plans for encrypting iCloud backups in 2020 after complaints from the FBI.
  • FaceTime is disabled on iPhones sold in the UAE and was only recently enabled in Saudi Arabia due to government pressure against end-to-end encryption.
  • Apple pulled an app by Russian opposition leader Alexei Navalny under pressure from the Russian government, which involved armed gunmen occupying Google's offices.
  • Apple's entire cloud infrastructure in China is subject to tight controls by the Chinese government, fundamentally altering the security model for users there. These instances demonstrate a clear pattern of Apple capitulating to state demands, making their "refusal" claim highly suspect.
  1. Technical Flaws in NeuralHash: The core technology for image fingerprinting, NeuralHash, was designed to create a perceptual hash that remains consistent even if an image is cropped, compressed, or altered. However, Kobeissi showed examples where distinct images could generate the same NeuralHash, indicating potential for collisions. While Apple stated manual human review would prevent false positives from triggering reports to authorities, the possibility of collisions opens an attack vector for harassment. Malicious actors could craft benign images that match known CSAM hashes and send them to targets, potentially causing their devices to flag reports to Apple, leading to unwarranted scrutiny.
  2. The Inevitability of Mission Creep: This was perhaps Kobeissi's most critical civic criticism. He argued that once the technology is legitimized for CSAM, it will inevitably expand to other categories. He cited the Electronic Frontier Foundation's documentation of how the Global Internet Forum for Counter Terrorism expanded its CSAM database to include "terrorist content." More alarmingly, Kobeissi warned of state-specific censorship: if Apple implements scanning universally, authoritarian regimes (e.g., Saudi Arabia, Russia, China) could demand the scanning of content deemed "homosexual," "insulting to the king," or "terrorist" (e.g., journalistic or dissident material). This sets a dangerous precedent where a privacy-preserving company becomes an enforcer of diverse, often oppressive, state laws.
  3. The "Bad Guys" Problem and Universal Scanning: Kobeissi argued that criminals would simply disable iCloud Photos, circumventing the system's initial activation condition. Since the technology works locally and offline, the next logical step, from a law enforcement perspective, would be to demand that Apple enable the scanning universally, regardless of iCloud Photos settings. This would lead to a situation where all devices are constantly scanning, fundamentally eradicating the concept of a private digital safe.

These findings collectively painted a picture of a well-intentioned but deeply flawed proposal that threatened to undermine fundamental privacy rights under the guise of child safety.

Technical Deep Dive

▶ Watch: Civic criticism: Device privacy and mission creep concerns (15:00)

The proposed Apple CSAM scanning system operates through a multi-stage, client-side process designed to identify illegal content while attempting to maintain user privacy. At its core, the system relies on perceptual hashing and a series of cryptographic techniques.

First, every Apple device, specifically iPhones, would ship with a database containing NeuralHash fingerprints of known CSAM images, provided by organizations like NCMEC. This database would reside locally on the device, not in the cloud.

When a user takes a photo or stores an image in the Photos app (and critically, if iCloud Photos is enabled, though Kobeissi stressed the technology's offline capability), the device generates a NeuralHash for that image. NeuralHash is a sophisticated perceptual hashing algorithm designed to produce the same hash for images even if they undergo minor transformations such as compression, cropping, resizing, or color adjustments. This resilience is crucial for detecting CSAM that might have been slightly altered to evade detection by traditional cryptographic hashes (like SHA-256). The newly generated hash is then compared against the on-device database of known CSAM hashes.

Apple's design incorporates a threshold mechanism: a single matching photo does not trigger an immediate report. Only if a "relevant quantity" of photos match the known CSAM hashes would the system proceed. This threshold was intended to minimize false positives and prevent accidental flagging.

If the threshold is met, the system then leverages several cryptographic techniques. Apple claimed to use private set intersection, secret sharing, and end-to-end encryption to ensure that only the photos matching the CSAM database are revealed to Apple. The idea is that the device would securely communicate with Apple's servers, revealing only that a match threshold has been crossed and which specific images matched, without initially exposing the actual images themselves or the entire library. Upon receiving such a report, Apple's engineers would then manually decrypt and review the flagged images. Only after this human review process confirms the presence of actual CSAM would a report be sent to the authorities.

However, Kobeissi highlighted significant technical weaknesses:

  1. NeuralHash Collisions: Despite its sophistication, NeuralHash is not collision-proof. Kobeissi presented visual examples of perceptually different images that could generate the same NeuralHash. This vulnerability means that a malicious actor could intentionally craft benign images that produce hashes identical to known CSAM, then distribute these images. If a user receives enough of these crafted images (e.g., via messaging apps that auto-save to the photo library), their device could cross the threshold and trigger a report to Apple. While manual review is a safeguard, it introduces a vector for harassment and places an undue burden on Apple's human reviewers, potentially leading to human error. Apple vaguely mentioned "two types of hashing" and a "secret hash database" as countermeasures, but these details were not elaborated upon, leaving their effectiveness unverified.
  2. Closed iOS Ecosystem and Lack of Auditability: The entire system operates within Apple's highly proprietary and closed iOS operating system. Kobeissi argued that this fundamental characteristic renders Apple's claims of "auditability" by security researchers disingenuous. Unlike open-source software, the internal workings of iOS are opaque, making it exceedingly difficult for independent researchers to verify that the system functions precisely as Apple describes, or that it cannot be modified for broader surveillance purposes. The "Apple Security Research Device Program" is highly restrictive, granting limited access only to approved researchers under Apple's terms, which is far from true independent auditability.
  3. Offline Functionality and Mission Creep Potential: The most alarming technical aspect, according to Kobeissi, was that the core scanning mechanism itself operates entirely on-device and offline. This means the technology has the inherent capability to scan all photos on a device, regardless of whether they are synced to iCloud Photos or if the device is connected to the internet. Apple's decision to activate it only for iCloud Photos users was a policy choice, not a technical limitation. This offline capability means that if public or governmental pressure mounts, Apple could technically "flip a switch" to enable universal, always-on scanning across all devices, regardless of user consent or service usage, thereby making every iPhone a persistent, self-reporting surveillance tool.

In essence, while Apple designed a system with cryptographic elements intended to limit data exposure, the underlying client-side scanning and the closed nature of iOS create an architecture ripe for abuse and expansion beyond its stated purpose.

Demo / Proof of Concept

▶ Watch: The 'bad guys' scenario and dangerous precedent (18:40)

Nadim Kobeissi's presentation was an analytical and critical discussion of Apple's proposed CSAM scanning system, rather than a demonstration of a working exploit or a proof-of-concept tool. The talk meticulously described how Apple's system was designed to function, highlighted its technical and civic vulnerabilities, and critiqued Apple's public statements. There was no live demonstration or technical proof of concept presented within the talk itself. Instead, Kobeissi relied on logical arguments, theoretical attack vectors (such as NeuralHash collisions for harassment), and historical examples of Apple's actions to support his claims.

Defensive Implications

▶ Watch: Q&A: Communicating the long-term precedent to the public (25:00)

The defensive implications stemming from Nadim Kobeissi's analysis are primarily rooted in policy, advocacy, and user awareness, rather than direct technical countermeasures for an already implemented system. Since the proposed system would be baked into the operating system, individual users have limited direct technical "defenses" once it's deployed. However, the talk provides crucial insights for individuals, the tech community, and policymakers:

  1. Understand the Precedent: The most significant defensive action is to recognize and actively resist the establishment of this kind of client-side scanning as a legitimate industry practice. Apple, as a trendsetter (e.g., removing headphone jacks, chargers), has immense power to normalize such technologies. Preventing its widespread adoption is paramount to safeguarding future digital rights.
  2. Advocacy and Public Pushback: As demonstrated by the temporary halt of Apple's plans, collective public outcry and expert advocacy can be effective. Users and privacy advocates must continue to educate the public, engage with policymakers, and support organizations like the Electronic Frontier Foundation (EFF) and Open Privacy that campaign against such surveillance technologies. Kobeissi's involvement in the "Apple privacy letter" signed by over 9000 people, including Edward Snowden, exemplifies this crucial defensive strategy.
  3. Critical Evaluation of Corporate Claims: Users should adopt a skeptical stance toward claims of "privacy-preserving" features in closed-source systems, especially when those systems introduce new forms of content scanning. Apple's assurances of auditability and resistance to government pressure were thoroughly debunked by Kobeissi, illustrating the need for independent verification and transparency.
  4. Awareness of Mission Creep: Defenders must remain vigilant against the inevitable mission creep that such a system would enable. Once a mechanism for client-side scanning is in place for CSAM, the pressure to expand its scope to include "terrorist content," "harmful but legal content," or state-specific censorship (e.g., homosexuality, political dissent) will be immense. The defense here is proactive resistance to any expansion of the hash database beyond its initial, narrowly defined purpose.
  5. Understanding Offline Capabilities: Users should be aware that the underlying technology for on-device scanning works offline and is not inherently tied to iCloud Photos. While disabling iCloud Photos might currently prevent the feature from activating, this is a policy decision by Apple, not a technical limitation. This means that a future update could potentially enable universal scanning regardless of user settings. There is no reliable "off switch" from the user's perspective if the core technology is integrated into the OS.
  6. Demand Openness and Transparency: A long-term defensive strategy involves pushing for greater transparency and, ideally, open-source components for any security-critical or privacy-impacting features within operating systems. This would allow for true independent auditability and verification that such systems are not being misused or expanded without public knowledge.

In summary, the "defensive implications" are less about personal technical fixes and more about a societal responsibility to understand, challenge, and prevent the normalization of on-device surveillance that fundamentally undermines the privacy of personal digital devices.

Key Takeaways

  • On-Device Scanning is a Paradigm Shift: Apple's proposed client-side scanning for CSAM represents a fundamental departure from traditional server-side content moderation, transforming private devices into potential surveillance tools.
  • Apple's Claims are Debunkable: Assurances regarding auditability, immunity to government pressure, and limited scope of scanning are contradicted by the closed nature of iOS and Apple's history of capitulating to state demands.
  • NeuralHash Vulnerabilities Exist: The NeuralHash technology, while robust, is susceptible to collisions, opening a vector for harassment and potential false positives, even with manual review.
  • Mission Creep is the Gravest Threat: The most significant long-term danger is the inevitable mission creep, where a system designed for CSAM detection can be expanded under pressure to include other forms of "objectionable" or politically sensitive content, leading to global censorship and surveillance.
  • Disabling iCloud Photos is Not a Panacea: The underlying scanning technology works offline; Apple's decision to link it to iCloud Photos is a courtesy, not a technical barrier, making future universal activation a distinct possibility.
  • Protecting Privacy Requires Vigilance: Even with noble intentions like child safety, the implementation of such technologies demands critical scrutiny and sustained public pushback to safeguard civil liberties and the concept of private digital property.

About the Speaker(s)

Nadim Kobeissi is a prominent cryptographer and privacy advocate. He is recognized for his deep understanding of cryptography not just as a technical discipline, but as a foundational building block for modern internet society and a crucial element in protecting civil liberties. Kobeissi has applied his expertise beyond academic research, having founded a startup focused on communication, demonstrating his interest in practical applications of secure technologies. Based out of Paris or Beirut, he is deeply engaged in the broader discourse surrounding privacy, as evidenced by his critical stance on Apple's content scanning proposal and his active participation in advocacy efforts, such as co-authoring the "Apple privacy letter" signed by thousands, including Edward Snowden. His work consistently highlights the societal implications of technological developments, urging a long-term perspective on privacy.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Kobeissi correctly identifies Apple's NeuralHash CSAM scanning for what it was: a backdoor into endpoint privacy dressed up as child safety. He strips away the Cupertino marketing and hammers the cryptographer consensus that client-side scanning is just distributed surveillance. This is the exact kind of systematic teardown that forced Apple to eventually kill the feature.

Heather Calloway (CISO) — STRONG ACCEPT

Kobeissi cuts through the cryptographic debates surrounding Apple's CSAM scanning to expose the massive institutional and geopolitical risks of client-side content moderation. This is an essential watch for any leadership team managing a global mobile fleet, as it directly challenges assumptions about corporate data sovereignty and executive privacy.

→ Top-rated talks at Swiss Cyber Storm 2021

All talks from Swiss Cyber Storm 2021