AppleStorm - Unmasking the Privacy Risks of Apple Intelligence

AppleStorm - Unmasking the Privacy Risks of Apple Intelligence

Speakers: Yoav Magid

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=BNmJ3qBP9GE

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Yoav%20Magid%20-%20AppleStorm%20-%20Unmasking%20the%20Privacy%20Risks%20of%20Apple%20Intelligence.pdf

Overview

AppleStorm is a research project by Yoav Magid that investigates the privacy implications of Apple Intelligence—Apple's on-device and cloud AI platform launched in late 2024. The research was sparked by a personal anomaly: Siri referenced a classified document title from a Notion workspace when it had no obvious reason to have that information. That observation led to a systematic investigation of how Apple Intelligence accesses, indexes, and transmits user data, and what the actual privacy boundaries are between on-device processing and Apple's servers.

The talk covers Apple's stated privacy architecture (Private Cloud Compute, on-device inference), the reality of what data actually flows where, threat modeling for AI-integrated operating systems, and several specific findings about information leakage and over-collection that Apple had not publicly documented. The work is relevant to any organization that deploys Apple devices in an enterprise environment and assumes on-device AI stays on-device.

Background

▶ Watch: Siri and AI assistant privacy implications (0:32)

Apple Intelligence was announced at WWDC 2024 and began rolling out with iOS 18, iPadOS 18, and macOS Sequoia. It is Apple's integrated AI layer, encompassing a redesigned Siri, a Writing Tools API, Image Playground, a Summarization engine, and deep integration with third-party apps via the App Intents framework.

Apple made strong privacy claims about Apple Intelligence from the outset. The primary claim is that most AI processing happens on-device using a local model (the on-device Apple Intelligence model). For tasks requiring more compute, Apple routes queries to Private Cloud Compute (PCC)—Apple-operated servers running a specially hardened OS that Apple claims provides "no persistent access" and "cryptographic proof" that queries are not retained or logged.

The research question Magid set out to answer was: does the system behave as advertised? Specifically: what triggers on-device vs. cloud processing, what context does Siri collect from the device before formulating a request, can third-party app data be sent to Apple servers, and does the user have meaningful visibility into or control over this?

Key Findings

▶ Watch: Recommended mitigations and Apple's response (4:15)

App Intents and cross-app context collection. Apple Intelligence integrates with third-party apps via App Intents—a framework that allows apps to expose actions and semantic entities (documents, contacts, tasks, messages) to the system. When the user invokes Siri with a request that touches a third-party app, Apple Intelligence may collect semantic metadata from that app—including titles, descriptions, and content snippets—to formulate the response. Magid found that this collection is broader than Apple's documentation implies: apps that implement App Intents expose their entity index to the system without per-query user consent.

The Notion case: document titles transmitted to Apple servers. The triggering observation was confirmed as a real behavior: Siri accessed Notion document titles through the App Intents entity index. When the query was processed using PCC (rather than fully on-device), those document titles were transmitted to Apple's servers as part of the request context. Apple's privacy documentation does not clearly disclose that third-party app data provided through App Intents is subject to PCC transmission.

Siri context window includes sensitive personal data. When Siri formulates a request, it builds a "context window"—similar to a prompt for a language model—that includes information about the user's current app state, recent documents, messages, emails, and calendar events. Magid instrumented the system to capture the context window content and found it regularly contained information the user had not explicitly shared with Siri, including email body text, calendar event details, and contact information.

Private Cloud Compute attestation is not user-verifiable in practice. Apple's PCC design uses a cryptographic attestation mechanism to allow devices to verify the code running on PCC servers. However, this verification is performed automatically by the device software—users cannot independently verify it. The security rests entirely on Apple's transparency log (which publishes PCC software versions) and the assumption that Apple does not deploy a different version to process user requests than the version published in the log. Magid notes this is a strong architectural control but argues it is not equivalent to user-level auditability.

Writing Tools process document content on PCC. The Writing Tools feature (rewrite, proofread, summarize) processes document content on PCC for documents above a certain size or complexity. The content of work documents—including drafts, internal memos, and source code—can be transmitted to Apple's servers when Writing Tools is invoked, regardless of whether the user believes they are using an on-device AI feature.

Insufficient disclosure in Settings. The Privacy & Security settings in iOS 18 and macOS Sequoia do not provide a clear, per-feature breakdown of which Apple Intelligence features process data on-device vs. on PCC. Users who opt into Apple Intelligence are not presented with a granular disclosure of what data each sub-feature may transmit.

Technical Deep Dive

▶ Watch: Private Cloud Compute architecture and its attack surface (5:15)

Apple Intelligence Architecture

Apple Intelligence uses a two-tier inference architecture:

1. On-device model: A 3-billion parameter model running locally using the Apple Neural Engine (ANE). This model handles simple text generation, classification, and system tasks where latency requirements preclude a network round-trip.

1. Private Cloud Compute (PCC): A larger model running on Apple silicon server hardware. PCC runs a hardened OS with no persistent storage and no administrative access. Requests are processed and results returned; Apple claims no logs are retained. The PCC software is published to a public transparency log and Apple invites security researchers to verify it.

App Intents Entity Index

App Intents allows developers to define AppEntity types that represent objects in their app (e.g., a NoteEntity with a title and body, or a ContactEntity). These entities are indexed by the system in a local Spotlight-like database. When Siri processes a query, it performs an intent classification step, determines which apps and entity types are relevant, and queries the entity index.

Magid's investigation showed that the entity index query results—including actual entity content—are bundled into the request context. For on-device inference, this stays local. For PCC inference, the context is transmitted as part of the inference request payload. The determination of whether to use on-device or PCC inference is made by the Apple Intelligence runtime based on model capability requirements, not on the sensitivity of the data in the context.

Request Interception Methodology

Magid used a combination of:

• Network traffic analysis (mitmproxy with a custom root certificate installed on a test device) to capture Apple Intelligence HTTP/2 requests to PCC endpoints.
• Private framework hooking (via Frida) to instrument the Apple Intelligence daemon (intelligenced) and capture context window construction before encryption.
• System log analysis using log stream with appropriate subsystem filters to understand request routing decisions.

This allowed him to reconstruct exactly what data left the device for PCC requests and correlate it with specific user actions.

Findings on Siri Context Window Size

In representative test scenarios, Magid measured context windows that included:

• The full subject and body of the last several received emails
• Calendar event titles and attendee lists for the next 48 hours
• The titles and first paragraphs of recently opened documents
• Third-party app entity data matching the semantic context of the query

The context window size was unbounded by any user-configurable limit.

Demo / Proof of Concept

▶ Watch: On-device vs. cloud processing boundaries and risks (17:29)

The talk includes live demonstrations:

• Notion document title leakage: A test Notion workspace with a deliberately unusual document title is created. Siri is invoked with a question that does not explicitly reference Notion. The PCC request is captured by mitmproxy, showing the Notion document title in the request payload.
• Writing Tools PCC transmission: A medium-length work document is opened in Pages. Writing Tools (Proofread) is invoked. The mitmproxy capture shows the document content being transmitted to a PCC endpoint.
• Email context in Siri request: A test email with sensitive content is sent to the test device. A Siri query tangentially related to the email's topic is issued. The captured context window contains the email body text.

Defensive Implications

▶ Watch: Network traffic analysis revealing Apple Intelligence data flows (27:29)

For individuals:

• Disable Apple Intelligence features (or Siri entirely) if you handle sensitive personal or professional information on your Apple devices.
• Understand that "on-device AI" is not a guarantee—specific features (Writing Tools, complex Siri queries) route to PCC.
• Review App Intents permissions in Settings > Privacy & Security and restrict which apps can expose data to Siri/Apple Intelligence.

For enterprise security teams:

• Apple Intelligence represents a new data exfiltration surface: an employee invoking Writing Tools on a confidential document may inadvertently transmit its contents to Apple's infrastructure.
• MDM (Mobile Device Management) profiles can disable Siri and Apple Intelligence features via restrictions. Organizations handling sensitive data should evaluate whether to enable these restrictions.
• Add Apple Intelligence endpoints to your DLP (Data Loss Prevention) monitoring lists.
• Update acceptable use policies to address AI assistant usage on corporate devices.

For Apple:

• Per-feature disclosure of on-device vs. PCC processing should be presented clearly to users before they use each feature for the first time.
• App Intents entity data transmission should require explicit per-query user consent or at minimum a prominent disclosure.
• A device-side log of PCC requests (analogous to location access logs) would substantially improve user visibility.

Key Takeaways

1. Apple Intelligence's privacy boundary between the device and Apple's servers is less clear than Apple's marketing implies—PCC transmission is triggered by normal user actions without explicit notification.
2. The App Intents framework creates a system-wide cross-app context collection mechanism that operates below the user's awareness.
3. Private Cloud Compute is a meaningful architectural improvement over conventional cloud AI, but "architecturally private" is not the same as "the data never leaves your device."
4. Enterprise security teams should treat AI-integrated operating systems as a new category of data exfiltration risk, distinct from traditional network-level DLP.
5. The research methodology—instrumenting OS-level AI frameworks using Frida and network proxies—provides a template for auditing other AI platform privacy claims.

About the Speaker(s)

▶ Watch: Vulnerabilities discovered and CVE details (37:29)

Yoav Magid is a security researcher with a focus on mobile and AI platform security. His research into Apple Intelligence began from a practical privacy concern—an unexpected Siri behavior involving a Notion document—and evolved into a systematic platform audit. The talk represents hands-on instrumentation work combining mobile security analysis techniques (Frida hooking, network interception) with privacy policy analysis. Magid's findings were presented to the DEF CON audience with an emphasis on empowering users and enterprise defenders to make informed decisions about AI feature adoption.