AI and the Moving Line of Privacy

AI and the Moving Line of Privacy

Speakers: Sounil Yu (Chief AI Safety Officer, Knostic); Jameeka Aaron (CISO, Headspace); Yassir Abousselham (CISO, Calendly)

Conference: Black Hat MEA

YouTube: https://www.youtube.com/watch?v=CsDyMlK1vpo

Overview

In an insightful panel discussion at Black Hat MEA, industry leaders Sounil Yu, Jameeka Aaron, and Yassir Abousselham explored the profound shifts AI introduces to the landscape of privacy. The session, moderated by Abousselham, delved into how traditional assumptions about machine data processing are being fundamentally challenged by AI's ability to infer, make decisions, and "remember" interactions. This discussion highlighted a critical transition from concerns focused solely on data privacy to the more nuanced and challenging realm of knowledge privacy, emphasizing that AI systems can derive sensitive insights even from anonymized data.

The panel underscored that the advent of AI necessitates a complete re-evaluation of our trust in machines, particularly given their emergent capabilities for bias, judgment, and a lack of shame—qualities previously attributed only to humans. With examples ranging from enterprise Data Loss Prevention (DLP) to highly sensitive mental health applications like Headspace's chatbots, the speakers illustrated the new risks and ethical dilemmas facing CISOs and privacy professionals. The core message was clear: AI is not merely automating existing processes; it is redefining the very essence of what privacy means and how it can be protected in an increasingly intelligent digital world.

This article dissects the panel's critical insights, examining the technical underpinnings of AI's privacy impact, the strategic implications for defenders, and the urgent need for new governance frameworks that extend beyond mere data protection to safeguard inferred knowledge.

Background

▶ Watch: Introduction: AI's impact on privacy (0:00)

For decades, the security industry has operated under a set of assumptions regarding how machines process data. Systems, servers, and applications were largely understood as entities that followed explicit rules, executing commands without independent judgment, memory, or bias. This foundational belief underpinned many security controls, including Data Loss Prevention (DLP), where organizations felt comfortable allowing machines to handle sensitive data, assuming they wouldn't misuse it or "pass judgment." The prevailing thought was that machines, devoid of human emotions like shame or ethical considerations, posed a limited risk in terms of privacy invasion beyond accidental exposure.

However, the rise of advanced AI systems, particularly Large Language Models (LLMs) and agentic AI, has shattered these long-held assumptions. As Yassir Abousselham noted, these new systems are "able to infer, they're able to make decisions, they're definitely able to remember conversations that they have with each one of us." This emergent capability creates a "black hole" of understanding regarding what truly happens behind the scenes. Sounil Yu emphasized that the reason we tolerate machines "invading our privacy" (e.g., in DLP) is the assumption they lack human flaws. With AI, this changes, forcing a re-evaluation of what we permit these machines to do when they can exhibit or seem to exhibit bias, judgment, and memory.

Jameeka Aaron further contextualized this challenge with her experience at Headspace, a company dealing with "highly, highly sensitive data" related to mindfulness, meditation, and mental health. The expectation of privacy when talking to a human therapist is absolute, but when interacting with an AI chatbot like Headspace's EB, the line blurs. While Headspace employs anonymization and de-identification to protect data, the AI's ability to be "empathetic" and "agreeable"—often to the point of being detrimental—raises profound questions. For instance, an AI might not recognize or act upon suicidal ideation in the same way a human therapist would, despite both being mechanisms ostensibly designed to protect a user's well-being and privacy. This highlights a critical deficiency in current privacy paradigms: the focus has historically been on explicit data, not the inferred insights.

This distinction led Sounil Yu to introduce two major classes of privacy: data privacy and knowledge privacy. Data privacy, he explained, pertains to explicit Personally Identifiable Information (PII) like gender or sexual orientation. Knowledge privacy, in contrast, is "actually inferred from our data" and is "far more valuable." He provided examples: an AI inferring suicidal thoughts from conversations with a mental health chatbot, or inferring a corporate merger from meeting schedules in a tool like Calendly, even without explicit mention of such events. The fundamental problem, as both Yu and Aaron articulated, is that while compliance frameworks like GDPR focus heavily on data privacy, there are "no good controls for knowledge privacy." This gap—between what legislation addresses and what AI can infer—forms the core of the privacy challenge in the age of AI.

Key Findings

▶ Watch: Distinguishing data privacy from knowledge privacy (5:45)

The panel's discussion yielded several critical findings that redefine the discourse on privacy in the AI era:

1. AI's Inferential Capabilities Shatter Traditional Privacy Assumptions: The most significant finding is that AI systems, unlike traditional machines, can infer, remember, and even exhibit qualities akin to judgment and bias. This capability fundamentally breaks the long-held assumption that machines are neutral processors of information. This shift moves the privacy goalpost from merely protecting raw data to safeguarding the insights and knowledge that can be derived from it.

1. The Critical Distinction Between Data Privacy and Knowledge Privacy: Sounil Yu eloquently articulated the difference between data privacy (protecting explicit PII) and knowledge privacy (protecting inferences drawn from data). The panel unanimously agreed that knowledge privacy is "far more valuable" and represents the true frontier of AI-related privacy concerns. Current compliance frameworks, such as GDPR, are largely ill-equipped to address knowledge privacy, leaving a significant gap in protective measures.

1. The Nuance of AI Model Training and User Data: A key debate centered on whether frontier model providers (e.g., OpenAI, Gemini) use user inputs to train their models. Jameeka Aaron expressed skepticism about contractual assurances, believing that advances in AI necessitate the use of user data, albeit perhaps for demographic understanding rather than specific personal details. Sounil Yu offered a nuanced perspective, distinguishing between core model training (a massive, infrequent effort where untrusted user input could "poison" the data) and fine-tuning (adjusting a model's behavior or "syllabus" without fundamentally altering its core knowledge base). Yu argued that while fine-tuning might use user input to adapt, responsible providers would avoid using untrusted input for core training due to the risk of data poisoning. He further posited that the greater risk might lie in the vulnerabilities of the applications running the AI models, rather than the models themselves being retrained on sensitive user data.

1. AI Agents Introduce Unprecedented Risks Due to Autonomy and Lack of Social Norms: The emergence of AI agents with "decision autonomy" presents a new class of privacy risks. These agents can "process inputs, make sense of it, and then determine what's the next course of action," but critically, they lack an understanding of consequences, shame, or human social and cultural norms. Jameeka Aaron highlighted their inability to read "context cues" or "body language," rendering them less effective than humans in sensitive roles like therapy and potentially leading to "nefarious experiences" for children.

1. AI Systems Operate Like "Teenagers": Sounil Yu provided a compelling analogy: most current AI systems operate like teenagers. They lack a fully developed "prefrontal cortex"—the human brain's center for risk management, understanding social norms, and feeling shame. This implies that AI systems can perform "stupid" actions without understanding why they shouldn't, much like a teenager. This fundamental limitation underscores the need to calibrate expectations and implement robust human oversight.

1. Significant Gaps in Governance and Technical Tooling for Knowledge Privacy: The panel concluded that there are currently "very few, if any, technical tools" specifically designed to enforce knowledge privacy. The "inference problem" remains the biggest challenge, highlighting the inadequacy of existing governance frameworks. This necessitates a greater reliance on "human guardrails in the form of ethics and morals" to bridge the technical and legislative void.

Technical Deep Dive

▶ Watch: Debate: Do AI models train on user data? (9:58)

The discussion offered a deep dive into several technical aspects that underpin the shifting privacy landscape in the age of AI.

At the heart of the problem is the fundamental nature of modern AI, particularly Large Language Models (LLMs), which Sounil Yu explicitly referred to as inference engines. Unlike traditional rule-based systems, these models are designed to derive meaning, patterns, and, crucially, inferences from vast datasets. This capability is what transforms raw data privacy concerns into the more complex challenge of knowledge privacy. For example, a traditional DLP system might flag a document containing specific keywords or PII. An AI-powered system, however, could analyze communication patterns, meeting schedules (as with Calendly), and travel itineraries to infer a potential corporate merger, even if no explicit mention of a merger exists in the data. This inference capability is not a bug; it's a core feature of how LLMs operate, making them powerful but also potent privacy threats.

The panel also delved into the technical nuances of how AI models are developed and updated, particularly concerning the use of user data. Sounil Yu distinguished between two forms of model training:

1. Core Curriculum Training: This refers to the initial, massive effort undertaken to build the foundational knowledge of an LLM. This process involves ingesting enormous, often curated, datasets. Yu argued that a "responsible frontier model provider" would be highly cautious about incorporating "untrusted user input" into this core training. The primary technical reason for this is the risk of data poisoning, where malicious or low-quality user data could degrade the model's overall performance, introduce biases, or cause it to "hallucinate" incorrect information. From a technical standpoint, poisoning a massive, carefully constructed foundational model would be "shooting themselves in the foot" for the provider.
2. Fine-tuning: This is a more common and less resource-intensive form of training where a pre-trained model is adapted to specific tasks or user interactions. Yu used the analogy of a professor "fine-tuning [their] syllabus" to address student misconceptions, rather than rewriting the entire course content based on individual homework assignments. In fine-tuning, the model's parameters are slightly adjusted based on new data, allowing it to improve its performance or adapt its conversational style without fundamentally altering its core knowledge base. While user input might be used for fine-tuning, the impact on the model's overall knowledge and the risk of widespread data poisoning are generally lower than with core training. This distinction is crucial for understanding how user data might (or might not) be incorporated into AI systems.

A significant technical development highlighted was the emergence of AI agents with decision autonomy. This refers to AI systems that can "process your its inputs, make sense of it, and then determine what's the next course of action" without constant human oversight. This "agentic AI workflow" is a departure from reactive chatbots, enabling AI to take proactive steps. The technical challenge here is that these agents, despite their sophisticated decision-making algorithms, lack intrinsic mechanisms for understanding human-centric concepts like "consequences" or "shame." Their decision trees and reward functions are not typically encoded with complex social or cultural norms. This technical limitation means that an agent, for instance, might delete critical data and express "sorrow" but not genuinely "feel shame" in a human sense, making its actions unpredictable from a human ethical standpoint.

Jameeka Aaron further elaborated on the technical limitations regarding cultural sensitivity and context cues. Current AI, she noted, struggles to "read body language" or understand the subtle nuances of human interaction that are critical for tasks like therapy. This is because the underlying neural networks, while powerful at pattern recognition, are not yet equipped with the equivalent of human sensory perception and emotional intelligence. The ambition for future AI development includes the creation of world models or mental models, which would give AI a more comprehensive understanding of its environment and the implications of its actions—analogous to the human "prefrontal cortex" that develops around age 25 and is responsible for risk management, social norms, and understanding consequences. Until then, the technical architecture of AI means it operates more like a "teenager," lacking a fully developed capacity for self-regulation and ethical reasoning.

These technical aspects collectively illustrate that AI's impact on privacy is not just a policy problem but is deeply rooted in how these systems are designed, trained, and deployed, demanding a technical understanding to formulate effective defensive strategies.

Demo / Proof of Concept

▶ Watch: Why AI systems operate like teenagers (20:00)

The panel discussion focused on theoretical and practical implications of AI on privacy, drawing on the speakers' extensive industry experience. There was no specific demonstration or proof of concept presented during the talk.

Defensive Implications

▶ Watch: CISO's advice: Take a risk-based approach to AI (22:50)

The insights shared by Sounil Yu and Jameeka Aaron provide critical guidance for security professionals navigating the complex intersection of AI and privacy. Defenders must fundamentally re-evaluate their strategies, moving beyond traditional data-centric approaches.

1. Re-evaluate Core Assumptions about Machine Trust: CISOs must discard the outdated assumption that machines are neutral, rule-following entities devoid of judgment or memory. With AI's ability to infer, make decisions, and "remember" interactions, every AI system, whether internal or third-party, must be treated with a higher degree of scrutiny regarding its potential privacy impact. This means questioning the implicit trust placed in systems like DLP when AI capabilities are integrated.

1. Prioritize Knowledge Privacy over Data Privacy: The most significant defensive shift required is to focus on knowledge privacy. While adherence to data privacy regulations like GDPR is still crucial, defenders must understand that AI's primary threat is the inference of sensitive knowledge, not just the exposure of raw data. This necessitates developing new risk assessment frameworks that identify what inferences an AI system could make and the potential harm from that inferred knowledge, even if the underlying data is anonymized or de-identified.

1. Scrutinize AI Model Training and Data Handling Practices: While Sounil Yu downplayed the risk of frontier models being poisoned by untrusted user data in core training, CISOs still need to engage vendors on their fine-tuning practices and overall data governance. Understanding whether and how user input influences model behavior is vital. Implement contractual agreements (e.g., BAAs in healthcare) that explicitly address data usage for training and fine-tuning, and seek transparency from providers on their data hygiene practices.

1. Strengthen Application-Layer Security for AI Implementations: As Sounil Yu pointed out, "the bigger problem is actually not... the company using your data to train their model. The bigger problem is a much more simpler one which is the application that the model is running on top of... still full of vulnerabilities." This highlights the enduring importance of traditional security controls. Defenders must prioritize third-party risk management, insider threat prevention, and robust vulnerability management for all applications that integrate or interact with AI models. The AI model itself might be secure, but the surrounding ecosystem (APIs, data pipelines, user interfaces) remains a fertile ground for exploitation.

1. Implement Human Guardrails and Ethical Frameworks: Given the current lack of "technical means for enforcing knowledge privacy," human oversight becomes paramount. Organizations must establish strong ethical guidelines and moral frameworks for AI deployment. This includes defining acceptable uses, setting boundaries for autonomous decision-making by AI agents, and ensuring human review processes for sensitive AI outputs. These "human guardrails" are essential where technical controls are immature or non-existent.

1. Calibrate Expectations for AI Agent Behavior: Defenders must recognize that current AI agents "operate like a teenager," lacking a fully developed "prefrontal cortex" for risk management or understanding shame. This means AI agents with decision autonomy should be deployed with extreme caution, especially in sensitive contexts. Implement strict controls on their scope of action, ensure robust logging and auditing of their decisions, and design for human intervention points to mitigate unintended consequences.

1. Adopt a Risk-Based Approach to AI Adoption: Jameeka Aaron advocated for a "calculated risk" approach, especially in fields like healthcare where AI promises significant benefits. CISOs must conduct thorough risk assessments for each AI use case, weighing potential benefits against privacy, security, and ethical risks. This involves understanding the type of data involved, the inferences that could be drawn, the level of autonomy granted to the AI, and the potential impact of failures or misuse. The decision to adopt AI should be a conscious, informed balancing act.

1. Advocate for and Contribute to New Governance and Tooling: The panel identified a significant gap in governance and technical tools for knowledge privacy. Defenders should actively participate in industry dialogues, collaborate with policymakers, and demand better solutions from AI developers. This includes pushing for standards around AI explainability, transparency in inference processes, and the development of new privacy-enhancing technologies specifically designed for inferred knowledge.

Key Takeaways

• Privacy is Shifting from Data to Knowledge: AI fundamentally redefines privacy, moving the focus from protecting explicit data (data privacy) to safeguarding inferred insights and patterns (knowledge privacy), for which current controls are inadequate.
• AI Breaks Traditional Assumptions of Machine Neutrality: AI's ability to infer, remember, make decisions, and exhibit bias or judgment shatters the long-held belief that machines are purely objective; this necessitates a complete re-evaluation of trust.
• AI Agents Pose Unique Risks Without Social Norms: AI agents with decision autonomy lack human concepts of shame, consequences, and cultural sensitivity, requiring stringent human guardrails and careful calibration of expectations.
• Application Security is as Crucial as Model Security: While concerns about model training persist, the immediate and often greater risk lies in vulnerabilities within the applications and ecosystems running AI models, highlighting the continued importance of traditional security practices.
• Human Ethics are the Primary Defense for Knowledge Privacy: In the absence of robust technical tools for knowledge privacy, ethical frameworks, moral responsibilities, and human oversight are paramount to guide AI's development and deployment.
• A Calculated, Risk-Based Approach is Essential for AI Adoption: Organizations, particularly in sensitive sectors like healthcare, must undertake thorough risk assessments to balance AI's transformative benefits against its inherent privacy and security challenges.

About the Speaker(s)

Sounil Yu serves as the Chief AI Safety Officer for Knostic, bringing a deep understanding of AI's potential impacts on security and ethics. His contributions to the panel highlighted the technical nuances of AI model training, the distinction between data and knowledge privacy, and the ethical considerations surrounding AI's emergent capabilities.

Jameeka Aaron is the Chief Information Security Officer (CISO) at Headspace, a company that deals with highly sensitive user data related to mental health and wellness. Her perspective provided crucial real-world context on managing privacy expectations in a consumer-facing, data-rich environment, particularly with the deployment of AI chatbots, and underscored the importance of a calculated risk approach to AI adoption.

Yassir Abousselham, the CISO at Calendly, moderated the panel. His role in guiding the discussion and framing the challenges of AI in the context of traditional security assumptions was instrumental. His company, Calendly, also served as a practical example in the discussion, illustrating how seemingly innocuous scheduling data could lead to significant knowledge privacy concerns.