Potential Pitfalls Protecting Patient Privacy
Brian Martin (Head of AI, R&D Information Research · AbbVie)
AI Village @ DEF CON 29 · Day 3 · AI Village
Overview
In this thought-provoking session at AI Village @ DEF CON 29, Brian Martin, Head of AI, R&D Information Research at the global biopharmaceutical company AbbVie, presented a critical ethical and legal dilemma facing the healthcare and AI communities: how to reconcile individual patient privacy rights with the collective benefit derived from AI-generated knowledge. The talk, framed as a question to the audience rather than a presentation of solutions, explored the "potential pitfalls protecting patient privacy" when artificial intelligence (AI) algorithms extract and infer new knowledge from consented patient data.

Key moments
- 0:00 Introduction: The patient privacy challenge in pharma AI
- 2:10 Defining AI and knowledge graphs in R&D
- 5:50 Core problem: Patient consent retraction and AI-derived knowledge
- 9:40 Legal vs. moral questions of the 'right to be forgotten'
- 11:30 Moral obligation to share knowledge vs. individual rights
- 14:05 Q&A: Differential privacy as a solution?
- 16:15 Q&A: Impact on rare diseases with limited data
- 18:40 Q&A: Data turnover vs. aggregate knowledge graphs
Potential Pitfalls Protecting Patient Privacy
Speakers: Brian Martin, Head of AI, R&D Information Research, AbbVie
Conference: AI Village @ DEF CON 29
YouTube: https://www.youtube.com/watch?v=kzsVFQnOas8
Overview
In this thought-provoking session at AI Village @ DEF CON 29, Brian Martin, Head of AI, R&D Information Research at the global biopharmaceutical company AbbVie, presented a critical ethical and legal dilemma facing the healthcare and AI communities: how to reconcile individual patient privacy rights with the collective benefit derived from AI-generated knowledge. The talk, framed as a question to the audience rather than a presentation of solutions, explored the "potential pitfalls protecting patient privacy" when artificial intelligence (AI) algorithms extract and infer new knowledge from consented patient data.
Martin highlighted a future challenge that the biopharmaceutical industry is proactively trying to address: does the right to be forgotten (RTBF), as enshrined in data protection regulations, extend beyond raw patient data to the sophisticated, derivative insights and knowledge generated by AI? This question carries immense weight, balancing the fundamental rights of individuals to control their personal information against the moral imperative to advance scientific discovery and develop life-saving therapeutics. The discussion underscored the urgent need for a framework that navigates this complex interplay, ensuring both legal compliance and ethical responsibility in the rapidly evolving landscape of AI-driven healthcare research.
The talk emphasized that while the industry is committed to patient well-being, the lack of clear legal precedent creates a "nebulous, gray area" where data scientists and legal teams must jointly determine not just what is technically feasible or legally permissible, but what is morally "right" for patients. This forward-looking approach seeks to pre-empt potential conflicts, ensuring that the immense potential of AI in drug discovery does not inadvertently compromise the trust and privacy of the very patients it aims to serve.
Background
▶ Watch: Introduction: The patient privacy challenge in pharma AI (0:00)
Brian Martin's perspective is deeply rooted in his role at AbbVie, a global biopharmaceutical company where the R&D community is singularly focused on making a tangible impact on patients' lives through faster, more effective therapeutics and cures. He clarified that the "AI" in question is not artificial general intelligence (AGI), but rather a broad spectrum of applied techniques including predictive algorithms, generative algorithms, classification mechanisms, traditional machine learning (ML), and deep learning (DL). These tools are employed to connect, comprehend, and transform vast amounts of information into actionable knowledge and insights.
The foundational premise for this challenge lies in the nature of patient data and consent. Patients participating in clinical trials provide their information, including Protected Health Information (PHI). Crucially, this PHI is subsequently blinded or removed to prevent re-identification, a standard practice aimed at safeguarding privacy. Furthermore, patients provide explicit consent for their data to be used not only for specific endpoints related to the trial but also for secondary analysis and use. This secondary consent is pivotal, as it allows companies like AbbVie to leverage the data beyond its initial purpose, fostering broader scientific inquiry.
From this consented and curated patient information, scientists and data scientists construct an elaborate knowledge graph. This graph is a sophisticated representation of interconnected concepts, where raw data—often spanning "tens to hundreds of petabytes"—is distilled into meaningful relationships. Within this graph, solid lines denote relationships backed by empirical data and extensive curation, representing established knowledge. However, the true power of AI in this context emerges through the inference of new knowledge, represented as hashed or dashed lines. These inferred associations are predictions made by AI algorithms—such as predicting missing edges in the graph (e.g., a compound binding to a target gene), forecasting property values, or classifying conditions. As these AI-driven inferences are subsequently validated through scientific experimentation, lab assays, and peer review, they transition from dashed lines of prediction to solid lines of established knowledge, continuously expanding the collective scientific understanding. This iterative process of data collection, AI inference, and scientific validation is at the heart of modern biopharmaceutical discovery.
Key Findings
▶ Watch: Core problem: Patient consent retraction and AI-derived knowledge (5:50)
The talk, rather than presenting traditional "findings," illuminated a critical, unresolved dilemma at the intersection of patient privacy, AI, and scientific progress. Brian Martin effectively identified the core problem: what are the implications when a patient retracts their consent for data use, after that data has contributed to the training of AI models and the generation of new, derivative knowledge within a persistent knowledge graph? This scenario triggers a cascade of complex questions that currently lack clear legal or ethical answers.
The central inquiry is whether the right to be forgotten (RTBF), which typically applies to the removal of personal data from databases, also extends to the derivative knowledge and insights that AI algorithms have extracted or inferred from that data. If an algorithm was trained on a patient's data, and that patient retracts consent, is there a moral or legal obligation to retrain the algorithm to "forget" that data? Martin drew an analogy to human learning: just as a jury cannot truly "unhear" testimony, an AI model, once trained, does not inherently forget the patterns it has learned, even if the source data is removed.
A significant concern arises if removing one patient's data causes a previously asserted piece of knowledge—for example, a predicted compound-target binding or a disease association—to fall below a significance threshold. This raises questions about the validity of the knowledge, the integrity of scientific assertions, and the potential impact on ongoing research. Conversely, if removing the data doesn't change the significance, what does that imply about the individual's contribution and the necessity of their data?
Martin also highlighted the profound ethical tension between the rights of the individual and the potential benefit to society. What if a piece of knowledge, derived in part from a patient's data, is a crucial step towards developing a therapeutic or cure with a "significant positive impact on society"? If the patient's retraction of consent leads to the "loss" of that knowledge, preventing the development of a life-saving treatment, it becomes a stark example of "the impact on the many for the rights of the one." This underscores the broader moral obligation to share knowledge for the greater good, a principle AbbVie's Chief Scientific Officer, Tom Hudson, champions. The current lack of legal precedent means that organizations are navigating a "nebulous, gray area," necessitating proactive engagement to define what is not only legally permissible but also ethically sound.
Technical Deep Dive
▶ Watch: Moral obligation to share knowledge vs. individual rights (11:30)
The technical core of Brian Martin's discussion revolves around the construction and utilization of knowledge graphs within biopharmaceutical R&D. These graphs are not merely static databases but dynamic, evolving repositories of scientific understanding, fueled by massive datasets and AI-driven inference.
Knowledge Graph Architecture and Data Flow:
The process begins with an immense volume of raw data, encompassing "tens to hundreds of petabytes." This includes diverse patient information from clinical trials, such as Protected Health Information (PHI) (which is subsequently blinded), genomics data, assay data, and various other clinical measurements. Scientists and data scientists undertake extensive data curation to connect and structure this disparate information. This forms the foundation of the knowledge graph, where entities (e.g., specific compounds, genes, diseases, patients) are represented as nodes, and the relationships between them (e.g., "compound X binds to gene Y," "gene Y is associated with disease Z") are represented as edges.
AI for Knowledge Inference:
The true innovation lies in applying AI methods to this structured graph. Martin specified the use of predictive algorithms, generative algorithms, and classification mechanisms to infer new knowledge. These algorithms are designed to:
- Predict missing edges: For instance, an AI might predict that a novel compound, previously unstudied, is likely to bind to a specific target gene, even without direct experimental evidence.
- Predict property values: The AI could infer a likely dosage response or a specific biomarker's value for a given patient cohort.
- Predict classifications: The AI might classify a patient's disease progression or response to a particular treatment based on their genetic profile and clinical history.
These AI-derived inferences initially exist as hashed or dashed lines within the graph, signifying potential new knowledge. The scientific method then comes into play: these inferences are treated as hypotheses, which are subsequently tested through rigorous lab experiments, assays, and peer review of published work. If validated, these dashed lines transform into solid lines in the knowledge graph, representing empirically confirmed knowledge. This continuous feedback loop ensures the graph is a living, growing entity of validated scientific understanding.
Challenges with Differential Privacy and Rare Diseases:
During the Q&A, the concept of differential privacy was raised as a potential technical solution. Differential privacy aims to ensure that the output of an algorithm is minimally affected by the presence or absence of any single individual's data, thereby preventing re-identification. While theoretically appealing, Martin highlighted significant practical challenges, particularly with rare diseases. If a disease affects only a "hundred people on the planet," each individual patient's data point becomes disproportionately influential. An AI model trained on such a dataset "definitely learned about it from one of those people." Removing even one patient's data in such a scenario could drastically alter or destroy the critical insights derived, making it difficult to achieve meaningful differential privacy without losing valuable, potentially life-saving, knowledge.
Another technical hurdle is the inherent bias in data collection. Martin noted the difficulty in obtaining healthy control data. Clinical trials for conditions like Parkinson's disease, for example, primarily enroll patients already experiencing symptoms (e.g., mild cognitive impairment). This leads to algorithms that are "highly biased towards those specific patients" and may not accurately predict disease onset or progression in a broader, healthy population. The absence of diverse, healthy datasets further complicates the generalizability of AI models and the impact of individual data removal. Unlike transactional security models that can be frequently retrained with new data, the cumulative and persistent nature of knowledge graphs in biopharma makes the "unlearning" of individual data points a far more complex technical and ethical challenge.
Demo / Proof of Concept
▶ Watch: Q&A: Differential privacy as a solution? (14:05)
Brian Martin's presentation at DEF CON 29 was explicitly framed as a solicitation for community input and discussion on an emerging ethical and legal challenge, rather than a demonstration of a technical solution or a proof of concept. He stated, "I am just here today not to pose an answer but to hopefully open discussion... and work towards answers with... the community." Therefore, no demo or proof of concept was presented as part of this talk. The focus was entirely on articulating the complex problem of patient privacy in the context of AI-derived knowledge.
Defensive Implications
▶ Watch: Q&A: Data turnover vs. aggregate knowledge graphs (18:40)
The challenges articulated by Brian Martin demand a proactive and multi-faceted defensive strategy from organizations leveraging AI with sensitive patient data. These implications span legal, ethical, and technical domains, requiring a comprehensive approach to safeguard patient privacy while advancing scientific discovery.
- Proactive Ethical and Legal Frameworks: Organizations must establish cross-functional working groups involving legal counsel, ethicists, data scientists, and clinical researchers. The goal is to develop clear policies and frameworks that address the right to be forgotten as it pertains to AI-derived knowledge. This involves interpreting regulations like GDPR in the context of persistent knowledge graphs, not just raw data. The aim is to define "what we should do" rather than just "what we can do."
- Advanced Consent Management Systems: Current consent forms for clinical trials, while evolving, may not fully capture the nuances of AI model training, subsequent knowledge inference, and the implications of consent retraction on derivative knowledge. Companies need to design more granular and transparent consent processes that explicitly inform patients about:
- How their data will be used to train AI models.
- The potential for new, inferred knowledge to be generated.
- The challenges and limitations of "unlearning" that knowledge if consent is retracted.
- The balance between individual rights and the potential for collective societal benefit.
- Investigate and Implement "Machine Unlearning" Mechanisms: While challenging, especially for persistent knowledge graphs, research into machine unlearning is crucial. This involves developing algorithms that can effectively remove the influence of specific data points from a trained model without retraining the entire model from scratch, or at least quantify the impact of such removal. For knowledge graphs, this might involve mechanisms for "decaying" inferred associations or tagging knowledge with its derivation source to facilitate targeted removal or re-evaluation.
- Strategic Application of Differential Privacy: For datasets where individual contributions are not disproportionately influential, exploring and implementing differential privacy techniques can provide a robust defense against re-identification and strengthen privacy guarantees. However, as discussed, its applicability is limited in contexts like rare diseases where preserving the influence of individual, unique data points is critical for scientific insight. Organizations need to assess where differential privacy is feasible and where alternative privacy-preserving techniques (e.g., homomorphic encryption, federated learning) might be more appropriate.
- Robust Data Governance for Knowledge Graphs: Implement strict data governance policies that track the provenance of knowledge within the graph. This includes logging which data contributed to which inferences and how those inferences were validated. This traceability is essential for responding to consent retraction requests, allowing for a more informed assessment of the impact of removing a patient's data on specific knowledge assertions. Policies for the "decay" or periodic re-evaluation of inferred knowledge, distinct from raw data expiry, should also be considered.
- Transparency and Public Engagement: Biopharmaceutical companies, as well as the broader AI community, should engage in open dialogue with patients, patient advocacy groups, and the public about these complex issues. Transparency builds trust and helps manage expectations regarding the capabilities and limitations of AI in healthcare, particularly concerning data privacy and the right to be forgotten. This includes sharing anonymized knowledge graphs with the broader research community, as AbbVie aims to do, while carefully navigating privacy concerns.
- Quantify Impact of Data Removal: Develop methodologies to quantify the statistical and scientific impact of removing individual patient data on specific knowledge assertions. This would provide objective data to inform decisions when a patient retracts consent, helping to weigh the loss of statistical significance against the individual's right to privacy. This is particularly relevant for scenarios where a "knowledge assertion [might] fall below a significance threshold."
By proactively addressing these defensive implications, the biopharmaceutical industry can strive to achieve a delicate balance: harnessing the transformative power of AI for medical advancements while upholding the fundamental ethical and legal rights of the patients whose data makes these advancements possible.
Key Takeaways
- The "right to be forgotten" (RTBF) for patient data in AI-driven knowledge graphs presents a profound, unresolved legal and moral dilemma for biopharmaceutical companies.
- AI algorithms infer new knowledge (e.g., drug-target interactions, disease associations) from vast, consented patient datasets, but retracting underlying patient consent raises complex questions about the validity and ethical retention of that derived knowledge.
- Unlike transactional data models, knowledge graphs represent aggregate, persistent learning, making "unlearning" or effective model re-training extremely challenging, especially when individual data points (e.g., for rare diseases) are highly influential.
- There is a critical tension between an individual's right to privacy and the potential for AI-driven insights to lead to significant societal benefits, such as new cures and therapeutics, highlighting the "impact on the many for the rights of the one."
- Organizations must proactively develop comprehensive frameworks that balance legal obligations (like GDPR) with ethical responsibilities, engaging cross-functional teams (legal, ethical, data science) to define what is not just possible but right in the context of AI-generated insights.
- The challenge underscores the urgent need for innovative technical solutions (e.g., advanced differential privacy for appropriate contexts, machine unlearning research) and clearer regulatory guidance regarding data ownership and the persistence of derivative knowledge in the age of AI.
About the Speaker(s)
Brian Martin is the Head of AI, R&D Information Research at AbbVie, a global biopharmaceutical company. In this role, he leads efforts to leverage artificial intelligence within the research and development pipeline, focusing on transforming information into actionable knowledge to accelerate drug discovery. Martin brings a diverse background to his current position, having previously worked in "big finance" before transitioning to the biopharma industry. He emphasizes that the R&D community at AbbVie is deeply committed to making a positive impact on patients' lives by delivering better, faster therapeutics and cures. Through his talk, Martin actively sought community input on the complex ethical and legal challenges surrounding patient privacy in the context of AI-derived knowledge, highlighting his dedication to a collaborative and responsible approach to technological advancement in healthcare.
Reviews
Dr. Zero (Offensive Security Researcher) — WEAK
Brian Martin showed up to AI Village to crowd-source AbbVie’s GDPR compliance strategy. He spent the entire slot asking the audience how to handle the 'Right to be Forgotten' in a knowledge graph without actually proposing a single technical or policy solution. If you want a 45-minute philosophical question about data governance, here you go, but defenders will learn absolutely nothing actionable.
Heather Calloway (CISO) — SOLID
This is exactly the GDPR nightmare your General Counsel is already losing sleep over. Martin perfectly articulates the upcoming collision between data privacy regulations and enterprise AI training, making it required viewing for any leadership team building models on customer or patient data.