Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric

Yan Shvartzshnaider (Paper author · York University), Vasisht Duddu (Paper author · University of Waterloo)

Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 2 · Moose

Overview

Large Language Models (LLMs) have rapidly become ubiquitous, influencing everything from personal advice to autonomous decision-making. However, their pervasive use introduces a significant conundrum: while immensely useful, LLMs are often criticized for their opaque and potentially inappropriate information handling practices, raising serious privacy concerns. In their talk at PETS, Yan Shvartzshnaider from York University and Vasisht Duddu from the University of Waterloo presented their collaborative research on identifying and measuring privacy bias in LLMs using a novel auditing metric based on Contextual Integrity (CI).

Watch on YouTube · Slides

Visual summary for Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric by Yan Shvartzshnaider, Vasisht Duddu
Visual summary for Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric by Yan Shvartzshnaider, Vasisht Duddu

Key moments

  1. 0:00 Introduction: LLMs, privacy conundrum, and decision bias.
  2. 1:55 Defining privacy bias with Contextual Integrity's 5 parameters.
  3. 3:35 Methodology: Prompting LLMs with CI parameters & Likert scale.
  4. 5:00 Multi-prompt assessment methodology for reliable bias detection.
  5. 6:00 Results: Variability in responses & different model biases.
  6. 6:50 Impact of LLM capacity on privacy bias (7B vs 13B).
  7. 8:00 Takeaways: Significant variability & societal implications for deployment.

Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric

Speakers: Yan Shvartzshnaider (Paper author, York University), Vasisht Duddu (Paper author, University of Waterloo)

Conference: PETS

YouTube: https://www.youtube.com/watch?v=MsK__RT-5Z8

Overview

Large Language Models (LLMs) have rapidly become ubiquitous, influencing everything from personal advice to autonomous decision-making. However, their pervasive use introduces a significant conundrum: while immensely useful, LLMs are often criticized for their opaque and potentially inappropriate information handling practices, raising serious privacy concerns. In their talk at PETS, Yan Shvartzshnaider from York University and Vasisht Duddu from the University of Waterloo presented their collaborative research on identifying and measuring privacy bias in LLMs using a novel auditing metric based on Contextual Integrity (CI).

This research addresses a critical gap in understanding how LLMs interpret and adhere to societal privacy norms. By defining privacy bias as a statistical measure of average error in appropriateness judgments, akin to Daniel Kahneman's work on decision bias, Shvartzshnaider and Duddu provide a robust framework for evaluating LLM behavior. Their findings underscore significant variability in how different models handle sensitive information, revealing a pressing need for developers and deployers to consider these biases as a core metric for responsible AI.

The presented methodology offers a pathway to reconcile the utility of LLMs with fundamental privacy expectations. By systematically auditing models for privacy bias, the work aims to prevent inappropriate information sharing that could lead to significant societal and ethical harms. The ultimate goal is to ensure that LLMs, as they become increasingly integrated into the fabric of daily life, operate in alignment with human values and established privacy norms.

Background

▶ Watch: Introduction: LLMs, privacy conundrum, and decision bias. (0:00)

The rapid adoption of Large Language Models has ushered in an era where AI assists, and increasingly dictates, decisions across a multitude of domains, from offering relationship advice to providing medical suggestions and acting as autonomous agents. This widespread integration, however, is not without its challenges. A central tension, dubbed the "GPT conundrum" by the speaker, lies in the conflict between the undeniable utility of these models and the legitimate concerns regarding their privacy and information handling practices. LLMs, by their very nature, process and often collect vast amounts of information, leading to justifiable anxieties about data stewardship and potential misuse.

To address this, the presented work builds upon the foundational concepts of decision bias and noise as articulated by the late Daniel Kahneman. Unlike cultural or societal bias, which focuses on prejudice against specific groups, Kahneman's definition of bias is a statistical measure: an average error or a consistent skew in judgments. An intuitive analogy is target shooting, where a consistent hit pattern to the left or right of the bullseye indicates a statistical bias. Noise, conversely, refers to variability or inconsistency in decisions where one would expect uniformity. Applying this lens to LLMs, the researchers introduce privacy bias as a statistical measure of inappropriate information flow.

The definition of privacy itself in this context is rooted in Contextual Integrity (CI), a widely accepted framework in privacy research developed by Helen Nissenbaum. CI defines privacy not as control over information, but as the appropriate flow of information, governed by contextual privacy norms. These norms dictate what information is appropriate to share, with whom, under what circumstances, and for what purpose. To operationalize CI, five essential parameters must be specified for any information flow:

  1. Sender: The entity initiating the information transfer.
  2. Receiver: The entity receiving the information.
  3. Information Type: The specific kind of data being shared (e.g., medical records, location data).
  4. Subject of Information: The individual or entity to whom the information pertains.
  5. Transmission Principle: The conditions or rules governing the information transfer (e.g., confidentially, with consent, publicly).

The problem, therefore, arises because LLMs, despite their advanced capabilities, do not inherently possess an understanding of these complex, context-dependent human privacy norms. Their training data might contain patterns reflecting various information flows, but without a principled framework like CI, their "judgments" on appropriateness can be inconsistent, biased, or outright inappropriate, leading to a risk of ethical harms and the erosion of trust. This research aims to provide a systematic way to identify and quantify these inherent biases.

Key Findings

▶ Watch: Methodology: Prompting LLMs with CI parameters & Likert scale. (3:35)

The research uncovered several critical insights into the privacy biases exhibited by Large Language Models, highlighting their variability, sensitivity, and the factors influencing their judgments:

1. Significant Variability Across Models:

A primary finding was the substantial variability in privacy biases across different LLMs. For instance, models like GPT-4 (or its mini variant) were observed to be "less conservative," meaning they tended to deem a greater number of information flows as acceptable. In contrast, models from the Llama family exhibited a more "conservative" privacy bias, frequently categorizing information flows as unacceptable. This divergence suggests that different training regimes, architectures, or alignment processes can lead to fundamentally different interpretations of privacy norms.

2. Sensitivity to Prompt Phrasing and Likert Scale Ordering:

LLMs demonstrated a remarkable sensitivity to subtle changes in input. The researchers found considerable variability in responses based on prompt paraphrasing (rephrasing the same underlying information flow in different ways) and the reordering of options within the Likert scale used for appropriateness judgments. This highlights a fragility in LLM decision-making, where minor linguistic or presentation shifts can alter a model's privacy assessment, underscoring the need for robust testing methodologies.

3. Impact of LLM Capacity:

The capacity of an LLM, often indicated by the number of parameters (e.g., 7B vs. 13B), was found to significantly influence its privacy biases, sometimes leading to directly opposing judgments. A striking example involved sharing information about a user's "at home" times with local police during an emergency:

  • A 13B Llama model considered this "strongly acceptable."
  • A 7B Llama model considered it "strongly unacceptable."

This polar privacy bias for the same scenario, differing only by model capacity, is a critical finding, suggesting that scaling alone does not guarantee a more aligned or consistent understanding of privacy, and in fact, can introduce divergent ethical stances.

4. Alignment Strategies Influence Bias:

While less detailed in the talk, the research also noted differences between base models and their aligned variants. These alignment processes, typically involving reinforcement learning from human feedback (RLHF), are designed to make models more helpful, harmless, and honest. However, the findings suggest that even these alignment strategies can introduce or modify privacy biases, sometimes leading to polar differences in judgments, particularly concerning sensitive entities like government intelligence agencies.

5. Nuances in Acceptability (e.g., User Consent):

The analysis revealed nuanced patterns in LLM judgments. For example, sharing information with intelligence agencies was generally considered "largely not acceptable" or "strongly not acceptable." However, there was a notable exception: when user consent was explicitly given, some models shifted their judgment towards acceptability. This indicates that while LLMs might generally default to a conservative stance on sensitive disclosures, they can, in certain contexts, recognize and incorporate the principle of user autonomy.

6. Reliability of Multi-Prompt Assessment:

Crucially, the research confirmed that the devised multi-prompt assessment methodology provides a reliable way to identify and quantify these privacy biases. By systematically varying prompts and Likert scales and then aggregating responses, the method effectively mitigates the noise and inconsistencies observed in single-prompt evaluations, allowing for a more accurate assessment of a model's underlying privacy stance.

These findings collectively emphasize that privacy bias is a pervasive and complex issue in LLMs, deeply influenced by model architecture, training data, capacity, and subtle input variations. Recognizing and measuring these biases is an essential first step toward building more privacy-aware and ethically aligned AI systems.

Technical Deep Dive

▶ Watch: Multi-prompt assessment methodology for reliable bias detection. (5:00)

The core of this research lies in its rigorous, CI-based methodology for defining and auditing privacy bias in LLMs. The definition of privacy bias is conceptualized as the "appropriateness skew" within the Contextual Integrity (CI) appropriateness measure. This measure, derived from the five essential CI parameters (sender, receiver, information type, subject of information, transmission principle), quantifies how appropriate a given information flow is according to specified norms.

Scalar vs. Tensor Bias:

When all five CI parameters are fully specified for a particular information flow, the CI appropriateness measure yields a scalar value, representing a single point on an "appropriateness queue" (e.g., strongly appropriate, strongly unacceptable). However, the methodology also allows for a more granular analysis. If not all parameters are fixed, this creates a multi-dimensional space of appropriateness values, which can be "sliced" to reveal patterns. This results in a tensor of appropriateness values, which is particularly useful for identifying biases across different dimensions—for example, how appropriateness changes across multiple potential receivers, various types of information, or in response to different prompts. This tensor-based approach allows researchers to observe "skew in inappropriateness" in a more complex, multi-faceted manner.

Auditing Metric and Ground Truth:

The process of auditing LLMs for privacy bias begins with constructing specific prompts. These prompts are meticulously designed using a template that explicitly specifies all five CI parameters for a hypothetical information flow. For example, a prompt might ask: "Is it acceptable for [Sender] to share [Information Type] about [Subject] with [Receiver] under [Transmission Principle]?" The LLM is then asked to respond on a Likert scale, typically ranging from "strongly unacceptable" to "strongly acceptable."

A key insight of this work is that even without a definitive ground truth (i.e., a universally accepted human judgment of appropriateness for every scenario), a normative analysis can still be performed. By observing the distribution of responses from multiple models or multiple prompts for the same model, one can identify patterns of clustering, skew, scattering, or noise. For instance, one model's responses might consistently cluster in the middle (neutral), while another's might be skewed towards "acceptable" or "unacceptable," or exhibit high variability (noise). This provides a preliminary basis for understanding a model's inherent tendencies.

However, the introduction of a ground truth significantly enhances the analysis. With a target appropriateness value (e.g., derived from human expert consensus), researchers can then calculate a privacy delta. This delta quantifies the deviation of the LLM's response from the ground truth, encompassing both systematic bias and random noise. The talk illustrated this with hypothetical models:

  • Model A: Zero bias, minimal noise (responses clustered around ground truth).
  • Model B: Some bias, but with some noise variability (responses consistently off-target but with some spread).
  • Model C: Some bias, with more noise (responses off-target and highly scattered).
  • Model D: No apparent bias, but very high noise (responses widely scattered, making it difficult to discern a consistent bias).

Identifying and capturing privacy bias effectively requires mitigating the impact of noise, especially when models exhibit high variability.

Multi-Prompt Assessment Methodology for Reliability:

To overcome the challenges posed by prompt sensitivity and response variability, the researchers developed a robust multi-prompt assessment methodology. This methodology is designed to reliably capture a model's underlying privacy bias by averaging out inconsistencies and reducing noise:

  1. Define Information Flow: A single, specific information flow is defined by its five CI parameters.
  2. Template and Paraphrasing: This information flow is embedded into a prompt template. Crucially, this template is then paraphrased multiple times to generate several semantically equivalent, but syntactically distinct, prompts. This tests the model's consistency across varied phrasings.
  3. Randomized Likert Scale Order: For each paraphrased prompt, the order of options on the Likert scale (e.g., "strongly unacceptable," "unacceptable," "neutral," "acceptable," "strongly acceptable") is randomized. This helps to mitigate any positional bias the LLM might have towards options appearing first or last.
  4. LLM Query: All these varied prompts (paraphrased, with randomized Likert scales) are then fed as input to the target LLM.
  5. Response Collection and Validation: The LLM's responses are collected. Only "valid responses" (e.g., those that clearly fall within the Likert scale and are above a certain confidence threshold if applicable) are retained for analysis.
  6. Consensus Determination: The final step involves a strategy (such as a majority vote or other statistical aggregation techniques) to derive a consensus on the privacy bias for that specific information flow. By aggregating multiple responses to varied but semantically identical prompts, the methodology aims to extract a more stable and reliable measure of the model's privacy stance.

This rigorous technical approach allows for a systematic and reliable identification of privacy biases, providing a powerful tool for auditing LLMs against established privacy norms. The findings, derived from applying this methodology to various models like GPT-4 and Llama (7B and 13B versions), provided empirical evidence of the significant and often polar privacy biases present in contemporary LLMs.

Demo / Proof of Concept

▶ Watch: Impact of LLM capacity on privacy bias (7B vs 13B). (6:50)

While the talk did not feature a live, interactive demonstration of a tool or a specific software proof of concept, the entire presentation serves as a robust proof of concept for the Contextual Integrity-based auditing metric itself. The researchers detailed their comprehensive multi-prompt assessment methodology, which is a systematic, step-by-step process for identifying and quantifying privacy bias in LLMs.

The "results highlights" section of the talk effectively acts as an empirical demonstration of this methodology in action. By presenting specific examples of how different LLMs (GPT-4, Llama 7B, Llama 13B) responded to various information flows, and how their responses varied based on factors like model capacity, prompt paraphrasing, and Likert scale ordering, the speakers provided concrete evidence that their proposed metric successfully captures and exposes privacy biases. The examples, such as the polar biases concerning sharing location data with police during an emergency, illustrate the practical application and the critical insights derived from their auditing framework. Therefore, while no software demo was shown, the detailed methodology and its empirical findings stand as a strong validation of the research's core claims.

Defensive Implications

▶ Watch: Takeaways: Significant variability & societal implications for deployment. (8:00)

The findings from this research carry profound implications for anyone involved in the development, deployment, or regulation of Large Language Models. Understanding and mitigating privacy bias is not merely an academic exercise; it is a critical step towards building responsible, trustworthy, and ethically aligned AI systems.

1. Integrate Privacy Bias as a Core Evaluation Metric:

Organizations and developers must move beyond traditional performance metrics (e.g., accuracy, fluency) and explicitly incorporate privacy bias as a non-negotiable evaluation criterion. Before deploying any LLM, especially in sensitive domains such as healthcare, finance, legal, or public safety, an audit for privacy bias using a CI-based framework should be mandatory. This ensures that the model's information handling practices align with societal expectations and regulatory requirements.

2. Enhance Model Selection and Configuration:

Given the significant variability in privacy biases across different LLMs (e.g., GPT-4 being less conservative, Llama being more conservative), model selection should be a deliberate process guided by a privacy bias assessment. Developers should choose models whose inherent biases are most aligned with the specific privacy norms of the intended application context. Furthermore, configuration parameters, including model capacity and alignment strategies, need to be carefully considered, as they demonstrably impact a model's privacy stance.

3. Implement Robust Prompt Engineering and Testing:

The observed sensitivity to prompt paraphrasing and Likert scale reordering highlights the fragility of LLM responses. Defenders must adopt rigorous prompt engineering practices, including:

  • Diversity in Prompting: Generate and test a wide array of prompt variations for critical information flows to ensure consistent and appropriate privacy judgments.
  • Likert Scale Randomization: Standardize the randomization of Likert scale options during testing to prevent positional bias from influencing evaluation results.
  • Adversarial Prompting: Proactively test for privacy vulnerabilities by crafting prompts designed to elicit inappropriate information sharing, similar to red-teaming efforts for safety.

4. Strengthen Alignment and Fine-tuning for Privacy Norms:

The differences observed between base and aligned models, and across model capacities, suggest that current alignment processes may not adequately embed contextual privacy norms. Future research and development should focus on:

  • Privacy-Centric RLHF: Incorporate human feedback specifically focused on privacy appropriateness according to CI principles during reinforcement learning from human feedback (RLHF).
  • Contextual Integrity Training: Fine-tune models on datasets explicitly annotated with CI parameters and appropriateness judgments to imbue them with a deeper understanding of privacy contexts.

5. Emphasize User Consent and Transparency:

The finding that models sometimes recognize user consent as an exception to otherwise inappropriate sharing underscores the importance of integrating robust consent mechanisms into LLM-powered applications. Furthermore, transparency about how LLMs handle user data and the potential for privacy biases should be communicated clearly to end-users.

6. Inform Policy and Regulatory Frameworks:

This research provides empirical evidence that can inform policymakers and regulators about the inherent privacy challenges in LLMs. Future regulations for AI development and deployment should consider mandating privacy bias assessments and establishing benchmarks for acceptable privacy behavior, particularly for models operating in regulated industries.

As Helen Nissenbaum warned, new technologies introduce "small little changes into our society that may be unnoticed at first, but they accumulate." Ignoring privacy bias in LLMs could lead to deeply entrenched, yet unaccepted, societal norms around information sharing. Proactive defensive measures are crucial to ensure that LLMs contribute positively to society without eroding fundamental privacy rights.

Key Takeaways

  • Significant Variability: LLMs exhibit substantial and often polar differences in privacy biases across models, capacities (e.g., 7B vs. 13B), and alignment strategies.
  • Contextual Integrity (CI) as a Framework: CI, with its five essential parameters (sender, receiver, information type, subject, transmission principle), provides a robust and principled framework for defining and auditing privacy bias.
  • Sensitivity to Input: LLM privacy judgments are highly sensitive to subtle variations in prompt phrasing and the ordering of options within Likert scales, underscoring the need for rigorous evaluation.
  • Reliable Auditing Methodology: The multi-prompt assessment methodology, which involves paraphrasing prompts and randomizing Likert scales, offers a reliable way to identify and quantify inherent privacy biases in LLMs.
  • Privacy Bias as a Deployment Metric: Privacy bias should be a core consideration and a critical evaluation metric for model selection, configuration, and deployment, especially in sensitive societal contexts.
  • Societal Implications: Unaddressed privacy biases in LLMs could lead to the gradual, unnoticed erosion of privacy norms, potentially resulting in deeply entrenched, yet socially unaccepted, information handling practices.

About the Speaker(s)

Yan Shvartzshnaider is a paper author affiliated with York University. He presented the research on privacy bias in language models, highlighting the collaborative nature of the work. His contribution focuses on the conceptualization of privacy bias using Contextual Integrity and the development of the auditing metric.

Vasisht Duddu is a paper author from the University of Waterloo and a collaborator on this research. His work, in conjunction with Yan Shvartzshnaider, contributed to the development and analysis of the contextual integrity-based auditing metric for identifying privacy bias in large language models.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Shvartzshnaider and Duddu brought an academic ruler to measure the obvious: LLMs hallucinate their morals just like they hallucinate facts. It's a competent measurement paper for PETS that applies Nissenbaum's Contextual Integrity to model outputs, but practitioners aren't going to operationalize Likert scale prompt testing to stop data leaks.

Heather Calloway (CISO) — STRONG ACCEPT

This research proves mathematically what CISOs have suspected: LLMs do not possess stable privacy norms, and their judgments fluctuate based on parameter size and prompt structure. It provides a concrete auditing framework for enterprise AI deployments, proving we cannot outsource contextual data stewardship to the models themselves. A necessary read for your AI governance committee.

→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026

All talks from Privacy Enhancing Technologies Symposium (PETS) 2026