Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale
Mst Eshita Khatun (Paper author · Louisiana State University), Lamine Noureddine (Paper author · Louisiana State University), Sideeq Bello (Paper author · Louisiana State University), Aisha Ali-Gombe (Paper author · Louisiana State University)
Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 2 · Caribou
Overview
In an era where digital privacy is paramount, apps are expected to clearly communicate their data handling practices to users. This talk, "Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale," presented by Aisha Ali-Gombe on behalf of her student Mst Eshita Khatun and co-authors Lamine Noureddine and Sideeq Bello from Louisiana State University, delves into a critical issue: the inconsistency between different data disclosure mechanisms provided by app developers. Specifically, the research investigates the alignment, or lack thereof, between an app's verbose privacy policy and the more structured, developer-reported Data Safety Level (DSL) on platforms like Google Play.

Key moments
- 0:00 Introduction to privacy policy and data safety misalignment
- 1:48 Study's research questions and analysis pipeline
- 3:58 Defining consistency, misalignment, and Cohen's Kappa
- 5:00 Initial findings: Naive vs. Cohen's Kappa consistency
- 6:12 Most misaligned data categories identified
- 7:15 Sensitivity Risk Scoring results and high-risk apps
- 8:34 Critical qualitative discoveries and the 'service provider' loophole
- 9:56 Recommendations for improved disclosure and transparency
Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale
Speakers: Mst Eshita Khatun (Paper author, Louisiana State University); Lamine Noureddine (Paper author, Louisiana State University); Sideeq Bello (Paper author, Louisiana State University); Aisha Ali-Gombe (Paper author, Louisiana State University)
Conference: PETS
YouTube: https://www.youtube.com/watch?v=oUFk_kzQgTU
Overview
In an era where digital privacy is paramount, apps are expected to clearly communicate their data handling practices to users. This talk, "Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale," presented by Aisha Ali-Gombe on behalf of her student Mst Eshita Khatun and co-authors Lamine Noureddine and Sideeq Bello from Louisiana State University, delves into a critical issue: the inconsistency between different data disclosure mechanisms provided by app developers. Specifically, the research investigates the alignment, or lack thereof, between an app's verbose privacy policy and the more structured, developer-reported Data Safety Level (DSL) on platforms like Google Play.
The core objective of this work is to empirically quantify the extent of misalignment between these two crucial documents. While both are intended to inform users about data collection and sharing, discrepancies can severely undermine user trust and their ability to make informed decisions. The study highlights that developers, despite having multiple avenues for disclosure, often fail to provide a consistent and accurate picture of their data practices, leading to a significant transparency deficit. This research not only identifies the prevalence and patterns of these inconsistencies but also proposes a Sensitivity Risk Scoring (SRS) model to weigh the impact of such misalignments, offering valuable insights for users, developers, and platform providers alike.
The findings underscore a systemic problem where current disclosure mechanisms, while well-intentioned, are frequently incomplete, outdated, or outright contradictory. The work is particularly relevant given the increasing global focus on data privacy regulations and the ongoing challenge of making complex legal documents accessible and actionable for average users. By shedding light on the "disclosure divergence," the researchers provide a foundation for improving transparency and fostering greater accountability within the mobile app ecosystem.
Background
▶ Watch: Introduction to privacy policy and data safety misalignment (0:00)
The proliferation of mobile applications has brought with it an increased demand for transparency regarding how these apps handle user data. To address this, app developers are typically required to provide disclosure mechanisms that explain their data access practices. Historically, the primary mechanism has been the privacy policy – a comprehensive, legally-oriented document detailing data collection, usage, storage, and sharing practices. However, as the speaker notes, privacy policies are notoriously complex, lengthy, and, as a practical matter, "nobody reads" them. This inherent usability flaw significantly diminishes their effectiveness in empowering users to make informed choices.
Recognizing the limitations of traditional privacy policies, platforms like Google Play introduced simplified, developer-reported summaries of data practices. For Android, this is known as the Data Safety Level (DSL). The DSL is designed to be more structured, digestible, and directly comparable across applications, offering users a quick overview of what data an app collects and shares. Ideally, both the detailed privacy policy and the summarized DSL should present a consistent and accurate representation of an app's data handling. However, the reality often diverges from this ideal.
The problem stems from several factors: developers might omit certain data categories in one disclosure but not another, misreport the purpose of data collection, fail to disclose third-party sharing, or simply provide outdated information. This creates a misalignment where the two documents, intended to convey the same information, offer conflicting narratives. Such inconsistencies weaken user trust, hinder informed decision-making, and can potentially expose users to privacy risks they are unaware of. The research presented aims to systematically investigate and quantify the extent of this disclosure divergence, moving beyond anecdotal observations to provide empirical evidence of the problem's scale and nature. The study's focus on both data collection and data sharing, particularly with third parties, is crucial, as the latter often represents a more significant privacy concern for users.
Key Findings
▶ Watch: Defining consistency, misalignment, and Cohen's Kappa (3:58)
The research yielded several significant findings, shedding light on the prevalence and patterns of disclosure divergence. The analysis began with a substantial dataset of approximately 9,000 applications, spanning over 400 different application categories. A notable preliminary finding, not directly part of the initial research questions but highly indicative of developer negligence, was that 3,000 of these applications (roughly one-third) lacked valid privacy policy statements or had broken links, rendering their primary disclosure mechanism non-functional.
For the remaining valid applications, the study employed a sophisticated analytical pipeline to compare data practices across privacy policies and Data Safety Levels. Initially, using a naive consistency score (simple agreement rate), the researchers found an apparent alignment of about 70% and a misalignment of 30%. This initial figure, while seemingly acceptable, was acknowledged as potentially overestimating true consistency. A deeper dive revealed that misalignment was more frequently observed within the Data Safety Level (DSL) compared to privacy policy documents. This suggests developers might be providing more comprehensive data details in their privacy policies, but failing to translate all of it into the structured DSL, reinforcing the idea that DSLs might be under-reporting.
To overcome the overestimation of naive consistency, the researchers adopted Cohen's Kappa score, a more robust statistical measure that accounts for agreement by chance. With Cohen's Kappa, the picture shifted dramatically: consistency dipped significantly, especially for data sharing practices, compared to data collection. This indicates that while developers might be somewhat consistent in reporting what they collect, they are far less consistent in disclosing what they share, highlighting a critical area of privacy concern.
Regarding specific data categories, the study identified clear patterns of misalignment:
- For data collection, photos and videos emerged as the top misaligned category, followed by location and app activity.
- For data sharing, personal information was overwhelmingly the most misaligned category, accounting for about 72% of inconsistencies. This is a particularly sensitive finding, as personal information is often considered high-risk.
The study also introduced a Sensitivity Risk Scoring (SRS) model, which weighs misalignment by the sensitivity of the underlying data category. Intriguingly, only about 1.2% of the analyzed dataset fell into the "high sensitivity risk" category. While this small percentage might seem positive, the researchers noted that these high-risk applications tended to have extensive data divergence, with many exhibiting misalignment across all 14 data categories studied. This suggests that a small number of apps contribute disproportionately to high-impact privacy risks. Furthermore, apps within the wearables and communication categories exhibited the highest SRS, though most apps across the board showed a medium SRS.
Beyond quantitative measures, qualitative analysis uncovered several critical issues:
- Among the 3,000 apps with invalid privacy policies, nearly a quarter stated they do not collect data, yet simultaneously claimed to share data with third parties. This constitutes a direct contradiction and a fundamental misunderstanding or misrepresentation of data practices.
- A particularly "mind-blowing" finding, as described by the speaker, was a loophole in Google's Data Safety Level documentation. It states that if a developer categorizes a third party as a "service provider," they are not required to disclose this as third-party sharing in the DSL. This creates a significant opportunity for developers to avoid transparency regarding data shared with analytics companies, ad networks, or other service providers, potentially underrepresenting third-party sharing overall. The researchers emphasize the need to further evaluate how widely developers exploit this loophole.
In summary, the key findings demonstrate a widespread and systematic problem of disclosure divergence, particularly concerning sensitive data categories and third-party sharing. The "service provider" loophole represents a critical vulnerability in current platform disclosure requirements, allowing developers to legally obscure significant data sharing practices.
Technical Deep Dive
▶ Watch: Most misaligned data categories identified (6:12)
The empirical study employed a rigorous analysis pipeline designed to systematically compare and quantify inconsistencies between privacy policies and Data Safety Levels (DSL). This pipeline comprised two main phases: data collection and data modeling/formulation.
The data collection phase involved the integration of various tools for web scraping and privacy policy analysis. The researchers initially collected data points for approximately 9,000 distinct applications, covering around 400 different application categories. A crucial preliminary step involved validating the presence and accessibility of privacy policies. From the initial 9,000 applications, it was discovered that 3,000 lacked valid privacy policy statements or had broken links. These applications were excluded from the final comparative analysis, highlighting a foundational issue of non-compliance even before examining content.
The data modeling and formulation phase was critical for normalizing and structuring the disparate information from privacy policies and DSLs into a unified schema. The researchers adopted 14 standardized data categories (e.g., location, personal information, photos/videos, app activity) and focused on two primary data operations: data collection and data sharing. For each data category and operation, a binary indicator was created: '1' if the data practice was disclosed, and '0' if it was not. This structured representation allowed for direct comparison between the two disclosure mechanisms for each app.
To formally define the metrics for evaluating consistency and misalignment, the following were established:
- Consistency: Occurs when both the privacy policy and the DSL agree on the disclosure (or non-disclosure) of a specific data category for a given operation.
- Misalignment: Occurs when the two disclosures disagree.
- Consistency Score (Naive): For an app and a given operation, this is the simple agreement rate across all 14 data categories. While straightforward, the researchers recognized its limitation in potentially overestimating true agreement.
- Cohen's Kappa Score: To address the overestimation of the naive consistency score, which doesn't account for agreement occurring by chance, Cohen's Kappa was adopted. This statistical measure provides a more robust indicator of inter-rater agreement, reflecting the true agreement beyond what would be expected by random chance. The Cohen's Kappa score considers both observed agreement and expected agreement.
- Sensitivity Risk Scoring (SRS) Model: This advanced metric was developed to weigh misalignment by the inherent sensitivity of the data category involved. The SRS function calculates a score for each application, for each operation (collection and sharing), based on the sum of misaligned sensitive data categories. The overall SRS score for an app is derived from the joint distribution of its collection and sharing SRS scores. A weighted function was also applied, giving more weight to data sharing disclosures than to data collection disclosures. This weighting reflects the general understanding that sharing data, especially with third parties, often carries a higher privacy risk for users than mere collection. The SRS was categorized into low (below 30%), medium, and high sensitivity risk bands to facilitate interpretation.
The technical approach ensured that the comparison was not just superficial but deeply analyzed the nuanced differences in data practices reported by developers. By standardizing data categories, defining clear operations, and employing robust statistical measures like Cohen's Kappa and the custom SRS model, the research provided a comprehensive and quantifiable assessment of disclosure divergence at scale. The identification of the "service provider" loophole further underscores the importance of a detailed analysis of documentation and real-world developer practices.
Demo / Proof of Concept
▶ Watch: Sensitivity Risk Scoring results and high-risk apps (7:15)
The presented talk focused on an empirical study and its quantitative and qualitative findings, derived from analyzing a large dataset of mobile applications and their associated privacy documentation. The talk does not indicate that a live demonstration or a proof-of-concept tool was presented as part of the talk. The research primarily involved data collection, statistical analysis, and qualitative review of existing disclosures rather than the development of a user-facing tool or exploit.
Defensive Implications
▶ Watch: Recommendations for improved disclosure and transparency (9:56)
The findings of this research carry significant implications for various stakeholders within the mobile app ecosystem, prompting actionable defensive strategies. The pervasive nature of disclosure divergence, particularly concerning sensitive data categories and third-party sharing, necessitates a multi-faceted approach to improve transparency and protect user privacy.
For App Developers:
The most direct implication is the urgent need for developers to review and harmonize their data disclosure practices. This includes:
- Ensuring Consistency: Developers must meticulously cross-reference their privacy policies with their Data Safety Level (DSL) declarations to eliminate discrepancies. Every data category collected or shared should be accurately represented in both documents.
- Comprehensive Disclosure: The research highlights underrepresentation of data sharing, especially with third parties. Developers must commit to fully disclosing all instances of data sharing, regardless of whether the third party is classified as a "service provider" or not. The "service provider" loophole, while technically permissible by platform rules, fundamentally undermines transparency and user trust.
- Regular Audits: Developers should conduct regular internal audits of their data practices and corresponding disclosures to ensure they remain accurate and up-to-date, especially after changes in app functionality or third-party integrations.
- Valid Privacy Policies: The 3,000 apps with invalid privacy policies underscore a basic failure. Developers must ensure their privacy policy links are functional and the documents are accessible.
For Platform Providers (e.g., Google Play):
Platform operators are in a unique position to enforce better disclosure practices:
- Strengthen Disclosure Requirements: Platforms should close loopholes like the "service provider" exemption, making it mandatory to disclose all third-party data sharing, irrespective of the relationship. This could involve clearly defining "third party" to include all entities receiving user data from the app.
- Automated Consistency Checks: Platforms could implement automated tools to detect significant discrepancies between privacy policies and DSLs, flagging potential misalignments for developer review or even imposing penalties.
- Inconsistency Ratings/SRS Scores: As recommended by the speaker, platforms could introduce a public "inconsistency rating" or display the Sensitivity Risk Score (SRS) directly on app listings. This would empower users by clearly indicating how reliable an app's privacy disclosures are, incentivizing developers to improve their practices.
- Mandatory Third-Party Naming: To enhance transparency, platforms should require developers to explicitly name all third-party entities with whom data is shared, rather than generic descriptions.
For End Users:
While users often bear the brunt of opaque practices, they can take proactive steps:
- Scrutinize DSLs (with Caution): Users should continue to check Data Safety Levels for quick insights, but remain aware of the potential for under-reporting, especially regarding data sharing.
- Seek Out Privacy Policies (if concerned): For apps handling highly sensitive data, or if the DSL seems vague, users may need to make the extra effort to review the full privacy policy, though acknowledging its complexity.
- Demand Transparency: Users should advocate for clearer, more consistent privacy disclosures through feedback channels and support for privacy-focused initiatives.
By addressing these defensive implications, the ecosystem can move towards a future where data disclosures are not just present, but are accurate, consistent, and genuinely informative, thereby rebuilding user trust and enabling truly informed decision-making.
Key Takeaways
- Widespread Disclosure Divergence: There is a significant and systematic misalignment between app privacy policies and Data Safety Levels (DSL), undermining user transparency.
- Under-reporting in DSLs: Misalignment is more prevalent in DSLs, suggesting developers provide more information in privacy policies but fail to translate it fully into the structured DSL, particularly for data sharing.
- High-Risk Data Categories: Sensitive data like "personal information" (72% of sharing inconsistencies), "photos and videos," and "location" are most frequently misaligned, posing significant privacy risks.
- The "Service Provider" Loophole: A critical finding is Google's policy allowing developers to omit disclosure of data shared with "service providers" from the DSL, creating a major transparency gap for analytics and other third-party data transfers.
- Few High-Risk Apps, Big Impact: While only 1.2% of apps fall into the high-sensitivity risk category, these apps often exhibit extensive misalignment across all data categories, indicating a concentrated risk.
- Call for Platform Intervention: The research recommends platform-level solutions such as inconsistency ratings, mandatory naming of third parties, and closing policy loopholes to enforce better developer accountability and improve user trust.
About the Speaker(s)
The talk was presented by Aisha Ali-Gombe, an Assistant Professor of Computer Science at Louisiana State University. She presented the work on behalf of her student, Mst Eshita Khatun, who is the primary author of the paper, alongside co-authors Lamine Noureddine and Sideeq Bello, also from Louisiana State University. Professor Ali-Gombe's role in the presentation highlighted her commitment to her students' research and her expertise in the field of computer science, particularly in areas related to privacy and data security. The research itself is a testament to the work being conducted at Louisiana State University to address critical issues in digital privacy and transparency.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Academic measurement study from LSU proving Google Play's Data Safety labels are pure fiction. Good statistical methodology across 9,000 apps, but the core findings just put concrete numbers to the ecosystem rot every practitioner already knew existed.
Heather Calloway (CISO) — SOLID
This research empirically proves what most practitioners already assume: app store privacy labels are largely fiction. The researchers mapped 9,000 apps and found structural divergence between legal privacy policies and developer-reported data safety summaries, heavily weighted toward obscuring third-party sharing. It is competent academic validation of a known regulatory and supply-chain reality.
→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026
All talks from Privacy Enhancing Technologies Symposium (PETS) 2026