DARKFLEECE: Probing the Dark Side of Android Subscription Apps

DARKFLEECE: Probing the Dark Side of Android Subscription Apps

Speakers: Chang Yue, Chen Zhong, Kai Chen, Zhiyu Zhang, Yeonjoon Lee

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=k66rcvv4_js

Overview

The proliferation of subscription-based applications has revolutionized the mobile app economy, offering developers a recurring revenue stream and users access to a vast array of services and content. However, this lucrative model has inadvertently paved the way for a deceptive and increasingly prevalent threat known as fleeceware. These applications, while not containing traditional malicious code, exploit user trust by charging excessive subscription fees, often after a seemingly "free" trial period, through subtle manipulation of user interfaces and information presentation. In 2021 alone, fleeceware was reported to have affected over 600 million users and generated an alarming $400 million in annual revenue, underscoring the urgent need for robust detection mechanisms.

The talk "DARKFLEECE: Probing the Dark Side of Android Subscription Apps" by researchers from The Institute of Information Engineering at the Chinese Academy of Sciences addresses this critical issue head-on. The team presents DARKFLEECE, a novel system designed to define, automatically detect, and explain the deceptive tactics employed by fleeceware applications. Unlike traditional malware, fleeceware thrives in a gray area, mimicking benign app functionalities while leveraging dark patterns – user interface designs that trick users into making unintended decisions. This makes them particularly challenging for existing app market review processes, which primarily rely on explicit UI display requirements and time-consuming manual inspections.

The research behind DARKFLEECE not only offers a clear definition of fleeceware but also provides a comprehensive framework for identifying its distinguishing features. By combining insights from user comments, expert domain knowledge, and established literature on dark patterns, the team has developed an automated approach that extracts both textual and visual cues from app subscription UIs. The ultimate goal of DARKFLEECE is to empower app markets with an efficient tool to curb the spread of fleeceware and to educate users about the insidious strategies used by these deceptive applications, thereby enhancing the overall security and trustworthiness of the mobile app ecosystem.

Background

▶ Watch: Introduction and defining the Fleeceware problem (0:00)

The mobile application landscape has witnessed an exponential surge in subscription-based services. By September 2023, the number of subscription apps had exceeded 1.7 million, constituting approximately 13% of all mobile applications and generating an annual revenue surpassing $45 billion. This model, which allows developers to sell content or services through recurring billing, often entices users with a free trial period before requiring a purchase. While beneficial for both developers and users when implemented transparently, this model has also been exploited by a new category of scam applications: fleeceware.

Fleeceware refers to applications that charge exorbitant subscription fees after a brief, often misleading, trial period. These apps are characterized not by the presence of traditional malicious code, but by their deceptive user interface (UI) designs and information presentation tactics. They appear to offer legitimate functionality, blurring the lines between benign and exploitative. This inherent characteristic makes fleeceware particularly challenging for conventional security analysis tools, which are primarily geared towards detecting overt malicious behaviors or vulnerabilities.

Current defenses employed by app markets, such as Google Play and Apple App Store, largely depend on stringent UI display requirements. These requirements mandate that subscription terms, pricing, and auto-renewal policies be clearly and explicitly presented to users. However, as the research highlights, these guidelines are often insufficient. Developers of fleeceware apps employ sophisticated tricks to obfuscate or ambiguously present crucial subscription information, effectively circumventing these rules without technically violating them in a way that is easily detectable by automated systems. Consequently, manual review of apps remains a necessary but time-consuming and unscalable endeavor, leaving a significant gap in protection against this growing threat.

Furthermore, a clear, widely accepted definition of fleeceware has been lacking, contributing to the difficulty in both academic research and practical detection efforts. Existing research on the distinguishing features between fleeceware and benign apps has been limited. The core problem, therefore, revolves around three critical questions that DARKFLEECE aims to answer: how to precisely define fleeceware, what are its common and identifiable features, and how can these apps be automatically detected without relying on time-intensive manual processes or searching for non-existent malicious code? Addressing these questions is fundamental to developing effective countermeasures and safeguarding users from financial exploitation.

Key Findings

▶ Watch: Novel technique for subscription information extraction (4:30)

The DARKFLEECE research yielded several significant findings that not only advance the understanding of fleeceware but also provide concrete tools and insights for combating it.

Firstly, the researchers established a precise and actionable definition of fleeceware. Through a comprehensive analysis of user comments regarding fleeceware experiences, expert domain knowledge, and existing literature on deceptive design patterns, they defined fleeceware as "apps that use dark patterns to trick users into subscription." This definition is crucial because it shifts the focus from traditional malware detection to identifying manipulative UI/UX design choices. Based on this definition and an analysis of app market requirements and previous dark pattern classifications, the team summarized 12 distinct types of risky forms commonly found in fleeceware. These forms were then formalized into a set of 'N' features, providing a structured basis for automated detection.

A core achievement of the DARKFLEECE system is its high accuracy in identifying fleeceware. When tested on its labeled dataset, DARKFLEECE achieved an impressive accuracy of 93.43%. This demonstrates the efficacy of its feature extraction and classification methodology. Beyond accuracy, the system also proved to be remarkably efficient, requiring only 7.17 seconds to analyze each UI, making it suitable for large-scale application scanning by app markets.

To gauge user perception and the practical impact of DARKFLEECE, a user study was conducted. The results were telling: 81.8% of participants reported having previously encountered similar fleeceware issues, highlighting the widespread nature of the problem. Crucially, the study also revealed that when DARKFLEECE's explanations were applied to highlight risky UI elements, an average of 13.51% more people noticed the fleeceware UI issues. This indicates the potential of the system's explainable AI component to enhance user awareness and empower them to make more informed decisions. Interestingly, the study also noted that individuals with a Computer Science background were more likely to be deceived, suggesting that even technically savvy users are not immune to these sophisticated dark patterns.

Perhaps one of the most sobering findings pertains to the prevalence of fleeceware in the wild. Applying DARKFLEECE to real-world applications, the researchers identified 443 apps containing a total of 62N fleeceware UIs. These apps have collectively amassed over 5 billion downloads, with some even appearing on various "top lists" within app stores. This quantitative evidence unequivocally confirms that fleeceware is not an isolated phenomenon but a pervasive threat deeply embedded within the mobile app ecosystem.

Further analysis of common strategies employed by fleeceware UIs revealed three dominant tactics:

1. Lack of clear subscription information: The most frequent strategy involves UIs that fail to clearly inform users about the subscription nature of the app, instead emphasizing the "free trial" aspect.
2. Undisclosed auto-renewal: A significant number of fleeceware apps neglect to explicitly inform users that the subscription will automatically renew after the trial period.
3. Inconsistent trial/cancellation deadlines: Many UIs present inconsistencies between the claimed free trial period and the actual deadline for canceling the trial, trapping users into unwanted subscriptions.

Finally, the research explored the evolution of fleeceware over time. While there has been some improvement in addressing fleeceware issues in recent years, developers continuously adapt. They persist in using new and evolving dark patterns for profit, leading to the emergence of new forms of fleeceware that demand ongoing vigilance and updated detection methodologies.

Technical Deep Dive

▶ Watch: DarkFleece detection model and explainable AI (6:00)

The DARKFLEECE system is engineered to systematically identify and categorize fleeceware by focusing on the deceptive dark patterns embedded within their user interfaces. This multi-stage process involves feature library construction, sophisticated feature extraction, and a robust classification model enhanced by explainable AI.

The foundational step was the construction of a comprehensive fleeceware feature library. This library was built upon three primary pillars:

1. User Comments: Analysis of user feedback on app stores provided invaluable insights into the common grievances and deceptive strategies users reported. These comments often highlighted how users were kept unaware of or misled about subscription information.
2. Expert Domain Knowledge: Security experts and UI/UX specialists contributed their understanding of manipulative design practices.
3. Literature on Dark Patterns: Drawing from established research on dark patterns – UI designs that trick users into making unintended actions – the team identified prevalent deceptive techniques.

Through this synthesis, the researchers formally defined fleeceware as "apps that use dark patterns to trick users into subscription." They then identified 12 distinct types of risky forms of dark patterns specific to fleeceware, which were subsequently formalized into a set of 'N' quantifiable features. These features covered both textual and visual aspects of the UI, crucial for capturing the nuances of deceptive design.

The next, and arguably most technically challenging, phase was the accurate extraction of these features from app UIs. The primary hurdles encountered were:

1. Fragmented Text Information: Subscription-related text on UIs is often fragmented. For instance, "Yearly" and its corresponding price might be split across different UI elements, making direct text parsing difficult.
2. Multiple Information Formats: Subscription details are presented in various linguistic and numerical formats, such as "some dollars/year," "some dollars per week," or "yearly some dollars."

To address the challenge of fragmented text, the researchers designed a novel layout-based linking technique. They observed that in Android UI layouts, content-related widgets (like text fields displaying "Yearly" and a price) are typically defined under specific structured parent widgets, such as ViewGroup (e.g., LinearLayout or RelativeLayout). By identifying these structured widget groups, DARKFLEECE can logically link fragmented pieces of subscription information that are visually and structurally related, even if they are not contiguous in the raw text stream. For example, if "Yearly" and "$XX.YY" are in separate TextView widgets but both reside within the same LinearLayout, the system can infer their association.

For the diverse formats of subscription information, DARKFLEECE employs a pattern matching approach. After analyzing a vast corpus of subscription UIs, the team summarized common patterns for each type of information. These patterns leverage specific keywords (e.g., "yearly," "weekly," "auto-renew"), symbols (e.g., "$", "€"), and particular expressions to accurately extract the desired information (e.g., price, duration, currency).

Beyond textual information, many fleeceware features are inherently visual, such as font size, text color, visibility of crucial information, or the presence/absence of certain UI elements. To extract these visual attributes, DARKFLEECE leverages dynamic execution of the target application. During dynamic execution, the system can obtain precise coordinate information for each widget on the screen. Once the relevant text information is identified using the layout-based linking and pattern matching techniques, these coordinates are used to locate and capture the target UI area. Then, advanced image processing techniques like Optical Character Recognition (OCR) or edge detection are applied to this captured area to extract visual properties like font size, color contrast, and overall visibility.

With the comprehensive set of textual and visual features extracted, the next step is classification. The researchers chose a decision tree model for several strategic reasons:

1. Boolean Features: A significant number of the extracted features are Boolean (e.g., "Is auto-renewal mentioned?"). Decision trees are well-suited for classifying based on such discrete features.
2. Prevention of Overfitting: Given the potentially smaller size of a meticulously labeled fleeceware dataset, a shallow decision tree model helps prevent overfitting, ensuring better generalization to unseen apps.
3. Interpretability: Decision trees offer excellent interpretability. The model's logic can be easily visualized and understood, making it straightforward to review the results and identify the specific combination of features that led to an app being flagged as fleeceware.

Finally, to enhance user understanding and provide actionable insights, DARKFLEECE integrates explainable AI (XAI) techniques. Specifically, SHAP (SHapley Additive exPlanations) values are applied to visualize the model's output. SHAP helps to attribute the prediction of a machine learning model to individual features. By doing so, DARKFLEECE can highlight the specific risky features on an app's UI that contributed to its classification as fleeceware, providing clear and understandable alerts to users. This transparency is crucial for educating users and empowering them to identify and avoid deceptive practices.

Demo / Proof of Concept

▶ Watch: Prevalence of Fleeceware and common dark patterns (8:00)

While the talk did not detail a live, interactive demonstration of the DARKFLEECE system in real-time, the presentation of its capabilities and the results of its application to real-world scenarios served as a compelling proof of concept. The researchers showcased the operational effectiveness of DARKFLEECE through several key aspects.

Firstly, the core functionality of DARKFLEECE – its ability to accurately identify fleeceware – was validated by its high detection accuracy of 93.43% on its labeled dataset. This quantitative metric is a direct demonstration that the system's underlying methodology, from feature extraction to classification, is robust and effective. The reported efficiency of 7.17 seconds per UI analysis further underscores its practical viability for large-scale deployment.

The integration of SHAP (SHapley Additive exPlanations) was presented as a key component for user understanding. The talk alluded to visualizations "as shown in the right figure" (referring to a slide), illustrating how DARKFLEECE can highlight specific risky features on an app's UI. This capability is a critical part of the proof of concept, demonstrating that the system not only detects fleeceware but can also explain why an app is deemed suspicious, making the findings actionable for both users and app store reviewers. This explainable aspect would be central to any user-facing demo, showing how the system pinpoints elements like obscured pricing or subtle auto-renewal clauses.

The most impactful demonstration of DARKFLEECE's capabilities as a proof of concept came from its application to the vast ecosystem of Android apps. The research team used DARKFLEECE to scan a significant portion of the app market, leading to the discovery of 443 apps containing a total of 62N fleeceware UIs. The staggering statistic that these apps have been collectively downloaded over 5 billion times provides irrefutable evidence of the system's power to uncover widespread deception. This large-scale identification of real-world fleeceware, including apps appearing on "top lists," serves as the ultimate validation of DARKFLEECE's practical utility and its ability to address a pervasive problem. The analysis of common strategies (emphasizing free trial, lack of auto-renewal info, inconsistent cancellation deadlines) further illustrates how the system can break down complex deceptive patterns into understandable categories.

In essence, the "demo" of DARKFLEECE was presented through its rigorous evaluation, its explainable output, and its successful deployment in a real-world audit, collectively proving its effectiveness as a tool for identifying and understanding the dark side of Android subscription apps.

Defensive Implications

▶ Watch: Summary of key contributions and conclusion (9:20)

The findings presented by the DARKFLEECE team carry significant defensive implications for various stakeholders within the mobile app ecosystem, from app market operators to individual users. The research clearly demonstrates that the current mechanisms for preventing fleeceware are insufficient and require substantial augmentation.

For app market operators (e.g., Google Play, Apple App Store), the primary takeaway is the urgent need to evolve their app review processes. The existing reliance on explicit UI display requirements and manual review, while foundational, has proven inadequate against the sophisticated and evolving nature of dark patterns. The fact that 443 fleeceware apps with over 5 billion downloads slipped through current gatekeeping mechanisms is a stark indicator of this deficiency. App markets should consider integrating automated tools like DARKFLEECE into their review pipelines. Such systems could efficiently scan app UIs for the identified 12 types of risky dark patterns and the 'N' formalized features, providing a scalable solution to detect fleeceware before it reaches users. The 7.17-second analysis time per UI makes this approach practically feasible for vetting a massive volume of app updates and new submissions. Furthermore, the explainable AI component (SHAP) of DARKFLEECE could empower human reviewers by providing clear, visual explanations of why an app is flagged, streamlining the decision-making process for app rejection or requiring modifications.

App developers also bear a responsibility. While the temptation to maximize profit through deceptive tactics might be present, the long-term impact on user trust and brand reputation can be severe. Developers should adhere to principles of transparent design, ensuring that all subscription terms, auto-renewal policies, and cancellation procedures are unequivocally clear and easily accessible to users. Avoiding the identified dark patterns – such as emphasizing "free trial" over "subscription," hiding auto-renewal details, or creating inconsistent cancellation deadlines – is crucial for ethical app development. App markets could enforce stricter guidelines, perhaps even mandating automated checks against dark patterns as part of the submission process, thereby incentivizing developers to build more trustworthy applications.

For end-users, the research underscores the critical importance of digital literacy and vigilance. The finding that 81.8% of participants had encountered fleeceware highlights a widespread vulnerability. Users need to be educated about the common strategies employed by fleeceware, such as the deceptive emphasis on "free" aspects, the subtle omission of auto-renewal warnings, and confusing trial cancellation terms. The DARKFLEECE project's ability to improve user detection by 13.51% through explanations suggests that providing users with tools or information that highlight risky UI elements could significantly empower them. Future app store interfaces or third-party security tools could integrate similar explanatory features, allowing users to make more informed decisions before subscribing. Users should cultivate habits like carefully reading all terms and conditions, especially around free trials, checking cancellation policies immediately after subscribing, and reviewing bank statements for unexpected charges.

Finally, the research highlights the need for continuous monitoring and adaptation. The observation that developers continue to evolve their dark patterns for profit necessitates an ongoing research effort to identify new deceptive techniques. Defensive systems must be dynamic, regularly updated with new fleeceware definitions and feature sets to stay ahead of malicious actors. This iterative process of detection, analysis, and defense is essential to maintain a secure and trustworthy mobile app ecosystem.

Key Takeaways

• Fleeceware is a Pervasive and Deceptive Threat: These apps leverage dark patterns to trick users into excessive subscriptions, lacking traditional malicious code but affecting billions of downloads and generating substantial revenue.
• Existing Defenses are Insufficient: Current app market requirements and manual review processes are unable to effectively combat the subtle and evolving nature of fleeceware's deceptive UI designs.
• DARKFLEECE Provides a Novel, Accurate, and Efficient Detection System: The system achieved 93.43% accuracy in identifying fleeceware, processing each UI in just 7.17 seconds, making it suitable for large-scale deployment.
• Comprehensive Feature Extraction is Key: DARKFLEECE employs innovative techniques like layout-based linking for fragmented text, pattern matching for diverse information formats, and dynamic execution with OCR for visual features (e.g., font size, visibility).
• Explainable AI Empowers Users: By using SHAP, DARKFLEECE can highlight specific risky features on an app's UI, increasing user awareness and improving their ability to detect fleeceware by 13.51%.
• Urgent Need for Proactive Measures: App markets must integrate automated, explainable detection systems, and users need to be educated on common fleeceware strategies (e.g., misleading "free trial" emphasis, hidden auto-renewal, inconsistent cancellation terms) to mitigate this widespread issue.

About the Speaker(s)

The research presented in "DARKFLEECE: Probing the Dark Side of Android Subscription Apps" was a collaborative effort by Chang Yue, Chen Zhong, Kai Chen, Zhiyu Zhang, and Yeonjoon Lee. The team is affiliated with The Institute of Information Engineering at the Chinese Academy of Sciences. Their work focuses on enhancing mobile app security, specifically addressing the growing challenges posed by deceptive practices within the Android subscription app landscape. Their expertise lies in analyzing complex app behaviors, identifying subtle security threats like fleeceware, and developing automated, intelligent systems to protect users and app ecosystems.