SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles

Guanchong Huang (Paper author · University of Oklahoma), Song Fang (Paper author · University of Oklahoma)

Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 1 · Moose

Overview

In an era where digital consumption deeply intertwines with personal identity, the privacy of our viewing habits has become a critical concern. This talk, "SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles," presented by Guanchong Huang from the University of Oklahoma at PETS, unveils a novel and unsettling side-channel attack. The research, primarily conducted by PhD student Song Fang, demonstrates that an attacker can identify the specific video a victim is watching, even when the screen content is blurred and unreadable, simply by analyzing the shapes of on-screen subtitles.

Watch on YouTube · Slides

Visual summary for SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles by Guanchong Huang, Song Fang
Visual summary for SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles by Guanchong Huang, Song Fang

Key moments

  1. 0:50 Introduction to the video identification privacy threat
  2. 2:20 New attack challenge: identifying videos despite user precautions
  3. 3:50 Core idea: leveraging blurred subtitle shapes (C-houses)
  4. 5:00 SilhouetteTell's threat model and attack assumptions
  5. 5:50 Methodology: Mask R-CNN for accurate subtitle shape extraction
  6. 7:00 Detailed methodology: extracting spatial and temporal features
  7. 8:30 Evaluation results demonstrating high video identification accuracy
  8. 8:50 Discussing defense strategies against the SilhouetteTell attack

SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles

Speakers: Guanchong Huang, Paper author, University of Oklahoma; Song Fang, Paper author, University of Oklahoma

Conference: PETS

YouTube: https://www.youtube.com/watch?v=MdKkQCWnKx0

Overview

In an era where digital consumption deeply intertwines with personal identity, the privacy of our viewing habits has become a critical concern. This talk, "SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles," presented by Guanchong Huang from the University of Oklahoma at PETS, unveils a novel and unsettling side-channel attack. The research, primarily conducted by PhD student Song Fang, demonstrates that an attacker can identify the specific video a victim is watching, even when the screen content is blurred and unreadable, simply by analyzing the shapes of on-screen subtitles.

This research challenges conventional wisdom regarding video privacy and attack vectors. Traditional video identification attacks often rely on direct content observation, network traffic analysis, or specialized optical equipment. SilhouetteTell, however, bypasses these established defenses by exploiting a previously overlooked information leakage channel: the unique geometric and temporal characteristics of subtitle regions. The ability to identify sensitive viewing habits—which can reveal political views, health information, religious beliefs, or sexual orientation—from a distance using only a standard smartphone camera represents a significant privacy threat, potentially leading to targeted advertising, discrimination, or cyberbullying.

The talk introduces a sophisticated methodology that combines advanced computer vision techniques, specifically Mask R-CNN for precise subtitle shape extraction, with spatio-temporal correlation algorithms. This approach allows for the creation of unique "fingerprints" for videos based solely on the blurred outlines of their subtitles. The findings underscore the pervasive nature of information leakage in seemingly innocuous data and necessitate a re-evaluation of privacy protections in video consumption platforms.

Background

▶ Watch: Introduction to the video identification privacy threat (0:50)

The act of watching videos, whether on YouTube, Amazon Prime Video, or Netflix, has become an integral part of daily life. However, what an individual chooses to watch can be deeply revealing, disclosing sensitive personal attributes such as political affiliations, health status, religious beliefs, or even sexual orientation. Such information, if compromised, can be weaponized for targeted advertising, or worse, lead to severe consequences like discrimination in employment or housing, or become fodder for cyberbullying. This forms the basis of the video identification attack, an unauthorized attempt to discern the specific videos a victim is watching.

Existing video identification attacks generally fall into three categories. The first is content matching, where an attacker directly observes the video content or listens to its audio. This is the most straightforward method but is easily thwarted if the victim takes simple precautions, such as tilting their screen away from observers or stopping playback when strangers are nearby. The second category involves traffic analysis, where attackers monitor network traffic patterns to infer video content. This often involves sophisticated techniques like video fingerprinting, which analyzes unique bandwidth consumption patterns or packet sizes associated with specific video streams. While powerful, traffic analysis often requires specialized equipment (e.g., USRP for passive collection) or the deployment of malware, and can be mitigated by offline viewing or strong encryption. The third type of attack leverages optical reflection, where reflections of the screen content from surfaces like glasses or walls are captured and analyzed. These methods, however, typically require specialized optical equipment and specific environmental conditions, making them less universally applicable.

Crucially, users are becoming increasingly aware of these threats and are adopting countermeasures. Tilting screens, watching downloaded content offline, or simply being in an environment where specialized equipment isn't present can effectively neutralize many existing attack vectors. This raises a fundamental question: in a scenario where such precautions are taken, where the video content is unreadable, audio is unavailable, and no specialized equipment or malware is present, can an attacker still identify the video? SilhouetteTell unequivocally answers this question with a resounding "yes."

The research highlights the often-overlooked role of subtitles in video consumption. Subtitles can be broadly categorized as open subtitles, which are embedded directly into the video stream and cannot be turned off, and closed subtitles, which can be toggled by the user and are often downloaded in specific formats like SRT (SubRip Subtitle) files. These files typically contain subtitle indices, start and end timestamps, and the corresponding text content. Subtitles are displayed in various regions of the frame, but most commonly at the bottom. Their prevalence is significant; surveys indicate that 63% of individuals under 30 years old prefer to watch videos with subtitles enabled. Beyond preference, subtitles are often essential for understanding foreign language content or for viewers in very quiet or very loud environments where audio comprehension is difficult. This widespread adoption and utility make subtitles a prime, yet previously unexamined, vector for information leakage.

Key Findings

▶ Watch: Core idea: leveraging blurred subtitle shapes (C-houses) (3:50)

The central and most impactful finding of the SilhouetteTell research is the discovery that even blurred recordings of video subtitles contain sufficient information to uniquely identify a video. This revelation uncovers a novel side-channel vulnerability that bypasses many traditional defenses against video identification attacks. The research demonstrates that the shape and temporal sequence of subtitle regions, referred to as C-houses (silhouette of subtitle region), are unique enough to serve as a reliable fingerprint for specific video content, even when the textual content itself is completely illegible.

Initial attempts to extract information using standard optical character recognition (OCR) or advanced image captioning algorithms like ClipCap proved unsuccessful on blurred subtitle frames. This failure underscores that the attack does not rely on deciphering the text, but rather on the geometric properties of the subtitle bounding boxes. The varying lengths of words, sentence structures, and the timing of their appearance create distinct visual patterns that persist even under significant blurring.

The practicality of the SilhouetteTell attack is another key finding. The attacker requires only a standard RGB camera (e.g., from a smartphone) and can operate from a considerable distance, demonstrated effectively from over 3 meters. Crucially, the attack is non-invasive, requiring no malware, no specialized equipment, and no user-specific training. It also does not rely on network traffic analysis, making it robust against common network-based privacy defenses. This low barrier to entry for attackers makes SilhouetteTell a particularly concerning threat.

Through extensive experimentation, the researchers validated the effectiveness and robustness of their approach. The SilhouetteTell system achieved a remarkably high Top-1 accuracy of over 91% in identifying videos from a diverse dataset of 300 movies. This high accuracy confirms the viability of extracting unique video fingerprints from blurred subtitle shapes and highlights the significant privacy implications for users who rely on subtitles for accessibility or preference. The research conclusively proves that the seemingly innocuous visual artifacts of subtitles, even when rendered unreadable, are a potent source of identifiable information.

Technical Deep Dive

▶ Watch: Methodology: Mask R-CNN for accurate subtitle shape extraction (5:50)

The SilhouetteTell attack operates under a specific and highly practical threat model. The victim is assumed to be watching subtitled videos on any device (e.g., laptop, tablet, smartphone) in a public place. The attacker, on the other hand, is equipped with only a standard RGB camera—typically that of a smartphone—and records the victim's screen from a distance exceeding 3 meters. Critically, the threat model explicitly excludes the need for malware, specialized surveillance equipment, user-specific training, or network traffic analysis, making the attack highly accessible and difficult to detect.

The technical architecture of SilhouetteTell is divided into two primary phases: a training phase and an inference phase.

Training Phase: Refined C-house Extraction

The foundation of SilhouetteTell lies in accurately extracting the C-houses, or the "silhouette of subtitle region," from video frames. These C-houses represent the bounding boxes or shapes of the subtitle text on the screen. The challenge is that existing coarse segmentation algorithms, such as MS ER (Maximally Stable Extremal Regions) or GrabCut, were found to be insufficiently accurate for this task. Their outputs were too imprecise to capture the subtle variations in subtitle shapes necessary for reliable video identification.

To overcome this limitation, the researchers developed a refined C-house extraction method utilizing Mask R-CNN (Mask Region-based Convolutional Neural Network). Mask R-CNN is a powerful deep learning model designed for object detection and instance segmentation. It can accurately identify objects within an image and generate a precise pixel-level mask for each instance. In the context of SilhouetteTell, Mask R-CNN is trained to recognize and precisely segment the subtitle regions, even when they are blurred or partially obscured. This precision is paramount because the attack relies on minute differences in the shape and dimensions of these regions. By leveraging Mask R-CNN, the system can extract highly accurate sequences of C-houses, forming the raw data for subsequent analysis.

Inference Phase: Video Identification

Once a sequence of refined C-houses has been extracted, the inference phase aims to match this sequence against a database of known video fingerprints. This phase involves several sophisticated steps:

  1. Distinguishing Successive C-houses: A crucial challenge is determining whether two successive C-houses in a video frame sequence belong to the same subtitle or represent a transition to a new subtitle. The system addresses this using the Intersection Over Union (IOU) metric. IOU measures the overlap between two bounding boxes. If the IOU between a C-house in frame t and a C-house in frame t+1 is low, it indicates they likely correspond to different subtitles. The talk illustrates this with several cases:
  • Case 1 (Totally Separate): C-houses are distinct with no overlap, clearly from different subtitles.
  • Case 2 (Different Lines): C-houses occupy different vertical positions, indicating separate subtitles.
  • Case 3 (Partial Overlap): C-houses partially overlap but have distinct boundaries, suggesting different subtitles.
  • Case 4 (Same Height, Partial Overlap): This is the most challenging scenario, where C-houses share similar heights and partially overlap, making IOU alone insufficient. For such ambiguous cases, the system utilizes "preprocessed frames" to further confirm whether they originate from the same subtitle. While the talk does not detail the exact nature of these "preprocessed frames," it implies additional visual cues or contextual analysis beyond simple bounding box overlap to resolve these difficult transitions.
  1. Feature Extraction: After accurately segmenting and associating C-houses, the system extracts both spatial and temporal features from sequences of these regions.
  • Spatial Features: These describe the intrinsic properties of individual C-houses within a frame. Examples include the width, height, aspect ratio, position (e.g., distance from bottom-right corner), and overall area of the subtitle region. The varying lengths of words and sentences in subtitles naturally lead to diverse spatial characteristics.
  • Temporal Features: These capture the dynamics and sequence of C-houses over time. This includes the duration a C-house is displayed, the time intervals between successive subtitles, and the patterns of changes in spatial features across a sequence. For instance, a common pattern might involve a short subtitle followed by a longer one, or a sequence of subtitles with increasing or decreasing heights.

The talk provides a specific example of creating a spatial-temporal feature vector by sorting C-houses based on their height. For a sequence of six C-houses, an attacker might identify two with lower heights and four with longer heights. This ordered arrangement, combined with other spatial and temporal attributes, forms a unique feature vector that acts as a fingerprint for that specific snippet of video.

  1. Video Matching: The final step involves matching the extracted spatial-temporal feature vector sequence from the observed video clip against a pre-built database of known video fingerprints. The system employs a sliding window matching algorithm. This algorithm likely compares the feature vector sequence of the observed segment against all possible contiguous sub-sequences in the database's video fingerprints. When a sufficiently high similarity score is achieved within a certain window, it identifies a candidate video. The more unique and longer the sequence of C-houses observed, the higher the confidence in the match and the smaller the set of potential candidate videos.

Demo / Proof of Concept

▶ Watch: Detailed methodology: extracting spatial and temporal features (7:00)

The practical viability and effectiveness of the SilhouetteTell attack were rigorously demonstrated through extensive experiments. The researchers constructed a comprehensive dataset comprising 300 movies sourced from various streaming services. This diverse dataset ensured that the attack was tested against a wide range of subtitle styles, fonts, and display characteristics inherent to different production houses and platforms.

For the recording phase, the attack simulated a real-world scenario where an attacker might surreptitiously record a victim's screen. All video clips were recorded using a standard smartphone camera from a distance exceeding 3 meters. This setup validates the low-cost and readily available nature of the attack, reinforcing its practical threat.

The primary metric used to evaluate the attack's performance was Top K accuracy. This metric assesses how often the correct video is found within the top K candidates identified by the system. The results were compelling: SilhouetteTell achieved an "almost like above 91%" Top-1 accuracy. This indicates that in over 91% of trials, the system correctly identified the specific movie as its very first prediction, demonstrating exceptional precision and reliability.

A specific case study discussed during the talk further illustrated the attack's effectiveness. It showed that for a 2-minute clip containing a sufficient number of subtitles, the number of potential candidate videos generated by the system significantly shrinks. This highlights the power of accumulating spatio-temporal features over time; longer observation periods with more subtitle events lead to increasingly precise and confident identifications, narrowing down the possibilities to a very small, often singular, correct video. This robust performance across a diverse dataset and under realistic recording conditions firmly establishes SilhouetteTell as a practical and potent video identification attack.

Defensive Implications

▶ Watch: Discussing defense strategies against the SilhouetteTell attack (8:50)

The emergence of SilhouetteTell necessitates a re-evaluation of current video privacy strategies and highlights new vectors for information leakage. The most straightforward, albeit often impractical, defense is to disable subtitles entirely. For many users, particularly those watching foreign language content, individuals with hearing impairments, or those in environments where audio is challenging, subtitles are indispensable. Therefore, simply turning them off is not a universally acceptable solution.

Another physical countermeasure mentioned is the use of privacy screen protectors. These physical overlays reduce the viewing angle of a screen, making it difficult for bystanders to see the content clearly. By further blurring or distorting the on-screen display from oblique angles, privacy screens could potentially make the precise extraction of C-houses more challenging for an attacker. However, their effectiveness against a direct head-on recording, even if blurred, might vary, and they do not fundamentally address the information leakage inherent in subtitle rendering.

The talk briefly mentions "fast get subtitles C-houses" in the context of defense. While the talk does not elaborate, this likely refers to advanced obfuscation techniques or dynamic display mechanisms designed to thwart C-house extraction. Potential strategies could include:

  • Dynamic Subtitle Rendering: Platforms could dynamically alter subtitle properties such as font, size, position, or even introduce subtle, random distortions to the bounding box shapes over short intervals. This would make it difficult for an attacker's Mask R-CNN model to consistently extract stable and identifiable C-house features.
  • Contextual Noise Injection: Introducing visual noise or subtle, rapidly changing background elements around subtitles could interfere with the segmentation process, making it harder to isolate clear C-houses.
  • Adaptive Display: A more advanced defense might involve systems that detect potential surveillance attempts (e.g., through camera detection or unusual viewing angles) and adaptively change subtitle rendering to minimize information leakage.

Beyond these specific measures, the SilhouetteTell research underscores a broader implication: video platforms and content providers must now consider subtitle rendering as a potential privacy vulnerability. Just as attention has been paid to encrypting network traffic or protecting content from direct observation, the visual characteristics of UI elements like subtitles must also be scrutinized for unintended information leakage. Future platform designs should incorporate privacy-by-design principles for all on-screen elements, ensuring that even non-textual visual cues do not inadvertently reveal sensitive user data. This could involve standardizing subtitle dimensions, introducing random variations in their display, or exploring entirely new, privacy-preserving methods of presenting textual information.

Key Takeaways

  • Blurred subtitle shapes (C-houses) constitute a novel side-channel that can be exploited to identify specific videos being watched, even when the content is otherwise unreadable.
  • The SilhouetteTell attack bypasses traditional defenses such as content blurring, offline viewing, network traffic analysis, and the absence of specialized equipment or malware.
  • Mask R-CNN is crucial for the precise extraction of subtitle silhouettes, and a sophisticated spatio-temporal correlation algorithm is key to creating unique video fingerprints.
  • The attack is highly practical and accessible, requiring only a standard smartphone camera to record from a distance of over 3 meters, achieving a Top-1 accuracy exceeding 91%.
  • Disabling subtitles is a direct defense, but more practical solutions may include privacy screen protectors or dynamic subtitle obfuscation techniques to vary C-house characteristics.
  • Video streaming platforms must recognize subtitle rendering as a privacy concern and integrate privacy-by-design principles into their UI elements to prevent such information leakage.

About the Speaker(s)

The research presented in "SilhouetteTell: Practical Video Identification Leveraging Blurred Recordings of Video Subtitles" was a collaborative effort from the University of Oklahoma. The talk at PETS was delivered by Guanchong Huang, who is listed as a paper author. The primary author and the individual who performed most of the work was Song Fang, a PhD student at the University of Oklahoma, who unfortunately could not attend the conference to present the findings in person. Their combined expertise in computer science and security research led to this significant contribution to understanding privacy vulnerabilities in video consumption.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

This is a classic, elegant side-channel attack that ignores the payload entirely and profiles the metadata. Fang and Huang realized that even if a privacy screen turns text into a smeared blob, the geometry and timing of those blobs are mathematically unique. It's exactly the kind of clever, low-barrier physical OPSEC threat that proves visual data leaks way more than we think.

Heather Calloway (CISO) — WEAK

This is academic side-channel research demonstrating how an attacker can identify a video playing on a blurred screen by fingerprinting the shape and timing of its subtitles. While the computer vision work is clever, the enterprise threat model is practically nonexistent since the attack requires a pre-existing database of the target videos to make a match.

→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026

All talks from Privacy Enhancing Technologies Symposium (PETS) 2026