Teenagers' Ability to Detect Synthetic Media

Teenagers' Ability to Detect Synthetic Media

Speakers: Aaliyah, High School Senior

Conference: BSides Seattle

YouTube: https://www.youtube.com/watch?v=6HeNx27Ww2k

Overview

In a standout presentation from a high school senior, Aaliyah shared original research on how well teenagers can detect AI-generated synthetic videos — a question that had not been studied before her work. Using OpenAI's Sora platform to generate synthetic videos and the Pexels stock video library for authentic footage, Aaliyah surveyed 32 participants across four content categories (people, nature, animals, objects) and found that teenagers detected synthetic media with 66% accuracy while consistently underestimating their own confidence.

This research fills a genuine gap in the literature. Prior deep fake detection studies focused exclusively on adult participants and traditional face-swap deep fakes. Aaliyah's work is the first to study teenagers' detection ability against fully synthetic AI-generated video (defined as "synthetic media" per a Harvard Kennedy School classification) produced by next-generation platforms like Sora. The finding that teenagers outperform adults from prior studies (66% vs. 57.6%) and underestimate rather than overestimate their confidence contradicts the Dunning-Kruger patterns observed in adult populations.

The work raises an important question for the security community: if casual, unintentional exposure to AI-generated content on social media platforms is creating a passive training effect that improves detection ability, should structured media literacy education formalize and accelerate that process rather than relying solely on regulation?

Background

▶ Watch: Introduction: high school senior's independent research on synthetic media de... (0:15)

Aaliyah began her research in September 2024 after reading a New Yorker article about AI voice deep fake scams — scenarios where attackers clone a family member's voice to demand ransom. Inspired by a dinner conversation with her parents, she set out to conduct fully independent research that would fill a gap in existing literature, a requirement of her research class.

A review of over 50 studies from more than 37,200 results on Google Scholar revealed that deep fake detection research focused on older technologies (face-swap deep fakes from 2015-2019 era platforms) and adult participants. When OpenAI launched Sora in December 2024, a platform that generates fully new videos from text prompts rather than altering existing footage, Aaliyah identified both a new technology category and an unstudied population.

Two prior studies were particularly influential on her methodology. "Fooled Twice" (2021) tested adult accuracy and self-assessed confidence in identifying traditional deep fakes using short video clips with 50/50 authentic-to-fake ratios. "Deep Fakes and Disinformation" (2020) used a single deep fake of President Obama with adult participants measuring accuracy and confidence. Both established the survey format (identify real vs. fake, rate confidence on a numeric scale) that Aaliyah adapted.

The Harvard Kennedy School's "Synthetic Media" paper from the Misinformation Review provided the taxonomic framework, defining Sora-generated content as "synthetic media" — fully AI-generated content not based on any pre-existing footage — distinguishing it from traditional deep fakes that alter existing video.

Key Findings

▶ Watch: Methodology pivot: from longitudinal study to cross-sectional design (6:00)

66% Detection Accuracy: Teenagers correctly identified synthetic vs. authentic videos 66% of the time, outperforming the 57.6% adult accuracy rate from comparable prior studies using traditional deep fake technologies.

Systematic Confidence Underestimation: 87.5% of participants (28 of 32) underestimated their detection ability, with an average underestimation of approximately 2 points on a 10-point scale. The mean self-assessed confidence was 5.91 out of 10 (standard deviation ~2.1), despite actual accuracy significantly exceeding their self-assessment.

Reversed Dunning-Kruger Pattern: In adult deep fake detection studies, participants overestimated their confidence. Teenagers showed the opposite pattern — the highest performers (scoring 13 out of 16) were among those who most underestimated their ability. This contradicts established patterns from adult populations.

Pearson Correlation: A Pearson correlation coefficient analysis (R = 0.35, P = 0.047) confirmed a statistically significant moderate connection between self-assessed confidence and actual accuracy, validating that the underestimation pattern is not random.

Category-Dependent Detection: Participants were significantly better at detecting synthetic videos of people (whom they interact with daily in real life) versus animals. One specific synthetic video of a lion with its mane moving in the wind was identified by zero participants — people lack the real-world reference frame to assess wildlife realism.

Exposure-Driven Detection: The top performers (13/16 accuracy) reported high exposure to AI-generated content on YouTube and social media. Average teenager screen time of 4.8 hours/day versus 2.35 hours/day for adults correlated with better detection, suggesting passive training through exposure. A 2024 Indonesian journal study on Generation X confirmed that self-taught digital literacy and exposure to AI-generated content improved detection ability in older populations as well.

Qualitative Detection Cues: Top performers reported looking for glitchy hands, oddly fluid movement, colors that were too vibrant, and textures that were too smooth — patterns they could only identify through prior exposure to AI-generated content.

Technical Deep Dive

▶ Watch: Exposure hypothesis: 4.8 hours daily screen time correlates with better detec... (10:00)

The study methodology used a within-subjects design with 32 participants (ages 14-18.5). The survey presented 8 videos: 4 authentic from Pexels (a free stock video platform) and 4 synthetic from OpenAI's Sora, distributed across four categories (people, nature, animals, objects). All selected videos were plausible — no obviously unrealistic content was included to avoid floor effects.

For each video, participants made a binary classification (AI-generated or authentic) and provided a confidence rating on a 1-10 scale. The maximum possible accuracy score was 16 (8 videos x 2 points per correct identification).

Statistical analysis included descriptive statistics (mean confidence 5.91, SD ~2.1), a Pearson correlation coefficient (R = 0.35, P = 0.047) testing the relationship between self-assessed confidence and accuracy, and categorical analysis of detection rates across video content types.

An initial longitudinal design (testing improvement over 5 days with feedback for an experimental group vs. control) was abandoned due to participant dropout below the threshold for statistical significance. The pivoted cross-sectional design sacrificed the ability to test improvement over time but achieved sufficient sample size for meaningful analysis.

Follow-up qualitative surveys were sent to the four highest-scoring participants to understand their detection strategies, revealing exposure-based pattern recognition as the common factor.

Demo / Proof of Concept

▶ Watch: Qualitative cues: glitchy hands, fluid movement, vibrant colors (12:00)

While this was a research presentation rather than a technical demonstration, Aaliyah showed the Sora interface used to generate synthetic videos and the Pexels stock video interface used for authentic footage. She highlighted specific videos that confused participants, including a synthetic video that the majority misidentified as authentic and the lion video that zero participants correctly identified as synthetic. Media playback was not functional during the presentation, but the video descriptions and participant response data were presented through slides.

Defensive Implications

▶ Watch: Q&A: category-dependent detection — people vs. animals vs. nature (16:00)

This research has direct implications for social engineering defense and organizational security awareness programs. The finding that passive exposure to AI-generated content improves detection ability suggests that structured media literacy training could meaningfully reduce susceptibility to synthetic media-based attacks (voice deep fake scams, synthetic video social engineering, non-consensual synthetic imagery).

The category-dependent detection finding (better at detecting synthetic people than synthetic animals/nature) indicates that training programs should focus on content types that people encounter less frequently in real life, where they lack the reference frame to spot inconsistencies.

The generational gap in detection ability (teenagers outperforming adults) has implications for organizational security posture: employees with less social media exposure may be more vulnerable to synthetic media attacks. Security awareness training should incorporate exposure to AI-generated content as a detection skill-building exercise.

The speaker noted that only one bill limiting synthetic content creation (the Take It Down Act, criminalizing non-consensual synthetic imagery) has been passed, and argued that education, not just regulation, is necessary to protect people in the AI era. For security leaders, this suggests that technical controls and user education must complement each other — relying solely on either regulation or detection technology is insufficient.

Key Takeaways

• Teenagers detect synthetic media (Sora-generated videos) with 66% accuracy, outperforming the 57.6% adult accuracy rate from prior deep fake studies, with higher daily screen time (4.8 hours vs. 2.35 hours) correlating with better detection
• 87.5% of teenage participants underestimated their detection confidence, reversing the overconfidence pattern observed in adult Dunning-Kruger studies on deep fake detection
• Detection accuracy is category-dependent: people are much better at detecting synthetic humans (familiar reference frame) than synthetic animals or nature scenes
• Passive exposure to AI-generated content on social media creates a training effect that improves detection — structured media literacy programs could formalize and accelerate this
• Pearson correlation (R = 0.35, P = 0.047) confirms a statistically significant moderate link between self-assessed confidence and actual accuracy in the teenage population
• The Take It Down Act is the only legislation addressing synthetic content; education must complement regulation to address the full scope of synthetic media threats

About the Speaker(s)

Aaliyah is a high school senior who conducted this research as an independent project for her research class, reviewing over 50 studies from more than 37,200 Google Scholar results to identify her research gap. The study was developed over approximately nine months, including a methodology pivot from a longitudinal feedback design to a cross-sectional design after participant dropout. Her research represents the first known study of teenage synthetic media detection using next-generation fully AI-generated video platforms, and she plans to continue exploring the relationship between exposure, education, and detection ability.