Thorn: AI for Child Safety at Scale

Thorn: AI for Child Safety at Scale

Speakers: Julie Cordua, CEO, Thorn

Conference: CS153 Infra @ Scale 2025

YouTube: https://www.youtube.com/watch?v=MBD0Ah9cpYU

Overview

In this compelling and critically important talk, Julie Cordua, CEO of Thorn, a nonprofit dedicated to protecting children online, unveiled the staggering scale of child sexual abuse material (CSAM) and the innovative technological solutions Thorn is deploying to combat it. The discussion navigated the harrowing realities of online child exploitation while simultaneously highlighting the profound potential of technology, particularly artificial intelligence (AI), to safeguard vulnerable populations. Cordua's presentation underscored the urgent need for tech companies to move beyond passive compliance and actively leverage advanced tools to enforce their terms of service, protect children, and assist law enforcement in the most challenging of investigations.

Thorn's mission, born 13 years ago amidst the rise of the iPhone and social media, has evolved significantly. Initially focused on bringing engineers together to acknowledge the emerging problem, the organization recognized a critical void: the lack of specialized software to address CSAM at scale. Today, Thorn develops and deploys sophisticated AI-powered platforms, such as Safer Match and Safer Predict, directly within tech companies and for law enforcement agencies. This talk not only detailed the technical architecture and impact of these tools but also emphasized the ethical considerations and human toll involved in content moderation, advocating for solutions that protect both children and the well-being of the moderators tasked with viewing horrific material.

The importance of this talk cannot be overstated. As digital platforms become increasingly pervasive and AI technologies advance at an unprecedented pace, the methods used by perpetrators of child abuse also evolve, creating new vectors for harm, including generative AI producing lifelike images and sophisticated AI personas for grooming and sextortion. Cordua's insights provide a roadmap for how the tech industry, in collaboration with nonprofits and law enforcement, can meet these challenges head-on, transforming technology from a facilitator of abuse into a powerful force for protection and justice.

Background

▶ Watch: Introduction to Thorn and its mission (0:00)

The problem of child sexual abuse material (CSAM) online predates the modern internet era, but its scale exploded with the advent of social media and smartphones. Prior to 2004, before platforms like Facebook and the iPhone democratized content creation and sharing, hundreds of thousands of CSAM images were already being reported to the National Center for Missing and Exploited Children (NCMEC). Law enforcement agencies, even then, were overwhelmed, struggling to investigate the origins of these images and locate the children involved.

The proliferation of smartphones equipped with cameras and easy access to cloud storage and social media platforms provided perpetrators with unprecedented tools. This led to a dramatic, exponential increase in CSAM files circulating online, with NCMEC eventually receiving over 100 million files annually from tech companies alone, not accounting for data seized from physical devices. This surge created a dual crisis: tech companies lacked the means to detect and remove this vast quantity of illegal content, and law enforcement, whose expertise lies in investigation rather than data science, was drowning in an unmanageable volume of data.

Legally, in the United States, tech companies are not mandated to proactively detect CSAM. However, if they do detect it, they are legally obligated to remove it from their platforms and report it to NCMEC, a designated repository for such content. This framework created a reactive environment where abuse often circulated widely before being identified.

Thorn, initially conceived as a research and convening organization 13 years ago, quickly recognized the need for practical software solutions. Their initial response was Safer Match, a system that allowed companies to perform hash matching against a known database of CSAM files. This technology identified exact duplicates of previously seen illegal content, preventing its re-upload and ensuring its removal.

However, the landscape shifted dramatically about two and a half years prior to the talk with the emergence of generative AI. Criminals, often early adopters of new technologies, rapidly leveraged open-source generative AI models to produce highly realistic, illegal material. This introduced a new, complex layer to the problem: discerning between images of real children and AI-generated fakes, or even images of real children manipulated into abusive contexts. This new form of CSAM not only added to the sheer volume but also created significant investigative challenges, as law enforcement could spend months or years pursuing a "child" who doesn't exist, diverting critical resources from real victims. Beyond images, generative AI also facilitated the development of sophisticated voices and personas for grooming and sextortion scams, often perpetrated by organized crime syndicates. This evolving threat landscape necessitated a new generation of defensive technologies.

Key Findings

▶ Watch: Law enforcement overwhelmed by scale of CSAM (3:00)

Thorn's work has revealed several critical findings about the nature of online child exploitation and the effectiveness of technological countermeasures:

1. Explosive Scale and Growth: The volume of CSAM is immense and continues to grow. NCMEC receives over 100 million files annually from tech companies. Thorn itself processed approximately 200 billion files for its 60+ partner companies in the past year alone. This sheer scale necessitates automated, AI-driven solutions.
2. Generative AI's Dual Threat: Generative AI has introduced a new and rapidly growing category of CSAM. While images of real children still constitute the vast majority of abuse, AI-generated content is "skyrocketing from zero." This includes completely fabricated images, as well as the manipulation of existing images of real children into abusive contexts, leading to new forms of victimization. This makes investigations significantly more complex, as law enforcement must now distinguish between real and AI-generated content.
3. Predictive AI's Transformative Impact: Thorn's Safer Predict models are proving indispensable. They now account for over half of all CSAM detections on partner platforms, a significant shift from just a few years ago. Crucially, these predictive models identify content that the world has never seen before, which most likely represents new abuse of currently endangered children. This capability directly leads to active investigations and child rescues, as demonstrated by the SmugMug case where a child was recovered in Austria within 24 hours of detection.
4. Criticality of Authentic Data: The accuracy and effectiveness of AI models for CSAM detection are directly tied to the quality and authenticity of their training data. Thorn's partnership with NCMEC, granting access to the world's largest repository of actual CSAM, has been vital. The speaker noted that models trained on pornography or general child imagery are significantly less effective because CSAM is fundamentally different—it is abuse, not pornography. Similarly, access to real grooming conversation data is crucial for developing robust text-based harm detectors.
5. Industry Collaboration is Possible and Necessary: Thorn successfully convened major generative AI companies, including OpenAI, Anthropic, Inflection, Microsoft, and Google, to agree on "by design" principles aimed at reducing the likelihood of their models being used to produce CSAM. This proactive engagement at the development stage is crucial for building safer AI from the ground up.
6. Pervasiveness of Harm: Cordua asserts that "anywhere that there is user-generated content, there is child abuse," a claim that has yet to be disproven. This highlights the universal applicability of Thorn's solutions across diverse online platforms, from social media to messaging apps and even profile images on services like Spotify and WhatsApp.
7. Mental Health of Moderators: The human element of content moderation, particularly for CSAM, is profoundly challenging. Thorn's development of a specialized review tool, designed with mental health in mind (e.g., blurred images, text descriptions, silenced audio, selective viewing), acknowledges and addresses this critical aspect, demonstrating that effective solutions must consider the well-being of those on the front lines.

These findings collectively underscore the evolving nature of online child exploitation, the necessity of advanced AI-driven solutions, and the critical role of collaboration between technology providers, nonprofits, and law enforcement in protecting children at scale.

Technical Deep Dive

▶ Watch: Thorn's initial solution: hash matching for CSAM (4:30)

Thorn's core offering is a suite of products under the umbrella of its Safer platform, designed to detect and remove CSAM and other related harms from online platforms. The architecture is built around two primary components: Safer Match and Safer Predict, complemented by specialized tools for content moderation and law enforcement.

Safer Match: Hash-Based Detection

The original solution, Safer Match, leverages hash matching technology. When an image or video is uploaded to a partner platform (e.g., X, SmugMug), its unique digital signature (hash) is generated. This hash is then compared against a comprehensive database of known CSAM hashes. If a match is found, it signifies that the content is a known piece of illegal material that has been seen and verified previously.

• Deployment: Safer Match is typically deployed as an enterprise system, sitting within the tech company's infrastructure (e.g., S3 buckets). This allows companies to maintain control over their data while utilizing Thorn's detection capabilities.
• Action: Upon a hash match, the system flags the content for the company's content moderation team. Because hash matches are generally "triple verified," they offer a high degree of certainty, allowing for automated removal and direct reporting to NCMEC via a dedicated API without requiring human review of the explicit content.

Safer Predict: AI Classifiers for Novel Content

Recognizing the limitations of hash matching for new or manipulated content, especially with the rise of generative AI, Thorn developed Safer Predict. This component utilizes AI classifiers (predictive models) based on computer vision and machine learning to identify CSAM in images, videos, and increasingly, text.

• Training Data: The efficacy of Safer Predict relies heavily on access to vast, authentic datasets. Thorn has established a critical partnership with NCMEC, gaining access to the largest repository of CSAM globally to train its image and video classifiers. For text-based harm detection, models are trained on actual grooming conversations, which has proven far more effective than training on general pornography or unrelated datasets.
• Functionality: Safer Predict analyzes new content for patterns and characteristics indicative of CSAM or grooming behavior. Unlike hash matching, it can identify novel forms of abuse, including AI-generated images, manipulated content, and previously unseen real-world abuse.
• Deployment: Safer Predict can be integrated via the enterprise system or through an easier-to-start API model, making it accessible to a wider range of companies, including startups.
• Human-in-the-Loop: Due to the nature of predictive AI, there is an inherent degree of false positives. Therefore, Safer Predict requires human content moderators to review flagged content and make final judgment calls. This ensures accuracy and compliance with legal reporting requirements.

Text-Based Harm Detection

An emerging area of focus for Safer Predict is the detection of text-based harms, particularly related to grooming and sextortion. By analyzing patterns in online conversations, Thorn's models aim to identify interactions that are likely to escalate into abusive situations, allowing for earlier intervention. This moves beyond reactive content removal towards proactive prevention.

Review Tool for Moderator Mental Health

A crucial, often overlooked, technical component is Thorn's specialized review tool for content moderators. Recognizing the severe psychological toll of viewing CSAM, this tool is designed with mental health considerations at its forefront:

• Blurred Content: All images and videos in the moderation queue are initially blurred.
• Text Descriptions: A textual description of the content is provided, allowing moderators to understand the nature of the material without immediate visual exposure.
• Audio Silencing: For videos, sound is automatically turned off.
• Selective Viewing: Moderators can use a "cursor pen" to selectively reveal only necessary portions of an image or video to make a judgment, minimizing exposure to graphic content.

Law Enforcement Integration

Thorn's predictive solutions have also been integrated into forensic tools used by law enforcement. When investigators seize hard drives containing vast amounts of data, Thorn's tools can rapidly process gigabytes of information, reducing millions of files down to a few thousand relevant images that require human attention, significantly accelerating investigations.

API to NCMEC

A direct API connection to NCMEC ensures that once illegal content is confirmed, it can be reported swiftly and efficiently, fulfilling legal obligations and facilitating the immediate initiation of investigations to find and rescue children.

In essence, Thorn's technical strategy combines robust, high-certainty hash matching for known content with advanced, data-intensive AI prediction for novel threats, all while incorporating vital human-centric design for moderator well-being and seamless integration with law enforcement workflows.

Demo / Proof of Concept

▶ Watch: New forms of victimization using generative AI (7:00)

While the talk did not feature a live demonstration of Thorn's software, Julie Cordua vividly described its functionality and impact through several compelling examples and explanations of its operational mechanics. These narratives served as powerful proof points for the effectiveness of Thorn's approach.

One of the most impactful "proof of concept" stories involved an early beta partner for the Safer Predict classifier, SmugMug. The head of content moderation at SmugMug, upon seeing a notification from the newly implemented classifier, initially suspected a false positive. However, upon opening the file (using Thorn's specially designed, blurred review tool), he discovered an image of abuse involving an 8- or 9-year-old girl. Further investigation, enabled by the timestamp and account details, revealed approximately 200 additional images of the same child within the user's backlog. This discovery, made possible by the predictive model, led to an immediate report. Within 24 hours, law enforcement in Austria acted on the report, recovered the child, and arrested the father who had been abusing her for years and publishing the content online. This real-world outcome powerfully illustrates how Safer Predict identifies new abuse that hash matching would miss, directly leading to child rescues.

Another key "demonstration" of Thorn's technical approach is the Review Tool itself, designed with mental health in mind. Cordua detailed how this tool mitigates the psychological burden on content moderators. Images are blurred by default, video sound is off, and text descriptions provide context. Moderators can use a "cursor pen" to selectively reveal only the necessary parts of an image to make a judgment, minimizing exposure to graphic content. This design, while not visually demonstrated, highlights Thorn's commitment to a holistic solution that protects both children and the individuals safeguarding online spaces.

Furthermore, Cordua explained how Thorn's predictive solutions are integrated into forensic tools for law enforcement. This "proof of concept" involves the ability to process gigabyte-sized hard drives, reducing potentially millions of images to a manageable few thousand relevant files in minutes. This dramatically accelerates investigations, allowing law enforcement officers to focus on critical evidence rather than sifting through irrelevant data.

Finally, the talk highlighted the sheer volume of data processed—200 billion files annually for 60 different tech companies—and the shift in detection methods, with predictive models now accounting for over 50% of detections. This statistical evidence underscores the operational scale and the growing reliance on AI-driven prediction over traditional hash matching for identifying novel CSAM. These examples, though verbal, painted a clear picture of Thorn's practical and life-saving impact.

Defensive Implications

▶ Watch: Call to action: responsible innovation and harm consideration (8:00)

The insights shared by Julie Cordua offer critical defensive implications for tech companies, law enforcement, and the broader online community:

1. Proactive Enforcement is Non-Negotiable: Companies can no longer afford to be reactive. While US law doesn't mandate detection, terms of service prohibiting illegal activity, especially CSAM, are meaningless without proactive enforcement. Platforms must implement robust systems like Thorn's Safer suite to actively scan for and remove illegal content.
2. Integrate Safety Early in the Development Cycle: Delaying safety measures until a platform is mature can lead to it becoming a "hub of abuse." Integrating solutions like Thorn's Safer Match and Safer Predict early prevents platforms from gaining a reputation as a safe haven for perpetrators, making it easier to scale securely.
3. Adopt a Hybrid Detection Strategy: Relying solely on hash matching is insufficient in the age of generative AI. Defenders need a dual approach, combining hash matching for known CSAM with predictive AI classifiers to identify novel, manipulated, or AI-generated abuse. This ensures comprehensive coverage against evolving threats.
4. Prioritize Moderator Mental Health: Content moderation, especially for CSAM, takes a severe psychological toll. Companies must invest in specialized review tools (like Thorn's) that minimize exposure to graphic content, offer blurred views, and provide textual context. This is crucial for retaining talent and ensuring the long-term sustainability of moderation efforts.
5. Leverage AI for Investigative Efficiency: Law enforcement agencies should integrate AI-powered forensic tools, such as Thorn's predictive solutions, to drastically reduce the volume of data requiring human review on seized devices. This frees up investigators to focus on actionable intelligence and victim identification.
6. Address Grooming and Sextortion Proactively: With the rise of AI-driven personas for grooming and sextortion, platforms must deploy text-based harm detectors and pattern recognition algorithms to identify suspicious conversations and organized crime networks. Early interception of these interactions can prevent abuse before it escalates.
7. Transparency with Users: Companies should clearly communicate their commitment to child safety and the measures they employ. Research suggests that users are generally supportive of platforms taking strong action against CSAM, even if it involves scanning non-encrypted public areas of their services (e.g., profile images).
8. Re-evaluate Encrypted Environments: While respecting privacy, platforms offering encrypted communications need to consider "other mechanisms" for creating safety, as suggested by the Australian e-safety commission. This might involve detection in non-encrypted elements (like profile pictures) or exploring privacy-preserving methods to identify criminal networks.
9. Promote Industry-Wide Collaboration: Engaging with initiatives like Thorn's "by design" principles for generative AI is vital. Tech companies must collaborate to establish ethical guidelines and build safety features into AI models from their inception, preventing their misuse for illegal purposes.
10. Emphasize Parental Education: Given the speed of technological change and the sophistication of online harms, a new era of "different kind of parenting" is required. Parents need to understand how technology can be used for harm and engage in continuous dialogue with their children about online safety.

The overarching message is clear: the fight against online child exploitation requires a proactive, multi-faceted, and continuously evolving defensive strategy, leveraging the best of technology, human expertise, and collaborative effort.

Key Takeaways

• The Scale of CSAM is Immense and Growing: Over 100 million CSAM files are reported annually to NCMEC, with Thorn processing 200 billion files for partners, highlighting the critical need for automated detection at scale.
• Generative AI is a Game-Changer: AI-generated and manipulated CSAM is skyrocketing, complicating investigations and creating new forms of victimization, necessitating advanced predictive models beyond traditional hash matching.
• Predictive AI Drives Child Rescues: Thorn's Safer Predict models now account for over 50% of detections, identifying new abuse that leads directly to investigations and the recovery of children, as exemplified by the SmugMug case.
• Data Access is Paramount for Model Accuracy: Training AI models on authentic CSAM data from sources like NCMEC is crucial for high-quality, effective detection, distinguishing abuse from other content.
• Human-Centric Design is Essential for Moderators: Tools like Thorn's specialized review interface, which blurs content and offers selective viewing, are vital for protecting the mental health of content moderators dealing with horrific material.
• Proactive Safety and Early Integration are Critical: Tech companies must embed safety solutions like Thorn's Safer platform early in their development cycles to prevent their platforms from becoming hubs for illegal activity and to enforce their terms of service effectively.

About the Speaker(s)

Julie Cordua is the CEO of Thorn, a nonprofit organization dedicated to leveraging technology to protect children from sexual exploitation online. Her career began in wireless technology, working at companies like Motorola and an early venture to bring live streaming television to the US. This background instilled in her a deep appreciation for technology's potential to change the world.

Thirteen years ago, Cordua was introduced to the complex intersection of technology and child sexual abuse, a field that profoundly shifted her professional focus. Under her leadership, Thorn evolved from a research and convening organization into a software company, actively building and deploying tools for tech companies and law enforcement. She is a passionate advocate for technological innovation, but equally emphasizes the responsibility of innovators to consider the "worst possible way" their creations could be used. Cordua's experience includes spearheading collaborations with major AI companies to establish "by design" safety principles, underscoring her commitment to proactive prevention and ethical technology development. She leads a team primarily composed of engineers and data scientists, inspiring them with the mission-driven impact of their work—directly contributing to child rescues.