Pwning User Phishing Training Through Scientific Lure Crafting

Pwning User Phishing Training Through Scientific Lure Crafting

Speakers: Christian DeMeff, Associate Professor of Emergency Medicine, Biomedical Informatics, and Computer Science, UCSD (co-director, Center for Healthcare Cybersecurity); Ariana Mirian, Security Researcher, Census

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=YKHlOHhKsvI

Reading time: ~7 minutes

Type: Briefing

TL;DR

A randomized controlled trial across 19,000+ hospital employees found that simulated phishing training delivered via off-the-shelf products produced only a 1.7% aggregate reduction in click rates — and the median user spent zero to ten seconds on the training. At the same time, lure crafting alone could swing failure rates from 2% to 30%, meaning a single AI-optimized email can erase whatever modest gains training achieves.

Introduction

Phishing awareness training is one of the most universally mandated security practices in the enterprise: required for cyber-insurance, cited as industry best practice, and delivered annually to virtually every employee at organizations of any size. Yet the scientific evidence for its effectiveness is thin, most of it sourced from small lab studies or from vendors with an obvious interest in reporting positive outcomes. Christian DeMeff and Ariana Mirian set out to run the kind of rigorous randomized controlled trial (RCT) that medicine uses to evaluate treatments — and the results should force a serious reconsideration of where organizations invest their security budget.

The study was published at IEEE S&P 2025, making it one of the few large-scale, peer-reviewed evaluations of enterprise phishing training in the literature. The implications cut across security operations, compliance programs, and security-awareness vendors alike.

The Study Design: A Healthcare RCT at Scale

The researchers drew on UCSD Health's workforce of more than 19,000 employees — spanning three hospitals and covering clinicians, administrative staff, cafeteria workers, and sanitation staff. Participants were randomly assigned to one of five groups:

• Control: no training; users who clicked a simulated phish received a 404 page
• Generic Static: general phishing-avoidance tips on a non-interactive web page
• Generic Interactive: same advice with interactive elements (drag-and-drop exercises)
• Tailored Static: content customized to healthcare-specific scenarios, static format
• Tailored Interactive: healthcare-specific content with interactive elements

▶ Watch: Study Design Explained (08:00)

Over eight months, ten different phishing lures were deployed on a rolling basis. The team custom-instrumented the training platform's back end to capture time-on-page metrics — something the commercial vendor's product did not track by default.

Lesson 1: Lure Crafting Controls the Failure Rate

The most striking single finding was the enormous variance in failure rates across lures. Among the ten emails deployed, click rates ranged from 1.8% to 30% — using identically formatted, plain-text messages.

The low end was a classic password-reset lure of the type employees have seen repeatedly. The high end was a dress-code violation notice and a vacation-policy change announcement — workplace-relevant messages that trigger an emotional or self-interested response. The vacation-policy email alone had roughly a 30% failure rate.

▶ Watch: Lure Variance Data (12:00)

The takeaway the researchers drove home: whoever controls the lures controls the failure rate. Security teams that deploy a single lure type repeatedly are not measuring actual user susceptibility — they are measuring familiarity with that specific template. And in the era of large language models, the gap between a 2% lure and a 30% lure is approximately one prompt.

A cumulative failure-rate graph showed that by the eight-month mark, more than 50% of users had clicked at least one simulated phish, regardless of training group. Given enough time and creative lure selection, the majority of any workforce will fall.

Lesson 2: Training as Deployed Is Not Efficacious

The annual cybersecurity training module — typically a 35-to-40-minute web-based course — showed no discernible protective effect. The team plotted failure rates against time elapsed since each employee last completed their annual training and found a flat line: users who had completed their training that same week were no more resistant to phishing simulations than those who had not taken it in months.

▶ Watch: Annual Training Effectiveness Analysis (16:00)

For embedded phishing simulations — the just-in-time training served immediately after a user clicks a simulated lure — the aggregate improvement across all training modalities was 1.7%. That number held whether the training was static or interactive, generic or tailored.

Lesson 3: Users Spend Almost No Time on Training

The custom instrumentation revealed why training is not working: the median user spent between zero and ten seconds on the post-click training page. With engagement so low, the modality differences between static and interactive — a variable researchers had hypothesized might drive outcomes — barely mattered for the bulk of the population.

▶ Watch: Time-on-Training Data (18:01)

When the researchers filtered to the subset of users who spent more than 90 seconds on training, results became more nuanced. Users assigned to interactive modalities who actually engaged with the content did show improvement; curiously, users who spent time on static content showed a slightly negative effect. The researchers were candid that this subsample is small and the finding would require independent replication.

The Asymmetry Problem

DeMeff and Mirian framed the core issue as a fundamental asymmetry. An attacker needs only a single user to have a bad day and act on a well-crafted lure. Training defenders have to successfully inoculate every employee, every time. With LLMs capable of transforming a low-efficacy lure into a high-efficacy one in under a second, any marginal training benefit risks being erased instantly.

The researchers asked plainly whether the juice is worth the squeeze — and offered two alternative investment directions: deploying hardware FIDO2 keys (such as YubiKeys), which defeat phishing categorically regardless of user behavior; and AI-powered email filtering that can analyze incoming messages for phishing characteristics at the organizational level, removing the burden from individual users.

They also called explicitly for ending punitive responses to employees who fail phishing simulations — a practice that creates a culture of shame without reducing risk, since more than half of all users will eventually click given enough targeted attempts.

The Larger Call: Evidence-Based Security

The talk situated the phishing-training question inside a broader critique of security practice. Drawing on the evidence-based medicine framework — which requires randomized controlled trials before treatments reach patients — the presenters argued that most security practices, including phishing training, are deployed based on vendor claims and expert opinion rather than rigorous experimental evidence.

▶ Watch: Call to Evidence-Based Security (24:01)

Their call to action: security practitioners should demand data-backed evidence before adopting or renewing security programs; organizations should share anonymized outcome data across the field; and the community should move toward peer-reviewed, pre-registered experiments for evaluating defensive tools.

Notable Quotes

"Whoever controls the lures controls the failure rate." — Ariana Mirian 12:00

"It takes an LLM less than a second to take a two percent click-through rate lure and make it a thirty percent click-through and erase any benefit you would get from your training." — Christian DeMeff 26:01

"Give me a long enough timeframe and give me some creative endeavors, and I can get the majority of the users to click on a phishing lure." — Ariana Mirian 14:00

"Is the juice worth the squeeze, y'all? Is this where we should be putting our time?" — Christian DeMeff 22:01

Key Takeaways

• Lure diversity matters more than training content. Deploying a single repeated template understates user susceptibility by an order of magnitude; rotate lures across emotional registers and organizational concerns.
• Annual compliance training showed zero measurable protective effect in a real-world RCT; organizations should not rely on it as a meaningful defense.
• Embedded post-click training produced only a 1.7% aggregate improvement — largely because median engagement time was under ten seconds.
• Stop punishing employees for failing simulations. Reprimands demoralize staff without reducing risk and distort the organization's ability to measure its actual exposure.
• Redirect budget toward phishing-resistant controls — FIDO2 hardware keys and AI-powered email filtering — that do not depend on human behavior under pressure.

Slides: No slides PDF was available for this talk. The peer-reviewed paper was published at IEEE S&P 2025 and is available on Christian DeMeff's UCSD webpage.