Machine Learning and the Optimization of Virtual Personae for Phishing Scams
Will Pearce (Machine Learning Security Researcher · Nvidia)
Swiss Cyber Storm 2022 · Day 1 · Scenario
Overview
In an era where digital identity underpins nearly every interaction, the proliferation of sophisticated machine learning (ML) techniques presents a formidable challenge to traditional security paradigms, particularly in the realm of phishing and social engineering. Will Pearce, a Machine Learning Security Researcher at Nvidia, delivered a compelling talk at Swiss Cyber Storm, dissecting how advanced ML models are being weaponized to optimize the creation of virtual personae for highly effective and scalable phishing scams. The presentation illuminated a critical shift in the threat landscape: from manual, template-based attacks to infinitely customizable, synthetically generated identities and content that defy conventional detection methods.

Key moments
- 0:00 Introduction and ML security background
- 4:00 Overview of the machine learning life cycle
- 8:30 Representing identity numerically for ML systems
- 16:00 Crafting unique phishing emails with GPT-3
- 17:20 Generating synthetic human personas with StyleGAN
- 21:40 Live demonstration of an audio deepfake
- 24:40 Conclusion: Identity as a composite distribution
Machine Learning and the Optimization of Virtual Personae for Phishing Scams
Speakers: Will Pearce, Machine Learning Security Researcher, Nvidia
Conference: Swiss Cyber Storm
YouTube: https://www.youtube.com/watch?v=fS02kmkwdfk
Overview
In an era where digital identity underpins nearly every interaction, the proliferation of sophisticated machine learning (ML) techniques presents a formidable challenge to traditional security paradigms, particularly in the realm of phishing and social engineering. Will Pearce, a Machine Learning Security Researcher at Nvidia, delivered a compelling talk at Swiss Cyber Storm, dissecting how advanced ML models are being weaponized to optimize the creation of virtual personae for highly effective and scalable phishing scams. The presentation illuminated a critical shift in the threat landscape: from manual, template-based attacks to infinitely customizable, synthetically generated identities and content that defy conventional detection methods.
Pearce's talk goes beyond merely showcasing deepfake technology; it delves into the fundamental principles of how ML models represent and manipulate data, illustrating how this numerical abstraction of identity can be exploited. He argues that the ability of algorithms to generate unique text, photorealistic faces, and convincing synthetic voices allows attackers to craft virtually untraceable and highly targeted phishing campaigns at an unprecedented scale. This evolution necessitates a re-evaluation of how organizations perceive and defend against identity-based attacks, emphasizing the critical need for robust, multi-modal defenses that account for the composite nature of identity in the digital age.
The implications of this research are profound, signaling a future where the line between real and artificial identities becomes increasingly blurred. As ML models become more accessible and powerful, the barrier to entry for creating advanced social engineering attacks diminishes, placing greater pressure on security professionals to develop innovative detection and prevention strategies. Pearce’s insights serve as a stark warning and a call to action for the security community to proactively address the inherent risks introduced by machine learning in production environments.
Background
▶ Watch: Introduction and ML security background (0:00)
To understand the sophisticated threats posed by machine learning in phishing, it's crucial to grasp the underlying principles of the ML lifecycle and how data is processed. Pearce meticulously walked through the six core stages: data collection, data processing, model training, model evaluation, model deployment, and system monitoring. This lifecycle, while designed for beneficial applications, inherently creates new attack surfaces and opportunities for malicious actors.
At the heart of ML is the transformation of raw, real-world data—be it images, text, or audio—into a numerical representation. As Pearce explained, "algorithms are math; they can't work on words or just straight images." This conversion process, often involving techniques like facial point maps, bounding boxes for image data, or algorithms such as Word2Vec and Doc2Vec for text, turns complex information into vectors of numbers. This numerical identity is then fed into models during training, where parameters are adjusted through processes like backpropagation to accurately classify or generate new data.
Once a model is trained and evaluated, it's deployed into various environments—from web endpoints and APIs to edge devices. A critical aspect at this stage is the introduction of business logic and thresholds, which determine whether a prediction (e.g., "this is Will Pearce," "this is malware") meets a certain confidence level. This introduces a probabilistic element to systems that were once more deterministic, complicating traditional security assessments.
Pearce provided concrete examples of how this numerical identity manifests in real-world systems. For instance, Windows Hello for Business uses infrared cameras to capture a user's face, converting it into a numerical representation stored in a .dat file within the C:\Windows\System32\WinBioDatabase. When a user attempts to log in, their current facial scan is similarly processed, and the system calculates the distance between the live scan and the stored numerical identity. If this distance falls within an acceptable error threshold, authentication is granted. Similarly, in an Active Directory environment, user attributes like usernames and group memberships can be transformed into numerical vectors, allowing for projection into multi-dimensional spaces where the proximity of users (e.g., to a "domain admin" cluster) can be measured algorithmically, as demonstrated by Pearce using a k-nearest neighbors projection. This fundamental concept—that any form of identity can be abstracted into numbers—forms the bedrock for both legitimate ML applications and their malicious exploitation.
Key Findings
▶ Watch: Representing identity numerically for ML systems (8:30)
The central revelation of Will Pearce's talk is the unprecedented capability of modern machine learning models to generate highly convincing, unique, and scalable synthetic personae for phishing and social engineering campaigns. This represents a significant leap from traditional, manually crafted attacks, offering attackers a virtually limitless supply of untraceable identities and content.
Pearce highlighted two primary tools enabling this shift:
- GPT-3 (Generative Pre-trained Transformer 3): This powerful language model can generate virtually infinite, unique textual content. For phishing, this means attackers are no longer limited to a finite set of templates or subject lines that can be easily detected. Instead, GPT-3 can produce bespoke job descriptions, email bodies, and other persuasive text that is contextually relevant and tailored to specific targets, making each phishing attempt distinct. This capability bypasses detection mechanisms that rely on identifying repetitive patterns or known phishing templates, significantly increasing the success rate and evasion capabilities of campaigns. Pearce demonstrated this by asking GPT-3 to generate a job description for a sales executive making $400k a year, showcasing its ability to produce highly plausible, unique content. He even humorously asked it to generate a job description for a cat banking executive, which it did with surprising detail, including "a lot of tuna" as a perk.
- StyleGAN3 (Style-based Generative Adversarial Network 3): This generative model excels at creating photorealistic images of human faces that do not correspond to any real person. Platforms like "thispersondoesnotexist.com" exemplify this technology. Attackers can now generate an endless supply of unique, untraceable profile pictures for social media platforms like LinkedIn, entirely eliminating the risk of reverse image searches or the need to scour obscure corners of the internet for images. This allows for the creation of hundreds, if not thousands, of completely synthetic online identities, each with a unique face, bolstering the credibility of phishing personae without leaving a digital footprint.
Beyond static images and text, Pearce also emphasized the growing threat of deepfakes, particularly in the audio domain. While visual deepfakes (like the Tom Cruise examples) are becoming incredibly sophisticated, Pearce argued that audio deepfakes are arguably more dangerous in a social engineering context due to lower user expectations for audio quality in phone calls. The inherent noise and signal degradation in phone communications can mask imperfections in synthetic voices, making them highly compelling for targeted attacks such as CEO fraud or impersonation.
In essence, the key finding is that these ML advancements provide attackers with an automated, scalable, and evasive toolkit for crafting highly personalized and credible phishing campaigns, fundamentally altering the economics and efficacy of social engineering. The ability to generate unique content and untraceable identities at will drastically lowers the operational overhead for attackers while simultaneously raising the bar for defenders.
Technical Deep Dive
▶ Watch: Crafting unique phishing emails with GPT-3 (16:00)
The technical core of Pearce's presentation lies in understanding how machine learning models abstract and manipulate identity, transforming it into a malleable, numerical construct that can be both analyzed and synthesized. This abstraction is the bedrock upon which sophisticated phishing personae are built.
As discussed, the ML lifecycle begins with raw data, which is then processed into numerical representations. For facial recognition, this involves techniques like facial landmark detection and bounding boxes to isolate and map key features, converting an image into a high-dimensional vector of numbers. Similarly, text processing utilizes algorithms like Word2Vec or Doc2Vec to embed words or entire documents into a numerical space, capturing semantic relationships. Pearce demonstrated this with an Active Directory example, where usernames and group memberships were transformed into vectors and projected onto a 2D plane using a k-nearest neighbors algorithm. This visualization clearly showed how algorithms inherently group similar identities, revealing potential pathways for lateral movement or privilege escalation based on numerical proximity.
The critical insight here is that once an identity (whether a face, a voice, or a network user profile) is represented numerically, it can be manipulated or synthesized. This is where generative adversarial networks (GANs) and large language models like GPT-3 come into play.
GPT-3 operates on the principle of predicting the next most probable word in a sequence, trained on vast datasets of text. This allows it to generate coherent, contextually relevant, and grammatically correct prose. For phishing, this means an attacker can programmatically request unique email subject lines, body paragraphs, or even entire fictitious job descriptions that are indistinguishable from human-written content. This capability directly addresses a long-standing challenge in phishing: the need for unique content to bypass signature-based detection and avoid being flagged as mass spam. By generating an infinite variety of plausible narratives, GPT-3 negates the effectiveness of simple keyword or template matching in security tools.
StyleGAN3, a prominent example of a GAN, consists of two neural networks: a generator and a discriminator. The generator creates synthetic images, while the discriminator attempts to distinguish between real and generated images. Through a competitive process, the generator learns to produce increasingly realistic images that can fool the discriminator. StyleGAN3's advanced architecture allows for fine-grained control over various stylistic features (e.g., hair color, age, facial expression), leading to incredibly high-fidelity, photorealistic faces. These faces can be used as profile pictures for fake social media accounts, adding a layer of authenticity to phishing personae that is difficult to debunk through reverse image search or visual inspection alone, as these individuals literally "do not exist."
The synthesis of deepfake audio relies on similar generative principles, often using techniques like voice cloning or text-to-speech (TTS) models trained on extensive datasets of human speech. These models learn the unique vocal characteristics (pitch, timbre, cadence, accent) of a target individual and can then generate new speech in that person's voice from arbitrary text. Pearce demonstrated this with a synthetic voice of Nvidia's CEO, highlighting its remarkable realism. He argued that audio deepfakes are particularly potent in phone-based social engineering because the inherent low fidelity and background noise of phone calls can mask subtle imperfections that might be noticeable in a high-quality video deepfake. This makes them highly effective for impersonating authority figures (e.g., a manager or even a CEO) to induce actions like transferring funds or divulging sensitive information. The probabilistic nature of ML models means that while a deepfake might not be "perfect," it can be "good enough" to bypass human scrutiny in a real-world, time-pressured scenario.
Pearce's emphasis on identity as a "composite of things" underscores the technical challenge for both attackers and defenders. For attackers, the goal is to create the "right representation" for their target—a blend of visual, textual, and auditory cues that form a convincing, albeit synthetic, identity. For defenders, this means moving beyond single-factor authentication or simple content analysis and embracing a holistic view of identity that integrates multiple data points and behavioral analytics to detect anomalies.
Demo / Proof of Concept
▶ Watch: Live demonstration of an audio deepfake (21:40)
Will Pearce provided a compelling real-time demonstration of an audio deepfake, showcasing the alarming fidelity achievable with current machine learning technology. The demo involved a synthetic voice model of Nvidia's CEO.
Pearce played an audio clip where the AI-generated voice of the CEO stated, "Hey everyone, Nvidia is happy to be at Swiss Cyber Storm." The quality was remarkably high, rendering it virtually indistinguishable from a genuine recording of a human voice. He further illustrated the capability by having the synthetic voice say, "Will is my favorite employee," demonstrating the ability to generate arbitrary phrases.
Pearce highlighted several key aspects of this demonstration:
- Realism: The synthetic voice exhibited natural intonation, rhythm, and clarity, making it highly convincing. He noted that Nvidia itself uses synthetic media in its GTC conferences, even having AI write and play outro music, further cementing the advanced state of this technology.
- Contextual Impact: Pearce argued that while visual deepfakes might be scrutinized more heavily due to the involvement of multiple senses (sight and sound), audio deepfakes are particularly potent for social engineering over the phone. He explained that "we kind of expect like rough phone calls or it to be like a noisier signal," meaning imperfections in synthetic audio are more likely to be attributed to poor connection quality rather than artificial generation. This lower expectation for fidelity makes audio deepfakes a "much more compelling" threat than their visual counterparts in many scenarios.
- Attack Scenarios: He specifically mentioned the common trope of a CEO calling for gift cards, acknowledging it might be "overdone." However, he stressed that the real danger lies in impersonating a direct manager or a peer, where a voicemail or a direct call from a seemingly legitimate voice asking for a quick action could be far more successful and less suspicious. The demo effectively illustrated that the technology is "not perfect, but it's good enough" to fool human perception in the right context.
The demonstration served as a powerful proof of concept for the immediate and practical threat posed by synthetic audio, underscoring how readily this technology can be leveraged to establish trust and execute sophisticated social engineering attacks with a high probability of success.
Defensive Implications
▶ Watch: Conclusion: Identity as a composite distribution (24:40)
The rise of machine learning-optimized virtual personae and deepfakes necessitates a profound re-evaluation of defensive strategies. Will Pearce outlined several critical areas where security professionals must adapt to counter these evolving threats.
Firstly, there is a clear need for new protocols and a greater sense of responsibility for media companies. As the creators and disseminators of digital content, these entities play a crucial role in authenticating the origin and integrity of media. While Pearce cautiously mentioned blockchain as a potential (though perhaps "dirty word") solution for content provenance, the underlying principle is to establish verifiable chains of trust for digital assets. This could involve embedding cryptographic signatures or watermarks that attest to the authenticity of images, videos, or audio.
From a technical detection standpoint, defenders must invest in advanced liveness detection and texture analysis for visual deepfakes. Liveness detection aims to determine if a presented biometric (e.g., a face during authentication) is from a live person rather than a static image, video, or 3D mask. Texture analysis can identify subtle, unnatural patterns or artifacts inherent in synthetically generated images that human eyes might miss. However, Pearce acknowledged that the "visual space is a much harder space to create deepfakes in" compared to audio, implying that visual detection might evolve faster, but the arms race will continue.
For audio deepfakes, the challenge is arguably greater due to the inherent noise and signal degradation in phone communications. While specific detection methods weren't detailed, the implication is that traditional audio forensics will need to evolve, possibly incorporating ML models trained to detect specific synthetic audio artifacts. However, the speaker's point about lower user expectations for phone call quality means that even imperfect deepfakes can be effective, making detection particularly difficult.
Pearce also highlighted the importance of robust system-level security. He mentioned Windows Hello for Business receiving Active Directory integrations, allowing face-based authentication directly against enterprise directories. While this enhances user convenience, it also means the integrity of the biometric data and the comparison process becomes even more critical. Event monitoring of such authentication systems is vital to detect anomalies or repeated failed attempts that might indicate deepfake-based attacks.
Crucially, Pearce concluded with a fundamental shift in mindset: defenders must view identity not as a single, static entity but as a composite or distribution of features. This means moving away from relying on a single authentication factor or a singular representation of a user. Instead, security systems should analyze a multitude of data points—behavioral patterns, network activity, geographic location, device characteristics, and the context of communication—to build a more holistic and dynamic understanding of identity. If a deepfake voice is heard from an unusual location, at an odd time, or requesting an out-of-policy action, these composite anomalies should trigger higher scrutiny.
In essence, defensive strategies against ML-powered phishing personae require a multi-layered approach: technological advancements in detection, industry-wide protocols for media provenance, continuous monitoring of authentication systems, and a fundamental shift in how organizations conceptualize and protect digital identities.
Key Takeaways
- ML Enables Scalable, Unique Phishing Personae: Advanced models like GPT-3 and StyleGAN3 allow attackers to generate an infinite supply of unique textual content and untraceable, photorealistic faces, bypassing traditional signature-based detection and greatly enhancing phishing credibility and scale.
- Deepfake Audio Poses Significant and Immediate Threat: Due to lower user expectations for audio fidelity in phone calls, synthetic voices are highly effective for impersonation, facilitating CEO fraud and other social engineering attacks that exploit trust.
- Digital Identity is a Numerical Composite: Any aspect of identity (face, voice, text, network attributes) can be represented as numbers, making it susceptible to algorithmic analysis, manipulation, and synthesis. Defenders must understand identity as a multi-faceted distribution rather than a single attribute.
- Traditional Defenses Are Insufficient: The ease of generating unique, untraceable content and identities renders static templates, simple keyword matching, and single-factor authentication largely ineffective against ML-powered attacks.
- Multi-Modal Detection and New Protocols Are Essential: Countermeasures require advanced techniques like liveness detection and texture analysis for visuals, alongside evolving audio forensics and potentially new protocols for media provenance and authentication.
- Adopt a Holistic View of Identity for Defense: Security systems must analyze a composite of behavioral, contextual, and multi-modal data points to detect anomalies and verify identity, moving beyond reliance on isolated features.
About the Speaker(s)
Will Pearce is a distinguished Machine Learning Security Researcher currently working at Nvidia. In this role, he specializes in attacking machine learning models, both from a system perspective (classic red teaming, including developing code execution techniques for ML libraries) and algorithmically (evasion, inversion, inference poisoning). Prior to joining Nvidia, Pearce served as the Azure Red Team Lead for the Trustworthy Machine Learning team at Microsoft, where he was instrumental in building methodologies for securing ML systems. His professional background also includes experience as a Senior Security Consultant at Silent Bridge Security, where he focused on red teaming and taught advanced malware development in courses like Dark Side Ops. Pearce is a passionate advocate for ML security, driven by the belief that any new technology deployed into production environments must inherently be secure and not weaken existing defenses. He is also actively involved in the AI Village at Defcon, contributing to initiatives like deepfake detection CTFs.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Will Pearce is a real operator, and it shows when he digs into Windows Hello .dat files and Active Directory KNN projections. But the core premise here is that GPT-3 and StyleGAN will unleash a tsunami of synthetic phishing, which is a threat model the industry has beaten to death. It's a well-delivered briefing, but the ML phishing apocalypse is overclaimed.
Heather Calloway (CISO) — STRONG ACCEPT
Pearce proves that generative AI has permanently broken template-based phishing detection and static identity verification. This talk forces a hard look at enterprise authentication and how we train employees to verify authority. It's required viewing for any security leadership team still relying on traditional email filtering and voice verification to protect the business.