DeepFakes: Seeing Is No Longer Believing

DeepFakes: Seeing Is No Longer Believing

Speakers: George Proorocu, IT OPS Manager - Cybersecurity & Fraud, ING Bank; Mihai Roman, Head of Security Engineering, ING Belgium

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=zqMddoHTqNM

Overview

In an era where digital interactions increasingly define our professional and personal lives, the rise of sophisticated deepfake technologies presents an unprecedented challenge to trust and security. George Proorocu and Mihai Roman, cybersecurity experts from ING Bank and ING Belgium, respectively, delivered a compelling DefCamp talk titled "DeepFakes: Seeing Is No Longer Believing," dissecting the escalating threat of AI-powered scams. The presentation served as a stark warning, illustrating how generative AI has transformed the landscape of fraud, making it easier, cheaper, and more effective for malicious actors to deceive individuals and infiltrate organizations.

The speakers opened with a chilling anecdote: a finance employee was duped into wiring an estimated $2.5 million after participating in a video call with what he believed were his CFO and high-level colleagues, all of whom were in fact deepfakes. This incident underscores the immediate and tangible financial risks posed by this technology. Proorocu and Roman meticulously traced the evolution of deepfakes from rudimentary manipulations to highly advanced, real-time impersonations, highlighting the dramatic increase in scam attempts and the projected global financial losses, which could reach $45 billion by the end of 2024. Their talk emphasized that deepfakes are not merely a futuristic concept but a present danger demanding immediate attention and robust defensive strategies.

Background

▶ Watch: Introduction and a $2.5M deepfake fraud example (0:00)

The proliferation of deepfake technology is intrinsically linked to the rapid advancements in generative AI over the past few years. The speakers provided a timeline illustrating this exponential growth, beginning in the latter half of 2020 when the concept of generative AI began to gain traction. At this stage, scams primarily relied on traditional methods like Photoshop for impersonation, with no significant AI involvement.

A pivotal shift occurred around 2022 with the public release of groundbreaking AI models such as ChatGPT and DALL-E. These tools ignited widespread enthusiasm for generative AI's potential, but also caught the attention of malicious actors. This period saw a significant increase in the number of scam attempts, with AI starting to be used in the preparatory phases of attacks. By 2023, the term "AI scams" entered the market, coinciding with the release of newer versions of ChatGPT and DALL-E models. Attackers, leveraging these enhanced capabilities, caused a tenfold increase in scam attempts. While the direct use of AI for deepfakes in successful attacks remained below 1% at this point, it was a clear indicator of the impending threat.

The landscape further transformed in 2024 with the emergence of multi-model generative AI, capable of generating not just text and images, but also highly convincing videos and audio based on simple prompts. The speakers noted that "the bad guys are the first ones to jump on the new technology," leading to a staggering 20 times more attempts compared to previous years. This rapid adoption by cybercriminals is driven by the fact that AI significantly lowers the barrier to entry for executing complex scams. What once required a team of 15-30 specialized individuals for intelligence gathering and execution can now be performed by a single individual or a small group, leveraging AI agents to automate reconnaissance, victim profiling, and even the creation of sophisticated deepfake content. This reduction in human effort and specialized skills makes large-scale, highly personalized attacks far more feasible and economically attractive for fraudsters.

Key Findings

▶ Watch: How AI agents drastically reduce scammer team size (10:00)

The talk unveiled several critical findings regarding the current state and future trajectory of deepfake-driven cybercrime:

1. Explosive Growth and Financial Impact: The most alarming finding is the unprecedented acceleration in deepfake technology adoption by attackers. From a tenfold increase in scam attempts in 2023 to a 20x increase in 2024 due to multi-model AI, the threat is escalating rapidly. The projected global financial losses are staggering, estimated to reach $45 billion by the end of 2024, demonstrating the immense economic toll of these sophisticated frauds.
2. Automation of the Entire Attack Chain: Generative AI, particularly through the use of AI agents and multi-agent systems, has enabled the near-complete automation of complex scam campaigns. These agents can autonomously gather information from social media (e.g., Facebook, LinkedIn), newspapers, and other public sources, profile targets, generate convincing phishing emails (e.g., using ChatGPT with Python instructions), and even make decisions on victim engagement. This drastically reduces the human resources required for an attack, making sophisticated fraud accessible to more actors.
3. Sophistication of Deepfake Content: The quality and realism of both video and audio deepfakes have reached a point where detection is exceedingly difficult for the human eye and ear. The speakers demonstrated how easily high-quality video deepfakes can be generated in mere seconds, even with basic tools, and how convincing audio deepfakes can be created from as little as 7-10 seconds of a person's voice. Crucially, these deepfakes can be integrated into live video and audio calls, enabling real-time impersonation during critical interactions like job interviews or urgent corporate requests.
4. Targeting Corporate Networks and MFA: Deepfakes are no longer limited to simple financial scams. Attackers are now leveraging them to target corporate environments, specifically aiming for network credentials and Multi-Factor Authentication (MFA) access. By impersonating high-level executives or managers, fraudsters can manipulate employees into divulging sensitive information or performing actions that compromise organizational security, as exemplified by the $2.5 million CFO scam.
5. The "Cat and Mouse" Game of Detection: While solutions exist for detecting AI-generated content, the technology for generating deepfakes is advancing so rapidly that detection algorithms struggle to keep pace. As new patterns are identified for detection, deepfake generation algorithms are quickly updated to address and circumvent these patterns, creating an ongoing cat and mouse game where perfect programmatic detection remains elusive.

Technical Deep Dive

▶ Watch: Live demonstration of an easy-to-use video deepfake tool (14:00)

The speakers outlined a detailed, end-to-end scam scenario, showcasing the technical sophistication and automation enabled by modern generative AI. The process begins with information gathering and audience definition. Attackers leverage social media platforms, online newspapers, and various public portals to collect extensive data on potential targets, allowing them to define specific high-value individuals or groups, such as corporate engineers.

Once targets are identified, the initial outreach is highly automated. Using tools like ChatGPT and custom Python scripts, scammers can generate highly legitimate-looking email templates, meticulously crafted to avoid grammatical errors and appear authentic. These emails are designed to hook potential victims, with job offers being a common lure. Feedback from initial email interactions is then analyzed, often by AI agents, to identify "high potential" victims who react positively or show interest.

From this point, the scam can bifurcate into two primary paths:

1. Elaborated Deepfake Interaction: This path involves direct, personalized engagement. Attackers schedule live video calls and audio calls, where they deploy sophisticated video deepfakes and audio deepfakes to impersonate individuals. For video, this might involve face swapping or manipulating facial expressions and head movements to appear natural. The speakers noted that even complex head rotations, once challenging, are now "almost perfect." For audio, voice cloning technology can replicate a target's voice from minimal samples (as little as 7-10 seconds), allowing for convincing conversations. This path is resource-intensive but yields high success rates due to the perceived authenticity.
2. Automated Multi-Agent Systems: This more advanced approach leverages a network of AI agents to manage the entire scam lifecycle. Instead of a human team, specialized agents are deployed: one for scrolling Facebook, another for LinkedIn, a third for analyzing news articles. These agents feed information into a central system. Critically, multi-modal agents can process diverse data types and even make autonomous decisions, such as determining if a victim is "high potential" without human intervention. This setup drastically reduces the human footprint required for an attack, potentially down to a single operator, or even a fully autonomous system in the future. While the hallucination tendency of AI remains a pragmatic concern, requiring "double, triple check" mechanisms, attackers are continuously refining these systems.

A specific example of corporate infiltration involved a multi-stage "job interview" scam. Victims undergo a series of fake interviews – with HR, technical teams, and even managers – designed to build rapport and trust. Throughout these interactions, scammers gather intelligence and subtly push the victim towards actions beneficial to the scam. The ultimate goal in such scenarios is often to obtain corporate network passwords and MFA access, which are the final keys to breaching an organization's defenses. The integration of deepfakes into these live, interactive stages makes the deception incredibly difficult to detect, as victims believe they are interacting with genuine company representatives.

Demo / Proof of Concept

▶ Watch: Essential strategies to protect against deepfake scams (20:00)

The speakers provided compelling, live demonstrations to illustrate the ease and quality of modern deepfake generation.

The first demonstration involved a video deepfake. Speaker George Proorocu recorded a short video of himself speaking. He then used an open-source tool (unnamed, but accessible for public use) to perform a face swap, superimposing Mihai Roman's face onto his own in the recorded video. The process was remarkably fast, taking approximately 20 seconds to generate the deepfake. Even without high-end hardware like a dedicated GPU, the output quality was impressive, showing how easily convincing video impersonations can be created. Proorocu highlighted that such tools can be integrated into live calls, though for security reasons, they did not demonstrate this real-time capability. The demo served to underscore the rapid advancement in video deepfake technology, noting that capabilities like realistic head rotation, which were complicated just a few years ago (e.g., in 2022), are now almost perfect and difficult to distinguish from genuine footage.

The second demonstration showcased an audio deepfake, focusing on its use in a corporate infiltration scenario targeting an engineer named "Mihai." The speakers explained that audio deepfakes have become particularly popular, especially in "accident scams" in the US, due to their ease of creation. They emphasized that a high-quality voice model can be built from as little as 7-10 seconds of a person's audio. The demo played an original voice sample, followed by a cloned voice (referred to as "Olivia's" voice), attempting to convince "Mihai" to provide MFA access and a password. The quality of the cloned voice was highly convincing, demonstrating how effectively it could be used to manipulate a victim. The speakers further elaborated on how AI agents could automate the entire process of gathering voice samples (from podcasts, conferences, social media, corporate content), building the voice model, and even generating a script for the scammer to use during a call, drastically streamlining the attack.

Defensive Implications

▶ Watch: Q&A: Challenges in deepfake detection and digital onboarding security (24:30)

Protecting against the sophisticated and rapidly evolving threat of deepfakes requires a multi-layered approach that combines technological solutions with human vigilance and robust organizational policies. The speakers outlined several critical defensive implications for individuals and corporations:

1. Enhanced Awareness and Education: The most fundamental defense is awareness. Through conferences, internal training, and public outreach, individuals must be educated about what deepfakes are, how they work, and the various forms they can take (video, audio, text). This knowledge empowers people to recognize potential scams and react appropriately. The goal is to instill a "double thoughts" mindset when encountering suspicious requests or communications.
2. Attention to Detail and Personal Questions: Encourage critical thinking and verification. If something feels "fishy" during a call or video conference, individuals should pay close attention to subtle details that might indicate a deepfake (though increasingly difficult). A crucial tactic is to ask personal questions that are not publicly available on social media or corporate profiles. As demonstrated by a famous case involving a CEO deepfake, an executive's simple personal question can expose the fraud and stop the scam.
3. Multi-Person Authorization for High-Risk Actions: For critical operations, especially those involving significant financial transactions like large bank transfers, no single person should have unilateral authority. Most large corporations already implement such segregation of duties, but small and medium-sized businesses (SMBs) often lack these controls. It is imperative for SMBs to establish and enforce policies requiring multiple approvals or verification steps for high-risk actions to prevent deepfake-induced financial fraud.
4. Strict Separation of Business and Private Devices: Attackers often target employees by tricking them into installing malicious software on their corporate devices, often disguised as part of a job interview or recruitment process. Maintaining a clear distinction between business and private devices and avoiding the use of personal devices for corporate activities (especially during interviews or when handling sensitive information) can significantly reduce the risk of corporate network infiltration.
5. Hybrid Human-AI Decision-Making for Onboarding and Verification: For critical processes like digital onboarding, relying solely on automated AI systems for identity verification is risky. While AI can assist in initial checks, the speakers advocated for a hybrid approach that incorporates human expertise. As per the European Union AI Act, automated onboarding is classified as high-risk AI, requiring that the final decision be human-based. Specialized human agents trained in detecting fake passports and IDs, working in conjunction with AI tools, provide a more robust defense against deepfake-based identity fraud.
6. Continuous Vigilance Against Evolving Detection Evasion: The "cat and mouse" nature of deepfake generation and detection means that static detection solutions are insufficient. Organizations must remain vigilant, investing in and continually updating their deepfake detection capabilities, while acknowledging that perfect programmatic detection is an ongoing challenge.

Key Takeaways

• Deepfakes are an escalating, real-world threat: Driven by advancements in generative AI, deepfakes are rapidly increasing in sophistication and prevalence, posing significant financial and security risks to individuals and organizations.
• AI automates and scales scams: Multi-agent AI systems can automate entire scam campaigns, from reconnaissance and target profiling to content generation and execution, drastically reducing the human effort required for complex attacks.
• Financial losses are immense: The global financial impact of deepfake-driven scams is projected to reach $45 billion by the end of 2024, highlighting the urgent need for proactive defense.
• Awareness is the primary defense: Educating individuals about deepfakes, how they operate, and the importance of critical thinking and verification is the most effective immediate countermeasure.
• Procedural controls are crucial: Implementing multi-person authorization for high-risk actions, separating business and private devices, and adopting hybrid human-AI decision-making processes are vital for organizational resilience.
• Detection remains a challenge: While tools exist, the rapid evolution of deepfake generation technology means that detection is a continuous "cat and mouse" game, requiring ongoing vigilance and adaptation.

About the Speaker(s)

George Proorocu serves as the IT OPS Manager for Cybersecurity & Fraud at ING Bank. His role involves overseeing operational security and fraud prevention strategies, providing him with firsthand insight into the evolving tactics of cybercriminals and the impact of emerging technologies like deepfakes on financial security.

Mihai Roman is the Head of Security Engineering at ING Belgium. His expertise lies in designing and implementing robust security architectures and engineering solutions. His work focuses on building resilient systems that can withstand advanced cyber threats, including those leveraging sophisticated AI and deepfake technologies. Together, their combined experience offers a comprehensive perspective on both the strategic and technical aspects of combating modern cyber fraud.