Inside M(AI)ware: Behavior, Analysis, and Detection

Fatou Sankare

CypherCon 9.0 (2026)

Overview

In "Inside M(AI)ware: Behavior, Analysis, and Detection," Dr. Fatou Sankare delivers a compelling exploration into the evolving landscape of malware, emphasizing the critical shift from signature-based detection to a deeper understanding of malware behavior and intent. The talk highlights how artificial intelligence (AI) is simultaneously transforming both offensive and defensive cybersecurity strategies, presenting both unprecedented opportunities for enhanced detection and formidable challenges in the form of AI-assisted evasion techniques, particularly with polymorphic malware.

Watch on YouTube

Visual summary for Inside M(AI)ware: Behavior, Analysis, and Detection by Fatou Sankare
Visual summary for Inside M(AI)ware: Behavior, Analysis, and Detection by Fatou Sankare

Key moments

  1. 2:00 Speaker introduction and talk overview
  2. 3:42 Understanding core malware behaviors: downloaders, backdoors, credential stealers
  3. 7:50 Malware persistence mechanisms and privilege escalation techniques
  4. 9:59 Static and dynamic malware analysis methods demonstrated
  5. 16:00 The role of AI in accelerating malware analysis and detection
  6. 18:33 Lab demo: Using AI to create polymorphic malware for evasion
  7. 23:00 Limitations of AI and the enduring importance of human analysis skills
  8. 30:50 Q&A: Latest evasive techniques and effectively interacting with AI tools

Inside M(AI)ware: Behavior, Analysis, and Detection

Speakers: Fatou Sankare, Cybersecurity Researcher and Offensive Security Engineer

Conference: CypherCon

YouTube: https://www.youtube.com/watch?v=uyObtFJKl8A

Overview

In "Inside M(AI)ware: Behavior, Analysis, and Detection," Dr. Fatou Sankare delivers a compelling exploration into the evolving landscape of malware, emphasizing the critical shift from signature-based detection to a deeper understanding of malware behavior and intent. The talk highlights how artificial intelligence (AI) is simultaneously transforming both offensive and defensive cybersecurity strategies, presenting both unprecedented opportunities for enhanced detection and formidable challenges in the form of AI-assisted evasion techniques, particularly with polymorphic malware.

Dr. Sankare, a distinguished cybersecurity researcher, offensive security engineer, and professor, underscores the necessity for security professionals to grasp the fundamental behaviors of malicious software. Her presentation provides a comprehensive guide to analyzing malware through both static and dynamic methods, integrating the burgeoning role of AI in accelerating these processes. The core message is clear: while AI offers powerful tools for scale and pattern recognition, the nuanced analysis and validation of threats remain an inherently human skill, crucial for distinguishing legitimate system processes from sophisticated attacks.

The significance of this talk lies in its timely address of AI's impact on malware. As AI tools become more accessible, attackers are leveraging them to craft more elusive and adaptive malware, exemplified by polymorphic variants that constantly alter their code to bypass traditional defenses. Dr. Sankare's research into these advanced evasion tactics, specifically targeting products like Windows Defender, provides invaluable insights for defenders seeking to bolster their resilience against future threats. This article will delve into the technical intricacies presented, offering a structured breakdown of malware behaviors, analysis techniques, AI's dual role, and practical defensive implications.

Background

▶ Watch: Speaker introduction and talk overview (2:00)

Understanding malware begins with recognizing its fundamental behaviors and the mechanisms it employs to achieve its malicious objectives. Malware is not merely "bad code"; it's a sophisticated program designed with specific intentions once it infects a system. Dr. Sankare categorizes these core behaviors into several types, each representing a distinct phase or goal in an attack chain.

Downloaders are foundational components, responsible for fetching additional malicious payloads from remote servers onto a local system. They often leverage legitimate Windows APIs such as URLDownloadToFileA to download files and then invoke the Windows executive function to execute them. While modern antivirus solutions like Windows Defender maintain frequently updated signature databases to block known downloader hashes, attackers continuously seek methods to evade these detections. Launchers or loaders are closely related, acting as executables that install malware for immediate or future execution, often designed to ensure the primary payload is properly set up on the target system.

Backdoors are critical for maintaining unauthorized access. This type of malware provides attackers with persistent remote access, often implementing a suite of capabilities without needing to download further tools. These capabilities can include manipulating registry keys, enumerating display windows, creating directories, and searching for files, granting the attacker a high degree of control over the compromised machine. Complementing backdoors are reverse shells, which are highly coveted by attackers. A reverse shell establishes an outbound connection from the infected machine back to an attacker-controlled server, providing command-line access. Common methods for establishing reverse shells include Netcat, dedicated Windows reverse shell executables, multi-threaded coding implementations utilizing functions like CreateProcess and CreatePipe, and frameworks like Metasploit.

Finally, credential stealers target the sensitive information that unlocks further access. Attackers often prioritize stealing credentials because human tendencies, such as reusing passwords, can lead to lateral movement across multiple systems. Techniques include hash dumping, where Windows hashes (e.g., NTLM hashes) are extracted from memory or the Security Account Manager (SAM) database, allowing attackers to crack them offline or use them in Pass-the-Hash attacks. Keystroke logging is another straightforward method, capturing user input as it's typed. This can be implemented through user-space keyloggers (software-based) or more stealthy kernel-based keyloggers, or even physical devices like USB keyloggers (e.g., a "rubber ducky") discreetly inserted between a keyboard and the computer.

Beyond initial compromise, malware seeks persistence to ensure continued access even after system reboots. Dr. Sankare outlines several common persistence mechanisms. Malware can install itself as a Windows service, often masquerading as legitimate processes like svchost DLLs (Service Host DLL), which is frequently used by Windows and thus less likely to raise suspicion in task managers. Another method involves Trojanized system binaries, where malware patches legitimate system executables, forcing them to execute the malicious code upon the next system restart. Task Scheduler is also exploited to schedule the malware to run at regular intervals, ensuring its continuous presence.

Finally, most initial compromises grant only local user privileges. To achieve full control, malware often employs privilege escalation techniques. This involves exploiting vulnerabilities or misconfigurations to elevate privileges to administrator or system level, allowing the attacker to perform more extensive actions on the compromised machine and further solidify persistence. The existence of these diverse behaviors and persistence tactics underscores the sophisticated nature of modern malware and the constant arms race between attackers and defenders.

Key Findings

▶ Watch: Malware persistence mechanisms and privilege escalation techniques (7:50)

Dr. Sankare's presentation highlights several pivotal findings regarding the contemporary state of malware and its analysis, particularly in the context of AI. A central theme is the inadequacy of solely relying on traditional, signature-based detection methods. While antivirus products like Windows Defender are constantly updated, their effectiveness against rapidly evolving and polymorphic malware is diminishing. The core finding here is that polymorphic malware, which alters its code and signature with each execution to evade detection, represents a significant leap in adversarial capabilities. This necessitates a shift towards behavioral analysis, where the focus is on what the malware does rather than just what it looks like.

Another key finding is the dual-edged nature of AI in cybersecurity. For defenders, AI offers immense potential to accelerate analysis, detect complex patterns that human analysts might miss, and manage vast quantities of data (e.g., logs) at scale. Tools leveraging AI can enhance anomaly detection, automate triage, and enrich threat intelligence. However, AI is equally accessible to attackers. Dr. Sankare demonstrates how AI can be leveraged to generate highly evasive malware, modify code to bypass security products, and create sophisticated self-modifying payloads. This means AI is not a silver bullet for defense; it's a new battleground where both sides are leveraging advanced capabilities.

Crucially, the talk emphasizes that despite the advancements in AI, human validation and expertise remain indispensable. AI tools, while powerful, are prone to false positives and often lack the nuanced environmental context required for accurate threat assessment. An AI might flag a process as malicious, but only a human analyst can determine if it's a legitimate, albeit unusual, activity within a specific network configuration. Furthermore, AI can be manipulated, and an over-reliance on it can lead to a degradation of critical human skills. The ability to ask "does this belong?" and to understand the intent behind observed behaviors is a uniquely human skill that AI cannot fully replicate.

In summary, the key findings underscore that malware is becoming more adaptive and elusive thanks to AI. While AI provides powerful tools for defenders, it also empowers attackers to create sophisticated evasion techniques, particularly with polymorphic malware. This necessitates a strong focus on behavioral analysis, continuous observation, and, most importantly, the unwavering application of human intelligence and critical thinking in conjunction with AI tools.

Technical Deep Dive

▶ Watch: The role of AI in accelerating malware analysis and detection (16:00)

The technical deep dive into malware behavior, analysis, and AI's role forms the core of Dr. Sankare's presentation. She meticulously breaks down the intricacies, from the fundamental actions of malware to advanced detection and evasion techniques.

Malware Behaviors in Detail:

  • Downloaders and Launchers: These initial stages often exploit common Windows APIs. Downloaders might use URLDownloadToFileA from urlmon.dll to fetch files from a URL and then execute them using standard process creation functions. Launchers ensure the primary malware payload is properly initialized and executed, potentially using techniques to bypass User Account Control (UAC) or inject into legitimate processes.
  • Backdoors: These typically involve modifying system configurations or injecting code into running processes. Examples include creating new user accounts, modifying firewall rules, or registering new services or tasks. They might interact with the Windows Registry to store configuration data or achieve persistence.
  • Reverse Shells: Beyond basic netcat usage, sophisticated reverse shells often involve creating a new process (e.g., using CreateProcess or ShellExecute) and redirecting its standard input/output/error streams to a network socket, effectively giving the attacker command-line control over the victim's machine. Multi-threaded implementations enhance stability and responsiveness. The Metasploit Framework is a common tool used to generate and manage these payloads.
  • Credential Stealers:
  • Hash Dumping: This involves extracting password hashes from the Local Security Authority Subsystem Service (LSASS) process memory or the SAM database. Tools like Mimikatz are widely used for this. These hashes can then be used in Pass-the-Hash attacks to authenticate to other systems without needing the plaintext password.
  • Keystroke Logging: Can be implemented by hooking keyboard input functions (user-space) or by injecting into the kernel to intercept hardware interrupts (kernel-based), offering a more stealthy approach. Physical keyloggers, such as those disguised as USB pass-through devices, are also effective.

Persistence Mechanisms:

  • svchost DLLs (Service Host DLL): Malware often registers itself as a Windows service that mimics or attaches to legitimate svchost.exe processes. This can be achieved by modifying Registry Run keys (HKLM\Software\Microsoft\Windows\CurrentVersion\Run, HKCU\Software\Microsoft\Windows\CurrentVersion\Run), creating new services, or injecting into existing svchost processes.
  • Trojanized System Binaries: This involves patching the executable code of legitimate Windows binaries (e.g., explorer.exe, winlogon.exe) to include calls to the malware's code. This ensures execution whenever the legitimate binary is run.
  • Task Scheduler: Malware can create scheduled tasks (using schtasks.exe or directly via COM objects) to execute itself at specific intervals or upon certain events (e.g., system startup, user logon), providing reliable persistence.

Malware Analysis Methods:

Dr. Sankare detailed both static and dynamic analysis techniques.

  • Static Analysis: Examines the malware without executing it.
  • strings Utility: Used to extract ASCII and Unicode strings from an executable. This can reveal API calls, URLs, error messages, and other indicators of compromise (IOCs).
  • xinfo (Flare Utilities): Provides information about the executable's packaging. It can determine if the malware is packed (compressed or encrypted to hinder analysis) or unpacked. Unpacked malware is easier to analyze statically as its code is readily accessible.
  • PE Studio: A comprehensive tool for analyzing Portable Executable (PE) files. It integrates with VirusTotal to check hashes against known malware databases. It also provides details from the DOS header and file header, such as confirming the executable is a 32-bit binary, identifying its type (e.g., GUI-based program), and crucially, verifying that the file extension matches its actual binary type (e.g., detecting an executable disguised as a JPEG).
  • Dynamic Analysis: Involves executing the malware in a controlled environment (e.g., a virtual machine) and observing its behavior.
  • Regshot: A tool used for taking snapshots of the system registry and file system before and after malware execution. By comparing these snapshots, analysts can identify changes made by the malware, such as new registry keys, dropped files, or modified system configurations.
  • IDA Pro: A powerful disassembler and debugger. When loading malware into IDA Pro, analysts can reverse engineer the code. Dr. Sankare highlighted an example where IDA Pro revealed a call to PSAPI.dll, a helper library that allows developers (and malware authors) to obtain information about running processes and device drivers. This seemingly innocuous call can provide critical reconnaissance for an attacker to identify suitable injection targets.

The Role of AI in Malware Analysis and Evasion:

AI plays a transformative role on both sides of the cybersecurity spectrum.

  • AI for Detection:
  • Accelerates Analysis: AI can quickly process vast amounts of data, identifying potential threats much faster than humans.
  • Detects Patterns: Machine learning algorithms can uncover subtle patterns and anomalies in system behavior, network traffic, and code that might indicate malicious activity, even if a signature isn't present.
  • Use Cases: Anomaly detection (flagging unusual process behavior or network connections), log analysis at scale (parsing and correlating events from numerous logs), automated triage (prioritizing or automatically remediating known threats), and threat intelligence enrichment (correlating new threats with existing intelligence).
  • Tools: Microsoft Defender for Endpoint, CrowdStrike Falcon, VirusTotal (leveraging ML), and even AI models for specific tasks.
  • AI for Evasion (Polymorphic Malware):
  • Attackers use AI, specifically GPT-based tools like OpenAI Codex (or similar large language models), to generate or modify malware code. This allows them to create self-modifying code that changes its hash, signature, and even subtle behavioral patterns each time it executes.
  • This polymorphism makes signature-based detection ineffective, as the malware presents a "new" identity every time it's encountered. AI can assist in obfuscating code, injecting junk instructions, or dynamically encrypting/decrypting payloads.
  • Tools like Py Armor, Shellter, and PEzor are used to hide Python code or pack executables, further complicating analysis.

This detailed exploration emphasizes that a multi-faceted approach, combining traditional analysis techniques with AI-driven tools, and crucially, human intelligence, is essential for navigating the complexities of modern malware.

Demo / Proof of Concept

▶ Watch: Lab demo: Using AI to create polymorphic malware for evasion (18:33)

Dr. Sankare elaborated on a practical lab demonstration she conducted, showcasing how polymorphic malware can be created and used to evade antivirus detection, specifically Windows Defender. This hands-on approach provided clear insights into the capabilities of AI in both generating and hiding malicious payloads.

Lab Setup and Prerequisites:

The demonstration utilized a standard ethical hacking environment:

  • Attacker Machine: Kali Linux, a popular distribution for penetration testing and ethical hacking. The speaker confirmed that Metasploit Framework and Veil Evasion were pre-installed.
  • Target Machine: A virtualized Windows environment, specifically Windows 10 or Windows 11. Crucially, Windows Defender was enabled on the target to simulate a real-world defensive posture.
  • Tools Required:
  • Python 3: For scripting the malware and its loader.
  • OpenAI Codex / GPT-based tool: Used for generating and modifying code to achieve evasion.
  • Metasploit Framework: To generate the initial reverse shell payload.
  • Veil Evasion: A tool designed to bypass security software like antivirus programs.
  • Virtualization Software: VirtualBox or VMware (or UTM for Mac users) for creating an isolated lab environment.
  • Hiding Tools: Py Armor, Shellter, and PEzor for obfuscating and packing Python code and executables.
  • Network Configuration: The virtual machines were configured with host-only or internal networking to ensure safe testing, preventing any malicious traffic from escaping to the broader network.

The Evasion Process:

  1. Payload Generation: The first step involved using Metasploit on the Kali Linux machine to generate a standard reverse shell payload. This payload, typically an executable (.exe) file, was designed to establish a connection back to the attacker.
  2. Initial Transfer and Detection: Dr. Sankare initially attempted to transfer this raw Metasploit-generated payload to the Windows 11 target. As expected, Windows Defender immediately detected and deleted the payload, demonstrating its effectiveness against known signatures. This confirmed that the signatures on Windows 11 were up-to-date.
  3. Targeting an Older System (for demonstration): To proceed with the evasion demonstration, the speaker pivoted to a Windows 10 virtual machine. While Windows Defender is also present on Windows 10, the slightly older signature database or configuration might have allowed the initial, unmodified payload to persist long enough for the next steps. This step highlights the continuous update cycle of security products and the slight differences in detection capabilities across versions.
  4. AI-Assisted Evasion: This was the core of the demonstration.
  • Dr. Sankare used OpenAI Codex (or a similar GPT-based tool) to analyze the detected Metasploit payload. The goal was to identify the specific code segments or characteristics that Windows Defender flagged as malicious (e.g., known API calls, specific byte sequences that contribute to its hash).
  • The AI was then prompted to modify the malware's code to change its signature and behavior. This involved techniques like code obfuscation, adding junk instructions, or restructuring the code logic while maintaining its core functionality (establishing a reverse shell).
  • The modified Python code was then converted into an executable using PyInstaller, allowing it to run on the Windows target.
  1. Polymorphic Implementation (Conceptual): The talk extended this concept to polymorphic malware with AI. The idea is to implement self-modifying code where the malware, with assistance from AI-generated variations, changes its own code every time it executes. This means:
  • If the malware is detected and deleted, the next time it attempts to run (perhaps from a persistence mechanism), it would have a different hash and signature, making it appear as a "new" and unknown threat to the antivirus.
  • The AI would guide these modifications, ensuring functionality while achieving evasion of signature-based detection.
  1. Monitoring and Iteration: After running the hidden payload on the Windows machine, the process involved monitoring Windows Defender logs for detection events. If detected, the attacker would iterate, using AI to further tweak the hiding techniques and code modifications. The speaker also suggested testing against other antivirus products (e.g., BitDefender) to evaluate broader evasion capabilities.

Cautionary Note: Dr. Sankare strongly emphasized that such experiments must be conducted in isolated virtualized environments only. This is crucial because malware behavior can be unpredictable, and unintended consequences on a host system can be severe. She also noted that malware might behave differently on a virtual machine compared to a physical machine due to environmental differences that the malware might detect.

This demonstration effectively illustrated the emerging reality where AI is not just a tool for defenders but also a powerful enabler for attackers to create highly adaptive and difficult-to-detect malware.

Defensive Implications

▶ Watch: Q&A: Latest evasive techniques and effectively interacting with AI tools (30:50)

Dr. Sankare's presentation offers crucial defensive implications for cybersecurity professionals grappling with the evolving threat landscape, particularly the rise of AI-assisted malware. The core message for defenders is a necessary paradigm shift from reactive, signature-based detection to proactive, behavior-centric analysis, augmented by intelligent observation.

1. Embrace Behavioral Analysis Over Signature-Based Detection:

Traditional antivirus relies heavily on signatures, which polymorphic and AI-generated malware can easily bypass. Defenders must prioritize tools and strategies that focus on malware behavior and intent. This means monitoring system processes, network connections, file system changes, and registry modifications for anomalous activities, rather than just known malicious hashes. For instance, observing a legitimate process like svchost.exe exhibiting unusually high memory consumption or making suspicious network connections should trigger an alert, even if its hash is clean.

2. Leverage AI as an Ally, But Validate with Human Expertise:

AI tools, such as those integrated into Microsoft Defender for Endpoint or CrowdStrike Falcon, are invaluable for anomaly detection, log analysis at scale, and automated triage. They can process vast amounts of data and identify patterns that human analysts might miss. However, defenders must understand the limitations of AI, including its susceptibility to false positives and its lack of environmental context. Every AI-generated alert or finding should be subjected to human validation. Analysts must cultivate the critical thinking skills to ask, "Does this belong?" and understand the why behind an alert, rather than blindly trusting automated systems.

3. Enhance Visibility and Continuous Monitoring:

Defenders need comprehensive visibility into their endpoints and networks. Tools like Process Explorer (or similar Endpoint Detection and Response - EDR solutions) should be used to monitor running processes, their parent-child relationships, network activity, and loaded DLLs (e.g., kernel32.dll, winlogon.exe). Any new or unusual processes, especially those that persist across reboots or exhibit unexpected behavior (e.g., an application starting automatically that wasn't intentionally installed), warrant immediate investigation. This continuous observation helps identify stealthy persistence mechanisms and privilege escalation attempts.

4. Understand Attackers' Evasion Tactics:

Knowledge of how attackers use AI to evade detection is a powerful defensive tool. By understanding techniques like code obfuscation, dynamic hash changes, and AI-assisted payload modification, defenders can better anticipate and counter these methods. This includes focusing on detecting the effects of evasion (e.g., unusual process injection, unexpected network traffic) rather than just the initial payload. Regularly testing defenses against advanced evasion techniques in isolated environments, similar to Dr. Sankare's demonstration, is crucial for hardening systems.

5. Prioritize Foundational Security Practices:

While advanced, AI-driven threats are emerging, basic security hygiene remains paramount. This includes:

  • Principle of Least Privilege: Users and applications should only have the minimum necessary permissions.
  • Regular Patching: Keeping operating systems and applications up-to-date mitigates known vulnerabilities that malware often exploits for privilege escalation or initial access.
  • Strong Authentication: Implementing multi-factor authentication (MFA) can significantly reduce the impact of credential theft.
  • Network Segmentation: Limiting lateral movement within a network can contain breaches.
  • Isolated Environments for Analysis: Always use virtual machines or sandboxes for malware analysis to prevent infection of production systems.

6. Cultivate Human Skills and Instinct:

Ultimately, Dr. Sankare stresses that analysis is still a human skill. Defenders must develop their intuition and critical thinking. The ability to discern "noise" from a "real threat" often comes down to asking the right questions and possessing a deep understanding of normal system behavior. AI can move faster, but human intellect provides the context, validation, and adaptability needed to combat the most sophisticated and novel threats. Attackers rely on hiding; defenders must become master observers.

Key Takeaways

  • Behavior and Intent are Paramount: Understanding what malware does and why it does it (e.g., downloader, backdoor, credential stealer) is more critical than just knowing its signature, especially against polymorphic threats.
  • AI is a Double-Edged Sword: While AI tools accelerate defensive analysis, detect subtle patterns, and improve threat intelligence, attackers are equally leveraging AI to generate highly evasive and self-modifying malware, making detection more challenging.
  • Polymorphic Malware Challenges Traditional AV: Malware that changes its code, hash, and behavior with each execution effectively bypasses signature-based antivirus solutions like Windows Defender, necessitating a shift to behavioral and anomaly detection.
  • Human Expertise Remains Indispensable: Despite the power of AI, human validation, critical thinking, and environmental context are crucial for accurate threat assessment, distinguishing false positives, and understanding the true intent of complex malicious activities.
  • Proactive Observation is Key for Defenders: Defenders must continuously monitor system processes, registry changes, network activity, and memory usage, developing an instinct for what "does not belong" to detect stealthy malware and persistence mechanisms.
  • Ethical Hacking and Research Drive Defense: Understanding and replicating attacker methodologies in isolated environments (e.g., using Metasploit, OpenAI Codex for evasion) is essential for security professionals to build resilient defenses against evolving threats.

About the Speaker(s)

Dr. Fatou Sankare is a distinguished figure in the cybersecurity community, serving as both a cybersecurity researcher and an offensive security engineer. Her professional work is primarily focused on cutting-edge areas such as polymorphic malware, adversarial AI, and advanced detection evasion techniques, with a particular emphasis on testing and bypassing widely used security products like Windows Defender. Beyond her research, Dr. Sankare is also an accomplished professor, teaching a range of cybersecurity courses at both undergraduate and graduate levels, including malware analysis, digital forensics, and penetration testing. In her personal life, she is an avid traveler, having visited 19 countries and 27 U.S. states, and is actively pursuing her Private Pilot License (PPL), reflecting a passion for exploration and mastering complex skills.

Reviews

Dr. Zero (Offensive Security Researcher) — WEAK

This is an undergraduate 101 lecture wearing a bullshit mustache that says 'AI'. Running a Metasploit payload through OpenAI Codex and packing it with PyInstaller doesn't make it advanced polymorphic malware; it makes it a parlor trick. Fine for absolute beginners, but heavily overclaimed.

Heather Calloway (CISO) — SOLID

A competent baseline on how adversaries use generative AI to automate polymorphic evasion against legacy antivirus. The technical findings validate the enterprise shift toward behavioral endpoint detection, but much of the presentation spends time defining foundational malware concepts rather than advancing the state of the art.

→ Top-rated talks at CypherCon 9.0 (2026)

All talks from CypherCon 9.0 (2026)