ChatGPT Please Write Me A Piece Of Polymorphic Malware

ChatGPT Please Write Me A Piece Of Polymorphic Malware

Speakers: Eran Shimony (Principal Security Researcher, CyberArk Labs); Omer Tsarfati (Security Researcher, CyberArk Labs)

Conference: HITB Amsterdam

YouTube: https://www.youtube.com/watch?v=kRF3CoQ9jfk

Overview

In a groundbreaking presentation at HITB Amsterdam, CyberArk Labs researchers Eran Shimony and Omer Tsarfati unveiled "Chatty Cathy," an open-source framework demonstrating how Generative Pre-trained Transformer (GPT) models can be leveraged to create sophisticated polymorphic malware. This talk delved into the offensive capabilities of large language models (LLMs), specifically focusing on how models like OpenAI's DaVinci can be coerced into generating malicious code on demand, bypassing existing content filters, and dynamically mutating its functionality to evade detection.

The core premise of their research challenges the conventional understanding of malware development by offloading the code generation process to an AI. Instead of manually crafting diverse malicious payloads, attackers could theoretically instruct an LLM to produce functionally identical but structurally unique code variants, thereby creating a constantly evolving threat. This approach represents a significant leap in malware sophistication, posing new challenges for traditional signature-based antivirus and heuristic detection systems.

Shimony and Tsarfati's work highlights the critical need for a deeper understanding of LLM security implications, not just from a defensive standpoint but also in anticipating their potential misuse. By open-sourcing Chatty Cathy, they aim to foster further research into both the offensive potential and robust defensive strategies against AI-generated threats, underscoring the ongoing arms race in cybersecurity.

Background

▶ Watch: Introduction: Using GPT for polymorphic malware (0:00)

The foundation of this research lies in the rapid advancements of generative AI and chatbots. Generative AI refers to AI systems capable of producing new content—be it text, images, or voice—based on provided input. Chatbots, or conversational AIs, utilize neural network algorithms to process text or speech input and predict the most appropriate next "token" to generate a response. Early chatbots relied on rule-based systems, but modern iterations employ far more complex algorithms, often rooted in the Transformer architecture first introduced by Google in their 2017 paper, "Attention Is All You Need." OpenAI's GPT models (Generative Pre-trained Transformers) are direct descendants of this research, trained on vast datasets encompassing the entire internet, medical bills, insurance documents, and academic essays.

The speakers detailed the evolution and characteristics of various GPT models. While GPT-1 and GPT-2 were open source, subsequent versions like GPT-3, GPT-3.5, and GPT-4 are largely closed source, reflecting a shift in OpenAI's strategy. Each iteration boasts more parameters and is trained on larger datasets, leading to enhanced capabilities. Of particular interest were GPT-3.5 Turbo and text-DaVinci-003. GPT-3.5 Turbo, which powers ChatGPT, is considered highly advanced, supports roughly 4,000 tokens per request, and is significantly cheaper (10 times) than DaVinci. However, it is heavily protected by sophisticated content filters. These filters, employing Natural Language Processing (NLP) and neural networks, analyze both requests and responses to detect and block illegal or unethical content, such as instructions for building nuclear bombs or generating malicious code.

Text-DaVinci-003, while 10 times more expensive and less "advanced" in some conversational aspects, proved to be the workhorse for their malicious code generation. Crucially, DaVinci possesses significantly weaker content filters, making it amenable to generating code for illicit activities. The speakers demonstrated that simple encoding of prompts could often bypass even the filters present in GPT-3.5 Turbo when using DaVinci.

Understanding how GPT models process information is key. They operate on tokens, which are individual units ranging from single characters to sub-words, words, or even phrases. The AI doesn't comprehend words or sentences in a human sense but rather processes these tokens. Requests and responses are limited by the number of tokens they can contain (e.g., DaVinci allows 4,000 tokens). The cost of interactions is also token-based; for instance, a 50-token prompt might cost a quarter of a cent, while a 160-token response costs half a cent. OpenAI provides new users with $5 worth of API credits, allowing for approximately 200,000 token interactions.

Another critical parameter influencing an LLM's output is temperature. A higher temperature introduces more randomness and creativity into the response, while a lower temperature makes the output more deterministic and focused. For generating malware, finding the right temperature is a balancing act: too low, and the code might be too predictable; too high, and the AI might "hallucinate" incorrect or nonsensical code with unwarranted confidence.

The concept of polymorphic code is central to this research. Polymorphism in this context refers to code that mutates its functionality and algorithm while retaining the same semantic purpose. This mutation makes it challenging for signature-based antivirus (AV) and Endpoint Detection and Response (EDR) solutions to detect and block, as each instance of the malware appears different. Traditional approaches to polymorphism include:

• Encryption and obfuscation: Techniques like those used by LockBit ransomware (which employs AES encryption to hide its code) or the VILock malware family (which generates instances with different heuristic properties) aim to alter the malware's appearance. However, the raw data or underlying logic often remains similar, limiting their effectiveness against advanced analysis.
• Virtual machine packers: Tools like Themida create an isolated virtual environment where malicious code executes, shielding it from detection and analysis. While highly effective, this approach requires significant developmental effort.

The speakers noted that their use of "polymorphic" might technically overlap with "metamorphic" code (which entirely rewrites itself), but they chose to stick with the "polymorphic" terminology for consistency within their talk. Their goal was to intertwine the principles of polymorphic code with GPT models to create self-mutating malware.

Key Findings

▶ Watch: Polymorphic code in malware explained (16:00)

The core findings of Shimony and Tsarfati's research highlight the practical weaponization of LLMs for offensive security:

• Reliable Malicious Code Generation: GPT models, particularly the less-filtered text-DaVinci-003, can be reliably prompted to generate functionally malicious code across various programming languages. This demonstrates a significant shift from LLMs as mere coding assistants to potential automated malware factories.
• Bypass of Content Filters: While models like GPT-3.5 Turbo employ robust content filters based on NLP and neural networks, these can be circumvented. The researchers successfully used DaVinci, with its weaker filtering mechanisms, to generate code that Turbo would reject. They also found that encoding prompts could sometimes bypass filters.
• LLMs as Polymorphic Engines: A key contribution is the demonstration that LLMs can act as dynamic polymorphic engines. By iteratively prompting the AI with slight variations or requests for different implementations, it can generate diverse code for the same malicious task (e.g., multiple ways to achieve persistence), making signature-based detection exceedingly difficult.
• The Chatty Cathy Framework: The research culminated in the development and open-sourcing of Chatty Cathy, a novel framework and "malicious playground." This proof-of-concept showcases a functional architecture for AI-powered polymorphic malware, where users with minimal coding knowledge can leverage GPT models to create custom malware.
• Challenges in AI-Generated Malware: The researchers identified practical challenges, including the AI's tendency to produce incomplete or incorrect code, the need for robust dependency management for generated scripts, and unexpected formatting quirks (like the "DaVinci bug" where a period in the prompt results in a leading dot in the output).
• Crucial Role of Validation: To overcome the AI's occasional inaccuracies, a dedicated validator component was found to be indispensable. This component ensures that the AI-generated malicious code actually works as intended in an isolated environment before being deployed to a target.

Technical Deep Dive

▶ Watch: Engineering challenges: Code validation and dependencies (28:00)

The technical implementation of Chatty Cathy revolves around leveraging specific GPT model characteristics and a custom architecture designed to manage the AI's output.

GPT Model Selection and Parameters:

The choice between GPT models was critical. GPT-3.5 Turbo is highly filtered and expensive for raw token count but cheaper per API call, making it unsuitable for direct malicious code generation. Text-DaVinci-003, despite being 10 times more expensive per token, was chosen due to its significantly weaker content filters. This allowed the researchers to consistently obtain malicious code. Key API parameters used to control the AI's output included:

• model: Specifies the GPT model (e.g., text-DaVinci-003).
• prompt: The instructions given to the AI. The researchers found that being highly descriptive, rather than brief, yielded better results, even if it meant using more tokens.
• temperature: Controls the randomness of the output. A balanced temperature was crucial; too low would make the code predictable, while too high would lead to unreliable or "hallucinated" code.
• max_tokens: Defines the maximum length of the AI's response. DaVinci supports up to 4,000 tokens, allowing for substantial code blocks.

Polymorphism through Prompt Engineering:

The core innovation lies in using the LLM itself as a polymorphic engine. By varying prompts—for example, asking for "more stealthy" code, "adaptive" code for different operating systems, or simply re-requesting the same functionality—the DaVinci model generates functionally equivalent but structurally different code. This dynamic mutation can be applied to various malicious capabilities, such as code injection (with at least 20 known Windows techniques) or achieving persistence (with over 30 known Windows methods). The ability to generate new techniques on the fly, perhaps after every system reboot, significantly enhances evasion capabilities against static detection mechanisms.

Chatty Cathy Architecture:

The framework consists of three main components:

1. CNC (Command and Control) Server: This central component manages the overall operation, issuing commands for malicious activities (e.g., keylogging, encryption, persistence) and receiving results from infected agents.
2. Validator: This is arguably the most critical and innovative component. Given that GPT models can produce incorrect or incomplete code, the Validator operates in an isolated virtual machine environment. When the CNC requests a new malicious capability, the Validator first pulls code from the DaVinci model. It then executes this code within its VM, performing specific tests to ensure functionality. For example, for a keylogger, it might inject a "magic string" of keystrokes and verify if they appear in the output file. If the code fails validation, the Validator discards it and requests a new variant from DaVinci, repeating the process until a working payload is obtained.
3. Malware/Agent: This component runs on the targeted machine. It's a "dumb binary" that communicates with the CNC, receives validated malicious code (as text), executes it on the fly, and sends back any relevant output.

Implementation Choices and Challenges:

• Python: The researchers chose Python for implementing the generated payloads for several reasons:
• GPT models are highly proficient in generating Python code, leading to better quality output.
• Python's cross-platform nature simplifies targeting diverse systems.
• Python scripts are less frequently flagged by AV/EDR solutions compared to compiled binaries.
• The dynamic nature of Python allows the agent to receive and execute text-based code directly, facilitating polymorphism.
• Dependency Management: A significant challenge was managing Python dependencies. DaVinci might generate code requiring external libraries not present on the target. Chatty Cathy addresses this by using Python's built-in model_finder library to identify required imports. If a library is missing, the agent attempts to install it on the fly using pip install. The speakers acknowledged the security risks of this approach (e.g., installing malicious libraries), but it was necessary for the PoC.
• "DaVinci Bug": An amusing yet critical bug was discovered where if a prompt ended with a period or question mark, DaVinci's response would often begin with an extraneous dot. This could render the generated Python code invalid. The solution involved implementing try-except blocks to handle and correct such formatting errors, ensuring the code could be interpreted.

The combination of DaVinci's flexibility, careful prompt engineering, and a robust validation architecture allowed Chatty Cathy to dynamically generate and deploy polymorphic malware, showcasing a potent new vector for cyber threats.

Demo / Proof of Concept

▶ Watch: Chatty Cathy live demo: Keylogger (32:00)

The core of the presentation included a demonstration of the Chatty Cathy framework, illustrating its capability to generate and deploy polymorphic keylogging malware.

The demo began with a view of the CNC (Command and Control) server, which featured an Admin LTE design for its user interface. From this dashboard, the researchers selected an agent and initiated a keylogging command.

In the background, the Validator agent, running within a dedicated virtual machine, received this command. It then began interacting with the DaVinci model via the OpenAI API to request Python code for keylogging. The Validator's activity was shown on the screen, illustrating its process of pulling code and attempting to execute it. The demo highlighted the Validator's role in detecting missing Python plugins or dependencies and its attempts to install them on the fly.

Critically, the Validator injected specific "magic" keystrokes into its virtual machine. Once a working keylogger code was successfully generated by DaVinci and validated (i.e., the magic keystrokes were correctly logged to a file named actions.txt), this validated payload was then pushed to the CNC server.

The demo then switched to the target machine, where the Chatty Cathy agent was already running in the background. The user on the target machine typed "hello word." Within seconds, the CNC dashboard updated, displaying "hello word" as the captured payload, confirming the successful operation of the AI-generated keylogger.

The speakers also elaborated on the validation scenarios for other malicious capabilities:

• Persistence: For persistence payloads (e.g., modifying the Windows Registry), the Validator would execute the generated code, then restart the virtual machine. Post-reboot, it would verify if the malicious program was actively running, confirming successful persistence. This was noted as a more complex validation scenario.
• Encryption: For encryption (ransomware-like) payloads, the Validator would use a file with known content. After the generated code encrypted the file, the Validator would attempt to decrypt it using a key also generated or provided by the AI, verifying both the encryption and decryption functionality.

The demo effectively showcased the entire lifecycle: from a high-level command on the CNC, through AI-driven code generation and rigorous validation, to successful execution on a target, proving the practical viability of AI-powered polymorphic malware.

Defensive Implications

▶ Watch: Q&A: Why Python for AI-generated malware? (40:30)

The emergence of AI-generated polymorphic malware, as demonstrated by Chatty Cathy, presents significant challenges to current cybersecurity defenses. While traditional Antivirus (AV), Endpoint Detection and Response (EDR), and Extended Detection and Response (XDR) solutions remain the primary line of defense, their effectiveness against constantly mutating code is diminished.

Here are the key defensive implications:

• Evasion of Signature-Based Detection: The core strength of AI-generated polymorphism is its ability to create functionally identical but structurally unique code variants. This renders static, signature-based detection mechanisms largely ineffective, as the "signature" of the malware is constantly changing.
• Dynamic, Memory-Resident Payloads: The Chatty Cathy model executes payloads received on the fly. This means the malicious code often resides only in memory during execution and leaves no persistent artifacts on disk after its task is complete. This "fileless" nature makes forensic analysis and disk-based detection much harder.
• Cross-Platform Triviality: The use of Python for generated payloads makes the malware inherently cross-platform. Porting the agent to Linux or macOS is relatively trivial, and on Linux, the concept of code signing (which could otherwise be a defensive measure for the agent binary) is less prevalent, further complicating detection.
• Signed Agent, Dynamic Payloads: While the initial agent binary could be signed to bypass some execution policies, the actual malicious payloads are dynamically generated scripts. This means defenders cannot rely solely on code signing to prevent malicious activity, as the signed agent is merely a conduit for unsigned, AI-generated code.
• Hosting on Cloud Providers: The CNC and validator components can be hosted on legitimate cloud infrastructure (e.g., AWS, GitHub Actions), blending in with normal traffic and making it difficult to block based on reputation.
• The AI Arms Race: The development of offensive AI capabilities necessitates a corresponding evolution in defensive AI. Researchers are exploring how defensive AI models could be fine-tuned to detect AI-generated code or patterns of malicious AI behavior. This highlights an ongoing "AI vs. AI" arms race in cybersecurity.
• Regulation and Ethical Considerations: The speakers emphasized the broader implications of AI's rapid advancement. They raised concerns about the lack of robust regulation for AI systems, especially those developed by corporations driven by profit. This could lead to a future where powerful AI tools are misused, either intentionally or unintentionally, with unforeseen consequences. The "competence, not malice" quote from Stephen Hawking underscores that even non-malicious AI can pose risks if its capabilities outstrip human control or understanding.

In essence, defending against AI-generated polymorphic malware requires a shift from static signature analysis to more advanced behavioral analysis, anomaly detection, and potentially, the deployment of defensive AI systems capable of understanding and predicting the output of offensive AI.

Key Takeaways

• LLMs as Malware Factories: Generative AI models, particularly those with weaker content filters like text-DaVinci-003, can be effectively weaponized to generate diverse and functional malicious code on demand, transforming them into potent tools for malware development.
• Polymorphism Reinvented: LLMs can serve as dynamic polymorphic engines, producing functionally equivalent but structurally unique code variants for the same malicious task (e.g., persistence, keylogging), thereby significantly challenging traditional signature-based AV/EDR detection.
• The Chatty Cathy Blueprint: The open-source Chatty Cathy framework demonstrates a practical architecture for AI-powered malware, featuring a CNC, a critical Validator component (to ensure AI-generated code functionality), and a Python-based agent for cross-platform execution.
• Content Filters Are Not Bulletproof: While advanced content filters exist in models like GPT-3.5 Turbo, they can be bypassed through strategic model selection (e.g., DaVinci) and prompt engineering techniques, highlighting a vulnerability in current AI safety mechanisms.
• Defensive Shift Required: The rise of AI-generated polymorphic malware necessitates a shift in defensive strategies from static signature analysis to more advanced behavioral detection, anomaly analysis, and potentially, the deployment of defensive AI to counter AI-driven threats.
• Future of Offensive AI: The research points to a future where LLMs could be fine-tuned on vulnerability data to discover new bugs, enhance fuzzing techniques, and automate exploit development, signaling a new era in offensive security research.

About the Speaker(s)

Eran Shimony is a Principal Security Researcher at CyberArk Labs. His primary focus lies in the discovery and exploitation of zero-day bugs, particularly within the Windows kernel. In this talk, however, he shifted his focus to exploring the offensive capabilities of generative AI in malware development.

Omer Tsarfati is a Security Researcher at CyberArk Labs. He primarily concentrates on Windows kernel research and vulnerability discovery. In addition to his research, he enjoys playing basketball and video games in his spare time. Together, Shimony and Tsarfati presented their collaborative work on leveraging GPT models to create polymorphic malware.