Generative Adversarial Network (GAN) based autonomous penetration testing for Web Applications
Ankur Chowdhary (Cybersecurity Researcher)
AppSec Village @ DEF CON 31 · Day 1 · AppSec Village
Overview
In an insightful presentation at AppSec Village @ DEF CON 31, Ankur Chowdhary unveiled his research on leveraging Generative Adversarial Networks (GANs) for autonomous penetration testing of Web Application Firewalls (WAFs). The talk, titled "Generative Adversarial Network (GAN) based autonomous penetration testing for Web Applications," addressed the perennial cat-and-mouse game between attackers seeking to bypass defenses and security teams striving to protect web applications. Chowdhary's work proposes an innovative, AI-driven methodology to automatically generate novel attack payloads capable of circumventing WAF rules, thereby identifying vulnerabilities in their signature databases.

Key moments
- 0:00 Introduction: GANs for WAF penetration testing
- 8:00 ModSecurity WAF and manual bypass challenges
- 13:45 Generative Adversarial Networks (GANs) explained
- 20:00 Adapting GANs for text-based attack payloads
- 27:00 Automated WAF bypass demonstration process
- 30:20 Conditional GAN's superior WAF bypass performance
- 34:00 Future: LLMs for payloads, commercial WAF testing
Generative Adversarial Network (GAN) based autonomous penetration testing for Web Applications
Speakers: Ankur Chowdhary (Cybersecurity Researcher)
Conference: AppSec Village @ DEF CON 31
YouTube: https://www.youtube.com/watch?v=V5YlZ-yYDrc
Overview
In an insightful presentation at AppSec Village @ DEF CON 31, Ankur Chowdhary unveiled his research on leveraging Generative Adversarial Networks (GANs) for autonomous penetration testing of Web Application Firewalls (WAFs). The talk, titled "Generative Adversarial Network (GAN) based autonomous penetration testing for Web Applications," addressed the perennial cat-and-mouse game between attackers seeking to bypass defenses and security teams striving to protect web applications. Chowdhary's work proposes an innovative, AI-driven methodology to automatically generate novel attack payloads capable of circumventing WAF rules, thereby identifying vulnerabilities in their signature databases.
The core motivation behind this research stems from the increasing sophistication of web application threats and the inherent limitations of static, signature-based WAFs. While WAFs provide a crucial layer of defense, their effectiveness is often challenged by attackers who craft subtle variations of known exploits to evade detection. Chowdhary's approach, rooted in his extensive background in AI and cybersecurity, demonstrates how GANs can learn to generate these evasive payloads, offering a proactive mechanism for both offensive security testing and defensive WAF improvement.
This talk is particularly significant for several reasons. Firstly, it pushes the boundaries of automated penetration testing beyond traditional fuzzing, introducing a learning-based system that can adapt and evolve its attack strategies. Secondly, it provides a tangible methodology for security teams to enhance their WAFs by using the AI-generated bypasses to update and strengthen their signature databases against unknown or highly obfuscated attack vectors. Finally, it highlights the growing intersection of advanced AI techniques and practical cybersecurity challenges, setting a precedent for future research in autonomous security analysis.
Background
▶ Watch: Introduction: GANs for WAF penetration testing (0:00)
Web applications continue to be a primary target for attackers, with a persistent array of vulnerabilities despite advancements in development practices and security tools. The OWASP Top 10, a widely recognized standard for web application security, consistently highlights issues like Broken Access Control, Cross-Site Scripting (XSS), Insecure Direct Object References (IDORs), and Vulnerable and Outdated Software Components. These threats underscore the complexity of securing modern web applications, which often involve diverse technologies, intricate logic flows, and multiple data stores.
To mitigate these risks, organizations deploy a range of defensive mechanisms. These include Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS), network firewalls, and crucially, Web Application Firewalls (WAFs). Traditional defenses often rely on signature databases, which detect known attack patterns. Complementary techniques like input sanitization, encoding, and truncation are also employed to prevent malicious input from reaching the backend. However, the efficacy of these methods is directly tied to the completeness and up-to-dateness of their signature sets and rules.
WAFs, operating at Layer 7 of the OSI model, are designed to inspect HTTP traffic for application-layer attacks. They combine various defense mechanisms, including user-agent blocking, rate limiting, and the application of attack signatures to filter out malicious requests. A prominent example is ModSecurity, an open-source WAF widely deployed across technologies like Apache, Nginx, and IIS. ModSecurity offers different paranoia levels, ranging from level 0 (minimal protection) to level 4 (full nuclear protection with extensive rules). As the paranoia level increases, so does the number of rules and, consequently, the difficulty of bypassing the firewall. However, higher paranoia levels also introduce a greater likelihood of false positives, making practical deployment a balancing act.
The fundamental challenge with signature-based WAFs is their reactive nature. While they effectively block known attack patterns, attackers constantly devise subtle variations, obfuscations, and novel techniques to evade detection. For instance, a direct command like cat /etc/passwd might be blocked by a WAF, but a carefully crafted variation using wildcards or character substitution (e.g., /?c?t /?t?/?p?s?w?) could potentially bypass the signature if the WAF's de-obfuscation logic is not robust enough. Manually discovering and testing these evasive payloads is an arduous, time-consuming, and often incomplete task for human penetration testers. This "game of cat and mouse" creates a clear demand for automated, intelligent systems that can proactively identify and generate such bypasses, leading to more resilient WAF defenses.
Key Findings
▶ Watch: Generative Adversarial Networks (GANs) explained (13:45)
Ankur Chowdhary's research demonstrates several pivotal findings regarding the application of GANs to autonomous WAF penetration testing:
- GANs can successfully generate novel attack payloads capable of bypassing WAFs. The core contribution is proving that a Generative Adversarial Network can learn the characteristics of attack payloads and generate new, unseen variations that evade WAF detection while still achieving their malicious objective. This moves beyond simple fuzzing to intelligent, context-aware payload generation.
- Conditional GANs (CGANs) significantly outperform vanilla GANs for this task. By providing labels to both the generator and discriminator, the CGAN model converges faster and exhibits superior performance in generating effective bypass payloads. This indicates that guiding the learning process with contextual information about successful and unsuccessful attacks is crucial for text-based adversarial generation.
- The generated samples effectively circumvent ModSecurity WAF rules at paranoia levels 1 and 2. The research specifically validated the model's ability to bypass the WAF's filters, highlighting its practical utility against commonly deployed WAF configurations. While higher paranoia levels (3 and 4) present greater challenges, the success at levels 1 and 2 represents a significant step.
- Reinforcement Learning (RL) is a critical component for handling the sequential nature of text data. Recognizing that attack payloads are sequences of characters with semantic meaning rather than just arbitrary noise, the integration of an RL model for the generator was key to overcoming issues like model collapse and vanishing gradients commonly faced when applying GANs to discrete text data.
- The generated payloads can be directly used to improve WAF signature databases. This is perhaps the most impactful finding for defenders. The successful bypasses generated by the GAN serve as invaluable data points to identify gaps in existing WAF rules. By adding these new, evasive signatures, WAFs can be proactively strengthened against advanced, polymorphic attacks, enhancing overall defensive posture.
- Current Large Language Models (LLMs) are not yet effective for generating WAF bypass payloads. Despite the rise of models like ChatGPT, Chowdhary's experiments indicated that LLMs currently lack the semantic and structural understanding required to reliably generate functional and evasive WAF bypasses without significant, intuitive engineering. This underscores the specialized nature of the GAN-based approach for this particular problem.
Technical Deep Dive
▶ Watch: Adapting GANs for text-based attack payloads (20:00)
The technical foundation of Chowdhary's research lies in the innovative application of Generative Adversarial Networks (GANs), adapted to the unique challenges of generating text-based attack payloads. At its core, a GAN consists of two neural networks: a generator and a discriminator, engaged in a zero-sum game. The generator's role is to create synthetic data that mimics real data, while the discriminator's role is to distinguish between real data and the generator's fakes. Through iterative training, both models continuously improve, with the generator striving to produce increasingly realistic fakes and the discriminator becoming more adept at identifying them.
However, directly applying GANs, which were originally conceived for continuous data like images, to discrete text data presents significant hurdles. Text has semantics, structure, and a vocabulary, unlike the continuous pixel values of an image. Chowdhary identified two primary problems:
- Model collapse and vanishing gradients: In text generation, beyond a certain point, the discriminator struggles to provide meaningful feedback to the generator, leading to a state where the generator produces limited variations or the training stagnates.
- Discrete nature of text: Generating individual characters or words means the output is discrete, making it difficult to backpropagate gradients directly from the discriminator to the generator, which is essential for learning.
To address these challenges, Chowdhary implemented several critical modifications:
1. Conditional GAN (CGAN) Architecture
Instead of a vanilla GAN, the research utilizes a Conditional GAN (CGAN). In a CGAN, labels (or some form of conditioning information) are provided to both the generator and the discriminator. For this application, these labels would represent information about the attack's effectiveness or the WAF's response (e.g., whether a payload was blocked, triggered a warning, or successfully executed). This conditioning helps the generator to produce more targeted and relevant payloads from the outset, leading to faster convergence and more stable training. The results showed that CGANs consistently performed better and converged faster than vanilla GANs.
2. Reinforcement Learning (RL) for the Generator
To overcome the discrete nature of text and the challenges of gradient propagation, the generator is modeled as a Reinforcement Learning (RL) agent. In this setup:
- States: Individual characters within a payload.
- Actions: The selection of the next character to append to the current sequence.
- Policy: The sequence of characters that constitutes a complete payload.
- Reward: The generator receives a positive reward if the generated payload successfully bypasses the WAF and triggers the intended malicious action (e.g., an XSS popup). Conversely, it receives a penalty if the payload is blocked by the WAF.
This RL approach allows the generator to learn optimal sequences of characters (policies) that are effective at bypassing the WAF. The discriminator, in turn, acts as a critic, providing feedback on the "realness" or "maliciousness" of the generated sequences.
3. Wasserstein Distance
To combat the issues of vanishing gradients and model collapse, Chowdhary's work likely incorporates techniques inspired by Wasserstein GANs (WGANs), which use the Wasserstein distance (also known as Earth Mover's distance) as a loss function. Unlike traditional GANs that use Jensen-Shannon divergence, the Wasserstein distance provides a more stable gradient even when the distributions of real and generated data have little overlap. This ensures that the discriminator always provides useful feedback to the generator, preventing the model from collapsing.
4. Semantic Tokenization and WAF Interaction
The process begins with an initial set of attack samples. These samples are fed into the generator, which produces variations. These generated payloads are then "thrown" at the target web application, which is protected by a WAF (e.g., ModSecurity). The system then meticulously observes the WAF's response and the application's behavior:
- WAF logs: Did the WAF trigger an alert? Was the payload denied?
- Application response: Did the payload lead to a 200 OK, 403 Forbidden, or a specific error? More importantly, did it achieve its objective (e.g., an XSS popup)?
This feedback is crucial for semantic tokenization, where each payload is associated with labels indicating its outcome (e.g., "bypassed," "blocked," "warning"). This labeled data is then used to update the discriminator and, through the RL mechanism, the generator. The objective is to maximize the generation of payloads that successfully execute and bypass the WAF.
5. ModSecurity's De-obfuscation Logic
A specific technical challenge addressed was ModSecurity's behavior. The WAF uses components like the PhantomJS library with WebKit and JavaScript API to de-obfuscate payloads. This means simple character substitution might not be enough. The model needed to learn semantic variations that could bypass ModSecurity's de-obfuscation logic, such as using specific partial non-alphanumeric characters or other encoding tricks that the WAF might misinterpret. The model's ability to learn and exploit these nuances was key to its success.
Through this sophisticated architecture, the GAN continuously refines its ability to generate stealthy, functional, and WAF-evading attack payloads. The discriminator's loss (D loss 1 for real attacks, D loss 2 for fake attacks) converges around 250 epochs, indicating that the model successfully learns to distinguish between effective and ineffective payloads, and the generator learns to produce the former.
Demo / Proof of Concept
▶ Watch: Conditional GAN's superior WAF bypass performance (30:20)
The demonstration showcased the practical application of the GAN-based system, albeit with the acknowledgement that the extensive pre-training phase, which can take "hours and hours," was not shown live. The core of the demo illustrated how the system automates the process of generating and testing WAF bypass payloads.
The scenario involved a vulnerable web application known to have a reflected XSS vulnerability, protected by a ModSecurity WAF. The initial steps involved:
- Input Parameter Identification: The system first identifies the specific input parameter in the web application that is susceptible to the XSS vulnerability.
- Initial Payload Bombardment (Burp Intruder): An initial list of payloads is compiled and then "hammered" against the application using tools like Burp Intruder. This step is crucial for gathering initial feedback.
- Response Analysis: For each payload, the system captures the HTML response from the application. It annotates this data based on the HTTP status code (e.g., 200 OK, 403 Forbidden), and whether it led to an error or a warning.
- WAF Log Monitoring (Docker): Concurrently, the system monitors the logs generated by the ModSecurity WAF (hosted, for instance, via Docker). This allows it to determine if a payload triggered a WAF alert or was explicitly denied by a signature.
- Label Generation: Both the application response and the WAF logs are used to update the labels for the training data. A "successful attack" is defined by two criteria: the payload bypasses the WAF (no alert/denial in logs) AND it successfully executes on the application (e.g., an XSS popup appears).
- Automated Testing (Selenium Web Driver): To scale the testing, the demo highlighted the use of Selenium Web Driver. Selenium automates the interaction with the web application, throwing the generated payloads and observing the client-side behavior. This is critical for detecting successful reflected XSS attacks, which manifest as pop-up messages in the browser. The system uses webhooks or other mechanisms within Selenium to identify when an alert is triggered.
- Signature Database Update: As the system identifies payloads that successfully bypass the WAF and trigger an alert, these "positive payloads" are then used to update the WAF's signature database. This iterative process allows the WAF to learn from its bypasses, effectively improving its defensive capabilities.
The demo visually represented the automated bombardment of payloads and the subsequent identification of those that successfully "pop up" an alert, signifying a successful XSS execution and WAF bypass. This concrete demonstration underscores the practical utility of the GAN-based approach in proactively identifying WAF weaknesses and strengthening defenses.
Defensive Implications
▶ Watch: Future: LLMs for payloads, commercial WAF testing (34:00)
The research presented by Ankur Chowdhary carries profound implications for defensive security strategies, particularly concerning the deployment and maintenance of Web Application Firewalls. The central message for defenders is clear: static, signature-based WAFs are insufficient against an intelligent, adaptive adversary, and continuous, AI-driven adversarial testing is essential for robust protection.
Here are the key defensive implications:
- Proactive WAF Signature Improvement: The most direct implication is the ability to proactively enhance WAF signature databases. Instead of waiting for real-world attacks to expose WAF weaknesses, security teams can leverage GANs to generate novel, evasive payloads. These successfully generated bypasses can then be immediately incorporated into WAF rules, effectively "immunizing" the WAF against previously unknown or highly obfuscated attack vectors. This shifts WAF management from a reactive patching model to a proactive, learning-based improvement cycle.
- Beyond Static Signatures: The research highlights the need for WAFs to evolve beyond simple pattern matching. WAF vendors and security engineers should investigate integrating more sophisticated semantic analysis, de-obfuscation capabilities, and potentially even behavioral anomaly detection into their WAF engines. The fact that the GAN needed to learn ModSecurity's specific de-obfuscation logic (e.g., PhantomJS, WebKit) underscores that current WAFs can be tricked by subtle variations.
- Adversarial AI for Defensive Testing: Organizations should consider adopting adversarial AI techniques, similar to what Chowdhary demonstrated, as a standard practice for WAF testing. This could be integrated into CI/CD pipelines or as part of regular red teaming exercises. An autonomous system that constantly tries to bypass the WAF can provide continuous feedback on its effectiveness, identifying blind spots before malicious actors do.
- Understanding Attack Variations: The research provides a deeper understanding of how attackers can craft variations of known exploits. Defenders gain insight into the types of obfuscation, encoding, and character substitutions that are most effective at bypassing WAFs. This knowledge can inform the development of more comprehensive and resilient WAF rule sets.
- Vendor Engagement and Evolution: The anecdote about Cloudflare reaching out to Chowdhary to test their WAF is a positive sign. It indicates that WAF vendors recognize the value of such adversarial research. Defenders should demand that their WAF providers demonstrate how their products are resilient against AI-generated attacks and how they incorporate advanced learning techniques into their threat intelligence and rule updates.
- Limitations of Current LLMs: The finding that current Large Language Models are not yet effective for generating WAF bypass payloads provides a temporary reprieve for defenders. While LLMs are rapidly advancing, their current inability to generate semantically correct and evasive attack payloads without significant engineering means that the specialized GAN-based approach remains a more immediate threat, and thus, a more immediate defensive concern.
In essence, Chowdhary's work advocates for an ongoing, intelligent "arms race" where defensive tools are continuously challenged and improved by equally intelligent offensive systems. Organizations that embrace this paradigm will be better equipped to defend against the next generation of web application attacks.
Key Takeaways
- Generative Adversarial Networks (GANs), particularly Conditional GANs (CGANs), offer a powerful and automated approach to penetration test and bypass Web Application Firewalls (WAFs).
- Reinforcement Learning is crucial for adapting GANs to the discrete and sequential nature of text-based attack payloads, enabling the generator to learn effective WAF bypass strategies.
- The research successfully demonstrated the generation of novel payloads capable of circumventing ModSecurity WAF rules at paranoia levels 1 and 2, highlighting the practical vulnerability of signature-based defenses to intelligent obfuscation.
- The output of these AI systems—the successfully generated bypass payloads—can be directly utilized to improve and update WAF signature databases, transforming a reactive defense mechanism into a proactively learning one.
- While rapidly evolving, current Large Language Models (LLMs) like ChatGPT are not yet effective for automatically generating sophisticated WAF bypasses without significant, specialized engineering.
- To maintain robust security, organizations must adopt a continuous, adversarial AI testing methodology for their WAFs, ensuring they can defend against evolving, AI-generated attack variations.
About the Speaker(s)
Ankur Chowdhary is a distinguished cybersecurity researcher with a specialized background in the application of Artificial Intelligence within the field of cybersecurity. He currently serves as a Senior AppSec Engineer at a company building AI-based sales software, which naturally aligns with his expertise. Chowdhary holds a PhD with a specialization in using artificial intelligence in cyber security, underscoring his deep academic and practical knowledge in this interdisciplinary domain. Beyond his corporate role, he is actively involved in mentoring students participating in cybersecurity defense competitions (CCDC programs), demonstrating his commitment to fostering new talent in the industry. His research insights, including the work presented at AppSec Village, are often shared on his personal website, reflecting his dedication to contributing to the broader cybersecurity community.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Chowdhary built a Conditional GAN wired to a Reinforcement Learning agent to fuzz ModSecurity, and the math is actually legitimate. But it only bypasses Paranoia Levels 1 and 2, which you can practically bypass by sneezing on your keyboard. It's a solid engineering exercise in applied ML, but it stops short of beating production-grade rule sets.
Heather Calloway (CISO) — SOLID
This research systematically proves what many of us already suspect: static WAF signatures are highly vulnerable to automated, AI-driven payload mutation. While the underlying GAN methodology is geared toward security engineers, the leadership takeaway is that relying on traditional WAFs as a permanent compensating control for bad code is a losing bet. It is a competent, practical AppSec presentation that makes a strong case for funding continuous adversarial testing.