WAF: Making a Problematic Security Tool Suck Less
Emile Spir (Security Researcher · Datadog), Emmanuelle Lejeail (Software Engineer · Datadog)
AppSec Village @ DEF CON 31 · Day 3 · AppSec Village
Overview
Web Application Firewalls (WAFs) have long been a cornerstone of application security, serving as a first line of defense against web-based attacks. However, as applications grow in complexity and attackers become more sophisticated, the efficacy and operational overhead of WAFs have come under scrutiny. In this insightful talk from AppSec Village @ DEF CON 31, Emmanuelle Lejeail and Emile Spir from Datadog delve into the inherent challenges of operating WAFs at a massive scale, drawing from their experience handling billions of requests per second across hundreds of thousands of organizations.

Key moments
- 2:19 WAF popularity and false sense of security
- 4:20 Initial WAF deployment challenges: 'we started crying'
- 6:09 Log4Shell: attacker bypasses and rule unpredictability
- 9:40 The WAF rule dilemma: accuracy vs. noise
- 18:10 WAF blind spots: obfuscation and performance
- 25:00 WAFs as monitoring tools, not protection
- 26:50 Improving WAFs: context and dynamic responses
- 29:00 Final takeaways for WAF users and attackers
WAF: Making a Problematic Security Tool Suck Less
Speakers: Emile Spir (Security Researcher, Datadog); Emmanuelle Lejeail (Software Engineer, Datadog)
Conference: AppSec Village @ DEF CON 31
YouTube: https://www.youtube.com/watch?v=4eDtrbPpZyU
Overview
Web Application Firewalls (WAFs) have long been a cornerstone of application security, serving as a first line of defense against web-based attacks. However, as applications grow in complexity and attackers become more sophisticated, the efficacy and operational overhead of WAFs have come under scrutiny. In this insightful talk from AppSec Village @ DEF CON 31, Emmanuelle Lejeail and Emile Spir from Datadog delve into the inherent challenges of operating WAFs at a massive scale, drawing from their experience handling billions of requests per second across hundreds of thousands of organizations.
The presentation critically examines why WAFs, despite their widespread adoption and compliance mandates, often provide a false sense of security. Lejeail and Spir dissect the fundamental limitations of WAFs, particularly their susceptibility to obfuscation and the constant struggle with false positives and false negatives. More importantly, they offer practical strategies and a paradigm shift in how organizations should perceive and utilize WAFs, advocating for their role as sophisticated monitoring and threat intelligence tools rather than infallible protective barriers. This talk is essential for anyone involved in application security, offering a candid look at WAF realities and a roadmap for making them genuinely more effective.
Background
▶ Watch: WAF popularity and false sense of security (2:19)
The concept of a Web Application Firewall emerged in the mid-1990s as web applications began to proliferate and attract malicious attention. Inspired by network firewalls, WAFs were designed to operate at the application layer, inspecting incoming HTTP requests for suspicious patterns using regexes and dictionaries. This approach made WAFs incredibly appealing: they were easy to explain, highly customizable (with readily available rule sets like the OWASP CRS), simple to demonstrate, and often mandated by compliance standards like PCI DSS. Their apparent simplicity and broad applicability led to their widespread popularity, establishing them as the backbone of runtime application security.
Despite their ubiquity, WAFs suffer from a fundamental flaw: they primarily look for suspicious requests. If an attacker can craft a malicious payload that appears innocuous, it will bypass the WAF. Conversely, legitimate user requests that happen to contain patterns resembling known attacks will trigger false positives. This inherent limitation has led to a false sense of security among many organizations. While alternatives like Runtime Application Self-Protection (RASP) and Interactive Application Security Testing (IAST) have emerged, none have achieved the same level of adoption as WAFs, making the improvement of WAF efficacy a critical challenge for the industry. The core problem persists: how to build a WAF rule that is both effective at catching attacks and precise enough to avoid disrupting legitimate traffic, especially when operating at enormous scale.
Key Findings
▶ Watch: Log4Shell: attacker bypasses and rule unpredictability (6:09)
Datadog's experience operating a WAF infrastructure processing approximately a billion requests per second across hundreds of thousands of organizations revealed several critical insights into the real-world behavior of WAFs. The most significant finding was the unpredictable accuracy of a WAF rule at scale. A rule that appears perfect in a controlled environment can generate an overwhelming number of false positives or fail to detect attacks when exposed to the diverse, real-world traffic of a vast user base. Even minor modifications, such as expanding a rule with a few additional obfuscation steps, necessitate exhaustive re-testing due to unforeseen interactions with application-specific traffic patterns. Conversely, surprisingly simple or "crazy" rules, which theoretically shouldn't work, sometimes prove highly effective in practice, underscoring the necessity of empirical testing over theoretical assumptions.
This unpredictability gives rise to the "Dilemma of Building a WAF Rule." The speakers illustrate this with a graph balancing rule breadth (catching everything vs. being very specific) against bypass difficulty and false positive noise. An ideal rule would be broad yet hard to bypass and generate minimal noise. However, in reality, broad rules often lead to high false positive rates, while highly specific rules are easily bypassed. The goal, therefore, is to find the optimal "space of operation" – a compromise between these extremes. The speakers emphasize that the sheer volume of false positives is a major operational challenge, leading to alert fatigue, potential business impact (blocking legitimate customers), and even the accidental collection of Personally Identifiable Information (PII) through logging. Effective WAF management, therefore, hinges on finding this delicate balance and continuously refining rules based on real-world data to minimize noise while maximizing detection.
Technical Deep Dive
▶ Watch: WAF blind spots: obfuscation and performance (18:10)
The architecture of a typical WAF involves placing a rule engine on a reverse proxy situated in front of the web application. This allows the WAF to inspect all incoming traffic. A rule set, such as the popular OWASP Core Rule Set (CRS), provides a collection of predefined patterns. The talk specifically mentions using Paranoia Level 1 (PL1) of OWASP CRS, which is designed to be less noisy. Finally, a robust monitoring system is crucial to collect and analyze alerts. However, as Datadog quickly discovered, simply deploying this setup can lead to an overwhelming influx of alerts, necessitating sophisticated strategies for rule refinement and alert management.
A compelling case study presented was the response to the Log4Shell vulnerability in late 2021. Initially, a simple rule matching jndi (the entry point for the exploit) followed by ldap and subsequent characters seemed effective. However, attackers rapidly evolved their bypasses:
- Alternative Schemas: Attackers moved beyond
ldapto other schemas. - String Interpolation: Using Log4j's string interpolation features (
${::}or${.}) to obfuscate patterns, requiring increasingly complex regexes. - Spurious Blocks: Inserting arbitrary, meaningless blocks within the payload that Log4j would ignore, further complicating detection.
- Non-ASCII Characters: Employing Unicode normalization tricks that Log4j would process, but WAFs might miss.
During this period, Datadog tested a "monstrosity" of a rule designed to catch all known obfuscations, alongside a surprisingly simple regex (j.n.d.*i) that unexpectedly performed well without triggering false positives. This highlighted the unpredictable nature of WAF rules and the critical importance of real-world testing. The Log4Shell experience alone generated roughly half a million triggers in a week at a smaller scale, underscoring the magnitude of the challenge.
Managing false positives is paramount. Datadog classified false positives into several families:
- Free Text: Applications like Stack Overflow where users paste code snippets (e.g., SQL injection) that resemble attacks.
- Admin Panels/Infrastructure Queries: Internal tools (e.g., PHPMyAdmin) running legitimate database queries that mimic SQL injection.
- Application-Specific Queries: Applications that inherently perform complex queries, leading to patterns that trigger WAF rules (e.g., Datadog's internal systems frequently use the keyword "query", which had to be allowlisted).
To detect these, Datadog employs statistical methods, including application/trigger trends. A rule whose trigger rate closely mirrors the application's overall traffic trend is likely a false positive. Similarly, if a rule is triggered by a very broad range of IPs, it suggests a widespread false positive rather than a targeted attack. More advanced techniques involve machine learning. Embeddings can map payloads into a multi-dimensional vector space, allowing similar payloads (e.g., false positives) to cluster together. New payloads can then be classified based on their proximity to these clusters. Machine learning classification models (including fine-tuned LLMs) can also be used, though building a robust dataset of true attacks and false positives is challenging. A notable example provided was an LLM that correctly identified a SQL injection payload but failed to detect the same payload with a simple comment appended, demonstrating a false negative that could be easily exploited.
WAF blind spots, or false negatives, primarily stem from obfuscation techniques that exploit discrepancies in how WAFs and target applications interpret requests. Examples include:
- Recursive URL Decoding and Null Bytes: Some web servers perform recursive URL decoding and support null byte syntax, allowing
%%30%30to become(for a SQL injection. - MySQL Conditional Comments: MySQL's unique
/!.../syntax allows comments to be interpreted as executable code in specific versions, bypassing WAFs that treat them as standard comments. - URL Parameter Arrays: When developers pass arrays of values in URL parameters, different web servers parse them inconsistently, leading to WAF bypasses if the WAF doesn't decode them precisely as the application would.
The most effective countermeasure is application-aware decoding, where the WAF decodes incoming parameters exactly as the target application would. While not perfect (e.g., for XSS where the final target is a browser), it's highly effective against URL encoding and HTTP shenanigans. However, this comes with performance implications. WAFs must operate within strict latency budgets; adding 100ms to a 2ms application response is unacceptable. Large payloads, especially when the attack is at the end, can tank performance, and strict time limits (e.g., 5ms) can be exploited by attackers sending benign but resource-intensive requests. Expensive decoding strategies must be carefully budgeted.
Finally, WAFs suffer from the "don't know what they don't see" problem. To overcome this, organizations need robust threat intelligence. Shadow deployments, where new rules are deployed in a non-blocking mode across various environments (internal services, customer environments), are crucial for understanding the true threat landscape and rule efficacy without impacting production. Once an attacker is identified, WAFs excel at leveraging their mistakes: common user agents, specific IPs or IP ranges, domains used for payload delivery, or even typos in attack strings. Identifying these patterns allows defenders to "dump all attacker traffic" and "burn" their toolkits by creating targeted, aggressive rules.
Demo / Proof of Concept
▶ Watch: WAFs as monitoring tools, not protection (25:00)
While no live "demo" in the traditional sense was presented, the speakers effectively used the Log4Shell vulnerability as a detailed, real-world proof of concept for demonstrating WAF bypasses and the challenges of rule development. They walked through the evolution of attacker obfuscation techniques – from simple LDAP schema changes to complex string interpolation, spurious blocks, and non-ASCII characters – and how Datadog's WAF rules had to adapt. This section served as a practical illustration of the unpredictability of rule accuracy at scale and the necessity of continuous, data-driven testing. The discussion of Log4Shell's jndi entry point, various obfuscation methods, and the surprising effectiveness of a "crazy rule" provided concrete examples of the technical deep dive in action, highlighting both WAF vulnerabilities and successful defense strategies.
Defensive Implications
▶ Watch: Final takeaways for WAF users and attackers (29:00)
For defenders, the core message is a paradigm shift: view your WAF more as a monitoring system than a protection system. While WAFs offer a layer of defense, their primary value lies in providing visibility into attack attempts and the evolving threat landscape. This means focusing on false positive analysis to ensure that alerts are actionable and don't lead to alert fatigue.
Key defensive strategies include:
- Rigorous Rule Testing: Implement shadow deployments of new or modified rules in non-blocking mode across diverse environments. This allows organizations to gauge a rule's true impact and noise level before enforcing it, avoiding accidental blocking of legitimate users or PII collection.
- Leverage Application Context: Provide the WAF with contextual information about the protected application, such as its programming language (e.g., JavaScript, PHP), frameworks (e.g., SilverStripe), and specific versions. A JavaScript application, for instance, doesn't need to be alerted about PHP-specific attacks. This significantly increases precision by filtering out irrelevant alerts and can even help identify sophisticated threat actors targeting the wrong technology stack.
- Dynamic Response Tuning: Instead of immediate blocking, implement multi-stage responses. For suspicious but uncertain traffic, a CAPTCHA challenge can be issued first. If malicious intent is confirmed, then blocking can occur. Furthermore, leverage false positive analysis to dynamically build allowlist rules, reducing noise and increasing confidence in blocking modes.
- Threat Intelligence Sharing: Actively collect and share threat intelligence, both internally and, for vendors, across their customer base. This collective knowledge helps identify emerging attack patterns and allows for proactive rule development.
- Flexible Rule Sets: When building rule sets, ensure they offer enough flexibility for engineering teams. Overly restrictive rules can break application functionality with minor code changes, hindering development velocity.
By embracing these proactive and analytical approaches, defenders can transform their WAFs from problematic security tools into powerful, intelligent components of their overall security posture.
Key Takeaways
- Reframe WAFs as Monitoring Systems: Defenders should prioritize the WAF's role in providing visibility and threat intelligence over its function as an infallible protection tool.
- Prioritize False Positive Analysis: Dedicate significant effort to understanding and mitigating false positives to ensure actionable alerts and prevent alert fatigue.
- Build Flexible Rule Sets: Ensure WAF rules are not overly restrictive, allowing engineering teams to evolve applications without constant WAF-related disruptions.
- Vendors Must Lead with Low-Noise Rules: WAF vendors have a collective mission to provide users with inherently low-noise rule sets, robust support for trigger analysis, and comprehensive threat intelligence sharing.
- Attackers Will Always Find Bypasses: Due to the WAF's inherent nature, bypasses will always exist. Attackers should focus their obfuscation efforts on ambiguous interpreted obfuscation, as HTTP-level obfuscation is often ineffective.
- Rotate Infrastructure to Evade Detection: Attackers are advised to rotate their infrastructure to avoid detection, as WAFs are highly effective at leveraging persistent attacker mistakes (IPs, user agents, typos) to "burn" their toolkits.
About the Speaker(s)
Emmanuelle Lejeail is a Software Engineer at Datadog, based in Paris. With a background primarily in machine learning models over the past four years, Emmanuelle transitioned to application security at Datadog, where she contributes to their application security products, bringing a unique data-driven perspective to WAF challenges.
Emile Spir is a Security Researcher at Datadog, also based in Paris. He has dedicated the last five years to working on RASP and WAF solutions, initially at Sqreen and subsequently at Datadog following an acquisition in 2021. Emile's extensive experience provides deep insight into the practical implementation and operational complexities of WAF technologies.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Datadog engineers step on stage to admit what we all already know: WAFs are fundamentally broken regex engines that generate more noise than signal. It's an honest, well-presented war story about operating at a billion requests per second, but the grand conclusion—treating your WAF as an alerting tool rather than a blocker—is a lesson the industry learned ten years ago.
Heather Calloway (CISO) — STRONG ACCEPT
DataDog exposes the operational reality of Web Application Firewalls at massive scale, confirming what mature security teams already know: WAFs are not a silver bullet, they are a high-maintenance sensor. The speakers argue convincingly for treating WAFs as threat intelligence feeds rather than infallible blocking layers. It's a required reality check for any program over-relying on edge blocking for application defense.