Another One Bites the DAST

Another One Bites the DAST

Speakers: István Albert-Tóth, DevSecOps and Product Development Director, VerSprite cybersecurity

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=caRdRRSORVg

Overview

In "Another One Bites the DAST," István Albert-Tóth, DevSecOps and Product Development Director at VerSprite cybersecurity, delivers a compelling and critically important talk on the pervasive shortcomings of enterprise Dynamic Application Security Testing (DAST) tools. Albert-Tóth, also known as the OWASP Curve Guard project leader and a contributor to open-source tools like Nuclei, shares his journey from a developer who manually found critical vulnerabilities to a security champion determined to expose the "false sense of security" these expensive tools often provide. The core message is clear: DAST tools, despite their hefty price tags and vendor promises, frequently fail to identify real-world security flaws in modern, complex applications, leaving organizations vulnerable.

The talk delves into two distinct, real-world use cases. The first involves a legacy, single-page application (SPA) with an obscure JavaScript framework and a non-RESTful JSON API, where a multi-thousand-dollar DAST tool proved utterly ineffective. The second case examines a modern, cloud-based application with complex, incremental authentication, where an even more expensive, multi-million-dollar DAST platform similarly faltered. Albert-Tóth meticulously dissects the reasons for these failures, from authentication challenges and poor endpoint discovery to lack of visibility and tool inflexibility. He then presents ingenious, "hacker-minded" solutions, emphasizing custom automation, open-source tools, and a deep understanding of application architecture to achieve meaningful security coverage.

This presentation is crucial for any organization investing in DAST, or indeed, any application security professional. It challenges the conventional wisdom that expensive tools equate to robust security, advocating instead for a pragmatic, observable, and adaptable approach. Albert-Tóth’s insights highlight the necessity of validating tool effectiveness, building custom integrations, and leveraging developer expertise to overcome the inherent limitations of off-the-shelf security solutions. Ultimately, the talk serves as a powerful reminder that security is about solving problems, not merely buying products.

Background

▶ Watch: The DAST tool promise vs. reality (0:00)

The allure of enterprise DAST tools often begins with a seductive promise from vendors: "just pay this tool, point, click, secure." Organizations are led to believe that by investing in these solutions, their application security will be automated, compliance checkboxes will be met, and they can "sleep well at night." However, as Albert-Tóth starkly illustrates, the reality often falls far short of this ideal. Initial scan reports frequently highlight only "informational" findings – self-signed certificates, missing X-Frame-Options headers, or easily improved Content Security Policies (CSPs) – with a conspicuous absence of critical vulnerabilities. This leads to a dangerous "mission accomplished" mentality and a pervasive false sense of security.

Albert-Tóth's personal journey into application security was ignited by this very discrepancy. As a lead developer and security champion at a Fintech Corporation, he manually discovered critical vulnerabilities like HTML injection, cross-site scripting (XSS), SQL injection, and widespread broken authorization, all of which the supposedly "secure" DAST tool had completely missed. This experience raised a fundamental question: "What did the scanner actually test?"

The problem is exacerbated by the evolving landscape of modern web applications. Traditional DAST tools struggle significantly with:

• Single Page Applications (SPAs): These applications often handle routing on the front-end and rely on complex internal logic within JSON payloads, which DAST crawlers find difficult to navigate and understand.
• Non-RESTful JSON-based APIs: Many legacy or specialized applications use APIs that don't conform to typical RESTful patterns, making it challenging for scanners to discover and interact with endpoints effectively.
• Obscure JavaScript Frameworks: Proprietary or less common frameworks (like XJS mentioned in the talk) can present unique parsing and interaction challenges for DAST tools.
• Complex Authentication Flows: Incremental authentication (username then next, then password then next), multi-step flows, token-based (e.g., OAuth, JWT), or certificate-based authentication mechanisms are often poorly handled, preventing scanners from even gaining access to protected areas of the application.
• Lack of Visibility: Enterprise DAST tools frequently operate as "black boxes," offering no insight into the payloads they send, the requests they make, or the reasons for their failures. This opacity makes debugging and validation nearly impossible.
• Test Data Dependency: Many vulnerabilities only manifest when specific data exists in the application's database. If a DAST scan runs against an unseeded environment, vulnerable code paths might never be reached, leading to missed findings (e.g., a SQL injection vulnerable only if a specific company record exists).

Crucially, DAST tools typically "don't report what they missed." This prevalence of false negatives is far more dangerous than false positives, as organizations remain unaware of critical vulnerabilities, creating silent failures that masquerade as security successes. This background sets the stage for Albert-Tóth's creative and technical solutions to these pervasive industry challenges.

Key Findings

▶ Watch: Achieving DAST code coverage with instrumentation (14:00)

István Albert-Tóth's extensive experience and the two detailed use cases presented in his talk reveal several critical findings regarding the limitations of enterprise DAST tools and the effective strategies to overcome them.

Firstly, a primary point of failure for DAST tools is their inability to handle complex authentication mechanisms. Whether it's multi-step, incremental authentication flows (like those seen in modern identity providers such as Okta), or applications relying on both security headers and cookies simultaneously, off-the-shelf DAST tools frequently fail to establish and maintain authenticated sessions. This immediately renders them ineffective against protected application areas.

Secondly, DAST tools often suffer from poor crawling and endpoint discovery, especially in modern application architectures. Single Page Applications (SPAs) with client-side routing, non-RESTful JSON APIs, or APIs with thousands of endpoints and complex structures (even with provided Swagger/OpenAPI specifications) prove challenging for scanners to fully map. This results in a significant portion of the application remaining untested.

A pervasive problem is the lack of visibility into the scanner's internal operations. Tools often do not provide detailed logs of requests, responses, or the reasons behind authentication failures or crawling issues. This "black box" nature prevents security teams from understanding why a scan failed or what areas were truly covered, contributing to the false sense of security.

Furthermore, Albert-Tóth highlighted tool inflexibility and bugs within commercial DAST platforms. This includes issues with parsing large API specifications (e.g., 70,000 lines of JSON/YAML), or limitations that prevent users from setting crucial HTTP headers like Referer (often required by Web Application Firewalls or application logic), or correctly combining authentication elements like cookies and headers. These are not minor inconveniences but fundamental flaws that can cripple a scan's effectiveness.

Finally, the talk underscored the critical role of application-level context, specifically the need for relevant test data to trigger certain vulnerable code paths. Without properly seeded databases, DAST tools might scan code that contains vulnerabilities but never actually execute the vulnerable logic, leading to missed findings. Similarly, understanding the application's code coverage during a DAST scan is vital to identify areas that were simply not reached by the dynamic scanner.

The overarching key finding, however, is that these shortcomings are not insurmountable. By adopting a "hacker mindset" – treating tools as APIs, building robust observability, extending and automating existing solutions, and leveraging deep developer skills – organizations can creatively circumvent these limitations. Custom automation, open-source tools like Burp Suite Pro, Playwright, and strategic use of cloud infrastructure (AWS Lambda) can be combined to create highly effective, tailored DAST solutions that significantly outperform expensive enterprise platforms, providing genuine security insights and coverage.

Technical Deep Dive

▶ Watch: Sitting in the middle: Reverse proxy for DAST visibility (20:00)

Albert-Tóth presented two detailed use cases, each illustrating distinct challenges with enterprise DAST tools and the creative, technical solutions devised to overcome them.

Use Case 1: Legacy Single Page Application with Non-RESTful API

The first scenario involved a Fintech company with a legacy XJS Single Page Application (SPA) and a Java backend. The application featured a complex, non-RESTful JSON-based API where most logic and routing were handled client-side, exposing only two or three actual API endpoints. The DAST tool in question was an expensive enterprise desktop solution, costing $8,000 per year per seat.

Initial scans with this tool yielded predictable "informational" findings like self-signed certificates and missing headers, but no critical vulnerabilities. Albert-Tóth's investigation revealed the tool managed to authenticate but then stopped at the landing page, failing to crawl any further. Instead, it resorted to directory brute-forcing for hours, generating numerous 404 errors in the application's debug logs. This indicated a fundamental failure in understanding the SPA's dynamic nature.

To gain visibility, Albert-Tóth employed a multi-pronged technical approach:

1. Application Debug Logging: As a developer, the first step was to enable debug logging in the target Java application, which immediately revealed the scanner's brute-forcing attempts and 404s.
2. Proxying DAST Traffic: The DAST tool had a proxy option, which was configured to route all its traffic through Burp Suite Pro. This provided full visibility into the requests being sent, confirming the scanner's inability to crawl beyond the login page.
3. Test Data Awareness: A crucial insight was the dependency on test data. Albert-Tóth highlighted a SQL injection vulnerability example where the vulnerable code path would only be reached if specific data (e.g., a company record) existed in the database. Without seeded data, even a perfectly crafted payload would be ineffective due to an early return statement in the code.
4. Ditching the Enterprise Scanner: Given the scanner's ineffectiveness, the bold decision was made to replace it with a custom, automated solution built around Burp Suite Pro.
5. Custom API Descriptor Generation: Since the API was non-RESTful and complex, a custom approach was needed to define its structure. The team parsed the Java codebase (leveraging techniques to analyze the application's syntax tree) to automatically generate API descriptors. These descriptors included all API endpoints, parameter definitions, enum permutations, and crucially, precise payload injection points within the JSON structures.
6. Burp Suite Plugins for Automation:

• Custom Scan Plugin: A Burp Suite plugin was developed to ingest the custom API descriptor. This allowed for highly targeted, "mini-scans" against specific endpoints, injecting payloads only into designated parameters (e.g., "between positions 10 and 15" of a JSON body), and then analyzing the responses. This approach generated thousands of minute-long scans.
• CI/CD Automation Plugin: Another Burp Suite plugin, coupled with a Maven plugin, enabled full automation from the CI/CD pipeline. This setup could launch Burp Suite Pro, load specific user and project configurations, disable interceptors, configure proxies, load necessary extensions, and then listen on a socket for commands. It would initiate scans against the defined endpoints, aggregate results, upload them as artifacts, and even break the pipeline if high-severity findings were detected.

1. Integration with Test Data and Code Coverage:

• Test Data Seeding: The existing integration tests were leveraged to seed the application's database with necessary test data. A critical step was to ensure this data was not deleted after the integration tests, allowing the subsequent DAST scans to operate against a data-rich environment.
• Code Coverage with Jakoco: To understand what the DAST scans actually tested, Jakoco (a Java code coverage tool) was integrated. By resetting the coverage counter after integration tests and then running DAST, the team could measure the dynamic scanner's byte code instrumentation coverage (e.g., "20% of your application"). This provided invaluable insights into untested components, which could then be cross-referenced with SAST findings to prioritize manual review.

The result of this sophisticated custom solution was a relevant DAST scan executed from CI/CD with clear coverage visibility, automated reports, pipeline breaking capabilities, and a cost that was 20-25 times cheaper than the original enterprise tool (excluding labor costs).

Use Case 2: Modern Cloud-Based Application with Complex Authentication

The second use case involved a much larger Fortune 500 enterprise client with a highly complex, cloud-based application featuring over a thousand REST endpoints. The client had invested in an "appsec platform with DAST capabilities" costing up to $1 million per year. A significant constraint was the lack of access to application logs or the DAST scanner's internal logs due to the consulting engagement nature and the cloud platform's black-box design.

The DAST platform immediately failed to handle the application's incremental authentication flow (e.g., username -> next -> password -> next, similar to Gmail or Okta). The tool's built-in simple form-based authentication was insufficient. Even manually providing session tokens initially failed.

Further issues included:

• Endpoint Discovery Failure: Despite the client providing a Swagger/OpenAPI specification (a massive 70,000 lines of JSON/YAML), the DAST tool struggled to parse it, initially throwing generic "parsing error" messages.
• Combined Authentication Requirements: The application required both security headers and cookies for authentication, but the DAST scanner could only send one or the other, leading to consistent 401 Unauthorized responses. This was identified as a scanner bug.
• Web Application Firewall (WAF) Bypass: The application's WAF dropped requests if the Referer header was missing or incorrect. However, the DAST tool's "deny lists" prevented users from setting or modifying critical headers like Host or Referer.

To overcome these "foggy in the clouds" challenges, Albert-Tóth's team adopted a "sit in the middle" strategy:

1. Reverse Proxy/Middleware: The core of the solution was deploying a reverse proxy or middleware between the DAST tool and the target web application. This was implemented using AWS Lambda and an Application Gateway, logging all requests and responses to a database for full visibility.
2. Headless Browser for Authentication: To handle the incremental authentication, a headless browser automation using Playwright (Selenium was also mentioned as an alternative) was implemented. This script would:

• Log into the identity provider (Okta) through the multi-step flow.
• Extract all necessary session cookies and headers after successful login.

1. Custom Cookie Header Transformation: Since the DAST tool couldn't send both headers and cookies, the extracted cookies were Base64-encoded and sent as a custom X-Custom-Cookie header. The AWS Lambda middleware was then configured to intercept this custom header, decode the Base64 string, and transform it back into standard HTTP Cookie headers before forwarding the request to the target application.
2. Referer Header Injection: The middleware was also configured to automatically inject the correct Referer header into all requests, bypassing the DAST tool's limitation and the WAF's blocking mechanism.
3. DAST API Integration: The enterprise DAST tool's API was used to configure scan templates, injecting the session information obtained from Playwright and directing the scanner to attack the URL of the controlled reverse proxy, rather than the actual target application.

The results were significant: successful authentication of the scanner, bypass of multiple tool limitations, full visibility into requests, and the ability to automate the process into CI/CD pipelines. While this added some costs for AWS infrastructure and development time, it enabled the client to operationalize DAST for critical applications, addressing issues that the multi-million-dollar tool could not handle out-of-the-box.

Demo / Proof of Concept

▶ Watch: Key message: Hack your own tools, solve problems (34:00)

István Albert-Tóth provided a concise yet effective demonstration of the custom authentication and proxy solution developed for the second use case. The demo showcased a simplified flow, illustrating how a headless browser and a custom middleware could successfully manage complex authentication and header manipulation for an enterprise DAST tool.

The demonstration began by showing the Playwright script in action. It initiated a login process against an Okta endpoint, simulating the incremental authentication flow. The headless browser successfully navigated the multi-step login, gathering all relevant session information.

Following the successful login, the demo highlighted how the extracted session cookies were Base64-encoded into a custom X-Custom-Cookie header. This custom header, along with other necessary authentication elements, was then passed to the DAST tool via its API.

The DAST tool was configured to target a controlled interact.sh subdomain, which served as the entry point for the custom AWS Lambda reverse proxy. This proxy then forwarded the requests to the actual target application. The demonstration showed the DAST tool initiating a scan, which was then stopped after a few moments to illustrate the setup without waiting for a full, potentially days-long scan to complete.

Crucially, the demo then shifted to the database where the AWS Lambda middleware logged all requests and responses. The logs clearly displayed the transformed requests, confirming the presence of a valid session ID cookie, the correctly injected Referer header (e.g., defcamp), the User-Agent, and an Authorization bearer token. This visual evidence unequivocally proved that the custom solution successfully authenticated the scanner, bypassed its limitations, and provided full visibility into the traffic, something the original enterprise DAST platform failed to do.

Defensive Implications

▶ Watch: Time taken for solutions and resource challenges (35:00)

The insights from István Albert-Tóth's talk carry profound defensive implications for organizations struggling with application security. The core message for defenders is to never blindly trust DAST tools or vendor claims. A "green check mark" from an expensive scanner does not equate to security. Instead, a proactive, analytical, and "hacker-minded" approach is essential.

1. Validate Tool Coverage Manually and Continuously: Before purchasing or deploying any DAST tool, conduct a thorough Proof of Concept (PoC) against your own complex applications, not just vendor-provided demos. Continuously validate what the tool actually covers by examining application logs, proxying scanner traffic, and using code coverage tools (like Jakoco for Java) to identify untested code paths. Understanding "what you don't know" – the false negatives – is paramount.
2. Prioritize Observability: Instrument your applications for comprehensive logging, metrics, and traces. This allows security teams to monitor DAST scanner activity, identify authentication failures, and debug crawling issues. When using cloud-based DAST, consider deploying a reverse proxy or middleware (e.g., AWS Lambda) to intercept and log all scanner traffic, providing the critical visibility that black-box tools often lack.
3. Embrace Extensibility and Automation: Recognize that off-the-shelf tools will always have limitations. Learn to extend them through plugins, API scripts, and custom integrations. Leverage open-source tools like Burp Suite Pro (with its extensive plugin ecosystem), OWASP ZAP (with its community extensions), or Nuclei (for targeted fuzzing and vulnerability scanning). Automate these custom solutions into your CI/CD pipelines to ensure consistent, scalable, and reproducible security testing.
4. Leverage Developer Skills: Security is not solely the domain of security professionals. Developers possess intimate knowledge of the application's architecture, logic, and potential vulnerabilities. Empower and train developers to think critically about security, extend security tools, and build security automation. As Albert-Tóth states, "developer skills are also security skills."
5. Architect for Testability: Design applications with testability in mind, especially regarding authentication flows and API structures. Ensure that test data can be easily seeded and managed (e.g., preventing automatic cleanup after integration tests) to allow DAST tools to reach vulnerable code paths.
6. Budget for Integration and Staffing: The real cost of ownership for DAST tools extends far beyond licensing. Organizations must budget for the significant effort required for integration, customization, and ongoing maintenance. Tools do not run themselves; dedicated security engineers are needed to configure, validate, analyze, and adapt them.
7. Consider Open-Source Alternatives: For smaller companies or those with limited budgets, open-source DAST solutions and custom frameworks built around tools like Burp Suite or ZAP can offer greater flexibility, transparency, and cost-effectiveness, allowing for scaling security efforts as needs evolve.
8. Don't Fight the Tool – Adapt It: Instead of getting frustrated by tool limitations, approach them with a problem-solving mindset. If a tool cannot set a specific header or handle a complex authentication flow, build middleware or use headless browsers to pre-process requests or manage sessions, effectively adapting the tool to your environment. Put all custom configurations and automation scripts under version control (e.g., GitHub) to prevent rework and ensure maintainability.

By adopting these defensive strategies, organizations can move beyond a superficial understanding of their application security posture and build a robust, adaptable, and genuinely effective DAST program that identifies real vulnerabilities.

Key Takeaways

• Enterprise DAST tools often provide a false sense of security: Despite high costs and vendor promises, they frequently fail to detect critical vulnerabilities in modern, complex applications due to limitations in handling authentication, crawling SPAs, and understanding non-RESTful APIs.
• Visibility is paramount to effective DAST: Organizations must actively gain insight into what DAST tools are actually testing by enabling application debug logs, proxying scanner traffic (e.g., through Burp Suite), and measuring code coverage (e.g., with Jakoco) to identify untested areas.
• Creative problem-solving and customization are essential: Overcoming DAST limitations requires a "hacker mindset," leveraging developer skills, and building custom solutions (e.g., API descriptor generation, Burp Suite plugins, Playwright for authentication, AWS Lambda for middleware) to adapt tools to unique application architectures.
• Complex authentication and header manipulation are common DAST stumbling blocks: Solutions often involve headless browsers for session management and custom middleware to transform requests, inject headers (like Referer), or translate custom cookie formats.
• Don't buy blindly; validate and budget for integration: Always conduct a Proof of Concept (PoC) against your own applications, not just vendor demos. Account for the total cost of ownership (TCO), including licensing, significant integration effort, and ongoing staffing needs, as tools do not run themselves.
• False negatives are more dangerous than false positives: Because DAST tools often don't report what they missed, organizations operate under a dangerous illusion of security. Proactive validation and observability are critical to uncover these hidden risks.

About the Speaker(s)

István Albert-Tóth is the DevSecOps and Product Development Director at VerSprite cybersecurity. With a strong background as a lead developer and security champion at a Fintech Corporation, he brings a unique perspective to application security, having personally experienced the gaps between perceived and actual security. Albert-Tóth is a prominent figure in the open-source security community, serving as the OWASP Curve Guard project leader and contributing to widely used tools such as Nuclei and Subfinder from Project Discovery. His expertise lies in security automation, DAST/SAST tools, CI/CD pipelines, and driving security initiatives within development-focused organizations.