Follow the Trace: How Traditional AppSec Tools Have Failed Us

Follow the Trace: How Traditional AppSec Tools Have Failed Us

Speaker: Kennedy Toomey

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: ~6 minutes

TL;DR

Traditional application security tools — SAST, DAST, WAFs — each carry significant blind spots that produce high false-positive rates and slow development teams. Kennedy Toomey, application security researcher at Datadog, argues that runtime tools, particularly Runtime Application Self-Protection (RASP), use execution traces to provide context that dramatically improves detection accuracy, demonstrating with a live SSRF attack on a Flask application.

Introduction

Application security teams are perpetually outnumbered. Developers ship code; security engineers chase vulnerabilities. The tools meant to bridge that gap — static scanners, dynamic testing suites, web application firewalls — generate noise as often as signal, training developers to ignore alerts and security teams to spend their days triaging false positives rather than eliminating real risk.

Kennedy Toomey's BSidesSF 2025 talk reframes the problem through the lens of runtime observability. If production applications already emit detailed execution traces for performance monitoring, why aren't those same traces being used for security? Toomey's answer, demonstrated on a purpose-built vulnerable Flask app, is that runtime tools have a fundamental accuracy advantage over every earlier generation of AppSec tooling because they know what the code actually did, not just what it looks like it might do.

▶ Watch: What traces are and why they matter for security (02:00)

The Landscape of Traditional AppSec Tools

Toomey surveys the three dominant categories of application security testing before explaining why each falls short:

Static Application Security Testing (SAST) scans non-running code via pattern matching against known vulnerability signatures. Its major advantage is early detection — it works without executing anything. Its major weakness is a high false-positive rate: SAST tools frequently flag code that matches a dangerous pattern but is, in context, safe. Developers receive alerts they cannot act on, creating friction and alert fatigue.

Dynamic Application Security Testing (DAST) simulates malicious traffic against a running application and observes the responses. It surfaces real runtime behavior but lacks deep application visibility — it cannot see logic flaws or understand the internal data flow, because it operates entirely from the outside. DAST also requires a running application, which pushes it to later SDLC stages.

Interactive Application Security Testing (IAST) instruments the running application to track data flow from input to sensitive function call. It follows four steps: track input sources, monitor data flow through the application, identify potentially vulnerable code points, and confirm whether suspicious input actually reached a dangerous sink. IAST reduces false positives significantly by using runtime context, but requires the application to be running and cannot assist with early-phase development.

On the protection side, Web Application Firewalls (WAFs) sit outside the application, inspecting inbound traffic for suspicious payloads. They excel at DDoS mitigation and threat landscape visibility but fail at exploit prevention: a WAF has no insight into whether a matched payload would actually exploit a real vulnerability in the target application. It can only enforce rules.

▶ Watch: SAST, DAST, and IAST compared (06:00)

Why Runtime Context Changes Everything

Toomey cites a figure from Datadog's State of DevSecOps report: only 18 percent of critical vulnerabilities in production applications actually require urgent attention when runtime context is applied. The remaining 82 percent can be deprioritized based on factors like whether the service is running in production, whether the vulnerable code path is reachable, and whether the service is being actively targeted. Traditional scanners treat all 100 percent as equally critical.

The mechanism behind this improvement is the combination of CVE data with runtime telemetry. A runtime tool that observes actual execution can adjust CVSS scores dynamically — lowering priority for services not in production, or for vulnerabilities in code paths that are never invoked. This transforms vulnerability management from a fire-hose of alerts into a manageable, prioritized queue.

▶ Watch: Runtime context and the 18% prioritization finding (14:00)

Live Demo: SSRF Attack on a Flask Application

The practical center of the talk is a walkthrough using a deliberately vulnerable Flask application with a login page, registration, and blog post functionality. The application accepts a user-supplied image URL when creating a blog post — a classic SSRF (Server-Side Request Forgery) entry point.

Toomey shows the baseline flame graph for a normal login request: a parent span representing the browser request, a Flask dispatch event, a tiny sliver of SQLite activity (about 0.2% of execution time), and cleanup. Understanding this baseline is the foundation for detecting anomalies.

With the SSRF payload injected — a URL pointing to an internal service rather than a real image — the flame graph changes visibly. Orange spans appear alongside the green spans, indicating execution that left the original service boundary. The attack succeeded: the application made a request to an internal location, the flame graph shows the pivot to a "get flag" service, and the full execution path is captured in the trace.

Toomey then enables the RASP and replays the identical attack. The flame graph returns to all-green. The orange spans are gone. The RASP intercepted the outbound call before it executed, blocking the exploit not by pattern-matching the request payload but by observing that the runtime execution was attempting to access a forbidden location.

▶ Watch: SSRF attack demonstration and RASP blocking (18:01)

RASP: How It Works and How to Configure It

Runtime Application Self-Protection embeds directly into the application — typically via an agent or library loaded at startup. Unlike a WAF, it operates on the already-interpreted request and has full visibility into application execution: function calls, database queries, file access, outbound network requests. This gives it an accuracy advantage over WAFs for exploit prevention, though it offers no DDoS protection (the request has already passed through the server by the time RASP sees it).

The RASP process, as Toomey describes it:

1. Monitor behavior — observe code execution, user inputs, and application context
2. Analyze in real time — intercept calls to sensitive functions
3. Respond — block the operation, alert, or both

Configuration requires several decisions: deployment mode (monitor-only vs. active blocking), what to block (by vulnerability type, all vulnerabilities, or by attacker identity/IP), which detections to use (out-of-the-box or custom rules), allow-listing for legitimate DAST scans and pen tests, and alerting thresholds.

Toomey's practical advice for teams adopting RASP: start in monitoring mode. Identify what attacks are actually happening before creating blocking rules. This prevents the tool from interfering with legitimate traffic and establishes a baseline for tuning.

▶ Watch: RASP configuration and deployment guidance (20:01)

Notable Quotes

"Why can't we use all of this information that we already have in security? Why can't application security people actually take advantage of this data?" — Kennedy Toomey at 06:00

"Eighteen percent — we only need to worry about eighteen percent of our critical vulnerabilities." — Kennedy Toomey at 14:00, on runtime-informed vulnerability prioritization

"Context really makes all the difference. It puts less stress on developers, which is huge. There are less false positives for developers to investigate, and they don't have to spend time fixing vulnerabilities if they don't have to." — Kennedy Toomey at 22:01

Key Takeaways

• Traditional AppSec tools are accurate in different ways at different SDLC stages. SAST catches issues early but generates false positives; DAST validates runtime behavior but misses logic flaws; IAST is more precise but arrives late in the cycle. None replaces the others.
• WAFs lack exploit-prevention capability. A WAF blocks on rules without knowing whether a payload would succeed against the actual application — limiting its utility for stopping real exploitation.
• Runtime context reduces critical vulnerability backlog by ~82%. Only 18% of critical CVEs require immediate attention when evaluated against runtime data, according to the Datadog State of DevSecOps report.
• RASP blocks exploits at execution time, not input time. By intercepting sensitive function calls during execution, RASP can detect and block attacks that bypass every earlier defense — as demonstrated by the SSRF example where an identical payload was blocked by the RASP but not the WAF.
• Start runtime tools in monitor mode. Deploying RASP in blocking mode without a tuning period risks interfering with legitimate traffic. Monitoring first establishes the baseline needed to write effective rules.