How Tree-of-AST Redefines the Boundaries of Dataflow Analysis

Thinking Outside the Sink: How Tree-of-AST Redefines the Boundaries of Dataflow Analysis

Speakers: Sasha (Alexander) Groshev, Security Researcher; Patrick (Kai) Reschke, Founder, Palano

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=VNBEoLE_bGA

Reading Time: ~7 minutes

Type: Briefing

TL;DR

Two researchers — one a recent high school graduate, the other a sixteen-year-old founder — developed Tree-of-AST, an LLM-powered dataflow analysis engine that applies Tree-of-Thoughts reasoning to taint analysis by traversing abstract syntax trees backward from sink to source. The system rediscovered known CVEs in large ML framework codebases and found new zero-days that CodeQL entirely missed, at a fraction of the false-positive rate.

Introduction

Vulnerability research in large open-source codebases is a grinding, human-intensive process. A researcher stares at millions of lines of code, manually traces how untrusted user input flows through functions and across files, and tries to determine whether any path leads to a dangerous sink like eval, exec, or a file write. Tools like CodeQL help, but traditional static analysis suffers from path explosion, high false positives, and an inability to reason across module boundaries the way a human researcher would.

Sasha Groshev and Patrick Reschke — introduced by their shared high school computer science teacher — came to Black Hat 2025 with a research project that applies cognitive reasoning frameworks borrowed from AI to the problem of automated vulnerability discovery. Their system, Tree-of-AST, is not a simple LLM wrapper that dumps code into a prompt and asks "is there a bug?" It is a structured traversal algorithm that mimics how an experienced researcher thinks, encoded as a stateful graph exploration.

The Backstory: From "Game of 24" to Security Research

▶ Watch: Origin of the Approach (02:00)

The intellectual foundation comes from a 2023 Google DeepMind paper: Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Where chain-of-thought prompting forces a model down a single linear reasoning path, Tree-of-Thoughts introduces branching, backtracking, lookahead, and voting — emulating the way humans explore multiple candidate strategies before committing to one.

Reschke's entry point was a bug-hunting program at Hunter, an open-source bug hunting platform for ML projects like Transformers and TensorFlow. The challenge was enormous: these codebases are built on C++ bindings, involve cross-file symbolic relationships, and require reasoning about data flow through hundreds of interdependent modules. Standard LLM approaches — send a file to a model and ask if it's vulnerable — fail badly on such complexity. False positives are overwhelming, and models miss deep, multi-hop vulnerabilities entirely.

The key insight came from recognizing a structural similarity. The "Game of 24" (combine four numbers to reach 24 using arithmetic operations) is the canonical benchmark for Tree-of-Thoughts because it requires exploring a combinatorial space where early decisions constrain later options. Taint analysis is precisely this kind of problem: tracing data from a sink backward through a tree of possible call paths to determine whether any path reaches a user-controlled source.

The Algorithm: Sink-to-Source Traversal with LLM Voting

▶ Watch: The Tree-of-AST Algorithm (08:00)

Tree-of-AST inverts the conventional direction of taint analysis. Instead of starting from sources (user inputs) and following data forward — a process that generates enormous numbers of low-value paths — it starts from high-priority sinks (dangerous functions) and traces backward to find any source of untrusted input.

Three reasons make sink-first analysis superior:

1. Sinks are easy to enumerate. Dangerous functions are literal, predictable, and catchable with AST pattern matching or regex. Sources are diffuse and context-dependent.
2. Sink-first eliminates low-severity findings. If you can backtrack a dangerous sink to a user-controlled source, you have a confirmed, exploitable vulnerability. Tracking forward from sources generates many paths that never reach anything dangerous.
3. Context relevance. Starting at the sink provides immediate semantic context about what kind of vulnerability exists, which guides path selection at each step.

The traversal works as follows: beginning at a confirmed dangerous sink node in the AST, the LLM votes on which adjacent predecessor nodes are most likely to lead toward a user-controlled input. It selects a path, continues voting step-by-step, and can backtrack to previously visited states when a path dead-ends. This is the "stateful" property Reschke calls state recovery — the algorithm maintains a record of explored branches and can rewind, a natural capability of tree structures.

Once a complete sink-to-source path is elected, the algorithm reverses direction — traversing source-to-sink — and generates a payload that satisfies all constraints and conditions along that path. This end-to-end automation, from vulnerability discovery to working proof-of-concept, distinguishes Tree-of-AST from tools that only report a potential issue.

Implementation: Stack Graphs, TreeSitter, and Parameter-Based Tainting

▶ Watch: Technical Implementation (14:00)

Groshev handled the engineering challenges, starting with the question of how to represent code internally in a way that supports cross-file symbol resolution, language portability, and efficient querying.

After experimenting with raw ASTs — and discovering their limitations (language-specific, no cross-file analysis, complex scope resolution) — the team settled on stack graphs, a data structure from the developer tools ecosystem built on top of TreeSitter. TreeSitter is a parsing library that supports many languages, which gave the project multi-language capability without rewriting the core logic. Stack graphs handle import resolution, name resolution, and visibility resolution automatically, building a unified graph that can answer questions like "where is this function defined, and where is it called?"

A practical setback: the team found a closed-source stack graph library that worked well until projects exceeded 200 files, at which point it would freeze indefinitely. Unable to fix the underlying bug (no source access), they implemented a chunking strategy — splitting large codebases into overlapping segments so that no single invocation exceeded the threshold.

For the taint analysis logic itself, they use parameter-based tainting, which tracks whether tainted data can reach a sink through any call path, including via caller functions. This outperforms both argument tracking and traditional sink-trace approaches, catching the three vulnerability code patterns that other methods miss. LLMs are used via LangChain for input source detection, path selection, and backtracking decisions. A higher-capability LLM is used in a final pass to re-weight confidence scores and reduce false positives.

Results: CVE Rediscovery and Performance vs. CodeQL

▶ Watch: Results and CVE Rediscovery (26:00)

The team validated Tree-of-AST against known CVEs in popular ML frameworks — the specific CVEs were not disclosed during the presentation, in keeping with responsible disclosure norms for any new findings. They confirmed the system successfully rediscovered vulnerabilities that had been documented in codebases within the Hunter platform's scope.

Performance comparisons with CodeQL — the industry-standard static analysis tool — showed that Tree-of-AST consistently outperformed it on large, multi-file projects, both in terms of finding vulnerabilities CodeQL missed and in producing fewer false positives. The gap was attributed to two structural advantages: Tree-of-AST's ability to reason across file boundaries via stack graphs, and its use of LLM voting for path prioritization rather than exhaustive path enumeration.

Groshev acknowledged measurement challenges: there is no standard benchmark dataset of large projects with fully mapped vulnerabilities, meaning manual verification is required for any new finding — a time-consuming process that limited the scale of quantitative evaluation presented at Black Hat.

Notable Quotes

"Sink to source is actually the Game of 24 for taint analysis."

— Patrick Reschke ▶ 10:00

"We never send any big amounts of code base into the LM. We do not process a whole program at one time. We take small pieces, and we deduct dynamically."

— Sasha Groshev ▶ 24:00

"Tree-of-AST doesn't stop at vulnerability discovery. It goes all the way to payload generation."

— Patrick Reschke ▶ 12:00

"If you have a cybersecurity task that involves human work, you need to separate the strategic decision part and the manual work part. Automate the manual work deterministically and delegate strategic decision-making to LMs."

— Sasha Groshev ▶ 26:00

Key Takeaways

• Start from sinks, not sources. Sink-first traversal eliminates enormous quantities of low-value paths and provides immediate exploit-relevant context at every reasoning step.
• Tree-of-Thoughts reasoning maps directly onto taint analysis. The stateful, backtracking, voting-based structure of Tree-of-Thoughts is a natural fit for navigating AST graphs toward vulnerability confirmation.
• Parameter-based tainting outperforms argument tracking. Tracking whether tainted data reaches a sink through any caller path, not just direct argument passing, catches vulnerability patterns that standard approaches miss.
• LLMs should handle strategic decisions; deterministic code should handle mechanical analysis. Feeding entire codebases to LLMs produces hallucinations and misses multi-hop paths; structured graph traversal with LLM-guided node selection is far more reliable.
• Cross-file symbol resolution is the hard problem in code analysis. Stack graphs built on TreeSitter solve scope resolution, import handling, and multi-language support in a unified framework that raw ASTs cannot match.

Slides: No slide PDF was available for this talk.