LLM-Driven Reasoning for Automated Vulnerability Discovery Behind Hall-of-Fame

LLM-Driven Reasoning for Automated Vulnerability Discovery Behind Hall-of-Fame

Speakers: Cheng Dai, Security Researcher; Yifei (collaborator), Undergraduate Researcher

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=WVjnipkKp4U

Reading time: 8 min

Type: Briefing

TL;DR

Cheng Dai and undergraduate collaborator Yifei built MinWhisper, an LLM-based pipeline that autonomously finds vulnerabilities in Samsung phones by decompiling stripped ARM64 binaries, reconstructing data structures, and running taint-style analysis with reasoning models. The tool contributed to the researchers earning recognition in Samsung's Mobile Security Hall of Fame in 2024 and has since found multiple new confirmed vulnerabilities in the SecVideoEngineService — including CVE-2024-34587 and SVE-2024-1490.

Introduction

The proposition sounds straightforward: point a large language model at a binary, ask it to find bugs. The reality of applying LLMs to real-world vulnerability discovery against stripped, compiled firmware is significantly more involved — and significantly more productive than most practitioners might expect.

Cheng Dai and his collaborator Yifei arrived at Black Hat with a concrete existence proof. Their tool, MinWhisper, targets Samsung's SecVideoEngineService — a network-accessible process that runs by default on Samsung phones, handles RTCP/RTP packet parsing, and can be triggered remotely without user interaction. That attack surface profile makes bugs in the service highly valuable, and finding them manually in stripped ARM64 binaries is a multi-week undertaking per vulnerability. MinWhisper compresses that process by replacing the most time-consuming human work — data structure reconstruction, call graph tracing, and taint analysis — with agent-driven LLM reasoning.

The Problem With Naïve LLM-Assisted Auditing

▶ Watch: Vulnerability Motivation and CVE Case Study (02:00)

Dai opened with the real-world bug that motivated the project: CVE-2024-34587, a heap overflow in SecVideoEngineService's RTCP APP packet parser. The vulnerable function performs a memcpy where the length parameter is a 16-bit attacker-controlled field (maximum value 0xFFFF), but both source and destination buffers are much smaller. The bug is conceptually simple — the manual work is verifying that the field is truly attacker-controlled all the way from the network receive function through several layers of callbacks and data structures.

Dai tested the obvious approach: paste the vulnerable function into a chatbot and ask "Is it vulnerable and why?" The model returned three findings, one of which matched the real bug — but qualified it with "if the app data list is large enough, memcpy may overflow." That kind of probabilistic hedging is useless in an audit context. When reviewing hundreds of functions, you need a firm yes or no, not a conditional that still requires manual triage.

The solution Dai identified is systematic: give the model precise preconditions — exact buffer sizes, the attacker-controlled field's type and range, the calling context — and it acts like a constraint solver rather than a guessing engine. With that context, the model flags the overflow with high confidence and explains exactly why.

MinWhisper's Architecture

▶ Watch: Pipeline Architecture Overview (06:02)

MinWhisper's pipeline consists of a core controller, environment interfaces for interacting with binaries and a physical Samsung device, and a set of LLM agents that handle specific tasks. The human operator makes exactly two decisions: selecting the target process at the start and verifying confirmed findings at the end. Everything between is automated.

The comparison Dai drew to Google Project Zero's Project Naptime is apt but highlights a key difference in scope. Project Naptime works at the source code level; MinWhisper operates on six stripped ARM64 binaries with no symbols, no type information, and no function names beyond those that survived stripping. That forces a mandatory preprocessing stage that source-code-based systems can skip.

The toolchain feeds binaries into a decompiler equipped with custom Ada plugins, which produce pseudo-C and build a global call graph. A Frida agent is attached to a real Samsung device to resolve indirect calls at runtime — static analysis breaks on virtual dispatch and function pointers, so the tool hooks every indirect call and reads the actual jump target from the register. Once the call graph is complete and indirect calls are resolved, the LLM agents take over.

Data Structure Reconstruction: The Core Innovation

▶ Watch: Data Structure Reconstruction in Detail (12:02)

The step that makes the difference between useful and useless LLM vulnerability analysis is data structure reconstruction. Without knowing the exact size of each buffer and the type of each field, any buffer overflow analysis will produce either false positives (model guesses the buffer is large enough) or false negatives (model assumes the field is bounded). Reconstruction gives the model the ground truth it needs to reason with high confidence.

The Ada plugin tags every allocation site (malloc, calloc, class constructors) and stores the decompiled function to a database. It also parses the AST at each allocation site to build an initial layout based on how the code initializes fields. The data structure analyzer agent then scans the codebase for functions that access the same struct, matching by field type, name, offset, or related class name.

Dai walked through a concrete example using the CTransportManager class from CVE-2024-34587. The agent starts with the constructor, identifies fields from offset accesses (this + 0x303, this + 0x1072), normalizes them based on type information, and progressively enriches the layout by following callers and callees of CTransportManager methods. When it encounters StartRtcpSend, it adds two new fields. When it follows RtpRtpCreate, it infers the structure of a related class and adds three more. When it sees CreateBuffer accessing this + 0x303, it adds subfields for that embedded object.

By the time the vulnerability analysis agent begins, the reconstructed data structures are expected to be as complete as if the researchers had access to the original source definitions.

Vulnerability Analysis and the Model Router

▶ Watch: Vulnerability Analysis Agent and Bug Findings (18:05)

With full data structure context in place, the vulnerability analysis agent performs inter-procedural taint analysis from packet receive functions to parse sinks, threading the argument list, buffer sizes, attacker-controlled field tags, and reconstructed structures through each function call as preconditions. The model is given a bug pattern description (in the CVE-2024-34587 case study, a simple out-of-bounds index pattern) and asked to flag any instances with high confidence.

One practical engineering challenge Dai disclosed: LLMs do not reliably produce valid JSON when asked to structure their output. They drop quotes, fail to escape characters, or truncate strings. The fix is a model router — a lightweight "judge" model (GPT-4o Mini or DeepSeek V3) that receives both the malformed JSON and an error report and repairs the output. The heavy reasoning work goes to OpenAI o3 or DeepSeek R1; format repair and lightweight tasks go to cheaper models. This hybrid routing reduces cost without compromising accuracy on the critical analysis steps.

Confidence thresholds matter significantly. Through validation against confirmed bugs, the team found that any finding below 80% confidence produces a high false-positive rate; findings at or above 80% confidence were, in their experiments, all confirmed true positives. The tool's output includes a confidence score per finding, a clear bug description, and a full reasoning trace for rapid human verification.

Benchmarking Results

▶ Watch: Bug Findings and Model Benchmarks (20:05)

Dai benchmarked four model families across the MinWhisper pipeline. The key findings:

• OpenAI o3 found all five bugs with high confidence in every run and completed the job in a reasonable timeframe. It is the recommended model for full runs.
• OpenAI o4-mini found all bugs most of the time but occasionally missed one or two on individual runs; cost is significantly lower than o3.
• DeepSeek R1 found all bugs but took up to twenty-one hours on the same job — a significant throughput constraint for iterative research. Cost per token is competitive.
• ChatGPT Plus (GPT-4o-class) found bugs at lower confidence and is not recommended for primary analysis.

The five confirmed bugs found by MinWhisper in the SecVideoEngineService include CVE-2024-34587 (the original motivating bug) and SVE-2024-1490, both involving out-of-bounds writes driven by unchecked counters that increment across function calls without bounds checking. The pattern — a counter that grows past a fixed-size allocation across multiple packet-processing calls — is simple but difficult to catch manually without reconstructed data structure sizes.

Dai was candid: "Such bug patterns might be the simplest you have ever seen, but I have had." His point is that the value of the tool is not finding novel vulnerability classes but finding all instances of known-dangerous patterns across a large, poorly documented binary surface faster than any human team could.

Notable Quotes

"When you are auditing hundreds of functions, you need firm yes or no answers, not maybes that still need further triage."

— Cheng Dai [[▶ 06:02]](https://www.youtube.com/watch?v=WVjnipkKp4U&t=362s)

"Give large language models enough and clear preconditions, then they act like a constraint solver — take conditions and concrete data, then output precise answers."

— Cheng Dai [[▶ 06:02]](https://www.youtube.com/watch?v=WVjnipkKp4U&t=362s)

"Will large language models replace security researchers? My answer is no. On the contrary, they take the boring parts, and we get much more efficient than before."

— Cheng Dai [[▶ 22:05]](https://www.youtube.com/watch?v=WVjnipkKp4U&t=1325s)

"The tool is still reporting new bugs, and I have to look at some tonight."

— Cheng Dai [[▶ 22:05]](https://www.youtube.com/watch?v=WVjnipkKp4U&t=1325s)

Key Takeaways

• Data structure reconstruction is the critical enabler for LLM-based binary vulnerability analysis. Without precise buffer sizes and field types, reasoning models produce either false positives or hedged answers that require the same manual triage as unassisted review.
• Frida-based runtime call resolution is necessary for accurate static analysis of stripped binaries. Virtual dispatch and function pointers break static call graphs; hooking indirect calls at runtime fills those gaps and enables correct inter-procedural analysis.
• An 80% confidence threshold is a practical filter for true positives. Below that threshold, false positives proliferate. Above it, findings were 100% confirmed true positives in the team's experiments — enabling very lean human verification workflows.
• Model routing reduces cost without sacrificing accuracy. Using heavy reasoning models (o3, DeepSeek R1) for vulnerability analysis and lightweight models (o4-mini, DeepSeek V3) for format repair and supporting tasks makes the pipeline economically viable for independent researchers.
• The approach generalizes beyond Samsung. Any network-accessible service running on ARM64 firmware with a meaningful attack surface is a candidate for the same pipeline — the architecture is target-agnostic once the decompilation and runtime instrumentation stages are adapted.

Slides: No slides PDF is available for this session.