Your Teammate Isn't Human: Mixing Decompilation and AI for Modern Reverse Engineering

Your Teammate Isn't Human: Mixing Decompilation and AI for Modern Reverse Engineering

Speakers: Zion Leonahenahe Basque, Ph.D. Student, Arizona State University

Conference: HITCON CMT

YouTube: https://www.youtube.com/watch?v=HbrebQiFLDs

Overview

In an era where Artificial Intelligence is rapidly permeating all facets of technology, its integration into cybersecurity, particularly in the demanding field of reverse engineering, is becoming increasingly critical. This talk, "Your Teammate Isn't Human: Mixing Decompilation and AI for Modern Reverse Engineering," delivered by Zion Leonahenahe Basque at HITCON CMT, delves into the transformative potential of blending large language models (LLMs) with traditional decompilation tools. Basque, a Ph.D. student at Arizona State University and a seasoned hacker with the Shellphish CTF team, introduces DAILA (Decompiler Artificial Intelligence Language Assistant), an innovative framework designed to address long-standing limitations in reverse engineering workflows.

The core premise of the talk is that while decompilers are indispensable for converting opaque assembly code back into human-readable pseudo-C, they inherently suffer from a loss of crucial contextual information such as meaningful names, precise types, and original control flow. Existing AI-driven approaches have attempted to mitigate these issues but often fall short due to token limitations, lack of interactivity, or decompiler-specific implementations. DAILA emerges as a sophisticated solution, offering a decompiler-agnostic, scriptable, and highly interactive pipeline that allows AI agents to dynamically assist reverse engineers, adapting suggestions as new context is uncovered. This work is particularly timely given initiatives like the DARPA AI Cyber Challenge, which underscores the significant investment and interest in advancing AI's role in cybersecurity, with a first-place prize of $4 million highlighting the stakes involved.

Background

▶ Watch: Analyzing common flaws in decompiled code (6:00)

Reverse engineering a binary involves transforming compiled machine code (assembly) back into a higher-level, more comprehensible representation, typically pseudo-C. This process is fundamental for tasks such as vulnerability analysis, malware dissection, and understanding proprietary software. Tools like IDA Pro, Ghidra, Binary Ninja, and Angr are the workhorses of this domain, providing decompilation capabilities that significantly reduce the cognitive load of analyzing raw assembly.

However, decompilation is inherently a lossy process. When source code is compiled, much of the original semantic information—such as meaningful function and variable names, precise data types, comments, and the programmer's intended control flow—is discarded or obscured for optimization and efficiency. As Basque illustrates, this is akin to taking a high-resolution image and pixelating it; while the general form remains, significant detail is lost. The decompiler then attempts to reverse this pixelation, often resulting in generic names like sub_4444 for functions, v1 or a3 for variables, and __int64 for types, alongside potentially distorted control flow structures. Recovering this lost information traditionally relies heavily on human intuition, experience, and painstaking manual analysis.

Previous attempts to leverage AI, especially Large Language Models (LLMs) like ChatGPT, for reverse engineering have faced several limitations. A common, rudimentary approach involves copying decompiled pseudo-C into an LLM and asking for summaries or name suggestions. This method is severely constrained by the LLM's token context window, typically limiting input to around 8,000 tokens, or roughly 500 lines of code. For larger functions, this approach is infeasible. Furthermore, these interactions are largely non-interactive and one-shot; the LLM provides a response, but it doesn't dynamically update its suggestions as the human reverse engineer gains more context or makes changes within the decompiler.

More structured AI integrations, such as Jepetto for IDA and GPT-WPR for Ghidra, have emerged. Jepetto offered a human-initiated, right-click summary and variable renaming feature, but it was decompiler-specific, couldn't handle large functions, and lacked deep, pervasive interaction. GPT-WPR aimed for "whole program reverse engineering" by iterating through functions and generating summaries, but it didn't push changes back into the decompiler, nor did it allow for iterative updates based on user feedback. These prior approaches highlighted a critical gap: the need for an AI teammate that could not only understand the context but also adapt and learn from the ongoing reverse engineering process, much like a human collaborator.

Key Findings

▶ Watch: Presenting Diala: decompiler-agnostic, interactive AI assistant (14:45)

The central contribution of Zion Basque's talk is the introduction of DAILA (Decompiler Artificial Intelligence Language Assistant), an innovative framework designed to bridge the gap between powerful AI capabilities and the practical demands of modern reverse engineering. DAILA represents a significant step forward by addressing the core limitations of both traditional decompilers and previous AI integrations.

The key findings and novelties of DAILA can be summarized as follows:

• Dynamic and Context-Aware Updates: Unlike prior static AI approaches, DAILA enables AI suggestions to dynamically update as the reverse engineer makes changes within the decompiler. This real-time feedback loop allows the AI agent to evolve its understanding of the binary, mirroring the adaptive learning process of a human teammate.
• Direct In-Decompiler Questioning: Users can ask specific questions directly within the decompiler's interface using a simple Q<question> syntax. The AI agent, leveraging its context-gathering mechanisms, provides targeted answers, significantly enhancing interactive exploration and understanding of obscure code segments.
• Support for Large Functions: DAILA overcomes the restrictive token limits of LLMs by employing sophisticated context gathering techniques. This allows the AI to focus on relevant sections (e.g., targeted lines, caller uses) of even very large functions (exceeding 1,000 lines), making AI assistance viable for complex binaries previously out of reach for LLM-based tools.
• Decompiler Agnosticism: A crucial design principle of DAILA is its compatibility across multiple major decompilers, including IDA Pro, Ghidra, Binary Ninja, and Angr. This broad support eliminates the need for decompiler-specific AI solutions, making the technology accessible to a wider range of reverse engineers regardless of their preferred toolchain.
• Scriptability Across the Entire Binary: The framework provides a Python scripting interface, enabling users to automate AI-driven tasks across an entire binary. This allows for batch processing of functions, custom prompt sequences, and integration into larger automated analysis pipelines.
• Interactive and Comprehensive Artifact Changes: DAILA facilitates the modification of a wide array of decompilation artifacts, including function names, variable names, data types, and even struct definitions (with specific context). These changes are not one-way; they feed back into the AI's understanding, influencing subsequent suggestions.
• Open-Source Availability: The project is open-source, hosted on GitHub at mahalos/DAILA, encouraging community contributions and further research into AI-assisted reverse engineering.

These findings collectively demonstrate that AI can move beyond simple summarization to become an integrated, intelligent, and adaptive partner in the intricate process of binary analysis.

Technical Deep Dive

▶ Watch: How Diala's interactive pipeline enhances reverse engineering (16:00)

The technical architecture of DAILA is designed around a flexible, interactive pipeline that integrates various components to deliver its novel capabilities. At its core, DAILA acts as an intermediary layer between the decompiler and the AI agent, processing information to maximize relevance and minimize the impact of LLM limitations.

The pipeline initiates with the user's chosen decompiler (IDA, Ghidra, etc.) providing a function's decompilation. This raw output then enters the DAILA working layer, which performs three critical operations:

1. Type Lifting and Information Lifting: Decompilers often use internal or generic type representations (e.g., __int64). DAILA first lifts these to standard C types (e.g., long long), making the code more understandable for the AI agent. This step also involves extracting other pertinent information from the decompiler's context.
2. Context Gathering: This is a novel and crucial aspect of DAILA. To overcome LLM token limitations and focus the AI, DAILA employs intelligent strategies to curate the information presented to the LLM:

• Targeted Lines: Instead of feeding an entire, potentially massive function, the system can be instructed to focus on a specific range of lines. This "blindfolding" technique directs the AI's attention to the most relevant code snippet, improving accuracy and efficiency.
• Caller Uses: A classic binary analysis technique, DAILA gathers context from functions that call the current function. Understanding how a function is used by its callers provides vital clues about its purpose and intended behavior.
• BinSync Integration: The entire backend of DAILA relies on BinSync, a synchronization framework used by the Shellphish team to share data across decompilers and users. BinSync records all changes (both human and AI) as Git-like commits. This commit-based system is fundamental to DAILA's interactivity, as every commit triggers a callback, notifying the AI agent of new information or modifications. This allows the AI to dynamically update its suggestions in response to user actions.

1. Prompting: DAILA includes six pre-written prompts: function summary, variable renaming, function renaming, retyping, source identification, and bug detection. These are easily extensible, allowing users to define custom prompts. The effectiveness of these prompts varies, with function summary, variable renaming, function renaming, and question response proving to be the most powerful. It's important to note that the order of execution for prompts matters significantly; for instance, variable renaming performed before summarization often yields more intelligent summaries.

After processing through the DAILA working layer, the contextualized information is sent to the chosen AI agent (e.g., GPT-4, or even local models like VAR model or Llama). The AI agent then generates suggestions, which are piped back through DAILA (with types being "lifted down" to the decompiler's specific format) and applied to the decompiler. This entire cycle is dynamic: if a user subsequently modifies a type (e.g., changing getChar to getInt), BinSync registers the change, triggers a callback, and the AI agent can update its suggestions accordingly.

Interactivity Features:

• Inline Questioning: The Q<question> syntax allows direct questions within the decompiler, providing immediate, context-specific answers. For generic questions not requiring decompilation context, ?X<question> can be used.
• Updates on Change: The BinSync backend ensures that any user modification (e.g., changing a variable type) triggers a recalculation by the AI, leading to updated and more accurate suggestions. For example, if a function was initially named getChar and the user changes its return type to int, the AI might suggest renaming it to getInt.
• Multi-User Environment: BinSync's capabilities extend to supporting multiple users, including various AI agents and human collaborators, all contributing to the same project. This creates a collaborative environment where distinguishing between human and AI contributions can become challenging, reinforcing the "teammate isn't human" motif.

Limitations and Considerations:

Basque highlights several limitations of current AI models in this context:

• Vulnerability Detection: LLMs are "super duper bad" at vulnerability detection. They tend to flag almost every function as having a bug, rendering this feature impractical and distracting.
• Retyping Dangers: While AI can suggest types, aggressively retyping variables can sometimes lead to incorrect inferences (e.g., char becoming int[186]), potentially corrupting the decompiler's stack and causing crashes.
• Source Identification: Identifying the original source code of a function has about a 30% accuracy rate, making it a "so-so" feature.
• Symbolic Reasoning: AI struggles with symbolic operations, such as interpreting p + 0x30 without explicit struct definitions, which is a common challenge for humans as well.
• Decompiler Quality Matters: The quality and clarity of the decompiler's pseudo-C output significantly influence the AI's ability to make correct judgments.

Despite these limitations, DAILA's architectural design provides a robust and extensible framework for integrating AI into reverse engineering, focusing on controlled context and dynamic interaction to maximize utility.

Demo / Proof of Concept

▶ Watch: "Your teammate isn't human": AI vs. human code analysis (34:30)

Zion Basque provided several compelling live demonstrations (with pre-recorded backups) using IDA Pro to showcase DAILA's capabilities.

1. Setting Up an AI User: The demo began by showing how to add an AI user, specifically GPT-4, within the DAILA panel in IDA. As the AI agent processed functions in the background, changes were pushed to a BinSync project, visible as Git-like commits. This illustrated the live, persistent nature of AI contributions, where suggestions (like function renames or comments) appeared as if another user (e.g., "Allan") was collaborating.

1. Scaling to Large Functions: A significant challenge for previous LLM integrations was their inability to handle functions exceeding the token limit (around 500 lines). Basque demonstrated DAILA processing a function over a thousand lines long. By intelligently minimizing context, DAILA enabled GPT-4 to successfully summarize and rename variables and the function itself, a task impossible with simple copy-pasting. This highlighted DAILA's effectiveness in managing context for large codebases.

1. Automated Whole-Binary Analysis: The speaker then showed how DAILA could be scripted to run across an entire binary, comprising 50-60 functions. While some renames were generic ("memory data manipulation"), others were remarkably accurate, such as identifying a function as md5_hash_imp and correctly describing its purpose. This demonstrated the potential for large-scale, automated assistance, even if some results require human refinement.

1. Contextual Questioning and Exploitation: This was the most impactful part of the demo, showcasing DAILA's interactive question-answering capabilities.

• Basque navigated to a complex section within a large main function from a recent CTF challenge ("Chicago"). This specific code segment implemented an unknown algorithm.
• Instead of summarizing the entire function, Basque used the Q<What algorithm is this code implementing?> prompt, targeting only the relevant 20 lines of code.
• The AI agent (via a qa1 BinSync user) correctly identified the algorithm as the Luhn algorithm, commonly used for credit card validation. This emphasized the critical importance of providing precise context to the AI.
• Leveraging this information, Basque then asked a generic question (?X<how to make a correct solution for Lun algorithm>) to the AI, which provided a valid 10-digit number.
• Inputting this AI-generated number into the "Chicago" challenge successfully yielded the flag, demonstrating a direct path from AI-assisted understanding to practical exploitation. Notably, asking the same question without targeted context (or in Ghidra) did not yield the correct algorithm, underscoring the value of DAILA's context management and decompiler integration.

1. Human vs. AI: Basque concluded the demo with an intriguing "who is human" poll, presenting two decompiled code snippets for a bytecode compiler/parser. One was analyzed by "Paul," a highly skilled human reverser, and the other by ChatGPT. The AI's analysis was strikingly similar to Paul's, accurately identifying the bytecode compiler's components and stack variables. This visually reinforced the talk's title, illustrating how AI's contributions can become indistinguishable from human work in a collaborative reverse engineering environment.

These demonstrations collectively proved DAILA's ability to provide dynamic, context-sensitive, and scalable AI assistance, transforming the reverse engineering workflow.

Defensive Implications

▶ Watch: Q&A: AI's weaknesses in reverse engineering and future improvements (48:00)

The advent of tools like DAILA carries significant implications for defensive cybersecurity postures, offering both powerful enhancements and new considerations for practitioners.

Enhanced Understanding and Efficiency:

• Faster Malware Analysis: Defenders can leverage DAILA to rapidly identify the purpose of unknown functions, rename obfuscated variables, and summarize complex code flows within malware samples. This accelerates the initial triage and in-depth analysis phases, allowing security teams to understand threats more quickly and develop countermeasures sooner.
• Vulnerability Triage and Patch Analysis: When analyzing patches or identifying vulnerabilities in third-party software, DAILA can help pinpoint critical code changes, understand their impact, and quickly map out the functionality of unfamiliar binaries.
• Improved Incident Response: During a security incident, understanding compromised binaries or custom tools used by attackers is paramount. AI-assisted decompilation can provide quick insights into attacker capabilities and persistence mechanisms.
• Collaboration: BinSync's multi-user backend, integrated into DAILA, fosters a collaborative environment where human analysts and AI agents can work concurrently on the same binary. This can streamline team efforts, especially in large organizations or during complex investigations.

Critical Limitations and Considerations for Defenders:

• Misleading Vulnerability Detection: A key takeaway from the talk is that current AI models are "super duper bad" at vulnerability detection. Defenders must not rely on AI to automatically find bugs. Such outputs are often inaccurate and can waste valuable time chasing false positives. Manual verification and established vulnerability analysis techniques remain indispensable.
• Context is King: The success of DAILA hinges on providing the AI with appropriate, minimized context. Defenders need to develop new skills in "prompt engineering" and context management to effectively guide the AI, rather than simply feeding it raw data.
• Data Integrity and Type Inference: While AI can suggest types, incorrect retyping can corrupt the decompiler's internal state. Defenders must exercise caution and verify AI-suggested type changes to maintain the integrity of their analysis environment.
• Decompiler Quality: The quality of the underlying decompiler's output directly impacts the AI's performance. Defenders should be aware that inconsistencies or errors in pseudo-C generated by the decompiler will propagate to the AI's analysis.
• Anonymity and Data Leakage: A question raised during the Q&A highlighted concerns about training LLMs on sensitive binaries (e.g., malware). If models are trained on proprietary or confidential code, there's a risk of training data leakage, where the AI might inadvertently reproduce or reveal information about the original source. While DAILA uses external LLM APIs (like GPT-4), the broader implications for custom, internal LLMs used in sensitive environments are significant.
• Blurring Lines of Attribution: As AI contributions become indistinguishable from human analysis, attributing specific insights or errors in a collaborative environment (especially with BinSync's multi-user features) could become complex. This may impact accountability and trust within security teams.

In conclusion, DAILA offers a powerful toolkit for defenders, promising significant boosts in efficiency and understanding. However, its effective deployment requires a nuanced approach: leveraging AI as an intelligent assistant for specific, well-defined tasks (naming, summarization, contextual questions) while maintaining human oversight, rigorous verification, and a keen awareness of its current limitations, particularly in areas like vulnerability detection.

Key Takeaways

• AI as an Interactive Teammate: Large Language Models, when properly integrated, can serve as dynamic, interactive teammates in reverse engineering, addressing traditional decompiler shortcomings like lost names, types, and context.
• DAILA's Novel Contributions: The DAILA framework introduces critical advancements: decompiler agnosticism (working across IDA, Ghidra, etc.), support for large functions via intelligent context gathering, and dynamic updates that allow the AI to adapt as the human reverse engineer works.
• The Power of Context and Prompting: Effective reverse engineering with AI is heavily reliant on providing precise, minimized context and well-engineered prompts. The order of operations for AI tasks (e.g., variable renaming before summarization) significantly impacts the quality of results.
• AI Strengths and Weaknesses: AI excels at pattern recognition (e.g., identifying cryptographic algorithms like Luhn, known code structures) and automating mundane tasks (naming, summarization). However, it is currently "super duper bad" at vulnerability detection and struggles with complex symbolic reasoning or ambiguous control flow structures.
• Collaboration Blurs Boundaries: Tools like DAILA and its BinSync backend facilitate seamless collaboration between human analysts and AI agents, making their individual contributions increasingly difficult to distinguish, hence the talk's title, "Your Teammate Isn't Human."
• Future Work and Open Source: The field of AI-assisted reverse engineering is still nascent, with much future work remaining in areas like control flow modification, advanced context techniques, and enhanced interactivity. The DAILA project is open-source (mahalos/DAILA), encouraging community contributions and further research.

About the Speaker(s)

Zion Leonahenahe Basque is a Ph.D. student in the sepcom lab at Arizona State University (ASU). His research focuses on the intersection of decompilation and reverse engineering with artificial intelligence, exploring how these domains can be merged to create more coherent and efficient analysis workflows. A self-proclaimed native Hawaiian hacker, Zion is passionate about hacking and competitive Capture The Flag (CTF) challenges, known online by his handle "mahalos." He is a previous co-captain of the renowned CTF team Shellphish, which has consistently reached the Defcon CTF finals for 20 consecutive years. His practical experience in high-stakes competitions, including using DAILA at the Defcon finals, underpins the real-world applicability and insights presented in his talk.