No Spec, No Problem: Automated ISA reverse engineering with Multi-Agent AI

No Spec, No Problem: Automated ISA reverse engineering with Multi-Agent AI

Speakers: Dan Borgogno (Security Researcher, Faraday); Gaston Aznarez (Security Researcher, Faraday)

Conference: Ekoparty Miami

YouTube: https://www.youtube.com/watch?v=c-Wfvo1kEuc

Overview

In the realm of embedded systems, custom Instruction Set Architectures (ISAs) often pose a significant barrier to security research and vulnerability discovery. When a device uses a proprietary or undocumented ISA, traditional reverse engineering tools like Ghidra are rendered ineffective, leaving firmware as an impenetrable black box. This talk, "No Spec, No Problem," presented by Faraday security researchers Gaston Aznarez and Dan Borgogno at Ekoparty Miami, tackles this precise challenge. They introduce a groundbreaking multi-agent AI pipeline designed to automatically reverse engineer unknown ISAs and generate the necessary specifications for decompilers, effectively turning previously unanalyzable firmware into actionable code.

The motivation for this ambitious project stemmed from a seemingly innocuous device: a "smart" vape. Driven by curiosity about its embedded screen, operating system, Bluetooth connectivity, and firmware update capabilities, the researchers embarked on a reverse engineering journey. What began as a standard hardware teardown quickly evolved into a deep dive into the complexities of undocumented architectures, ultimately leading to the development of a novel AI-driven solution that promises to democratize firmware analysis for a wide array of embedded devices.

This article delves into the methodology, technical intricacies, and significant findings of their work. It highlights how a sophisticated blend of large language models (LLMs) and deterministic agents can overcome the limitations of traditional reverse engineering, providing a pathway for security researchers to analyze and secure devices powered by obscure or proprietary instruction sets, thereby expanding the frontier of embedded security.

Background

▶ Watch: Introduction: Reverse engineering a vape and Ghidra's challenge (0:00)

The journey into automated ISA reverse engineering began with a practical, albeit unexpected, target: a smart vape. Upon disassembling the device, the Faraday team identified its core components: a Shelly CY712F6 Microcontroller Unit (MCU) and a Boy 25Q1 SPI flash memory. Initial attempts to extract useful information involved standard reverse engineering techniques, including analyzing its Flutter-based Android companion app, performing Bluetooth Low Energy (BLE) sniffing to reverse the communication protocol, and capturing partial firmware updates. However, these methods yielded only peripheral data like assets, images, and sounds, but not the full, runnable firmware.

The breakthrough came with the discovery that a specific programming command, sent with precise timing, could enable the device's bootloader, allowing for a full firmware dump using an ESP32 and a relay—a technique reverse-engineered by an online community member. With the complete firmware in hand, the natural next step was to load it into Ghidra for decompilation. This is where the core problem emerged: Ghidra, like most reverse engineering frameworks, relies on predefined Instruction Set Architecture (ISA) specifications to interpret binary code. The Shelly CY712F6 MCU utilized a highly specific, undocumented ISA known as PI32V2, for which no public specification or Ghidra support existed. An existing community plugin for PI32V1 proved incompatible, leaving the researchers with a raw, unintelligible binary.

This scenario is far from unique; custom architectures are prevalent in various embedded domains, including IoT devices, automotive systems, and Industrial Control Systems (ICS) PLCs, often due to specific performance, power, or cost requirements. The lack of ISA documentation for these devices creates significant blind spots for security researchers, making vulnerability discovery exceptionally challenging. The pivotal discovery that unlocked their approach was the availability of the vendor's toolchain, which included an object dump utility. This utility could take the compiled binary and produce a human-readable assembly interpretation, providing a crucial "ground truth" for what the binary instructions actually meant. Faced with either laboriously reverse engineering the toolchain itself (a long, architecture-specific endeavor) or developing a more general solution that could infer ISA definitions from object dump outputs, the team chose the latter, paving the way for their multi-agent AI pipeline.

Key Findings

▶ Watch: Automated ISA reverse engineering with multi-agent AI pipeline (10:00)

The Faraday team's primary contribution is the successful development and validation of a hybrid multi-agent AI pipeline capable of automatically reverse engineering unknown Instruction Set Architectures (ISAs) from their corresponding object dump outputs. This pipeline generates Ghidra-compatible SLEIGH specifications, effectively enabling high-level decompilation for previously inaccessible binaries.

Key findings include:

• Feasibility of Automated ISA Recovery: The project demonstrated that it is possible to infer complex ISA definitions, including instruction mnemonics, operand types, register usage, and intricate literal encoding schemes, purely from the textual output of an object dump utility. This overcomes the critical barrier of undocumented ISAs.
• Optimal Hybrid AI Architecture: A crucial discovery was the effectiveness of combining Large Language Models (LLMs) with deterministic agents. LLMs proved exceptional at semantic reasoning, context interpretation, and making high-level decisions (e.g., splitting instruction clusters based on observed discrepancies). However, they struggled with precise, bit-level arithmetic and structural consistency, often "hallucinating" or losing context. Deterministic agents, conversely, handled these exact, repeatable tasks (e.g., grouping instructions, bit correlation, mathematical literal recovery) with high accuracy and efficiency. This hybrid approach significantly improved performance, reduced token usage, and enhanced reliability compared to a purely LLM-driven system.
• Successful Ghidra SLEIGH Generation: The pipeline is able to construct comprehensive SLEIGH specifications, covering both the structural definition of instructions and their semantic meaning via P-code. This direct integration with Ghidra is critical, as it allows security researchers to immediately leverage their existing tools and workflows for analyzing the newly understood firmware.
• Demonstrated Generalizability and Efficiency:
• MIPS32 Validation: The pipeline successfully generated a SLEIGH specification for the well-known MIPS32 architecture, achieving 98% code coverage using approximately 1.6 million tokens and costing around $7. The generated specification closely matched existing, manually crafted Ghidra definitions, validating the approach.
• PI32V2 for "Smart Vape": For the target PI32V2 architecture, the pipeline achieved approximately 90% coverage using more tokens (due to the architecture's complexity) and costing around $50. This allowed the researchers to decompile the vape's firmware, a task previously impossible.
• Identification of Key Challenges and Solutions: The team identified and addressed challenges such as "bit hallucination" in LLMs (by offloading bit-level tasks to deterministic agents) and the difficulty of generating syntactically correct SLEIGH (by making structural generation deterministic and letting LLMs focus on P-code semantics). They also acknowledged limitations inherent to using object dumps, such as the inability to infer calling conventions or instructions not present in the sample data.

In essence, the project demonstrates a powerful paradigm shift in embedded security research, providing an automated, cost-effective, and generalizable method for tackling the pervasive problem of undocumented instruction sets.

Technical Deep Dive

▶ Watch: Live demonstration of the automated ISA extraction tool (36:00)

The core of the "No Spec, No Problem" solution is a sophisticated multi-agent AI pipeline designed to infer ISA specifications from an object dump output. This pipeline comprises several specialized agents, each contributing to the overall goal of generating a Ghidra-compatible SLEIGH specification. The architectural design is a hybrid model, leveraging the strengths of Large Language Models (LLMs) for reasoning and semantics, and deterministic agents for structural analysis, arithmetic, and consistency checks.

The pipeline's journey begins with the raw object dump output, which provides pairs of binary instruction data and their corresponding assembly text representation.

1. Bootstrap Agent (Deterministic):

The initial step involves grouping similar instructions. The Bootstrap agent deterministically analyzes the object dump output to identify patterns and cluster instructions that share common characteristics (e.g., the same mnemonic or similar operand structures). This process aims to reduce redundancy and provide the subsequent agents with more focused sets of data. For instance, all "add" operations, despite varying registers or immediates, would be grouped together. This pre-processing is crucial for managing the vast amount of data and preventing LLMs from losing context or "hallucinating" due to excessive input.

1. Text Interpreter Agent (Pure AI/LLM):

Once instructions are grouped, the Text Interpreter—a purely LLM-driven agent—takes over. Its role is to assign semantic meaning to the grouped textual representations. It analyzes the mnemonics, registers, and literals, identifying which parts of an instruction are fixed (e.g., the operation code) and which are variable (e.g., registers, immediate values). For example, given a group of "add R1, R2, #imm" instructions, the LLM will identify "add" as the operation, R1/R2 as registers, and "#imm" as a literal.

A key capability of this agent is its ability to detect discrepancies within seemingly similar groups. If, after initial analysis, it finds two distinct patterns within a single cluster (e.g., an "add" operation sometimes using an immediate and sometimes a memory address), it will flag this, prompting a split in the cluster for further, more granular analysis. This reasoning capability is difficult to achieve deterministically, making it an ideal task for an LLM that can infer context and patterns across diverse textual representations found in different architectures.

1. Bit Interpreter Agent (Hybrid AI/Deterministic):

This is one of the most critical and complex stages, responsible for correlating the textual fields identified by the Text Interpreter with the actual bits in the binary instruction. The Bit Interpreter is a hybrid agent, with deterministic components handling the precise bit-level logic and an LLM component making high-level decisions like splitting.

• Field-to-Bit Correlation: For each instruction field (e.g., a specific register, an immediate value), the deterministic part of the agent analyzes numerous samples where that field changes. By comparing the binary representations, it identifies which specific bits remain constant (opcode) and which bits change to encode the varying field. This process results in a "mask" that defines the bit-field for each component.
• Complex Literal Encoding Recovery: The recovery of immediate (literal) values is particularly challenging due to various encoding schemes. The agent employs a "recipe" or equation-based approach to represent these: literal = (Transform Instruction_Bits) + K + (H Program_Counter).
• Linear Cases:
• Direct Fields: Simple bit-ranges directly mapping to a value.
• Zero/Sign Extension: Identified by how the most significant bit (MSB) propagates. For sign extension, the MSB is copied to fill higher bits; for zero extension, zeroes are used. The agent determines this by comparing literal values with their bit patterns.
• PC-Relative: Jumps or branches often use program counter (PC)-relative offsets. The H * Program_Counter term accounts for adding the current PC or PC+offset to the literal. The K term handles constant offsets, such as PC - 4 in ARM branches.
• XORed Values: Pathological cases, like a specific PI32V2 instruction, might involve XORing the bit field with a constant K value.
• Non-Linear Cases (Shifted Immediates): For encodings like shifted immediates (e.g., value << shift_amount), the agent models these as functions applied to bits. A mathematical equation is formulated and solved to determine the base value and the shift amount.
• Decision Making (LLM): While the bit-level arithmetic is deterministic, the LLM is involved when a decision regarding instruction splitting is required. For instance, if an instruction's encoding changes based on a specific bit (e.g., using a stack pointer vs. a named register), the LLM is given all statistical and bit-level information to decide if a cluster needs to be split. This prevents "bit hallucination" where LLMs might incorrectly interpret bit patterns.

1. Knowledge Manager Agent (Hybrid AI/Deterministic):

As information about the ISA is extracted, it's stored in a knowledge base. The Knowledge Manager is responsible for maintaining the consistency and integrity of this data. It performs deterministic pre-checks to ensure new information doesn't conflict with existing entries. When conflicts arise (e.g., two different interpretations for the same bit pattern), the LLM component of this agent engages in reasoning to resolve the discrepancy, ensuring the knowledge base remains accurate.

1. Supervisor Agent (Hybrid AI/Deterministic):

The Supervisor acts as the orchestrator and quality control for the entire pipeline. It monitors the progress, detects potential errors, and identifies instances of LLM hallucination or logical inconsistencies. If an agent produces an illogical output or if the overall process stalls, the Supervisor can decide to relaunch previous steps, provide hints, or, as a last resort, escalate the problem to a human operator. Its primary goal is to ensure the pipeline progresses efficiently and produces correct results.

1. SLEIGH Generator Agent (Hybrid AI/Deterministic):

The final stage is the generation of the SLEIGH specification, which Ghidra uses for decompilation. SLEIGH consists of two main parts:

• Structure Generation (Deterministic): This part defines the instruction format, operand fields, and register banks. A pre-analysis step involves building graphs from register usage patterns across instructions to deterministically identify and group registers into their respective banks (e.g., T-registers, F-registers). This structural generation is made deterministic to avoid the common problem of LLMs producing syntactically incorrect code that enters compilation error loops.
• P-code Generation (LLM): P-code is Ghidra's intermediate representation, defining the semantic meaning of each instruction (e.g., an "add" instruction translates to a P-code operation that adds two values and stores the result). The LLM is ideally suited for this task, as it can accurately translate the high-level semantic descriptions derived from the Text Interpreter into the precise P-code operations. This leverages the LLM's strength in contextual understanding and code generation, focusing it on the semantic layer where it excels.

By meticulously orchestrating these hybrid agents, the pipeline efficiently and accurately reverse engineers unknown ISAs, delivering a functional SLEIGH specification that unlocks the full potential of Ghidra for firmware analysis.

Demo / Proof of Concept

▶ Watch: Results: MIPS32 and P32V2 architecture recovery (42:00)

The practical application and effectiveness of the multi-agent AI pipeline were showcased through a compelling terminal-based User Interface (UI), demonstrating its capabilities in real-time. The demo highlighted the system's ability to process and understand an unknown ISA, providing transparency into each agent's activity and the overall progress.

The terminal UI, designed for Docker deployment, provides a comprehensive overview of the pipeline's status. Key metrics displayed include the number of instruction clusters identified, the size of the knowledge base, the current pipeline status (e.g., "running," "finished"), and, crucially, the code coverage percentage. The demo, which was sped up for presentation, showed the coverage rapidly increasing, particularly in the initial phases, as the system prioritized analyzing clusters with a high number of samples. The speakers noted that Anthropic's models were used, though users could select other LLM providers.

During the demonstration, the UI allowed for a detailed inspection of individual agents and their outputs:

• Cluster View: This showed the identified clusters of instructions, their interpreted structure, and raw samples from the object dump. For the demo, a MIPS32 binary extracted from a router was used, illustrating the pipeline's ability to handle a known, but still complex, architecture.
• Coverage View: A visual representation of the progress, indicating how much of the instruction set had been analyzed.
• Token Usage: A live counter tracking the number of LLM tokens consumed, providing insight into the computational cost.
• Knowledge Base: This critical section displayed the evolving understanding of the ISA, including identified addressing modes, register definitions, endianness, and other general architectural information. The UI allowed drilling down into specific instruction entries, showing the Text Interpreter's semantic breakdown and the Bit Interpreter's bit-field correlations. This feature was highlighted as essential for debugging and refining the pipeline's output.
• Supervisor Interaction: The UI allowed users to interact with the Supervisor agent, checking the process status or even providing hints (e.g., if external information about the ISA was found).

A significant achievement demonstrated was the speed of the process. For the MIPS32 architecture, the pipeline achieved 90% code coverage in less than 20 minutes. Upon completion, the system could export the generated SLEIGH specification as a Ghidra plugin.

The results for MIPS32 were particularly impressive: 1.6 million tokens consumed at a cost of approximately $7, yielding 98% code coverage. The generated SLEIGH closely matched the existing, manually developed Ghidra specifications for MIPS32, confirming the accuracy of the automated approach. While some minor differences in interpretation existed, the overall semantic meaning was identical.

For the original target, the PI32V2 architecture of the smart vape, the process consumed more tokens and cost around $50, achieving slightly less coverage (around 90%). This was attributed to the obscurity and "pathological" nature of the PI32V2 instruction set, which features highly compressed and unusual encodings (e.g., 8-bit register list bitmaps for push instructions). Despite these complexities, the pipeline successfully generated a Ghidra output, providing a foundation for future reverse engineering and vulnerability research on the device, a task that was previously impossible. The speakers proudly displayed the Ghidra decompilation, acknowledging that while some aspects like calling conventions and delay slot interpretation still required manual refinement, the core code was now understandable.

Defensive Implications

▶ Watch: Q&A: Initial firmware extraction from the vape (54:00)

The "No Spec, No Problem" project has profound defensive implications, fundamentally shifting the landscape of embedded device security. For years, custom or undocumented Instruction Set Architectures (ISAs) have served as a de facto security by obscurity mechanism. Manufacturers, particularly in IoT, automotive, and industrial control systems (ICS), often rely on the sheer difficulty of reverse engineering proprietary silicon to deter attackers. This research directly challenges that paradigm.

Empowering Defenders:

• Broader Firmware Analysis: The most significant implication is the ability for security researchers and defenders to analyze firmware from devices that were previously black boxes. This tool democratizes access to low-level code, enabling vulnerability research on a far wider range of embedded systems.
• Proactive Vulnerability Discovery: With automated ISA recovery, defenders can proactively audit the firmware of their deployed devices, identifying vulnerabilities before they are exploited in the wild. This is especially critical for long-lifecycle industrial systems or critical infrastructure where manual reverse engineering is prohibitively time-consuming and expensive.
• Reduced Time to Insight: The pipeline drastically reduces the time required to generate an initial, usable Ghidra specification from weeks or months of manual effort to mere hours or even minutes. This accelerated analysis cycle allows defenders to respond more quickly to emerging threats or conduct comprehensive security assessments within tighter deadlines.
• Enhanced Supply Chain Security: Organizations can use this capability to perform deeper security audits of third-party hardware and firmware components in their supply chain, verifying that devices adhere to security best practices and do not contain hidden backdoors or critical flaws.

Implications for Attackers:

While primarily a defensive tool, the open-source nature of this project means that attackers could also leverage it. The barrier to entry for analyzing custom ISAs is lowered for everyone. However, this is largely a net positive for defenders:

• Leveling the Playing Field: For sophisticated attackers, manual ISA reverse engineering was always a possibility, albeit resource-intensive. This tool simply makes the process more efficient. For less resourced attackers, it might open new avenues, but the overall benefit to the security community (who can now find and patch vulnerabilities more widely) is significant.
• Shift in Focus: The difficulty of ISA reverse engineering previously absorbed significant attacker resources. With that hurdle reduced, both defenders and attackers can now focus on the actual vulnerability discovery within the decompiled code, shifting the battleground from "understanding the architecture" to "finding the flaws."

Recommendations for Defenders:

• Assume Undocumented ISAs are Vulnerable: No longer rely on the obscurity of a custom ISA as a security measure. Assume that tools like this can and will be used to analyze your firmware.
• Invest in Secure Development Practices: Focus on secure coding, robust input validation, memory safety, and strong cryptographic implementations, as the underlying architecture is becoming increasingly transparent.
• Proactive Firmware Audits: Integrate automated ISA reverse engineering into your security testing pipeline for embedded devices, especially those with custom silicon.
• Monitor Open-Source Tools: Keep an eye on the open-source release of this and similar tools to understand the evolving capabilities of both defensive and offensive researchers.

In essence, "No Spec, No Problem" transforms "no spec" from a security advantage into a potential liability, urging manufacturers and defenders to double down on fundamental security principles rather than relying on the obscurity of their proprietary hardware.

Key Takeaways

• Automated ISA Reverse Engineering is Achievable: The project successfully demonstrates the feasibility of automatically deriving Instruction Set Architecture (ISA) specifications, even for previously undocumented and "pathological" custom architectures, from object dump outputs.
• Hybrid AI is Key for Complex RE Tasks: The most effective approach combines the semantic reasoning capabilities of Large Language Models (LLMs) with the precision and consistency of deterministic agents. LLMs excel at understanding context and making high-level decisions, while deterministic code handles bit-level arithmetic, structural analysis, and consistency checks, mitigating issues like "bit hallucination."
• Ghidra Integration Unlocks Firmware Analysis: The pipeline's ability to generate Ghidra-compatible SLEIGH specifications is crucial, directly enabling security researchers to decompile and analyze firmware that was previously an unintelligible binary blob.
• Significant Efficiency and Cost Savings: The automated process drastically reduces the time and cost associated with ISA reverse engineering. For MIPS32, it achieved 98% coverage for just $7 in LLM tokens in under 20 minutes, a fraction of the manual effort. For the complex PI32V2, it provided a usable specification for approximately $50.
• Broadened Scope for Security Research: This tool lowers the barrier to entry for analyzing firmware across a wide range of embedded systems, including IoT, automotive, and industrial control devices, opening new avenues for vulnerability discovery and enhancing overall cybersecurity.
• Open-Source Contribution to the Community: The project is open source, fostering collaboration and allowing the broader security research community to leverage, contribute to, and further develop this innovative methodology.

About the Speaker(s)

Gaston Aznarez and Dan Borgogno are Security Researchers at Faraday, a company specializing in security research and solutions. Their work, as demonstrated in "No Spec, No Problem," focuses on pushing the boundaries of reverse engineering and vulnerability discovery in complex systems, particularly those involving embedded devices and custom architectures. Their expertise lies in understanding low-level hardware and software interactions and developing innovative tools to address significant challenges in the security landscape.