Writing A New Decompiler Just For A Security Audit

Writing A New Decompiler Just For A Security Audit

Speakers: Dominic Couture, Staff Security Engineer, GitLab

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=h-x5Igpp9bY

Overview

In "Writing A New Decompiler Just For A Security Audit," Dominic Couture, a Staff Security Engineer at GitLab and an avid bug bounty hunter, shares his journey into building a specialized decompiler named Yburge for JRuby applications. This talk delves into the fascinating rabbit hole of reverse engineering a niche technology driven purely by curiosity and the desire to perform a comprehensive security audit. Couture's work highlights a critical gap in existing security tooling for JRuby, demonstrating how a lack of direct source code access can hinder effective vulnerability assessment.

The presentation outlines the technical challenges posed by JRuby's compiled format, particularly its "dark horse" application type, which encapsulates Ruby code within a Java binary blob. By methodically dissecting JRuby's internal representation (IR), Couture developed Yburge to reconstruct human-readable Ruby source code. This capability transforms the security auditing process for JRuby applications, moving from laborious black-box testing to efficient source-code analysis, enabling the discovery of otherwise hidden vulnerabilities and, equally importantly, confirming the absence of others. The talk serves as both a technical guide and an inspirational account of how pursuing intellectual curiosity can yield valuable tools for the broader security community.

Background

▶ Watch: Introduction and the 'rabbit hole' motivation (0:00)

JRuby is an implementation of the Ruby programming language that runs on the Java Virtual Machine (JVM). As Couture explains, its existence is often attributed to a desire for high performance (relative to standard Ruby), seamless integration with Java ecosystems, and improved garbage collection. He humorously speculates that JRuby might have been a "sneaky way to use Rails in a Java enterprise environment," catering to developers who wanted to leverage Ruby on Rails within Java-centric organizations.

JRuby applications typically manifest in three formats:

1. Standard Ruby: These are plain Ruby files, executed directly by JRuby. No reverse engineering is needed here, as the source code is readily available.
2. Java Class with Embedded JRuby Code: In this format, Ruby source code is embedded as a string literal within a Java class. Standard Java decompilers like JD-GUI or JD-CLI can easily extract the original Ruby source, often even preserving comments. Again, this poses no significant reverse engineering challenge for security auditors.
3. Java Class with Compiled JRuby Code (The "Dark Horse"): This is where the challenge lies. Applications in this format contain a Java class that, upon initial inspection, appears to hold a large, opaque binary blob. While strings from the original Ruby code (e.g., file names like greeter.rb, class names greeter, method names say hello) might be visible, the actual source code is not directly present. This format uses an internal representation (IR) of the Ruby code, which JRuby then processes. This binary blob, often seen in functions like load IR or referred to as script IR, was the "interesting research question" that prompted Couture's deep dive.

The primary problem was the lack of existing tools to reverse engineer this "dark horse" format. For a security auditor, the inability to access the original source code forces a reliance on black-box testing, which is often inefficient, incomplete, and prone to missing subtle vulnerabilities. Couture's motivation stemmed from a bug bounty engagement where he encountered such a JRuby application. Faced with the choice of moving to an easier target or delving into this unknown, his curiosity, combined with a passion for source code auditing, led him to embark on the ambitious task of building a decompiler. He acknowledged the many reasons not to pursue it—JRuby's niche nature, the time investment versus potential bounty payout, and the option to contribute to more mainstream projects—but ultimately, the excitement of uncharted territory, the joy of learning, and the desire to create a tool for the community outweighed these concerns.

Key Findings

▶ Watch: Motivational reasons for diving into niche security research (8:05)

Couture's journey into decompiling JRuby's internal representation began with a focused investigation into the mysterious script IR binary blob and the omnipresent IR keyword. His initial breakthrough came from discovering that "IR" stood for Internal Representation, a common concept in compilers where source code is translated into an intermediate, machine-agnostic format before final compilation or execution.

The critical insight arrived when he explored the JRuby source code itself. He found a debugging utility called IR dumper, which, as its name suggests, could dump the JRuby IR in a human-readable format. This tool, likely used by JRuby developers for debugging their compiler, proved invaluable. By dumping the IR for a simple Ruby script, Couture observed several key characteristics:

• Verbosity: The IR was significantly more verbose than the original Ruby code, expanding a few lines of Ruby into dozens of lines of IR.
• Loss of Control Flow: High-level control flow statements like if, while, and for were replaced with low-level jumps, similar to assembly language.
• Preservation of Key Information: Crucially, the IR retained line numbers from the original Ruby source, as well as meaningful strings like class names (greeter), method names (say hello), and literal strings (hello). This preservation of metadata strongly suggested that the original source code was, in principle, recoverable.

The IR dumper output revealed a structured sequence of instructions, such as def class (define class), define instance method, create string, and call puts. This granular breakdown of operations convinced Couture that it was possible to automate the reverse process: read the IR and reconstruct the Ruby code.

The next significant finding was the discovery of JRuby's CompiledScriptLoader. Instead of having to parse the raw binary blob himself, Couture found that the JRuby codebase provided a public class designed precisely for this purpose. The CompiledScriptLoader could take the binary IR blob, deserialize it, and return a structured list of individual instructions, each with its associated operands. This eliminated a massive hurdle, as it meant Yburge could leverage JRuby's own parsing logic rather than reinventing it. This finding solidified the feasibility of the decompiler project, transforming it from a "maybe possible" to a "just a matter of doing it."

Technical Deep Dive

▶ Watch: Live demonstration of the Yburge decompiler tool (18:50)

The core technical challenge addressed by Yburge is the transformation of JRuby's compiled Internal Representation (IR) back into human-readable Ruby source code. JRuby's compilation process involves several stages: it first parses Ruby source, translates it into its IR, and then "Java-izes" it for execution on the JVM. The "dark horse" application format stores this IR in a binary blob within a Java class file.

Couture's approach leverages the open-source nature of JRuby, specifically targeting how JRuby itself handles this compiled IR. The initial hurdle of understanding the binary blob was significantly simplified by two key discoveries:

1. The IR dumper utility: This internal JRuby debugging tool allowed Couture to observe the structure of the IR. He noted that while verbose and low-level (replacing high-level Ruby constructs with jumps), the IR still contained critical metadata like original line numbers, class names, method names, and string literals. This confirmed that enough information was retained to attempt a reconstruction. For example, a simple Ruby class definition def class Greeter would translate into a series of IR instructions that explicitly define a class named "Greeter."

1. The CompiledScriptLoader class: This was the most significant technical enabler. Instead of parsing the intricate binary format of the IR blob directly, Couture discovered that JRuby provides a public Java class, org.jruby.ir.CompiledScriptLoader, specifically designed to load and deserialize this IR. When invoked, CompiledScriptLoader takes the binary blob and returns a structured list (an array) of IR instructions. This effectively delegated the complex parsing task to JRuby itself.

How Yburge Works:

Yburge's operation is essentially a reverse mapping process:

1. Loading the IR: Yburge first takes a JRuby-compiled .class file as input. It then calls into the CompiledScriptLoader (a Java class, but accessible from Ruby if running on JRuby itself, or via FFI/JNI from other languages) to extract the list of IR instructions. This list represents the entire program logic in an intermediate, abstract form.

1. Instruction and Operand Analysis: Each item in the list returned by CompiledScriptLoader is an IR instruction. These instructions correspond to fundamental operations or constructs in Ruby. Examples include:

• DefineClassInstruction: Represents the definition of a Ruby class.
• DefineInstanceMethodInstruction: Represents the definition of a method within a class.
• CallInstruction: Represents a method call (e.g., puts).
• CreateStringInstruction: Represents the creation of a string literal.
• LoadConstantInstruction: Loading various types of constants (integers, booleans).
• JumpInstruction: Low-level control flow.

Each instruction also has associated operands, which provide the specific details for that operation. For a DefineClassInstruction, operands would include the class name. For a CallInstruction, operands would specify the method name and arguments.

1. Syntactic Reconstruction: Yburge iterates through this list of IR instructions. For each instruction, it applies a rule to generate the corresponding Ruby syntax.

• If it encounters a DefineClassInstruction with the operand greeter, it outputs class Greeter.
• If it sees a DefineInstanceMethodInstruction for say_hello, it outputs def say_hello.
• A CreateStringInstruction followed by a CallInstruction for puts would translate to puts "Hello".
• Conditional if/else blocks, which are represented by jump instructions and conditional checks in the IR, are analyzed to reconstruct the higher-level Ruby if and else constructs. This often involves tracking the flow of jumps and the conditions that trigger them.

Couture notes that the initial implementation is a straightforward, iterative process: "if I'm seeing a defined class instruction, then I just output the string defined class with the name of the class." While this produces functional Ruby code, it often results in more verbose output than the original, with explicit parentheses for method calls (puts("Hello") instead of puts "Hello") and sometimes redundant constructs. The goal was primarily functionality—to recover the source code for auditing—rather than perfect stylistic fidelity.

The reliance on JRuby's CompiledScriptLoader is a pragmatic design choice that significantly accelerated development. However, Couture also acknowledges this as a potential future dependency to break, ideally by re-implementing the deserialization logic directly to make Yburge entirely independent of the JRuby runtime for parsing. This would allow Yburge to be rewritten in pure Ruby, rather than needing to run within a JRuby environment or bridge to Java.

Demo / Proof of Concept

▶ Watch: Real-world vulnerabilities found using the decompiler (23:50)

Dominic Couture provided a live demonstration of his decompiler, Yburge, showcasing its command-line functionality. Despite a humorous battle with "demo gods" (which he had ironically tried to avoid by initially planning to skip the demo), he successfully illustrated the tool's core capability.

The demonstration process involved:

1. Compiling Ruby Code with JRuby: Couture first created a simple Ruby script, for instance, test.rb, containing basic Ruby code. He initially tried a slightly complex if true block, which caused a temporary hiccup, but then simplified it to a single puts "Hello, Hack in the Box" statement. He then used the JRuby compiler (jrubyc) to compile this .rb file into a JRuby .class file (e.g., Test.class). This .class file contains the opaque binary IR blob that Yburge aims to reverse.

1. Decompiling with Yburge: With the compiled .class file generated, Couture then invoked Yburge from the command line:

(Note: The exact command shown in the video was JRuby then Y Birch. We want to decompile this class. Output it and uh this one as Ruby.).

Yburge then processed the Test.class file, extracted the IR instructions using JRuby's internal mechanisms, and reconstructed the Ruby source code, writing it to test_decompiled.rb.

1. Viewing the Decompiled Output: The output file test_decompiled.rb contained the recovered Ruby code. For the simple puts example, it successfully produced puts("Hello, Hack in the Box"). Couture noted that Yburge currently adds parentheses to method calls even when Ruby doesn't strictly require them, making the output slightly more verbose than the original but functionally identical. He later demonstrated it working correctly with an if x == true block, showing its ability to reconstruct conditional logic, albeit with some extra verbosity.

The demo, despite its minor live-coding challenges, effectively proved Yburge's ability to take a JRuby compiled class file and reconstruct its original Ruby source code. This transformation is crucial for security auditors, as it converts an opaque binary into an analyzable script, enabling static analysis and code review.

Defensive Implications

▶ Watch: Broader takeaways: unexpected benefits of deep dives (33:00)

Yburge significantly enhances the capabilities of security defenders and auditors working with JRuby applications, primarily by unmasking security through obscurity. When applications rely on the obscurity of their compiled code to hide logic or sensitive data, Yburge provides the means to bypass this.

Couture provided several compelling examples from a real-world security audit of an unnamed JRuby Ruby on Rails application (dubbed "Druby on Rails"):

1. Undocumented API Routes and Sensitive Data Exposure: In black-box testing, discovering all API endpoints can be a major challenge without comprehensive documentation (like Swagger). By decompiling the application, Couture was able to recover all route definitions. This led to the discovery of an undocumented API endpoint that, when called, listed the API keys for every single user. This was a critical information disclosure vulnerability that would have been extremely difficult, if not impossible, to find through fuzzing or other black-box techniques. The source code revealed the endpoint's existence and its sensitive functionality directly.

1. Mass Assignment Vulnerabilities: Mass assignment occurs when an application allows a user to set more properties on an object than intended. For example, a user might submit a JSON payload containing {"username": "attacker", "email": "[email protected]", "role": "admin"} even if the UI only exposes username and email fields. In the audited application, Couture found a mass assignment vulnerability related to API key creation. While the Ruby on Rails permit function was correctly used to whitelist parameters (preventing arbitrary fields like role from being set), the developers failed to implement permission checks for specific, sensitive parameters. A low-privilege user could submit a request to create an API key with the parameter access_all_engines set to true. Because there was no code checking if the user had administrative privileges to set this property, the low-privilege user could effectively elevate their access, gaining control over "every single thing" in the application. Again, the existence of access_all_engines and the lack of permission checks were only evident from the decompiled source code.

Perhaps one of the most underrated but significant defensive implications highlighted by Couture is the ability to confirm the absence of vulnerabilities. In traditional black-box testing, auditors often spend considerable time attempting various payloads for common vulnerabilities like XML External Entity (XXE) injection or command injection. Even after extensive testing, a nagging doubt often remains: "Did I miss a specific encoding? A subtle bypass? Could this still be vulnerable?"

With the decompiled source code, this uncertainty is largely eliminated:

• Command Injection: Couture encountered an endpoint he suspected of shelling out to the operating system. By reviewing the decompiled code, he could see that the developers were using the correct and safe method for executing external commands in Ruby, effectively sanitizing inputs. This allowed him to confidently conclude that it was not vulnerable to command injection, saving days of fruitless bypass attempts.
• XXE Injection: For XML inputs, XXE is a common concern. By reviewing the source code, Couture could inspect the configuration of the XML parser. He could verify that settings known to enable XXE (e.g., allowing external entities) were disabled. This allowed him to definitively state that the application was not vulnerable to XXE, avoiding the complexities of testing different XML parsers, environments (Linux/Windows), and encoding schemes.

This capability to definitively rule out vulnerabilities is a massive time-saver for security professionals, reducing audit scope, improving report accuracy, and alleviating the mental burden of "what ifs." Yburge transforms the JRuby security landscape by enabling thorough, source-code-level analysis, moving beyond guesswork to verifiable conclusions.

Key Takeaways

• Embrace Curiosity-Driven Research: Dominic Couture's journey began with a "rabbit hole" question about an obscure compiled format. This pursuit of intellectual curiosity, even without immediate financial incentive, led to the creation of a valuable security tool.
• Source Code Auditing is Paramount: The talk strongly advocates for the benefits of source code auditing over black-box testing. Direct access to code enables the discovery of subtle vulnerabilities (like undocumented APIs or mass assignment flaws) that are nearly impossible to find otherwise.
• Confirming Non-Vulnerabilities Saves Time and Reduces Risk: A significant, often overlooked, benefit of source code access is the ability to definitively confirm that certain vulnerabilities (e.g., XXE, command injection) are not present. This saves immense amounts of auditor time and provides greater confidence in security assessments.
• Niche Tools Can Have Broad Impact: While JRuby might be considered niche, Yburge addresses a critical gap. The unexpected interest from other JRuby users demonstrates that even highly specialized tools can fulfill a real need and foster community collaboration.
• Leverage Existing Open Source: Yburge's development was significantly accelerated by leveraging JRuby's own internal CompiledScriptLoader and IR dumper utilities. Understanding and utilizing the target's open-source codebase is a powerful strategy in reverse engineering.
• Tool Building is a Fulfilling Endeavor: Beyond the security findings, the act of building a tool to solve a specific, complex problem is presented as a deeply satisfying and educational experience, fostering continuous learning and skill development.

About the Speaker(s)

Dominic Couture is a Staff Security Engineer at GitLab. His work presented in this talk, however, is explicitly stated to be unrelated to his job at GitLab, stemming instead from his personal pursuits as a bug bounty hunter. Couture describes himself as a long-time developer who found his way into security, driven by a strong curiosity and a passion for understanding how things work. He is a staunch advocate for source code auditing, believing it offers a superior approach to vulnerability discovery compared to black-box testing. His talk highlights his dedication to building tools that empower the security community and his willingness to dive deep into technical challenges for the sheer joy of learning and problem-solving.