Advancements in JavaScript Engine Fuzzing

Advancements in JavaScript Engine Fuzzing

Speakers: Carl Smith, Security Engineer, V8 Security, Google

Conference: HITCON CMT

YouTube: https://www.youtube.com/watch?v=cDvzySoALoE

Overview

Carl Smith, a Security Engineer on the V8 Security team at Google, delivered an insightful talk at HITCON CMT, delving into the sophisticated world of JavaScript engine fuzzing. The presentation, aptly subtitled "Finding Cool Bugs with Little Compute," explored the persistent challenges of discovering complex vulnerabilities within high-performance JavaScript engines like Google's V8, which powers Chrome. Smith highlighted how traditional coverage-guided fuzzing, while effective, often struggles with bugs that require a precise sequence of operations or specific data interactions.

The core of Smith's talk revolved around the innovative techniques developed by the V8 security team to overcome these limitations. He introduced specialized mutators – Probing and Exploration – designed to dynamically learn API usage and object capabilities, significantly enhancing the fuzzer's intelligence. Furthermore, he unveiled a powerful human-assisted fuzzing approach through program templates, enabling researchers to guide the fuzzer towards specific code shapes and known bug patterns. This talk is crucial for security researchers, vulnerability hunters, and browser developers seeking to understand the cutting edge of JavaScript engine security and how targeted, intelligent fuzzing can uncover deeply hidden flaws.

Background

▶ Watch: Introduction to JavaScript engine fuzzing (0:00)

Fuzzing, at its essence, is an automated software testing technique that involves feeding a program with large amounts of random or semi-random data in an attempt to make it crash or behave unexpectedly. The fundamental principle often relies on coverage-guided fuzzing, where the fuzzer monitors the program's control flow graph and attempts to maximize edge coverage. When a new execution path (edge) is discovered, the input that triggered it is deemed "interesting" and added to a corpus for further mutation, guiding the fuzzer towards unexplored code. For simple programs, this approach is highly effective; for complex targets like JavaScript engines, the inputs are not mere numbers but entire JavaScript programs.

Google's V8 security team employs a custom fuzzer named Fusili, which operates on an Intermediate Language (IL) called Fuzzilli. This IL represents JavaScript programs in a structured format, making them significantly easier to mutate and combine compared to raw JavaScript code. Fusili uses various mutation strategies, including changing inputs, constants, and inserting instructions. A particularly effective mutation is splicing, which intelligently copies parts of one Fuzzilli program into another, respecting variable dependencies and contextual constraints (e.g., a return instruction only within a function). The mutated Fuzzilli is then "lifted" back into JavaScript and fed into V8, with observed behavior guiding the fuzzer.

Despite its sophistication, Fusili, like all coverage-guided fuzzers, faces a fundamental challenge: the sheer immensity of the JavaScript program space. While it excels at finding bugs that are "one mutation away" from existing corpus samples or those that manifest through simple code path coverage, it struggles with vulnerabilities requiring a complex, multi-step sequence of operations or specific data interactions that don't immediately increase edge coverage. The problem lies in the fuzzer's inability to "understand" how to meaningfully combine disparate, individually interesting code paths to trigger a more complex bug, leading to a vast "unreachable" space of potential vulnerabilities.

Key Findings

▶ Watch: Classic JavaScript engine bug: array shrinking vulnerability (10:00)

The core innovation presented by Carl Smith lies in addressing the limitations of traditional coverage-guided fuzzing by introducing intelligent, specialized mutators and a human-assisted fuzzing paradigm. The key findings and contributions include:

1. Dynamic API Learning with Specialized Mutators:

• Probing Mutator: This technique allows Fusili to dynamically infer how specific JavaScript APIs or built-in functions use their arguments. By temporarily inserting a JavaScript proxy object (the "probe") as an argument, the fuzzer observes all property accesses and method calls made on that object by the target function. This information is then used to construct concrete, intelligent arguments, effectively teaching the fuzzer how to interact with an API. This led to the discovery of a bug in ArrayBuffer resizing, demonstrating its power in uncovering side-effect-related vulnerabilities.
• Exploration Mutator: Complementing probing, the exploration mutator focuses on inferring the capabilities and return types of JavaScript APIs. By leveraging JavaScript's runtime type introspection features, it determines what can be done with an object (e.g., available properties, callable methods). This allows Fusili to dynamically discover and interact with new or less-frequented methods, leading to the identification of a significant type confusion vulnerability in the relatively new Array.prototype.at method within V8's optimizing compiler.

1. Human-Assisted Fuzzing with Program Templates: To tackle bugs requiring a very specific, high-complexity code shape, the team developed a hybrid fuzzing engine that incorporates program templates. These templates, written in Swift, allow researchers to express a rough "code shape" or structure that the fuzzer should adhere to. Within these templates, a powerful build function enables Fusili to generate and mutate code iteratively, filling in the blanks while maintaining the desired high-level structure. This approach significantly reduces the search space for complex bugs by focusing compute power on researcher-identified areas of interest.

1. Demonstrated Effectiveness on Complex Bugs: The talk highlighted several real-world bugs found using these techniques:

• A harmless out-of-bounds access in ArrayBuffer resizing (via Probing).
• A critical type confusion in Array.prototype.at (via Exploration), which was actively exploited.
• An old, persistent V8 bug related to regular expressions (via Program Templates).
• A bug in V8's serialization API leading to an inconsistent object state (via Program Templates), requiring further interaction to manifest.

These findings collectively demonstrate a significant advancement in JavaScript engine fuzzing, moving beyond purely coverage-guided approaches to incorporate dynamic API understanding and human domain knowledge, enabling the discovery of vulnerabilities that were previously intractable.

Technical Deep Dive

▶ Watch: Human-assisted fuzzing with program templates for complex bugs (24:00)

The technical foundation of Fusili lies in its use of Fuzzilli, an Intermediate Language specifically designed for JavaScript fuzzing. Fuzzilli abstracts away the complexities of JavaScript syntax, representing programs as a sequence of operations like DefineFunction, CreateArray, LoadInt, and CallMethod. This structured representation makes programs highly amenable to automated mutation. For example, splice mutations are particularly powerful: if the fuzzer wants to copy a return V4 instruction, it automatically identifies and copies all dependent instructions (e.g., CallMethod V2, V3) and their arguments, ensuring the copied code remains valid. Fuzzilli also incorporates the concept of contexts, allowing the fuzzer to understand where certain instructions are semantically valid (e.g., a return instruction only inside a function). After mutation, the Fuzzilli is "lifted" into executable JavaScript code, which is then fed to V8.

The core challenge for Fusili, even with advanced splicing, is the combinatorial complexity of finding bugs that require multiple, precise interactions. Smith illustrated this with a classic JavaScript engine bug from 2016 involving Array.prototype.slice and a side effect in a valueOf callback. In this scenario, an evil object with a valueOf method sets an array's length to zero, effectively removing its backing store, while slice continues to operate on the now-invalidated memory, leading to an out-of-bounds access. A traditional fuzzer might discover three distinct code paths: one for defining the evil object, one for shrinking an array, and one for calling slice with an out-of-bounds argument. However, combining these three disparate elements on the same array, in the correct sequence, and with the evil object as the slice argument, requires multiple, intelligent splice and input mutations. Fusili needs to understand variable dependencies, type compatibility, and then perform an input mutation to replace a literal with the evil object. This multi-step process, where each step has hundreds of options, highlights why many complex bugs remain elusive.

To address this, the Probing Mutator introduces dynamic API learning. When Fusili encounters a built-in function, it can insert a temporary JavaScript proxy object as an argument. This proxy acts as a "probe," recording every property access (e.g., object.value) and method call (e.g., object.valueOf()) performed by the built-in function on that argument. This telemetry is fed back to Fusili, which then "learns" the API's argument requirements. For instance, if a built-in function accesses value.valueOf(), Fusili can then generate a concrete object with a valueOf callback, converting the temporary probe into a permanent, intelligent input. This mechanism was instrumental in finding a bug where an ArrayBuffer was resized via a toPrimitive conversion callback, leading to an out-of-bounds access.

The Exploration Mutator complements probing by focusing on the output or capabilities of objects. When Fusili has an object (e.g., the result of a new ArrayBuffer call), it can insert a temporary "exploration call" that uses JavaScript's runtime type introspection (e.g., typeof, Object.keys, Object.getOwnPropertyNames) to discover its properties and methods. This allows Fusili to learn what can be done with the object. This technique was crucial for discovering the Array.prototype.at type confusion bug. The at method was relatively new, and Fusili's environment model didn't initially know about it. By exploring array objects, Fusili discovered the at method, called it, and subsequently found a bug in V8's optimizing compiler that incorrectly handled type checks, leading to out-of-bounds access.

Finally, for bugs requiring an even higher degree of human guidance, Program Templates were introduced. These templates are expressed in Swift, defining the high-level structure of a JavaScript program. For example, a template can specify that a program should have two functions, call a specific "gadget" function (e.g., getObjectIntoInterestingState), and then execute a loop to trigger the optimizing compiler. The key is the build function within these templates, which allows Fusili to "go wild" within a defined scope, iteratively generating and mutating code, splicing from the corpus, and utilizing any variables or objects created so far. This hybrid approach combines the fuzzer's generative power with a researcher's intuition about complex bug patterns, effectively creating "mini-fuzzers" tailored to specific components or attack surfaces. This was demonstrated by converting an existing C++ fuzzer for regular expressions into a Fusili template, which then found an old, persistent V8 bug that the original fuzzer had missed due to Fusili's greater flexibility in exploration.

Demo / Proof of Concept

▶ Watch: Serializer API fuzzer template example and bug discovery (31:00)

While the talk did not feature live demonstrations, Carl Smith effectively presented the "proof of concept" for his advanced fuzzing techniques by detailing several critical vulnerabilities discovered directly through their application. These bug descriptions serve as compelling evidence of the methods' efficacy:

1. ArrayBuffer Resizing Out-of-Bounds (Probing Mutator): The Probing Mutator successfully uncovered a bug related to ArrayBuffer resizing. The scenario involved creating an ArrayBuffer, then passing an object with a toPrimitive conversion callback into a function that processes the ArrayBuffer. This callback would resize the ArrayBuffer to zero while the original operation was still in progress, leading to an out-of-bounds access. This demonstrated the mutator's ability to learn that the API was accessing a primitive conversion method, prompting Fusili to construct an input that exploited this interaction.

1. Array.prototype.at Type Confusion (Exploration Mutator): A particularly impactful bug was a type confusion vulnerability in the Array.prototype.at method, found using the Exploration Mutator. This bug involved calling array.at(large_index) on an object that was not a true array, but was mistakenly treated as one by V8's optimizing compiler. The at method was relatively new, and Fusili's existing environment model didn't account for it. The Exploration Mutator, by introspecting array objects, discovered this new method. Upon calling it with a large index, the optimizing compiler's type checks were bypassed, allowing out-of-bounds access and leading to a highly exploitable primitive. Smith noted that a "really cool exploit" was written for this bug, underscoring its severity.

1. Old V8 Regex Bug (Program Templates): The effectiveness of Program Templates was showcased by converting an existing internal C++ regex fuzzer into a Swift-based Fusili template. This template precisely defined the structure of a program involving regular expressions (e.g., loading a pattern, loading a string, calling exec, and then having Fusili's build function explore further). This hybrid approach generated a sample that exploited a long-standing bug in V8's regex engine that the original C++ fuzzer had failed to find. The flexibility of Fusili's build function to further explore the generated state was crucial for triggering this elusive bug.

1. V8 Serialization API Inconsistent State (Program Templates): Another bug found via Program Templates involved V8's serialization API. The template was designed to fuzz the interaction between serializer.serialize() and serializer.deserialize(). A specific program was generated that serialized a large integer (bigint), then mutated the resulting serialized data, and finally deserialized it. The deserialized object was left in an inconsistent state. Crucially, the bug wouldn't manifest until the inconsistent object was touched by a subsequent operation. This highlights the importance of the build function within templates, allowing Fusili to perform additional, exploratory operations on the result of the deserialization, thus triggering the latent vulnerability.

These examples collectively demonstrate that by augmenting traditional coverage-guided fuzzing with dynamic API learning and human-guided, structured generation, Fusili can uncover a wider range of complex and critical vulnerabilities in highly optimized JavaScript engines.

Defensive Implications

▶ Watch: Q&A: Fuzzer evaluation, effectiveness, and restarting strategies (42:00)

The advancements in JavaScript engine fuzzing presented by Carl Smith offer several critical defensive implications for developers and security teams working on browsers and other JavaScript-intensive applications:

1. Prioritize New APIs and Features: The discovery of the Array.prototype.at bug highlights that new or recently changed APIs are often ripe for vulnerabilities. Optimizing compilers, in particular, may struggle to correctly apply all necessary checks to new language features, creating opportunities for type confusion or other memory safety issues. Defenders should prioritize rigorous security reviews and targeted fuzzing for newly introduced JavaScript features or modifications to existing ones.

1. Beware of Side Effects in Callbacks: The classic valueOf callback bug and the ArrayBuffer resizing vulnerability underscore the danger of unexpected side effects within callbacks during built-in function execution. Any JavaScript API that accepts user-defined callbacks or objects with toPrimitive, valueOf, or toString methods should be treated as a high-risk area. Developers must ensure that internal operations are robust against state changes initiated by these callbacks, especially concerning underlying data structures like array backing stores.

1. Embrace Targeted Fuzzing for Complexity: Smith's work strongly advocates for moving beyond purely generic, coverage-guided fuzzing. When specific components (e.g., optimizing compilers, serialization logic, garbage collectors) are known to be complex or historically bug-prone, human-assisted fuzzing with program templates offers a powerful defensive tool. Security teams should leverage their domain knowledge to craft templates that target these specific areas, significantly increasing the chances of finding deep-seated bugs that generic fuzzers might miss.

1. Investigate Data Flow and State Inconsistency: The serialization bug, which only manifested after further interaction with an inconsistently deserialized object, emphasizes the importance of understanding data flow and how operations can leave objects in a vulnerable state. Defenders should look for scenarios where an object's internal state can be manipulated or become inconsistent without immediately crashing, as these latent bugs can be exploited later. Future fuzzing feedback mechanisms that track data flow more precisely could be invaluable here.

1. Continuous, Sophisticated Fuzzing is Essential: Even for mature and heavily fuzzed engines like V8, sophisticated techniques like Probing, Exploration, and Program Templates continue to uncover critical vulnerabilities. This indicates that the search space for JavaScript engines is vast, and attackers are constantly seeking these complex, multi-step exploits. Organizations developing browser engines or similar high-performance runtimes must maintain a continuous, evolving fuzzing strategy that incorporates the latest advancements in fuzzer intelligence and guidance.

1. Understand the Optimizing Compiler's Role: Multiple bugs discussed (e.g., Array.prototype.at, the in-the-wild bug requiring an optimizing compiler trigger) highlight the optimizing compiler as a frequent source of complex vulnerabilities. These components are designed for performance and often involve intricate assumptions and optimizations that can be bypassed with carefully crafted inputs. Deep understanding and targeted fuzzing of optimizing compiler logic are crucial defensive measures.

Key Takeaways

• The JavaScript engine search space is immensely vast, making comprehensive fuzzing a significant challenge, even with abundant compute resources.
• Traditional coverage-guided fuzzing, while effective, struggles to find bugs requiring complex, multi-step sequences or specific data interactions that don't immediately increase code coverage.
• Specialized mutators like Probing and Exploration enable fuzzers to dynamically learn API argument usage and object capabilities, leading to the discovery of vulnerabilities related to side effects and new features.
• Human-assisted fuzzing through Program Templates empowers researchers to guide the fuzzer towards specific, high-complexity code shapes and known bug patterns, drastically improving targeted fuzzing efficiency.
• Even for highly mature and well-fuzzed engines like V8, these advanced, targeted fuzzing techniques continue to uncover critical and long-standing vulnerabilities.
• Future advancements in fuzzing may involve new feedback mechanisms that track data flow more precisely, helping to bridge the gap between individual code coverage and the meaningful combination of operations.

About the Speaker(s)

Carl Smith is a Security Engineer on the V8 Security team at Google, based in Switzerland. His work focuses on enhancing the security of the V8 JavaScript engine, the core component powering Google Chrome and other applications. Smith specializes in advanced fuzzing techniques and vulnerability research within complex software environments. He expressed a keen interest in compilers, JIT (Just-In-Time) engines, and browsers in general, reflecting his deep technical expertise in these areas. His presentation at HITCON CMT was his first time in Taiwan, and he aimed to inspire other researchers to engage in V8 fuzzing.