HyperMirage: Direct State Manipulation in Hybrid Virtual CPU Fuzzing

HyperMirage: Direct State Manipulation in Hybrid Virtual CPU Fuzzing

Speakers: Manuel Andreas

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=eUAZbEaN6C0

Overview

Hypervisors form the cornerstone of cloud security, and while fuzzing has proven effective at finding bugs in device virtualization interfaces, the virtual CPU (vCPU) component -- typically implemented in kernel space and thus more security-critical -- has received far less attention. This talk introduces HyperMirage, a hybrid fuzzing approach that uses direct state manipulation to make a hypervisor handle VM exits that never actually occurred, combined with a novel bare-metal symbolic execution runtime built on SIMCC. The result is a fuzzer that achieves nearly 100x throughput improvement over prior art, covers the majority of VM exit handlers (not just a manually-selected subset), and discovered 11 new bugs in Xen and KVM, including 4 CVEs with security-critical impact.

Presented by Manuel Andreas, the work targets the Intel VMX platform and demonstrates that prior work in hypervisor fuzzing only covered a fraction of the virtual CPU attack surface because crafting valid VM states for each exit reason required enormous manual effort. HyperMirage eliminates this bottleneck entirely by directly setting VMCS fields, general-purpose registers, and VM memory to arbitrary values, ignoring architectural constraints during fuzzing and handling false positives during triage.

Background

▶ Watch: Motivation: why virtual CPUs are underexplored in hypervisor fuzzing (0:00)

Under Intel VMX virtualization extensions, a hypervisor manages virtual machines through a Virtual Machine Control Structure (VMCS) -- a hardware structure that configures the VM's initial state (guest state fields), isolation properties (control fields), and receives information about VM exits (exit information fields). The lifecycle works as follows: the hypervisor configures the VMCS, launches the VM via VM entry, the CPU natively executes VM code, and when the VM performs a privileged operation, the CPU triggers a VM exit -- a hardware trap that transfers control back to the hypervisor.

During a VM exit, the CPU implicitly stores the VM's current state in the VMCS guest state fields and populates VM exit information fields to tell the hypervisor what caused the exit. The hypervisor then fetches these fields and uses a large switch-case statement over the exit reason (an integer identifier) to decide how to handle the event -- emulating instructions, denying access, servicing hypercalls, etc.

The problem with existing virtual CPU fuzzers is that they require manually crafted seed states: for the CPU to generate a specific VM exit reason, the virtual machine needs to be in a precise architectural state and execute a specific instruction. With the Intel manual spanning thousands of pages, creating valid states for each exit reason is an enormous manual undertaking. Prior work (e.g., Hyperpill) only covered a small, manually-selected subset of VM exit reasons.

Key Findings

▶ Watch: Direct state manipulation: making hypervisors see a mirage (4:00)

HyperMirage's core innovation is direct state manipulation: instead of trying to coerce a real VM into producing specific VM exits, the fuzzer directly sets the VMCS fields, general-purpose registers, and VM memory to arbitrary values, making the hypervisor handle an artificial VM exit that it thinks happened but never actually occurred. This eliminates the need for manual seed crafting and enables fuzzing of all VM exit reasons, not just a manually-selected subset.

The approach dramatically outperforms prior art:

• Coverage: Drastically more edge coverage than Hyperpill across both Xen and KVM, covering VM exit handlers that were previously unreachable
• Throughput: Nearly 100x faster than Hyperpill, because HyperMirage runs everything natively rather than on QEMU's CPU emulator
• Bug finding: 11 new bugs discovered (4 security-critical CVEs) in already battle-tested hypervisors

A critical finding was that one CVE was only discoverable through direct state manipulation because it required an architectural edge case not described in the Intel manual: transitioning from 64-bit long mode to 32-bit mode while retaining 64-bit values in the upper halves of general-purpose registers, then triggering a hypercall. The hypervisor panicked because it didn't expect 32-bit guests to have non-zero upper register halves. This bug would have been missed by any approach that enforced known architectural constraints.

Interestingly, the other bugs were found in VM exit handlers that had already been fuzzed by prior work, demonstrating that HyperMirage's superior coverage and symbolic execution can find bugs even in previously-explored code paths.

Technical Deep Dive

▶ Watch: SIMCC record-and-replay runtime for bare-metal symbolic execution (8:00)

The fuzzing architecture uses nested virtualization: a host hypervisor runs the target hypervisor (Xen or KVM) inside a VM, which in turn spawns its own VM (the "VM agent"). Because the target hypervisor is itself virtualized, the host hypervisor has full control over the target's virtual CPU state.

Fuzzing inputs model three components that influence VM exit handling code:

1. Full VMCS -- all guest state fields and VM exit information fields (including the exit reason)
2. General-purpose registers -- all register values
3. 512 bytes of memory -- repeated to fill an entire page, covering any memory accesses the hypervisor makes during VM exit handling

To apply a fuzzing input, the host hypervisor writes the VMCS and register values directly to the target hypervisor's virtual CPU, and remaps all VM agent physical pages (via second-level address translation) to point to the memory bytes from the fuzzing input. Then an artificial VM exit is injected, and the target hypervisor handles it as if it were real.

The hybrid fuzzing combines coverage-guided greybox fuzzing (using Intel Processor Trace for coverage) with symbolic execution via SIMCC -- a compiler-based instrumentation tool. A novel record-and-replay runtime for SIMCC enables symbolic execution on bare-metal targets: instead of performing constraint solving inside the hypervisor (which lacks a standard library, memory allocator, etc.), symbolic operations are recorded during execution and replayed later in a managed environment where Z3 can be linked in for actual constraint solving.

Demo / Proof of Concept

▶ Watch: Performance: 100x throughput over Hyperpill (10:00)

The prototype was built for the Intel VMX platform targeting Xen and KVM. In 24-hour fuzzing runs, HyperMirage drastically outperformed Hyperpill in edge coverage for both hypervisors. The symbolic execution component (hybrid mode) provided significantly more coverage than greybox-only mode, demonstrating its value in systematically exploring deep code paths in VM exit handlers.

The 11 discovered bugs include 4 security-critical CVEs. The edge case CVE (64-bit to 32-bit mode transition with non-zero upper register halves) is particularly notable: it exists in the Intel architecture's specification ambiguity, not in an obviously buggy code path. The EP violation VM exit handler dominates coverage due to its massive instruction emulator, but bugs were found across multiple exit reasons.

The prototype will be open-sourced (placeholder repository exists, expected within approximately two weeks of the talk). The record-and-replay SIMCC runtime is reusable for fuzzing other bare-metal targets beyond hypervisors.

Defensive Implications

▶ Watch: 11 new bugs, 4 CVEs, and the architectural edge case discovery (14:00)

For cloud providers and organizations running hypervisor-based infrastructure, this research has several implications:

1. Xen and KVM both have security-critical bugs in their virtual CPU implementations, even after years of security scrutiny. Organizations should ensure they are running patched versions that address the four CVEs discovered.

1. The virtual CPU attack surface is larger than commonly assessed. Most hypervisor security testing focuses on device virtualization (user-space), but virtual CPU bugs are more impactful because they execute in kernel space. Security teams should advocate for virtual CPU fuzzing as part of their hypervisor security program.

1. Architectural edge cases create bugs that are not detectable by approaches that enforce known CPU constraints. The 64-bit-to-32-bit transition CVE demonstrates that hypervisor developers cannot assume guests will behave according to the Intel manual.

1. The SIMCC record-and-replay runtime is applicable beyond hypervisors to any bare-metal target needing symbolic execution, suggesting a broader tooling improvement for firmware and kernel security testing.

Key Takeaways

• Direct state manipulation eliminates the manual effort bottleneck in virtual CPU fuzzing by making hypervisors handle artificial VM exits that never actually occurred
• HyperMirage achieves nearly 100x throughput over prior art (Hyperpill) and covers the majority of VM exit handlers, not just a manually-selected subset
• 11 new bugs found in Xen and KVM (4 security-critical CVEs), including one that was only discoverable by ignoring architectural constraints
• The SIMCC record-and-replay runtime enables symbolic execution on bare-metal targets without managed runtime dependencies
• Bugs were found even in VM exit handlers that prior work had already fuzzed, demonstrating the value of deeper exploration
• The prototype will be open-sourced for the Intel VMX platform

About the Speaker(s)

Manuel Andreas presented the work with deep technical fluency, demonstrating detailed knowledge of both the Intel VMX architecture and the fuzzing infrastructure. He engaged thoughtfully with audience questions about false positive management, explaining the deliberate trade-off of accepting false positives during fuzzing to avoid over-specifying architectural constraints that could cause real bugs to be missed. He described the project as built on top of the Nyx fuzzing framework and confirmed plans to open-source the prototype.