Bug Tamer: Turning Limited Heap Overflow into Full VMware Escape
Yongkang Jia (Security Researcher · Singular Security Lab), Xiao Lei (Security Researcher · Singular Security Lab)
POC2025 (Power of Community) · Main Stage
Overview
This talk, "Bug Tamer: Turning Limited Heap Overflow into Full VMware Escape," delivered by Yongkang Jia and Xiao Lei from Singular Security Lab at POC, delves into the intricate process of escalating a constrained heap overflow vulnerability within VMware products into a full VMware Escape. The researchers meticulously detail their journey from identifying a subtle time of check to time of use (TOCTOU) bug in VMware's USB device emulation to leveraging it for arbitrary code execution in the host operating system. Their work is a compelling demonstration of how seemingly minor vulnerabilities, when chained with sophisticated exploitation techniques, can lead to critical security bypasses in complex virtualization environments.

Key moments
- 1:40 Why VMware is a compelling target
- 8:00 Initial Time-of-Check to Time-of-Use (TOCTOU) bug
- 10:40 Exploiting TOCTOU for limited OOB read/write
- 14:00 Locating the exploitable virtual video device
- 17:30 ASLR bypass and VMware base address leakage
- 20:00 Hijacking control flow for arbitrary function calls
- 28:00 Critical discovery: Fixed guest RAM address on M1
Bug Tamer: Turning Limited Heap Overflow into Full VMware Escape
Speakers: Yongkang Jia (Security Researcher, Singular Security Lab); Xiao Lei (Security Researcher, Singular Security Lab)
Conference: POC
YouTube: https://www.youtube.com/watch?v=Ulwmbmh5g1o
Overview
This talk, "Bug Tamer: Turning Limited Heap Overflow into Full VMware Escape," delivered by Yongkang Jia and Xiao Lei from Singular Security Lab at POC, delves into the intricate process of escalating a constrained heap overflow vulnerability within VMware products into a full VMware Escape. The researchers meticulously detail their journey from identifying a subtle time of check to time of use (TOCTOU) bug in VMware's USB device emulation to leveraging it for arbitrary code execution in the host operating system. Their work is a compelling demonstration of how seemingly minor vulnerabilities, when chained with sophisticated exploitation techniques, can lead to critical security bypasses in complex virtualization environments.
The significance of this research lies in its target: VMware, a perennially high-value target in virtualization security competitions like Pwn2Own. Unlike open-source hypervisors such as QEMU or VirtualBox, VMware's closed-source nature presents significant reverse engineering challenges, making successful exploits particularly noteworthy. Jia and Lei's presentation not only uncovers a practical exploit path but also shares valuable methodologies for tackling such formidable targets, including advanced heap manipulation on macOS and novel ASLR bypass techniques applicable to both Apple Silicon and Intel-based Macs.
Their research culminates in a successful VMware Fusion escape on both M1 Pro and Intel platforms, demonstrating the profound impact of their findings. The talk serves as a critical resource for security researchers, penetration testers, and system administrators seeking to understand the intricacies of hypervisor exploitation and the defensive strategies required to mitigate such threats. By transforming a limited memory corruption primitive into a complete system compromise, the Singular Security Lab team underscores the importance of deep architectural understanding and creative problem-solving in the realm of virtualization security.
Background
▶ Watch: Why VMware is a compelling target (1:40)
VMware products, including VMware Fusion and ESXi, are foundational components of many IT infrastructures, making their security paramount. Historically, hypervisor escapes have been a coveted achievement in the security community, offering full control over the host system from a compromised guest virtual machine. The speakers chose VMware for their research due to its consistent high-value status in security competitions, the availability of prior public research, and crucially, its closed-source nature, which presents a significant reverse engineering challenge. This challenge, they noted, made the endeavor particularly appealing.
Their research methodology began by leveraging public knowledge, specifically a finding from Black Hat USA 2024 that ESXi unified releases contain debugging symbols, greatly aiding reverse engineering efforts. They also followed advice from Hexacon 2024, emphasizing that reproducing and analyzing known vulnerabilities is an effective way to accelerate learning and potentially uncover new, similar issues. This led them to focus on USB devices emulation within VMware, a common attack surface for virtualization escapes. They specifically sought to reproduce a vulnerability demonstrated at Pwn2Own 2023, which targeted a virtual Bluetooth device implemented over USB.
VMware supports three types of USB controllers: UHCI, EHCI, and XHCI. When a guest OS communicates with a USB device, it sends a USB packet to the virtual USB controller, which then processes these into USB Request Blocks (URBs). These URBs are subsequently passed to specific USB backend devices (e.g., Bluetooth, Smart Card, HID, or virtual video) for higher-level protocol processing. Vulnerabilities frequently arise at two critical points: within the USB controller itself, which parses raw USB packets from the guest OS, and within the USB backend devices, which interpret the higher-level protocols embedded in the URBs. For UHCI and EHCI controllers, data transfer from guest to host utilizes Transfer Descriptions (TDs) and Queue Heads (QHs), whose structures and data are fully controllable by the guest, making them prime targets for manipulation.
Key Findings
▶ Watch: Exploiting TOCTOU for limited OOB read/write (10:40)
The researchers' initial plan involved reverse engineering USB packet parsing in UHCI and EHCI controllers, then delving into the Bluetooth backend, and finally discovering new vulnerabilities. However, they stumbled upon an unexpected discovery almost immediately: a time of check to time of use (TOCTOU) bug within the UHCI and EHCI handlers, specifically in the HCQ function. This classical race condition occurs because the setup packet data is read twice.
Here's the sequence of events leading to the TOCTOU:
- The
HCQfunction first reads theWlen(length) field from the guest-controlled setup packet. - It then allocates a URB object whose size is calculated based on this
Wlen(specifically,0x98plusWlenplus8bytes). - Crucially, it reads the setup packet data a second time to store it into the newly allocated URB data buffer.
This two-stage read creates a window where an attacker in the guest VM can modify the Wlen value between the first read (for allocation) and the second read (for data copying). If the Wlen is changed to a larger value after allocation, it results in a heap overflow. If changed to a smaller value, it could lead to a heap underflow. This TOCTOU bug was found to exist in both UHCI and EHCI devices.
While the TOCTOU provided a powerful primitive, its immediate exploitability was not straightforward. The team then focused on how URBs are processed by various USB backend devices (Bluetooth, HID, virtual video) to find a way to leverage this bug for memory corruption. They eventually found an interesting opportunity in the virtual video device. This device, when processing URBs, offered a path to achieve a limited out-of-bounds (OOB) read and OOB write primitive. The maximum allowed Wlen for this specific vulnerability was 0x22 bytes.
The OOB read/write primitive works as follows:
- OOB Read:
- A first URB (
URB0) is crafted withWleninitially set to0. This means the extra data buffer forURB0is allocated with zero length. - The TOCTOU vulnerability is then exploited to modify
Wlenfrom0to0x22afterURB0's allocation. - The virtual video device copies
0x22bytes into its UVC parameter structure, effectively reading0x22bytes out-of-bounds from the heap memory immediately followingURB0.
- OOB Write:
- A first URB (
URB0) is crafted with a legitimateWlenof0x22. The virtual video device allocates a properly sized buffer and safely copies data. - A second URB (
URB1) is crafted withWleninitially set to0. - The TOCTOU vulnerability is exploited to modify
Wlenfrom0to0x22afterURB1's allocation. - When the virtual video device processes
URB1, it attempts to write0x22bytes into its UVC parameter structure using the now-manipulatedWlen, triggering an OOB write of0x22bytes.
A critical challenge arose: the virtual video device is not initialized in VMware Workstation or ESXi, despite the code existing in the binaries. After extensive testing, they discovered it was only activated in VMware Fusion on macOS when a compatible USB video periphery was connected. This made VMware Fusion on macOS their viable exploitation target.
Technical Deep Dive
▶ Watch: Locating the exploitable virtual video device (14:00)
With a limited 0x22-byte OOB read/write primitive in hand, the next phase involved escalating this into a full VMware Escape on macOS. This required overcoming ASLR (Address Space Layout Randomization) and achieving control flow hijacking.
ASLR Bypass via Information Disclosure
The initial 0x12-byte OOB read (from the URB object) proved insufficient for direct information leakage to bypass ASLR, as the VMware process memory layout was largely unknown. The researchers devised an ingenious method combining the limited OOB write with URB objects themselves to achieve effective information disclosure.
- Heap Spray and Layout: The strategy relies on carefully manipulating the heap to place two
URBobjects,URB0andURB1, adjacently in memory. actual_lenManipulation: TheURBstructure contains a critical member variable,actual_len, at offset0x0c. This field specifies the length of response data within theURBthat is readable by the guest OS.- OOB Write to
actual_len: By exploiting the0x22-byte OOB write onURB0, the researchers could overwriteURB0'sactual_lenfield. For example, settingactual_lento0x98would trick the emulated USB device into believing thatURB0's valid extra data field now extends0x98bytes beyond its original boundary. - Leaking
URB1: IfURB1is placed immediately afterURB0(due to heap manipulation), thenURB1's entire object structure effectively becomes the "extra data" field ofURB0. - Base Address Leak: The
URBstructure also contains a data pointer that, duringURBcreation, points to a specific location in the BSS section of the VMware process. By submitting a USB transfer descriptor with a specific packet identification type, the emulated USB device copiesURB0's "extra data" (which isURB1's object) into guest memory. This process successfully leaks the base address of the VMware process, thus bypassing ASLR.
Control Flow Hijacking
Once ASLR was bypassed, the goal was to hijack control flow for arbitrary code execution. The researchers identified the vUSB_CancelPipe function within the emulated USB device code as a suitable target. This function is called when the guest OS submits two setup transfer descriptors with inconsistent setup data.
vUSB_CancelPipeParameter: ThevUSB_CancelPipefunction takes a single parameter,pipe, which originates from thepipepointer variable within theURBobject itself, located at offset0x18.- Hijacking
pipePointer: By using the0x22-byte OOB write primitive, thepipepointer within aURBobject can be overwritten to point to a crafted pipe object controlled by the attacker. - Arbitrary Function Call: The crafted
pipeobject can then contain an attacker-controlledop_cancel_pipefunction pointer. WhenvUSB_CancelPipeis invoked, it will indirectly call this hijackedop_cancel_pipewith a fully controllable first parameter (also derived from the craftedpipeobject). This provides a powerful primitive for making arbitrary function calls.
Heap Manipulation on macOS (libmalloc Nanozone)
A significant challenge was the lack of prior knowledge regarding macOS user-space heap management (libmalloc) implementation. The researchers analyzed libmalloc's internals, focusing on the nanozone, which handles small allocations (from 0x10 to 0x100 bytes) relevant to URB objects.
- Nanozone Structure: The nanozone pre-allocates a
512MBregion, divided into eight64MBarenas. Each arena is split into409616KBblocks (Apple's page size). - Block Types: Blocks are categorized as data blocks (holding user data) or metablocks (managing allocation state). Notably, block zero of each arena is the only metablock.
- Randomization: Due to XOR-based shuffling of block indices, the memory layout of blocks is randomized. However, slots (the smallest allocation units within a block) have non-shuffled indices.
- Multicore Optimization:
libmalloc's multicore optimizations introduce complexity: different CPU cores use distinct blocks for even identical-sized allocations, making heap layout unpredictable across cores. - Free List: Freed slots are managed by a free list, following a Last-In, First-Out (LIFO) order. A
double_free_guard(XOR result of slot address and a random cookie) is used to detect double-free situations.
To ensure reliable adjacent allocation for information disclosure and control flow hijacking, the researchers developed a specific heap spray strategy:
- Initial Spray: Spray
URBobjects to ensure every CPU core's active block has at least one slot allocated. - Contiguous Allocation: Allocate three
URBs(URB0,URB1,URB2) expecting them to reside contiguously within a single core's block. - Freeing in Order: Free these
URBsin the orderURB0,URB1,URB2. This populates the free list with slots 3, then 2, then 1 (LIFO). - Reallocation and Reversal: Reallocate three
URBs. Due to LIFO, the newestURB(e.g., the one that wasURB2's slot) occupies the lowest address, effectively reversing their memory order. This ensuresURB2hasURB1adjacent in memory during processing. - Single USB Request: Crucially, steps 2-4 must execute within a single USB request to minimize the chance of CPU core switching, which would disrupt the contiguous allocation.
This setup allows triggering the OOB write on URB2 to overwrite URB1's actual_len for information disclosure, and similarly, to overwrite URB1's pipe pointer with the crafted object for control flow hijacking.
Finding a Home for the Crafted Pipe Object (Guest RAM)
Initially, finding a reliable location for the crafted pipe object was difficult. However, a critical discovery simplified this significantly: on Apple Silicon (M1) Macs, the guest RAM's virtual address within the VMware Fusion process was consistently fixed at 0x280000000.
MacVMAllocate: Further reverse engineering revealed that VMware Fusion allocates guest RAM using theMacVMAllocateAPI, similar tommapon Linux. Tests confirmed that sufficiently large allocation requests consistently returned this fixed address.- Exploiting Fixed Address: This meant that crafted
pipeobjects could be placed directly in guest RAM. TheURB'spipepointer could then be hijacked to0x280000000plus the guest physical address of the crafted object. This eliminated the need for additional address leakage for control flow hijacking on M1 Macs.
Intel-based Macs: ASLR Bypass via Address Collision
For Intel-based Macs, the guest RAM's virtual address is protected by ASLR. However, the researchers found that the offset between the VMware process base address and the guest RAM's virtual address only changed within a limited range (approximately 0x5 million to 0x6 million bytes).
- Address Collision Strategy: Since the guest RAM is typically several gigabytes, its size is significantly larger than this
0x1million byte variance. The strategy involved populating a16MBregion of guest RAM with craftedpipeobjects at every page boundary. Regardless of the actual offset, the hijackedpipepointer would always "collide" with and hit a valid craftedpipeobject within this densely populated region. This effectively bypassed guest RAM's ASLR on Intel Macs, allowing the same M1 methodology for control flow hijacking to be applied.
Demo / Proof of Concept
▶ Watch: Hijacking control flow for arbitrary function calls (20:00)
The researchers demonstrated their exploit on two distinct platforms: a MacBook M1 Pro and an Intel-based Mac, both running VMware Fusion version 13.5.2.
For the MacBook M1 Pro demonstration, the exploit successfully popped a calculator on the host macOS, signifying a full VMware escape. A notable detail for the M1 demo was the requirement of manually adding an extra virtual video device. This is because M1 Macs lack native UHCI devices, and other USB devices are routed through XHCI, whose implementation details the researchers had not fully reverse-engineered. The vulnerability, however, does not inherently require two virtual video devices; it was a workaround for the demo environment.
The Intel-based Mac demonstration, presented via a recorded video, also showed a calculator popping up on the host, confirming the successful escape on that architecture as well, leveraging the address collision technique for ASLR bypass. The Intel Macs, having native UHCI devices, did not require the additional virtual video device.
These demonstrations served as clear proof of concept for the robust nature of their exploitation framework, showcasing its adaptability across different macOS architectures despite varying ASLR implementations.
Defensive Implications
▶ Watch: Critical discovery: Fixed guest RAM address on M1 (28:00)
The research presented by Yongkang Jia and Xiao Lei highlights several critical defensive implications for users and administrators of VMware products, particularly VMware Fusion:
- Prompt Patching and Updates: The most immediate defensive action is to ensure all VMware products, especially VMware Fusion, are kept up-to-date with the latest security patches. Although the bug was fixed during their exploitation process, similar vulnerabilities can emerge, and timely patching is crucial to mitigate known attack vectors.
- Principle of Least Privilege: Hypervisors and virtual machines are complex systems. Adhering to the principle of least privilege for guest operating systems and their applications can limit the impact of a successful escape. Restricting USB device access or specific USB device types for sensitive VMs could add a layer of defense.
- Monitoring and Anomaly Detection: Host systems running virtual machines should be rigorously monitored for unusual activity, especially related to memory access patterns, process creation, or network connections originating from the hypervisor process (e.g.,
vmware-vmx). Unusual guest OS behavior that might indicate attempts to manipulate virtual hardware, such as USB devices, should also be flagged. - Secure Configuration: Reviewing and hardening VMware configurations, such as disabling unnecessary virtual hardware or features, can reduce the attack surface. For example, if virtual video devices are not required, ensuring they are not initialized or accessible can prevent exploitation paths that leverage them.
- Understanding Attack Surfaces: This talk underscores that virtual hardware, particularly complex components like USB controllers and their backends, remains a fertile ground for vulnerabilities. Security teams should prioritize auditing and understanding the security posture of these components in their virtualization deployments.
Key Takeaways
- Reproducing and analyzing known vulnerabilities can effectively lead to the discovery of new, similar issues, providing a strong foundation for advanced security research.
- Subtle time of check to time of use (TOCTOU) race conditions, even in seemingly robust systems, can be leveraged to create powerful memory corruption primitives like heap overflows.
- Closed-source software, despite its challenges, offers unique opportunities for deep security research, with debugging symbols and existing public research proving invaluable for initial reverse engineering efforts.
- Escalating limited memory corruption primitives (e.g., a
0x22-byte OOB write) into full VMware Escapes requires sophisticated heap manipulation, ASLR bypass, and control flow hijacking techniques. - Understanding the specifics of host OS memory management (e.g.,
libmalloc's nanozone on macOS) is crucial for reliable heap exploitation, including strategies like heap spraying and careful freeing/reallocation to achieve desired memory layouts. - Novel ASLR bypass techniques, such as exploiting fixed guest RAM virtual addresses on Apple Silicon or using address collision for a limited range of ASLR variance on Intel Macs, are essential for achieving reliable exploits across different architectures.
About the Speaker(s)
Yongkang Jia and Xiao Lei are Security Researchers from Singular Security Lab. Their research primarily focuses on virtualization security, encompassing various hypervisors such as QEMU, KVM, VirtualBox, and VMware. Over the past three years, they have specifically concentrated on virtual I/O devices, achieving three QEMU escape exploits which they have shared at previous security conferences. Their work demonstrates a deep expertise in identifying and exploiting vulnerabilities within complex virtualization environments.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
Real VMware escape research with a novel TOCTOU-to-escape chain, complete heap manipulation framework for macOS libmalloc, and working demos on both M1 and Intel. This is what Pwn2Own-grade work looks like when the researchers actually explain the machinery.
Heather Calloway (CISO) — STRONG ACCEPT
This is serious hypervisor escape research on a high-value target. Any CISO running VMware Fusion in their environment—particularly for executive devices or sensitive workloads—needs to understand that guest-to-host escape is a live capability, not a theoretical risk. The work is technically dense but the strategic implication is clear: your VM boundary is not your trust boundary.