Timekiller: Escape From QEMU/KVM

Timekiller: Escape From QEMU/KVM

Speakers: Yongkang Jia (Zhejiang University), Xiao Lei (Zhejiang University), Yiming Tao (Zhejiang University), Gaoning Pan (Zhejiang University), Chunming Wu (Security Researcher, Zhejiang University)

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=Zm32_ekspEc

Overview

In a compelling presentation at HITB Phuket, researchers from Zhejiang University unveiled "Timekiller," a novel exploitation technique targeting QEMU/KVM. The talk, delivered by Yongkang Jia, Xiao Lei, and Chunming Wu, detailed their journey into a previously underexplored attack surface within QEMU: its asynchronous mechanisms. Their research culminated in the discovery of multiple critical vulnerabilities within the virtio-crypto device, including a potent heap-based buffer overflow, which they successfully leveraged to achieve a guest-to-host escape.

The significance of this research cannot be overstated. QEMU/KVM forms the backbone of countless cloud environments and virtualized infrastructures globally. As traditional attack surfaces become increasingly hardened, security researchers are compelled to explore more intricate and subtle vulnerabilities. The "Timekiller" exploit demonstrates a sophisticated understanding of QEMU's internal workings, specifically how its asynchronous clock and throttling mechanisms can be weaponized to manipulate memory and bypass security mitigations like ASLR, ultimately leading to arbitrary code execution on the host. This work not only highlights the persistent challenges in securing complex virtualization platforms but also provides a blueprint for future vulnerability research in similar asynchronous contexts.

Background

▶ Watch: Introduction to QEMU/KVM and research background (0:00)

QEMU/KVM serves as a cornerstone of modern virtualization, with QEMU operating in user mode to provide device virtualization and KVM residing in kernel mode for efficient CPU, memory, and interrupt virtualization. For years, security researchers have diligently probed QEMU for vulnerabilities, making it increasingly challenging to discover new flaws on conventional attack surfaces. This necessitates a shift towards exploring less obvious vectors, such as GPU virtualization, race conditions, or identifying novel exploit primitives and useful internal structures.

The research team recognized that race conditions and asynchronous mechanisms could represent a fertile ground for new vulnerabilities. Asynchronous operations are ubiquitous in complex systems like QEMU, designed to prevent blocking and enhance responsiveness. These are typically implemented through multithreading or, critically for this research, through timers, often referred to as an asynchronous clock. QEMU's main thread manages events, callbacks, and includes a sophisticated timer system. A qemu_timer object allows a callback function to be invoked after a specified time interval. QEMU categorizes these timers into several clock types: realtime (runs when VM is stopped), watchdog (runs when VM is running), host (runs even when VM is suspended but is sensitive to system clock changes), and realtime_co (for I/O accounting). These timers are managed within qemu_timer_list groups, which maintain pointers to active timers for each clock type. Beyond their role in handling asynchronous requests for devices like network, USB, or disk, qemu_timer objects can also be exploited, most commonly by hijacking their callback functions.

Another key component in QEMU is its throttle mode, which imposes limits on operations, such as bytes per second (BPS) or operations per second (OPS) for disk I/O. This mechanism, designed for resource management, proved to be instrumental in the "Timekiller" exploit by allowing precise control over the timing of asynchronous operations.

The researchers chose to focus their efforts on the virtio-crypto device, a virtual cryptography device within the virtio framework. This device provides a unified interface for various cryptographic services. Their selection was driven by several factors:

1. Widespread Use: Cryptography is fundamental in cloud environments, protecting data in transit and at rest.
2. Continuous Updates: The virtio-crypto device is actively maintained, with recent additions like the RSA algorithm and, significantly, support for an asynchronous mode introduced in November 2022. New features often introduce new bugs.
3. Lack of Security Research: The device had seen little security scrutiny in recent years, with the last public CVE related to virtio-crypto dating back to 2017.
4. Asynchronous Nature: Its inherent support for asynchronous operations aligned perfectly with their research hypothesis regarding this new attack surface.

The virtio-crypto device communicates with the guest via two types of queues: a Control Queue for managing symmetric and asymmetric sessions, and a Data Queue for transporting cryptographic service requests. It supports symmetric encryption (e.g., AES) and asymmetric encryption (e.g., RSA, with future plans for DSA and ECDSA). Guest drivers (user or kernel space) interact with a unified host interface that, in turn, calls underlying software libraries like QCrypto, libgcrypt, or OpenSSL. Crucially, virtio-crypto supports two operational modes:

• Synchronous Mode: Requests are processed immediately, and memory chunks are allocated and freed promptly.
• Asynchronous Mode: Requests are added to a timer queue, and encryption/decryption operations are deferred until triggered by the clock. In this mode, relevant data structures remain in memory for a longer duration, as chunks are merely blocked and used, not immediately freed. This asynchronous behavior can be enabled by configuring throttle-bps and throttle-ops parameters. This seemingly minor difference proved to be a critical enabler for the "Timekiller" exploit.

Key Findings

▶ Watch: Virtio Crypto device: chosen attack surface (8:00)

The research team embarked on a fuzzing campaign targeting the virtio-crypto device, leveraging their hypothesis about asynchronous mechanisms. They adapted an existing fuzzer, wehle, which was originally designed for DMA devices. To make wehle effective for virtio-crypto, they implemented several key improvements:

1. Initialization Operations: Added specific initialization routines to trigger virtio-crypto control and data queue handlers, including creating virtual buffers with correct addresses and initializing virtio-crypto via virtio_pci_common_write functions.
2. I/O Buffer Hooking: Hooked the I/O buffer for transport fuzzing seeds.
3. Log Redirection: Redirected QEMU's log information to a file to monitor stack frames upon crashes.

This customized fuzzing setup proved highly effective, quickly yielding a high coverage in the virtio-crypto device. Over a few days of fuzzing, the team discovered and reported four distinct vulnerabilities, all found within the virtio-crypto device:

1. Null Pointer Dereference in virtio_crypto_free_request: This vulnerability occurs in the function responsible for cleaning up and freeing request data structures after encryption or decryption. The code attempts to memset and g_free the op_info structure without first checking if op_info holds a non-null value. This can be triggered in two main ways:

• If the encryption type is set to anything other than PL_CIPHER or ALGORITHM_CHAIN, the op_info allocation fails, returning NULL.
• If an excessively large length is provided for op_info, leading to an allocation failure and a NULL op_info.

In both scenarios, a NULL op_info is passed to memset and g_free, resulting in a denial-of-service (DoS) due to a null pointer dereference.

1. Null Pointer Dereference in virtio_crypto_d_backend_cb_account: This issue arises when a guest requests an RSA cryptographic service, but the QEMU host was compiled without the necessary RSA library (e.g., libgcrypt). The virtio_crypto_d_backend_cb_account function, intended to account for encrypted bytes, attempts to access a structure that holds the byte count. If the RSA backend library is missing, this structure is NULL. Consequently, accessing its members in the virtio_crypto_d_backend_cb_account function leads to a null pointer dereference and a DoS.

1. Null Pointer Dereference in virtio_crypto_d_build_op: This vulnerability concerns an inconsistent matching between the chosen encryption/decryption algorithm and the session type. The virtio_crypto_d_build_op function is triggered during symmetric encryption/decryption. QEMU's built-in backend supports both AES and RSA interfaces, and both AES (cipher) and RSA (x_cipher) session structures share the same memory area within a larger session structure. During session initialization, only one of these (either cipher or x_cipher) is initialized, while the other remains NULL. If a guest attempts to use an AES session with an RSA operation (or vice-versa), the code checks for a non-null session pointer but fails to verify the type of the session. This mismatch leads to an attempt to access the uninitialized (NULL) cipher or x_cipher pointer within the virtio_crypto_d_build_op function, causing a null pointer dereference.

1. Heap-based Buffer Overflow (CVE-2023-XXXX): This is the most critical finding, forming the foundation of the "Timekiller" exploit. It occurs during symmetric encryption/decryption operations. The virtio-crypto device allocates space for both plaintext (src) and ciphertext (dst) within the op_info structure. While the lengths of plaintext and ciphertext are typically equal in symmetric cryptography, virtio-crypto allows the guest to completely control both src_len and dst_len. If the guest specifies a src_len that is larger than dst_len, the resulting ciphertext, which has a length of src_len, will overflow the dst_len-sized buffer allocated for it. Crucially, the plaintext content is also fully controlled by the guest, meaning the attacker can precisely control the data written during the overflow. This heap-based buffer overflow, assigned a CVE ID (referred to as "a three number" in the talk, implying a formal assignment), provides a powerful primitive for subsequent exploitation.

Technical Deep Dive

▶ Watch: Detailed analysis of heap buffer overflow vulnerability (26:40)

The ultimate goal for the researchers was to leverage the discovered heap-based buffer overflow to achieve arbitrary code execution and a guest-to-host escape. This typically requires two key primitives: a malloc primitive for controlled memory allocation and an exploitable structure on the heap that can be overwritten. An exploitable structure usually contains data pointers, function pointers, or length members that, when corrupted, can alter program flow or expose sensitive information. Achieving this often demands sophisticated heap manipulation techniques, such as heap spraying or heap function spraying, to place the victim object adjacent to the overflow buffer in a predictable manner.

The researchers first reviewed existing QEMU/KVM escape techniques to inform their approach. They considered:

• An exploit leveraging a vulnerability in Virtual GPU that used an uninitialized buffer, but this had already been fixed by replacing malloc with calloc.
• A Blackhat USB device escape that involved an out-of-bounds (OOB) read and write within the USB device structure. This was straightforward for information leakage but offered limited heap manipulation capabilities.
• An exploit involving Network components by SLP that used the IP fragment mechanism to achieve an OOB write, leveraging the mbuf structure for arbitrary read/write. However, reproducing this exploit was noted as being quite difficult.

Finding these existing techniques unsuitable or impractical for their specific heap overflow, the team turned their focus back to the virtio-crypto device itself, surprisingly discovering that it offered ideal primitives for their exploit:

malloc Primitives within virtio-crypto:

• Symmetric encryption requests: When a guest initiates a symmetric encryption request, QEMU allocates an op_info object. The size of this object is largely controlled by guest-supplied length arguments, subject to minimal constraints. This provides a controlled malloc primitive.
• Asymmetric encryption requests: Even more powerfully, asymmetric encryption requests (e.g., RSA) involve helper functions that allocate source and destination buffers with virtually no size limitations from the guest, offering an extremely flexible malloc primitive.

Exploitable Structures within virtio-crypto:

• The virtio_crypto_sym_op_info and virtio_crypto_asym_op_info structures both contain crucial members such as data pointers and length fields. Overwriting a source_length field could transform an OOB write into an OOB read. Corrupting these pointers could directly lead to arbitrary address read and write capabilities.
• The virtio_crypto_request structure is particularly valuable. It contains a callback member and an opaque member, which are prime targets for control flow hijacking. Furthermore, its in member, a pointer to guest memory space, could be used to leak guest memory addresses. The callback itself, if its value is leaked, could reveal the QEMU image base address.

The "Timekiller" Technique: Asynchronous Clock Exploitation

A significant hurdle was the default synchronous nature of encryption processes in virtio-crypto. When an encryption request was made, QEMU would atomically handle the request, allocate structures, perform encryption, return results, and then free all allocated structures. This malloc-use-free cycle meant that only one instance of an exploitable structure would reside in memory at any given time, making sophisticated heap manipulation and reliable overwrites extremely difficult.

The "Timekiller" technique directly addresses this challenge by leveraging QEMU's asynchronous clock mechanism in conjunction with the throttle BPS (bytes per second) setting. The core idea is to introduce delays that allow multiple exploitable structures to coexist in memory simultaneously:

1. Introduce Delay: The attacker sends a large encryption request (containing a large amount of data to be processed).
2. Activate Throttling: By configuring throttle-bps, QEMU is forced to process this large request slowly, consuming a significant amount of time.
3. Queue Subsequent Requests: During this extended processing window, subsequent encryption requests are received by QEMU but are kept waiting in the timer queue.
4. Heap Manipulation Window: Crucially, the objects allocated for these waiting requests remain in memory for the duration of the delay. This creates an extended time window during which the attacker can perform precise heap manipulation and reliably overwrite targeted exploitable structures.

Information Leakage (ASLR Bypass)

The first step in a modern exploit chain is often to bypass ASLR (Address Space Layout Randomization) by leaking sensitive memory addresses. The researchers devised a method to turn their heap-based OOB write into an OOB read for this purpose:

1. Prepare Vulnerable Object: A vulnerable virtio_crypto_sym_op_info (sym_op_info) object is allocated on the heap. Its des (destination) member is made to point to the tail of the object itself, effectively creating an inline buffer.
2. Heap Layout: An virtio_crypto_asym_op_info (asym_op_info) object is then allocated immediately adjacent to the vulnerable sym_op_info. Its source buffer is allocated concurrently. A victim virtio_crypto_request structure is then placed next to the asym_op_info's source buffer.
3. Trigger OOB Write: The heap-based buffer overflow from the sym_op_info is triggered. This overflow overwrites the source_length member of the adjacent victim asym_op_info to a larger, controlled value.
4. OOB Read: When the asymmetric encryption process for the asym_op_info begins, its now-corrupted source_length causes its source buffer to read beyond its allocated boundaries, effectively overflowing into and encrypting the adjacent virtio_crypto_request members.
5. Decrypt and Leak: The resulting ciphertext is sent back to the guest. By decrypting this ciphertext, the attacker can recover the original values of the request members:

• The in member yields a guest memory address.
• The callback member reveals a QEMU image address (bypassing QEMU ASLR).
• The opaque member provides a heap address.

This successfully bypasses ASLR.

The heap manipulation for this leakage was meticulously orchestrated:

• An initial large encryption request creates the time window.
• Heap spray using asym_op_info D buffers prepares a 0x1C0 chunk and clears small bins (0x20, 0x70).
• The 0x1C0 chunk is freed, clearing 0x1A0 and 0x1B0 small bins, leaving a single 0x1C0 chunk.
• The vulnerable sym_op_info (size 0x1A0) is allocated, splitting the 0x1C0 bin and leaving a 0x20 small bin.
• The asym_op_info structure is allocated, taking the 0x20 small bin. Its source buffer takes 0x70 from the unsorted bin.
• Finally, the victim request is allocated from the large bin, ensuring it is adjacent to the asym_op_info's source buffer, completing the ideal heap layout for the OOB read.

Control Flow Hijack

With ASLR bypassed and arbitrary address information obtained, the next step is to hijack control flow to execute arbitrary code. The researchers demonstrated two primary methods:

Method 1: Overwriting qemu_timer_list

• Heap Layout: The heap is manipulated such that a victim sym_op_info is located next to other sym_op_info memory.
• Payload in Guest Memory: A payload is prepared in guest memory, consisting of a pointer to a malicious qemu_timer structure.
• OOB Write: The OOB write from the vulnerable sym_op_info is triggered. This overwrites the victim sym_op_info's source and destination pointers to point to the payload in guest memory and, critically, to the main_loop_tlg's qemu_timer_list pointer area within QEMU's memory.
• Hijack: When the victim symmetric encryption process completes, the pointer from guest memory (pointing to the malicious timer structure) is written into the qemu_timer_list pointer area, effectively hijacking the qemu_timer_list and allowing the attacker to control future timer-related callbacks.

Method 2: Overwriting request structure's callback and opaque

• Heap Layout: In this approach, the vulnerable sym_op_info object is positioned directly adjacent to a virtio_crypto_request structure.
• Payload in Guest Memory: The payload in guest memory consists of the address of system@PLT (Program Linkage Table entry for the system function) and the address of a command string (e.g., "calc.exe" on Windows or a shell command on Linux).
• OOB Write: The OOB write is used to overwrite the callback member of the adjacent request object with the system@PLT address and the opaque member with the address of the command string.
• Execute Code: When the victim request is subsequently handled, its callback function is invoked. Since callback now points to system@PLT and opaque holds the command string, the system(command_string) function is executed, leading to arbitrary code execution (e.g., popping a calculator on the host).
• Heap Manipulation: To ensure the sym_op_info and request structures are adjacent, heap spray is used to clear large bins of small sizes, forcing both objects to be allocated sequentially from the same large bin.

By combining these techniques, the researchers demonstrated a full guest-to-host escape from QEMU/KVM.

Demo / Proof of Concept

▶ Watch: Timekiller: leveraging async clock for heap manipulation (36:00)

The presentation included a live demonstration of the "Timekiller" exploit in action. After booting the guest VM, which took approximately one minute, the exploit successfully executed a command on the host. The visual proof of concept showed a calculator application (e.g., calc.exe on Windows) appearing on the host operating system, unequivocally demonstrating the successful guest-to-host escape and arbitrary code execution. This practical demonstration underscored the severity and effectiveness of their novel exploitation technique.

Defensive Implications

▶ Watch: Q&A: CFI protection in user space QEMU (51:10)

The "Timekiller" research provides crucial insights for both QEMU developers and cloud operators. The immediate defensive implication is the need for patching the identified vulnerabilities. The speakers indicated that patches were developed, specifically mentioning a check for src_len not being equal to dst_len before allocation in the heap-based buffer overflow, which would prevent the bug.

Beyond immediate fixes, the research highlights broader implications for secure coding and system design:

• Robust Input Validation: The null pointer dereferences stemmed from insufficient validation of guest-supplied type and length parameters, as well as inadequate checks for backend library availability and session type matching. Developers must implement comprehensive input validation and null pointer checks, especially in code paths handling guest input or managing asynchronous operations.
• Careful Memory Management in Asynchronous Contexts: The core of the "Timekiller" exploit lies in manipulating memory through asynchronous operations and throttles. Developers of virtualization devices and other security-critical components that utilize asynchronous clocks, timers, or throttling mechanisms must meticulously review their memory allocation, deallocation, and state management logic to prevent structures from remaining in memory longer than intended or in predictable locations.
• Continuous Fuzzing and Security Audits: The success of their modified wehle fuzzer in discovering these bugs underscores the importance of continuous, targeted fuzzing, particularly for new features and complex device models like virtio-crypto. Regular security audits focusing on concurrency, race conditions, and asynchronous programming patterns are also vital.
• Reinforced Isolation: While QEMU operates in user space, an escape still allows an attacker to compromise the host system. This research reinforces the need for strong isolation mechanisms, including sandboxing QEMU processes, implementing robust privilege separation, and potentially exploring technologies like Control Flow Integrity (CFI) in user space, although the speakers noted that CFI is not typically applied to QEMU's user-space execution.
• Awareness of New Attack Surfaces: The "Timekiller" technique itself serves as a warning: asynchronous mechanisms, often overlooked in favor of more direct data path vulnerabilities, can be a powerful new attack surface. Defenders should be aware of these subtle interaction points and their potential for exploitation.

Key Takeaways

• Asynchronous mechanisms and throttles in QEMU/KVM represent a novel and exploitable attack surface, offering unique opportunities for heap manipulation and overcoming traditional exploit challenges.
• The virtio-crypto device, despite its security-critical function, was found to contain several vulnerabilities, including multiple null pointer dereferences and a critical heap-based buffer overflow.
• The "Timekiller" technique leverages QEMU's asynchronous clock and throttling mechanism to introduce controlled delays, enabling precise heap manipulation and allowing multiple exploitable structures to coexist in memory simultaneously.
• A guest-controlled heap-based buffer overflow in virtio-crypto can be transformed into arbitrary read/write primitives, which are then used to bypass ASLR and ultimately lead to a full guest-to-host escape.
• Information leakage (ASLR bypass) and control flow hijacking can be achieved by carefully orchestrating heap layouts and overwriting specific pointers (e.g., qemu_timer_list pointers) or callback functions (e.g., request structure's callback and opaque members).
• Thorough security audits and continuous, targeted fuzzing are essential for complex virtualization platforms, especially when new features or asynchronous operational modes are introduced, to prevent critical vulnerabilities from being leveraged for guest-to-host escapes.

About the Speaker(s)

The "Timekiller" research was a collaborative effort by a team from Zhejiang University in China. The speakers presenting the talk included Yongkang Jia, Xiao Lei, and Chunming Wu. Chunming Wu, a key contributor to the research, mentioned that he had recently graduated and was transitioning to a role as a security researcher at Singular Security Lab. The team's expertise lies in uncovering and exploiting vulnerabilities within widely used virtualization frameworks such as QEMU/KVM, contributing significantly to the understanding of cloud security challenges.