CacheWarp: Software-based Fault Injection using Selective State Reset

CacheWarp: Software-based Fault Injection using Selective State Reset

Speakers: Ruiyi Zhang, PhD student at CISA; Youheng Lü, Security Consultant at Schwarz; Andreas Kogler; Michael Schwarz

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=34qHV9fxO8s

Overview

In a groundbreaking presentation at USENIX Security '24, Ruiyi Zhang and Youheng Lü unveiled CacheWarp, a novel software-based fault injection attack that fundamentally compromises the integrity guarantees of AMD's Secure Encrypted Virtualization-Secure Nested Paging (SEV-SNP). This talk, a collaborative effort with Lucas Daniel Lans, Andy, and Michael, introduces the first attack capable of fully breaking the integrity of SEV-SNP, a trusted execution environment (TEE) designed to protect virtual machines (VMs) from malicious hypervisors. CacheWarp achieves its devastating effects through the precise abuse of a single, often overlooked, x86 instruction: INVD.

The core of CacheWarp lies in its ability to selectively erase modified data from the CPU cache without writing it back to main memory, effectively causing a software-controlled data loss. This unprecedented level of control over cache state enables attackers to inject faults at critical points in a program's execution, leading to powerful exploit primitives such as remote code execution (RCE) and privilege escalation. The research highlights a critical vulnerability in how privileged instructions interact with modern TEEs, challenging the long-held assumption that such instructions, when executed by a privileged entity, cannot be malicious within a protected environment.

The implications of CacheWarp are profound, particularly for cloud security models that rely heavily on hardware-based TEEs like SEV-SNP to isolate customer workloads from potentially compromised cloud infrastructure. By demonstrating a full-chain exploit against SEV-SNP VMs, the researchers have exposed a fundamental flaw that necessitates a re-evaluation of hardware-software interaction within TEEs. AMD has since addressed this vulnerability through firmware and microcode updates, underscoring the severity and practical impact of the CacheWarp discovery.

Background

▶ Watch: CacheWarp: First attack breaking SCV-SNP integrity (0:00)

To understand the intricacies of CacheWarp, it's essential to first grasp the foundational concepts of modern computing architecture, memory hierarchies, and the evolution of trusted execution environments. Modern CPUs operate at speeds far exceeding main memory access times, creating a significant performance bottleneck. To mitigate this, CPUs employ a multi-level cache hierarchy – smaller, faster buffers that store frequently accessed data and instructions closer to the processing cores. When the CPU needs data, it first checks the cache; if present (a cache hit), access is fast. If not (a cache miss), the data must be fetched from slower main memory.

Memory write operations are often buffered in the cache before being committed to main memory. This mechanism relies on cache coherence protocols to ensure consistency across multiple cores and memory levels. However, certain x86 instructions can manipulate the cache directly, sometimes in ways that bypass standard coherence mechanisms. One such instruction is INVD (Invalidate Cache), which clears all internal caches without writing back any modified data to memory. This means any "dirty" cache lines (data that has been modified in the cache but not yet written to main memory) are simply discarded, resulting in data loss. Due to this destructive behavior, both Intel and AMD explicitly warn against using INVD where memory coherence is critical, recommending WBINVD (Write Back and Invalidate Cache) instead, which ensures all modified data is written back before invalidation. Historically, INVD has been considered a benign threat vector because it is a privileged instruction, meaning only highly privileged software (like the operating system kernel or hypervisor) can execute it. The assumption was that privileged software is trusted and would not intentionally corrupt its own data or that of other processes.

This security model, however, was fundamentally challenged with the advent of Trusted Execution Environments (TEEs). TEEs, introduced by chip vendors since 2016, aim to enhance cloud security by creating isolated execution environments where code and data are protected even from a malicious or compromised hypervisor. AMD's Secure Encrypted Virtualization (SEV) and its subsequent iterations, SEV-ES (Encrypted State), SEV-SNP (Secure Nested Paging), are prominent examples. With SEV, each VM is assigned a unique encryption key, managed solely by hardware. The entire VM's memory is encrypted, preventing the hypervisor from directly reading or writing sensitive data. SEV-SNP, the latest iteration, specifically adds integrity protection to these confidentiality guarantees. It aims to ensure that even if a malicious hypervisor attempts to tamper with a VM's memory, the hardware will detect and prevent such modifications, thereby providing a strong assurance that the VM's code and data remain untainted. The threat model for SEV-SNP explicitly includes a malicious hypervisor, making its integrity guarantees paramount. This is precisely the security boundary that CacheWarp targets and ultimately shatters.

Key Findings

▶ Watch: TEEs (SCV-SNP) make INVD exploitable for integrity breaks (3:30)

CacheWarp's primary and most impactful finding is its status as the first attack to fully break the integrity guarantees of AMD SEV-SNP protected virtual machines. This achievement is significant because SEV-SNP was widely considered a robust solution for protecting VM workloads from a malicious hypervisor, offering both confidentiality and integrity. The attack demonstrates that even with hardware-enforced memory encryption and integrity checks, a sufficiently sophisticated software-based fault injection technique can bypass these protections.

The core mechanism enabling CacheWarp's success is the precise, software-controlled abuse of the INVD instruction. While INVD is a privileged instruction, the researchers found a method for a malicious hypervisor to execute it with fine-grained control over its timing and effect on a guest VM. By strategically invalidating cache lines without write-back, CacheWarp can selectively prevent data from being committed to main memory, effectively "dropping" memory writes. This capability transforms INVD from a dangerous but seemingly benign instruction (in the context of a trusted privileged entity) into a potent fault injection primitive.

The research further introduces a novel precise control framework that allows the attacker to orchestrate these cache invalidations with unprecedented accuracy. This framework leverages techniques such as page fault handling, APIC timer-based single-stepping, and careful use of WBINVD to ensure reliable fault injection at specific instruction boundaries or data writes. This precise control is crucial for turning a chaotic cache flush into a targeted attack.

Ultimately, CacheWarp demonstrates how this low-level cache manipulation can be escalated into high-impact security breaches. The researchers successfully developed full-chain exploits, achieving both remote code execution (RCE) and privilege escalation within an SEV-SNP protected guest VM. These exploits showcase the practical viability of CacheWarp, moving beyond theoretical concerns to concrete demonstrations of compromised system security. The findings led to AMD releasing firmware and microcode updates to mitigate the vulnerability, confirming its severity and the necessity of patching.

Technical Deep Dive

▶ Watch: Precise control with WBINVD for reliable single-stepping (5:40)

The technical foundation of CacheWarp hinges on exploiting the specific behavior of the INVD instruction within the context of an AMD SEV-SNP protected VM. As discussed, INVD clears the CPU's internal caches without writing back any modified ("dirty") data. In a normal system, this is catastrophic for data integrity, which is why it's a privileged instruction. The assumption has always been that a privileged entity would not intentionally cause such data loss to itself or its guests. However, the SEV-SNP threat model explicitly assumes a malicious hypervisor, which is a privileged entity. The critical insight of CacheWarp is that the hypervisor, despite being outside the TEE's trust boundary, can still execute INVD and affect the guest VM's execution.

The challenge for the attackers was not just executing INVD, but doing so with surgical precision to achieve targeted data corruption rather than a general system crash. To this end, the researchers developed a precise control framework consisting of several key components:

1. Page Granularity Code Pattern Identification: The first step involves gaining insight into the guest VM's execution flow. This is achieved by clearing the "present" bit of VM pages, triggering page faults when the guest attempts to access them. The hypervisor's page fault handler can then identify the code pattern at a page granularity, allowing it to deduce which part of the guest's code is being executed. This provides a coarse-grained understanding of the guest's state.

1. APIC Timer-based Single-Stepping: To achieve finer-grained control, the framework employs the APIC (Advanced Programmable Interrupt Controller) timer for single-stepping. The APIC timer can be configured to generate interrupts after a very short, precise duration, allowing the hypervisor to interrupt the guest VM's execution almost instruction-by-instruction. This enables the attacker to monitor and control the guest's progress at a micro-architectural level.

1. Reliable Single-Stepping with WBINVD: A significant hurdle for reliable single-stepping, especially in a VM-based TEE like SEV-SNP, is the impact of context switching on the CPU cache state. Every time the VM is interrupted and resumed, the cache state can change unpredictably. To address this, the CacheWarp framework utilizes the WBINVD instruction. Before each interrupt, WBINVD is executed, ensuring that all modified data in the cache is written back to memory and the cache is then cleared. This guarantees a clean cache state every time the VM is interrupted and resumed, leading to highly reliable and predictable single-stepping. This also has the secondary benefit of ensuring that unrelated data is less likely to be dropped accidentally by a subsequent INVD instruction.

1. Selective Write-Back using Cache-Set Specific PPMAN: Even with reliable single-stepping, a blanket INVD can still be too destructive. The attackers need to ensure that only the targeted data is dropped, while other critical data remains intact. To achieve this, CacheWarp employs a technique similar to Prime+Probe or Flush+Reload attacks, but adapted for selective write-back, referred to as PPMAN (likely a variant for "Prime, Probe, Manipulate, and Attack"). By selectively performing cache operations on specific cache sets, the attacker can force certain values in those sets to be written back to memory before the INVD instruction is executed. This ensures that essential data is preserved, while the targeted data, residing in other cache sets or specifically manipulated to remain "dirty," is vulnerable to INVD.

With this precise control framework, the hypervisor can monitor the guest VM, identify specific instruction points where critical data is about to be written to memory (e.g., a return address on the stack, a flag variable), and then execute INVD at the exact moment the data is in the cache but not yet written to main memory. This effectively causes the intended memory write to be lost, leading to a controlled fault injection.

Demo / Proof of Concept

▶ Watch: TimeWarp: Generic control flow hijacking by reusing return addresses (8:00)

The researchers presented two compelling proof-of-concept exploits to demonstrate the practical capabilities of CacheWarp, showcasing its potential for both privilege escalation and control flow hijacking.

The first exploit targeted a common scenario: bypassing a password check in the sudo utility. In this simplified example, sudo might have a variable initialized to zero, and if it remains zero, the user gains root privileges without a password check. The exploit works as follows:

1. The sudo program initializes a critical variable to zero.
2. Later, a legitimate write operation (e.g., to store the result of a password check) would update this variable.
3. Using the CacheWarp precise control framework, the malicious hypervisor monitors the guest VM's execution.
4. At the exact moment the program attempts to write a non-zero value to this critical variable (after a failed password check, for instance), the hypervisor executes the INVD instruction.
5. This INVD instruction invalidates the cache line containing the variable without writing the new, non-zero value back to memory.
6. Consequently, the variable retains its initial zero value in main memory.
7. The sudo program proceeds, finds the variable still at zero, and grants root privileges, effectively bypassing the password check. This simple demonstration highlights the power of dropping a single, critical memory write.

The second, more generic, and sophisticated exploit is dubbed TimeWarp. TimeWarp focuses on control flow hijacking rather than direct data manipulation. The core idea is to invalidate the write operation of a return address onto the stack, causing the program to reuse a stale or previously written return address. This effectively allows the attacker to "warp back in time" to an earlier point in the execution flow or to an entirely different, attacker-controlled location.

The presentation provided a toy example to illustrate TimeWarp:

• Imagine a program where if (A == B) { win(); }.
• Normally, function Red1 sets A to 10, and function Red2 sets B to 30. Thus, A != B, and the win() function is not called.
• With TimeWarp, the attacker intervenes when Red2 is called.
• Specifically, when Red2 attempts to write its legitimate return address onto the stack, the CacheWarp INVD instruction is triggered.
• This prevents Red2's actual return address from being committed to the stack.
• Instead, the program might reuse the stale return address that was previously pushed by Red1 (or another earlier function).
• By manipulating which return address is reused, the attacker can cause the program to execute a different path. In the animated example, the goal was to make A and B equal. If Red2's return address is invalidated, and a stale value is used, it can lead to a scenario where both A and B end up being 30. When the program resumes, A and B are now equal, triggering the win() function. The speakers state that this "effectively injects a new line of code," referring to the altered execution path.

A real-world application of TimeWarp was demonstrated against a simplified SSH password login mechanism.

• In a typical SSH login, the user's input password is hashed (hash(user_input)) and compared against the stored hash of the actual password (hash(real_password)). If they match, login is successful.
• The TimeWarp attack targets a scenario where these comparisons might be performed in separate function calls or stages.
• By triggering INVD when a critical return address is written during the second function call (e.g., the function responsible for hashing the user's input or comparing it), the attacker can manipulate the control flow.
• The outcome of this manipulation is that the comparison hash(user_input) == hash(real_password) is effectively bypassed or forced to evaluate as true. For instance, the attack might cause the system to compare hash(user_input) with hash(user_input) itself, leading to a guaranteed match and successful login, granting remote code execution within the guest VM.

These demonstrations clearly illustrate how CacheWarp, by selectively dropping memory writes, can transition from a low-level hardware vulnerability to high-level system compromise, achieving both privilege escalation and remote code execution within an SEV-SNP protected environment.

Defensive Implications

▶ Watch: Animation: TimeWarp demonstrating control flow manipulation (9:00)

The CacheWarp attack exposed a critical vulnerability in the security model of AMD's SEV-SNP, necessitating immediate and robust defensive measures. The primary implication is that hardware-based trusted execution environments, even those designed with strong integrity guarantees, are not immune to sophisticated software-based fault injection techniques that leverage low-level privileged instructions.

The most direct and immediate defensive action was taken by AMD, who acknowledged the vulnerability and released firmware and microcode updates to mitigate CacheWarp. This highlights the critical role of hardware vendors in maintaining the security of TEEs. Such updates typically involve modifying the CPU's internal logic or microcode to change how the INVD instruction behaves in the context of an SEV-SNP guest VM, or to introduce additional integrity checks that detect attempts by the hypervisor to manipulate cache states in a malicious manner.

For cloud providers and users relying on SEV-SNP for VM isolation and data protection, the CacheWarp vulnerability underscores the importance of promptly applying all available firmware and microcode updates. Failure to do so leaves their protected VMs susceptible to a malicious hypervisor. It also serves as a reminder that while TEEs significantly raise the bar for attackers, they are not a silver bullet and require continuous scrutiny and patching.

More broadly, CacheWarp forces a re-evaluation of how privileged instructions are handled within the TEE security model. The traditional assumption that privileged instructions are benign because only trusted software can execute them no longer holds when the privileged software (the hypervisor) is part of the attacker's threat model. Hardware architects designing future TEEs must carefully scrutinize every privileged instruction, considering its potential for fault injection or side-channel leakage, even when executed by an untrusted privileged entity. This may involve:

• Restricting INVD behavior: Modifying INVD to behave like WBINVD when an SEV-SNP guest is active, ensuring that all dirty cache lines are written back before invalidation, thus preserving data integrity.
• Enhanced Integrity Checks: Implementing more granular hardware-level integrity checks that can detect unauthorized cache manipulations or unexpected memory state changes that result from fault injection.
• Secure Microcode Updates: Ensuring that microcode updates are securely delivered and applied, as they are a critical vector for patching such low-level hardware vulnerabilities.

For software developers building applications intended for TEEs, CacheWarp is a stark reminder that even with hardware protection, software should ideally employ robust defensive programming practices. While TEEs aim to protect against external tampering, vulnerabilities like CacheWarp show that the boundary can be breached. Developers should still strive for resilient code, although the primary defense against CacheWarp lies at the hardware/firmware level.

In conclusion, CacheWarp has demonstrated a potent new class of attack against TEEs, emphasizing that the interaction between low-level CPU instructions and hardware security features requires continuous vigilance. The defense against such attacks will continue to be a collaborative effort between hardware vendors, cloud providers, and the security research community.

Key Takeaways

• CacheWarp is the first attack to fully break AMD SEV-SNP integrity: It demonstrates that even with hardware-enforced memory encryption and integrity, a malicious hypervisor can compromise guest VM integrity.
• Abuse of the INVD instruction is the core primitive: The attack leverages the privileged INVD instruction to selectively invalidate cache lines without writing back dirty data, leading to controlled data loss.
• Precise control framework enables targeted fault injection: CacheWarp employs techniques like page fault handling, APIC timer-based single-stepping, and WBINVD for reliable state control, allowing INVD to be triggered at critical moments.
• Leads to full-chain exploits: RCE and Privilege Escalation: The researchers demonstrated practical exploits, including bypassing sudo password checks and hijacking control flow in an SSH login scenario (TimeWarp), achieving remote code execution.
• Highlights TEE vulnerability to privileged instruction abuse: The attack exposes a fundamental flaw in the assumption that privileged instructions executed by a hypervisor are benign, challenging the security model of modern TEEs.
• Mitigated by firmware and microcode updates: AMD has released patches for the vulnerability, emphasizing the critical need for users and cloud providers to apply these updates promptly.

About the Speaker(s)

Ruiyi Zhang is a PhD student at CISA. His research focuses on low-level system security and hardware vulnerabilities, particularly within the context of trusted execution environments. He was a primary presenter of the CacheWarp research at USENIX Security '24.

Youheng Lü is a Security Consultant at Schwarz. His expertise lies in identifying and exploiting security vulnerabilities in complex systems, with a particular emphasis on hardware-software interfaces and cloud security. He collaborated on the CacheWarp project and co-presented the findings.