Exploiting Blind Memory Corruption in Cloud Services

Exploiting Blind Memory Corruption in Cloud Services

Speakers: Anthony Weems (Staff Security Engineer, Google Cloud Vulnerability Research); Stefan Schiller (Security Engineer, Google Cloud Vulnerability Research); Simon Scannell (Vulnerability Researcher, Google)

Conference: CODE BLUE

YouTube: https://www.youtube.com/watch?v=MLNA0Y8aRY8

Overview

This talk, "Exploiting Blind Memory Corruption in Cloud Services," delivered by Simon Scannell, Anthony Weems, and Stefan Schiller from the Google Cloud Vulnerability Research team, addresses a critical yet often underestimated area of cloud security: memory corruption. While traditionally associated with on-premise software and client-side attacks, memory corruption is frequently considered a theoretical risk in modern cloud environments, which heavily leverage memory-safe languages and sophisticated infrastructure. The speakers challenge this perception, arguing that memory corruption remains a practical and viable threat in cloud backends, despite the unique challenges it presents.

The core objective of the presentation is to demystify why successful memory corruption chains are rarely reported in cloud services and to outline the specialized techniques required to overcome the inherent difficulties of exploiting such vulnerabilities in a cloud context. Through a detailed, end-to-end case study within Google Cloud Platform (GCP), the team demonstrates that remote code execution (RCE) via memory corruption is not only feasible but achievable with patience and advanced exploitation methodologies. This research highlights the need for a shift in perspective, urging security professionals to recognize memory corruption as a tangible risk that demands sophisticated defensive and offensive strategies within cloud ecosystems.

Background

▶ Watch: Introduction: Memory corruption in cloud services (0:00)

The prevailing view often separates memory corruption from cloud security, largely due to the widespread adoption of memory-safe languages like Go, Python, Rust, and JavaScript in cloud service development. Furthermore, the operational characteristics of cloud environments—such as unknown binaries, heavy load balancing, and the inability to replicate production setups—complicate traditional memory corruption exploitation workflows.

In a classical server-side exploitation scenario, attackers typically have access to the target binary, allowing them to reverse engineer, fuzz, or analyze source code to find vulnerabilities. An information leak (info leak) can reveal memory pointers, which are then used to craft a buffer overflow payload for remote code execution (RCE). Crucially, an exploit working on one server usually works on all servers running the same version.

Cloud environments fundamentally alter this dynamic:

• Unknown Binaries: Cloud services run on the provider's backend, meaning attackers rarely have access to the exact binary or library versions. This renders leaked memory pointers largely meaningless without knowledge of the binary's layout.
• Load Balancing: Cloud services are massively distributed. An info leak request might hit "Worker A," but a subsequent exploit request could be routed to "Worker B," leading to a crash on Worker B without achieving RCE, and potentially triggering detection mechanisms due to service instability. Brute-forcing is often impractical with hundreds or thousands of workers.
• Memory-Safe Languages: While many cloud services are written in memory-safe languages, they frequently rely on underlying C/C++ libraries (e.g., image processing libraries, database engines like SQLite) where memory corruption issues can still arise.

To overcome the unknown binary problem, the speakers proposed several strategies:

• Arbitrary File Read: If an attacker can achieve an arbitrary file read vulnerability, they can leak binaries directly from the server, then analyze them to understand memory layouts and gadget locations. Previous work by the Google VRP team, for instance, involved using a JPEG 2000 vulnerability to leak binaries.
• Arbitrary Memory Read: Similarly, an arbitrary memory read primitive can dump binaries directly from memory. CVE-2016-5771, a PHP memory corruption vulnerability, served as an example where researchers dumped the PHP binary to prepare a second-stage exploit.
• Fingerprinting Common Docker Images: Attackers can infer or fingerprint common base images used by cloud providers. Google VRP hunters successfully used this to leak libc.so from Google production services in Gemini's Python execution sandbox, which provided crucial information for crafting ROP chains.

Addressing the load balancing problem requires more specialized techniques:

• Single-Shot Exploits: If the entire exploit chain can be delivered in a single, self-sufficient payload (e.g., dynamic execution in JavaScript), load balancing is circumvented. However, most common libraries (like image parsers) don't support dynamic execution.
• Conditional Corruption: A more pragmatic approach involves finding a specialized bug that allows the exploit to "bail out" gracefully if it lands on the wrong worker or if ASLR offsets are incorrect. This prevents crashes and allows repeated attempts until the correct worker and ASLR offset are hit. The exploit payload effectively checks: "Am I on the right worker with the correct ASLR?" If yes, proceed; if no, exit cleanly without crashing. This expands the range of exploitable vulnerabilities beyond those that support dynamic execution.

These concepts underscore the need for a new playbook for memory corruption in the cloud, one that acknowledges the unique environment and leverages specialized vulnerabilities to achieve reliable exploitation.

Key Findings

▶ Watch: Solving Challenges: Leaking binaries & conditional corruption (6:00)

The talk's central finding is the successful demonstration of an end-to-end memory corruption exploit chain against Google Cloud's Artifact Analysis service, achieving remote code execution on a production worker. This chain elegantly combines multiple vulnerabilities and specialized techniques to navigate the complexities of a cloud environment, including blind exploitation and load balancer evasion.

The key discoveries and contributions are:

1. Path Traversal to Arbitrary File Write in Excalibur: A vulnerability was identified in the Excalibur library, used by Artifact Analysis to unpack container images. Specifically, a flaw in how the Go standard library function path.Clean handles relative paths at the beginning of an input, combined with path.Join, allowed an attacker to write arbitrary files to disk outside the intended temporary directory.
2. Conditional Corruption Primitive for ASLR Bypass and Load Balancer Evasion: Leveraging the arbitrary file write, the researchers developed a novel technique to create an ASLR oracle. By attempting to write to specific paths within /proc/self/map_files (a Linux procfs directory containing symlinks to memory mappings), the exploit could determine if a guessed ASLR offset for libc.so was correct. If the guess was wrong, the write would fail to a read-only file, causing the service to gracefully exit processing the malicious input without crashing. If the guess was correct, the file would exist, causing the service to skip the write and proceed with the exploit. This allowed for reliable ASLR bypass and ensured the full memory corruption exploit only triggered on the targeted worker.
3. Heap Buffer Overflow in SQLite's concat_ws Function: A previously unpatched integer overflow vulnerability was discovered in SQLite version 3.47.2 (the version used by Artifact Analysis) within the concat_ws function. This bug, which had no assigned CVE but was noted in a later change log, resulted in an incorrect memory allocation size due to a 32-bit integer multiplication overflow. Subsequent memcpy operations then wrote data significantly past the end of the allocated buffer, leading to a heap buffer overflow.
4. Exploiting Go Heap Predictability for Instruction Pointer Control: To exploit the SQLite buffer overflow for RCE, the team leveraged the predictable memory layout of the Go heap. Unlike the native C heap, which is randomized by ASLR, the Go heap's base address often remains consistent. This allowed the attackers to spray a fake sqlite3_module structure onto the Go heap at a known address range. The SQLite overflow was then used to overwrite a pointer within an FTS3 table structure (a suitable target object) to point to this fake module on the Go heap. Upon cleanup, a function pointer within the fake module (xDisconnect) was called, granting instruction pointer control and allowing the execution of a ROP chain.
5. End-to-End RCE in GCP: By chaining these vulnerabilities and techniques, the researchers achieved full remote code execution on a production Google Cloud Artifact Analysis worker, demonstrating that memory corruption is a practical and exploitable risk in even highly secured cloud environments.

Technical Deep Dive

▶ Watch: Path Traversal: ASLR bypass oracle via /proc/self/map_files (18:00)

The exploit chain developed by the Google Cloud Vulnerability Research team targets Artifact Analysis, a critical Google Cloud service responsible for scanning user-submitted container images for vulnerabilities. This service utilizes an open-source library called Excalibur (Software Composition Analysis library) to parse various package metadata formats, including RPM. A crucial detail is that the RPM parser within Excalibur relies on SQLite, which in turn uses a C implementation (typically a single large sqlite.c file) for its core functionality. This setup perfectly aligns with the "rule of two" for identifying promising targets: code processing untrusted inputs (user container images) that is written in a memory-safe language (Go) but incorporates memory-unsafe C code (SQLite), all operating without a sandbox.

Path Traversal in Excalibur (Arbitrary File Write)

The initial entry point for the exploit leverages how Excalibur unpacks container images. Container images are essentially layered tar files, and the tar format is permissive of relative paths and symlinks. Excalibur extracts these layers to a local disk using Go code that iterates through tar entries. For each entry, the header.name (which is attacker-controlled) is passed to path.Clean, a Go standard library function designed to sanitize paths, before being joined with a static temporary directory and written to disk.

The vulnerability lies in the specific behavior of path.Clean. While it removes . and .. elements, it does so lexically and does not fully resolve paths if .. elements appear at the very beginning without a preceding parent directory. For example, if an attacker provides ../../x as header.name, path.Clean alone does not modify it because there's no "parent" to remove. When this "cleaned" path is then concatenated with the temporary directory (e.g., /tmp/layer) using path.Join("/tmp/layer", "../../x"), the full path becomes /tmp/x. This effectively bypasses the intended sanitization, granting an arbitrary file write primitive outside the designated temporary directory.

While an arbitrary file write is powerful, directly achieving RCE on a Google service is challenging due to minimal, often read-only file systems and the fact that most services run as large, monolithic Go binaries with no simple executables to overwrite. However, this primitive proved invaluable for the next stage of the exploit.

Conditional Corruption Primitive (ASLR Bypass and Load Balancer Evasion)

To overcome ASLR randomization and the load balancing challenge, the arbitrary file write was weaponized into a conditional corruption primitive. This technique relies on the Linux /proc/self/map_files directory. This directory contains symlinks for every memory mapping of the current process, pointing to the actual executable file mapped at that specific virtual address. For example, /proc/self/map_files/<libc_base_address> would be a symlink to libc.so.

The exploit constructs a malicious tar archive where one entry attempts to write to a path like ../../../../proc/self/map_files/<guessed_libc_address>. The logic within Excalibur's unpacker has two critical behaviors:

1. Existence Check: Before writing, it checks if the target file already exists.
2. Error Handling: If the write operation fails (e.g., due to read-only permissions), the function returns, and further processing of the tar archive is halted.

This creates an ASLR oracle:

• Incorrect ASLR Guess: If the attacker's guess for <libc_base_address> is wrong, the symlink in /proc/self/map_files will not exist. The unpacker will then attempt to write to /proc/self/map_files/<wrong_address>. Since /proc/self/map_files is read-only, this write will fail, causing the unpacker function to return. Crucially, the malicious SQLite database (the next stage of the exploit) will not be processed, and the service will not crash.
• Correct ASLR Guess: If the attacker's guess for <libc_base_address> is correct, the symlink will exist. The unpacker's existence check will return true, causing it to skip the write operation entirely. The function will then continue processing the rest of the tar archive, including the malicious SQLite database.

By iterating through potential libc.so ASLR offsets and uploading multiple container images, the attackers could reliably determine the correct ASLR base address for a specific worker. Once the correct offset was found, a subsequent image containing the full exploit payload would only trigger the memory corruption on the worker with that specific ASLR layout, effectively bypassing load balancing and preventing widespread service crashes.

Heap Buffer Overflow in SQLite's concat_ws Function

With ASLR bypassed and load balancing handled, the focus shifted to finding a memory corruption vulnerability. The team examined the SQLite version used by Artifact Analysis (3.47.2), noting that it was slightly behind the latest version (3.49.1). A review of the change log for SQLite revealed an intriguing entry: "Fix a bug in the concat WS function that could lead to a memory error, specifically a right past the end of allocated space." This bug had no CVE and no explicit security implications, yet it immediately signaled a potential heap buffer overflow.

The concat_ws function in SQLite concatenates multiple strings using a specified separator. The vulnerability lies in the calculation of the total required memory size for the resulting string. The relevant line of code involved arcC - 1 nSap, where arcC is the number of strings and nSap is the length of the separator. Both arcC and nSap are 32-bit integers (int). The multiplication arcC - 1 nSap is performed using 32-bit arithmetic before the result is added to a 64-bit integer (n) that holds the total size.

If arcC - 1 and nSap are sufficiently large (e.g., 18 strings and a separator length of 0x10000000 or 256 MB), their product (e.g., 17 * 0x10000000) will overflow a 32-bit integer. The most significant bits will be discarded, leading to a much smaller, incorrect result (e.g., 0x10000000). This smaller, incorrect size is then passed to malloc, allocating only 256 MB instead of the required 4GB+. Subsequent memcpy calls, which copy the actual strings and separators into this undersized buffer, will then write significantly out-of-bounds, overwriting whatever data follows the allocated chunk.

Initially, this overflow caused an immediate segmentation fault because it typically wrote into unmapped memory. To prevent this, heap grooming was employed. By carefully crafting SQL SELECT queries with UNION SELECT statements, the attackers could make multiple large allocations. Due to the behavior of TCMalloc (the memory allocator used), these allocations were not always contiguous and could be interspersed with read-only segments. However, by precisely controlling the sizes of individual allocations to align them, a continuous block of over 4GB of mapped memory could be prepared on the heap. These allocations were then freed, and a second SQL query was used to trigger the overflow. TCMalloc's block-wise repurposing of freed chunks allowed the attackers to ensure that the undersized allocation for concat_ws was placed immediately before the groomed, continuous 4GB block, allowing the overflow to write successfully into mapped, writable memory without crashing.

Exploiting for RCE (Fake sqlite3_module struct)

To transform the arbitrary write into RCE, a suitable target object on the heap was needed. This object had to be allocatable via a SELECT query, have a controllable size to fit within the groomed heap, and contain a function pointer that could be overwritten. The FTS3 table structure (Full Text Search) was identified as an ideal candidate. Referencing an FTS3 table in a SELECT statement allocates this structure, and its size can be controlled by the table's name. Crucially, the FTS3 structure contains a pointer, pModule, which references a sqlite3_module struct. This sqlite3_module struct, in turn, holds various function pointers, including xDisconnect, which is called during cleanup after a SQL query.

The goal was to overwrite the pModule pointer within the FTS3 table structure to point to a fake sqlite3_module structure controlled by the attacker. This fake structure would contain a malicious address in place of the xDisconnect function pointer. When xDisconnect is called during cleanup, the instruction pointer would be hijacked.

The final challenge was to place this fake sqlite3_module structure at a known address, as the native C heap is randomized by ASLR. The breakthrough came from the observation that the Go heap, where the Artifact Analysis service itself runs, is often located at a predictable, non-randomized base address. While Go is memory-safe, its heap's consistency provides a unique opportunity when exploiting native C libraries it uses. The attackers could spray many copies of their fake sqlite3_module structure onto the Go heap, ensuring a high probability that the pModule overwrite would land within one of these controlled regions.

Once pModule was overwritten to point into the Go heap spray, and the xDisconnect function pointer was controlled, the instruction pointer was successfully hijacked. From this point, a ROP chain (included within the container image) was executed, leading to the deployment of a C2 implant and full remote code execution.

Demo / Proof of Concept

▶ Watch: SQLite Vulnerability: concat_ws memory corruption bug (24:00)

The talk concluded with a compelling live demonstration of the end-to-end exploit chain. The demo showcased the practical viability of the techniques discussed:

1. C2 Client Initialization: The demonstration began by initiating a Command and Control (C2) client, set up to listen for incoming connections from the C2 implant that would be deployed on the compromised Google Cloud worker.
2. Targeted ASLR Exploitation: The attackers had previously used the conditional corruption primitive to identify the libc.so base address of a specific worker in the Artifact Analysis fleet. For the demo, they then uploaded approximately 30 malicious container images. Each image was crafted such that the conditional corruption logic would only allow the full SQLite exploit to proceed if it landed on the worker with the pre-determined libc.so base address. This illustrated the efficiency of load balancer evasion and targeted exploitation.
3. Successful Back Connection: After a short wait, one of the uploaded images successfully hit the target worker, and the C2 implant executed. The C2 client instantly received a back connection, confirming the compromise.
4. Remote Code Execution: To prove full control, the attackers issued the hostname command through their C2 client. The command was executed on the compromised production Google Cloud worker, and its output was displayed in the C2 client, unequivocally demonstrating successful remote code execution on a sensitive cloud service.

This demonstration powerfully validated the talk's premise: memory corruption, even in blind cloud environments, is a practical and exploitable risk when armed with sophisticated techniques.

Defensive Implications

▶ Watch: Live Demo: Remote code execution on production system (35:00)

This research carries significant defensive implications for both cloud providers and users, fundamentally challenging the notion that memory corruption is a theoretical risk in cloud services.

For Cloud Providers:

• Rethink Memory Safety Assumptions: Cloud providers must acknowledge that services written in memory-safe languages are still vulnerable if they interface with memory-unsafe (C/C++) libraries. This necessitates rigorous security reviews of all third-party dependencies, especially those handling untrusted input.
• Aggressive Patching and Updates: Even minor bugs in C/C++ libraries, like the SQLite integer overflow without an assigned CVE, can be critical primitives in an exploit chain. Providers should prioritize updating to the latest stable versions of all third-party libraries and actively monitor change logs for any memory-related fixes, regardless of explicit security advisories.
• Enhanced Sandboxing: The absence of a sandbox for Excalibur's processing of user-submitted container images was a critical factor. Implementing robust sandboxing (e.g., gVisor, namespaces, seccomp-bpf) around any component that processes untrusted data, especially those using native code, is paramount to contain potential breaches.
• Input Validation and Path Sanitization: The path.Clean vulnerability highlights the subtlety required for path sanitization. Developers must understand the nuances of standard library functions and ensure they are robust against all forms of relative path traversal, particularly when combining or joining paths.
• Randomize Go Heap: While Go's heap predictability aided the attackers, cloud providers should explore methods to randomize Go heap addresses, if technically feasible without significant performance impact, to remove this exploitation advantage.
• Sophisticated Crash Monitoring: While conditional corruption aims to prevent crashes, robust monitoring for any anomalous behavior, even non-crashing errors or unusual resource consumption, is crucial. Additionally, analyzing crash dumps for root causes, even in seemingly unrelated crashes, can reveal underlying memory corruption issues.
• Supply Chain Security: A deep understanding of the software supply chain, including all open-source libraries and their versions, is vital. This enables proactive vulnerability hunting and patching before exploitation.

For Cloud Users:

• Understand Shared Responsibility: Users should recognize that while cloud providers handle infrastructure security, vulnerabilities can still exist in the underlying services. While direct mitigation for this specific vulnerability was a provider's responsibility, it underscores the need for vigilance.
• Secure Container Practices: Implement robust practices for building and deploying container images, including:
• Minimal Base Images: Use minimal, hardened base images to reduce the attack surface.
• Regular Scanning: Employ container image scanning tools (like Artifact Analysis itself, once patched) to identify known vulnerabilities in your application dependencies.
• Least Privilege: Configure container runtimes with the principle of least privilege, limiting their access to host resources.
• Monitor Service Advisories: Stay informed about security advisories and patches from your cloud provider regarding managed services you use.
• Consider Data Sensitivity: For highly sensitive workloads, understand the security posture of all underlying cloud services and consider additional layers of protection, such as confidential computing, if available and appropriate.

This research serves as a stark reminder that security in the cloud is a complex, multi-layered challenge requiring continuous vigilance and adaptation from all stakeholders.

Key Takeaways

• Memory corruption is a practical and viable risk in cloud services, not merely a theoretical concern, even when services are primarily written in memory-safe languages.
• Cloud environments present unique exploitation challenges such as unknown binaries and pervasive load balancing, but these can be overcome with specialized techniques and a deep understanding of the underlying infrastructure.
• Conditional corruption is a powerful primitive for bypassing ASLR and evading load balancers, allowing targeted exploitation without crashing services and alerting defenders.
• Seemingly minor bugs without CVEs, like the integer overflow in SQLite's concat_ws function, can be critical components of an end-to-end RCE chain when combined with other vulnerabilities.
• Leveraging predictable memory layouts (e.g., the Go heap) in otherwise randomized environments can be crucial for achieving reliable instruction pointer control and remote code execution.
• Thorough security analysis of all third-party C/C++ libraries used by cloud services, even those written in memory-safe languages, is essential, along with robust sandboxing for components processing untrusted input.

About the Speaker(s)

The speakers for this talk are Simon Scannell, Anthony Weems, and Stefan Schiller. They are all members of the Google Cloud Vulnerability Research team. Their work focuses on proactively identifying and exploiting vulnerabilities within Google Cloud's infrastructure and services. By engaging in this internal vulnerability research, they aim to gain a deeper understanding of the limitations, challenges, and possibilities of exploiting cloud environments, ultimately contributing to the enhanced security of Google Cloud for all users.