Forensic Examination of Ceph

Florian Bausch (Incident Responder and Pentester · ERNW Research)

TROOPERS23 · Day 1 · Attack & Research · Attack & Research

Overview

In this insightful talk from TROOPERS, Florian Bausch, an Incident Responder and Pentester at ERNW Research, presented his extensive work on the forensic examination of Ceph, a prominent software-defined storage solution. Drawing from his master's thesis, Bausch delved into the intricacies of Ceph's on-disk data structures, particularly focusing on its BlueStore backend. The presentation highlighted the critical need for forensic understanding of such complex distributed storage systems, which are increasingly deployed in cloud environments, OpenStack, and enterprise setups like SAP HANA.

Watch on YouTube · Slides

Visual summary for Forensic Examination of Ceph by Florian Bausch
Visual summary for Forensic Examination of Ceph by Florian Bausch

Key moments

  1. 0:00 Introduction to Ceph and its storage types
  2. 3:55 Evolution of Ceph's OSD storage: FileStore to BlueStore
  3. 17:45 Decoding object metadata from RocksDB key-value store
  4. 24:00 Reconstructing CephFS files and directory structures
  5. 28:00 Capabilities of the 'Vampire' Ceph forensics tool
  6. 37:00 Live demonstration of the Vampire forensics tool
  7. 46:30 Q&A: Ceph users, encryption, and tool use cases

Forensic Examination of Ceph

Speakers: Florian Bausch, Incident Responder and Pentester, ERNW Research

Conference: TROOPERS

YouTube: https://www.youtube.com/watch?v=UFBqaoVU8mU

Overview

In this insightful talk from TROOPERS, Florian Bausch, an Incident Responder and Pentester at ERNW Research, presented his extensive work on the forensic examination of Ceph, a prominent software-defined storage solution. Drawing from his master's thesis, Bausch delved into the intricacies of Ceph's on-disk data structures, particularly focusing on its BlueStore backend. The presentation highlighted the critical need for forensic understanding of such complex distributed storage systems, which are increasingly deployed in cloud environments, OpenStack, and enterprise setups like SAP HANA.

The core challenge Bausch addressed was the significant gap in forensic tools and methodologies for analyzing Ceph's internal data organization. Prior to his work, there was limited public documentation or tooling to systematically recover data, reconstruct file systems, or analyze metadata from Ceph OSDs (Object Storage Devices). His research not only reverse-engineered the underlying data formats but also resulted in the creation of Vampire, an open-source Python tool designed to facilitate these complex forensic investigations. This talk is essential for incident responders, digital forensic analysts, and security professionals who need to understand how data is stored, retrieved, and potentially recovered from compromised or failed Ceph clusters.

Background

▶ Watch: Introduction to Ceph and its storage types (0:00)

Ceph is a highly scalable, open-source, software-defined storage platform that provides object, block, and file system interfaces from a single distributed computer cluster. At its heart lies RADOS (Reliable Autonomic Distributed Object Store), which stores all data as objects. These objects are managed across numerous physical storage devices known as OSDs (Object Storage Devices). Clients can interact with Ceph via several interfaces: CephFS (Ceph File System), offering a POSIX-compatible file system mount; RBD (RADOS Block Device), presenting block devices to clients; and RADOS Gateway, providing S3/Swift-compatible object storage via a REST API. Florian Bausch's research primarily focused on CephFS and RBD due to their rich metadata potential for forensic analysis.

A pivotal change in Ceph's architecture, which motivated Bausch's research, occurred around 2017. Prior to this, Ceph OSDs used FileStore, an implementation that relied on a traditional XFS file system to store objects as individual files, with metadata tracked by a LevelDB key-value store. However, FileStore encountered performance bottlenecks, particularly with XFS's journaling mechanism. To address these issues, Ceph introduced BlueStore, a new, more performant backend where the Ceph daemon directly manages raw OSDs. BlueStore includes a minimalistic internal file system called BlueFS, whose sole purpose is to store the RocksDB key-value store, which in turn holds all object metadata. The actual object content, however, is written directly to the raw device by the Ceph daemon, bypassing traditional file system overheads. This shift meant that existing file system forensic tools were rendered ineffective for BlueStore, creating a significant void that Bausch aimed to fill.

To systematically categorize the data he uncovered, Bausch adopted the well-established forensic framework introduced by Brian Carrier. This framework divides data into five categories:

  1. File System Category: Data describing the file system itself (e.g., type, size, attributes).
  2. Content Category: The actual data stored in files or objects, and metadata tracking allocation.
  3. Metadata Category: Metadata specific to files or objects (e.g., timestamps, permissions, owners).
  4. File Name Category: Maps internal file system IDs (like inodes) to human-readable names.
  5. Application Category: Additional data for specific applications (e.g., quotas, journals).

A crucial aspect of Ceph's operation is its CRUSH (Controlled, Scalable, Decentralized Placement of Replicated Data) algorithm. This algorithm deterministically distributes data objects across OSDs to ensure even load balancing and data redundancy. Consequently, any single OSD will only contain a subset of the total data stored in the cluster, typically based on a configured replication factor (e.g., three copies of each object). This distribution poses a challenge for forensic analysis, as a complete picture often requires access to multiple, if not all, OSDs.

Key Findings

▶ Watch: Decoding object metadata from RocksDB key-value store (17:45)

Florian Bausch's master's thesis and subsequent tool development yielded several critical findings regarding the forensic analysis of Ceph's BlueStore. His work effectively demystified the complex internal workings of BlueStore, providing a blueprint for future forensic investigations.

Firstly, Bausch successfully reverse-engineered the on-disk format of Ceph BlueStore. This involved meticulous analysis of source code and raw disk images, leading to a comprehensive understanding of how data blocks are laid out and interconnected. He applied Brian Carrier's five categories to every piece of data discovered, providing a structured and systematic way to understand and document BlueStore's forensic artifacts.

A central discovery was the critical role of BlueFS and RocksDB. BlueFS, though minimal, is indispensable as it houses the RocksDB key-value store, which in turn contains nearly all the essential metadata about Ceph objects, including their location on the OSD. Bausch developed a methodology to reconstruct BlueFS, enabling the extraction and subsequent analysis of the RocksDB database. This was a foundational step, as interpreting RocksDB's schema was key to unlocking object metadata.

Within RocksDB, Bausch identified specific key prefixes that delineate different types of metadata. For instance, keys starting with O denote primary object metadata, containing details like internal object IDs (onodes), sizes, extended attributes, and crucial extent maps that point to the object's physical data blocks on the OSD. Keys starting with M provide additional object metadata, vital for interpreting CephFS structures like directory contents and file permissions. He also found s keys for OSD-level metadata and b/B keys for block allocation bitmaps.

Furthermore, his research demonstrated the feasibility of reconstructing both RBDs and CephFS file systems from individual OSDs. For RBDs, he found that objects are named systematically (e.g., RBD_data_<ID>_<BlockID>), allowing for ordered reconstruction of block device content. For CephFS, he elucidated how large files are split into multiple objects, and how directory structures and file paths can be inferred using extended attributes and specific metadata entries within RocksDB.

Another significant finding was the ability to extract OSD map data. These maps provide crucial topology information about the entire Ceph cluster, including the number of OSDs, their IDs (e.g., whoami0), and historical events such as OSD downtime or uptime. This allows an analyst examining a single OSD to gain insights into the broader cluster context.

Finally, Bausch's work highlighted the persistence of data in slack space and unallocated areas on OSDs. Even after objects are deleted or overwritten, remnants of previous data can often be recovered from these regions, underscoring the importance of secure erasure practices for Ceph deployments. The ability of his tool to carve for these artifacts further enhances forensic capabilities.

Technical Deep Dive

▶ Watch: Reconstructing CephFS files and directory structures (24:00)

The forensic analysis of Ceph BlueStore, as detailed by Florian Bausch, hinges on a deep understanding of its layered on-disk structure and the interpretation of its internal metadata. The OSD's basic structure begins with a BlueStore superblock, which holds fundamental information about the OSD and its belonging Ceph cluster. Following this are a BlueFS superblock and an OSD superblock, which provide pointers and configuration details for BlueFS and the OSD itself, respectively. The majority of the drive is then allocated for storing actual Ceph objects, interspersed with allocated areas for BlueFS.

The first critical step in forensic reconstruction is analyzing BlueFS. BlueFS is a very basic file system, primarily designed to store the RocksDB key-value store. It can only create first-level directories and uses a transaction log (a journal) to track all file system operations (creation, renaming, moving, resizing). The BlueFS superblock contains pointers to the initial part of this transaction log. By sequentially reading and interpreting each transaction entry, an analyst can reconstruct the current state of BlueFS, identifying all its files, their extents (physical locations on the OSD), and their sizes. This process is crucial because RocksDB, residing within BlueFS, is the gateway to all object metadata.

Once RocksDB files are extracted from BlueFS, the ldb debugging tool (a common utility for LevelDB/RocksDB) can be used to dump their contents. The dumped data consists of key-value pairs, where the key's prefix is highly indicative of the value's content:

  • s prefix: Contains OSD metadata, such as the OSD's creation time and its unique ID within the cluster (e.g., whoami0).
  • o prefix (uppercase O): Contains basic object metadata. A key like O<object_name>o points to a value containing an onode data structure. This onode includes the object's internal ID (oid), its size, extended attributes, flags, and the beginning of its extent map. The extent map is a list of physical disk regions where the object's data is stored. If an object is large and its extent map exceeds the space in this entry, additional entries with keys ending in X (e.g., O<object_name>o<counter>x) store further extent map segments.
  • m prefix (uppercase M): Contains additional object metadata, often specific to the object's context (e.g., CephFS). A key like M<object_ID><type> points to an encoded data structure that, depending on its type, can reveal information such as directory contents or file permissions within CephFS.
  • b (lowercase b) and B (uppercase B) prefixes: Relate to bitmaps that track allocated and unallocated blocks on the OSD. The b keys contain the bitmap data itself, while B keys provide metadata for interpreting these bitmaps correctly.

RBD reconstruction is relatively straightforward due to its systematic object naming. When an RBD is created and data is written, Ceph generates objects with names like RBD_data_<RBD_ID>_<BlockID>. The <RBD_ID> uniquely identifies the block device, and <BlockID> represents the sector count, allowing for easy ordering and reconstruction of the block device's content. However, RBDs employ lazy allocation, meaning objects are only created when data is actually written to a specific block. Therefore, a missing RBD_data object could mean the block was never written to or that it resides on another OSD. Additional metadata, such as the RBD_header_<RBD_ID> object, provides crucial information like the RBD's creation time and features (e.g., locking status).

CephFS reconstruction is more complex as it doesn't have a one-to-one mapping between files and objects. A single large file (e.g., 20 MB) might be split into multiple 4 MB objects, all linked to the same inode ID. Reconstructing a file requires identifying all objects belonging to its inode and concatenating their contents. To reconstruct directory paths, CephFS uses an extended attribute called parent, which lists the inode IDs of parent directories. Directory contents and file permissions are often found in specific M entries within RocksDB, sometimes associated with objects named _head within the directory's metadata. The CephFS journal (object ID 600) also contains vital information about recent file system operations, though Bausch noted this was not fully explored in his initial work.

Beyond file and block device content, Bausch's research revealed the presence of OSD map objects. These objects contain topology information about the entire Ceph cluster, allowing an analyst to determine the total number of OSDs and identify periods of OSD downtime or uptime, even from a single OSD. The tool can also identify and extract data from slack spaces (unused portions of allocated blocks) and unallocated areas of the OSD, potentially revealing remnants of previously stored or deleted data.

Demo / Proof of Concept

▶ Watch: Live demonstration of the Vampire forensics tool (37:00)

Florian Bausch demonstrated his open-source Python tool, Vampire, named after the "vampire squid" (Vampyroteuthis infernalis), a nod to Ceph's cephalopod namesake. Vampire is written in Python 3 and notably does not rely on any official Ceph libraries, ensuring it can operate independently of Ceph's internal logic and potentially expose more raw data.

The demonstration showcased Vampire's core functionalities:

  1. BlueStore Label Information: Using the bluestore-label flag, Vampire can extract fundamental metadata from the OSD. This includes the BlueStore's initial creation timestamp, the OSD's size (e.g., 100 GB), its version (e.g., "version 26 of the OSD data"), and critically, its unique identifier within the cluster (e.g., whoami0). This whoami ID is invaluable for inferring the minimum size of the cluster (e.g., whoami100 implies at least 100 OSDs existed). It also displays information about the RocksDB database and potential slack space beyond the OSD data.
  1. BlueFS Superblock and File Listing: Vampire can parse the BlueFS superblock, revealing details about the internal file system where RocksDB resides. This includes the transaction log's size and location, and a list of files within BlueFS, complete with their metadata. This step is crucial for then extracting the RocksDB database.
  1. Object Listing (ls-object): The ls-object flag provides a comprehensive list of all objects stored on the analyzed OSD. This output distinguishes between various object types:
  • Numbered objects: Represent files and directories within CephFS. The demo showed objects related to inode 15, consisting of five separate chunks, demonstrating how a single file is fragmented across multiple objects.
  • OSD map objects: Contain the cluster topology information.
  • MDS snap table objects: Related to the CephFS metadata service.
  • RBD data objects: Identified by RBD_data_<ID>_<BlockID>, showing the first few sectors of an RBD (e.g., 11 objects for a specific RBD ID). Bausch explained that comparing the number of observed RBD objects with the RBD's theoretical size (obtained from other metadata) helps determine if the full RBD content is available on the analyzed OSD.
  1. Data Extraction and Reconstruction: Vampire can extract all found objects to a specified directory, organizing them into a logical structure. For CephFS, the tool generates symbolic links to reconstruct the directory hierarchy. The demo illustrated navigating this reconstructed file system, identifying directories like "test_files" and "images", and eventually locating a JPEG image file. Bausch then demonstrated opening a reconstructed image file, emphasizing that for larger files, the tool automatically concatenates the multiple object chunks into a single, usable file.

The validation of Vampire's capabilities was performed using both a controlled three-VM test setup with small OSDs and real-world data from SAP HANA installations on Ceph clusters. This dual approach ensured that the tool worked effectively with both known states and large, complex datasets, confirming its practical utility.

Defensive Implications

▶ Watch: Q&A: Ceph users, encryption, and tool use cases (46:30)

The forensic insights provided by Florian Bausch's research into Ceph BlueStore carry significant implications for defenders and system administrators managing Ceph environments. Understanding how data persists and is organized on OSDs is crucial for implementing robust security measures and preparing for incident response.

  1. Secure Erasure of OSDs: The most critical defensive implication is the absolute necessity of securely wiping OSD drives before repurposing or decommissioning them. Simply deleting objects or even logical volumes (if OSDs are wrapped in LVM, as Bausch mentioned is common) is insufficient. Data remnants can persist in unallocated areas and slack spaces, making them recoverable by tools like Vampire. Defenders should employ cryptographic erasure or multi-pass overwriting tools (dd if=/dev/zero, shred) directly on the raw OSD devices to prevent data leakage.
  1. Understanding Data Distribution: Defenders must be aware that a single OSD does not contain all cluster data. While this distributed nature enhances resilience, it means that a single compromised or leaked OSD can still reveal significant metadata (e.g., cluster topology, OSD IDs, creation times, partial file contents, RBD headers) and a subset of actual user data. The CRUSH algorithm's deterministic nature means that if an attacker knows object names, they might infer where other related data could be stored.
  1. Forensic Readiness and Incident Response: Organizations operating Ceph clusters should incorporate Bausch's findings into their incident response plans. Knowing the internal structure of BlueStore, the role of BlueFS and RocksDB, and the types of metadata available allows IR teams to:
  • Identify relevant forensic artifacts more quickly.
  • Estimate the scope of a data breach or compromise by analyzing OSD maps and object metadata.
  • Potentially recover data from failed OSDs or partially corrupted clusters.
  • Validate whether data has been properly removed or overwritten.
  1. Encryption at Rest: While Bausch did not deeply explore Ceph encryption, he acknowledged its possibility, often implemented via underlying technologies like LVM encryption. For defenders, full disk encryption or object-level encryption within Ceph is a paramount control. If OSDs are encrypted, forensic tools like Vampire would struggle to recover data without the decryption keys, significantly mitigating the risk of data exposure from leaked drives. This highlights a critical area for further defensive research and implementation.
  1. Monitoring and Auditing: The ability to extract OSD map information, including OSD uptime/downtime, provides a valuable auditing capability. Defenders can use this to correlate unusual cluster events with potential security incidents or unauthorized changes. Monitoring the creation and modification times of RBDs and CephFS entities can also provide an audit trail for specific data operations.
  1. CephFS Journal Analysis: Bausch noted that he did not deeply analyze the CephFS journal. This implies a potential blind spot for defenders, as the journal could contain critical forensic artifacts regarding recent file system activity that might be missed if not properly parsed. Further research or development in this area could enhance the completeness of CephFS forensic analysis.

In essence, Bausch's work transforms Ceph from a "black box" into a forensically transparent system, empowering defenders to better secure their deployments and respond effectively to security incidents.

Key Takeaways

  • Ceph BlueStore's Complexity Demystified: Florian Bausch's research provides the first comprehensive forensic analysis of Ceph's BlueStore backend, detailing its on-disk data structures and how they replace traditional file systems for performance.
  • BlueFS and RocksDB are Metadata Hubs: The minimalistic BlueFS, hosting the RocksDB key-value store, is central to BlueStore's metadata management. Reconstructing BlueFS and interpreting RocksDB keys (e.g., O, M, s) are essential for understanding object properties, extents, and cluster topology.
  • Vampire Tool Enables Forensic Analysis: The open-source Python tool, Vampire, allows for the extraction of BlueStore metadata, reconstruction of BlueFS, interpretation of RocksDB, and reassembly of RBDs and CephFS files/directories from raw OSD images.
  • Data Distribution Impacts Scope: Due to the CRUSH algorithm and lazy allocation, a single OSD contains only a subset of the cluster's data. Forensic investigations often require access to multiple OSDs to fully reconstruct files or block devices.
  • Secure Erasure is Critical: Data persists in slack space and unallocated areas on OSDs even after deletion. Thorough, secure wiping of OSD drives is imperative to prevent data leakage.
  • OSD Maps Reveal Cluster State: OSD map objects contain vital cluster topology information, including the number of OSDs and historical uptime/downtime events, which can be invaluable for incident response.

About the Speaker(s)

Florian Bausch is an Incident Responder and Pentester at ERNW Research. He has been working in the field of incident response and analysis for over four years. His presentation at TROOPERS was based on his master's thesis, which he completed about five years prior at FAU Erlangen. His academic background in computer forensics, combined with his practical experience, drove his in-depth exploration of Ceph's forensic landscape, culminating in the development of the Vampire tool.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Bausch did the miserable, grinding work of reverse-engineering an undocumented storage backend so the rest of us don't have to. Ripping apart BlueStore and building a standalone parser from scratch without leaning on Ceph's own libraries is exactly what deep forensic research should look like.

Heather Calloway (CISO) — SOLID

This is deep forensic research on Ceph storage, but the executive takeaway is simple: standard data deletion on these systems leaves data intact, and most IR firms lack the tooling to investigate it. It's a highly credible technical talk that exposes a gap in enterprise hardware decommissioning and post-breach visibility.

→ Top-rated talks at TROOPERS23

All talks from TROOPERS23