Consistent Volume Group Snapshots, Unraveling the Magic - Leonardo Cecchi, EDB & Xing Yang

Consistent Volume Group Snapshots, Unraveling the Magic

Speakers: Leonardo Cecchi, Principal Software Engineer, EDB; Xing Yang, Co-chair of Kubernetes SIG Storage, VMware by Broadcom

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=urRefZ0KnU4

Overview

In the dynamic and often chaotic world of cloud-native infrastructure, ensuring robust data protection and disaster recovery for stateful applications remains a paramount concern. This talk, "Consistent Volume Group Snapshots, Unraveling the Magic," delivered by Xing Yang from VMware by Broadcom and Leonardo Cecchi from EDB, addresses a critical gap in Kubernetes' native storage capabilities: the ability to take crash-consistent snapshots of multiple persistent volumes simultaneously. While Kubernetes has long supported individual volume snapshots, the challenge of maintaining data integrity for applications that distribute their data across several volumes—such as databases with separate data and transaction log volumes—has persisted.

The speakers introduce the Volume Group Snapshot feature, a significant enhancement to the Kubernetes storage API that moved to Beta in Kubernetes 1.32. This innovation provides a unified mechanism to capture the state of multiple volumes at a single point in time, thereby guaranteeing write-order consistency. For application developers, site reliability engineers (SREs), and storage administrators managing mission-critical stateful workloads on Kubernetes, this feature is a game-changer. It simplifies backup and restore processes, drastically reduces recovery times, and enhances the overall resilience of complex applications, moving beyond the limitations of individual volume snapshots that can lead to data inconsistencies.

Background

▶ Watch: Speakers' introduction and talk agenda (0:00)

The journey towards comprehensive data protection in Kubernetes began with the introduction of the Volume Snapshot API, which reached General Availability (GA) in Kubernetes 1.20. This API provided a fundamental building block, allowing users to take crash-consistent snapshots of individual Persistent Volumes (PVs) and use them for data restoration. This was a crucial step, offering a basic level of protection against data loss for single-volume applications.

However, many sophisticated, mission-critical applications, particularly databases like PostgreSQL, do not confine their data to a single volume. They often utilize multiple volumes for distinct purposes—for instance, one volume for core data and another for transaction logs (Write-Ahead Logs, or WALs). When attempting to back up such an application using individual volume snapshots, a significant problem arises. If snapshots are taken sequentially (e.g., volume 1 at time T1, volume 2 at T2, volume 3 at T3), the resulting snapshots represent different points in time. Restoring from these time-divergent snapshots inevitably leads to data inconsistency, requiring lengthy and complex application-level recovery procedures, if possible at all.

To ensure application consistency, traditional backup strategies often require "quiescing" the application—temporarily pausing writes or putting it into a read-only state—before taking snapshots. While effective, quiescing is often a time-consuming and resource-intensive operation, making frequent backups impractical or too disruptive for production environments. The absence of a native Kubernetes mechanism to coordinate snapshots across multiple volumes simultaneously meant that users were either forced into expensive application quiescing, risked inconsistent backups, or relied on complex, ad-hoc, and often vendor-specific solutions outside the Kubernetes ecosystem. The Volume Group Snapshot feature emerges as the solution to this long-standing challenge, providing a native, consistent, and efficient way to protect multi-volume applications.

Key Findings

▶ Watch: Introducing consistent group snapshots and their benefits (4:05)

The core contribution of this talk is the detailed exposition of the Volume Group Snapshot feature, which fundamentally enhances Kubernetes' data protection capabilities for multi-volume applications. The key findings and contributions presented are:

• Introduction of Crash-Consistent Group Snapshots: The primary innovation is the ability to take a snapshot from multiple volumes at the exact same point in time, ensuring write-order consistency across all included volumes. This is paramount for applications like databases that require synchronized data states to avoid corruption or lengthy recovery processes.
• Enhanced Efficiency and Performance: By coordinating the snapshot process across multiple volumes simultaneously, the group snapshot mechanism offers improved efficiency and performance compared to taking individual snapshots one by one. This streamlines backup operations and reduces the overall impact on application performance.
• New Kubernetes APIs: The feature introduces three new Kubernetes API objects, providing a standardized and extensible framework for managing group snapshots:
• VolumeGroupSnapshotClass: An administrative object defining how group snapshots should be created, including the CSI driver to use and specific parameters.
• VolumeGroupSnapshot: A user-facing object representing a request to create a group snapshot for a set of volumes, typically identified by a label selector.
• VolumeGroupSnapshotContent: An object representing the physical group snapshot resource on the underlying storage system, managed by the snapshot controller.
• CSI Spec Integration: To enable heterogeneous storage systems to support this feature, the Container Storage Interface (CSI) specification has been extended. This includes a new Group Controller Service and three new Remote Procedure Calls (RPCs): CreateVolumeGroupSnapshot, DeleteVolumeGroupSnapshot, and GetVolumeGroupSnapshot, which reached GA in CSI spec 1.11.
• Simplified Restore Process: Despite the underlying complexity of group snapshot creation, the restore process is intentionally designed to be as straightforward as restoring from an individual volume snapshot. Users can reference the created Volume Snapshots (which are part of the group) as data sources for new Persistent Volume Claims (PVCs), ensuring ease of use during critical recovery scenarios.
• Beta Status in Kubernetes 1.32: The feature was introduced as an Alpha feature in Kubernetes 1.29 and has since progressed to Beta in Kubernetes 1.32, indicating a higher level of maturity and readiness for broader adoption, albeit with feature gates still enabled by default.

These findings collectively represent a significant leap forward in Kubernetes' ability to manage and protect stateful workloads, particularly those with complex storage requirements, by providing a native, consistent, and efficient group snapshot capability.

Technical Deep Dive

▶ Watch: Overview of new Kubernetes APIs for group snapshots (5:30)

The Volume Group Snapshot feature is a sophisticated orchestration of new Kubernetes API objects, extended CSI specifications, and collaborative controllers. Its implementation ensures that multiple volumes can be snapshotted in a crash-consistent manner.

New Kubernetes APIs

The feature introduces three new custom resources, managed by the Kubernetes API server:

1. VolumeGroupSnapshotClass:

• This administrative object is analogous to StorageClass or VolumeSnapshotClass.
• It defines the characteristics and parameters for creating VolumeGroupSnapshot objects.
• Key fields include:
• driver: The name of the CSI driver responsible for provisioning the group snapshot.
• parameters: A map of opaque key-value pairs passed directly to the CSI driver, allowing vendor-specific configurations.
• deletionPolicy: Specifies whether the physical group snapshot on the storage system should be Deleted or Retained when the VolumeGroupSnapshot object is removed from Kubernetes.

1. VolumeGroupSnapshot:

• This is the user's request to create a group snapshot.
• Its spec defines the source volumes to be snapshotted:
• selector: A label selector used to identify the Persistent Volume Claims (PVCs) that should be included in the group snapshot. All PVCs matching this selector and provisioned by the specified CSI driver will be part of the snapshot. This is the common method for dynamic provisioning.
• volumeGroupSnapshotContentName: Used for pre-provisioning, where an existing group snapshot on the storage system is imported into Kubernetes.
• The status field provides runtime information, including:
• volumeGroupSnapshotContentName: The bound VolumeGroupSnapshotContent object.
• creationTime: Timestamp of the snapshot.
• readyToUse: A boolean indicating if the snapshot is ready for restoration.

1. VolumeGroupSnapshotContent:

• This object represents the actual, physical group snapshot resource on the underlying storage system.
• It's typically created dynamically by the snapshot controller in response to a VolumeGroupSnapshot request, or manually by an administrator for pre-provisioned scenarios.
• Its spec includes:
• deletionPolicy, driver, volumeGroupSnapshotClassName: Similar to VolumeGroupSnapshotClass.
• source: Defines the source of the content, which can be a list of volumeHandles (for dynamic provisioning) or a groupSnapshotHandle along with individual snapshotHandles (for pre-provisioning).
• The status field contains the volumeGroupSnapshotHandle and a list of volumeHandle-snapshotHandle pairs, linking Kubernetes objects to their storage-system counterparts.

CSI Spec Integration

The ability for various storage vendors to support group snapshots is facilitated by extensions to the CSI specification, which reached GA in CSI spec 1.11. This includes:

• New Group Controller Service: A dedicated service within the CSI driver responsible for group snapshot operations.
• New RPCs:
• CreateVolumeGroupSnapshot: Instructs the CSI driver to create a group snapshot on the storage system for a given set of volume handles.
• DeleteVolumeGroupSnapshot: Requests the CSI driver to delete a specific group snapshot.
• GetVolumeGroupSnapshot: Retrieves the status of a group snapshot from the storage system.

Architecture and Workflow

The feature relies on several Kubernetes components working in concert:

1. Snapshot Controller: This core Kubernetes component monitors VolumeGroupSnapshot objects. For dynamic provisioning, it creates the corresponding VolumeGroupSnapshotContent object and handles the binding between the user's request and the physical snapshot resource. It also orchestrates the creation of individual VolumeSnapshot and VolumeSnapshotContent objects for each volume within the group.
2. CSI Snapshot Sidecar: Deployed alongside the CSI driver in the same pod, this sidecar watches VolumeGroupSnapshotContent objects. When a new content object is created or its status needs updating, the sidecar makes gRPC calls to the CSI driver's Group Controller Service to initiate or query the group snapshot operation on the storage system.
3. CSI Driver: The storage vendor's CSI driver is responsible for implementing the new Group Controller Service and its RPCs. It translates the Kubernetes group snapshot requests into vendor-specific API calls to the underlying storage array, which performs the actual, synchronized snapshot operation.

Dynamic Provisioning Workflow (The "Opera" Metaphor)

Leonardo Cecchi described the dynamic provisioning process using an "opera" metaphor with four acts and four personas (Kubernetes administrator, CSI driver, snapshot sidecar, snapshot controller):

• Act One: The Administrator's Request: A Kubernetes administrator (or an operator) defines a VolumeGroupSnapshotClass and creates PVCs with a common label (e.g., instance-name). This sets the stage for future group snapshots.
• Act Two: The Operator's Trigger and Controller's Magic:
• An application operator (e.g., a database operator) creates a VolumeGroupSnapshot object, specifying the volumeGroupSnapshotClassName and a selector matching the PVCs.
• The Snapshot Controller sees this new object, gathers details about the matching PVCs, and creates a VolumeGroupSnapshotContent object.
• The CSI Snapshot Sidecar detects the new VolumeGroupSnapshotContent and calls the CreateVolumeGroupSnapshot RPC on its adjacent CSI Driver.
• The CSI Driver interacts with the storage system to create the physical group snapshot, returning unique identifiers for the group snapshot and its constituent individual volume snapshots.
• The Snapshot Controller then creates individual VolumeSnapshot and VolumeSnapshotContent objects for each volume in the group, using the identifiers provided by the CSI driver. These individual snapshots are marked as members of the group.
• The CSI Snapshot Sidecar continuously polls the CSI Driver using GetVolumeGroupSnapshot until the snapshot is ready, updating the status of the VolumeGroupSnapshotContent and individual VolumeSnapshotContent objects.
• Act Three: The Restore: When a restore is needed, users simply create new PVCs, referencing the individual VolumeSnapshot objects (which are part of the group) in their dataSource field. This process is designed to be identical to restoring from a single volume snapshot, ensuring simplicity during high-stress recovery situations.
• Act Four: The Conclusion: The process is complete, with all components having collaborated to achieve a consistent group snapshot.

Database Consistency (PostgreSQL Example)

For databases like PostgreSQL, the group snapshot feature is particularly valuable. PostgreSQL uses a data store (where tables and indexes reside) and a transaction log (WAL). Changes are first written to the WAL, then periodically flushed to the data store during checkpoints.

• Traditional (Inconsistent) Snapshots: If individual snapshots are taken sequentially, the data store and WAL volumes will be inconsistent. Upon restore, PostgreSQL must perform a crash recovery: it finds the last consistent checkpoint before the earliest snapshot and then applies all subsequent WAL entries to bring the data store to a consistent state. This process can be lengthy and resource-intensive.
• Group Snapshots: With group snapshots, both the data store and WAL volumes are snapshotted at the exact same instant. To achieve a truly application-consistent snapshot, a database operator would still ideally trigger a pg_backup_start() command before the group snapshot. This command forces a checkpoint and prepares the database for a consistent state. Even without pg_backup_start(), the group snapshot provides crash consistency across all volumes. Upon restore, the database's crash recovery process is significantly faster because the data and WAL are already synchronized to the same point in time, minimizing the amount of WAL replay required. This makes group snapshots a highly desirable feature for database administrators.

Feature Gates and Troubleshooting

The feature is currently in Beta, meaning feature gates must be explicitly enabled in both the snapshot controller and the CSI snapshot sidecar. Additionally, users must ensure the CRDs for the new APIs are installed. Troubleshooting involves checking logs of all involved components (controller, sidecar, driver) to identify the point of failure in the orchestrated workflow. All PVCs to be snapshotted together must be provisioned by the same CSI driver, and that driver must be referenced in the VolumeGroupSnapshotClass.

Demo / Proof of Concept

▶ Watch: Architecture: Snapshot controller and CSI sidecar roles (9:35)

Leonardo Cecchi presented a live demonstration using the Cloud Native PostgreSQL (CNPG) operator to showcase the dynamic provisioning of a Volume Group Snapshot.

The demo began by showing a Kubernetes Cluster object managed by CNPG. This cluster was configured with a single PostgreSQL instance, but crucially, its data was split across multiple volumes: one for the primary data and another for the Write-Ahead Log (WAL). This setup perfectly illustrates the multi-volume scenario that Volume Group Snapshots are designed to address.

To initiate the backup, Leonardo executed a simple cnpg backup command. This command, rather than directly interacting with the snapshot APIs, triggered the CNPG operator. The operator, in turn, performed the necessary database-specific actions (like potentially calling pg_backup_start() to ensure application consistency, though the talk focused on the underlying snapshot mechanics) and then created a VolumeGroupSnapshot object in Kubernetes.

The output of the demo clearly showed the creation of several Kubernetes resources:

1. VolumeGroupSnapshot Object: This was the initial request from the CNPG operator. Leonardo highlighted its selector field, which used the instance-name label to identify the PostgreSQL data and WAL PVCs that needed to be snapshotted together. The volumeGroupSnapshotClass was also referenced here.
2. VolumeGroupSnapshotContent Object: This object, dynamically provisioned by the snapshot controller, represented the physical group snapshot on the storage system. Its definition included a volumeGroupSnapshotHandle and a list mapping the UIDs of the original volumes to the UIDs of their corresponding individual snapshots.
3. Individual VolumeSnapshot and VolumeSnapshotContent Objects: For each PVC (data and WAL), the system automatically created a VolumeSnapshot object and its corresponding VolumeSnapshotContent. These individual snapshots were clearly linked back to the parent VolumeGroupSnapshotContent, indicating their membership in the group. The demo showed that these objects were marked as readyToUse almost immediately, demonstrating the efficiency of the underlying storage system in creating the snapshots.

Leonardo emphasized that while CNPG adds its own annotations and logic, the underlying Volume Group Snapshot mechanism is generic and applicable to any multi-volume application. He also mentioned that future versions of CNPG would further simplify the restore process, allowing users to directly reference a VolumeGroupSnapshot object to recreate a cluster from a consistent point in time, eliminating the need to manage individual snapshots during restoration.

The demo visually confirmed the "magic" of dynamic provisioning, where a single VolumeGroupSnapshot request orchestrates the creation of all necessary Kubernetes objects and triggers the underlying storage system to perform a synchronized, crash-consistent backup across multiple volumes.

Defensive Implications

▶ Watch: Feature gate status for the beta API (10:10)

The introduction of the Volume Group Snapshot feature provides significant advantages for Kubernetes users tasked with protecting stateful applications. To leverage this capability effectively and enhance defensive postures, several implications and recommended actions should be considered:

1. Enable Feature Gates: Since Volume Group Snapshots are currently in Beta (as of Kubernetes 1.32), platform administrators must explicitly enable the corresponding feature gates in both the Kubernetes snapshot controller and the CSI snapshot sidecar. This is a prerequisite for the functionality to be available and operational within the cluster. Regular monitoring of Kubernetes release notes is essential for understanding when these gates might be enabled by default or removed upon GA.
2. CSI Driver Adoption: Storage vendors must update their CSI drivers to implement the Group Controller Service and its associated RPCs (CreateVolumeGroupSnapshot, DeleteVolumeGroupSnapshot, GetVolumeGroupSnapshot) from CSI spec 1.11. Defenders should inquire with their storage providers about their roadmap for supporting this feature. Without CSI driver adoption, the Kubernetes API objects alone cannot interact with the underlying storage system to perform group snapshots.
3. Application Operator Integration: Developers of Kubernetes application operators (especially for stateful applications like databases, message queues, or distributed file systems) should integrate with the Volume Group Snapshot API. This allows operators to orchestrate application-consistent backups by creating VolumeGroupSnapshot objects at appropriate times, potentially after quiescing the application or initiating database-specific backup commands (e.g., pg_backup_start() for PostgreSQL). This integration streamlines backup workflows and reduces operational complexity for end-users.
4. Simplified Disaster Recovery Plans: With consistent group snapshots, disaster recovery plans for multi-volume applications become significantly more reliable and faster. The ability to restore all related volumes to a single, consistent point in time drastically reduces the time and effort required for application-level recovery procedures, minimizing Recovery Time Objectives (RTOs). Defenders should update their DR playbooks to incorporate this new capability.
5. Granular Backup Policies: VolumeGroupSnapshotClass allows administrators to define different group snapshot policies (e.g., deletion policy, vendor-specific parameters) based on application needs or storage tiers. This enables more granular and efficient backup strategies for various workloads within the cluster.
6. Improved Data Integrity: The primary defensive benefit is the assurance of data integrity for multi-volume applications. By eliminating the risk of inconsistent data across related volumes during backup, organizations can have higher confidence in their ability to recover accurate and usable data after an incident, whether it's a hardware failure, accidental deletion, or cyber-attack.
7. Resource Planning: While group snapshots are more efficient than individual sequential snapshots, the actual performance (speed of creation and impact on storage system) depends heavily on the underlying storage implementation. Defenders should consult their storage vendor's documentation and perform testing to understand the performance characteristics for their specific environment and volume sizes, especially for large volumes.
8. Monitor and Alert: Implement monitoring for VolumeGroupSnapshot and VolumeGroupSnapshotContent objects to track their status, readiness, and any potential errors during creation or deletion. Robust alerting ensures that backup failures are detected promptly.

By strategically adopting and integrating the Volume Group Snapshot feature, organizations can significantly strengthen their data protection strategies for complex, stateful applications running on Kubernetes, leading to more resilient and recoverable systems.

Key Takeaways

• Addresses Multi-Volume Inconsistency: The Volume Group Snapshot feature in Kubernetes solves the critical problem of ensuring crash-consistent snapshots for applications that utilize multiple persistent volumes, preventing data corruption or lengthy recovery processes.
• Beta in Kubernetes 1.32: This capability was introduced as an Alpha feature in 1.29 and promoted to Beta in Kubernetes 1.32, indicating its increasing maturity and readiness for broader adoption, though feature gates must be explicitly enabled.
• New API Objects and CSI Extensions: The feature introduces three new Kubernetes APIs (VolumeGroupSnapshotClass, VolumeGroupSnapshot, VolumeGroupSnapshotContent) and extends the CSI specification (GA in 1.11) with a new Group Controller Service and RPCs, enabling storage vendor integration.
• Simplified Backup and Restore: It streamlines backup workflows, especially for stateful applications like databases (e.g., PostgreSQL), by capturing all related volumes simultaneously. The restore process is designed to be as simple as restoring from a single volume snapshot.
• Collaborative Ecosystem: Its functionality relies on the coordinated efforts of the Kubernetes snapshot controller, the CSI snapshot sidecar, and the underlying CSI driver, highlighting the importance of a robust cloud-native storage ecosystem.
• Call for Adoption: Widespread adoption requires storage vendors to implement the new CSI Group Controller Service and application operators to integrate the new Kubernetes APIs into their backup and recovery strategies.

About the Speaker(s)

Xing Yang is a prominent figure in the Kubernetes storage community, serving as a Co-chair of Kubernetes SIG Storage. She works at VMware by Broadcom, contributing to the advancement of storage solutions within the cloud-native ecosystem.

Leonardo Cecchi is a Principal Software Engineer at EDB (EnterpriseDB), a leading provider of PostgreSQL products and services. He is a long-time PostgreSQL user, a dedicated contributor to the Kubernetes CSI project, and a maintainer of the Cloud Native PostgreSQL (CNPG) project, an operator designed to manage PostgreSQL clusters on Kubernetes.